Welcome back, aspiring cyberwarriors!

Many software-as-a-service (SaaS) platforms are adding AI chatbots to their products. But these bots do more than just chat; they can access internal tools and perform tasks for users. Each of these actions creates a potential security risk, and developers are releasing these features faster than they can secure them.

In this article, I want to share a type of vulnerability I found while testing an AI chatbot – LLM02:2025 Sensitive Information Disclosure – which allowed me to obtain users’ first and last names and email addresses just by interacting with the chatbot. Let’s get rolling!

What is LLM02:2025 Sensitive Information Disclosure?

LLM02:2025 Sensitive Information Disclosure is a critical security issue listed in the OWASP Top 10 for LLM Applications 2025. This problem occurs when a Large Language Model (LLM) application accidentally reveals confidential or personal information in its responses.

This vulnerability happens because LLMs are designed to be helpful and use all available context, such as training data, system prompts, and runtime inputs, to create replies. If sensitive information is included in these sources without proper protections, the model may accidentally disclose information it should keep private.

Step 1: Find the Hidden Feature Behind the Feature

Our target had a chat feature that looked, on its face, completely trivial: type an arithmetic expression wrapped in double curly braces, and the bot evaluates it and replies with the answer.

{{5-5+0}} = 0
{{5*5+0}} = 25

Just like a calculator. But the interesting bugs live in the features nobody thought to test twice. So I started varying the input systematically, the way you’d fuzz any parameter in a pentest. And the calculator started talking back with things that were most definitely not numbers:

{{5-0+0}} → collectionPropertyOption://CollectionName
{{1-0+0}} → https://domain/p/CollectionName
{{2-0+0}} → user://UUID

This shows that the “calculator” is really a thin cover for a much riskier part of the code, which allows direct access to the platform’s internal system without any authentication.

Step 2: Isolate the Real Trigger

When you notice a leak like this, do not just note it down. Understand what is causing it.

My first guess was that the final answer of the expression is used as a lookup index. This is easy to test. Try expressions that all result in the same number through different math operations:

{{2+2}}    = 4        (plain number)
{{2+2+0}}  = 4        (plain number)
{{2-2+0}}  = user://UUID    ← NOT a plain number!

That last result contradicted my guess. The expression 2-2+0 simplifies to 0, but the result came back as an object reference instead of a plain number, similar to when the expression equals 2. This was a clue. I ran a few similar tests to confirm:

{{4-2}}    (=2)  → collectionProperty:// reference (type 4, not type 2)
{{5-2+0}}  (=3)  → collectionPropertyOption:// reference (type 5, not type 3)
{{0+2-2}}  (=0)  → plain number (type 0)

In conclusion, the lookup key is based on the first operand’s type, not the final arithmetic result. The developer’s evaluator takes the first number from the input and puts it directly into an internal array, sorted by object type, completely separate from what the expression calculates. I confirmed this by testing the limits:

{{99-2+0}}              = 97 (plain math, outside the enum)  
{{99999999999-0+0}}     = 99999999999 (same, safely out of range)

Step 3: Turn the Reference Into a Record

The next question is obvious: will the system actually resolve that reference if I hand it back to it?

Yes. It will.

{{user://<uuid>}} → returns first name, last name, and email address of that user.

This AI chatbot successfully provided information about any user by knowing his UUID.

Summary

Finding an issue like this, such as index leak, reference resolution, and PII disclosure, is not just luck; it is a systematic approach. This is the practical skill we teach at Hackers-Arise. As AI systems become part of every platform you will test, hackers who understand these new areas of attack will have the best tools and the highest salaries. Check out our Subscriber training package to start building these skills effectively and methodically.

The post Hacking: How a “Calculator” Feature Can Leak Internal Object References (and Real User PII) in AI Chatbots first appeared on Hackers Arise.