The AI phone assistant told us it had updated a customer account.
It hadn't.
During an AI agent pentest, we called the assistant twice using the same account number but two different names. Each time, it appeared to find an account, but the details changed depending on which name we gave it.
We then asked it to correct information on the account. The assistant confidently confirmed that the change had been made.
Nothing changed in the backend.
In this case, the affected data was a billing record. But the same failure mode becomes much more serious in systems handling things like prescription refills, payment authorisations, address changes, or other consequential actions: the agent reports success when the underlying system has done nothing.
Note: the scenario and details in this post have been reframed around a utility/telco account assistant rather than the original engagement, and identifying specifics have been changed. The underlying behaviour described is unchanged from what we observed.
How These Assistants Usually Work
Assistants like this are usually built in one of a few ways. Sometimes the model itself decides when it needs to look something up and calls a tool or API directly. Other times, the model works out what the caller is asking for, and separate application code decides which lookup or API to call.
Some systems may also use retrieval to provide additional context to the model. For example, a RAG pipeline might search a knowledge base or other indexed data and return relevant information for the model to reason over. That is different from an exact account lookup, where the application would normally query a structured backend using an identifier such as an account number or customer ID.
In many implementations, the model does not usually reach into the backend itself. Something else sits in between.
One version of that lookup might look something like this:
{
"type": "tool_use",
"name": "lookup_account",
"input": {
"account_number": "4021998",
"name": "..."
}
}
That kind of separation makes sense. The application handles access to the underlying systems, while the model handles the conversation.
From the phone interface, we could not see exactly what the backend returned. What we could see was that the assistant was willing to present contradictory information as though it were authoritative.
Testing the Same Account Twice
On the first call we supplied an account number and one name. The assistant gave us an account manager, billing information and an open service ticket.
On a second call, we used the same account number with a different name. This time, the details were completely different. It gave us another account manager and different billing information, as though we were looking at someone else's account.
We could not tell from the outside exactly how the lookup was implemented, so we cannot say whether this came from partial matching, application logic, or the model filling in missing details. What we could establish was that the same account number produced two incompatible account records.
We saw the same drift within a single call. We asked who the account manager was three times using slightly different wording. The first answer gave us a name, the second said no manager was listed, and the third returned to the original name.
Nothing in the conversation had changed that should have affected the answer.
Confirming a Change That Never Happened
Later in the same call, we told the assistant that the date of birth on the account was wrong and gave it a replacement.
It told us the date of birth had been updated. We checked separately afterwards and found that no change had been made.
We repeated the test with a mobile number and got the same result. The assistant confirmed the number had been added, but it had not.
Getting a name wrong is one thing. Telling a caller that an account change has been made when nothing actually changed is much more serious.
If an update fails, the assistant should say so. It should not tell the caller, "that's been updated for you," when the backend says otherwise.
The Instruction Was There, It Just Didn't Hold
In a separate finding, which is not covered in this post, we were able to extract the system prompt. One of the instructions in that prompt was, in effect, not to invent details that had not been provided.
Despite that instruction, the assistant still produced incorrect information during normal conversation. We were not using a jailbreak or unusual prompt syntax. We simply provided account details, asked questions and requested changes.
The bigger issue is that a prompt should not be relied on to determine whether an account update succeeded. The assistant should report the actual result returned by the backend.
That flow should look something like this:
Why This Matters
There was no classic unauthorised access finding here. We did not prove that one customer's real data was exposed to another.
The issue was that the assistant could not always be relied on to describe the state of the system behind it accurately.
That becomes more serious when the assistant is confirming actions. If a customer changes a phone number, address, payment instruction or cancellation request, they may assume the change has gone through when it has not.
This is not just a hypothetical concern. In Moffatt v. Air Canada, 2024 BCCRT 149, a Canadian tribunal held Air Canada responsible after its website chatbot gave a customer incorrect information about a bereavement fare policy and the customer relied on it when booking a flight.
Air Canada argued that it should not be responsible for information provided by the chatbot. The tribunal rejected that argument, finding that the chatbot was still part of Air Canada’s website and that the airline was responsible for the information it provided.
The circumstances were different from what we found here. That case involved incorrect policy information rather than a false confirmation that a backend change had completed. The broader point is still useful though: once a company puts an AI assistant in front of customers, the answers it gives are part of the service the customer is relying on.
OWASP's 2025 LLM Top 10 covered this broader class of problem under LLM09: Misinformation, where generated output can appear credible despite being wrong.
How We Tested It
The only interface available to us was a phone number, so every test had to work as something a real caller could say out loud.
That ruled out a lot of techniques that depend on exact formatting, special characters or structured text. Instead, we focused on simple conversational tests: repeating questions, changing one detail at a time, comparing separate calls and independently checking whether claimed account changes had actually happened.
Because some responses could contain sensitive information, transcription of the assistant's side of the call was done locally.
We will cover the voice testing setup and how we made repeated testing more efficient in a separate post.
What to Test if You're Building One of These
- Use the same account details across separate sessions and check for consistency.
- Change one identifying field and see whether unrelated account details change with it.
- Ask the same factual question several times using different wording.
- Try details that should not resolve to a real account and see whether the assistant still produces plausible account information.
- Ask it to make a change, then verify the backend independently.
- Test failed updates and make sure the assistant reports them as failures.
Final Thoughts
A lot of AI security testing focuses on whether an assistant can be tricked into revealing information it should not have access to. It is also worth checking whether you can trust what it tells you about the system behind it.
Nothing about this required a clever technique. Normal conversation, asked a couple of different ways, was enough to get conflicting account details and confirmations for changes that were never actually made.
Worth checking whether yours would do the same.
Further Reading: