Testing AI Agents in Production: What Changes When the Model Can Call Your Tools

You shipped an agent. It reads from a database, calls two internal APIs, files tickets, and sends email on a user's behalf. Your last app pentest predates all of it, and the question sitting in your next scoping call is what testing this thing actually involves and what you should be asking for.

The change is easy to state. Talk a chatbot into a bad answer and you get a bad answer. Talk an agent into a bad action and something in a system you own is now different. The model and the prompt handling can be identical in both cases. Agents usually bring more with them than the ability to act, but that ability is the critical difference, and it is the one that changes what a test has to prove.

That is also where the exploitable part usually sits: in what your app permits the model to do. The model weights are rarely the interesting piece.

This has a practical consequence for what you buy. Most of what a good AI engagement finds is ordinary, and a fair amount of it maps to categories your team already knows how to fix, which helps the remediation timeline. It also explains why a scope written around the model comes back thin while one written around the integration does not.

What changes when the model can act

Separate the model from the integration and most of the confusion clears up. The model is a component you bought or called. The integration is the thing you built: the tool definitions, the permission grants, the code that decides what to do with whatever comes back.

OWASP has a name for the failure mode on the integration side. LLM06:2025 Excessive Agency in the OWASP Top 10 for LLM Applications covers what happens when an LLM system is granted the ability to call functions or reach other systems and that grant is wider than the job requires. OWASP splits it into three root causes, and the split is useful when you are scoping a test:

• Excessive functionality, where the agent can reach functions it does not need for its intended operation

• Excessive permissions, where the tools it calls hold rights on downstream systems beyond what the intended operation requires

• Excessive autonomy, where a high-impact action goes through without independent verification or approval

All three are primarily integration and configuration decisions, which means they are decisions you can test and control. Mike Shelton made the same argument from the defensive side in what we allow AI to do, and the testing view is the mirror image of it.

Where the findings actually sit

Tool and function permissions

Start with what the agent can reach and what credentials it carries. An agent usually gets its own identity, and that identity accumulates: a tool built to look up an order also updates one, a service account provisioned for a read grows a write, a connector added for a pilot outlives the pilot.

A tool exposed to the model as "look up customer" that accepts a customer ID and no tenant filter. A ticketing integration that can close and delete as well as create, because the API token was issued once with a broad scope. A deployment helper reachable from a chat surface that nobody expected to be conversational. And the case a permission review catches that a single-tool test does not: a tool that is safe on its own and dangerous in sequence with another.

Whose authority the agent acts under matters as much as what it can reach. OWASP's prevention guidance is to track user authorization and security scope so that actions taken on behalf of a user run on downstream systems in that user's context. When an agent holds one broad set of credentials and serves every user through them, that property is gone, and a request that should have failed for this user succeeds because the agent was allowed to make it.

Prompt injection that reaches a function call

Prompt injection sits at LLM01:2025 on that list and it is well covered ground. We have written the primer on what prompt injection is and how it plays out in web apps with integrated LLMs.

What changes with an agent is how far a successful injection reaches. Against a chatbot, injected instructions can already cause disclosure and other real impact. Against an agent, that impact extends into downstream actions: a tool call fires, and the result is a row written, a message sent, or a file moved. The indirect version deserves particular attention here, because an agent that reads documents, web pages, tickets or retrieved records is taking instructions from content that no user typed and nobody reviewed.

What happens to the output downstream

LLM05:2025 Improper Output Handling attracts less attention than prompt injection. Model output is untrusted input, and the system that receives it frequently does not treat it that way. Generated text lands in a query, a shell argument, a rendered page or a webhook payload, and the validation that would have caught a malicious user may be skipped because the output is treated as trusted.

This is the class most likely to turn an AI finding into an ordinary one. The consequence lives in your app, which is why an engagement here often ends up reading the surrounding web app and API surface too.

Retrieval, embeddings and tenant boundaries

An agent that retrieves brings its retrieval layer into the attack surface. LLM08:2025 covers vector and embedding weaknesses, and the practical version for a multi-tenant product is whether filtering happens at query time or is assumed from how the index was built. Shared indexes, embeddings generated before a tenant model existed, and documents whose access control was enforced in the source system but never carried into the vector store all belong in scope. The failure looks like an ordinary authorization flaw, because it is one.

What the context window can reach

LLM02:2025 Sensitive Information Disclosure and LLM07:2025 System Prompt Leakage both land here. System prompts get treated as configuration rather than as something an attacker may read, so they accumulate internal URLs, table and field names, business rules and tool metadata. If a system prompt holds a credential, that is a finding in its own right before anyone tries to extract it. Tool definitions carry the same exposure, and so does whatever a prior turn left in the window.

Testing an agent is not red-teaming a chatbot

Both are useful and they answer different questions. Red-teaming a chatbot asks what you can get the model to say. Testing an agent asks what you can get it to do, and then follows the consequence into the system that received the action.

Following the consequence is most of the work, and it is why an engagement needs a real environment rather than a prompt playground. It also creates a practical problem to raise with any provider you talk to. Model behavior is not deterministic, which raises the bar on evidence rather than on the number of attempts. A single confirmed unauthorized action or high-impact state change is reportable on its own. What the report has to carry is the detail around it: the input, the path it took, the tool call it triggered, the state it changed, the conditions it needed, and what happened on repeat attempts.

Mike wrote up where this line falls in what AI and LLM penetration testing actually looks like, including the parts that sit outside it. Read that before you write a scope.

What to ask a provider for

Proposals in this service line still vary a lot. Ask for:

• A written scope that names the tools, connectors and data sources inside it, not just the app it belongs to

• Reproduction steps for every finding, including how often it was reproduced and under what conditions

• Findings mapped to a public reference such as the OWASP Top 10 for LLM Applications, so you can separate an integration flaw from a complaint about the model

• An explicit statement of what was not tested, which for most engagements includes the foundation model itself

• Remediation guidance aimed at your permission model and your code, not only at rewording the system prompt, since prompt-only restrictions are not a substitute for authorization enforced at the downstream system boundary

• Whether remediation testing is included after you fix things, and inside what window

If a proposal is priced entirely on prompt attacks and returns transcripts, you are buying a chatbot review. That is a useful thing to buy, and it answers a different question than the one you have.

What a tester needs from you before starting

An agent engagement often requires additional scoping inputs, because the interesting surface is not discoverable from the outside. A tester poking at your chat window cannot see your tool definitions. Expect to map:

• The model and provider, and whether you call it hosted or self-host it

• The use case and what the agent is permitted to do, as designed

• User roles and the authentication model

• Environments available for testing, and any restrictions on them

• The tenant model, if you have one

• MCP servers, connectors and tools the agent can reach

• RAG data sources, and who controls the documents in them

• API endpoints in scope

Two of those tend to be the bottleneck. Nobody has written down the full list of tools the agent can actually reach, and nobody is sure which environment is safe to test against. Both are cheaper to resolve before the engagement than during it.

Where this testing tends to earn its place

Scoping depends on what your agent touches and what it is allowed to do, so read these as situations where an AI/LLM engagement belongs on the table, not as a verdict:

• The agent can write to a production system, not just read from one

• It acts on behalf of users, and you serve more than one tenant through the same integration

• It retrieves from documents or feeds you do not fully control, which is where indirect injection enters

• It reaches third-party connectors or MCP servers, so its effective permissions are wider than your own code

• You are pre-launch, and the permission model is still cheap to change

If your agent only summarizes text it was handed and calls nothing, the risk profile is closer to an ordinary app feature, and scoping should say so.

Scope, stated plainly

An AI/LLM engagement tests the integration and the app around it. It does not test the foundation model, and it is not an evaluation of model accuracy, bias or performance. Testing against a production system is scoped and controlled to minimize risk, with the destructive actions agreed in advance rather than discovered during the test.

At Red Sentry, that engagement is AI/LLM Penetration Testing, which you can scope and buy on its own. Web and API testing often sit alongside it, because a tool call ends somewhere in your app and that part needs testing too. Neither is a prerequisite for the other. The service page lists what comes back in the report.

Book a scoping call and bring the tool list, even a rough one. We will tell you which parts of your agent to test first, and which parts are ordinary app surface you may already have covered.