An agent that calls tools
What you are building
Your model proposes structured calls that server code turns into reads or actions. The execution handler decides whether those proposals can touch a real record, recipient or payment.
Where untrusted text gets in
- Messages and earlier conversation turns that shape a proposed action.
- Documents, web pages and tool results read before the model selects its next call.
- Model-generated tool names and arguments, even when their JSON is well formed.
- A changed action proposed after the user has approved an earlier set of arguments.
What it can do if it goes wrong
- Execute a validly shaped call with an attacker-chosen price, ID or recipient.
- Use a privileged service account to reach a record the current user cannot access.
- Send, pay, delete or publish without approval for the actual action.
- Continue an unbounded loop of model calls and tools while appearing to handle one request.
Know the limit
Schema validation establishes argument shape, not user intent or permission. Approval also needs the same authorization checks as an ordinary API action. Test the execution boundary directly and keep benign controls in the regression pack, rather than deciding safety from the model's final reply.
Minimum controls
Validate every tool call on the server
Every tool validates its arguments on the server and recomputes prices and IDs from your own data.
Run tools with the user's permissions, not the agent's
Every tool takes user and tenant identity from the session and checks the same permissions as the normal API.
Ask before any action that cannot be undone
Irreversible actions need an explicit approval bound to the exact arguments, with an expiry checked at execution.
Treat retrieved and uploaded content as data, not instructions
Outside text is labeled as untrusted, hidden document text is checked, and tools that send data or change state are removed or gated.
Cap spend and rate-limit every AI endpoint
AI endpoints require authentication, limit each user and IP, and cap input, output, agent iterations and tool calls.
Run injection tests on every change
A fixed pack of attacks and benign controls checks tool calls and arguments in CI when prompts, models or tools change.