Guide

An agent that calls tools

Draft, not yet reviewed

What you are building

Your model proposes structured calls that server code turns into reads or actions. The execution handler decides whether those proposals can touch a real record, recipient or payment.

Where untrusted text gets in

  • Messages and earlier conversation turns that shape a proposed action.
  • Documents, web pages and tool results read before the model selects its next call.
  • Model-generated tool names and arguments, even when their JSON is well formed.
  • A changed action proposed after the user has approved an earlier set of arguments.

What it can do if it goes wrong

  • Execute a validly shaped call with an attacker-chosen price, ID or recipient.
  • Use a privileged service account to reach a record the current user cannot access.
  • Send, pay, delete or publish without approval for the actual action.
  • Continue an unbounded loop of model calls and tools while appearing to handle one request.

Know the limit

Schema validation establishes argument shape, not user intent or permission. Approval also needs the same authorization checks as an ordinary API action. Test the execution boundary directly and keep benign controls in the regression pack, rather than deciding safety from the model's final reply.

Minimum controls

  1. Validate every tool call on the server

    30 min

    Every tool validates its arguments on the server and recomputes prices and IDs from your own data.

  2. Run tools with the user's permissions, not the agent's

    30 min

    Every tool takes user and tenant identity from the session and checks the same permissions as the normal API.

  3. Ask before any action that cannot be undone

    45 min

    Irreversible actions need an explicit approval bound to the exact arguments, with an expiry checked at execution.

  4. Treat retrieved and uploaded content as data, not instructions

    30 min

    Outside text is labeled as untrusted, hidden document text is checked, and tools that send data or change state are removed or gated.

  5. Cap spend and rate-limit every AI endpoint

    15 min

    AI endpoints require authentication, limit each user and IP, and cap input, output, agent iterations and tool calls.

  6. Run injection tests on every change

    45 min

    A fixed pack of attacks and benign controls checks tool calls and arguments in CI when prompts, models or tools change.