The moment you give a model the ability to take actions, the failure mode stops being a wrong sentence and becomes a wrong side effect. A hallucinated fact is embarrassing; a hallucinated tool call that issues a refund or drops a table is an incident. The design problem is containment: assume the model will occasionally do the wrong thing and make sure the blast radius is small.
Design tools like a public API, not internal helpers
Each tool the model can call is an untrusted client of your system. Give it a narrow, typed interface, validate every argument server-side, and never let it pass raw SQL or shell strings. A tool called 'refund_order' that takes an order id and a capped amount is safe; a tool called 'run_query' that takes arbitrary SQL is a loaded gun.
- Scope each tool to one capability with explicit, validated parameters.
- Enforce authorization inside the tool, on the current user — never trust the model to respect permissions.
- Make destructive tools idempotent so a retried call cannot double-charge or double-send.
- Return structured errors the model can recover from rather than raw stack traces.
Separate read from write, and gate the writes
Read-only tools can run freely; anything that mutates state, moves money or contacts a person deserves a checkpoint. For high-stakes actions, have the agent propose the call and require a human or a rules engine to approve it. Confirmation does not have to be a person every time — a spend limit or an allowlist of recipients is a cheaper gate that still bounds the damage.
Bound the loop and watch it run
Agents loop, and loops that plan their own next step can spiral — repeating a failing call, wandering off task, or burning tokens without converging. Cap the number of steps, set a wall-clock and token budget per task, and stop cleanly when a limit is hit. Log every step, tool call and result so a bad run is fully reconstructable after the fact.
- Set a hard ceiling on steps and total tokens per task.
- Trace the full plan-act-observe loop with inputs and outputs at each step.
- Detect and break repetition — the same failing call three times means stop, not try again.
Test the agent against adversarial inputs
Retrieved documents and user messages can carry prompt injection that tries to hijack the agent's instructions. Treat all external content as data, keep it clearly delimited from your system instructions, and include injection attempts in your evaluation suite. If a poisoned document can talk your agent into calling a tool it shouldn't, you want to learn that in a test, not from a customer.