The agent threat model

AI agent security: the threat model for agentic systems

Why agents are a new attack surface

A chatbot returns text. An agent takes actions: it queries databases, calls APIs, writes files, sends messages — often on content it fetched from the open web or a shared inbox. Three properties combine to make this dangerous in a way a single completion never is.

• It acts. The model's output is no longer just shown to a person; it drives tool calls that change state. A bad decision has consequences, not just a wrong answer.
• It reads untrusted input. Anything the agent ingests — a web page, a PDF, an email, a tool result — can carry instructions. To the model, retrieved content and your system prompt are the same kind of text.
• It decides for itself. The agent chooses the next step in a loop. Autonomy is the feature; it's also what lets a single hijacked instruction cascade into many actions.

Simon Willison calls the worst combination the lethal trifecta: an agent that has access to private data, is exposed to untrusted content, and can communicate externally. Hold all three and prompt injection becomes data exfiltration. Remove any one leg — scope the data, sandbox the input, or cut the outbound channel — and the attack path collapses. That framing is the fastest way to audit a design: which legs does this agent hold at once?

The agent threat model (OWASP LLM Top 10)

The OWASP Top 10 for LLM Applications (2025) is the shared vocabulary for these risks. Not all ten are agent-specific, but the ones below are where agentic systems are won or lost. Map each threat to a mitigation you've actually built — design for them, don't patch them on.

OWASP's list is descriptive, not a checklist you complete once. The point is coverage: for every action your agent can take, you should be able to name which of these threats it exposes and which control answers it.

Excessive agency and insecure tool use

Excessive agency (LLM06) is the risk most unique to agents, because it only exists once a system can act. OWASP breaks it into three root causes, and each maps to a concrete design lever:

• Excessive functionality — the agent has access to tools or capabilities the task never needs (a leftover delete endpoint, a shell it rarely uses). Fix: expose the minimum set of tools, scoped to the job.
• Excessive permissions — a tool's credential is broader than required: a read task holding write access, a token scoped to a whole tenant instead of one user. Fix: least-privilege, per-user credentials, and enforce authorisation in the downstream system, not in the prompt.
• Excessive autonomy — the agent executes high-impact actions without a human in the loop. Fix: require approval for irreversible or sensitive operations; let the agent draft, a person commit.

Insecure tool use is the same problem from the tool's side. A tool is an attack surface: its inputs come from a model that may be under an attacker's influence, so it must validate arguments, refuse out-of-scope requests, and fail closed. The cardinal rule is that the model is not a security boundary — never rely on instructions in the prompt to keep an agent in line when the credential itself could simply be narrower. The MCP security guidance says the same for standardised tools: fine-grained access, mutual authentication, and sanitised tool results.

Designing a secure agent (principles)

Security for agents is architectural, not a filter you bolt on at the end. Five principles cover most of the surface above:

• Least-privilege tools. Each tool exposes the minimum capability, with the narrowest credential, scoped per user or session. A confused or hijacked agent can't reach further than its tools allow — so make the tools small.
• Bounded loops. Cap turns and tool calls with a budget and an escalation path, so a hijacked or looping agent stops instead of running away. This is the structural answer to excessive agency and unbounded consumption — detailed in the bounded agentic loop.
• Human-in-the-loop. Put a person on the high-impact, irreversible actions — sending money, deleting data, emailing customers. The agent proposes; a human approves. Autonomy is graduated, not all-or-nothing.
• Input and output handling. Treat everything the model reads as untrusted and everything it emits as unvalidated. Keep untrusted content out of the instruction channel where you can, and never pass output into a shell, query, or browser without validation. This is where prompt injection is contained and guardrails earn their keep.
• Isolation. Sandbox tool execution, separate per-user state and credentials, and break the lethal trifecta by design — so a single compromise can't read private data and exfiltrate it in the same breath.

None of these is exotic. They're the operational plane of a production agentic architecture — the layer that separates a demo from something you can safely leave running. Anthropic frames the same goal in its framework for safe and trustworthy agents: keep humans in control, scope what the agent can touch, and make its behaviour observable.

Frequently asked questions

What is AI agent security?

AI agent security is the practice of designing an agentic AI system — one that takes actions via tools, often on untrusted input — so those actions cannot be subverted. It covers the threats agents add over a plain chatbot (prompt injection, excessive agency, insecure tool use, data exposure, supply chain) and the controls that contain them: least-privilege tools, bounded loops, human-in-the-loop, and disciplined input/output handling.

What are the main security risks of AI agents?

The agent-relevant entries in the OWASP Top 10 for LLM Applications: prompt injection (LLM01, including indirect injection from ingested content), excessive agency (LLM06), sensitive information disclosure (LLM02), improper output handling (LLM05), supply chain (LLM03), and unbounded consumption (LLM10). Prompt injection plus excessive agency is the combination that turns a model error into a real-world action.

What is excessive agency?

Excessive agency (OWASP LLM06) is the risk that an agent can do more than its task requires, so manipulated or ambiguous output causes damaging actions. OWASP names three root causes: excessive functionality (too many tools), excessive permissions (credentials broader than needed), and excessive autonomy (acting on high-impact operations without human approval). You reduce it with least-privilege tools, scoped credentials, and human-in-the-loop on sensitive actions.

How do you secure an AI agent?

Design for it across the whole system: give each tool the minimum capability and a narrow, per-user credential; bound the loop with turn and tool-call budgets; require human approval for irreversible actions; treat all model input as untrusted and all output as unvalidated before it reaches a sink; and isolate execution and per-user state. Enforce authorisation in the downstream system, never in the prompt — the model is not a security boundary.

What is the OWASP Top 10 for LLMs?

The OWASP Top 10 for LLM Applications is an open, community-maintained list of the most critical security risks in applications built on large language models, published by the OWASP Gen AI Security Project. The 2025 edition ranks prompt injection first and includes excessive agency, sensitive information disclosure, improper output handling, supply chain, and unbounded consumption — the shared vocabulary for the agent threat model.

How do you limit what an agent can do?

Through least privilege and bounded autonomy. Expose only the tools the task needs, scope each tool's credential to the narrowest permission and to a single user or session, and enforce that authorisation in the tool and downstream system rather than trusting the prompt. Cap the loop with a turn and tool-call budget, and gate high-impact or irreversible actions behind human approval so the agent drafts but a person commits.

As-built: what an internal security audit of our own agent found

This platform runs a per-learner coach agent with tools, memory and a database behind it, and it has been through several adversarial review passes. The findings are worth more than the principles, because none of them was an exotic model attack — they were ordinary application-security bugs that only became agent bugs because a model sat in the loop.

• The agent's own output completed a stored self-XSS. A learner's saved reading position round-tripped through the API and reached an innerHTML href unescaped. Nothing about it is LLM-specific; what makes it an agent finding is that the value was written by one surface, stored, and rendered by another, which is the normal shape of agent state. Fixed on both sides — encodeURIComponent in the client href, and sanitizeReadingPos in src/lib/db.ts restricting anchors to a safe id charset — the slug shape our heading-id generator mints, plus deliberate headroom for hand-authored ids — with anything outside it dropped to empty rather than escaped, so an invalid anchor degrades to "resume at the top" instead of rendering.
• Untrusted text escaped its fence by including the fence. Short-answer and practical grading wrap the learner's submission in a delimiter before handing it to a model. A submission containing that delimiter could close its own wrapper. The fix is the one every escaping problem has: escape the token inside the untrusted text (src/lib/grade.ts, src/lib/practicalGrade.ts). Delimiting untrusted content is a real control, and it is only as strong as your handling of a submission that knows the delimiter.
• Excessive agency is enforced as a latch, not an instruction. The coach loop (src/lib/coach.ts) hard-caps at six model turns and eight tool calls, and it carries a capability latch: once a turn has requested a documentation search — i.e. once untrusted external content is on its way in — state-changing tools are refused for the remainder of that turn. The flag is set from the whole turn's tool array before any of it dispatches, so a same-turn [write, search] pair cannot beat it on ordering. That ordering detail is the entire control; an implementation that latched per dispatched call would have looked identical in review and failed in production. The full write-up is the capability-latch pattern.
• Reads and writes are separated at the HTTP-method level, because scanners click links. An unsubscribe route mutated state on GET, and mail scanners prefetching the footer link caused silent opt-outs. No attacker, no model — just an automated client doing exactly what the spec permits. GET now renders a confirmation form and writes nothing.

Two enforcement notes for anyone copying the architecture. Everything above is enforced in code, not in the system prompt: tool arguments are validated at the tool, doc fetches are host-allowlisted, editorial and spend-cap actions write audit rows, and per-user rate limits are applied per agent instance. And the one invariant we treat as non-negotiable is answer-key hygiene — read endpoints never return an item's key, and grading happens in exactly one place — because "the model was told not to reveal it" is not an access control. Prompt-injection-specific defences, including the metadata-channel bug that produced the strictest of these rules, are on the prompt-injection page; the review process that surfaced these findings is the security self-audit workflow.