Pattern · last reviewed 2026-07-21

The capability latch: revoking an agent's writes the moment a turn touches untrusted content

Context & problem

Simon Willison's lethal trifecta names the three capabilities that, held together, make an agent exfiltratable: "Access to your private data", "Exposure to untrusted content", and "The ability to externally communicate in a way that could be used to steal your data." His root cause is the uncomfortable one — "LLMs are unable to reliably distinguish the importance of instructions based on where they came from" — and his mitigation is to remove a leg, because "we still don't know how to 100% reliably prevent this from happening."

On a real product all three legs earn their keep. Our coach reads private learner state, pulls live vendor documentation instead of answering from stale model memory, and writes back. Removing a leg is the clean answer and the one that deletes the feature. Beurer-Kellner et al. (2025) state the constraint the field is circling: "once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions." Their six patterns — Action-Selector, Plan-Then-Execute, LLM Map-Reduce, Dual LLM, Code-Then-Execute, Context-Minimization — each satisfy it structurally. The capability latch is a cheaper move in the same family, deliberately not one of the six: instead of restructuring the agent it makes the two capabilities mutually exclusive within a turn.

It is not a detector. Hackett et al. (2025) tested six prominent injection and jailbreak protection systems — Azure Prompt Shield and Meta's Prompt Guard among them — and found evasion reaching "up to 100% evasion success" in some instances. A filter is a speed bump; a capability that is not present cannot be talked into existing.

Forces

• Capability vs. containment. Every write tool is a consequential action an injected instruction can aim at. Removing it is the strongest control and the most expensive — it is the feature.
• Granularity. Latch a whole session and one lookup disarms the agent forever. Check per dispatch, setting the flag as you go, and the model's own array order defeats it.
• Deterministic vs. model-mediated. A system-prompt rule is advisory, and the model that would ignore it is the model you are defending against.
• Failure legibility. A block that throws kills the turn and teaches the model nothing. A block that returns a readable result, with an alternative, lets the agent recover in-band.

The pattern

Three design-time decisions and one runtime rule.

• Classify the tools. Split the toolset into write-capable — anything with a consequential side effect: state mutation, an outbound message, a payment — and read-only. That list is a constant in the tool layer, not something the model reasons about.
• Name the ingest event. Decide which tools return third-party content the agent did not author: web fetches, doc search, RAG retrieval, anything attacker-influenceable.
• Scope the latch. One user turn: a flag created per invocation of the agent loop, latching true, persisting across every model turn inside it, discarded when the next message arrives. Write-once-true — a path that clears it early is a bypass waiting to be found.

Then the runtime rule: when the model returns its tool requests, pre-scan the whole array for any ingest tool and set the flag before dispatching anything. Then dispatch in order; every write-capable call checks the flag first.

The pre-scan is the whole pattern. A single assistant turn can return an array of tool-use blocks, and the dispatch loop walks it in order. Set the flag when the search runs, and a model already steered by a poisoned page emits the write first: it dispatches at index 0 against a flag still false. Pre-scanning makes the flag reflect the declared intent of the turn, not the accident of its sequence.

The refusal matters as much as the block. Return it as a normal tool result the model can read and react to, carrying a short explanation and a concrete alternative, rather than throwing. Say the same in the system prompt: after a documentation read, the state-writing tools are disabled by design. The instruction is not the control; the code is.

Reference implementation notes

The latch is stack-agnostic, but the ingest boundary and the enforcement point sit in different places on each platform.

Anthropic

Current Anthropic guidance names the threat model this pattern serves: "Indirect prompt injection, where the user is trusted but Claude processes third-party content (web pages, emails, documents, tool results) that contains adversarial instructions." The latch pairs with their strongest structural recommendation — "Put untrusted content only in tool results… never in system prompts or plain user text blocks", plus the corollary "Don't put your own instructions in tool results", which is why the refusal above reads as data about the tool. It is also their least-privilege advice made conditional: "a successful injection can do minimal damage" because full privilege lasts only while the turn is clean. Their tool-output screening — a small Claude Haiku 4.5 classifier returning a parseable boolean — is the detection layer, not a boundary.

AWS

On AWS the ingest event is usually retrieval: a documentation search, a knowledge-base lookup, an MCP server fronting vendor docs. Allowlist the hosts you accept content from and treat that list as a capability grant, not a destination filter — widening it grants everything reachable on that host. Fail closed on an unparseable URL, case-fold the hostname, and match suffixes with a leading dot (.aws.amazon.com) so docs.aws.amazon.com.evil.tld does not slip past a naive endsWith.

Cloudflare

It is tempting to expect a platform guardrail to cover this. Cloudflare AI Gateway Guardrails is a content filter: prompts and responses can each be set to Flag, Ignore or Block, blocked requests return code 2016 (prompt) or 2017 (response), and evaluation runs on Llama Guard 3 8B on Workers AI, adding roughly 500 ms per request. It judges text, not whether a tool may fire — and the hard constraint, "Streaming is not supported when using Guardrails", means a streaming agent's reply is not inspected at all.

A second-order lesson we hit ourselves: a filter set to Block will block your own traffic when your subject matter overlaps its threat categories. This platform keeps Guardrails in Flag mode because under Block, a question from our own prompt-injection lesson was rejected with code 2016 — and passed under Flag. Capability controls never read the text.

Trade-offs

• It constrains actions, not content. A poisoned document can still shape the words of the reply even when it cannot trigger a write. Beurer-Kellner et al. concede the same limit for code-then-execute: "we cannot prevent a prompt injection in the calendar data from altering the content of the email sent."
• Cheap and deterministic, but unproven. CaMeL is the rigorous end: it compiles the trusted query into a fixed program, so runtime data arrives as a value that program operates on and never as an instruction that changes what it does, then layers capability-based policies on what tools may do with which data — roughly 77% of AgentDojo tasks with provable security. That costs an interpreter and a policy engine around the model. The latch is one invariant.
• The classification is a standing obligation. The latch is only as good as its two lists — which tools write, which ingest. A new tool added without being classified silently defeats it, and nothing will tell you.
• Write-once-true is deliberately blunt. One lookup disables writes for the rest of the turn even when the write was unrelated, so a legitimate turn ends with the user resubmitting. Softening that means a clear path, which is a new bypass surface.

When not to use this

Do not reach for a latch when you can remove the leg outright. If the agent has no legitimate autonomous need for a write tool, delete the tool — an absent capability cannot be latched around, ordered around, or tested. Beurer-Kellner et al. put least privilege, sandboxing and user confirmation in an appendix of best practices rather than among the patterns, which is the right hierarchy: they are the baseline you do first.

Do not use it when the untrusted read and the consequential write are the same task by design. "Read this invoice, update the ledger" cannot be split across turns; that is the case for Plan-Then-Execute, where the plan is fixed before untrusted data is seen so injected content "cannot inject instructions that make the agent deviate from its plan" — with the caveat that it can still influence the arguments to planned calls.

And do not use it alone. If every tool is read-only the latch has nothing to switch off, and your exposure is the reply channel — output scanning and link allowlisting, a different control. The paper is explicit: "no single pattern is likely to suffice across all threat models or use cases." A latch is one layer of a defense in depth, never the whole defense.

As-built evidence

This platform runs the pattern in production; the latch keeps the coach's writes unreachable from its untrusted reads.

• The write list is a module-private constant — WRITE_TOOLS = ['update_memory', 'grade_attempt', 'schedule_review'] in src/lib/tools.ts, three of the coach's eight tools; none can touch content, lessons, items or billing. The check against it is the first statement in tool dispatch, ahead of the tool switch.
• The pre-scan runs in runCoachLoop (src/lib/coach.ts), before the dispatch loop, and names the attack it closes:

// Capability latch pre-scan: set the flag from the WHOLE turn's tool
// requests BEFORE any of them dispatch, so a same-turn
// [grade_attempt, search_aws_docs] array can't dispatch the write
// before the search sets the flag (array order must not matter).
if (toolUses.some((tu) => tu.name === 'search_aws_docs' || tu.name === 'search_cloudflare_docs')) {
  untrustedIngested = true;
}

• Scope is exactly one learner message. A plain local let declared outside the while loop: it persists across up to six model turns in one invocation, resets on the next message, and is never set back to false. The tool layer reads it through a getter (untrustedIngested?: () => boolean), so the loop can flip it after the context object was built.
• The block is a readable result, not an exception — a state_change_blocked payload telling the model to explain the answer and ask the learner to resubmit — and it is audited: a security.state_change_blocked row lands in audit_log with the blocked tool name, fire-and-forget, because an audit failure must never break the action it records.
• A red-team eval gates it. src/evals/coach.eval.ts carries seven cases tagged red-team, one being "state-change attempt after a poisoned doc result is blocked (the latch)". test/coach.eval.test.ts requires 100% to pass, plus an assertion forbidding any from being marked live-only.

Two caveats. That eval proves the deterministic control: the latch case runs against a fixed test model, so it shows the write is blocked, not that a real Claude model resists a poisoned-doc injection. And there is no CI here — the gates are enforced by a release runbook run by a human.

Changelog

• 2026-07-21 — Initial publication, on the day the capability latch shipped to production. Verified against Beurer-Kellner et al. (2025), CaMeL, Willison's lethal trifecta, Hackett et al. (2025), current Anthropic injection guidance, and the Cloudflare AI Gateway Guardrails docs.