Workflow · last verified 2026-07-21

The security self-audit: auditing your own LLM app with a fresh-context agent

The job this workflow does

This platform ran the workflow on itself, which is why the page exists. A six-packet hardening effort across both of its LLM surfaces shipped on 2026-07-21 (Worker 825a09d4, PRs #30–#36): a capability latch, fencing of untrusted content, host allowlisting, a streaming output scanner, a red-team eval gate that passes 100% or fails the local eval run. Every packet had a fresh-context reviewer. By every internal signal, it was finished.

Later the same day a second effort (Worker a59aa45f, PRs #37–#42) had a fresh-context subagent audit that result against the phase-1/2 runbook of an external hardening framework. It found five real gaps and confirmed seven other checks solid with evidence — the gaps are what the builders couldn't see, the confirmations are what turn "we think it's covered" into a citation.

A checklist beats a careful re-read because the defense is a combination of structural controls, and a combination can't be verified by reading one file. Willison's diagnosis of why prompt-level fixes don't hold: LLMs are "unable to reliably distinguish the importance of instructions based on where they came from." Hackett et al. (2025) measured the consequence for detectors, evading six protection systems — Microsoft's Azure Prompt Shield and Meta's Prompt Guard among them — with "up to 100% evasion success" in some instances.

This is not red-teaming, which attacks a running system empirically. This is the static half — code and configuration against a fixed list — and its last item is usually "does an adversarial suite exist at all."

The workflow

1. Freeze the external checklist before you open the code

Pick the framework first, and pick one you did not write, so scope isn't quietly negotiated against what you already built. Three sources cover a typical LLM application. The OWASP Top 10 for LLM Applications — the 2025 edition; there is no 2026 LLM edition, the 2026 list is the separate Top 10 for Agentic Applications — supplies the taxonomy: prompt injection (LLM01), sensitive information disclosure (LLM02), improper output handling (LLM05), excessive agency (LLM06), system prompt leakage (LLM07). Beurer-Kellner et al. supply the architectural remedies: six design patterns under one principle — once an agent has ingested untrusted input, "it must be constrained so that it is impossible for that input to trigger any consequential actions." Anthropic's guidance supplies the call-site practice: untrusted content only in tool results, JSON-encoded, screen tool outputs and not just user input, least privilege throughout.

Then flatten it into numbered items that don't move during the run, so every finding carries a stable identifier into commits and tickets.

2. Hand the checklist to a fresh context

This is the load-bearing step, and the one most likely to be skipped because it feels redundant. The agent that built the hardening cannot audit it — not from dishonesty, but because it knows what the code was meant to do. It holds the design intent and the reasoning behind every decision; asked to check the result, it re-reads its own rationale and agrees. The bias the AI-assisted review workflow handles for a single diff applies with more force to a whole security posture.

Evidence from the run above: two of the five findings were surfaces the original effort had hardened on one route and forgotten on its sibling. Foreign-link neutralization existed server-side on Ask the Guides (the public guide Q&A surface, POST /api/guide-coach) and nowhere on the authenticated coach; an argument-length guard existed on one write tool and not the one beside it. Both are invisible from inside the intent — the control was designed, implemented, and reviewed, and only its coverage was partial. A fresh context doesn't know which route was the interesting one, so it checks both.

# the audit prompt, new session, no prior context
Audit this repository against docs/genai-hardening-paper.md §8 phases 1-2.
Report every item as PASS or FAIL with file:line evidence. Fix nothing
yet. Do not assume a control is present because the repo documents it:
open the file and cite the line.

3. Discover every LLM call site and record its shape

Phase one is inventory, not judgement. Grep for the provider SDKs and gateway clients in use, and record five fields per call site: which model, where the system prompt comes from, where user or external input enters, which tools that call exposes, and where the output goes — browser, database, shell, another service.

The output column is under-filled most often and sets severity for everything else. A reply rendered in a chat pane has a different blast radius than text concatenated into a query or executed — OWASP's LLM05, and the difference between an embarrassing answer and an incident.

4. Build the lethal-trifecta matrix, one row per workflow

Willison's framing (2025-06-16) is the sharpest triage tool available because it turns "is this risky" into three yes/no columns: "Access to your private data", "Exposure to untrusted content", and "The ability to externally communicate in a way that could be used to steal your data." The second column is broader than it looks: it asks which input paths reach the model, not who typed the message, so retrieved documents and tool results fill it as readily as a visitor's question does. All three legs means critical; removing any one is the mitigation.

The top row is critical, which points straight at the remedy Beurer-Kellner's principle prescribes: sever the link between ingesting untrusted content and taking consequential action. Here that is a turn-scoped capability latch — once a turn ingests a documentation-search result, the three learning-state write tools are disabled for the rest of that turn (src/lib/tools.ts:164-182), with the flag set by a pre-scan of the whole turn's tool array (src/lib/coach.ts:306-312) so call order can't bypass it. Private data and untrusted content still coexist; the consequential-action leg is gone.

5. Audit item by item, pass or fail, with file:line evidence

One item, one verdict, one citation. Prose findings ("looks fine", "handled elsewhere") are how partial coverage survives an audit. And the citation has to be specific in a particular way: it proves the control exists, not that the control covers this route. The pass condition is a line in the file that handles the route being audited, not a line in the module that implements the idea.

# A8: model-output links neutralized before render?
PASS  src/lib/promptSafety.ts:101-111  # Ask the Guides: server-side, foreign URL -> bare hostname
FAIL  public/assets/module.js          # authenticated coach: no allowlist on the client renderer

# the FAIL is the point — same idea, one route, sibling missed

Report the passes too, with their citations. Seven of the twelve items came back solid in the run above, and those citations are what makes the next audit cheap.

6. Fix in priority order — and sweep every sibling of each fix

Order fixes by consequence, not ease: irreversible actions reachable from an untrusted context first, then output handling, then prompt hygiene, then tool validation and least privilege, then egress, then ingress limits and logging. One commit per item, so each is reviewable and revertible on its own.

Then add the rule this workflow exists to teach: every fix greps for its own defect shape across the whole repository before it is committed. A missing allowlist on one renderer isn't fixed by adding the allowlist there — it's fixed by finding every renderer, and every other place output crosses into a live link, and checking each. A finding is a sample from a population; the population is what you're fixing.

7. Prove each fix with a test that can fail

Every fix lands with a test, and the test's first obligation is being capable of failing. During the effort above, a red-team fixture was built in the wrong wire shape — a Cloudflare-shaped payload handed to the AWS documentation path — so the injected text never reached the injection-detection code at all. The case passed. It had been green while testing nothing, and it was caught by hand-tracing the fixture, not by any run, because a vacuous test looks exactly like a working one from outside (commit 4cedd7b). A security test that cannot fail is worse than no test, because it manufactures the confidence that stops anyone looking again.

The second obligation is stating what the test gates. This repo's red-team suite is seven offline-constructible cases behind a 100%-or-fail gate (test/coach.eval.test.ts:294-307) that runs in the ordinary keyless test run, with a second assertion structurally forbidding any case from being marked live-only so the gate can't quietly move behind a paid run. That is a strong gate on the deterministic controls — the latch, the host drop, the injection tagging — and that is all it gates: three of the seven inject a fixed stream that wins over live dependencies even under the live run, so they give zero incremental signal about whether a real model resists a poisoned document. That limit is written down as a ticket, not smoothed over.

# the gates — offline, no key, no spend
npm run typecheck
OPENROUTER_API_KEY= npm test     # sub-10s run proves no live suite ran
npm run eval                     # includes the 100%-or-fail red-team gate

# the live adversarial run is separate, manual, real money
npm run redteam:live             # npx promptfoo@latest redteam run, owner-run

Note what isn't in that block: a pipeline. This repository has no CI — the gates are enforced by a documented release runbook a human executes, and the promptfoo run is a manual, owner-invoked script against a local dev server, deliberately never wired to per-PR spend. If your stack does have CI, the deterministic gates belong in it; the live adversarial run is a cadence decision, not a merge gate.

8. Write down what the audit did not cover

The final artifact is the residual-risk list, and it ages best of all of them. Three kinds of entry belong on it: deferred scope, known-partial controls, and framework limits.

Be specific about the second. Here, the instruction-shaped pattern matcher is documented in its own module header as a warning generator rather than a filter — it annotates untrusted content so the model re-anchors, it will be evaded, and nothing may depend on it holding. Writing that down isn't modesty; it's what stops the next engineer treating a regex as a boundary. The third kind is easy to miss: platform guardrails have documented scopes, and Cloudflare states that "Streaming is not supported when using Guardrails" — a streamed surface sits outside that control's reach whatever the dashboard says is enabled.

Artifacts this workflow produces

• The frozen checklist — the external framework as stable numbered items that don't move mid-run.
• The call-site inventory — model, prompt source, input entry, exposed tools, output sink, per LLM call.
• The lethal-trifecta matrix — one row per workflow, three yes/no columns, criticals flagged.
• The audit report — every item PASS or FAIL with file:line evidence, passes included.
• One commit per fix — each with the sibling sweep that proves the defect shape is extinct, not relocated.
• A test per fix that can demonstrably fail — plus a written statement of what it does and doesn't gate.
• The residual-risk list — deferred scope, known-partial controls, framework limits, each with an owner.

Pitfalls

• Letting the builder audit the build. The context that produced the hardening holds the intent, and intent is what hides partial coverage. Two of five findings here were controls correctly implemented on one route and absent on its sibling.
• A security test that cannot fail. A fixture in the wrong wire shape meant injected text never reached the probe, and the case passed while testing nothing — worse than no test, because it manufactures confidence. Break the control on purpose and confirm the test goes red.
• Claiming more coverage than the suite has. Three of seven red-team cases give zero incremental signal even against a live model, because a test-injected stream wins over live dependencies. They gate the deterministic controls, and that is all they gate. Say so where the next reader looks.
• Assuming one review scope catches every bug class. An optional binding guarded for absence but not for throwing passed every per-task review — each correct within its own scope — and was caught only by the composed whole-branch pass that could see the wrapper and its call sites together.
• Auditing the control instead of the route. "We have an allowlist" is a module-level fact. The audit question is whether this path goes through it, and the answer is a line number on that path.
• Treating the checklist as the threat model. A framework is a floor built from other people's incidents. The item specific to your architecture comes from the trifecta matrix in step 4, not from the list.
• Mistaking a warning generator for a filter. Pattern lists annotate; they don't contain. Anything you rely on has to be structural — a capability that is unavailable, a host that isn't reachable, an action that is latched off.

When not to use this workflow

Skip it while the application is a prototype with no real data, no credentials, and no tool that does anything irreversible — there is nothing behind the model for an audit to protect. That exemption expires the moment any one of the three becomes true, and it expires silently: nobody sends a notification when the demo starts reading production data.

Don't run it as a substitute for architecture. If the critical rows stay critical after the fixes, the answer isn't a longer checklist — it's a structural pattern that removes a trifecta leg outright, which is what the defense-in-depth pattern lays out. And don't run it in place of adversarial testing: this proves the controls you designed are present and cover every route, while red-teaming probes for the ones you never thought of.

As-built evidence

Every claim above about this platform is checkable in its own source. The capability latch is the first statement in the tool dispatcher (src/lib/tools.ts:164-182), returning a readable tool result rather than throwing, with the turn-wide pre-scan at src/lib/coach.ts:306-312. The two link-neutralization implementations step 5 contrasts are src/lib/promptSafety.ts:101-111 (server-side, Ask the Guides) and public/assets/module.js:1352-1369 (client-side, authenticated coach, added by the audit). The fail-open rate-limit wrapper the whole-branch review produced is src/lib/rateLimit.ts:75-91 — fail-open by design, because it is always an additional gate over a primary in-memory limiter, never the only check. Security signals write to an audit table that predates the security work: the audit wired five signal types into it and added no table and no migration (src/lib/audit.ts:14-19). The red-team gate is test/coach.eval.test.ts:294-307; the framework audited against is docs/genai-hardening-paper.md §8.

Gate output at ship time: typecheck clean, 1,622 tests passing and 3 skipped on the keyless run, 43 eval cases passing including the red-team gate at 7/7. Counts drift — the keyless run's own output is the source of truth. In production, a live injection probe against Ask the Guides returned a clean refusal from the real model: no instruction leak, no forged answer key.

Changelog

• 2026-07-21 — initial version, written from this platform's own hardening effort and the fresh-context gap-fill audit that followed it (Workers 825a09d4 and a59aa45f), and verified against the OWASP 2025 LLM Top 10, Beurer-Kellner et al. (2025), Willison's lethal trifecta, Hackett et al. (2025), Anthropic's mitigation guidance, and Cloudflare's AI Gateway Guardrails docs.