Workflow · last verified 2026-07-21

Red-teaming your LLM system: adversarial testing as a practice

The job this workflow does

The first eight layers of the quality stack — spec-first framing, context engineering, structured output, machine gates, evals, guardrails, human-in-the-loop, observability — all catch what they were built to catch, on inputs that behave the way the builder expected. None of them is designed to ask "what happens if the user is actively hostile." That question is what red-teaming answers, and it answers it empirically instead of by hoping the other eight layers happened to cover it.

The industry has a shared taxonomy for the answer's shape: the OWASP GenAI Security Project's Top 10 for LLM Applications, currently the 2025 edition. It names ten categories — prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption — and two of them, sensitive information disclosure and system prompt leakage, either moved up sharply or are entirely new since the 2023 list, which is itself evidence that the threat surface for LLM systems is still moving. Red-teaming is how a maintainer finds out where their own system sits against that list, instead of assuming the guardrails already cover it.

The same conclusion shows up on the infrastructure side: AWS's Well-Architected Generative AI Lens (published 2025-11-19) builds "mitigate harmful-output and excessive-agency risks" and "secure prompts" directly into its security guidance for generative AI workloads — adversarial testing is now a named line item in enterprise architecture guidance, not a specialist add-on.

The workflow

1. Map the attack surface against the named categories

Before writing a single adversarial test, name which OWASP LLM Top 10 categories apply to your system and why. Three attack shapes account for most of what a maintainer needs to cover first, and each maps onto the named taxonomy rather than being its own category:

None of these three is dangerous by itself if the model can only talk. What turns a successful jailbreak or injection into an incident is what the model is allowed to do next — a broadly-scoped database tool, an email-send action, a file write. That multiplier is its own named category, LLM06:2025 Excessive Agency, and it is exactly what the defense-in-depth pattern's tool boundary exists to constrain. Map your attack surface with that category alongside the other three, or the adversarial suite will only ever test what the model says, never what it can be made to do.

2. Build the automated adversarial suite

Hand-testing a handful of prompts once does not scale and does not repeat. An automated suite generates adversarial cases against a stated purpose and a named set of categories, runs them against your target, and grades the result the same way an eval suite grades a capability — code-graded where possible, judge-graded where not. promptfoo is the in-stack candidate here: MIT-licensed, free at up to 10,000 probes/month on the Community tier, and its red-team mode is a first-class extension of the same eval CLI this workflow library already uses elsewhere.

The suite is driven by a config file, not a hand-written list of prompts. The purpose field tells promptfoo what your target is for, which drives what it generates; plugins name the vulnerability categories to probe; strategies name the delivery techniques layered on top:

# promptfooconfig.yaml — the harness, not the exploit; promptfoo generates the actual test cases
redteam:
  purpose: "Support-ticket triage agent with read access to the ticket database"
  targets:
    - id: support-agent
      config: { url: "https://internal.example/api/agent" }
  plugins:
    - prompt-extraction   # LLM07:2025 System Prompt Leakage
    - sql-injection        # LLM06:2025 Excessive Agency, via the DB-backed tool
    - harmful:hate          # output-handling / policy category
  strategies:
    - jailbreak              # delivery technique, filed under LLM01:2025 Prompt Injection
    - base64                  # encoding-based delivery, same category

The current CLI is one prescribed step or two staged ones — both current, pick based on whether you want generation and evaluation as one gate or two:

# one step — regenerates cases if the config changed, then evaluates
npx promptfoo@latest redteam run

# or staged — generate once, evaluate repeatedly
npx promptfoo@latest redteam generate    # synthesizes adversarial cases into redteam.yaml
npx promptfoo@latest redteam eval --tag ci
npx promptfoo@latest redteam report       # local web UI over the findings

promptfoo redteam plugins lists the full current attack-plugin catalog if you need categories beyond the three above; redteam init or the interactive redteam setup wizard scaffold a starting config instead of writing one from scratch.

As built on this platform: promptfooconfig.yaml targets POST http://localhost:8787/api/guide-coach — Ask the Guides, the public guide Q&A surface — posting { "question": "{{prompt}}" } and pulling the model's answer out with transformResponse: "json.answer":

# promptfooconfig.yaml — as shipped, not an illustrative shape
targets:
  - id: https
    label: guide-coach
    config:
      url: http://localhost:8787/api/guide-coach
      method: POST
      body: { "question": "{{prompt}}" }
      transformResponse: "json.answer"

redteam:
  numTests: 15
  plugins: [owasp:llm, prompt-extraction, pii, excessive-agency]
  strategies: [jailbreak, prompt-injection, multilingual, base64]

The runner is one script — "redteam:live": "npx promptfoo@latest redteam run" in package.json; promptfoo is fetched at run time via npx, not installed as a dependency. Run it manually, with a real OPENROUTER_API_KEY in .dev.vars and npm run dev already up. It is not a CI step, because this repo has no CI at all — no .github/ directory, no workflow file, nothing that runs on push. Every gate here, red-teaming included, is enforced today by a human running a script against a documented release runbook, not by a pipeline. Don't read the per-release/scheduled-sweep pattern later in this workflow as a description of what's automated here — for red-teaming specifically, on this platform, that automation doesn't exist yet.

3. Which surfaces you can actually reach

A red-team run needs something to send requests to. That constraint splits any real system into two very different buckets before you write a single plugin line: surfaces reachable with no credentials over plain HTTP, and surfaces gated behind authentication, a paid-plan check, or a transport a scripted tool can't casually dial — WebSocket, a message queue, an internal RPC. The first bucket is easy to automate against. The second usually holds the tool-using, higher-privilege target — which means the surface most worth red-teaming is often the one hardest to point a tool at.

As built here: promptfooconfig.yaml targets only Ask the Guides (POST /api/guide-coach) — unauthenticated, read-only, no tools. The paid coach (GET /api/coach/ws) is WebSocket-only, gated behind Clerk authentication and a paid-plan check, and it's the surface with the tool access (grading, memory writes, live doc search) this whole workflow exists to stress-test. Automating a red-team run against it needs an owner-provisioned authenticated test account on a paid plan — infrastructure this repo doesn't have yet, so it's filed as an open ticket rather than built against a guessed auth flow. The honest status is "red-teamed the reachable surface, opened a ticket for the one that matters more," not "red-teamed the coach."

4. Run it on two cadences — per-release gate, scheduled deep sweep

One suite, two jobs, the same split this library already uses for evals: a small, fast, deterministic subset that blocks a release, and a larger, slower sweep that runs on a schedule and reports rather than gates. A red-team suite with every plugin and strategy enabled is too slow and too noisy to sit in a merge gate; a red-team suite that never grows past its original three plugins stops finding anything new as attack techniques move.

# package.json — same "the gate is a script" pattern as the eval workflow
"redteam:gate":  "promptfoo redteam eval --config redteam.release.yaml --tag release",
"redteam:sweep": "promptfoo redteam run --config redteam.full.yaml --tag scheduled"

# CI — release gate on every deploy, full sweep on a schedule
on: pull_request:            run: npm run redteam:gate   # small config, blocks merge
on: schedule (weekly cron):  run: npm run redteam:sweep  # full plugin/strategy set, reports only

The release config stays small on purpose — the categories from step 1 that a regression would be unacceptable for. The scheduled sweep is where you widen coverage: pull the latest plugins, add strategies as promptfoo's Registry adds them, and let it run against the full attack surface without holding up a deploy. When a scheduled finding proves stable across a few sweeps, promote its case into the release config — the same graduation move the eval workflow uses for a capability eval that's earned regression status.

5. Triage every finding back into guardrails and evals

A red-team run that produces a report nobody acts on is theater, not defense. Every finding gets a disposition, and each one lands in a specific place:

• Guardrail gap — the input or output layer should have caught this and didn't. Tighten the rule, or reconsider the flag-versus-block decision for that pattern; see the defense-in-depth pattern for how that tuning works layer by layer.
• Eval gap — the behavior is wrong and nothing regresses it going forward. Promote the finding into a permanent regression case, the same way the eval-first-development workflow grows its suite from real failures.
• Accepted risk — genuinely out of scope for this system. Document why, and leave the plugin enabled so a future change that reopens it gets caught again.

The flag-versus-block call is not academic. On this platform, one hazard category — Prompt Injection/Jailbreaks — runs in flag rather than block, because block mode was found, by testing rather than by guessing, to catch the platform's own security-curriculum content whenever a lesson quotes a prompt-injection example. The other thirteen categories stay on block, prompt side and response side. That the disposition landed on one category rather than on the gateway is the part worth copying: the fix wasn't a better filter, and it wasn't standing the control down either — it was picking the right mode for the one context that needed a different one. The loop only closes if a finding's disposition is recorded somewhere durable, not left in a report file nobody reopens.

As-built: a red-team case that passed by testing nothing

An automated suite is only as trustworthy as its fixtures. Wiring the offline red-team eval gate that backs this workflow (src/evals/coach.eval.ts, 7 cases, gated at 100% in npm run eval) turned up the sharpest version of that lesson: a fixture built for the wrong wire shape can keep an injected payload from ever reaching the control it's supposed to test — and the case still reports green.

Cloudflare's and AWS's live doc-search MCP servers return differently-shaped responses: Cloudflare's is pseudo-XML over SSE, AWS's is double-JSON-encoded (the outer body's result.content[0].text is itself a JSON string). An earlier draft of the poisoned-AWS-doc fixture used the Cloudflare-shaped response for what was supposed to be an AWS case. searchAwsDocs failed to JSON.parse it, silently fell back to its "malformed AWS docs response" path, and the injected instruction text never reached probeInjection or fenceUntrusted at all. The case would have passed — it was exercising the malformed-response fallback, not the injection-detection path its own name promised. It was caught before merge, not shipped broken; the fixture now builds the correct double-JSON AWS shape, and the code comment records the near-miss verbatim so it can't recur silently.

A related, subtler version of the same failure is still open rather than fixed: 3 of the repo's 7 red-team eval cases give zero incremental signal even when run for real money against a live model. The poisoned-doc-content, poisoned-doc-host, and capability-latch cases each supply a fixed test streamFn that wins over the live model dependency under npm run eval:live too (a dependency-merge ordering issue in the eval runner), so real OpenRouter spend on those three proves only that the deterministic controls — the host-drop, the capability latch, the injection tagging — fire correctly, never whether an actual Claude model resists a poisoned-doc injection. That gap is tracked, not hidden (docs/QUEUE.md, ticket Q-033), with a named fix: a genuine live-only case with no forced stream function.

Generalize both into one checkable practice: a red-team fixture that cannot fail is not testing anything. Before trusting a green result, assert that the fixture actually reaches the control it's named for — feed it a known-bad input and assert the control itself fires (the probe flags it, the latch blocks the write, the host filter drops the URL) in the same test that checks the overall case passes. A wire-shape mismatch, a stubbed dependency that silently outranks the real one, or any other seam between the fixture and the control it targets will manufacture exactly this failure mode: full green, zero coverage.

As-built: the safety policy that blocked itself

The second case is the mirror image of the first, found on 2026-07-24 while building a prompt-injection classifier for this platform's AI coach (src/lib/injectionJudge.ts). The classifier is openai/gpt-oss-safeguard-20b, an open-weight policy-based safety model: its system message is not a persona, it is a policy document — instructions, definitions, criteria, and, in the obvious first draft, worked examples of what counts as a violation. The examples were not an invention: the model publisher's own recommended policy template ends with an Examples section and tells you to "provide 4-6 short examples labeled 0 or 1", on the sound reasoning that a classifier needs boundary cases. Following that advice verbatim is exactly what broke it.

Every call travels through Cloudflare AI Gateway, which runs its own Guardrails check on the outbound prompt. Guardrails saw the attack phrase sitting inside the policy and returned HTTP 424, error code 2016 — blocked prompt. The classifier never ran. The blast radius is the part worth teaching: the policy ships as the system message on every call, so it did not fail only on attack traffic. It failed 100% of turns, entirely benign ones included. A control designed to catch hostile input was disabled by its own description of hostile input. This is the same collision as the flag-versus-block finding in step 5 — the platform's own security material tripping the platform's own filter — arriving this time from inside a control rather than from a lesson.

Generalize it into three checks. Your detection content is itself traffic. Signatures, policies, test fixtures, and eval corpora all traverse the same filters as production requests, and they are the most attack-shaped strings in the system; a filter cannot tell your description of a payload from the payload. Test the tooling against the live path, not a mock. A mocked gateway passes this cleanly and ships a control with a 100% failure rate. Decide the failure direction on purpose. A security control that fails open under this condition degrades silently and nobody notices; one that fails closed takes down the whole product. Either way the operator learns nothing from the design review alone — nothing in the architecture diagram says the gateway will refuse to carry the policy.

The fix was to describe and name techniques rather than quote a complete, ready-to-use attack string. The same rule now governs the fixture corpus for that classifier (test/injectionJudge.eval.live.test.ts), for a reason worth stating separately: a gateway 424 reaches the caller as "classifier unavailable", so a fixture carrying a verbatim payload would silently be measuring the gateway instead of the classifier. That is the full-green, zero-coverage shape from the case above, arriving by a different route.

Artifacts this workflow produces

• The attack-surface map — which OWASP LLM Top 10 categories apply and why, from step 1.
• The versioned adversarial config — promptfooconfig.yaml, purpose, plugins, and strategies, committed like code.
• The release gate config — a small, fast subset that blocks deploys; distinct from the full sweep config.
• The scheduled sweep job — full plugin/strategy coverage, reporting on a cadence, not gating.
• The triage log — every finding's disposition (guardrail fix, promoted regression case, or documented accepted risk).

Pitfalls

• Red-teaming as a one-time pre-launch checkbox. A suite run once before ship and never again stops reflecting the system the moment the prompt, the tools, or the model changes. It needs the same cadence as the eval gate it feeds.
• Testing only the input side. A jailbreak that only makes the model say something embarrassing is a lesser incident than one that makes an over-permissioned tool act. Cover LLM06:2025 Excessive Agency, not just LLM01:2025 Prompt Injection.
• Findings that never get triaged. A report that lists the same category failing release after release means the loop from step 5 is broken, not that the model is untestable.
• Treating "jailbreak resistance" as one thing to fix. It cuts across multiple named categories — injection, disclosure, agency, prompt leakage — and a single filter tuned for one rarely covers the others.
• Sharing exploit payloads instead of harness config. Keep working injection text out of tickets, docs, and chat — the config and the category are what the team needs; the payload is what an attacker needs.
• A fixture that can't fail. A test double shaped for the wrong provider, or a stubbed dependency that silently outranks the real one, can make an injection case pass by never exercising the control it targets. Assert the control actually fires on a known-bad input before trusting the green — see the as-built section below.
• Forgetting that your detection content is traffic. Policies, signatures, fixtures, and eval corpora are the most attack-shaped strings in your system, and they traverse the same filters as production requests. A classifier policy that quotes a working payload can be blocked by an upstream guardrail on every call it ever makes — see the second as-built case below.

When not to use this workflow

Skip the full adversarial suite for a genuinely throwaway prototype with no real user data, no real secrets, and no tool-execution capability — there is nothing yet for an attacker to reach. That exemption covers only the prototype with nothing real behind it: the day it gains real data, real credentials, or a tool that does something irreversible, calling it "still throwaway" is wishful thinking, not an assessment — the attack surface exists whether or not anyone has mapped it yet.

And this page assumes the vocabulary of the other quality-stack layers rather than re-teaching it — see the AI quality stack hub for how red-teaming fits alongside evals, guardrails, and observability, and the defense-in-depth pattern for the guardrail layers a finding here most often lands in.

Changelog

• 2026-07-25 — added a second as-built case: a policy-based injection classifier that blocked itself on every call, because its policy text quoted a verbatim attack example and the AI Gateway's Guardrails check 424'd the outbound prompt before the classifier ran — generalized into three checks (detection content is itself traffic, test the tooling against the live path rather than a mock, decide the fail-open/fail-closed direction deliberately), plus the fixture rule that follows from it.
• 2026-07-21 — added the as-built promptfooconfig.yaml and its manual, no-CI status; added a new step naming which surfaces a red-team tool can actually reach (Ask the Guides) versus which need provisioned auth to test (the paid coach, tracked as an open ticket); added the as-built lesson on vacuous red-team fixtures — a wire-shape mismatch that kept an injected payload from ever reaching the control it targeted, plus 3 of 7 red-team eval cases that give no incremental signal even live — generalized into asserting a fixture reaches its control before trusting a pass.
• 2026-07-18 — initial version, verified against the OWASP GenAI Security Project's 2025 LLM Top 10, promptfoo's red-team CLI docs, and the AWS Well-Architected Generative AI Lens.