Workflow · last verified 2026-07-18

Red-teaming your LLM system: adversarial testing as a practice

The job this workflow does

The first eight layers of the quality stack — spec-first framing, context engineering, structured output, machine gates, evals, guardrails, human-in-the-loop, observability — all catch what they were built to catch, on inputs that behave the way the builder expected. None of them is designed to ask "what happens if the user is actively hostile." That question is what red-teaming answers, and it answers it empirically instead of by hoping the other eight layers happened to cover it.

The industry has a shared taxonomy for the answer's shape: the OWASP GenAI Security Project's Top 10 for LLM Applications, currently the 2025 edition. It names ten categories — prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption — and two of them, sensitive information disclosure and system prompt leakage, either moved up sharply or are entirely new since the 2023 list, which is itself evidence that the threat surface for LLM systems is still moving. Red-teaming is how a maintainer finds out where their own system sits against that list, instead of assuming the guardrails already cover it.

The same conclusion shows up on the infrastructure side: AWS's Well-Architected Generative AI Lens (published 2025-11-19) builds "mitigate harmful-output and excessive-agency risks" and "secure prompts" directly into its security guidance for generative AI workloads — adversarial testing is now a named line item in enterprise architecture guidance, not a specialist add-on.

The workflow

1. Map the attack surface against the named categories

Before writing a single adversarial test, name which OWASP LLM Top 10 categories apply to your system and why. Three attack shapes account for most of what a maintainer needs to cover first, and each maps onto the named taxonomy rather than being its own category:

None of these three is dangerous by itself if the model can only talk. What turns a successful jailbreak or injection into an incident is what the model is allowed to do next — a broadly-scoped database tool, an email-send action, a file write. That multiplier is its own named category, LLM06:2025 Excessive Agency, and it is exactly what the defense-in-depth pattern's tool boundary exists to constrain. Map your attack surface with that category alongside the other three, or the adversarial suite will only ever test what the model says, never what it can be made to do.

2. Build the automated adversarial suite

Hand-testing a handful of prompts once does not scale and does not repeat. An automated suite generates adversarial cases against a stated purpose and a named set of categories, runs them against your target, and grades the result the same way an eval suite grades a capability — code-graded where possible, judge-graded where not. promptfoo is the in-stack candidate here: MIT-licensed, free at up to 10,000 probes/month on the Community tier, and its red-team mode is a first-class extension of the same eval CLI this workflow library already uses elsewhere.

The suite is driven by a config file, not a hand-written list of prompts. The purpose field tells promptfoo what your target is for, which drives what it generates; plugins name the vulnerability categories to probe; strategies name the delivery techniques layered on top:

# promptfooconfig.yaml — the harness, not the exploit; promptfoo generates the actual test cases
redteam:
  purpose: "Support-ticket triage agent with read access to the ticket database"
  targets:
    - id: support-agent
      config: { url: "https://internal.example/api/agent" }
  plugins:
    - prompt-extraction   # LLM07:2025 System Prompt Leakage
    - sql-injection        # LLM06:2025 Excessive Agency, via the DB-backed tool
    - harmful:hate          # output-handling / policy category
  strategies:
    - jailbreak              # delivery technique, filed under LLM01:2025 Prompt Injection
    - base64                  # encoding-based delivery, same category

The current CLI is one prescribed step or two staged ones — both current, pick based on whether you want generation and evaluation as one gate or two:

# one step — regenerates cases if the config changed, then evaluates
npx promptfoo@latest redteam run

# or staged — generate once, evaluate repeatedly
npx promptfoo@latest redteam generate    # synthesizes adversarial cases into redteam.yaml
npx promptfoo@latest redteam eval --tag ci
npx promptfoo@latest redteam report       # local web UI over the findings

promptfoo redteam plugins lists the full current attack-plugin catalog if you need categories beyond the three above; redteam init or the interactive redteam setup wizard scaffold a starting config instead of writing one from scratch.

3. Run it on two cadences — per-release gate, scheduled deep sweep

One suite, two jobs, the same split this library already uses for evals: a small, fast, deterministic subset that blocks a release, and a larger, slower sweep that runs on a schedule and reports rather than gates. A red-team suite with every plugin and strategy enabled is too slow and too noisy to sit in a merge gate; a red-team suite that never grows past its original three plugins stops finding anything new as attack techniques move.

# package.json — same "the gate is a script" pattern as the eval workflow
"redteam:gate":  "promptfoo redteam eval --config redteam.release.yaml --tag release",
"redteam:sweep": "promptfoo redteam run --config redteam.full.yaml --tag scheduled"

# CI — release gate on every deploy, full sweep on a schedule
on: pull_request:            run: npm run redteam:gate   # small config, blocks merge
on: schedule (weekly cron):  run: npm run redteam:sweep  # full plugin/strategy set, reports only

The release config stays small on purpose — the categories from step 1 that a regression would be unacceptable for. The scheduled sweep is where you widen coverage: pull the latest plugins, add strategies as promptfoo's Registry adds them, and let it run against the full attack surface without holding up a deploy. When a scheduled finding proves stable across a few sweeps, promote its case into the release config — the same graduation move the eval workflow uses for a capability eval that's earned regression status.

4. Triage every finding back into guardrails and evals

A red-team run that produces a report nobody acts on is theater, not defense. Every finding gets a disposition, and each one lands in a specific place:

• Guardrail gap — the input or output layer should have caught this and didn't. Tighten the rule, or reconsider the flag-versus-block decision for that pattern; see the defense-in-depth pattern for how that tuning works layer by layer.
• Eval gap — the behavior is wrong and nothing regresses it going forward. Promote the finding into a permanent regression case, the same way the eval-first-development workflow grows its suite from real failures.
• Accepted risk — genuinely out of scope for this system. Document why, and leave the plugin enabled so a future change that reopens it gets caught again.

The flag-versus-block call is not academic. On this platform, the AI Gateway's guardrails run in flag mode rather than block, because block mode was found — by testing, not by guessing — to flag the platform's own security-curriculum content whenever a lesson quotes a prompt-injection example. That is a red-team-shaped finding that fed directly into a guardrail configuration decision: the fix wasn't a better filter, it was picking the right mode for the context. The loop only closes if a finding's disposition is recorded somewhere durable, not left in a report file nobody reopens.

Artifacts this workflow produces

• The attack-surface map — which OWASP LLM Top 10 categories apply and why, from step 1.
• The versioned adversarial config — promptfooconfig.yaml, purpose, plugins, and strategies, committed like code.
• The release gate config — a small, fast subset that blocks deploys; distinct from the full sweep config.
• The scheduled sweep job — full plugin/strategy coverage, reporting on a cadence, not gating.
• The triage log — every finding's disposition (guardrail fix, promoted regression case, or documented accepted risk).

Pitfalls

• Red-teaming as a one-time pre-launch checkbox. A suite run once before ship and never again stops reflecting the system the moment the prompt, the tools, or the model changes. It needs the same cadence as the eval gate it feeds.
• Testing only the input side. A jailbreak that only makes the model say something embarrassing is a lesser incident than one that makes an over-permissioned tool act. Cover LLM06:2025 Excessive Agency, not just LLM01:2025 Prompt Injection.
• Findings that never get triaged. A report that lists the same category failing release after release means the loop from step 4 is broken, not that the model is untestable.
• Treating "jailbreak resistance" as one thing to fix. It cuts across multiple named categories — injection, disclosure, agency, prompt leakage — and a single filter tuned for one rarely covers the others.
• Sharing exploit payloads instead of harness config. Keep working injection text out of tickets, docs, and chat — the config and the category are what the team needs; the payload is what an attacker needs.

When not to use this workflow

Skip the full adversarial suite for a genuinely throwaway prototype with no real user data, no real secrets, and no tool-execution capability — there is nothing yet for an attacker to reach. That exemption covers only the prototype with nothing real behind it: the day it gains real data, real credentials, or a tool that does something irreversible, calling it "still throwaway" is wishful thinking, not an assessment — the attack surface exists whether or not anyone has mapped it yet.

And this page assumes the vocabulary of the other quality-stack layers rather than re-teaching it — see the AI quality stack hub for how red-teaming fits alongside evals, guardrails, and observability, and the defense-in-depth pattern for the guardrail layers a finding here most often lands in.

Changelog

• 2026-07-18 — initial version, verified against the OWASP GenAI Security Project's 2025 LLM Top 10, promptfoo's red-team CLI docs, and the AWS Well-Architected Generative AI Lens.