Governance & risk

AI agent governance: controlling autonomous AI in production

Why agentic AI needs governance

A chatbot generates text; an agent acts — it calls APIs, moves money, changes records, sends messages. The moment a model is wired to tools, every loop iteration becomes a decision that someone in the organisation is accountable for, taken at machine speed and without a human in the path by default. That is the shift that turns governance from a compliance checkbox into an engineering concern.

Three properties of agents force the issue. They are autonomous — they choose their own next action, so you cannot enumerate every behaviour in advance. They are non-deterministic — the same input can produce a different action, so you govern by bounding and logging, not by testing every path. And they operate at scale and speed — a misconfigured agent can take thousands of wrong actions before anyone notices. Add the regulatory layer — the EU AI Act imposes legal obligations on higher-risk AI systems — and governance becomes something you design in, not bolt on.

The governance controls

Governance is concrete: a handful of controls that, together, make an agent's actions attributable, bounded, and reviewable. These are organisation-level controls — they define who is accountable and what is permitted, and the runtime guardrails enforce them per request.

Governance vs security vs guardrails

These three are routinely conflated, and conflating them leaves gaps. They answer different questions:

• Governance — accountability and authority. Who owns this agent, what is it permitted to do, and how do we prove what it did? Organisation-level, mostly designed before a single request runs.
• Security — defence against adversaries. Prompt injection, tool poisoning, data exfiltration, privilege escalation. This is the agent security surface and the OWASP threat model.
• Guardrails — runtime control of one agent run. Bounded loops, input/output filtering, allow-lists, the human-approval gate firing on a specific action.

The connection: governance sets the policy, guardrails enforce it on every request, and security protects both from being subverted. Least privilege is the clearest example — governance decides an agent's mandate, the permissions system enforces it as a guardrail, and that same boundary contains a security breach when injection succeeds. Get one without the others and you have accountability you cannot enforce, or enforcement no one is accountable for.

Mapping to frameworks: NIST AI RMF, ISO 42001, EU AI Act

You do not have to invent governance from scratch — three recognised frameworks already define what good looks like, and mapping your controls to them is how you demonstrate diligence to auditors, customers, and regulators.

NIST AI RMF and ISO 42001 give you the management scaffolding; the EU AI Act sets the legal floor; OWASP supplies the threat detail. A mature programme cites all four: the first two for how you govern, the Act for what you must do, OWASP for what you defend against.

Building governance into the architecture

Governance fails when it lives in a policy document the runtime never reads. The architect's job is to make the controls structural — properties of the system, not promises about it. In practice that means wiring each control into a specific layer of the agent architecture:

• Identity in the tools layer — every agent authenticates as itself; tool calls carry that identity so the audit trail attributes each action.
• Permissions as least-privilege bindings — scope each tool to the minimum capability, so the mandate is enforced by construction, not by prompt.
• Audit as a first-class output — log every loop iteration, tool call, and decision to an immutable store; observability and audit are the same pipeline.
• Approval gates as code — high-risk actions pause for human sign-off in the orchestration loop, with the bounded-loop budget as the backstop.
• Policy as enforcement, not documentation — encode the rules in the operational plane so they apply on every request and map cleanly to a framework's requirements.

Done this way, governance and the operational plane are the same surface viewed two ways: engineers call it guardrails and observability; auditors call it oversight and audit trail. Build it once, satisfy both.

Frequently asked questions

What is AI agent governance?

AI agent governance is the set of organisation-level controls that keep autonomous AI agents accountable: agent identity, scoped least-privilege permissions, audit trails, human oversight and approval, and policy enforcement, mapped to frameworks like NIST AI RMF, ISO/IEC 42001, and the EU AI Act. It defines who is responsible for an agent's actions, what it is permitted to do, and how you prove what happened.

Why do AI agents need governance?

Because agents take real actions — calling APIs, changing records, moving money — autonomously, non-deterministically, and at machine speed. Each action is a decision someone is accountable for, and a misconfigured agent can take many wrong actions before anyone notices. Governance bounds what agents may do and records what they did, so the organisation stays accountable and can meet regulatory obligations such as the EU AI Act.

What is the difference between AI governance and AI security?

Governance is about accountability and authority: who owns an agent, what it is permitted to do, and how you prove its actions. Security is about defending against adversaries: prompt injection, tool poisoning, data exfiltration, and privilege escalation. Governance sets the policy and security protects it from being subverted; you need both, plus runtime guardrails to enforce them on every request.

What frameworks apply to AI governance?

Four recur. The NIST AI Risk Management Framework structures governance around Govern, Map, Measure, and Manage. ISO/IEC 42001 is the certifiable AI management system standard. The EU AI Act is binding law that imposes oversight, logging, and transparency duties on high-risk systems. OWASP's GenAI threat models (the LLM Top 10 and the agentic guides) supply the attack detail your controls must cover.

How do you govern an autonomous agent?

Give it its own identity, scope its permissions to least privilege, log every prompt, tool call, and decision to an immutable audit trail, gate high-risk actions behind human approval, and enforce policy as code in the runtime rather than as a document. Then map those controls to a recognised framework so the governance is demonstrable, not just claimed.

What is agent identity?

Agent identity is a distinct machine identity for each agent, separate from the human user it may act on behalf of. It lets every action be attributed to a named agent, lets you scope permissions per agent, and makes the audit trail meaningful. Without it, agent actions disappear into a shared service account and accountability breaks.