The operational plane

LLM observability: tracing, monitoring, and debugging agents in production

The trace of one production run — every model call, tool call, and retrieval it made — is what layer 8 of the AI quality stack adds: observability. Evals and guardrails only tell you whether a run should have passed or been blocked; when one didn't, the trace is what shows you why, catching failures your eval set never sampled and your guardrails never flagged.

Part of the AI quality stack — the layered gate chain for knowing your LLM is delivering quality.

What LLM observability means

Observability is a property borrowed from systems engineering: a system is observable if you can infer its internal state from its outputs. For traditional services, those outputs are the classic three signals — traces, metrics, and logs. LLM observability applies the same idea to a model-driven system, where the hardest-to-see internal state is the model's own decisions: which tool it chose, what it sent, what came back, and why the loop kept going.

The reason it is a named discipline at all is non-determinism. A normal function given the same input returns the same output; a stack trace pins the bug. An LLM given the same input can return different text, call a different tool, or take a different number of steps. You cannot reproduce a production failure by re-running it locally, so the only way to debug is to have captured what happened the first time. Observability is that capture. It is one part of the operational plane that wraps every production agent system.

The signals: traces, metrics, logs

LLM observability rests on the same three signals as any observable system, specialised for model and agent behaviour.

Traces are the signal that makes LLM observability distinct. In a single completion there is little to trace; in an agent that loops, the trace is the whole story — a parent span for the run, child spans for each model turn, grandchild spans for each tool call inside that turn. Token, latency, and cost are attached to spans as attributes, which is also where cost attribution comes from: you can see exactly which step in which run spent the budget.

Observability vs evaluation vs monitoring

These three are routinely conflated, and the distinction is worth holding precisely because the tooling overlaps.

Monitoring is best read as a use of observability data, not a separate thing: you monitor by putting alerts on the metrics that observability emits — a latency spike, an error-rate climb, a cost-per-request jump. Evaluation is the genuinely separate discipline. It needs a notion of ground truth or a judge, and it answers a question observability never can: was the answer right? For how to build that correctness harness — datasets, LLM-as-judge, regression gates — see the LLM evaluation guide. The practical link between them is that traces captured by observability become the raw material for evaluation datasets.

Observability for agents: per-step tracing

Single-call observability is nearly trivial: log the prompt, the completion, and the token count. Agents are where it earns its keep, for two reasons.

• Multi-step. An agent run is a loop of model turns and tool calls, sometimes dozens deep. A failure three tool calls in is invisible unless every step is traced with its own span. Agent tracing is exactly this — capturing the nested span tree of a loop so you can open the failing run and walk it.
• Non-deterministic control flow. The agent decides its own path, so two runs of the same task can diverge. Without a trace you cannot tell whether a bad result came from a wrong tool choice, a bad tool result, a context that overflowed, or the loop hitting its budget. The trace makes the branch point visible.

This is why a bounded agent loop and observability are designed together: the same loop that enforces a turn and tool-call budget is the natural place to emit a span per iteration. See agentic AI architecture for where this sits in the system, and note that a gateway in front of the model — such as an AI gateway — gives you a second, infrastructure-level vantage point: request logs, token metrics, and caching visibility without touching application code.

OpenTelemetry and the GenAI semantic conventions

The portability question for LLM observability is the same one that vendor-neutral tracing solved years ago: if every tool defines its own span format, you are locked in. OpenTelemetry — the open standard for traces, metrics, and logs — addresses this for GenAI through its GenAI semantic conventions, a common vocabulary of gen_ai.* attributes for model calls, token-usage metrics, and tool and agent spans, so an instrumented app can emit to any compatible backend.

The honest caveat: as of mid-2026 these conventions are still experimental (OpenTelemetry marks them in Development status), so attribute names can change and you should pin a version. They are nonetheless already widely adopted, and several tools below speak them natively. Treat OpenTelemetry as the wire format you instrument against, and a tool as the backend that stores and visualises what it carries.

The main tools

The space spans open-source projects, dedicated commercial platforms, and LLM modules inside established APM suites. The list below is representative, not a ranking — the right choice depends on whether you want to self-host, how much you value an open standard, and whether LLM observability needs to live next to your existing infrastructure monitoring. Verify licensing and features against each project's own docs before committing.

A reasonable default heuristic: if you want an open standard and self-hosting, start from an OpenTelemetry-native open-source tool; if LLM observability must sit next to existing infrastructure monitoring, an APM vendor's module reduces moving parts; if you want the deepest LLM-specific tracing and eval workflow out of the box, a dedicated platform is usually furthest ahead. None of these removes the need for a separate evaluation harness.

Frequently asked questions

What is LLM observability?

LLM observability is runtime visibility into an LLM or agent system through three signals — traces, metrics, and logs. A trace records the full execution path of one request: every model call, tool call, and retrieval, with inputs, outputs, tokens, latency, and cost. It exists because LLMs are non-deterministic, so you cannot reproduce a production failure by re-running it; the only way to debug is to have captured what happened the first time.

How is observability different from evaluation?

Observability tells you what the system did on a given run; evaluation tells you whether the output was correct or good. Observability operates on live traffic one request at a time and needs no ground truth. Evaluation operates on a dataset with expected behaviour or a judge, and produces a quality score. They are complementary: traces captured by observability often become the dataset that evaluation runs against. See the evaluation guide for the correctness side.

What is agent tracing?

Agent tracing is capturing the full nested span tree of an agent run — a parent span for the run, child spans for each model turn, and grandchild spans for each tool call inside that turn. Because an agent loops and decides its own path, a failure several steps deep is invisible without per-step tracing. The trace lets you open a failing run and walk it to the exact step that went wrong.

What should you monitor in an LLM app?

The core metrics are token usage, latency, cost per request, error rate, and throughput, plus per-step counts for agents such as tool calls per run and how often the loop hits its budget. You monitor by putting alerts on these — a latency spike, an error-rate climb, or a cost-per-request jump. Quality signals such as refusal rate or failed-tool-call rate are useful too, but correctness itself belongs to evaluation, not monitoring.

What are the best LLM observability tools?

There is no single best; the choice depends on whether you want to self-host, how much you value an open standard, and whether LLM observability needs to sit next to existing infrastructure monitoring. Representative options include the open-source Langfuse, the commercial LangSmith, the source-available Arize Phoenix, and Datadog LLM Observability inside a wider APM platform. OpenTelemetry's GenAI conventions are the instrumentation standard these can ingest. Verify licensing and features against each project's own docs.

Does OpenTelemetry support LLMs?

Yes. OpenTelemetry has GenAI semantic conventions — a common vocabulary of gen_ai.* attributes for model calls, token-usage metrics, and tool and agent spans — so an instrumented app can emit to any compatible backend. As of mid-2026 these conventions are still experimental, marked Development status, so attribute names can change and you should pin a version. They are nonetheless already widely adopted, and several observability tools speak them natively.

As-built: the failures we actually miss are the silent-wrong ones

The alerting advice above is the easy half. Every outage this platform has had that mattered was silently wrong rather than loudly down — a pipeline that kept exiting 0, kept writing a file, kept producing a plausible number, and was wrong for days. Four of ours, because the shape is the point:

• A pinned model that the vendor's catalogue did not have. A blanket "refresh model references" commit set our AI-visibility tracker to a slug that is correct for this repo's own routing and absent from DataForSEO's catalogue, which lags model releases. Every Claude call in that job returned 40501 Invalid Field for 13 days. Nothing alerted, because nothing was watching a job that had no consumers yet. The fix was not an alert: scripts/seo/citation-tracker.mjs now validates its pins against the vendor's free live model catalogue at startup and refuses to run, printing the available list, before spending anything. A related flaw fell out of the same read — errored engines were dropped from the share-of-model denominator with nothing in the summary saying so, so a broken engine quietly turned a four-engine panel into a three-engine one while still reporting four.
• An ageing figure that counted finished work as backlog. Our review_queue rows store created_at in epoch milliseconds, and the weekly roll-up reporting how long editorial items have been waiting grouped by status and aged every group. But approved and rejected are terminal states: no code path rewrites created_at, and the approve/reject route only transitions rows WHERE status = 'draft' (src/lib/db.ts), so a decided row keeps its original timestamp for good. Read as backlog, it yields a real date, correctly formatted, in range, that gets one day older every day. Ours carried three settled rows as work sitting since 2026-06-23 across two successive roll-ups before anyone questioned it; all three targeted a single page that was already live. Nothing was null and nothing threw, so there was nothing for an alert to fire on — the fix was to narrow the population to status = 'draft'. Its loud sibling is worth naming as the contrast: feed those same millisecond values straight to datetime(…, 'unixepoch'), which reads its argument as seconds, and 13 digits fall outside the supported date range and return null. That one was caught while the query was being written, because null looks broken — it never shipped. The silent one ran for weeks.
• A registry that drifted 42 routes behind the artefact it generates. Our sitemap generator kept a hand-maintained page registry; public/sitemap.xml had grown past it by 42 URLs. Running the generator would have produced a valid, well-formed, correctly-signed sitemap that silently dropped 42 pages. There is no error state for that — the output is well-formed either way. scripts/build-sitemap.mjs now cross-checks the registry against the real route table in src/index.ts in both directions and fails on either mismatch.
• A render that failed by producing a black image. In our offline media pipeline, a background path outside the project directory failed silently and rendered black. Exit 0, file written, correct dimensions. The only usable signal was output size — roughly 100 KB against a normal ~500 KB, because a flat black frame compresses. When a job has no error channel, a cheap physical property of the artefact is often the only detector you have.

The common thread is that none of these were visible in latency, error rate, or cost — the three things the advice above tells you to watch. Each needed a check on the content of the output: does this pin exist upstream, is this age measured in the unit we think it is and over the rows that are actually waiting, does this registry match the route table, is this file the size a real one would be. Our own instrumented surface is the coach loop (src/lib/coach.ts), where per-turn token usage and cost are recorded and surfaced; the lesson from the four above is that the instrumented path was never the one that hurt us. Budget an assertion on every job that can succeed while being wrong.