AI Agent Observability: What to Log for Every Run
When an agent misbehaves, the difference between a five-minute diagnosis and a day of guessing is what you recorded at runtime. Agents are non-deterministic, so reproduction is never guaranteed — the trace of the original run is often the only evidence you will ever have. Here is what that trace has to contain.
Traces, not log lines
Application logging habits undersell agents. A scatter of log lines tells you the run happened; a trace tells you what the run was — a chronological, structured record of every LLM call and tool invocation with inputs, outputs, and timing, navigable as one object. The unit of debugging is the run, so the unit of recording must be the run too.
In Dezifi every execution — production, test, agent, or workflow — lands in Monitor with a one-click path into its full trace. That uniformity matters more than it sounds: the trace you inspect while building is the same artifact you will pull up during an incident, so you learn to read it before you need it under pressure. If your stack treats test runs and production runs differently, you are practicing on the wrong instrument.
Record every LLM call in full
For each model call, capture the resolved prompt — after templates, retrieved context, and memory are spliced in — plus the raw completion, the model and version, token counts, and latency. The resolved prompt is the piece teams skip and regret: most “the agent went weird” investigations end at a context window that contained something nobody expected, and without the resolved prompt you cannot see it.
Model identity matters for a subtler reason: providers revise models under stable names, and behavior shifts arrive silently. When quality moves and your traces pin every run to a model version, correlation takes minutes. This is also the raw material for evaluation — a failing run whose full context is preserved can be replayed against a candidate fix; one without is just an anecdote.
Tool spans are your side-effect ledger
Tool calls are where agents touch the world, so they deserve the strictest recording: the tool and action invoked, the exact arguments, the response or error, and duration. Together these spans form a ledger of side effects — the authoritative answer to “what did the agent actually do?”, which during an incident matters far more than what it said.
Two disciplines keep the ledger trustworthy. Record arguments as sent, not as the model first drafted them — if a guardrail redacted a field in between, the trace should show the redacted version that hit the API. And record failed calls with the same fidelity as successes: a sequence of malformed retries against a tool is usually the first visible symptom of a prompt regression, and it is invisible if you only log outcomes.
Governance events belong in the trace
Policy checks and guardrail inspections are decisions, and decisions belong in the run record. When a guardrail fires, the trace should show which rule matched, the matched span, and the action taken — blocked, redacted, or logged. When a policy halts a run, the violation event should carry the policy and rule id and the payload that triggered it.
This serves two audiences at once. Engineers debugging a “stuck” agent discover in seconds that a PII scanner redacted the field the next step depended on — a class of bug that is maddening without governance events inline. Compliance teams get an audit trail showing controls firing on real traffic, which converts “we have guardrails” from a claim into evidence. Approval decisions complete the picture: who reviewed, what they saw, what they chose.
Cost, and closing the loop
Record dollar cost per run from day one — token spend plus tool calls, visible on every row. Per-run cost is how you catch the retry loop that burns budget overnight, and it is the number that makes pilot economics credible when someone asks what scale costs. Aggregates hide exactly the outliers you need to see.
Finally, treat observability as the intake for quality work, not a terminal archive. The failing runs you find while filtering Monitor are your next eval dataset — Dezifi makes this literal with Open in Eval on every run detail. And check your retention against your incident reality: runs are retained 90 days on standard plans, a year on enterprise, with export via the Runs API beyond that. The question arriving in month four should not meet an empty archive.