Agent Observability: Log, Trace, and Read a Run After It Fails

Every other guide in this library assumes something this one supplies: a record of what the agent actually did. You cannot diagnose a tool call failure you did not capture, price a run you did not measure, or evaluate output you cannot retrieve. Yet most agent setups keep almost nothing - a final answer, maybe an error message, and a monthly bill. When a run goes wrong, the team is left interviewing the model about its own behaviour, which is roughly as reliable as it sounds. Observability is the discipline of recording runs well enough that questions about them have factual answers.

The good news is that the recording is mostly free. The data already passes through your code: every model response reports how many tokens it used, every tool call returns through your harness, every step starts and ends on your clock. Observability is deciding, in advance, to write those facts down in a shape you can query later - and deciding what to leave out, because a run record can just as easily become a liability as an asset.

The unit of record is the run, and the run is a tree

A chatbot exchange is a line: prompt in, answer out. An agent run branches and loops. The model calls a tool, reads the result, calls another tool or the same one again, maybe hands work to a second agent, and only then answers. Recording that as a flat list of log lines loses the one thing you most need later: which step produced which. The standard shape, borrowed from distributed tracing in ordinary software, is a tree of spans. One span represents the whole run. Under it, a span for each model call and each tool execution, each with a start time, a duration, a parent, and a set of attributes. Read top-down, the tree is the run's table of contents. Read at any node, it answers what happened here, what it cost, and what it fed into.

This shape is no longer a matter of taste. OpenTelemetry, the vendor-neutral instrumentation standard most tooling now builds on, publishes Generative AI semantic conventions that name exactly these spans: an invoke_agent span for the run, chat spans for model calls, and execute_tool spans for tool executions, with shared attribute names for the model used, the input and output token counts, and the tool name. The conventions are still marked as under development, so individual attribute names can shift between versions; pin the version you build against and expect to adjust. The shape, though, has settled: if your records follow the tree, tools that speak OpenTelemetry can read them, and you are not locked into whichever dashboard you picked first.

invoke_agent  research-brief            41.2s  outcome: completed
  chat  model call 1                     3.1s  in 1,240 / out 386 tokens
  execute_tool  search                   1.8s  status: ok
  execute_tool  fetch_page               2.4s  status: ok
  chat  model call 2                     4.0s  in 6,930 / out 512 tokens
  execute_tool  fetch_page               0.3s  status: error (timeout)
  execute_tool  fetch_page (retry)       2.2s  status: ok
  chat  model call 3                     5.6s  in 11,480 / out 1,940 tokens

Even in this compressed shape, the tree tells a story in seconds: three model calls, input tokens growing as the history accumulates, one tool failure absorbed by a retry, and a final call that did the heavy writing. A flat log of the same run holds the same facts in principle and hides every one of them in practice.

The run record: what to capture every time

Capture the same fields for every run, whether it succeeded or not. A record that exists only for failures cannot tell you whether this failure is new, and a record that exists only for successes is marketing. The checklist below is the minimum that pays for itself; each field is on it because some real question needs it.

• Run id and task identity: a unique id per run, the workflow or task name, and the version of the instructions it ran under. Without the version, you cannot tell whether a change helped.
• The model for each call, with input and output token counts as reported by the provider's response. This is the raw material of every cost calculation and most slowness investigations.
• Each tool call: the tool name, the arguments as sent (or a redacted form; see below), the status, the error class if it failed, and the size of the result. The arguments matter - most tool failures are argument failures.
• Timing per span, not just per run. A run that took four minutes spent them somewhere, and the total does not say where.
• Retries and how they ended. A step that failed twice and then succeeded is a warning, and it is invisible in any record that only keeps the final state.
• The outcome in the workflow's own terms: completed, stopped by a limit, escalated to a human, abandoned. Plus the human verdict where one exists - accepted, edited, rejected - because that field is what turns records into evaluation data.
• Cost attribution: the run's total token and tool cost, computed from the recorded counts. Cheap to store at write time, painful to reconstruct later.

Logs, traces, and metrics answer different questions

Teams often pick one of the three and then wonder why it serves them badly. They are three views of the same records, and each is good at one kind of question. Logs are the raw events: searchable text for what happened in this run, on this input, at this time. Traces are the structured tree: the shape of one run, and the tool for following cause and effect across steps and services. Metrics are the aggregates: counts and rates over many runs - cost per run this week against last week, tool error rates by tool, the share of runs escalated. A question about a single run goes to the log or the trace. A question with the words usually, trending, or how often in it is a metrics question, and no amount of staring at individual runs answers it. Build the records once, in the span shape, and all three views are derivations of the same stored facts.

Reading a failed run: a short walkthrough

Consider a hypothetical failure, invented to show the reading order. A nightly workflow that summarizes new support tickets stops producing summaries; the final answer field is empty and no error was raised. Start at the root span: the run completed, in about its usual time, which rules out the dramatic failures. Move to the outcome attributes and the last model call: its output was short and its finish state normal. Look at its inputs next - in the tree, that means the tool spans feeding it. The retrieval span returned a result of size zero with a status of ok. That is the fault line: the tool succeeded at returning nothing, the model summarized nothing into a polite empty answer, and every downstream step behaved correctly on empty input. The fix belongs at the retrieval step (an empty result should be an error class of its own) and at the run level (an empty final answer should fail the outcome check). Neither fix is findable from the final answer alone, and both are obvious from the tree. This is the ordinary shape of agent failures: rarely a crash, usually one quiet step passing an empty or malformed result to a model that papers over it.

What never goes in the record

A run record is a copy of your most sensitive traffic, and it deserves the suspicion you would give any such copy. Three rules cover most of the risk. First, credentials never appear: not API keys, not authorization headers, not the secret values inside tool arguments. Redact by field name before writing, not by pattern-matching after. Second, personal data gets a decision, not a default. Prompts and tool results routinely contain names, addresses, and account details; store the minimum the debugging purpose needs, which is often a truncated or hashed form, and keep full content only where a human will genuinely read it. Third, retention is finite. Content-level records should expire on a schedule you choose in advance, while the counts, timings, and outcomes - the parts with lasting analytical value and little sensitivity - can be kept indefinitely. The OpenTelemetry conventions treat message content as opt-in for the same reason: the standard assumes you capture content only when you have decided to, not because the pipe made it easy.

How much observability is enough

Start with the boring version and let failures argue for more. Structured records written to files or a database table, in the span shape, with the checklist fields above, cover a surprising share of real needs: any single run can be found and read, costs can be summed, and error classes can be counted with ordinary queries. Add sampling when volume makes full content storage expensive: keep counts and outcomes for every run, full content for failures and a slice of successes. Reach for a platform when the questions turn aggregate and frequent, or when finding a run in the files stops being a two-minute job. What a platform buys is not the records - you were already writing those - but the aggregation, search, and alerting on top of them, plus evaluation tooling that scores outputs against the traces you captured. Langfuse, profiled in this site's catalog, covers tracing, prompt management, and evaluation and can be self-hosted as open source software; LangSmith is the managed option from LangChain; Helicone takes a proxy approach that logs requests with minimal setup. The catalog profiles linked below carry the setup and limits for each, checked against primary sources.

Where this fits in the library: the production harness checklist treats logging as one control among several and gives the audit questions; this guide is the design underneath that line item. Diagnosing tool call failures assumes you can see the failing call and its arguments - the run record is what makes that true. Evaluating agent output quality needs runs you can retrieve and replay, and the ROI guide needs per-run cost, which falls out of the same record. One capture discipline, four uses. Build it once, early, and the rest of operating an agent gets measurably easier; skip it, and every other practice in this library is running on memory and vibes.

Sources and further reading

• OpenTelemetry, Generative AI semantic conventions (opentelemetry.io/docs/specs/semconv/gen-ai and the GenAI conventions repository): the invoke_agent, chat, and execute_tool span model, the gen_ai attribute names for model, token usage, and tool calls, and the development status of the conventions.
• Langfuse, documentation (langfuse.com/docs): tracing of LLM and non-LLM calls, sessions, and evaluation on captured traces, on an OpenTelemetry base, from the platform's own docs.
• Anthropic, Messages API reference (docs.anthropic.com/en/api/messages): the usage object returned with every response, reporting input and output token counts per call - the source of the token fields in the run record.