A dashboard shows that model requests are fast, yet users complain that the assistant takes too long. A billing chart looks stable, yet one unresolved task keeps consuming tool calls. Both problems arise when operations are measured below the level of the user’s actual job.
The useful unit is the run: a request with an objective, a lifecycle, a resource budget, and an outcome. Model calls are only part of it. Retrieval, queues, retries, tools, approval waits, and reconciliation all contribute to the experience.
Trace the work from entry to outcome
Give each run a stable identifier. Create child spans for meaningful stages: input validation, retrieval, model inference, tool execution, and response assembly. Preserve relationships across queues and worker handoffs.
OpenTelemetry describes traces as the path of a request through services and spans as units of work within that trace. Those concepts provide a useful structure for following agent work across component boundaries. OpenTelemetry traces
A run can outlive an HTTP request. Return a run reference when work continues asynchronously, and retain its state after the initial connection closes. Otherwise the frontend may call the task failed while a background worker continues changing things.
Record decisions and outcomes, not hidden reasoning
Useful attributes include workflow version, model identifier, tool-contract version, operation identifier, attempt count, terminal state, and references to evidence or approved actions. Record the observable decision: which tool was requested and why the application allowed or rejected it.
Do not require private chain-of-thought to debug a workflow. Inputs, outputs, validated proposals, policy decisions, and external receipts are concrete artifacts. They are also easier to connect to application behavior.
Avoid putting raw documents, credentials, or complete user messages into every span. Decide which content may be retained, who can inspect it, and when it expires. A source reference with access-controlled retrieval can be more useful than another unrestricted copy of sensitive text.
Keep high-cardinality identifiers in traces or structured logs rather than unbounded metric labels. A per-run metric series can make the monitoring system expensive while making aggregate behavior harder to see.
Separate different kinds of time
Measure end-to-end elapsed time, active execution time, queue delay, external service time, and waiting for human input. An approval wait is real user-visible time, but treating it as model latency sends optimization work to the wrong component.
For interactive responses, distinguish time to first useful feedback from time to final completion. A fast acknowledgement is valuable only if it honestly describes ongoing work. Do not show “complete” when the operation merely entered a queue.
Inspect distributions and failure groups rather than relying only on averages. A small number of very slow investigations may dominate the user experience. Compare like tasks: a document summary and a multi-system investigation do not have the same expected work.
Budget before dispatch, reconcile afterward
Set limits for elapsed time, model calls, tool calls, output size, and estimated spend. The appropriate limits depend on the product; do not adopt arbitrary numbers as universal defaults.
A coordinator should reserve budget before starting concurrent work. If several workers independently check the same remaining allowance, each can conclude that its request fits while their combined requests exceed it.
After completion, reconcile the reservation with actual provider usage where available. Keep the pricing version and unit interpretation used for estimates. Token estimates, cache treatment, tool charges, and provider billing records can differ; label estimates as estimates.
For money, use explicit units and appropriate numeric representations. Repeated floating-point addition of fractional currency can introduce accounting inconsistencies. A budget mechanism should also specify how it handles missing usage reports rather than silently charging zero.
Work through a bounded investigation
Consider an assistant investigating a failed deployment. It retrieves logs, reads a runbook, asks a model to choose the next read-only check, and drafts a diagnosis.
Give the run a total deadline and a bounded number of investigations. Reserve capacity for producing a final status report before spending the remaining budget on another search. If the last tool call finds a useful clue, the system should still have enough allowance to explain it.
When a call would exceed the budget, stop before dispatch and record the reason. Return the evidence already gathered, unresolved questions, and a clear continuation option. Continuing should be an explicit new budget decision, not an automatic reset hidden in another model turn.
If a write was already dispatched, reaching the budget does not make its outcome disappear. Move that operation to reconciliation if necessary. The retry guide explains why timing out locally cannot prove a remote effect was canceled.
Distinguish expensive progress from expensive repetition
A long run can be justified when each step resolves uncertainty. A cheap run can still be useless when it repeats the same failed lookup. Record enough structure to identify repeated tool arguments, unchanged evidence, and cycles between workflow states.
Use these signals to select an intervention: stop an unproductive loop, improve a tool description, add a missing retrieval source, or route the task to a human. Do not assume every inefficiency requires a smaller model.
Compare cost per useful outcome, not only cost per request. Include retries and abandoned runs in the denominator you report. Define “useful” with task-specific evidence, such as a supported diagnosis or an accepted draft, rather than equating successful HTTP responses with successful work.
Alert on states someone can act on
| Signal | Actionable question |
|---|---|
| Growing queue age | Is capacity sufficient for accepted work? |
| Increasing uncertain writes | Can remote outcomes be reconciled? |
| Budget exhaustion concentrated in one task | Is the plan too broad or a dependency failing? |
| More permission denials | Did a contract, role, or routing rule change? |
| Missing terminal run states | Can workers recover or does work disappear? |
Attach the run and operation references needed for investigation. Choose an owner for unresolved work; an alert without a recovery path merely transfers confusion to someone else.
Instrument the smallest useful workflow first, then expand observability with new capabilities. The goal is to answer what the assistant attempted, what it actually accomplished, what it consumed, and what still needs attention. Those answers make reliability work concrete and make budget decisions defensible.