An agent can return a polished answer and still fail the task. It may have selected the wrong tool, queried stale documents, retried a failing API until it hit a timeout, or retrieved the right policy and ignored it. A final-output test sees only the last symptom. Production debugging needs the path.
That is the premise of trace-based evaluation: evaluate an agent from a structured record of what it did, not only from its input and final text. The record is an execution trace: a causally connected sequence of operations such as model calls, retrieval, memory access, tool execution, and orchestration. The idea is also called telemetry-defined or framework-agnostic evaluation. The name is not standardized; the architectural move is what matters.
Amazon’s current implementation is a useful signal, rather than the definition of the concept. Its AgentCore Evaluations service can assess agents built with different frameworks when they emit compatible OpenTelemetry or OpenInference telemetry, reconstructing their activity from sessions, traces, and spans. AWS’s August 26 update names LangGraph, LlamaIndex, the OpenAI Agents SDK, Google ADK, Claude Agent SDK, and Strands Agents among the supported ecosystems.
Plain English: Instead of grading only “Was this answer good?”, preserve enough evidence to ask “What did the agent do, where did it go wrong, and does that failure recur?”
Why this matters now
Classic software testing works best when a function has bounded inputs and a predictable contract. Agent systems stretch both assumptions. A request may cause a model to choose among tools, search a knowledge base, call a service, store or retrieve memory, and then decide whether its result is sufficient. The model can introduce variation even when the surrounding code is unchanged.
This does not make conventional tests obsolete. You still need unit tests for argument validation, integration tests for service boundaries, and end-to-end checks for known workflows. But those tests alone do not describe the route an agent took in a real execution. Agent evaluation has to cover complete behavior—tool selection, multi-step reasoning, memory retrieval, and task completion—not simply response quality, as AWS argues in its discussion of real-world agent evaluation. Source
The important shift is an interface boundary. With framework-specific evaluation, a scoring system often depends on a framework’s callback objects or test APIs. Trace-based evaluation instead makes telemetry—machine-readable operational records—the boundary. If two runtimes emit compatible records, one evaluation pipeline can inspect both without embedding itself in either runtime.
That portability is increasingly valuable as teams mix frameworks, replace models, or split agent work across services. It also makes development and production evaluation look less like separate disciplines. The same structural record can support curated regression cases in CI and sampled live-traffic analysis, even though the evaluators and privacy rules should differ.
How the trace becomes an evaluation surface
A span is one timed operation inside a trace. A parent span can represent an overall agent invocation; child spans can represent a chat request, a retrieval step, a tool call, or a memory operation. A session groups related interactions over time. This hierarchy gives an evaluator both detail and context: a malformed tool argument is a local defect, while failure to complete the user’s task is a trace- or session-level outcome.
The records need common meanings. OpenTelemetry is an open observability framework for collecting and exporting traces, metrics, and logs. Its GenAI semantic conventions define shared attributes for model interactions, token usage, messages, tool calls, retrieval-related data, and evaluation scores. They also caution that message content can contain sensitive information. Specification Background
A practical pipeline works in four steps:
- Instrument meaningful operations. Emit structure for the agent invocation and for each model, retrieval, memory, and tool action. Preserve parent-child relationships, not merely a timestamped log line.
- Normalize meaning. Use semantic conventions where possible so downstream consumers can distinguish a tool execution from text generation without custom framework parsing.
- Reconstruct and score. The evaluator rebuilds the execution and applies an evaluator: deterministic code, human review, an LLM-based judge, or a combination. AWS says AgentCore Evaluations includes built-in measures such as goal success rate, correctness, and helpfulness, alongside custom LLM-as-a-judge evaluation. Source
- Compare versions and investigate failures. Join quality scores to latency, token use, errors, tool results, and the path taken. A regression then becomes inspectable rather than a mysterious drop in a single aggregate score.
Plain English: The trace is not the grade. It is the evidence that lets many different grading methods inspect the same run.
What this is not
Trace-based evaluation is not just distributed tracing with a new dashboard. Distributed tracing supplies the structure for linked operations. Evaluation adds judgments: did the agent use an allowed tool, retrieve relevant material, satisfy a policy, or finish the task correctly?
It is also not synonymous with LLM-as-a-judge. That is one technique in which a language model scores another model’s output against a rubric. It can be useful for qualities that are difficult to encode as assertions, such as helpfulness, but it remains model-dependent. The same trace can be checked with exact rules for tool arguments, reference-based checks for answers, and human review for disputed cases.
Nor does a common trace schema magically make results comparable. Two teams can emit similar spans but sample different traffic, use different references, define “success” differently, or change their scoring model. Version the instrumentation, evaluator prompt or code, dataset, sampling policy, and score definition together.
A realistic engineering example
Consider a hypothetical internal support agent. It receives “Can I enable this feature for a customer on the enterprise plan?” It must identify the account, retrieve the current entitlement policy, inspect account state through a tool, and explain the permitted next action. It must not make the change itself.
A useful trace begins with the incoming request and an agent-invocation span. Child spans show the model choosing a policy-search tool, the retrieval result, the account lookup, a later model response, and the final message. A deterministic evaluator can verify that no write-capable tool was invoked and that account lookup used a valid identifier. A retrieval evaluator can inspect whether a cited policy was actually among the retrieved material. A task-level evaluator can score whether the final answer matches an expected decision for a curated test case.
Suppose a prompt change raises apparent answer helpfulness but reduces entitlement correctness. The trace may show why: the model skipped policy retrieval and answered from general language knowledge. That diagnosis is substantially more actionable than “quality fell.” You can require retrieval for this class of request, adjust routing, or add a regression case that asserts the retrieval operation occurred.
For CI, run the curated cases and retain their traces. For production, sample runs and look for behavior that does not require a known reference: forbidden tool use, repeated retries, missing retrieval in regulated flows, latency outliers, or low judge scores. AgentCore Evaluations supports both on-demand evaluation for development and CI/CD and online evaluation of sampled production traffic. Source
Where it breaks
Instrumentation quality sets the ceiling. Missing spans, wrong parent-child links, or framework-specific fields that other consumers cannot interpret produce incomplete stories and misleading scores. A trace containing only the final answer is not enough to judge tool choice, retrieval quality, retries, or the cause of latency.
Privacy is equally fundamental. Prompts, completions, tool arguments, results, and conversation history can carry personal or confidential data. Capture only what evaluators need; apply filtering, truncation, access control, and retention policies before treating traces as an analytics dataset. Do not assume observability storage is automatically an appropriate place for raw customer content.
Live traffic has another hard limit: it often has no ground truth. An online evaluation can measure some signals without a reference answer, but reference-based correctness and assertion checks need expectations supplied through an appropriate workflow. AWS explicitly notes this constraint for online evaluation. Source
Finally, the contract is moving. OpenTelemetry describes its GenAI semantic conventions as under active development. Treat attributes and migrations as versioned dependencies, not settled infrastructure. Source
Plain English: Portable evaluation is possible only when the recorded execution is complete enough, safe enough to retain, and stable enough for your evaluators to understand.
What you can do this quarter
Start with one high-value workflow, not every agent. Write down the quality questions that the final answer cannot answer: Was a restricted tool called? Was retrieval required and used? Did the agent loop? Did it complete the task within a latency budget?
Then define a minimal trace contract. Include operation type, parent-child relationships, stable version identifiers, timing, outcome status, and carefully selected input/output metadata. Adopt relevant OpenTelemetry GenAI conventions where they fit, while keeping an explicit mapping for anything application-specific.
Build a small evaluation matrix. Put deterministic checks around safety and tool contracts. Use curated reference cases for correctness. Use an LLM judge only where a rubric is genuinely subjective, and calibrate it against human review before making it a release gate. Keep trace samples with failures so a future regression reproduces the execution pattern, not merely the final text.
Most importantly, make the trace a product interface. Frameworks, models, prompts, and tools will change. A well-versioned execution record gives your evaluation system a chance to survive those changes.