Evaluation-Driven Development: Agent Behavior as a Release Contract

What is evaluation-driven development for agents?

Evaluation-driven development is the practice of treating an AI agent’s behavior as a versioned release contract. You define representative scenarios, decide what acceptable behavior means, run those scenarios against a candidate change, and use the results in delivery decisions.

That sounds close to testing, and it is. The difference is the thing under test. A conventional function normally has known inputs and deterministic outputs. An agent combines a model’s probabilistic language generation with instructions, Tools, permissions, external data, and multi-step decisions. A harmless-looking prompt edit, model swap, credential change, or tool-schema update can change not just the final answer but which tool the agent chooses, which arguments it sends, and whether it completes the task at all.

Plain English: Don’t release an agent because its prompt looks sensible. Release it because it repeatedly handles the important situations you have defined well enough.

The current name is not a fully standardized methodology. You will also encounter “evaluation-first development,” continuous agent evaluation, and agent quality gates. The durable idea is simpler: make behavior observable, measurable, and reviewable before it reaches production.

That does not mean chasing one universal score. It means establishing a small, explicit set of promises for your application. A support agent might need to select the right account lookup tool and avoid unsupported policy claims. An internal engineering agent might need to identify the correct repository, stay within its authorized scope, and surface uncertainty instead of fabricating a conclusion. Those are software-quality properties, even where a traditional assertion cannot fully describe a good natural-language response.

Why this matters now

On September 8, AWS published a reference CI/CD pattern that deploys an agent and an MCP server, runs evaluation prompts, scores the behavior, and blocks pull requests when evaluation scores regress. The implementation is specific to Amazon Bedrock AgentCore and GitHub Actions, but the engineering pattern is portable: agent behavior can be a merge criterion rather than a manual demo performed after the merge.

The reason to care is not that every team needs another platform. It is that agents have widened the regression surface. In ordinary service development, a change to an endpoint can be checked against a contract. In agent development, a change can alter instructions, retrieval context, a model, orchestration logic, an MCP integration, permissions, or all of them together. Looking at a few chat transcripts catches obvious breakage but cannot tell you whether a known fragile case silently regressed.

AWS documents several lifecycle modes: on-demand evaluation for build-time testing, batch evaluation for baselines, before-and-after comparisons, regression tests, and audits, plus online evaluation of production activity. Its evaluation-types documentation describes these distinct modes. That separation is useful even if you use no AWS services. Fast, targeted checks belong near a pull request; broader and slower suites belong before promotion; sampled production sessions reveal failures your curated set did not anticipate.

Plain English: Evaluation is not a one-off benchmark. It is a feedback loop that starts in CI and continues after deployment.

How the loop works

Start with a scenario, a controlled task that represents behavior worth preserving or preventing. Each scenario should carry enough context to be rerun: user request, relevant fixture data, allowed tools, relevant permissions, and the expected condition. Cover normal work, awkward but valid requests, permission-sensitive operations, and bugs that have already occurred. A failure found in production is especially valuable: turn it into a versioned scenario once you understand it.

Next, run the candidate agent in an environment that is close enough to the runtime being assessed. “Close enough” matters. If CI has broader permissions, different tool behavior, or different external data than production, a passing result proves less than it appears. Capture the interaction as a trace: structured evidence of the agent run. Depending on the system, this can include a complete session, intermediate steps, and tool-call spans, which are individual recorded tool invocations. AgentCore can target sessions, traces, or individual tool-call spans for evaluation. AWS documents those evaluation targets.

Then apply more than one kind of evaluator. The strongest release checks tend to be deterministic. A code-based evaluator can verify that an argument has an allowed value, a required policy condition is present, an identifier matches a format, or an agent did not invoke a prohibited operation. AgentCore’s documentation says code-based evaluators can implement deterministic checks, external API calls, regular-expression matching, and business-specific rules. See the evaluator documentation.

For tasks with a known acceptable answer or action sequence, add ground truth: explicit reference material against which a run can be checked. AWS’s evaluation API supports expected responses, assertions, and expected tool trajectories. That allows checks of both correctness and behavior. An expected trajectory should not mean “there is only one valid route” when several routes are genuinely safe. Use it when the required or forbidden action is materially important.

Some qualities cannot be reduced cleanly to an exact assertion. An LLM-as-a-judge evaluator uses a model and a written rubric to assess a response or interaction. It can help assess whether an explanation is useful, whether a response follows a policy, or whether a multi-step result addresses the request. Reference-free judging relies on the evaluator model’s own assessment rather than ground-truth data, so it is useful for scalable qualitative signals but is not the same thing as a proof of correctness. AWS makes that distinction explicitly.

Finally, aggregate the results into a clear quality gate. A gate may require every safety-critical scenario to pass, prohibit a regression on specific dimensions, and allow a non-critical semantic score to remain advisory while it is being calibrated. Keep per-scenario results visible. A global average can stay healthy while a single high-risk task gets worse.

Plain English: Use exact checks for promises that must not break. Use model-based judging for qualities humans recognize but code cannot describe neatly—and do not confuse that judgment with certainty.

What it is not

Evaluation-driven development is not ordinary unit testing with a model bolted on. Unit tests remain essential for deterministic code: authorization logic, parsers, validators, tool implementations, and state transitions should still be tested directly. Agent evaluation extends the surface to language-mediated decisions and execution paths where the result may vary while still being acceptable.

It is also not merely observability. Observability tells you what happened through logs, telemetry, traces, and inspection. Evaluation adds a judgment about whether what happened met a requirement. You need both: a low score without a trace is hard to diagnose, while a trace without an explicit criterion is easy to admire and hard to act on.

Nor is it a generic benchmark. A benchmark usually compares systems on a fixed task collection. Your suite should instead reflect your users, policies, tools, and incidents. It should evolve with the application. And it is not an agent Harness, although a Harness may provide the controlled tools, fixtures, simulated environment, and inputs needed to run tests. The Harness executes the experiment; the evaluation policy decides what counts as a pass.

A practical engineering example

Consider a hypothetical internal incident-triage agent. It can read alerts, query a service catalog, retrieve recent deployment information, and draft an incident update. It must not make production changes. A team changes the system instructions and replaces one catalog integration.

A useful CI suite would not ask only, “Was the final update well written?” It could include a scenario where an alert names an ambiguous service. The expected condition is that the agent queries the catalog before attributing ownership. A deterministic evaluator checks that it does not call any write-capable tool and that every tool argument conforms to the schema. A ground-truth assertion checks that the response names the verified owning service, or explicitly states that ownership remains unresolved. A judge-based rubric scores whether the draft separates observed facts from hypotheses.

Another scenario can reproduce an earlier mistake: the deployment system returns no matching release. The acceptance criterion is not a fluent invented explanation. It is that the agent reports the missing evidence, avoids a false causal claim, and proposes an allowed next step. The trace tells reviewers whether the fault was retrieval, tool selection, argument construction, model reasoning, or an unexpected permission response.

In CI, run the compact set for every pull request. Before a production promotion, run a broader batch against stable fixtures and compare it with a baseline. After deployment, sample real sessions with appropriate privacy and access controls, inspect low-scoring flows, and promote recurring failure patterns back into the suite. AWS’s online evaluation documentation describes sampling, filtering, aggregate trends, and investigation of low-scoring sessions. See its online-evaluation guidance.

Where it breaks

A green suite is evidence about the scenarios it contains, not a certificate of general correctness. It can miss novel requests, adversarial instructions, unusual tool interactions, external-service faults, and distribution changes in production. A suite that never tests an awkward permission boundary cannot tell you whether that boundary is safe.

Judge-based scores add another failure mode: the judge has its own model behavior, prompt sensitivity, and rubric interpretation. Calibrate it against reviewed examples. Keep a set of human-reviewed cases, look for disagreement, and avoid making a single opaque judge score the only release blocker for a critical action.

Cost and time are real constraints. Deploying dependencies, running inference, calling tools, and collecting traces make agent suites slower than a normal unit-test run. Tier the suite rather than abandoning it: small mandatory checks per pull request, broader regression runs at promotion, and sampled evaluation online. Also treat flaky external dependencies as an engineering problem. Separate a genuine behavioral failure from unavailable test data, changed credentials, or a degraded provider.

What you can do this week

Choose one agent workflow that has real user or operational impact. Write ten to twenty scenarios from support tickets, incidents, and expected happy paths. For each, identify one condition that must be deterministic: a prohibited tool, a required permission check, a mandatory data source, or a response field that must be present.

Add trace capture around the agent and its tools. Put a small scenario subset into CI first, with results attached to the pull request. Do not begin with a complicated scorecard. Make one or two critical conditions merge-blocking, retain the evidence for failures, and use other measurements as visible but non-blocking signals while you learn their behavior.

Then make the suite part of normal maintenance. Every escaped defect should lead to a question: can we represent this as a scenario, an assertion, a fixture, or a Tool boundary check? Over time, the suite becomes a record of what your agent is allowed to do and what it must reliably avoid.