The Trajectory Is the Test: Critical User Journeys and Layered Gates for Production Agent Evaluation

Why the Final Answer Is Not Enough

An agent can return a polished, correct-sounding answer while having skipped a required verification, called the wrong tool, looped, or performed an irreversible side effect. The text looks fine. The system state does not.

This is the central failure mode of production agents: silent process errors that surface only as higher cost, duplicate actions, user distrust, or compliance issues. Google Cloud’s methodical evaluation work and recent practitioner write-ups (including Google’s October 2026 guidance on critical user journeys) make the same point: grade the trajectory, not the closing pitch.

A trajectory is the ordered sequence of reasoning steps, tool calls, parameters, intermediate results, and final outcome. Evaluating it requires capturing traces (OpenTelemetry-style spans from frameworks or observability tools) and scoring both process and result.

Start With Critical User Journeys, Not Abstract Benchmarks

Before writing metrics, define success for the actual people who will use the agent.

A critical user journey (CUJ) states:

• Persona: who is acting.
• Goal: the concrete outcome they need.
• Constraints: time, policy, data, cost, or safety limits.
• Verification conditions: the observable checks that prove the goal was met correctly.

Example (adapted from recent Google Cloud practitioner guidance):

As an operations analyst, I expect the agent to ingest a counterparty trade-discrepancy email, identify the disputed transaction, verify the trade economics against our internal ledger within 30 seconds, and draft a response that reverse-maps our internal trade IDs to the broker’s nomenclature without human intervention.

Every clause becomes a testable point: correct transaction identified, ledger check performed before drafting, IDs mapped accurately, time bound respected.

Vague goals (“be helpful,” “book a trip”) produce vague tests. Specific CUJs produce the golden paths and edge cases you will actually run.

Four Practical Layers of Checks

Effective suites combine speed, determinism, and judgment. Recent frameworks (DeepEval trajectory metrics, LangSmith/agentevals trajectory match, Arize Phoenix tracing + judges, Google’s agent-eval patterns) converge on similar layers:

1. Unit / component tests — Fast, deterministic checks on individual tools, parsers, and prompt builders. Example: given a sample email, does the extraction function return the expected transaction ID?
2. Deterministic trajectory metrics — Exact or mode-based assertions on the tool-call sequence: required tools called, forbidden tools not called, parameter schema valid, latency or step-count budgets respected. LangChain’s trajectory match modes (strict, unordered, subset, superset) are a clean example of this.
3. LLM-as-judge with boolean rubrics — For qualitative conditions (“Did the agent verify the ledger before drafting?”). Prefer binary pass/fail questions over 1–5 scales. Calibrate the judge against a small human-labeled set so it does not systematically disagree with domain experts.
4. Outcome / end-state checks — Did the external system reach the expected state (reservation exists, ledger updated, ticket created)? Combine with cost and escalation-rate metrics.

Cost, step count, and unsafe-action counts belong in the same scorecard. A correct answer that burned 4× the expected tokens or skipped a required human approval fails the suite.

From Golden Path to CI Gate

Start small:

• Write 10–30 CUJ-derived scenarios covering happy path, common edges, and known failure modes.
• Capture traces on every run.
• Score with a mix of the four layers above.
• Set explicit thresholds (e.g., task success ≥ baseline, zero unsafe tool calls, cost per task under limit).

Wire the suite into CI so a prompt, model, or tool-schema change cannot merge if scores regress. DeepEval’s pytest integration and similar patterns in LangSmith/Phoenix make this straightforward: failing metrics fail the test, which fails the build.

For longer or multi-turn agents, include both single-step decision tests (cheaper, more targeted) and full trajectory runs. Use repeated trials or controlled mocks for non-deterministic tools so comparisons stay meaningful.

Production Sampling and the Feedback Loop

Offline suites catch regressions before deploy. Production still drifts.

• Sample a small percentage of live trajectories (often 3–10% is enough to surface issues within a day).
• Score them with the same metrics or a lighter online subset.
• Alert on drops and route low-scoring traces to human review.
• Periodically promote real failures and novel successes into the offline golden set (anonymized).

This is the virtuous cycle described across Google Cloud, LangChain, and observability tooling write-ups: production findings become the next regression tests.

A Minimal Practical Checklist

Use this as a starting gate for any agent that can take external actions:

• At least one CUJ written with persona, goal, constraints, and verification conditions.
• Trace capture on every evaluation run (tool calls, parameters, latency, cost).
• Deterministic checks for required/forbidden tools and parameter validity.
• At least one boolean LLM-judge rubric calibrated on a small human set.
• Explicit cost and step budgets.
• CI job that blocks merge on regression below threshold.
• Sampling + alert path for production traces.
• Process to promote interesting production cases back into the offline suite.

You do not need a perfect golden dataset on day one. Binary scores from early human review plus deterministic trajectory checks are enough to start gating. Expand as you learn the real failure modes.

References & Community Insights

• Google Cloud: A methodical approach to agent evaluation — success statements, three pillars (quality, trajectory, trust/safety), and operationalization into CI and production monitoring.
• Recent practitioner coverage of Google’s critical user journeys guidance (October 2026) and the open-source agent-eval toolkit patterns.
• LangChain Docs: How to evaluate your agent with trajectory evaluations — strict/unordered/subset/superset match modes.
• DeepEval: Trajectory-based metrics (TaskCompletion, StepEfficiency, PlanAdherence) and pytest CI integration.
• Arize AI workshop (Laurie Voss, October 2026): practical tracing + code evals + LLM judges + CI gates with Phoenix, using a real “confidently wrong” example.
• Supporting context from Lumenalta, Open Data Science, and ContextQA on release gates, regression suites, and the gap between demo success and production reliability.

The trajectory is the test. Final answers are necessary but never sufficient.