Stop Asking for the Best Model: Build a Policy-Aware Router for Agent Workflows

A production agent rarely needs one model. It needs a model portfolio.

A lightweight classifier may be ideal for intent detection. A long-context model may be necessary for document synthesis. A reasoning model may be justified for a high-impact decision. A private deployment may be mandatory when the request contains regulated data. A fast fallback may be better than a perfect answer when the user’s latency budget is already exhausted.

The mistake is to encode this as one global model choice or one undifferentiated fallback chain. In an agent workflow, every planning step, retrieval step, tool call, and final response can have a different capability profile. A model that is excellent at planning may be wasteful for formatting. A model that is cheap for a short answer may fail when it must emit a valid tool call. A nominally available fallback may violate the user’s data-residency policy.

Model routing is not primarily a cost optimization. It is a policy and SLO decision made at every agent boundary.

Why model selection becomes a systems problem

For a single chat completion, model selection looks like a quality-versus-price decision. For an agent, the call is one step in a trajectory. A bad choice can cause an invalid tool call, a retry, a longer context, a human escalation, or a partially completed workflow. The cost of a model is therefore not only its token price; it includes the downstream work created by failure.

The router should consider at least these dimensions:

Dimension	Routing question
Capability	Does this step need reasoning, coding, vision, multilingual quality, or reliable tool use?
Sensitivity	Which providers and regions are allowed to see this data?
Context	Will the request fit safely, or should it be summarized, retrieved, or sent to a long-context model?
Latency	What p95 time-to-first-token and completion budget does this step have?
Availability	Is the deployment healthy, within quota, and below its queue threshold?
Cost	What is the expected cost per successful task, including retries and fallbacks?
Risk	What is the harm of an incorrect, overconfident, or unsafe output?
Continuity	Does the same session need model affinity or a compatible reasoning state?

The current research literature frames routing as a utility function over model performance and cost, with performance varying by task type. Task-aware routing research such as TRouter introduces a latent task profile so the router can predict which model is likely to perform well for a particular kind of request instead of relying on a single global ranking. 1

The vLLM Semantic Router project makes a related point through its Workload–Router–Pool architecture: the routing policy, the workload mix, and the inference pool are coupled. A policy that works for short chat requests may be poor for long-running agents, multimodal inputs, or tool-heavy sessions. 2

Use a decision gate before calling any model

The first version of a router should be explicit and inspectable. Do not start with a learned classifier that nobody can explain. Establish policy rules first, log the decisions, and only then use the resulting production data to train or tune a more adaptive component.

A practical order is:

1. Data policy: classify sensitivity, tenant restrictions, region, and approved providers.
2. Task policy: classify the step as extraction, planning, retrieval synthesis, code, tool selection, or final response.
3. Context policy: estimate input and expected output size; decide whether to retrieve, summarize, or use a long-context route.
4. SLO policy: filter models that cannot meet the step’s latency or availability target.
5. Quality policy: select the least expensive remaining model that meets the measured quality threshold.
6. Recovery policy: define which failures justify retry, cross-provider fallback, degradation, or human review.

Represent the decision as data rather than scattering if/else branches throughout agent code:

{
"request_id": "run_123.step_04",
"tenant": "acme",
"step_kind": "tool_argument_generation",
"sensitivity": "confidential",
"region": "eu-west",
"estimated_input_tokens": 6200,
"latency_budget_ms": 4500,
"quality_floor": 0.92,
"allowed_capabilities": ["function_calling", "structured_output"],
"allowed_model_groups": ["private-reasoning-eu", "approved-hosted-eu"]
}

This policy object is also the explanation you need later. If a customer asks why a model handled a request, you should be able to answer with the data classification, task type, model health, policy version, and measured quality threshold—not “the router picked it.”

Route by agent step, not just by conversation

MLflow’s current routing guidance points out that each tool call inside an agent loop is its own routing decision. Planning often benefits from a stronger model, while execution can use a smaller model when the required behavior is constrained. Treating an entire agent run as one opaque model call hides where latency and spend accumulate. 3

A useful portfolio might look like this:

Agent step	Default route	Escalation condition
Intent classification	Small, fast structured model	Ambiguous confidence or unsupported language
Retrieval query rewrite	Small model with strong instruction following	Multiple failed retrieval attempts
Evidence synthesis	Medium long-context model	Conflicting sources or high-risk domain
Plan generation	Strong reasoning model	Multi-tool dependency graph or irreversible action
Tool arguments	Model with proven function-calling accuracy	Schema repair, repeated validation failure
Final response	Fast model with grounded context	Missing evidence, sensitive claim, or low confidence

The route should be allowed to change between steps, but the policy must preserve continuity where it matters. A fallback that cannot interpret the previous provider’s encrypted reasoning state or structured tool format may create a new failure. Store a normalized, provider-neutral state envelope at step boundaries.

Separate retries, fallbacks, and circuit breakers

These mechanisms solve different problems:

• Retry: the same model group may succeed after a transient timeout or rate limit.
• Fallback: a different approved model group handles the request after the primary path is exhausted.
• Circuit breaker: the router temporarily stops sending traffic to a deployment that is repeatedly failing.
• Degradation: the business workflow deliberately reduces scope, quality, or freshness instead of failing outright.
• Escalation: the request is routed to a stronger model or human when the quality or risk floor cannot be met.

LiteLLM documents these as separate controls: deployment order, retries, timeouts, cooldowns, fallbacks, and context-window pre-checks. It also records the original model group and fallback attempts in spend logs, which is essential for understanding the actual cost of recovery. 4

A simple policy file can make the separation concrete:

routes:
classify:
primary: fast-structured
retry:
max_attempts: 2
backoff_ms: 250
fallback: approved-fast-eu
on_failure: continue_with_low_confidence

plan:
primary: strong-reasoning
retry:
max_attempts: 1
backoff_ms: 800
fallback: private-reasoning-eu
on_failure: require_human_review

tool_call:
primary: function-calling-medium
retry:
max_attempts: 2
backoff_ms: 300
fallback: function-calling-private
on_failure: stop_before_side_effect

circuit_breakers:
error_rate_threshold: 0.15
consecutive_failures: 3
cooldown_seconds: 45

budgets:
per_run_usd: 0.50
per_step_latency_ms: 6000

Never let a generic fallback bypass a data policy. “Any available model” is not a safe recovery rule. The fallback graph should be filtered by region, contract, capability, tenant, and risk before health or price is considered.

Quality floors are better than model rankings

A static ranking such as “Model A is best, Model B is second” decays quickly. Model performance depends on task distribution, prompt version, retrieval quality, tool schema, provider load, and output constraints.

Instead, define a quality floor per task. For example:

from dataclasses import dataclass

@dataclass(frozen=True)
class Candidate:
name: str
quality: float
p95_ms: int
cost_usd: float
private: bool
supports_tools: bool
healthy: bool


def choose(candidates, *, quality_floor, latency_budget, private, needs_tools):
eligible = [c for c in candidates if (
c.healthy
and c.quality >= quality_floor
and c.p95_ms <= latency_budget
and (not private or c.private)
and (not needs_tools or c.supports_tools)
)]
if not eligible:
return None
return min(eligible, key=lambda c: (c.cost_usd, c.p95_ms))

The example is intentionally simple. It makes a crucial distinction: price is optimized only after the candidate satisfies policy, capability, quality, and SLO constraints.

Build the quality estimate from representative production tasks. MLflow recommends comparing candidate routes on the same real queries and tracking route-level cost, p95 time-to-first-token, fallback rate, cache hit rate, and task-specific quality. It also recommends rule-based routing and semantic caching before introducing a predictive classifier, because the logged decisions become the labeled evidence needed to train and recalibrate the next stage. 3

Fallbacks can silently lower quality

A healthy fallback is not necessarily an equivalent fallback. If the primary model fails during a planning step and the fallback can only produce a shallow answer, the workflow may continue while making a worse decision. This is more dangerous than a visible error because the system looks available.

Record every route transition in the same trace as the agent step:

{
"run_id": "run_123",
"step_id": "step_04",
"requested_group": "strong-reasoning",
"attempts": [
{"model": "provider-a/strong", "error": "429", "latency_ms": 2100},
{"model": "provider-b/strong", "error": "timeout", "latency_ms": 6000},
{"model": "private-reasoning-eu", "status": "ok", "latency_ms": 3300}
],
"fallback_count": 2,
"quality_check": {"score": 0.94, "floor": 0.92},
"policy_version": "router-2026-10-15.3"
}

A rising fallback rate is an early warning. It may signal provider instability, quota pressure, a changing traffic mix, or a cheap model that no longer meets the task distribution. A falling average cost can hide a rising review rate or an increase in failed agent trajectories.

Measure cost per successful business task, not only cost per request. Include retries, fallback tokens, tool calls, cache misses, infrastructure, and human review. A more expensive model that finishes in one reliable step can be cheaper than a low-cost model that triggers three retries and a manual escalation.

Route the tool-model pair when agents use tools

The model is not the only decision. The tool interface changes what the model must understand and how much context it needs.

The WRP research paper highlights joint tool-model selection: a web-search summary may need only a small text model, image analysis needs a vision-capable model, and code execution benefits from stronger reasoning. A session’s total cost and success depend on the sequence of tool and model choices, not only the price of the current completion. 2

Add tool requirements to the route contract:

type StepRequirements = {
task: "plan" | "tool_args" | "summarize" | "vision";
needsStructuredOutput: boolean;
needsTools: string[];
sensitivity: "public" | "internal" | "confidential";
maxLatencyMs: number;
};

function compatible(model: ModelProfile, req: StepRequirements): boolean {
return model.healthy
&& model.maxLatencyMs <= req.maxLatencyMs
&& model.supportsStructuredOutput >= req.needsStructuredOutput
&& req.needsTools.every(tool => model.tools.includes(tool))
&& model.dataClasses.includes(req.sensitivity);
}

• This prevents a common failure mode: a fallback model is technically online but cannot reliably emit the structured tool call required by the next step.

Start simple, then learn from real traffic

A routing program should mature in stages:

1. Rules and cache: handle sensitivity, region, task type, obvious context limits, and repeated requests.
2. Health-aware routing: add quotas, latency, error rates, cooldowns, and provider availability.
3. Semantic routing: use embeddings or a classifier to map varied requests to task profiles.
4. Quality-aware routing: estimate which model meets a measured floor for each task category.
5. Adaptive routing: recalibrate from production outcomes, feedback, and route-level evaluations.

Do not skip directly to online learning. You need stable telemetry, a held-out evaluation set, safety constraints, rollback controls, and a way to explain every change. Google ADK’s RoutedLlm illustrates the basic failover boundary: the router receives failed model keys and error context before a response is produced, but once streaming begins, the error is propagated rather than switching models mid-stream. 6

That is a useful operational rule for every framework: choose and record a model at the step boundary; do not pretend a partially streamed response can be transparently moved to another model.

References & Community Insights

Production checklist

1. Does every agent step declare capability, sensitivity, context, latency, quality, and cost requirements?
2. Are data-residency and tenant policies applied before health or price optimization?
3. Are retries, cross-provider fallbacks, circuit breakers, degradation, and escalation modeled separately?

4. Can the router explain why a particular model handled a particular step?

5. Are fallback models proven to support the required tool and structured-output contract?

6. Are model snapshots and routing policies versioned and pinned?

7. Are quality floors measured on real production tasks rather than only generic benchmarks?
8. Are fallback rate, p95 latency, cost per successful task, and route-level quality monitored together?
9. Does the system stop before a side effect when a fallback cannot meet the required quality or capability?
10. Can a new routing policy be shadow-tested and rolled back without changing the agent workflow?

There is no universally best model for a production agent. There is only a model that is appropriate for a particular step, under a particular policy, with a particular budget and failure consequence. A policy-aware router turns that decision into an observable engineering control instead of a hidden provider preference.