ADR-006: Manual Overrides Are Auto-Routing Failure Signals

Superseded in part by ADR-009

ADR-009 owns the v0.11 public routing surface. The override-as-failure-signal decision remains active, but Profile and ModelRef are no longer public routing-intent fields.

Context

ADR-005 replaced model_routes with smart routing: the service combines catalog policy, provider liveness, prompt characteristics, and per-(provider, model) signals to pick (harness, provider, model) automatically. ADR-005 left the framing of pin precedence (Harness / Provider / Model / ModelRef / Profile) as a primary user surface — implicitly treating manual pins as a normal way to drive Execute.

That framing is wrong at first principles.

The system exists to complete prompts at minimum cost and time. Auto-routing is the optimizer. Manual overrides exist to correct the optimizer when it picks badly. Therefore every override is, by definition, a signal that auto-routing failed for that prompt — the user is asserting a different decision than what auto would produce. Without measuring overrides, we cannot improve the optimizer.

Today RouteStatus exposes per-(provider, model) success rates and recent decisions. Neither captures “the user disagreed with auto’s pick on this request.” The most important quality signal for routing — how often does auto get it right? — is invisible.

This ADR re-frames the user surface around that signal.

Decision

1. Override vs. intent

The auto-routing pipeline produces a decision tuple (harness, provider, model). Inputs that assert a value on one of those axes are overrides; inputs that declare goals or constraints for the optimizer are intent.

• Overrides (each axis tracked independently):

  • Harness — assertion on the harness axis.
  • Provider — assertion on the provider axis.
  • Model — assertion on the model axis.

• Intent (never an override):

  • Profile — caller’s cost-vs-time preference. cheap = minimize spend, willing to wait; fast = minimize latency, willing to spend; smart = quality-optimized balanced default. Implementation may map each profile to a tier as a discrete approximation; the principle is continuous.
  • ModelRef — catalog tier alias (or, when Profile is unset, a profile alias). Coarser than concrete model selection; the optimizer still picks the model within the indicated tier.
  • EstimatedPromptTokens, RequiresTools, Reasoning — capability constraints, not assertions.
  • CachePolicy — request-scoped cache opt-out; orthogonal to routing per ADR-005 §10 amendment.

2. Coincidental agreement still counts as override

A pin that happens to match what auto would have picked anyway is still recorded as an override on its axis. The user intended to assert; the assertion itself is the signal regardless of whether auto agreed.

This is deliberate. A high rate of coincidental-agreement overrides means users do not trust the optimizer and pin defensively. That is a UX failure (auto’s reasoning is opaque) even when the optimizer is technically correct. It deserves its own metric.

3. Override event

Execute emits an override event whenever any override axis is set on the request. The event carries both the user-pinned decision and the unconstrained auto-decision (computed by a second in-process ResolveRoute call with override axes stripped):

{
  "type": "override",
  "session_id": "svc-...",
  "user_pin": {"harness": "claude", "provider": "", "model": "opus-4.7"},
  "auto_decision": {"harness": "agent", "provider": "openrouter", "model": "claude-opus-4.7"},
  "axes_overridden": ["harness", "model"],
  "match_per_axis": {"harness": false, "model": false},
  "auto_score": 0.67,
  "auto_components": {"cost": 0.12, "latency_ms": 240, "success_rate": 0.95, "capability": 0.82},
  "prompt_features": {
    "estimated_tokens": 12500,
    "requires_tools": true,
    "reasoning": "high"
  },
  "reason_hint": "<optional --override-reason flag>",
  "outcome": {
    "status": "success|stalled|failed|cancelled",
    "cost_usd": 0.42,
    "duration_ms": 12345
  }
}

outcome is populated post-execution, mirroring the final-event status fields. It does not provide a counterfactual (“what auto would have cost”) — that would require speculatively running the auto pick. Outcome correlation across many overrides is sufficient to diagnose whether overrides on a prompt class are completing better or worse than the cohort.

4. Rejected-override event

When a pin fails pre-dispatch (orphan model, unknown provider, harness/provider incompatibility), Execute emits a rejected_override event with the same shape minus outcome. This measures how often callers mis-pin and surfaces common typos in route-status diagnostics.

5. Routing-quality metrics

Three first-class metrics, exposed on RouteStatus and UsageReport:

• auto_acceptance_rate = no-override requests ÷ total requests. Higher is better. The headline number for routing health.
• override_disagreement_rate = overrides where pin ≠ auto on at least one overridden axis ÷ overrides. Lower is better. Coincidental-agreement overrides pull this metric down (they’re in the denominator but not the numerator).
• override_class_breakdown = pivot of (prompt_features, axis_overridden, match_per_axis) → count and outcome aggregates. Where diagnosis lives. Operators answer “should I tune the optimizer for prompts with X tokens and tools?” by reading this breakdown.

These are labeled routing quality in the UI. The existing per-(provider, model) success rate stays — labeled provider reliability. The two compose: routing quality × provider reliability ≈ end-to-end completion rate. Conflating them today is a UI bug we fix as part of this ADR’s bead chain.

6. Soft-supersedes ADR-005’s framing of §1

ADR-005’s mechanics are unchanged: when no pin is set, auto-fills run; when pins are set, the engine resolves them per the precedence rules. What flips is the user-facing framing: pins are override hatches, not a normal way to drive Execute. CONTRACT-003’s “Selection precedence” section gets a disclaimer paragraph naming this; the rules below the disclaimer are now implementation reference.

7. Implementation invariants

• Auto-decision capture is mandatory on every Execute, including pinned ones. The second ResolveRoute call (with override axes stripped) is in-process and bounded by catalog size; expected sub-millisecond.
• The override event MUST be emitted before the corresponding final event, so consumers correlating per-session can join cleanly.
• Test-only harnesses (virtual, script) emit overrides like any other. Test fixtures consuming overrides should expect them.
• prompt_features.estimated_tokens is populated from EstimatedPromptTokens if the caller provided it, else from the harness’s tokenizer (best-effort; field nullable).
• The reason_hint field accepts a free-form caller-supplied string via a --override-reason flag (CLI) or Metadata["override.reason"] (programmatic). Optional; encourages users to explain pins for later diagnosis.

Consequences

Positive

• Routing quality becomes a first-class measurable signal. Per-prompt-class diagnosis becomes possible without manually correlating session logs.
• Pin precedence demotes to implementation reference. The user-facing surface is profile + auto.
• Override-tracking creates a path to closed-loop optimizer improvement: operators tune scoring weights based on override_class_breakdown; eventually a future ADR may specify automatic per-prompt-class scoring priors.
• UX defect surfacing: high coincidental-agreement override rate means auto-routing is opaque. Distinct metric forces the team to address that, not just “make auto right more often.”

Negative

• Small per-Execute cost: a second ResolveRoute call when any override axis is set. Bounded by catalog size; expected sub-millisecond. Negligible against any actual model dispatch.
• Telemetry volume grows: one extra event per overridden request. Session logs are already JSONL; impact is marginal.
• UI complexity: three routing-quality metrics + provider-reliability metric must be presented distinctly. Conflation risk during the transition.
• Caller mental-model shift: users who have been pinning routinely now learn pinning is a failure signal. Communication problem more than a technical one.

Migration

Three beads. Step 1 lands first; steps 2 and 3 can parallelize.

1. telemetry: emit override and rejected_override events from Execute (P2). Adds the second ResolveRoute call with override axes stripped. Captures prompt_features. Outcome populated by the existing final-event path. Tests asserting both event shapes plus the AST guard that Execute emits the override event before the final event.

2. metrics: routing-quality vs provider-reliability separation in RouteStatus / UsageReport (P2). Adds auto_acceptance_rate, override_disagreement_rate, and override_class_breakdown as first-class metrics. Renames or relabels the existing per-(provider, model) success-rate UI to “Provider reliability” so the two are not conflated.

3. cli: fiz route-status --overrides --since DURATION (P3, depends on 2). Operator surface — prints override_class_breakdown with outcome aggregates over the time window. Includes a mode to filter by axis. Note explicitly: operator-driven feedback loop; automatic learning is a future ADR.

A separate small commit amends three locations with framing disclaimers (no behavior change):

• CONTRACT-003 “Selection precedence” section gets a leading disclaimer naming pin precedence as implementation reference.
• SD-005 D13 (RequiresTools filter scope) gets a one-line note.
• SD-005 routing.health_cooldown gets a one-line note.

Out of scope (deferred)

• Automatic learning loop. Override breakdown adjusting per-prompt-class scoring priors. Real, useful, and a follow-up ADR. Don’t build it before we have weeks of override data to validate the metric shape.
• Continuous Profile weights. Today Profile maps to a tier; principle is continuous (cost-time tradeoff weights). Future ADR may make this explicit if discrete tiers stop being expressive enough.
• Counterfactual outcome. Speculatively running the auto pick to compare actual cost. Too expensive; we rely on cohort-level outcome correlation.
• Pre-commit override-reason prompt. UI nudge to ask “why are you pinning?” before running. Out of band; CLI nicety.

Related

• ADR-005 — smart routing replaces model_routes. This ADR soft-supersedes its framing of §1 (auto-selection rules) without invalidating the mechanics.
• CONTRACT-003 — Selection precedence section now framed as implementation reference.
• SD-005 — D13 (RequiresTools filter) and routing.health_cooldown get framing disclaimers; mechanics unchanged.
• Future ADR-007 (proposed): automatic learning loop driving scoring priors from override_class_breakdown.