feat(proxy/model-router): add shadow mode (HEADROOM_MODEL_ROUTER_LOG_ONLY) (#2395)

## Description

Adds a shadow mode to the cost-aware model router (#1706) so operators
can validate their routes against real traffic before switching routing
on for real.

Enabling routing today immediately rewrites the upstream model. That is
a big first step to take on faith: an operator wants to see which
requests each rule would catch (and confirm nothing important gets
downgraded) before letting it change live traffic. Shadow mode provides
exactly that dry run.

## How it works

Set `HEADROOM_MODEL_ROUTER_LOG_ONLY=1` (alongside the usual
`HEADROOM_MODEL_ROUTER_ENABLED` / `HEADROOM_MODEL_ROUTES`). Rules are
evaluated normally and the intended route is logged, but the outgoing
model is left unchanged.

It is implemented entirely inside the pure `ModelRouter.select()` and
needs no handler changes: in `log_only` mode a matched rule returns a
`ModelDecision` whose `routed_model` equals the original model, so
`changed` is `False` and every caller (the Anthropic and OpenAI routing
helpers) already leaves the model untouched. The decision `reason`
carries the shadow intent, e.g.

```
[shadow] would route gpt-5.4 -> gpt-5.4-mini via rule 'small' (input_tokens=1200, has_tools=False); log_only, model unchanged
```

which the handlers already log at INFO, so the would-be routing is fully
observable.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/model_router.py`: add `ModelRouterConfig.log_only`
(parsed from `HEADROOM_MODEL_ROUTER_LOG_ONLY` in `from_env`), a
`ModelRouter.log_only` property, and the shadow branch in `select()`
that returns a matched-but-unchanged decision with a `[shadow]` reason.
- `headroom/proxy/server.py`: pass `HEADROOM_MODEL_ROUTER_LOG_ONLY` into
`from_env`.
- `tests/test_proxy/test_model_router.py`: cover env parsing, the
default-off case, that a matched rule leaves the model unchanged in
shadow mode (with the shadow reason), and that shadow off still routes.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/proxy/model_router.py headroom/proxy/server.py tests/test_proxy/test_model_router.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/model_router.py
Success: no issues found in 1 source file
# model_router is a pure module, so I ran its test file against the real code
# in the project venv (uv sync): 33 no-arg tests pass, including the new
# shadow-mode ones.
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`.
- Exact command / steps: built a config via
`ModelRouterConfig.from_env("1", "[{...}]", "true")` and called
`ModelRouter.select(...)` on a request a rule matches, plus the same
with `log_only` off.
- Observed result: in shadow mode the decision is `matched=True`,
`changed=False`, `routed_model` equals the original, and `reason` starts
with `[shadow] would route strong -> cheap`; with shadow off the same
request is routed (`changed=True`, `routed_model="cheap"`); `log_only`
defaults to `False`. Because `model_router` has no heavy imports, this
ran against the actual module, not a replica.
- Not tested: a live proxy request in shadow mode; the router is pure
and the handlers already consume `ModelDecision`/log its reason
unchanged.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

`model_router` is a pure component, so I ran its unit tests against the
real code in the venv (33 pass). No handler changes are required: shadow
mode works by returning a non-changing decision, which the existing
Anthropic and OpenAI routing helpers already handle. Disabled by
default; only takes effect when routing is enabled and
`HEADROOM_MODEL_ROUTER_LOG_ONLY` is truthy.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2395"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `aff3be8e38c361809d98ca19414bfb019c265cef`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean current-main0ddaa56c integration; contributor ModelRouter.select
and ModelRouterConfig.from_env ASTs unchanged. Main-relative scope
remains three intended source/test files. Fresh
validation349passed,11skipped,1warning:60router/wiring tests
plus289incoming-main regressions. Three changed Python files pass
Rufflint/format,candidate offline lock and whitespace pass. Complete
native tracked-input set/all recorded hashes/artifact checksum verified
before3982DLL reuse; no freshbuild.

Fresh runtime proof closes the outstanding select-only evidence gap:
real local proxy TCP/ASGI lifespan forwards to loopback HTTP upstream
with synthetic auth/model/content. From canonical environment-factory
config, enabled+shadow forwards original claude-sonnet-4-6 and logs
exactly one '[shadow] would route ... -> claude-haiku-4-5'; active
forwards claude-haiku-4-5 and logs one matched route; disabled forwards
original and emits no routing decision. All three HTTP200. Each proxy
child stopped and upstream server closed. These are actual local
forwarding/log observations, not cloud/provider-account,
model-inference, billing or CLI-env parsing proof. Main's separate CLI
construction was not changed; do not infer CLI env wiring from
environment-factory coverage. Synthetic fixture/log artifacts preserved
in pr2395-0dd-runtime-proof.json and associated scripts/logs.

Read author/other-human discussion and current Devin runtime-proof note.
Current fresh REST audit contains no new external findings. Only three
reviewed source/test files differ from main; no
workflow/setup/dependency/native/CLI delta, central publication gate
passes. Contributor shadow semantics/defaults preserved. Exact
published-head CI/Actions/rollup,main ancestry,mergeability and fresh
discussion remain before approval. Maintainers handle CI; no PR merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


<!-- maintainer-refresh-12864fc8f01a1ca6c58440eccef0dabdd3e595b8 -->

Maintainer current-main refresh: exact head
`41a0f5b920f48f669e6e8ea0a85593c02d6622c0`, authoritative main
`12864fc8f01a1ca6c58440eccef0dabdd3e595b8`. Clean integration patches
exactly match incoming main; contributor behavior preserved. Fresh
focused source tests plus tokenizer/response-header/HF/JSON-parser
incoming-main controls: 149 passed, 1 warning in 15.46s.
Ruff/format/whitespace pass. Router tests and mocked proxy wiring are
not live provider/billing proof; shadow mode preserves the actual
requested model. Native artifact reuse verified against full recorded
input set/every hash and artifact checksum; no fresh native build
claimed. Static publication gate excludes inherited
workflow-validation/native Docker/Init/Wrap installer paths, with no
workflow/setup/dependency/native/CLI changes in PR diff. Actual
exact-head hosted workflows and final independent review remain pending.
No merge performed.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Abhay Singh
2026-11-07 20:41:30 -06:00
committed by GitHub
co-authored by JD Davis
parent f896161340
commit 861ef95620
3 changed files with 111 additions and 3 deletions
+40 -3
View File
@@ -107,9 +107,18 @@ class ModelRouterConfig:
enabled: bool = False
routes: tuple[ModelRoute, ...] = ()
log_only: bool = False
"""Shadow mode: evaluate rules and log what routing *would* do, but leave the
model unchanged. Lets an operator validate routes against real traffic before
switching them on for real."""
@classmethod
def from_env(cls, enabled_raw: str | None, routes_raw: str | None) -> ModelRouterConfig:
def from_env(
cls,
enabled_raw: str | None,
routes_raw: str | None,
log_only_raw: str | None = None,
) -> ModelRouterConfig:
"""Build config from env-style strings, failing open to disabled.
``routes_raw`` is a JSON array of rule objects, e.g.::
@@ -117,15 +126,20 @@ class ModelRouterConfig:
[{"name": "small->mini", "max_input_tokens": 4000,
"require_no_tools": true, "to_model": "gpt-5.4-mini"}]
``log_only_raw`` (``HEADROOM_MODEL_ROUTER_LOG_ONLY``) turns on shadow
mode: rules are evaluated and the intended route is logged, but the
model is left unchanged.
A malformed value logs a warning and disables routing rather than
raising, so a bad config can never take the proxy down.
"""
enabled = _truthy(enabled_raw)
log_only = _truthy(log_only_raw)
routes = _parse_routes(routes_raw)
if enabled and not routes:
logger.warning("model router enabled but no valid routes configured; disabling")
return cls(enabled=False, routes=())
return cls(enabled=enabled and bool(routes), routes=routes)
return cls(enabled=False, routes=(), log_only=log_only)
return cls(enabled=enabled and bool(routes), routes=routes, log_only=log_only)
@dataclass(frozen=True)
@@ -154,6 +168,11 @@ class ModelRouter:
def enabled(self) -> bool:
return self._config.enabled and bool(self._config.routes)
@property
def log_only(self) -> bool:
"""True when routing runs in shadow mode (decisions logged, not applied)."""
return self._config.log_only
def select(
self,
*,
@@ -176,6 +195,24 @@ class ModelRouter:
if route.matches(
model=model, input_tokens=input_tokens, has_tools=has_tools, max_tokens=max_tokens
):
if self._config.log_only:
# Shadow mode: record what routing WOULD do, but keep
# ``routed_model == model`` so ``changed`` is False and the
# handlers leave the outgoing model untouched. The reason is
# what the handlers log, so the intended route is observable.
reason = (
f"[shadow] would route {model} -> {route.to_model} via rule "
f"{route.name or route.to_model!r} "
f"(input_tokens={input_tokens}, has_tools={has_tools}); "
f"log_only, model unchanged"
)
return ModelDecision(
original_model=model,
routed_model=model,
matched=True,
reason=reason,
rule_name=route.name,
)
reason = (
f"matched rule {route.name or route.to_model!r}: "
f"{model} -> {route.to_model} "
+1
View File
@@ -6651,6 +6651,7 @@ def _proxy_config_from_env() -> ProxyConfig:
model_router=ModelRouterConfig.from_env(
os.environ.get("HEADROOM_MODEL_ROUTER_ENABLED"),
os.environ.get("HEADROOM_MODEL_ROUTES"),
os.environ.get("HEADROOM_MODEL_ROUTER_LOG_ONLY"),
),
)
+70
View File
@@ -324,6 +324,58 @@ def test_estimate_input_tokens_counts_system_blocks() -> None:
assert estimate_input_tokens([{"content": "hi"}], system=blocks) > 100
# ---------------------------------------------------------------------------
# Shadow mode (HEADROOM_MODEL_ROUTER_LOG_ONLY)
# ---------------------------------------------------------------------------
def test_from_env_parses_log_only() -> None:
cfg = ModelRouterConfig.from_env("1", '[{"to_model":"cheap","max_input_tokens":10000}]', "true")
assert cfg.enabled is True
assert cfg.log_only is True
def test_log_only_defaults_false() -> None:
cfg = ModelRouterConfig.from_env("1", '[{"to_model":"cheap"}]')
assert cfg.log_only is False
assert ModelRouter(cfg).log_only is False
def test_log_only_matches_but_leaves_model_unchanged() -> None:
router = _router(
ModelRoute(to_model="cheap", max_input_tokens=10_000, name="small"),
)
# Force shadow mode via config.
router = ModelRouter(
ModelRouterConfig(
enabled=True,
routes=(ModelRoute(to_model="cheap", max_input_tokens=10_000, name="small"),),
log_only=True,
)
)
d = router.select(model="strong", input_tokens=100, has_tools=False)
# A rule matched, but the model is NOT rewritten (changed is False).
assert d.matched is True
assert d.changed is False
assert d.routed_model == "strong"
assert d.rule_name == "small"
# The reason records what routing WOULD have done, for the handler log.
assert d.reason.startswith("[shadow] would route strong -> cheap")
def test_log_only_off_still_routes() -> None:
router = ModelRouter(
ModelRouterConfig(
enabled=True,
routes=(ModelRoute(to_model="cheap", max_input_tokens=10_000),),
log_only=False,
)
)
d = router.select(model="strong", input_tokens=100, has_tools=False)
assert d.changed is True
assert d.routed_model == "cheap"
def test_select_passes_max_tokens_to_rules() -> None:
router = ModelRouter(
ModelRouterConfig(
@@ -361,3 +413,21 @@ def test_request_max_tokens() -> None:
assert request_max_tokens({"max_tokens": "4000"}) is None
assert request_max_tokens({"max_tokens": 1.5}) is None
assert request_max_tokens(None) is None
def test_shadow_mode_respects_max_output_tokens_condition() -> None:
router = ModelRouter(
ModelRouterConfig(
enabled=True,
log_only=True,
routes=(ModelRoute(to_model="cheap", max_output_tokens=512, name="small-output"),),
)
)
small = router.select(model="strong", input_tokens=10, has_tools=False, max_tokens=256)
assert small.matched
assert not small.changed
assert small.routed_model == "strong"
assert "[shadow]" in small.reason
large = router.select(model="strong", input_tokens=10, has_tools=False, max_tokens=4000)
assert not large.matched
assert not large.changed