feat(router): add max_output_tokens condition to ModelRoute (#3898)

## Description

`ModelRoute` could bound the estimated input size (`max_input_tokens`)
but had no condition on the request's own `max_tokens`. That meant a
tiny prompt asking for a long answer (for example, `max_tokens: 4000`)
still matched a "trivial turn" downshift rule and was sent to the
cheaper model.

This PR adds `max_output_tokens`, using the shape proposed in the issue.
A rule with it set matches only when the request's `max_tokens` is `<=
N`. As with the other conditions, `None` means "ignore this condition",
set conditions AND together, and the first matching rule wins.

**Open question from the issue, decided here:** a request with no
`max_tokens`, or with a non-integer or boolean value, does **not**
match. An unbounded response is not a trivial turn, so the rule can
never widen to cover unbounded requests. This follows the same
fail-closed approach as the existing parser.

Closes #2765

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/model_router.py`: new `ModelRoute.max_output_tokens`
field. `matches()` and `ModelRouter.select()` take an optional
`max_tokens` keyword, which defaults to `None`, so existing callers are
unchanged. `max_tokens` is added to the decision reason.
`max_output_tokens` is added to the allowed route keys and parsed with
the existing `_strict_opt_int`, so a malformed, boolean or negative
value skips the route. New `request_max_tokens(body)` helper.
- `headroom/proxy/handlers/anthropic.py`: `_maybe_route_model` passes
`request_max_tokens(body)` to the router.
- `docs/content/docs/configuration.mdx`: new field in the rule table,
plus a note on combining it with `max_input_tokens`.
- Tests for matching, absent budget, select and reason, env parsing
(valid and malformed), the helper, and handler wiring.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): not run successfully. My
local mypy stops on an unrelated numpy stub syntax error in this env
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_proxy/test_model_router.py tests/test_proxy/test_model_router_wiring.py -q
54 passed, 1 warning in 5.22s

$ pytest tests/test_proxy -q -k "anthropic or router"
119 passed, 237 deselected, 1 warning in 14.41s

$ ruff check headroom/proxy/model_router.py headroom/proxy/handlers/anthropic.py tests/test_proxy/test_model_router*.py
All checks passed!
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, branch head
- Exact command / steps: the rule from the issue (`from_models:
[claude-opus-4-8]`, `max_input_tokens: 4096`, `max_output_tokens: 512`,
`require_no_tools`, `to_model: claude-haiku-4-5`) built via
`ModelRouterConfig.from_env`. I then ran `select()` for the issue's
"Reply with the single word OK." prompt with three different
`max_tokens` values.
- Observed result: `max_tokens=4000` stays on `claude-opus-4-8`,
`max_tokens=256` routes to `claude-haiku-4-5`, and a request with no
`max_tokens` stays on `claude-opus-4-8`:
  ```text
  max_tokens=4000: routed_model=claude-opus-4-8 reason=no rule matched
max_tokens=256: routed_model=claude-haiku-4-5 reason=matched rule
'trivial-opus->haiku': claude-opus-4-8 -> claude-haiku-4-5
(input_tokens=7, has_tools=False, max_tokens=256)
  max_tokens=None: routed_model=claude-opus-4-8 reason=no rule matched
  ```
- Not tested: an end-to-end request through a running proxy against a
live upstream.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This only extends the opt-in model
router (`HEADROOM_MODEL_ROUTER_ENABLED`).
- Minimum rollout channel: N/A
- Stable/default behavior changed: No. The router is still off by
default, and rules without `max_output_tokens` behave exactly as before.
- Kill switch / disable path: unset `HEADROOM_MODEL_ROUTER_ENABLED`, or
remove the field from the rule.
- Unsafe override required: No
- Qualification impact: None
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Routing still only applies on the Anthropic `/v1/messages` path, where
`max_tokens` is required. If routing is later extended to the OpenAI
paths (#2354), `request_max_tokens` is the single place to add
`max_completion_tokens` / `max_output_tokens`.


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3898"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Sudhindra Desai
2026-10-06 10:30:01 -05:00
committed by GitHub
co-authored by JD Davis
parent a961a6fea0
commit 01185e32d3
5 changed files with 158 additions and 6 deletions
+2
View File
@@ -480,6 +480,7 @@ Enable it with `HEADROOM_MODEL_ROUTER_ENABLED=1` and declare ordered rules in `H
| `to_model` | string (required) | Model to route to when the rule matches. |
| `max_input_tokens` | integer | Match only when the estimated input size is at or below this. |
| `min_input_tokens` | integer | Match only when the estimated input size is at or above this. |
| `max_output_tokens` | integer | Match only when the request's own `max_tokens` is at or below this. A request without `max_tokens` does not match. |
| `require_no_tools` | boolean | Match only when the request declares no tools (a proxy for low-risk work). |
| `from_models` | list of strings | Restrict the rule to these source models. Omit for any source model. |
| `name` | string | Label surfaced in the decision log. |
@@ -494,6 +495,7 @@ export HEADROOM_MODEL_ROUTES='[
Notes:
- Combine `max_input_tokens` with `max_output_tokens` to downshift only turns that are cheap on both sides: a tiny prompt that asks for a long answer (a large `max_tokens`) stays on the original model.
- Input size is a fast, tokenizer-free estimate over the messages, tools, and top-level `system` prompt, meant for tier selection rather than exact accounting.
- A malformed rule fails open: it is skipped (never silently widened), and the rest of the rules still apply.
- Routing is skipped for byte-faithful passthrough requests (`x-headroom-bypass: true` or `x-headroom-mode: passthrough`), so those are never model-rewritten.
+2 -1
View File
@@ -61,7 +61,7 @@ from headroom.proxy.identity import resolve_memory_identity
from headroom.proxy.image_isolation import run_image_compression_isolated
from headroom.proxy.memory_decision import MemoryDecision
from headroom.proxy.memory_query import MemoryQuery
from headroom.proxy.model_router import estimate_input_tokens
from headroom.proxy.model_router import estimate_input_tokens, request_max_tokens
from headroom.proxy.nonstream_sse_policy import should_recover_sse_reply
from headroom.proxy.outcome import RequestOutcome
from headroom.proxy.output_shaper import shaper_enabled_for, steering_allowed_for
@@ -910,6 +910,7 @@ class AnthropicHandlerMixin:
model=model,
input_tokens=estimate_input_tokens(messages, body.get("tools"), body.get("system")),
has_tools=bool(body.get("tools")),
max_tokens=request_max_tokens(body),
)
logger.info("model routing decision: %s", decision.reason)
if not decision.changed:
+52 -5
View File
@@ -61,7 +61,25 @@ class ModelRoute:
name: str = ""
"""Human-readable label surfaced in decision logs."""
def matches(self, *, model: str, input_tokens: int, has_tools: bool) -> bool:
# Appended after the original fields so existing positional
# ``ModelRoute(...)`` calls keep their meaning.
max_output_tokens: int | None = None
"""Match only when the request's own ``max_tokens`` is <= this.
The caller's declared output budget is the best "this turn is trivial"
signal at routing time: a tiny prompt asking for 4000 output tokens may
want a long answer from the strong model. A request that declares no
``max_tokens`` does NOT match (an unbounded response is not a trivial
turn), so the rule can never widen to unbounded requests."""
def matches(
self,
*,
model: str,
input_tokens: int,
has_tools: bool,
max_tokens: int | None = None,
) -> bool:
"""True when every set condition is satisfied for this request."""
if self.from_models and model not in self.from_models:
return False
@@ -73,6 +91,10 @@ class ModelRoute:
return False
if self.min_input_tokens is not None and input_tokens < self.min_input_tokens:
return False
if self.max_output_tokens is not None and (
max_tokens is None or max_tokens > self.max_output_tokens
):
return False
# A rule whose ``to_model`` equals the current model still MATCHES (strict
# first-match-wins): it is a no-op (``changed`` is False) that short-circuits
# later rules, which lets an operator write an explicit exemption rule.
@@ -132,7 +154,14 @@ class ModelRouter:
def enabled(self) -> bool:
return self._config.enabled and bool(self._config.routes)
def select(self, *, model: str, input_tokens: int, has_tools: bool) -> ModelDecision:
def select(
self,
*,
model: str,
input_tokens: int,
has_tools: bool,
max_tokens: int | None = None,
) -> ModelDecision:
"""Return the routing decision for a request.
Never raises: on a disabled router or no matching rule, returns a
@@ -144,11 +173,14 @@ class ModelRouter:
return ModelDecision(model, model, matched=False, reason="no source model")
for route in self._config.routes:
if route.matches(model=model, input_tokens=input_tokens, has_tools=has_tools):
if route.matches(
model=model, input_tokens=input_tokens, has_tools=has_tools, max_tokens=max_tokens
):
reason = (
f"matched rule {route.name or route.to_model!r}: "
f"{model} -> {route.to_model} "
f"(input_tokens={input_tokens}, has_tools={has_tools})"
f"(input_tokens={input_tokens}, has_tools={has_tools}, "
f"max_tokens={max_tokens})"
)
return ModelDecision(
original_model=model,
@@ -187,6 +219,18 @@ def estimate_input_tokens(messages: object, tools: object = None, system: object
return 0
def request_max_tokens(body: object) -> int | None:
"""Return the request's ``max_tokens``, or ``None`` if absent or not an integer.
Booleans and non-integers are treated as absent so a malformed client value
can never satisfy a ``max_output_tokens`` bound. It never raises.
"""
value = body.get("max_tokens") if isinstance(body, dict) else None
if isinstance(value, int) and not isinstance(value, bool):
return value
return None
def _truthy(value: str | None) -> bool:
return (value or "").strip().lower() in {"1", "true", "yes", "on", "enable", "enabled"}
@@ -219,6 +263,7 @@ _ALLOWED_ROUTE_KEYS = frozenset(
"to_model",
"max_input_tokens",
"min_input_tokens",
"max_output_tokens",
"require_no_tools",
"require_tools",
"from_models",
@@ -252,7 +297,8 @@ def _route_from_entry(entry: object, index: int) -> ModelRoute | None:
max_tokens = _strict_opt_int(entry, "max_input_tokens", index)
min_tokens = _strict_opt_int(entry, "min_input_tokens", index)
if max_tokens is _INVALID or min_tokens is _INVALID:
max_output = _strict_opt_int(entry, "max_output_tokens", index)
if max_tokens is _INVALID or min_tokens is _INVALID or max_output is _INVALID:
return None
require_no_tools = entry.get("require_no_tools", False)
@@ -280,6 +326,7 @@ def _route_from_entry(entry: object, index: int) -> ModelRoute | None:
to_model=to_model,
max_input_tokens=max_tokens, # type: ignore[arg-type]
min_input_tokens=min_tokens, # type: ignore[arg-type]
max_output_tokens=max_output, # type: ignore[arg-type]
require_no_tools=require_no_tools,
require_tools=require_tools,
from_models=tuple(from_models_raw),
+79
View File
@@ -8,6 +8,7 @@ from headroom.proxy.model_router import (
ModelRouter,
ModelRouterConfig,
estimate_input_tokens,
request_max_tokens,
)
# ---------------------------------------------------------------------------
@@ -52,6 +53,45 @@ def test_route_require_tools_and_require_no_tools_never_matches() -> None:
assert not route.matches(model="m", input_tokens=1, has_tools=False)
def test_route_max_output_tokens_bounds_declared_budget() -> None:
# Issue #2765: a tiny prompt asking for a long answer must not downshift.
route = ModelRoute(to_model="cheap", max_input_tokens=4096, max_output_tokens=512)
assert route.matches(model="strong", input_tokens=10, has_tools=False, max_tokens=512)
assert route.matches(model="strong", input_tokens=10, has_tools=False, max_tokens=1)
assert not route.matches(model="strong", input_tokens=10, has_tools=False, max_tokens=4000)
def test_route_max_output_tokens_absent_budget_does_not_match() -> None:
# An unbounded response is not a trivial turn: absent max_tokens never
# satisfies the bound, so the rule cannot widen to unbounded requests.
route = ModelRoute(to_model="cheap", max_output_tokens=512)
assert not route.matches(model="strong", input_tokens=10, has_tools=False)
assert not route.matches(model="strong", input_tokens=10, has_tools=False, max_tokens=None)
def test_route_without_max_output_tokens_ignores_budget() -> None:
route = ModelRoute(to_model="cheap", max_input_tokens=4096)
assert route.matches(model="strong", input_tokens=10, has_tools=False, max_tokens=100_000)
assert route.matches(model="strong", input_tokens=10, has_tools=False)
def test_route_positional_constructor_keeps_original_field_order() -> None:
# max_output_tokens is appended last, so positional ModelRoute(...) calls
# written before #2765 keep their meaning: here the 4th positional arg is
# still require_no_tools, not the new output bound.
route = ModelRoute("cheap", 4000, None, True)
assert route.require_no_tools is True
assert route.max_output_tokens is None
assert not route.matches(model="strong", input_tokens=10, has_tools=True, max_tokens=1)
assert route.matches(model="strong", input_tokens=10, has_tools=False, max_tokens=1)
route = ModelRoute("cheap", None, None, False, True, ("strong",), "agentic")
assert route.require_tools is True
assert route.from_models == ("strong",)
assert route.name == "agentic"
assert route.max_output_tokens is None
def test_route_matches_even_for_same_model() -> None:
# A same-model rule still MATCHES (strict first-match-wins); it is a no-op
# that short-circuits later rules, enabling explicit exemption rules.
@@ -282,3 +322,42 @@ def test_estimate_input_tokens_counts_system_string() -> None:
def test_estimate_input_tokens_counts_system_blocks() -> None:
blocks = [{"type": "text", "text": "x" * 4000}]
assert estimate_input_tokens([{"content": "hi"}], system=blocks) > 100
def test_select_passes_max_tokens_to_rules() -> None:
router = ModelRouter(
ModelRouterConfig(
enabled=True,
routes=(ModelRoute(to_model="cheap", max_output_tokens=512, name="trivial"),),
)
)
small = router.select(model="strong", input_tokens=10, has_tools=False, max_tokens=256)
assert small.changed and small.routed_model == "cheap"
assert "max_tokens=256" in small.reason
large = router.select(model="strong", input_tokens=10, has_tools=False, max_tokens=4000)
assert not large.matched and large.routed_model == "strong"
def test_from_env_parses_max_output_tokens() -> None:
cfg = ModelRouterConfig.from_env(
"1", '[{"to_model": "cheap", "max_input_tokens": 4096, "max_output_tokens": 512}]'
)
assert cfg.enabled
assert cfg.routes[0].max_output_tokens == 512
def test_from_env_malformed_max_output_tokens_skips_route() -> None:
for bad in ('"lots"', "true", "-1"):
cfg = ModelRouterConfig.from_env(
"1", f'[{{"to_model": "cheap", "max_output_tokens": {bad}}}]'
)
assert not cfg.routes, bad
def test_request_max_tokens() -> None:
assert request_max_tokens({"max_tokens": 4000}) == 4000
assert request_max_tokens({}) is None
assert request_max_tokens({"max_tokens": True}) is None
assert request_max_tokens({"max_tokens": "4000"}) is None
assert request_max_tokens({"max_tokens": 1.5}) is None
assert request_max_tokens(None) is None
@@ -159,6 +159,29 @@ def test_maybe_route_model_routes_when_enabled() -> None:
tracker.mark_mutated.assert_called_once_with("model_router")
def test_maybe_route_model_honors_max_output_tokens() -> None:
# Issue #2765: the request's own max_tokens reaches the router.
host = _RouterHost()
host.model_router = ModelRouter(
ModelRouterConfig(
enabled=True,
routes=(ModelRoute(to_model="claude-haiku-4-5", max_output_tokens=512),),
)
)
long_answer = {"model": "claude-opus-4-8", "max_tokens": 4000}
out = host._maybe_route_model(
"claude-opus-4-8", [{"content": "OK?"}], long_answer, MagicMock(), False
)
assert out == "claude-opus-4-8"
assert long_answer["model"] == "claude-opus-4-8"
trivial = {"model": "claude-opus-4-8", "max_tokens": 256}
out = host._maybe_route_model(
"claude-opus-4-8", [{"content": "OK?"}], trivial, MagicMock(), False
)
assert out == "claude-haiku-4-5"
@pytest.mark.parametrize(
("routes", "expected_reason"),
[