mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Adds a shadow mode to the cost-aware model router (#1706) so operators can validate their routes against real traffic before switching routing on for real. Enabling routing today immediately rewrites the upstream model. That is a big first step to take on faith: an operator wants to see which requests each rule would catch (and confirm nothing important gets downgraded) before letting it change live traffic. Shadow mode provides exactly that dry run. ## How it works Set `HEADROOM_MODEL_ROUTER_LOG_ONLY=1` (alongside the usual `HEADROOM_MODEL_ROUTER_ENABLED` / `HEADROOM_MODEL_ROUTES`). Rules are evaluated normally and the intended route is logged, but the outgoing model is left unchanged. It is implemented entirely inside the pure `ModelRouter.select()` and needs no handler changes: in `log_only` mode a matched rule returns a `ModelDecision` whose `routed_model` equals the original model, so `changed` is `False` and every caller (the Anthropic and OpenAI routing helpers) already leaves the model untouched. The decision `reason` carries the shadow intent, e.g. ``` [shadow] would route gpt-5.4 -> gpt-5.4-mini via rule 'small' (input_tokens=1200, has_tools=False); log_only, model unchanged ``` which the handlers already log at INFO, so the would-be routing is fully observable. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/model_router.py`: add `ModelRouterConfig.log_only` (parsed from `HEADROOM_MODEL_ROUTER_LOG_ONLY` in `from_env`), a `ModelRouter.log_only` property, and the shadow branch in `select()` that returns a matched-but-unchanged decision with a `[shadow]` reason. - `headroom/proxy/server.py`: pass `HEADROOM_MODEL_ROUTER_LOG_ONLY` into `from_env`. - `tests/test_proxy/test_model_router.py`: cover env parsing, the default-off case, that a matched rule leaves the model unchanged in shadow mode (with the shadow reason), and that shadow off still routes. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/proxy/model_router.py headroom/proxy/server.py tests/test_proxy/test_model_router.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/model_router.py Success: no issues found in 1 source file # model_router is a pure module, so I ran its test file against the real code # in the project venv (uv sync): 33 no-arg tests pass, including the new # shadow-mode ones. ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`. - Exact command / steps: built a config via `ModelRouterConfig.from_env("1", "[{...}]", "true")` and called `ModelRouter.select(...)` on a request a rule matches, plus the same with `log_only` off. - Observed result: in shadow mode the decision is `matched=True`, `changed=False`, `routed_model` equals the original, and `reason` starts with `[shadow] would route strong -> cheap`; with shadow off the same request is routed (`changed=True`, `routed_model="cheap"`); `log_only` defaults to `False`. Because `model_router` has no heavy imports, this ran against the actual module, not a replica. - Not tested: a live proxy request in shadow mode; the router is pure and the handlers already consume `ModelDecision`/log its reason unchanged. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes `model_router` is a pure component, so I ran its unit tests against the real code in the venv (33 pass). No handler changes are required: shadow mode works by returning a non-changing decision, which the existing Anthropic and OpenAI routing helpers already handle. Disabled by default; only takes effect when routing is enabled and `HEADROOM_MODEL_ROUTER_LOG_ONLY` is truthy. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2395" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `aff3be8e38c361809d98ca19414bfb019c265cef`, upstream main `0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`. Clean current-main0ddaa56c integration; contributor ModelRouter.select and ModelRouterConfig.from_env ASTs unchanged. Main-relative scope remains three intended source/test files. Fresh validation349passed,11skipped,1warning:60router/wiring tests plus289incoming-main regressions. Three changed Python files pass Rufflint/format,candidate offline lock and whitespace pass. Complete native tracked-input set/all recorded hashes/artifact checksum verified before3982DLL reuse; no freshbuild. Fresh runtime proof closes the outstanding select-only evidence gap: real local proxy TCP/ASGI lifespan forwards to loopback HTTP upstream with synthetic auth/model/content. From canonical environment-factory config, enabled+shadow forwards original claude-sonnet-4-6 and logs exactly one '[shadow] would route ... -> claude-haiku-4-5'; active forwards claude-haiku-4-5 and logs one matched route; disabled forwards original and emits no routing decision. All three HTTP200. Each proxy child stopped and upstream server closed. These are actual local forwarding/log observations, not cloud/provider-account, model-inference, billing or CLI-env parsing proof. Main's separate CLI construction was not changed; do not infer CLI env wiring from environment-factory coverage. Synthetic fixture/log artifacts preserved in pr2395-0dd-runtime-proof.json and associated scripts/logs. Read author/other-human discussion and current Devin runtime-proof note. Current fresh REST audit contains no new external findings. Only three reviewed source/test files differ from main; no workflow/setup/dependency/native/CLI delta, central publication gate passes. Contributor shadow semantics/defaults preserved. Exact published-head CI/Actions/rollup,main ancestry,mergeability and fresh discussion remain before approval. Maintainers handle CI; no PR merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. <!-- maintainer-refresh-12864fc8f01a1ca6c58440eccef0dabdd3e595b8 --> Maintainer current-main refresh: exact head `41a0f5b920f48f669e6e8ea0a85593c02d6622c0`, authoritative main `12864fc8f01a1ca6c58440eccef0dabdd3e595b8`. Clean integration patches exactly match incoming main; contributor behavior preserved. Fresh focused source tests plus tokenizer/response-header/HF/JSON-parser incoming-main controls: 149 passed, 1 warning in 15.46s. Ruff/format/whitespace pass. Router tests and mocked proxy wiring are not live provider/billing proof; shadow mode preserves the actual requested model. Native artifact reuse verified against full recorded input set/every hash and artifact checksum; no fresh native build claimed. Static publication gate excludes inherited workflow-validation/native Docker/Init/Wrap installer paths, with no workflow/setup/dependency/native/CLI changes in PR diff. Actual exact-head hosted workflows and final independent review remain pending. No merge performed. Co-authored-by: JD Davis <mxjerrett@gmail.com>
434 lines
17 KiB
Python
434 lines
17 KiB
Python
"""Tests for cost-aware model routing (issue #1706)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from headroom.proxy.model_router import (
|
|
ModelDecision,
|
|
ModelRoute,
|
|
ModelRouter,
|
|
ModelRouterConfig,
|
|
estimate_input_tokens,
|
|
request_max_tokens,
|
|
)
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ModelRoute.matches
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_route_matches_on_max_tokens_and_no_tools() -> None:
|
|
route = ModelRoute(to_model="cheap", max_input_tokens=4000, require_no_tools=True)
|
|
assert route.matches(model="strong", input_tokens=1000, has_tools=False)
|
|
# too many tokens
|
|
assert not route.matches(model="strong", input_tokens=5000, has_tools=False)
|
|
# tools present
|
|
assert not route.matches(model="strong", input_tokens=1000, has_tools=True)
|
|
|
|
|
|
def test_route_min_tokens() -> None:
|
|
route = ModelRoute(to_model="strong", min_input_tokens=10000)
|
|
assert route.matches(model="cheap", input_tokens=20000, has_tools=True)
|
|
assert not route.matches(model="cheap", input_tokens=5000, has_tools=True)
|
|
|
|
|
|
def test_route_from_models_restriction() -> None:
|
|
route = ModelRoute(to_model="cheap", from_models=("gpt-5.5", "gpt-5.4"))
|
|
assert route.matches(model="gpt-5.5", input_tokens=1, has_tools=False)
|
|
assert not route.matches(model="claude-sonnet-4-6", input_tokens=1, has_tools=False)
|
|
|
|
|
|
def test_route_require_tools_matches_only_with_tools() -> None:
|
|
# Inverse of require_no_tools: route agentic (tool-using) turns to a
|
|
# stronger model, leaving plain chat alone.
|
|
route = ModelRoute(to_model="strong", require_tools=True)
|
|
assert route.matches(model="cheap", input_tokens=1, has_tools=True)
|
|
assert not route.matches(model="cheap", input_tokens=1, has_tools=False)
|
|
|
|
|
|
def test_route_require_tools_and_require_no_tools_never_matches() -> None:
|
|
# Contradictory conditions on one rule are an AND that can never be true —
|
|
# a harmless operator error, not a crash.
|
|
route = ModelRoute(to_model="x", require_tools=True, require_no_tools=True)
|
|
assert not route.matches(model="m", input_tokens=1, has_tools=True)
|
|
assert not route.matches(model="m", input_tokens=1, has_tools=False)
|
|
|
|
|
|
def test_route_max_output_tokens_bounds_declared_budget() -> None:
|
|
# Issue #2765: a tiny prompt asking for a long answer must not downshift.
|
|
route = ModelRoute(to_model="cheap", max_input_tokens=4096, max_output_tokens=512)
|
|
assert route.matches(model="strong", input_tokens=10, has_tools=False, max_tokens=512)
|
|
assert route.matches(model="strong", input_tokens=10, has_tools=False, max_tokens=1)
|
|
assert not route.matches(model="strong", input_tokens=10, has_tools=False, max_tokens=4000)
|
|
|
|
|
|
def test_route_max_output_tokens_absent_budget_does_not_match() -> None:
|
|
# An unbounded response is not a trivial turn: absent max_tokens never
|
|
# satisfies the bound, so the rule cannot widen to unbounded requests.
|
|
route = ModelRoute(to_model="cheap", max_output_tokens=512)
|
|
assert not route.matches(model="strong", input_tokens=10, has_tools=False)
|
|
assert not route.matches(model="strong", input_tokens=10, has_tools=False, max_tokens=None)
|
|
|
|
|
|
def test_route_without_max_output_tokens_ignores_budget() -> None:
|
|
route = ModelRoute(to_model="cheap", max_input_tokens=4096)
|
|
assert route.matches(model="strong", input_tokens=10, has_tools=False, max_tokens=100_000)
|
|
assert route.matches(model="strong", input_tokens=10, has_tools=False)
|
|
|
|
|
|
def test_route_positional_constructor_keeps_original_field_order() -> None:
|
|
# max_output_tokens is appended last, so positional ModelRoute(...) calls
|
|
# written before #2765 keep their meaning: here the 4th positional arg is
|
|
# still require_no_tools, not the new output bound.
|
|
route = ModelRoute("cheap", 4000, None, True)
|
|
assert route.require_no_tools is True
|
|
assert route.max_output_tokens is None
|
|
assert not route.matches(model="strong", input_tokens=10, has_tools=True, max_tokens=1)
|
|
assert route.matches(model="strong", input_tokens=10, has_tools=False, max_tokens=1)
|
|
|
|
route = ModelRoute("cheap", None, None, False, True, ("strong",), "agentic")
|
|
assert route.require_tools is True
|
|
assert route.from_models == ("strong",)
|
|
assert route.name == "agentic"
|
|
assert route.max_output_tokens is None
|
|
|
|
|
|
def test_route_matches_even_for_same_model() -> None:
|
|
# A same-model rule still MATCHES (strict first-match-wins); it is a no-op
|
|
# that short-circuits later rules, enabling explicit exemption rules.
|
|
route = ModelRoute(to_model="cheap")
|
|
assert route.matches(model="cheap", input_tokens=1, has_tools=False)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ModelRouter.select
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _router(*routes: ModelRoute, enabled: bool = True) -> ModelRouter:
|
|
return ModelRouter(ModelRouterConfig(enabled=enabled, routes=tuple(routes)))
|
|
|
|
|
|
def test_disabled_router_is_passthrough() -> None:
|
|
router = _router(ModelRoute(to_model="cheap", max_input_tokens=10_000), enabled=False)
|
|
d = router.select(model="strong", input_tokens=10, has_tools=False)
|
|
assert not d.matched and not d.changed
|
|
assert d.routed_model == "strong"
|
|
|
|
|
|
def test_first_matching_rule_wins() -> None:
|
|
router = _router(
|
|
ModelRoute(to_model="nano", max_input_tokens=2000, name="tiny"),
|
|
ModelRoute(to_model="mini", max_input_tokens=8000, name="small"),
|
|
)
|
|
d = router.select(model="gpt-5.5", input_tokens=1500, has_tools=False)
|
|
assert d.changed and d.routed_model == "nano" and d.rule_name == "tiny"
|
|
|
|
d2 = router.select(model="gpt-5.5", input_tokens=5000, has_tools=False)
|
|
assert d2.changed and d2.routed_model == "mini" and d2.rule_name == "small"
|
|
|
|
|
|
def test_exemption_rule_short_circuits_later_rules() -> None:
|
|
# An explicit same-model rule wins first and stops a later downgrade rule.
|
|
router = _router(
|
|
ModelRoute(to_model="keep", from_models=("keep",), name="exempt"),
|
|
ModelRoute(to_model="cheap", max_input_tokens=10_000, name="downgrade"),
|
|
)
|
|
d = router.select(model="keep", input_tokens=100, has_tools=False)
|
|
assert d.matched and not d.changed
|
|
assert d.routed_model == "keep" and d.rule_name == "exempt"
|
|
|
|
|
|
def test_no_rule_matches_is_passthrough() -> None:
|
|
router = _router(ModelRoute(to_model="mini", max_input_tokens=1000))
|
|
d = router.select(model="gpt-5.5", input_tokens=50_000, has_tools=True)
|
|
assert not d.matched and not d.changed and d.routed_model == "gpt-5.5"
|
|
assert d.reason == "no rule matched"
|
|
|
|
|
|
def test_empty_source_model_is_passthrough() -> None:
|
|
router = _router(ModelRoute(to_model="mini"))
|
|
d = router.select(model="", input_tokens=10, has_tools=False)
|
|
assert not d.matched and d.routed_model == ""
|
|
|
|
|
|
def test_enabled_requires_routes() -> None:
|
|
assert not ModelRouter(ModelRouterConfig(enabled=True, routes=())).enabled
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ModelDecision
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_decision_changed_only_when_model_differs() -> None:
|
|
assert ModelDecision("a", "b", matched=True, reason="x").changed
|
|
assert not ModelDecision("a", "a", matched=True, reason="x").changed
|
|
assert not ModelDecision("a", "b", matched=False, reason="x").changed
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ModelRouterConfig.from_env (fail-open parsing)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_from_env_disabled_by_default() -> None:
|
|
cfg = ModelRouterConfig.from_env(None, None)
|
|
assert not cfg.enabled and cfg.routes == ()
|
|
|
|
|
|
def test_from_env_parses_routes() -> None:
|
|
routes = (
|
|
'[{"name":"small","max_input_tokens":4000,"require_no_tools":true,'
|
|
'"to_model":"gpt-5.4-mini","from_models":["gpt-5.5"]}]'
|
|
)
|
|
cfg = ModelRouterConfig.from_env("true", routes)
|
|
assert cfg.enabled
|
|
assert len(cfg.routes) == 1
|
|
r = cfg.routes[0]
|
|
assert r.to_model == "gpt-5.4-mini"
|
|
assert r.max_input_tokens == 4000
|
|
assert r.require_no_tools is True
|
|
assert r.from_models == ("gpt-5.5",)
|
|
|
|
|
|
def test_from_env_enabled_but_no_routes_disables() -> None:
|
|
cfg = ModelRouterConfig.from_env("true", None)
|
|
assert not cfg.enabled
|
|
|
|
|
|
def test_from_env_malformed_json_fails_open() -> None:
|
|
cfg = ModelRouterConfig.from_env("true", "{not json")
|
|
assert not cfg.enabled and cfg.routes == ()
|
|
|
|
|
|
def test_from_env_non_array_json_ignored() -> None:
|
|
cfg = ModelRouterConfig.from_env("true", '{"to_model":"x"}')
|
|
assert cfg.routes == ()
|
|
|
|
|
|
def test_from_env_skips_bad_entries_keeps_good() -> None:
|
|
routes = '[{"no_to_model":true}, {"to_model":"mini","max_input_tokens":"3000"}]'
|
|
cfg = ModelRouterConfig.from_env("1", routes)
|
|
assert len(cfg.routes) == 1
|
|
assert cfg.routes[0].to_model == "mini"
|
|
# numeric string coerced
|
|
assert cfg.routes[0].max_input_tokens == 3000
|
|
|
|
|
|
def test_from_env_malformed_int_skips_route() -> None:
|
|
# A bool or non-numeric token bound must fail open (skip the route), never
|
|
# silently widen to "no cap".
|
|
assert (
|
|
ModelRouterConfig.from_env("yes", '[{"to_model":"m","max_input_tokens":true}]').routes == ()
|
|
)
|
|
assert (
|
|
ModelRouterConfig.from_env("yes", '[{"to_model":"m","min_input_tokens":"abc"}]').routes
|
|
== ()
|
|
)
|
|
|
|
|
|
def test_from_env_malformed_require_no_tools_skips_route() -> None:
|
|
# A string "false" must not be coerced to True.
|
|
cfg = ModelRouterConfig.from_env("yes", '[{"to_model":"m","require_no_tools":"false"}]')
|
|
assert cfg.routes == ()
|
|
|
|
|
|
def test_from_env_parses_require_tools() -> None:
|
|
cfg = ModelRouterConfig.from_env(
|
|
"yes", '[{"name":"agentic","require_tools":true,"to_model":"strong"}]'
|
|
)
|
|
assert len(cfg.routes) == 1
|
|
route = cfg.routes[0]
|
|
assert route.require_tools is True
|
|
assert route.to_model == "strong"
|
|
|
|
|
|
def test_from_env_malformed_require_tools_skips_route() -> None:
|
|
# A non-boolean must fail open (skip), never be coerced.
|
|
cfg = ModelRouterConfig.from_env("yes", '[{"to_model":"m","require_tools":"yes"}]')
|
|
assert cfg.routes == ()
|
|
|
|
|
|
def test_from_env_malformed_from_models_skips_route() -> None:
|
|
assert (
|
|
ModelRouterConfig.from_env("yes", '[{"to_model":"m","from_models":"gpt-5.5"}]').routes == ()
|
|
)
|
|
assert ModelRouterConfig.from_env("yes", '[{"to_model":"m","from_models":[1,2]}]').routes == ()
|
|
|
|
|
|
def test_from_env_negative_bound_skips_route() -> None:
|
|
# A negative bound would match everything; it must fail open (skip the route).
|
|
assert (
|
|
ModelRouterConfig.from_env("yes", '[{"to_model":"m","min_input_tokens":-1}]').routes == ()
|
|
)
|
|
assert (
|
|
ModelRouterConfig.from_env("yes", '[{"to_model":"m","max_input_tokens":-5}]').routes == ()
|
|
)
|
|
|
|
|
|
def test_from_env_unknown_key_skips_route() -> None:
|
|
# A misspelled condition key must not be silently ignored (which would widen
|
|
# the rule to match everything).
|
|
assert ModelRouterConfig.from_env("yes", '[{"to_model":"m","max_input_token":5}]').routes == ()
|
|
assert ModelRouterConfig.from_env("yes", '[{"to_model":"m","typo":true}]').routes == ()
|
|
|
|
|
|
def test_from_env_valid_bool_and_ints_kept() -> None:
|
|
cfg = ModelRouterConfig.from_env(
|
|
"yes",
|
|
'[{"to_model":"m","require_no_tools":false,"max_input_tokens":10,"min_input_tokens":0}]',
|
|
)
|
|
assert len(cfg.routes) == 1
|
|
r = cfg.routes[0]
|
|
assert r.require_no_tools is False and r.max_input_tokens == 10 and r.min_input_tokens == 0
|
|
|
|
|
|
def test_from_env_various_truthy_values() -> None:
|
|
for v in ("1", "true", "YES", "on", "enabled"):
|
|
assert ModelRouterConfig.from_env(v, '[{"to_model":"m"}]').enabled, v
|
|
for v in ("0", "false", "", "off", None):
|
|
assert not ModelRouterConfig.from_env(v, '[{"to_model":"m"}]').enabled
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# estimate_input_tokens
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_estimate_input_tokens_basic() -> None:
|
|
messages = [{"role": "user", "content": "a" * 400}]
|
|
assert estimate_input_tokens(messages) == 100
|
|
|
|
|
|
def test_estimate_input_tokens_includes_tools() -> None:
|
|
with_tools = estimate_input_tokens([{"content": "x" * 40}], tools=[{"name": "y" * 40}])
|
|
without = estimate_input_tokens([{"content": "x" * 40}])
|
|
assert with_tools > without
|
|
|
|
|
|
def test_estimate_input_tokens_never_raises() -> None:
|
|
assert estimate_input_tokens(None) == 0
|
|
assert estimate_input_tokens("not a list") == 0
|
|
assert estimate_input_tokens([123, {"content": "ok"}]) >= 0
|
|
|
|
|
|
def test_estimate_input_tokens_counts_system_string() -> None:
|
|
# A large top-level system prompt must not be ignored.
|
|
small = estimate_input_tokens([{"content": "hi"}])
|
|
with_system = estimate_input_tokens([{"content": "hi"}], system="s" * 4000)
|
|
assert with_system >= small + 900
|
|
|
|
|
|
def test_estimate_input_tokens_counts_system_blocks() -> None:
|
|
blocks = [{"type": "text", "text": "x" * 4000}]
|
|
assert estimate_input_tokens([{"content": "hi"}], system=blocks) > 100
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Shadow mode (HEADROOM_MODEL_ROUTER_LOG_ONLY)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_from_env_parses_log_only() -> None:
|
|
cfg = ModelRouterConfig.from_env("1", '[{"to_model":"cheap","max_input_tokens":10000}]', "true")
|
|
assert cfg.enabled is True
|
|
assert cfg.log_only is True
|
|
|
|
|
|
def test_log_only_defaults_false() -> None:
|
|
cfg = ModelRouterConfig.from_env("1", '[{"to_model":"cheap"}]')
|
|
assert cfg.log_only is False
|
|
assert ModelRouter(cfg).log_only is False
|
|
|
|
|
|
def test_log_only_matches_but_leaves_model_unchanged() -> None:
|
|
router = _router(
|
|
ModelRoute(to_model="cheap", max_input_tokens=10_000, name="small"),
|
|
)
|
|
# Force shadow mode via config.
|
|
router = ModelRouter(
|
|
ModelRouterConfig(
|
|
enabled=True,
|
|
routes=(ModelRoute(to_model="cheap", max_input_tokens=10_000, name="small"),),
|
|
log_only=True,
|
|
)
|
|
)
|
|
d = router.select(model="strong", input_tokens=100, has_tools=False)
|
|
# A rule matched, but the model is NOT rewritten (changed is False).
|
|
assert d.matched is True
|
|
assert d.changed is False
|
|
assert d.routed_model == "strong"
|
|
assert d.rule_name == "small"
|
|
# The reason records what routing WOULD have done, for the handler log.
|
|
assert d.reason.startswith("[shadow] would route strong -> cheap")
|
|
|
|
|
|
def test_log_only_off_still_routes() -> None:
|
|
router = ModelRouter(
|
|
ModelRouterConfig(
|
|
enabled=True,
|
|
routes=(ModelRoute(to_model="cheap", max_input_tokens=10_000),),
|
|
log_only=False,
|
|
)
|
|
)
|
|
d = router.select(model="strong", input_tokens=100, has_tools=False)
|
|
assert d.changed is True
|
|
assert d.routed_model == "cheap"
|
|
|
|
|
|
def test_select_passes_max_tokens_to_rules() -> None:
|
|
router = ModelRouter(
|
|
ModelRouterConfig(
|
|
enabled=True,
|
|
routes=(ModelRoute(to_model="cheap", max_output_tokens=512, name="trivial"),),
|
|
)
|
|
)
|
|
small = router.select(model="strong", input_tokens=10, has_tools=False, max_tokens=256)
|
|
assert small.changed and small.routed_model == "cheap"
|
|
assert "max_tokens=256" in small.reason
|
|
large = router.select(model="strong", input_tokens=10, has_tools=False, max_tokens=4000)
|
|
assert not large.matched and large.routed_model == "strong"
|
|
|
|
|
|
def test_from_env_parses_max_output_tokens() -> None:
|
|
cfg = ModelRouterConfig.from_env(
|
|
"1", '[{"to_model": "cheap", "max_input_tokens": 4096, "max_output_tokens": 512}]'
|
|
)
|
|
assert cfg.enabled
|
|
assert cfg.routes[0].max_output_tokens == 512
|
|
|
|
|
|
def test_from_env_malformed_max_output_tokens_skips_route() -> None:
|
|
for bad in ('"lots"', "true", "-1"):
|
|
cfg = ModelRouterConfig.from_env(
|
|
"1", f'[{{"to_model": "cheap", "max_output_tokens": {bad}}}]'
|
|
)
|
|
assert not cfg.routes, bad
|
|
|
|
|
|
def test_request_max_tokens() -> None:
|
|
assert request_max_tokens({"max_tokens": 4000}) == 4000
|
|
assert request_max_tokens({}) is None
|
|
assert request_max_tokens({"max_tokens": True}) is None
|
|
assert request_max_tokens({"max_tokens": "4000"}) is None
|
|
assert request_max_tokens({"max_tokens": 1.5}) is None
|
|
assert request_max_tokens(None) is None
|
|
|
|
|
|
def test_shadow_mode_respects_max_output_tokens_condition() -> None:
|
|
router = ModelRouter(
|
|
ModelRouterConfig(
|
|
enabled=True,
|
|
log_only=True,
|
|
routes=(ModelRoute(to_model="cheap", max_output_tokens=512, name="small-output"),),
|
|
)
|
|
)
|
|
small = router.select(model="strong", input_tokens=10, has_tools=False, max_tokens=256)
|
|
assert small.matched
|
|
assert not small.changed
|
|
assert small.routed_model == "strong"
|
|
assert "[shadow]" in small.reason
|
|
large = router.select(model="strong", input_tokens=10, has_tools=False, max_tokens=4000)
|
|
assert not large.matched
|
|
assert not large.changed
|