mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(providers): isolate resolved Anthropic context limits (#4593)
## Description
Addresses the order-dependent Anthropic context-limit resolution
reported in #4390. The proxy reuses its provider instance: resolving
`gpt-4` first currently makes a later `gpt-4-turbo` lookup inherit 8,192
tokens through alias matching, although LiteLLM reports 128,000.
Keep dynamically resolved limits in an exact-model cache so each model
reaches its own metadata lookup. The broader client-supplied
context-limit/header proposal in #4390 remains a separate feature.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Separate resolved limits from the built-in/configured entries used for
release and alias matching.
- Preserve exact-result caching, configured aliases, `[1m]` handling,
and recognition of previously resolved models.
- Cover both lookup orders, both LiteLLM limit fields, unknown-model and
Claude fallbacks, date aliases, repeated lookups, and explicit
overrides.
## Testing
- [x] Relevant unit tests pass
- [x] Linting passes (`python -m ruff check .`)
- [x] Type checking passes for the Linux target (`python -m mypy
headroom --ignore-missing-imports --platform linux`)
- [x] New regression tests added
- [x] Real provider-runtime replay performed
### Test Output
```text
New regressions against fc1ba891: 14 failed, 3 passed
python -m pytest tests/test_providers tests/test_provider_model_fallback.py tests/test_provider_model_metadata.py tests/test_provider_registry.py tests/test_provider_registry_extended.py tests/test_proxy_anthropic_model_sanitization.py
304 passed
Makefile ci-precheck-python test selection, using the locally built Rust extension:
178 passed, 5 skipped
python -m ruff check .
All checks passed!
python -m ruff format . --check
1860 files already formatted
python -m mypy headroom --ignore-missing-imports --platform linux
Success: no issues found in 579 source files
cargo fmt --all -- --check: passed
cargo clippy --workspace -- -D warnings: passed
commitlint --from origin/main --to HEAD --config .commitlintrc.json: passed
```
Windows validation limits: native Windows mypy reports seven existing
POSIX-attribute errors (`os.fchmod`, `O_DIRECTORY`, `O_NOFOLLOW`,
`O_NONBLOCK`), with the same errors reproduced using the original
provider source. The local mypy hook was skipped after the equivalent
Linux-target check above passed. Other applicable pre-commit and
commit-message hooks passed.
`cargo test --workspace` compiled successfully with Rust 1.95.0,
`x86_64-pc-windows-gnu`, then the unchanged `headroom-core` test
executable exited with `0xc0000005` before running tests; invoking it
with `--list` produced the same failure. Rust tests therefore have no
local passing result. The Windows Make wrapper also fails to interpret
the shell recipe (`command was unexpected at this time`). Push uses the
hook's documented `--no-verify` option after the explicit checks above.
The Rust extension was built from this checkout, installed editable, and
import-verified with `DiffCompressor` and `SmartCrusher`.
## Real Behavior Proof
- Environment: Windows x64, Python 3.14.5, LiteLLM 1.101.3, default
`ProxyConfig`, locally built editable Headroom. Each sequence uses the
actual shared Anthropic provider from `build_proxy_provider_runtime` and
LiteLLM's installed model catalog.
- Exact steps, run with the checkout's Python:
```python
from headroom.providers.registry import build_proxy_provider_runtime
from headroom.proxy.models import ProxyConfig
import litellm
for order in [("gpt-4", "gpt-4-turbo"), ("gpt-4-turbo", "gpt-4")]:
runtime = build_proxy_provider_runtime(ProxyConfig())
provider = runtime.pipeline_provider("anthropic")
assert runtime.pipeline_provider("anthropic") is provider
actual = {model: provider.get_context_limit(model) for model in order}
expected = {model: litellm.get_model_info(model)["max_input_tokens"] for model in order}
print(order, actual, expected)
assert actual == expected
```
- Before: short-ID-first returns `gpt-4=8192, gpt-4-turbo=8192`; reverse
order returns the correct values.
- After: both orders return `gpt-4=8192, gpt-4-turbo=128000`, matching
the real catalog.
- Not tested: live model-provider requests, OpenCode traffic, or
provider billing. The reproduction exercises the production
runtime/provider objects without mocks or online inference calls. The
complete Python suite was not run.
## Runtime Rollout Safety
- Rollout-managed features/channel: existing default provider
resolution; no new rollout flag.
- Stable/default behavior changed: resolved model IDs retain their own
context limits across successive requests.
- Disable/rollback: configure an exact context limit through the
existing configuration, or revert this commit.
- Unsafe override required: none.
- Qualification impact: provider-resolution regressions and runtime
replay above.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] Code follows the project's style guidelines
- [x] Added regressions that fail before the fix
- [x] Relevant new and existing Python tests pass locally
- [x] `CHANGELOG.md` is unchanged
- Documentation: existing configuration and public API remain
applicable.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4593"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -811,6 +811,10 @@ class AnthropicProvider(Provider):
|
||||
if context_limits:
|
||||
self._context_limits.update(context_limits)
|
||||
|
||||
# Dynamically resolved limits belong to an exact model id. Keep them
|
||||
# separate from the configured entries used for release/alias matching.
|
||||
self._context_limit_cache: dict[str, int] = {}
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "anthropic"
|
||||
@@ -856,6 +860,8 @@ class AnthropicProvider(Provider):
|
||||
# Check explicit and loaded limits
|
||||
if model in self._context_limits:
|
||||
return self._context_limits[model]
|
||||
if model in self._context_limit_cache:
|
||||
return self._context_limit_cache[model]
|
||||
|
||||
# Releases and aliases of a known id (e.g. "claude-3-5-sonnet" matches
|
||||
# "claude-3-5-sonnet-20241022"); see `_is_release_of`.
|
||||
@@ -871,11 +877,11 @@ class AnthropicProvider(Provider):
|
||||
if info:
|
||||
if "max_input_tokens" in info and info["max_input_tokens"] is not None:
|
||||
limit = int(info["max_input_tokens"])
|
||||
self._context_limits[model] = limit
|
||||
self._context_limit_cache[model] = limit
|
||||
return limit
|
||||
if "max_tokens" in info and info["max_tokens"] is not None:
|
||||
limit = int(info["max_tokens"])
|
||||
self._context_limits[model] = limit
|
||||
self._context_limit_cache[model] = limit
|
||||
return limit
|
||||
except Exception as e:
|
||||
logger.debug(f"LiteLLM get_model_info failed for {model}: {e}")
|
||||
@@ -886,20 +892,20 @@ class AnthropicProvider(Provider):
|
||||
limit = cast(int, _PATTERN_DEFAULTS[tier]["context"])
|
||||
self._warn_unknown_model(model, limit, f"inferred from '{tier}' tier")
|
||||
# Cache for future calls
|
||||
self._context_limits[model] = limit
|
||||
self._context_limit_cache[model] = limit
|
||||
return limit
|
||||
|
||||
# Fallback for unknown Claude models
|
||||
if model.startswith("claude"):
|
||||
limit = cast(int, _UNKNOWN_CLAUDE_DEFAULT["context"])
|
||||
self._warn_unknown_model(model, limit, "using default Claude limit")
|
||||
self._context_limits[model] = limit
|
||||
self._context_limit_cache[model] = limit
|
||||
return limit
|
||||
|
||||
# Non-Claude model - use conservative default
|
||||
limit = 128000
|
||||
self._warn_unknown_model(model, limit, "unknown provider, using conservative default")
|
||||
self._context_limits[model] = limit
|
||||
self._context_limit_cache[model] = limit
|
||||
return limit
|
||||
|
||||
def _warn_unknown_model(self, model: str, limit: int, reason: str) -> None:
|
||||
@@ -916,7 +922,7 @@ class AnthropicProvider(Provider):
|
||||
def supports_model(self, model: str) -> bool:
|
||||
"""Check if this provider supports the given model."""
|
||||
model = sanitize_anthropic_model_id(model)
|
||||
if model in self._context_limits:
|
||||
if model in self._context_limits or model in self._context_limit_cache:
|
||||
return True
|
||||
# Check prefix matches - support all Claude models
|
||||
return model.startswith("claude")
|
||||
|
||||
@@ -3,6 +3,73 @@
|
||||
import pytest
|
||||
|
||||
|
||||
class TestContextLimitCache:
|
||||
@pytest.mark.parametrize("metadata_field", ["max_input_tokens", "max_tokens"])
|
||||
@pytest.mark.parametrize(
|
||||
("first", "second", "metadata", "expected"),
|
||||
[
|
||||
("gpt-4", "gpt-4-turbo", {"gpt-4": 8192, "gpt-4-turbo": 128000}, (8192, 128000)),
|
||||
("gpt-4-turbo", "gpt-4", {"gpt-4": 8192, "gpt-4-turbo": 128000}, (128000, 8192)),
|
||||
("gateway", "gateway-large", {"gateway-large": 1000000}, (128000, 1000000)),
|
||||
("gateway", "gateway-unknown", {"gateway": 64000}, (64000, 128000)),
|
||||
(
|
||||
"claude-future-sonnet",
|
||||
"claude-future-sonnet-extended",
|
||||
{"claude-future-sonnet-extended": 1000000},
|
||||
(200000, 1000000),
|
||||
),
|
||||
("gateway", "gateway-sonnet", {}, (128000, 200000)),
|
||||
(
|
||||
"claude-future",
|
||||
"claude-future-long",
|
||||
{"claude-future-long": 640000},
|
||||
(200000, 640000),
|
||||
),
|
||||
(
|
||||
"gateway-20260101",
|
||||
"gateway",
|
||||
{"gateway-20260101": 64000, "gateway": 128000},
|
||||
(64000, 128000),
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_each_model_keeps_its_own_resolved_limit(
|
||||
self, monkeypatch, first, second, metadata, expected, metadata_field
|
||||
):
|
||||
import headroom.providers.anthropic as anthropic_module
|
||||
|
||||
lookups = []
|
||||
|
||||
def get_model_info(model):
|
||||
lookups.append(model)
|
||||
return {metadata_field: metadata[model]} if model in metadata else None
|
||||
|
||||
monkeypatch.setattr(
|
||||
anthropic_module, "_get_litellm_clients", lambda: (None, get_model_info)
|
||||
)
|
||||
provider = anthropic_module.AnthropicProvider(warn=False)
|
||||
|
||||
assert (provider.get_context_limit(first), provider.get_context_limit(second)) == expected
|
||||
assert (provider.get_context_limit(first), provider.get_context_limit(second)) == expected
|
||||
assert lookups == [first, second]
|
||||
assert provider.supports_model(first)
|
||||
assert provider.supports_model(second)
|
||||
|
||||
def test_configured_aliases_and_1m_suffix_keep_their_limits(self, monkeypatch):
|
||||
import headroom.providers.anthropic as anthropic_module
|
||||
|
||||
monkeypatch.setattr(anthropic_module, "_get_litellm_clients", lambda: (None, None))
|
||||
provider = anthropic_module.AnthropicProvider(
|
||||
context_limits={"gateway": 64000, "gateway-exact": 96000}, warn=False
|
||||
)
|
||||
|
||||
assert provider.get_context_limit("gateway-20260101") == 64000
|
||||
assert provider.get_context_limit("gateway-exact") == 96000
|
||||
assert provider.get_context_limit("gateway[1m]") == 1000000
|
||||
assert provider.get_context_limit("gateway") == 64000
|
||||
assert provider.get_context_limit("claude-sonnet-4-5-20250929") == 200000
|
||||
|
||||
|
||||
class TestAnthropicModelSanitization:
|
||||
def test_sanitize_model_id_removes_ansi_escape_sequences(self):
|
||||
from headroom.providers.anthropic import sanitize_anthropic_model_id
|
||||
|
||||
Reference in New Issue
Block a user