mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): break the compact-every-turn loop near the real context window (#2942)
## Description Closes #3998 Compression makes the usage reported back to the client reflect the *forwarded* (compressed) request, not the client's raw transcript. Clients whose auto-compaction keys off reported usage (Claude Code) therefore defer compaction — the selling point — but with no floor: the raw transcript can grow until even the compressed request exceeds the model's real context window. From that point every turn fails with a 400 `prompt is too long`, the client force-compacts on each error, and the session degrades into a compact-every-other-prompt loop. This came from a churned-user field report (2026-08-12): "Headroom cut off my context and started compacting every two prompts." A second, related bug: `get_context_limit()` resolves purely from the model id, so a session running with the `context-1m` beta is treated as 5x over budget by compression pressure (`context_pressure` pins at 1.0) and gets maximum-aggression lossy compression from early on — the "cut off my context" half of the report. This PR adds `headroom/proxy/context_guard.py` with three cooperating pieces, all passthrough until the danger zone: 1. `effective_context_limit` — the window the forwarded request is really subject to: the registry limit, raised to 1M when the client sent a `context-1m` beta token, capped by any limit learned from a real `prompt is too long` error. Also used for compression pressure at the `context_limit` call site. 2. `note_prompt_too_long` — parses `prompt is too long: N tokens > M maximum` from upstream 400 bodies (streaming and buffered error paths) and records M keyed by (model, has-1m-beta). This is how a client that believes 1M but whose account is capped at 200k stops looping after exactly one error — and why the learned limit must not leak between beta shapes. 3. `MessageStartGuard` — an SSE transformer applied to the client-bound bytes only: when the `message_start` usage total reaches 90% of the effective limit, it inflates the reported `input_tokens` so the client's gauge reads 95% of the window the client believes in, which trips the client's own graceful compaction *before* the 400 wall. Inert after a single decision (steady-state overhead is one `if`); savings metrics and telemetry parse the original upstream bytes, never the nudged ones. Kill switch: `HEADROOM_CONTEXT_GUARD=0`. Out of scope, deliberately: the buffered (non-streaming) success path is not nudged — the affected clients (Claude Code) always stream, and rewriting buffered bodies would break response byte-fidelity for a path with no known symptom. The batch endpoint's `context_limit` site is also untouched (batch has no interactive compaction loop). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - New `headroom/proxy/context_guard.py`: limit resolution (`believed_context_limit`, `effective_context_limit`), 400-learning (`note_prompt_too_long`), and the `MessageStartGuard` SSE usage rewriter, plus the `HEADROOM_CONTEXT_GUARD` kill switch. - `headroom/proxy/handlers/streaming.py`: build the guard for anthropic streams (inside try/except, passthrough on any setup failure), route yielded chunks through `guard.feed()`, flush held bytes at stream end, and learn from streaming 400 bodies. - `headroom/proxy/handlers/anthropic.py`: compression `context_limit` now uses `effective_context_limit` (1m-beta aware, learned-cap aware); the existing UPSTREAM_ERROR block also feeds `note_prompt_too_long`. - `tests/test_proxy/test_context_guard.py`: 25 tests — unit coverage of limits/learning/rewriter (chunk splits, pings, malformed JSON, buffer cap, never-deflate) and four integration tests driving `_stream_response` with a mocked upstream (nudge on the wire, byte-identical passthrough far from the limit, kill switch, 400 learning). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_proxy/test_context_guard.py -q 25 passed in 0.53s $ uv run --frozen --extra dev pytest tests/test_proxy/ -q 272 passed, 1 warning in 71.42s (0:01:11) $ uv run --frozen --extra dev pytest tests/test_sse_thinking_blocks.py tests/test_h2_stream_reset_retry.py \ tests/test_proxy_streaming_ratelimit_headers.py tests/test_request_outcome.py tests/test_proxy_retry_429.py -q 81 passed in 1.53s $ uv run --frozen --extra dev mypy headroom/proxy/context_guard.py Success: no issues found in 1 source file $ uvx ruff check headroom/proxy/context_guard.py headroom/proxy/handlers/streaming.py \ headroom/proxy/handlers/anthropic.py tests/test_proxy/test_context_guard.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS 15 (Darwin 24.6.0), Python 3.10 project venv via uv, repo at this branch; live runs used a sandboxed proxy instance (this module injected) with Claude Code 2.1.206 as the real client. - Exact command / steps: ran the four integration tests in `TestStreamResponseIntegration`, which drive the real `_stream_response` code path with a mocked upstream httpx response: (a) a `message_start` carrying a 185k-token usage against a 200k model limit (above the 90% trigger, below the 95% target, so the assertion can only pass if the rewrite happened), (b) the same stream at 50k tokens, (c) a 190k stream with `HEADROOM_CONTEXT_GUARD=0`, (d) a 400 body containing `prompt is too long: 213021 tokens > 200000 maximum` with `anthropic-beta: context-1m-2025-08-07`. - Observed result: (a) the client-received `message_start` reports `input_tokens` 185000 inflated to 190000 (the 95% target) while the `message_stop` bytes pass through unmodified; (b) the client-received bytes are byte-identical to the upstream bytes; (c) byte-identical passthrough with the kill switch set; (d) the 400 is forwarded unchanged and `effective_context_limit("claude-sonnet-4-5", 200000, "context-1m-2025-08-07")` returns 200000 afterwards. - Live end-to-end (added after the initial PR): drove a real Claude Code 2.1.206 session through a proxy running this code, with a learned limit seeded to the state a real prompt-too-long 400 produces. Result: the client-visible usage total read exactly 190000 (95% of the believed 200k window) and the interactive TUI showed "100% context used". This run is also why the second commit exists: with only message_start rewritten, the client kept displaying the raw values because it merges the final cumulative-usage message_delta over message_start. Forcing a prompt-too-long 400 on the next turn produced a clean "Compacting conversation..." -> summarization request -> retried turn -> correct response, with no user-facing error. - Re-validated 2026-10-06 at `739bacfa`, merged with upstream/main `855390d6` (0.40.0): the repro in #3998 runs the real app with only the upstream HTTP call mocked. On main a forwarded 185k turn reaches the client as 185000 in both `message_start` and `message_delta`, a `prompt is too long ... > 200000 maximum` 400 is forgotten (the next 185k turn reads 18.5% of the 1M the client believes in), and a `context-1m` request is compressed against 200k (`context_pressure` 1.00 at 400k). On this branch: 190000 (95% of 200k), 950000 (95% of the believed 1M) after the learned limit, and 1M (`context_pressure` 0.40). - Merge note: main's memory tool stream filter (#3947) changed the same `yield chunk` line. The guard now runs last, on the frames the filter releases, including the filter's closing frames and the memory continuation rounds, and the end-of-stream drain moved after the memory handling so the `message_delta` the filter holds back is still nudged. `test_guard_nudges_what_the_memory_filter_releases` covers it (fails with `185000 == 190000` if the drain moves back). `tests/test_proxy/` 432 passed, the memory tool suites 95 passed. - Not tested: an organic (unseeded) 180k+-token conversation against the production API; the buffered non-streaming success path is deliberately not nudged (see Description). Note for reviewers: Claude Code 2.1.206 compacts reactively at prompt-too-long (with a threshold-armed precomputed summary) rather than proactively pre-send, so the nudge's role there is an honest gauge/warnings and threshold arming; older clients and other harnesses that compact proactively off reported usage get the pre-wall compaction directly. ## Runtime Rollout Safety - Rollout-managed feature(s): None. Not in the rollout FEATURES registry; gated by its own env switch, `HEADROOM_CONTEXT_GUARD` (default on). - Minimum rollout channel: N/A (not rollout-managed; applies on every channel) - Stable/default behavior changed: Yes, for Anthropic requests only. Compression pressure uses the effective context window (1M when the client sent the `context-1m` beta, capped by any limit learned from a real `prompt is too long` 400), and a client-bound `message_start` whose usage reaches 90% of that window reports `input_tokens` raised to 95% of it so the client compacts before the wall. Below 90% the client receives byte-identical upstream bytes. Upstream requests and error bodies are never modified. - Kill switch / disable path: `HEADROOM_CONTEXT_GUARD=0` (or `false`). Every call site checks it before using the effective limit or rewriting usage, so with it off the proxy behaves exactly as before this PR (covered by the kill-switch integration test). - Unsafe override required: No - Qualification impact: Adds `tests/test_proxy/test_context_guard.py`; no existing test changed or relaxed. - Rollback path: Set `HEADROOM_CONTEXT_GUARD=0` or revert the PR. Learned limits live in process memory only, so nothing persisted needs cleanup. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from the Conventional Commit PR title (a CI guard enforces this) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2942" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Fable 5
JD Davis
parent
0d7d61d5cc
commit
76cceeb487
@@ -0,0 +1,516 @@
|
||||
"""Context-limit guard: keep client auto-compaction working under compression.
|
||||
|
||||
Compression makes the usage reported back to the client reflect the
|
||||
*forwarded* (compressed) request, not the client's raw transcript. Clients
|
||||
whose auto-compaction keys off reported usage (Claude Code) therefore defer
|
||||
compaction — the selling point — but with no floor: the raw transcript can
|
||||
grow until even the compressed request exceeds the model's real context
|
||||
window. From that point every turn fails with a 400 ``prompt is too long``,
|
||||
the client force-compacts on each error, and the session degrades into a
|
||||
compact-every-other-prompt loop (field report 2026-08-12).
|
||||
|
||||
Three cooperating pieces, all passthrough until the danger zone:
|
||||
|
||||
1. ``effective_context_limit`` — the window the forwarded request is really
|
||||
subject to: the model's registry limit, raised to 1M when the client sent
|
||||
a ``context-1m`` beta token, capped by any limit *learned* from an actual
|
||||
``prompt is too long`` error (piece 2). Also used for compression
|
||||
pressure, so 1M sessions stop being treated as 5x over budget.
|
||||
2. ``note_prompt_too_long`` — parses ``prompt is too long: N tokens > M
|
||||
maximum`` from upstream 400 bodies and records M per (model, 1m-beta)
|
||||
key. This is how a client that *believes* 1M but whose account is capped
|
||||
at 200k stops looping after exactly one error.
|
||||
3. ``StreamUsageGuard`` — an SSE transformer for the streamed bytes only:
|
||||
when the ``message_start`` usage total reaches ``TRIGGER_FRACTION`` of
|
||||
the effective limit, it inflates the reported ``input_tokens`` — in both
|
||||
``message_start`` and the final cumulative-usage ``message_delta``, which
|
||||
clients merge over the former — so the client's gauge reads
|
||||
``REPORT_FRACTION`` of the window the client believes in, which trips
|
||||
the client's own graceful compaction before the 400 wall. Savings
|
||||
metrics and telemetry parse the original upstream bytes, never the
|
||||
nudged ones.
|
||||
|
||||
The nudged number is deliberately unmarked on the wire. It exists to be read
|
||||
by the client's own compaction gauge, which is a third-party parser we do not
|
||||
control, so anything added beside it is either ignored or a wire-shape change
|
||||
against a client we cannot test. Nothing inside headroom needs the marker:
|
||||
savings, telemetry and billing all read the original upstream bytes, never the
|
||||
client-bound ones, and each rewrite emits ``event=context_guard_nudge`` with
|
||||
the forwarded and reported totals for reconciliation.
|
||||
|
||||
Kill switch: ``HEADROOM_CONTEXT_GUARD=0``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from headroom.proxy.sse_byte_buffer_policy import find_sse_event_terminator
|
||||
|
||||
logger = logging.getLogger("headroom.proxy")
|
||||
|
||||
# Forwarded-usage fraction of the *effective* (real) limit at which the nudge
|
||||
# arms. 0.90 leaves one normal turn of headroom before the hard wall.
|
||||
TRIGGER_FRACTION = 0.90
|
||||
# Reported-usage fraction of the *believed* limit the nudge inflates to.
|
||||
# Claude Code auto-compacts around 92%; 0.95 clears that with margin.
|
||||
REPORT_FRACTION = 0.95
|
||||
|
||||
_CONTEXT_1M_PREFIX = "context-1m"
|
||||
|
||||
_PROMPT_TOO_LONG_RE = re.compile(
|
||||
r"prompt is too long:\s*(\d+)\s*tokens?\s*>\s*(\d+)\s*maximum", re.IGNORECASE
|
||||
)
|
||||
|
||||
# A learned maximum is believed only inside this band. Below the floor it is
|
||||
# not a context window: gateways return the same "prompt is too long: N > M"
|
||||
# shape for per-request quota and per-key throttles, and one such number would
|
||||
# clamp every later request on that model for the life of the process. Above
|
||||
# the ceiling it cannot cap anything anyway.
|
||||
_MIN_LEARNED_LIMIT = 8_000
|
||||
_MAX_LEARNED_LIMIT = 10_000_000
|
||||
|
||||
# A learned limit is evidence about one account at one moment, not a fact about
|
||||
# the model: entitlements get provisioned, gateways get repointed. It expires so
|
||||
# a stale cap cannot hold a budget down forever -- the cost of expiry is at most
|
||||
# one more prompt-too-long error, which is what taught it the first time.
|
||||
_LEARNED_LIMIT_TTL_SECONDS = 3600.0
|
||||
|
||||
# (model, has_1m_beta, credential scope) -> (maximum, learned-at monotonic time).
|
||||
#
|
||||
# The 1m-beta dimension matters: an account with real 1M access must not inherit
|
||||
# a 200k limit learned from a request that never sent the beta. The credential
|
||||
# scope matters for the same reason one step out -- this dict is process-global,
|
||||
# and on a shared proxy one tenant's capped key would otherwise clamp the budget
|
||||
# of every other tenant on that model.
|
||||
_learned_limits: dict[tuple[str, bool, str], tuple[int, float]] = {}
|
||||
|
||||
# Give up scanning for message_start beyond this many buffered bytes.
|
||||
_MAX_GUARD_BUFFER = 256 * 1024
|
||||
|
||||
|
||||
def context_guard_enabled() -> bool:
|
||||
"""True unless explicitly disabled via HEADROOM_CONTEXT_GUARD=0."""
|
||||
from headroom.proxy import runtime_env
|
||||
|
||||
return (runtime_env.getenv("HEADROOM_CONTEXT_GUARD", "1") or "1").strip().lower() not in (
|
||||
"0",
|
||||
"false",
|
||||
"no",
|
||||
"off",
|
||||
)
|
||||
|
||||
|
||||
def has_context_1m_beta(beta_header: str | None) -> bool:
|
||||
"""True when the anthropic-beta header carries a context-1m token."""
|
||||
if not beta_header:
|
||||
return False
|
||||
return any(
|
||||
token.strip().lower().startswith(_CONTEXT_1M_PREFIX) for token in beta_header.split(",")
|
||||
)
|
||||
|
||||
|
||||
def believed_context_limit(model_limit: int, beta_header: str | None) -> int:
|
||||
"""The context window the *client* thinks it has."""
|
||||
if has_context_1m_beta(beta_header):
|
||||
return max(model_limit, 1_000_000)
|
||||
return model_limit
|
||||
|
||||
|
||||
def credential_scope(credential: str | None) -> str:
|
||||
"""A stable, non-reversible tag for the upstream credential.
|
||||
|
||||
Only ever a truncated digest: the credential itself is never stored, never
|
||||
logged, and never compared outside this function.
|
||||
"""
|
||||
if not credential:
|
||||
return ""
|
||||
return hashlib.sha256(credential.strip().encode("utf-8", "ignore")).hexdigest()[:16]
|
||||
|
||||
|
||||
def credential_scope_from_headers(headers: Any) -> str:
|
||||
"""Credential scope for an outbound header mapping, best effort.
|
||||
|
||||
Never raises: an unusable header mapping just means the unscoped bucket,
|
||||
which is the behaviour of a single-tenant proxy anyway.
|
||||
"""
|
||||
try:
|
||||
for name in ("x-api-key", "authorization"):
|
||||
value = headers.get(name) or headers.get(name.title())
|
||||
if isinstance(value, str) and value:
|
||||
return credential_scope(value)
|
||||
except Exception:
|
||||
return ""
|
||||
return ""
|
||||
|
||||
|
||||
def _learned_key(model: str, beta_header: str | None, scope: str) -> tuple[str, bool, str]:
|
||||
return (model, has_context_1m_beta(beta_header), scope)
|
||||
|
||||
|
||||
def _learned_limit(key: tuple[str, bool, str]) -> int | None:
|
||||
"""The unexpired learned maximum for ``key``, dropping it once stale."""
|
||||
entry = _learned_limits.get(key)
|
||||
if entry is None:
|
||||
return None
|
||||
maximum, learned_at = entry
|
||||
if time.monotonic() - learned_at >= _LEARNED_LIMIT_TTL_SECONDS:
|
||||
_learned_limits.pop(key, None)
|
||||
return None
|
||||
return maximum
|
||||
|
||||
|
||||
def effective_context_limit(
|
||||
model: str,
|
||||
model_limit: int,
|
||||
beta_header: str | None,
|
||||
*,
|
||||
scope: str = "",
|
||||
) -> int:
|
||||
"""The window the forwarded request is actually subject to.
|
||||
|
||||
Optimistically the believed limit; capped by a limit learned from a real
|
||||
``prompt is too long`` error for the same (model, beta, credential) shape,
|
||||
and only while that lesson is still fresh.
|
||||
"""
|
||||
believed = believed_context_limit(model_limit, beta_header)
|
||||
learned = _learned_limit(_learned_key(model, beta_header, scope))
|
||||
if learned is not None:
|
||||
return min(believed, learned)
|
||||
return believed
|
||||
|
||||
|
||||
def note_prompt_too_long(
|
||||
model: str,
|
||||
beta_header: str | None,
|
||||
error_text: Any,
|
||||
*,
|
||||
scope: str = "",
|
||||
) -> int | None:
|
||||
"""Learn the real context limit from an upstream prompt-too-long error.
|
||||
|
||||
Returns the learned maximum, or None when there is nothing to learn.
|
||||
|
||||
This runs on the error path, where the client is still waiting for the real
|
||||
upstream body, so it accepts whatever the body turned out to be: upstreams
|
||||
and gateways send ``"message": null``, nested objects, and lists in the
|
||||
slot this reads, and a diagnostic helper must not turn one of those into an
|
||||
exception that replaces the error the caller was about to forward.
|
||||
"""
|
||||
if isinstance(error_text, bytes | bytearray):
|
||||
error_text = bytes(error_text).decode("utf-8", errors="replace")
|
||||
if not isinstance(error_text, str):
|
||||
return None
|
||||
match = _PROMPT_TOO_LONG_RE.search(error_text)
|
||||
if not match:
|
||||
return None
|
||||
maximum = int(match.group(2))
|
||||
if not _MIN_LEARNED_LIMIT <= maximum <= _MAX_LEARNED_LIMIT:
|
||||
# Not a context window. Learning it would clamp the budget on a number
|
||||
# that never described one.
|
||||
logger.debug(
|
||||
"event=context_guard_limit_out_of_band model=%s limit=%d (ignored)",
|
||||
model,
|
||||
maximum,
|
||||
)
|
||||
return None
|
||||
key = _learned_key(model, beta_header, scope)
|
||||
previous = _learned_limit(key)
|
||||
_learned_limits[key] = (maximum, time.monotonic())
|
||||
if previous != maximum:
|
||||
logger.warning(
|
||||
"event=context_guard_learned_limit model=%s context_1m=%s limit=%d "
|
||||
"attempted=%s (client compaction gauge will be nudged near this limit)",
|
||||
model,
|
||||
key[1],
|
||||
maximum,
|
||||
match.group(1),
|
||||
)
|
||||
return maximum
|
||||
|
||||
|
||||
def reset_learned_limits() -> None:
|
||||
"""Test hook: clear limits learned from prompt-too-long errors."""
|
||||
_learned_limits.clear()
|
||||
|
||||
|
||||
def _data_line(payload: dict[str, Any], original: bytes) -> bytes:
|
||||
"""A rewritten ``data:`` line keeping the original's CRLF ending."""
|
||||
eol = b"\r" if original.endswith(b"\r") else b""
|
||||
return b"data: " + json.dumps(payload, separators=(",", ":")).encode() + eol
|
||||
|
||||
|
||||
def _sse_event_type(event: bytes) -> bytes | None:
|
||||
"""The event type of one SSE frame, or ``None`` when it carries none.
|
||||
|
||||
Read from the ``event:`` line, not matched as a substring of the frame: a
|
||||
substring match reads the *payload* too, so an assistant that writes the
|
||||
words ``message_delta`` into its own reply ended the scan and lost the
|
||||
final nudge.
|
||||
|
||||
``None`` means the frame is not part of the message protocol at all --
|
||||
an SSE comment, a ``retry:`` field, the keepalives an intermediary shim
|
||||
injects. Those pass through without arming or disarming anything; treating
|
||||
one as an unknown event disabled the guard for the whole stream.
|
||||
"""
|
||||
saw_data = False
|
||||
for line in event.split(b"\n"):
|
||||
if line.startswith(b"event:"):
|
||||
return line[len(b"event:") :].strip()
|
||||
if line.startswith(b"data:"):
|
||||
saw_data = True
|
||||
if not saw_data:
|
||||
return None
|
||||
# A data frame with no event: line defaults to the type inside the payload.
|
||||
for line in event.split(b"\n"):
|
||||
if not line.startswith(b"data:"):
|
||||
continue
|
||||
try:
|
||||
payload = json.loads(line[len(b"data:") :].strip())
|
||||
except Exception:
|
||||
return b""
|
||||
value = payload.get("type") if isinstance(payload, dict) else None
|
||||
return value.encode() if isinstance(value, str) else b""
|
||||
return b""
|
||||
|
||||
|
||||
def nudged_input_tokens(
|
||||
usage: dict[str, Any],
|
||||
*,
|
||||
believed_limit: int,
|
||||
effective_limit: int,
|
||||
) -> int | None:
|
||||
"""The inflated ``input_tokens`` for this usage, or ``None`` to leave it be.
|
||||
|
||||
One implementation for every client-bound surface -- ``message_start``, the
|
||||
final ``message_delta``, and the non-streaming JSON body -- so a request
|
||||
cannot get the raised compression budget on one path and the un-nudged
|
||||
gauge on another.
|
||||
"""
|
||||
if believed_limit <= 0 or effective_limit <= 0:
|
||||
return None
|
||||
input_tokens = int(usage.get("input_tokens") or 0)
|
||||
cache_read = int(usage.get("cache_read_input_tokens") or 0)
|
||||
cache_creation = int(usage.get("cache_creation_input_tokens") or 0)
|
||||
total = input_tokens + cache_read + cache_creation
|
||||
if total < TRIGGER_FRACTION * effective_limit:
|
||||
return None
|
||||
target_total = int(believed_limit * REPORT_FRACTION)
|
||||
if total >= target_total:
|
||||
# Already reads as full to the client -- never deflate.
|
||||
return None
|
||||
return target_total - cache_read - cache_creation
|
||||
|
||||
|
||||
def nudge_response_usage(
|
||||
resp_json: Any,
|
||||
*,
|
||||
believed_limit: int,
|
||||
effective_limit: int,
|
||||
request_id: str = "",
|
||||
) -> bool:
|
||||
"""Nudge the usage of a non-streaming Anthropic message body, in place.
|
||||
|
||||
The buffered path raises the compression budget exactly like the streaming
|
||||
one, so it has to move the client's gauge exactly like the streaming one
|
||||
too; otherwise a non-streaming client gets the larger forwarded request and
|
||||
none of the warning that it is near the wall.
|
||||
|
||||
Returns True when the body was modified.
|
||||
"""
|
||||
if not isinstance(resp_json, dict):
|
||||
return False
|
||||
usage = resp_json.get("usage")
|
||||
if not isinstance(usage, dict):
|
||||
return False
|
||||
new_input = nudged_input_tokens(
|
||||
usage, believed_limit=believed_limit, effective_limit=effective_limit
|
||||
)
|
||||
if new_input is None:
|
||||
return False
|
||||
cache_read = int(usage.get("cache_read_input_tokens") or 0)
|
||||
cache_creation = int(usage.get("cache_creation_input_tokens") or 0)
|
||||
forwarded_total = int(usage.get("input_tokens") or 0) + cache_read + cache_creation
|
||||
usage["input_tokens"] = new_input
|
||||
logger.warning(
|
||||
"[%s] event=context_guard_nudge path=non_streaming forwarded_total=%d "
|
||||
"effective_limit=%d reported_total=%d believed_limit=%d (nudging client to "
|
||||
"compact before the prompt-too-long wall)",
|
||||
request_id,
|
||||
forwarded_total,
|
||||
effective_limit,
|
||||
new_input + cache_read + cache_creation,
|
||||
believed_limit,
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
class StreamUsageGuard:
|
||||
"""Rewrites near-the-wall usage in the client-bound SSE bytes.
|
||||
|
||||
Feed every chunk headed to the client through :meth:`feed`; call
|
||||
:meth:`flush` when the stream ends. The first ``message_start`` decides
|
||||
whether the nudge is active. When it is not (the common case), the guard
|
||||
goes inert immediately and chunks pass through untouched — steady-state
|
||||
overhead is one ``if``. When it is, the guard rewrites the
|
||||
``message_start`` usage AND stays armed for the final ``message_delta``:
|
||||
since mid-2025 that event carries cumulative input/cache usage and
|
||||
clients (verified live with Claude Code 2026-08-12) let it override the
|
||||
``message_start`` values, so nudging only the first event loses the
|
||||
merge. Events are released as soon as their ``\\n\\n`` terminator
|
||||
arrives, so no latency is added beyond SSE's own event framing.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
believed_limit: int,
|
||||
effective_limit: int,
|
||||
request_id: str = "",
|
||||
) -> None:
|
||||
self._believed_limit = believed_limit
|
||||
self._effective_limit = effective_limit
|
||||
self._request_id = request_id
|
||||
self._buf = bytearray()
|
||||
self._seen_message_start = False
|
||||
self._target_total: int | None = None
|
||||
self._start_cache_read = 0
|
||||
self._start_cache_creation = 0
|
||||
# Inert immediately when limits are unusable.
|
||||
self._done = believed_limit <= 0 or effective_limit <= 0
|
||||
|
||||
def feed(self, chunk: bytes) -> bytes:
|
||||
if self._done:
|
||||
return chunk
|
||||
self._buf.extend(chunk)
|
||||
out = bytearray()
|
||||
while not self._done:
|
||||
# LF or CRLF framing: a CRLF stream never contains ``\n\n`` and
|
||||
# was held until the buffer cap, so it stopped streaming.
|
||||
terminator = find_sse_event_terminator(self._buf)
|
||||
if terminator is None:
|
||||
break
|
||||
end = terminator[0] + terminator[1]
|
||||
event = bytes(self._buf[:end])
|
||||
del self._buf[:end]
|
||||
out += self._process_event(event)
|
||||
# Capped on what is still unframed, so one large chunk carrying
|
||||
# message_start is read before the cap can disarm the guard.
|
||||
if self._done or len(self._buf) > _MAX_GUARD_BUFFER:
|
||||
out += self._finish()
|
||||
return bytes(out)
|
||||
|
||||
def flush(self) -> bytes:
|
||||
"""Return any held-back bytes; the guard goes inert."""
|
||||
return self._finish()
|
||||
|
||||
def _finish(self) -> bytes:
|
||||
self._done = True
|
||||
remaining = bytes(self._buf)
|
||||
self._buf = bytearray()
|
||||
return remaining
|
||||
|
||||
def _process_event(self, event: bytes) -> bytes:
|
||||
event_type = _sse_event_type(event)
|
||||
# Pings, comments, retry fields, keepalives: not part of the message
|
||||
# protocol, so they neither arm nor disarm the guard.
|
||||
if event_type is None or event_type == b"ping":
|
||||
return event
|
||||
if not self._seen_message_start:
|
||||
# Anything data-bearing that isn't message_start (error events,
|
||||
# protocol changes) ends the scan.
|
||||
self._seen_message_start = True
|
||||
if event_type != b"message_start":
|
||||
self._done = True
|
||||
return event
|
||||
try:
|
||||
rewritten = self._rewrite_message_start(event)
|
||||
except Exception:
|
||||
logger.debug("context_guard: message_start rewrite skipped", exc_info=True)
|
||||
self._done = True
|
||||
return event
|
||||
if self._target_total is None:
|
||||
# Below trigger (or already reads full): nothing to nudge.
|
||||
self._done = True
|
||||
return rewritten
|
||||
# Armed: pass content deltas through, rewrite the final usage delta.
|
||||
if event_type != b"message_delta":
|
||||
return event
|
||||
self._done = True
|
||||
try:
|
||||
return self._rewrite_message_delta(event)
|
||||
except Exception:
|
||||
logger.debug("context_guard: message_delta rewrite skipped", exc_info=True)
|
||||
return event
|
||||
|
||||
def _rewrite_message_start(self, event: bytes) -> bytes:
|
||||
lines = event.split(b"\n")
|
||||
for i, line in enumerate(lines):
|
||||
if not line.startswith(b"data:"):
|
||||
continue
|
||||
payload = json.loads(line[len(b"data:") :].strip())
|
||||
if payload.get("type") != "message_start":
|
||||
return event
|
||||
usage = payload.get("message", {}).get("usage")
|
||||
if not isinstance(usage, dict):
|
||||
return event
|
||||
cache_read = int(usage.get("cache_read_input_tokens") or 0)
|
||||
cache_creation = int(usage.get("cache_creation_input_tokens") or 0)
|
||||
total = int(usage.get("input_tokens") or 0) + cache_read + cache_creation
|
||||
new_input = nudged_input_tokens(
|
||||
usage,
|
||||
believed_limit=self._believed_limit,
|
||||
effective_limit=self._effective_limit,
|
||||
)
|
||||
if new_input is None:
|
||||
return event
|
||||
target_total = new_input + cache_read + cache_creation
|
||||
self._target_total = target_total
|
||||
self._start_cache_read = cache_read
|
||||
self._start_cache_creation = cache_creation
|
||||
usage["input_tokens"] = new_input
|
||||
logger.warning(
|
||||
"[%s] event=context_guard_nudge forwarded_total=%d effective_limit=%d "
|
||||
"reported_total=%d believed_limit=%d (nudging client to compact "
|
||||
"before the prompt-too-long wall)",
|
||||
self._request_id,
|
||||
total,
|
||||
self._effective_limit,
|
||||
target_total,
|
||||
self._believed_limit,
|
||||
)
|
||||
lines[i] = _data_line(payload, line)
|
||||
return b"\n".join(lines)
|
||||
return event
|
||||
|
||||
def _rewrite_message_delta(self, event: bytes) -> bytes:
|
||||
assert self._target_total is not None
|
||||
lines = event.split(b"\n")
|
||||
for i, line in enumerate(lines):
|
||||
if not line.startswith(b"data:"):
|
||||
continue
|
||||
payload = json.loads(line[len(b"data:") :].strip())
|
||||
if payload.get("type") != "message_delta":
|
||||
return event
|
||||
usage = payload.get("usage")
|
||||
# A delta without cumulative input usage can't override the
|
||||
# message_start values we already rewrote — leave it alone.
|
||||
if not isinstance(usage, dict) or "input_tokens" not in usage:
|
||||
return event
|
||||
cache_read = int(usage.get("cache_read_input_tokens", self._start_cache_read) or 0)
|
||||
cache_creation = int(
|
||||
usage.get("cache_creation_input_tokens", self._start_cache_creation) or 0
|
||||
)
|
||||
new_input = self._target_total - cache_read - cache_creation
|
||||
if int(usage.get("input_tokens") or 0) >= new_input:
|
||||
# Never deflate.
|
||||
return event
|
||||
usage["input_tokens"] = new_input
|
||||
lines[i] = _data_line(payload, line)
|
||||
return b"\n".join(lines)
|
||||
return event
|
||||
@@ -1857,6 +1857,24 @@ class AnthropicHandlerMixin:
|
||||
from headroom.proxy.helpers import COMPRESSION_TIMEOUT_SECONDS
|
||||
|
||||
context_limit = self.anthropic_provider.get_context_limit(model)
|
||||
# Context pressure must be computed against the window the
|
||||
# request is really subject to: raised to 1M when the
|
||||
# client sent a context-1m beta (a 1M session is NOT 5x
|
||||
# over budget), capped by a limit learned from an actual
|
||||
# prompt-too-long error. See headroom/proxy/context_guard.py.
|
||||
from headroom.proxy.context_guard import (
|
||||
context_guard_enabled,
|
||||
credential_scope_from_headers,
|
||||
effective_context_limit,
|
||||
)
|
||||
|
||||
if context_guard_enabled():
|
||||
context_limit = effective_context_limit(
|
||||
model,
|
||||
context_limit,
|
||||
headers.get("anthropic-beta"),
|
||||
scope=credential_scope_from_headers(headers),
|
||||
)
|
||||
result = None
|
||||
biases = (
|
||||
self.config.hooks.compute_biases(messages, _hook_ctx)
|
||||
@@ -4122,6 +4140,7 @@ class AnthropicHandlerMixin:
|
||||
outcome_provider=provider_name,
|
||||
session_key=session_key,
|
||||
server_memory_tool_names=server_memory_tool_names,
|
||||
client_beta=client_anthropic_beta or "",
|
||||
)
|
||||
else:
|
||||
# Whatever set it — the client's own ``stream: false`` or
|
||||
@@ -4251,6 +4270,30 @@ class AnthropicHandlerMixin:
|
||||
)
|
||||
err_type = "parse_error"
|
||||
|
||||
if response.status_code == 400:
|
||||
# Diagnostic only, and outside the parse above:
|
||||
# `err_msg` is whatever upstream put in that slot
|
||||
# (null, an object, a list have all been seen), and
|
||||
# the client is still waiting for the real error
|
||||
# body below.
|
||||
try:
|
||||
from headroom.proxy.context_guard import (
|
||||
credential_scope_from_headers,
|
||||
note_prompt_too_long,
|
||||
)
|
||||
|
||||
note_prompt_too_long(
|
||||
model,
|
||||
headers.get("anthropic-beta"),
|
||||
err_msg,
|
||||
scope=credential_scope_from_headers(headers),
|
||||
)
|
||||
except Exception:
|
||||
logger.debug(
|
||||
f"[{request_id}] context_guard: limit learning skipped",
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
logger.warning(
|
||||
f"[{request_id}] UPSTREAM_ERROR "
|
||||
f"status={response.status_code} "
|
||||
@@ -5022,7 +5065,14 @@ class AnthropicHandlerMixin:
|
||||
)
|
||||
if not buffered_stream_ccr:
|
||||
return Response(
|
||||
content=response.content,
|
||||
content=self._context_guard_nudge_body(
|
||||
response.content,
|
||||
status_code=response.status_code,
|
||||
model=model,
|
||||
headers=headers,
|
||||
request_id=request_id,
|
||||
client_beta=client_anthropic_beta or "",
|
||||
),
|
||||
status_code=response.status_code,
|
||||
headers=response_headers,
|
||||
)
|
||||
@@ -5144,6 +5194,15 @@ class AnthropicHandlerMixin:
|
||||
status_code=502,
|
||||
)
|
||||
|
||||
# This stream is resynthesized from the parsed message,
|
||||
# so the streaming guard never sees it: nudge here.
|
||||
self._context_guard_nudge_message(
|
||||
resp_json,
|
||||
model=model,
|
||||
headers=headers,
|
||||
request_id=request_id,
|
||||
client_beta=client_anthropic_beta or "",
|
||||
)
|
||||
try:
|
||||
sse_events = self._response_to_sse(resp_json, "anthropic")
|
||||
except ValueError as sse_err:
|
||||
@@ -5175,7 +5234,14 @@ class AnthropicHandlerMixin:
|
||||
)
|
||||
|
||||
return Response(
|
||||
content=response.content,
|
||||
content=self._context_guard_nudge_body(
|
||||
response.content,
|
||||
status_code=response.status_code,
|
||||
model=model,
|
||||
headers=headers,
|
||||
request_id=request_id,
|
||||
client_beta=client_anthropic_beta or "",
|
||||
),
|
||||
status_code=response.status_code,
|
||||
headers=response_headers,
|
||||
)
|
||||
@@ -5306,6 +5372,96 @@ class AnthropicHandlerMixin:
|
||||
# permanently. The emit function is idempotent.
|
||||
await _finalize_pre_upstream()
|
||||
|
||||
def _context_guard_nudge_body(
|
||||
self,
|
||||
content: bytes,
|
||||
*,
|
||||
status_code: int,
|
||||
model: str,
|
||||
headers: dict[str, str],
|
||||
request_id: str,
|
||||
client_beta: str | None = None,
|
||||
) -> bytes:
|
||||
"""Apply the context guard to a buffered (non-streaming) message body.
|
||||
|
||||
The buffered path raises the compression budget through
|
||||
``effective_context_limit`` exactly like the streaming path, so it has
|
||||
to move the client's gauge exactly like the streaming path too. Without
|
||||
this, a non-streaming client got the larger forwarded request and none
|
||||
of the warning, and walked into the prompt-too-long wall the guard
|
||||
exists to keep it away from.
|
||||
|
||||
Returns ``content`` unchanged unless the usage genuinely needed
|
||||
nudging, so byte-faithful forwarding still holds for every response
|
||||
that is not near the wall.
|
||||
"""
|
||||
if status_code != 200 or not content:
|
||||
return content
|
||||
try:
|
||||
payload = json.loads(content)
|
||||
except Exception:
|
||||
return content
|
||||
if not self._context_guard_nudge_message(
|
||||
payload,
|
||||
model=model,
|
||||
headers=headers,
|
||||
request_id=request_id,
|
||||
client_beta=client_beta,
|
||||
):
|
||||
return content
|
||||
return json.dumps(payload).encode()
|
||||
|
||||
def _context_guard_nudge_message(
|
||||
self,
|
||||
payload: Any,
|
||||
*,
|
||||
model: str,
|
||||
headers: dict[str, str],
|
||||
request_id: str,
|
||||
client_beta: str | None = None,
|
||||
) -> bool:
|
||||
"""Nudge a parsed message's usage in place; True when it changed.
|
||||
|
||||
Shared by the buffered body above and the buffered-CCR stream, which
|
||||
resynthesizes SSE from the parsed message and so never passes through
|
||||
the streaming guard. ``client_beta`` is the beta the client itself
|
||||
sent (its gauge follows it; session-sticky merging can add context-1m
|
||||
to the outbound header); None falls back to the outbound header.
|
||||
"""
|
||||
try:
|
||||
from headroom.proxy.context_guard import (
|
||||
believed_context_limit,
|
||||
context_guard_enabled,
|
||||
credential_scope_from_headers,
|
||||
effective_context_limit,
|
||||
nudge_response_usage,
|
||||
)
|
||||
|
||||
if not context_guard_enabled():
|
||||
return False
|
||||
if not isinstance(payload, dict) or payload.get("type") != "message":
|
||||
return False
|
||||
model_limit = self.anthropic_provider.get_context_limit(model)
|
||||
beta_header = headers.get("anthropic-beta")
|
||||
return bool(
|
||||
nudge_response_usage(
|
||||
payload,
|
||||
believed_limit=believed_context_limit(
|
||||
model_limit, beta_header if client_beta is None else client_beta
|
||||
),
|
||||
effective_limit=effective_context_limit(
|
||||
model,
|
||||
model_limit,
|
||||
beta_header,
|
||||
scope=credential_scope_from_headers(headers),
|
||||
),
|
||||
request_id=request_id,
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
logger.debug(f"[{request_id}] context_guard: buffered nudge skipped", exc_info=True)
|
||||
return False
|
||||
|
||||
def _anthropic_batch_capability_error(self) -> Response | None:
|
||||
"""Return the stable client error for a Copilot batch target."""
|
||||
if not is_copilot_upstream_url(self.ANTHROPIC_API_URL):
|
||||
@@ -5478,6 +5634,23 @@ class AnthropicHandlerMixin:
|
||||
original_tokens = 0 # Initialize before try to prevent UnboundLocalError
|
||||
try:
|
||||
context_limit = self.anthropic_provider.get_context_limit(model)
|
||||
# Same window the streaming path computes: a batch item for a
|
||||
# 1M-beta model was otherwise compressed against the registry
|
||||
# limit, so identical requests got two different budgets
|
||||
# depending on which endpoint they arrived through.
|
||||
from headroom.proxy.context_guard import (
|
||||
context_guard_enabled,
|
||||
credential_scope_from_headers,
|
||||
effective_context_limit,
|
||||
)
|
||||
|
||||
if context_guard_enabled():
|
||||
context_limit = effective_context_limit(
|
||||
model,
|
||||
context_limit,
|
||||
headers.get("anthropic-beta"),
|
||||
scope=credential_scope_from_headers(headers),
|
||||
)
|
||||
frozen_message_count = (
|
||||
self._strict_previous_turn_frozen_count(original_messages, 0)
|
||||
if is_cache_mode(self.config.mode)
|
||||
|
||||
@@ -1029,6 +1029,7 @@ class StreamingMixin:
|
||||
conversation_key: str | None = None,
|
||||
conversation_tokens_saved: int | None = None,
|
||||
server_memory_tool_names: frozenset[str] | None = None,
|
||||
client_beta: str | None = None,
|
||||
) -> Response | StreamingResponse:
|
||||
"""Stream response with metrics tracking and memory tool handling.
|
||||
|
||||
@@ -1075,6 +1076,7 @@ class StreamingMixin:
|
||||
conversation_key=conversation_key,
|
||||
conversation_tokens_saved=conversation_tokens_saved,
|
||||
server_memory_tool_names=server_memory_tool_names,
|
||||
client_beta=client_beta,
|
||||
)
|
||||
except (Exception, asyncio.CancelledError):
|
||||
self._cleanup_mid_turn_stream(session_key)
|
||||
@@ -1108,6 +1110,7 @@ class StreamingMixin:
|
||||
conversation_key: str | None = None,
|
||||
conversation_tokens_saved: int | None = None,
|
||||
server_memory_tool_names: frozenset[str] | None = None,
|
||||
client_beta: str | None = None,
|
||||
) -> Response | StreamingResponse:
|
||||
"""Actual streaming implementation, guarded by _stream_response's cleanup wrapper."""
|
||||
from fastapi.responses import Response, StreamingResponse
|
||||
@@ -1128,7 +1131,63 @@ class StreamingMixin:
|
||||
# of being swallowed. (#1608)
|
||||
if supports_mid_turn_coalescing(client):
|
||||
self._active_streams.add(session_key)
|
||||
|
||||
# Before the guard: its learned-limit lookup is keyed on the credential
|
||||
# actually sent, the same one the 400 learning below keys on.
|
||||
headers = await apply_copilot_api_auth(headers, url=url)
|
||||
|
||||
# Context-limit guard (see headroom/proxy/context_guard.py): rewrites
|
||||
# only the client-bound message_start bytes when the forwarded request
|
||||
# is near the model's real window, so clients whose auto-compaction
|
||||
# keys off reported usage compact gracefully instead of looping on
|
||||
# prompt-too-long 400s. Metrics below parse the original upstream
|
||||
# bytes and are unaffected.
|
||||
context_guard = None
|
||||
if provider == "anthropic":
|
||||
from headroom.proxy.context_guard import (
|
||||
StreamUsageGuard,
|
||||
believed_context_limit,
|
||||
context_guard_enabled,
|
||||
credential_scope_from_headers,
|
||||
effective_context_limit,
|
||||
)
|
||||
|
||||
try:
|
||||
if context_guard_enabled():
|
||||
_guard_model_limit = self.anthropic_provider.get_context_limit(model)
|
||||
_guard_beta = headers.get("anthropic-beta")
|
||||
_guard_scope = credential_scope_from_headers(headers)
|
||||
context_guard = StreamUsageGuard(
|
||||
# The client's gauge follows the beta the client
|
||||
# sent; session-sticky merging can add context-1m to
|
||||
# the outbound header after the client dropped it.
|
||||
# None: the caller did not say, use the outbound one.
|
||||
believed_limit=believed_context_limit(
|
||||
_guard_model_limit,
|
||||
_guard_beta if client_beta is None else client_beta,
|
||||
),
|
||||
effective_limit=effective_context_limit(
|
||||
model, _guard_model_limit, _guard_beta, scope=_guard_scope
|
||||
),
|
||||
request_id=request_id,
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("context_guard setup skipped", exc_info=True)
|
||||
|
||||
def _guard_client_bytes(data: bytes) -> bytes:
|
||||
"""Client-bound bytes through the context guard, when there is one."""
|
||||
return context_guard.feed(data) if context_guard is not None else data
|
||||
|
||||
def _drain_context_guard() -> bytes:
|
||||
"""Held-back guard bytes, once. Safe to call on any exit path."""
|
||||
if context_guard is None:
|
||||
return b""
|
||||
try:
|
||||
return context_guard.flush()
|
||||
except Exception:
|
||||
logger.debug("context_guard flush skipped", exc_info=True)
|
||||
return b""
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# Byte-faithful forwarding (PR-A3, fixes P0-2). Resolve outbound
|
||||
@@ -1403,6 +1462,24 @@ class StreamingMixin:
|
||||
finally:
|
||||
await upstream_response.aclose()
|
||||
|
||||
if provider == "anthropic" and upstream_response.status_code == 400:
|
||||
# Diagnostic only: whatever it makes of the body, the client
|
||||
# still gets the upstream error below.
|
||||
try:
|
||||
from headroom.proxy.context_guard import (
|
||||
credential_scope_from_headers,
|
||||
note_prompt_too_long,
|
||||
)
|
||||
|
||||
note_prompt_too_long(
|
||||
model,
|
||||
headers.get("anthropic-beta"),
|
||||
error_content,
|
||||
scope=credential_scope_from_headers(headers),
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("context_guard: limit learning skipped", exc_info=True)
|
||||
|
||||
if _codex_wire_debug:
|
||||
_error_text: str | None = None
|
||||
_error_body: Any = None
|
||||
@@ -1542,11 +1619,16 @@ class StreamingMixin:
|
||||
|
||||
# Always stream immediately — buffering breaks
|
||||
# real-time clients (LangGraph, LangChain, etc.)
|
||||
if memory_filter is None:
|
||||
yield chunk
|
||||
else:
|
||||
for frame in memory_filter.feed(chunk):
|
||||
yield frame
|
||||
# The context guard runs last, on exactly what the
|
||||
# client receives (after the memory filter), and may
|
||||
# hold back bytes only until the first complete event.
|
||||
client_frames = (
|
||||
[chunk] if memory_filter is None else memory_filter.feed(chunk)
|
||||
)
|
||||
for frame in client_frames:
|
||||
guarded_frame = _guard_client_bytes(frame)
|
||||
if guarded_frame:
|
||||
yield guarded_frame
|
||||
|
||||
if _codex_wire_debug:
|
||||
capture_codex_wire_debug(
|
||||
@@ -1625,7 +1707,12 @@ class StreamingMixin:
|
||||
)
|
||||
if memory_filter is not None:
|
||||
for frame in memory_filter.closing_frames():
|
||||
yield frame
|
||||
guarded_frame = _guard_client_bytes(frame)
|
||||
if guarded_frame:
|
||||
yield guarded_frame
|
||||
guarded_tail = _drain_context_guard()
|
||||
if guarded_tail:
|
||||
yield guarded_tail
|
||||
return
|
||||
|
||||
# Parse SSE to get response JSON
|
||||
@@ -1644,7 +1731,9 @@ class StreamingMixin:
|
||||
stream_state=stream_state,
|
||||
request_id=request_id,
|
||||
):
|
||||
yield frame
|
||||
guarded_frame = _guard_client_bytes(frame)
|
||||
if guarded_frame:
|
||||
yield guarded_frame
|
||||
memory_filter = None
|
||||
parsed_response = (
|
||||
stream_state.pop("memory_client_message", None) or parsed_response
|
||||
@@ -1687,7 +1776,20 @@ class StreamingMixin:
|
||||
stream_state=stream_state,
|
||||
request_id=request_id,
|
||||
):
|
||||
yield frame
|
||||
guarded_frame = _guard_client_bytes(frame)
|
||||
if guarded_frame:
|
||||
yield guarded_frame
|
||||
|
||||
# A stream that ended before its first complete event may
|
||||
# leave held-back bytes in the context guard; release them,
|
||||
# after the memory filter's held tail so the final
|
||||
# message_delta is still nudged. Every exit does this,
|
||||
# including the error paths below: bytes the guard is holding
|
||||
# are bytes the upstream already sent, and dropping them
|
||||
# silently truncates the client's stream.
|
||||
guarded_tail = _drain_context_guard()
|
||||
if guarded_tail:
|
||||
yield guarded_tail
|
||||
|
||||
# CCR Feedback: Record headroom_retrieve tool calls for TOIN learning.
|
||||
# In streaming mode, the client handles actual retrieval, but we
|
||||
@@ -1725,6 +1827,9 @@ class StreamingMixin:
|
||||
|
||||
except (httpx.ConnectError, httpx.ConnectTimeout, httpx.PoolTimeout) as e:
|
||||
logger.error(f"[{request_id}] Connection error to upstream API: {e}")
|
||||
guarded_tail = _drain_context_guard()
|
||||
if guarded_tail:
|
||||
yield guarded_tail
|
||||
error_event = public_errors.anthropic_error_body(
|
||||
public_errors.classify_or_internal(e),
|
||||
request_id=request_id,
|
||||
@@ -1733,6 +1838,9 @@ class StreamingMixin:
|
||||
yield f"event: error\ndata: {json.dumps(error_event)}\n\n".encode()
|
||||
except httpx.HTTPStatusError as e:
|
||||
logger.error(f"[{request_id}] HTTP error from upstream API: {e}")
|
||||
guarded_tail = _drain_context_guard()
|
||||
if guarded_tail:
|
||||
yield guarded_tail
|
||||
# Forward the upstream error response
|
||||
yield e.response.content
|
||||
except MemoryToolStreamOverflowError as e:
|
||||
@@ -1745,6 +1853,9 @@ class StreamingMixin:
|
||||
yield f"event: error\ndata: {json.dumps(error_event)}\n\n".encode()
|
||||
except Exception as e:
|
||||
logger.error(f"[{request_id}] Unexpected streaming error: {e}")
|
||||
guarded_tail = _drain_context_guard()
|
||||
if guarded_tail:
|
||||
yield guarded_tail
|
||||
error_event = public_errors.anthropic_error_body(
|
||||
public_errors.classify_or_internal(e),
|
||||
request_id=request_id,
|
||||
|
||||
@@ -179,3 +179,124 @@ def test_buffered_ccr_records_usage_with_compression_on(monkeypatch, ccr_marker:
|
||||
f"buffered-CCR turn dropped usage under compression: "
|
||||
f"cr={o.cache_read_tokens} cw={o.cache_write_tokens} out={o.output_tokens}"
|
||||
)
|
||||
|
||||
|
||||
@respx.mock
|
||||
def test_buffered_ccr_stream_gets_the_context_guard_nudge(monkeypatch, ccr_marker: str) -> None:
|
||||
"""The buffered-CCR stream is resynthesized from the parsed message, so the
|
||||
streaming guard never sees it. Near the wall its gauge must move anyway, or
|
||||
the client gets the raised compression budget without the warning."""
|
||||
import json
|
||||
|
||||
from headroom.proxy.context_guard import REPORT_FRACTION, reset_learned_limits
|
||||
|
||||
reset_learned_limits()
|
||||
app, _ = _app_and_outcomes(monkeypatch)
|
||||
near_wall = {**_RESPONSE, "usage": {**_USAGE, "cache_read_input_tokens": 900_000}}
|
||||
respx.post("https://api.anthropic.com/v1/messages").mock(
|
||||
return_value=httpx.Response(200, json=near_wall)
|
||||
)
|
||||
|
||||
r = _post(app, stream=True, tools=[_RETRIEVE_TOOL], marker=ccr_marker)
|
||||
|
||||
assert r.status_code == 200
|
||||
start = next(
|
||||
json.loads(line[len("data:") :])
|
||||
for line in r.text.splitlines()
|
||||
if line.startswith("data:") and '"message_start"' in line
|
||||
)
|
||||
usage = start["message"]["usage"]
|
||||
reported = (
|
||||
usage["input_tokens"]
|
||||
+ usage["cache_read_input_tokens"]
|
||||
+ usage["cache_creation_input_tokens"]
|
||||
)
|
||||
# claude-opus-5 is a 1M model; 909k forwarded is past the trigger.
|
||||
assert reported == int(1_000_000 * REPORT_FRACTION)
|
||||
|
||||
|
||||
def _reported_input(usage: dict) -> int:
|
||||
return (
|
||||
usage["input_tokens"]
|
||||
+ usage["cache_read_input_tokens"]
|
||||
+ usage["cache_creation_input_tokens"]
|
||||
)
|
||||
|
||||
|
||||
@respx.mock
|
||||
def test_non_stream_json_gets_the_context_guard_nudge(monkeypatch) -> None:
|
||||
"""The plain non-stream JSON body is nudged near the wall, like the stream."""
|
||||
from headroom.proxy.context_guard import REPORT_FRACTION, reset_learned_limits
|
||||
|
||||
reset_learned_limits()
|
||||
app, _ = _app_and_outcomes(monkeypatch)
|
||||
near_wall = {**_RESPONSE, "usage": {**_USAGE, "cache_read_input_tokens": 900_000}}
|
||||
respx.post("https://api.anthropic.com/v1/messages").mock(
|
||||
return_value=httpx.Response(200, json=near_wall)
|
||||
)
|
||||
|
||||
r = _post(app, stream=False, tools=None)
|
||||
|
||||
assert r.status_code == 200
|
||||
usage = r.json()["usage"]
|
||||
# claude-opus-5 is a 1M model; 909k forwarded is past the trigger.
|
||||
assert _reported_input(usage) == int(1_000_000 * REPORT_FRACTION)
|
||||
assert usage["output_tokens"] == _USAGE["output_tokens"]
|
||||
|
||||
|
||||
@respx.mock
|
||||
def test_non_stream_json_is_untouched_with_the_guard_off(monkeypatch) -> None:
|
||||
from headroom.proxy.context_guard import reset_learned_limits
|
||||
|
||||
reset_learned_limits()
|
||||
monkeypatch.setenv("HEADROOM_CONTEXT_GUARD", "0")
|
||||
app, _ = _app_and_outcomes(monkeypatch)
|
||||
near_wall = {**_RESPONSE, "usage": {**_USAGE, "cache_read_input_tokens": 900_000}}
|
||||
respx.post("https://api.anthropic.com/v1/messages").mock(
|
||||
return_value=httpx.Response(200, json=near_wall)
|
||||
)
|
||||
|
||||
r = _post(app, stream=False, tools=None)
|
||||
|
||||
assert r.status_code == 200
|
||||
assert r.json()["usage"] == near_wall["usage"]
|
||||
|
||||
|
||||
def test_buffered_nudge_follows_the_client_beta_not_the_sticky_one(monkeypatch) -> None:
|
||||
"""Session-sticky merging can keep context-1m outbound after the client
|
||||
dropped it. The learned cap keys on the forwarded beta; the client's gauge
|
||||
is its own 200k window, so the nudge targets 95% of 200k, not of 1M."""
|
||||
from headroom.proxy.context_guard import (
|
||||
REPORT_FRACTION,
|
||||
credential_scope,
|
||||
note_prompt_too_long,
|
||||
reset_learned_limits,
|
||||
)
|
||||
|
||||
reset_learned_limits()
|
||||
beta_1m = "context-1m-2025-08-07"
|
||||
note_prompt_too_long(
|
||||
"claude-sonnet-4-5",
|
||||
beta_1m,
|
||||
"prompt is too long: 213021 tokens > 200000 maximum",
|
||||
scope=credential_scope("sk-sticky"),
|
||||
)
|
||||
app, _ = _app_and_outcomes(monkeypatch)
|
||||
proxy = app.state.proxy
|
||||
monkeypatch.setattr(proxy.anthropic_provider, "get_context_limit", lambda _model: 200_000)
|
||||
payload = {
|
||||
**_RESPONSE,
|
||||
"model": "claude-sonnet-4-5",
|
||||
"usage": {**_USAGE, "cache_read_input_tokens": 185_000 - 300 - 8197},
|
||||
}
|
||||
|
||||
changed = proxy._context_guard_nudge_message(
|
||||
payload,
|
||||
model="claude-sonnet-4-5",
|
||||
headers={"x-api-key": "sk-sticky", "anthropic-beta": beta_1m},
|
||||
request_id="test-sticky-buffered",
|
||||
client_beta="",
|
||||
)
|
||||
|
||||
assert changed
|
||||
assert _reported_input(payload["usage"]) == int(200_000 * REPORT_FRACTION)
|
||||
|
||||
@@ -0,0 +1,694 @@
|
||||
"""Tests for the context-limit guard (headroom/proxy/context_guard.py)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.proxy.context_guard import (
|
||||
REPORT_FRACTION,
|
||||
StreamUsageGuard,
|
||||
believed_context_limit,
|
||||
context_guard_enabled,
|
||||
credential_scope,
|
||||
effective_context_limit,
|
||||
has_context_1m_beta,
|
||||
note_prompt_too_long,
|
||||
nudge_response_usage,
|
||||
reset_learned_limits,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_learned_limits():
|
||||
reset_learned_limits()
|
||||
yield
|
||||
reset_learned_limits()
|
||||
|
||||
|
||||
def _message_start_event(
|
||||
input_tokens: int,
|
||||
cache_read: int = 0,
|
||||
cache_creation: int = 0,
|
||||
) -> bytes:
|
||||
payload = {
|
||||
"type": "message_start",
|
||||
"message": {
|
||||
"id": "msg_test",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": "claude-sonnet-4-5",
|
||||
"content": [],
|
||||
"usage": {
|
||||
"input_tokens": input_tokens,
|
||||
"cache_read_input_tokens": cache_read,
|
||||
"cache_creation_input_tokens": cache_creation,
|
||||
"output_tokens": 1,
|
||||
},
|
||||
},
|
||||
}
|
||||
return b"event: message_start\ndata: " + json.dumps(payload).encode() + b"\n\n"
|
||||
|
||||
|
||||
def _parse_usage(event_bytes: bytes) -> dict:
|
||||
for line in event_bytes.split(b"\n"):
|
||||
if line.startswith(b"data:"):
|
||||
return json.loads(line[5:].strip())["message"]["usage"]
|
||||
raise AssertionError("no data line in event")
|
||||
|
||||
|
||||
def _message_delta_event(
|
||||
input_tokens: int,
|
||||
cache_read: int = 0,
|
||||
cache_creation: int = 0,
|
||||
) -> bytes:
|
||||
payload = {
|
||||
"type": "message_delta",
|
||||
"delta": {"stop_reason": "end_turn", "stop_sequence": None},
|
||||
"usage": {
|
||||
"input_tokens": input_tokens,
|
||||
"cache_read_input_tokens": cache_read,
|
||||
"cache_creation_input_tokens": cache_creation,
|
||||
"output_tokens": 42,
|
||||
},
|
||||
}
|
||||
return b"event: message_delta\ndata: " + json.dumps(payload).encode() + b"\n\n"
|
||||
|
||||
|
||||
class TestBetaAndLimits:
|
||||
def test_has_context_1m_beta_detects_token(self):
|
||||
assert has_context_1m_beta("context-1m-2025-08-07")
|
||||
assert has_context_1m_beta("oauth-2025-04-20, context-1m-2025-08-07")
|
||||
assert not has_context_1m_beta("oauth-2025-04-20")
|
||||
assert not has_context_1m_beta(None)
|
||||
assert not has_context_1m_beta("")
|
||||
|
||||
def test_believed_limit_raised_by_1m_beta(self):
|
||||
assert believed_context_limit(200_000, "context-1m-2025-08-07") == 1_000_000
|
||||
assert believed_context_limit(200_000, "oauth-2025-04-20") == 200_000
|
||||
|
||||
def test_believed_limit_never_lowered(self):
|
||||
assert believed_context_limit(2_000_000, "context-1m-2025-08-07") == 2_000_000
|
||||
|
||||
def test_effective_limit_optimistic_until_learned(self):
|
||||
beta = "context-1m-2025-08-07"
|
||||
assert effective_context_limit("claude-sonnet-4-5", 200_000, beta) == 1_000_000
|
||||
|
||||
def test_learned_limit_caps_effective(self):
|
||||
beta = "context-1m-2025-08-07"
|
||||
learned = note_prompt_too_long(
|
||||
"claude-sonnet-4-5",
|
||||
beta,
|
||||
'{"error": {"message": "prompt is too long: 213021 tokens > 200000 maximum"}}',
|
||||
)
|
||||
assert learned == 200_000
|
||||
assert effective_context_limit("claude-sonnet-4-5", 200_000, beta) == 200_000
|
||||
|
||||
def test_learned_limit_keyed_by_beta_presence(self):
|
||||
# A limit learned WITHOUT the 1m beta must not clamp sessions that
|
||||
# send it (their account may have real 1M access).
|
||||
note_prompt_too_long(
|
||||
"claude-sonnet-4-5",
|
||||
None,
|
||||
"prompt is too long: 201000 tokens > 200000 maximum",
|
||||
)
|
||||
assert (
|
||||
effective_context_limit("claude-sonnet-4-5", 200_000, "context-1m-2025-08-07")
|
||||
== 1_000_000
|
||||
)
|
||||
|
||||
def test_note_prompt_too_long_ignores_other_errors(self):
|
||||
assert note_prompt_too_long("m", None, "rate limit exceeded") is None
|
||||
assert note_prompt_too_long("m", None, b"") is None
|
||||
|
||||
def test_note_prompt_too_long_accepts_bytes(self):
|
||||
assert (
|
||||
note_prompt_too_long("m", None, b"prompt is too long: 250000 tokens > 200000 maximum")
|
||||
== 200_000
|
||||
)
|
||||
|
||||
def test_enabled_by_default_and_kill_switch(self, monkeypatch):
|
||||
monkeypatch.delenv("HEADROOM_CONTEXT_GUARD", raising=False)
|
||||
assert context_guard_enabled()
|
||||
monkeypatch.setenv("HEADROOM_CONTEXT_GUARD", "0")
|
||||
assert not context_guard_enabled()
|
||||
monkeypatch.setenv("HEADROOM_CONTEXT_GUARD", "false")
|
||||
assert not context_guard_enabled()
|
||||
|
||||
|
||||
class TestLearnedLimitHygiene:
|
||||
"""Regressions for what a learned limit is allowed to do."""
|
||||
|
||||
def test_non_string_error_bodies_do_not_raise(self):
|
||||
# Upstreams and gateways put null, objects and lists in the message
|
||||
# slot. This runs while the client is still waiting for the real error
|
||||
# body, so it must return empty-handed rather than raise over it.
|
||||
for body in (None, {"message": {"nested": True}}, ["prompt is too long"], 42):
|
||||
assert note_prompt_too_long("claude-sonnet-4-5", None, body) is None
|
||||
assert effective_context_limit("claude-sonnet-4-5", 200_000, None) == 200_000
|
||||
|
||||
def test_out_of_band_maximum_is_not_learned(self):
|
||||
# Same wire shape, different subject: gateways report per-request quota
|
||||
# and per-key throttles as "prompt is too long: N > M". A limit that is
|
||||
# not a context window must not clamp the budget for the process.
|
||||
assert (
|
||||
note_prompt_too_long(
|
||||
"claude-sonnet-4-5", None, "prompt is too long: 900 tokens > 1 maximum"
|
||||
)
|
||||
is None
|
||||
)
|
||||
assert (
|
||||
note_prompt_too_long(
|
||||
"claude-sonnet-4-5",
|
||||
None,
|
||||
"prompt is too long: 99000000 tokens > 50000000 maximum",
|
||||
)
|
||||
is None
|
||||
)
|
||||
assert effective_context_limit("claude-sonnet-4-5", 200_000, None) == 200_000
|
||||
|
||||
def test_learned_limit_is_scoped_to_one_credential(self):
|
||||
# One tenant's capped key on a shared proxy must not clamp another's.
|
||||
note_prompt_too_long(
|
||||
"claude-sonnet-4-5",
|
||||
None,
|
||||
"prompt is too long: 213021 tokens > 200000 maximum",
|
||||
scope=credential_scope("sk-tenant-a"),
|
||||
)
|
||||
assert (
|
||||
effective_context_limit(
|
||||
"claude-sonnet-4-5", 400_000, None, scope=credential_scope("sk-tenant-a")
|
||||
)
|
||||
== 200_000
|
||||
)
|
||||
assert (
|
||||
effective_context_limit(
|
||||
"claude-sonnet-4-5", 400_000, None, scope=credential_scope("sk-tenant-b")
|
||||
)
|
||||
== 400_000
|
||||
)
|
||||
|
||||
def test_credential_scope_never_echoes_the_credential(self):
|
||||
scope = credential_scope("sk-ant-secret-value")
|
||||
assert "secret" not in scope
|
||||
assert scope and scope != credential_scope("sk-ant-other-value")
|
||||
assert credential_scope(None) == credential_scope("") == ""
|
||||
|
||||
def test_learned_limit_expires(self, monkeypatch):
|
||||
# Entitlements get provisioned and gateways get repointed; a lesson
|
||||
# from an hour ago must not hold the budget down forever.
|
||||
from headroom.proxy import context_guard
|
||||
|
||||
clock = {"now": 1_000.0}
|
||||
monkeypatch.setattr(context_guard.time, "monotonic", lambda: clock["now"])
|
||||
note_prompt_too_long(
|
||||
"claude-sonnet-4-5", None, "prompt is too long: 213021 tokens > 200000 maximum"
|
||||
)
|
||||
assert effective_context_limit("claude-sonnet-4-5", 400_000, None) == 200_000
|
||||
clock["now"] += context_guard._LEARNED_LIMIT_TTL_SECONDS + 1
|
||||
assert effective_context_limit("claude-sonnet-4-5", 400_000, None) == 400_000
|
||||
|
||||
|
||||
class TestNonStreamingNudge:
|
||||
"""The buffered path raises the same budget, so it owes the same warning."""
|
||||
|
||||
def test_near_limit_body_is_nudged(self):
|
||||
payload = {
|
||||
"type": "message",
|
||||
"usage": {
|
||||
"input_tokens": 185_000,
|
||||
"cache_read_input_tokens": 0,
|
||||
"cache_creation_input_tokens": 0,
|
||||
"output_tokens": 7,
|
||||
},
|
||||
}
|
||||
assert nudge_response_usage(payload, believed_limit=200_000, effective_limit=200_000)
|
||||
assert payload["usage"]["input_tokens"] == int(200_000 * REPORT_FRACTION)
|
||||
|
||||
def test_far_from_limit_body_is_untouched(self):
|
||||
payload = {"type": "message", "usage": {"input_tokens": 5_000, "output_tokens": 7}}
|
||||
assert not nudge_response_usage(payload, believed_limit=200_000, effective_limit=200_000)
|
||||
assert payload["usage"]["input_tokens"] == 5_000
|
||||
|
||||
def test_bodies_without_usage_are_untouched(self):
|
||||
for payload in ({"type": "message"}, {"type": "error"}, "not a body", None):
|
||||
assert not nudge_response_usage(
|
||||
payload, believed_limit=200_000, effective_limit=200_000
|
||||
)
|
||||
|
||||
|
||||
class TestStreamUsageGuard:
|
||||
def test_below_trigger_passes_through_byte_identical(self):
|
||||
guard = StreamUsageGuard(believed_limit=200_000, effective_limit=200_000)
|
||||
event = _message_start_event(50_000, cache_read=100_000)
|
||||
assert guard.feed(event) == event
|
||||
# Inert afterwards: later chunks untouched even if they look like events.
|
||||
tail = b"event: content_block_delta\ndata: {}\n\n"
|
||||
assert guard.feed(tail) == tail
|
||||
|
||||
def test_above_trigger_inflates_to_report_fraction_of_believed(self):
|
||||
guard = StreamUsageGuard(believed_limit=1_000_000, effective_limit=200_000)
|
||||
# 185k total forwarded = 92.5% of the real 200k window.
|
||||
event = _message_start_event(5_000, cache_read=170_000, cache_creation=10_000)
|
||||
out = guard.feed(event)
|
||||
usage = _parse_usage(out)
|
||||
target_total = int(1_000_000 * REPORT_FRACTION)
|
||||
assert (
|
||||
usage["input_tokens"]
|
||||
+ usage["cache_read_input_tokens"]
|
||||
+ usage["cache_creation_input_tokens"]
|
||||
== target_total
|
||||
)
|
||||
# Cache components are never touched — only input_tokens absorbs the nudge.
|
||||
assert usage["cache_read_input_tokens"] == 170_000
|
||||
assert usage["cache_creation_input_tokens"] == 10_000
|
||||
assert usage["output_tokens"] == 1
|
||||
|
||||
def test_same_limits_nudges_gauge_over_compact_threshold(self):
|
||||
guard = StreamUsageGuard(believed_limit=200_000, effective_limit=200_000)
|
||||
event = _message_start_event(185_000)
|
||||
usage = _parse_usage(guard.feed(event))
|
||||
assert usage["input_tokens"] == int(200_000 * REPORT_FRACTION)
|
||||
|
||||
def test_never_deflates(self):
|
||||
guard = StreamUsageGuard(believed_limit=200_000, effective_limit=200_000)
|
||||
event = _message_start_event(199_000) # above the 95% report target
|
||||
assert guard.feed(event) == event
|
||||
|
||||
def test_split_chunks_across_event_boundary(self):
|
||||
guard = StreamUsageGuard(believed_limit=1_000_000, effective_limit=200_000)
|
||||
event = _message_start_event(190_000)
|
||||
first, second = event[:40], event[40:]
|
||||
assert guard.feed(first) == b"" # held back: no complete event yet
|
||||
out = guard.feed(second)
|
||||
assert _parse_usage(out)["input_tokens"] == int(1_000_000 * REPORT_FRACTION)
|
||||
|
||||
def test_crlf_framed_events_are_released_and_rewritten(self):
|
||||
guard = StreamUsageGuard(believed_limit=1_000_000, effective_limit=200_000)
|
||||
event = _message_start_event(190_000).replace(b"\n", b"\r\n")
|
||||
out = guard.feed(event)
|
||||
# Released at its own terminator, not held until the buffer cap.
|
||||
assert out.endswith(b"\r\n\r\n")
|
||||
assert b"\n" not in out.replace(b"\r\n", b"")
|
||||
assert _parse_usage(out)["input_tokens"] == int(1_000_000 * REPORT_FRACTION)
|
||||
|
||||
def test_a_chunk_over_the_buffer_cap_still_gets_its_message_start_read(self):
|
||||
guard = StreamUsageGuard(believed_limit=1_000_000, effective_limit=200_000)
|
||||
filler = b"event: content_block_delta\ndata: " + b"x" * (300 * 1024) + b"\n\n"
|
||||
out = guard.feed(_message_start_event(190_000) + filler)
|
||||
assert _parse_usage(out)["input_tokens"] == int(1_000_000 * REPORT_FRACTION)
|
||||
assert out.endswith(filler)
|
||||
|
||||
def test_ping_events_pass_through_before_message_start(self):
|
||||
guard = StreamUsageGuard(believed_limit=1_000_000, effective_limit=200_000)
|
||||
ping = b'event: ping\ndata: {"type": "ping"}\n\n'
|
||||
assert guard.feed(ping) == ping
|
||||
out = guard.feed(_message_start_event(190_000))
|
||||
assert _parse_usage(out)["input_tokens"] == int(1_000_000 * REPORT_FRACTION)
|
||||
|
||||
def test_non_message_start_first_event_disarms(self):
|
||||
guard = StreamUsageGuard(believed_limit=1_000_000, effective_limit=200_000)
|
||||
error_event = b'event: error\ndata: {"type": "error"}\n\n'
|
||||
assert guard.feed(error_event) == error_event
|
||||
# Even a later message_start is untouched (protocol says it is first).
|
||||
event = _message_start_event(190_000)
|
||||
assert guard.feed(event) == event
|
||||
|
||||
def test_malformed_json_passes_through(self):
|
||||
guard = StreamUsageGuard(believed_limit=1_000_000, effective_limit=200_000)
|
||||
event = b"event: message_start\ndata: {not json\n\n"
|
||||
assert guard.feed(event) == event
|
||||
|
||||
def test_buffer_cap_flushes_verbatim(self):
|
||||
guard = StreamUsageGuard(believed_limit=1_000_000, effective_limit=200_000)
|
||||
blob = b"x" * (300 * 1024) # no event boundary anywhere
|
||||
out = guard.feed(blob)
|
||||
assert out == blob
|
||||
assert guard.feed(b"more") == b"more" # inert afterwards
|
||||
|
||||
def test_flush_returns_held_bytes(self):
|
||||
guard = StreamUsageGuard(believed_limit=1_000_000, effective_limit=200_000)
|
||||
partial = b"event: message_start\ndata: {"
|
||||
assert guard.feed(partial) == b""
|
||||
assert guard.flush() == partial
|
||||
|
||||
def test_unusable_limits_make_guard_inert(self):
|
||||
guard = StreamUsageGuard(believed_limit=0, effective_limit=200_000)
|
||||
event = _message_start_event(190_000)
|
||||
assert guard.feed(event) == event
|
||||
|
||||
def test_rewritten_event_is_valid_sse(self):
|
||||
guard = StreamUsageGuard(believed_limit=200_000, effective_limit=200_000)
|
||||
out = guard.feed(_message_start_event(185_000))
|
||||
assert out.startswith(b"event: message_start\n")
|
||||
assert out.endswith(b"\n\n")
|
||||
payload = _parse_usage(out) # parses => data line is intact JSON
|
||||
assert payload["input_tokens"] > 185_000
|
||||
|
||||
def test_armed_guard_rewrites_final_message_delta(self):
|
||||
# Clients merge the final cumulative-usage message_delta over
|
||||
# message_start (verified live with Claude Code 2026-08-12), so an
|
||||
# armed guard must rewrite both or the nudge loses the merge.
|
||||
guard = StreamUsageGuard(believed_limit=200_000, effective_limit=200_000)
|
||||
guard.feed(_message_start_event(180_000, cache_creation=5_000))
|
||||
content = b'event: content_block_delta\ndata: {"type":"content_block_delta"}\n\n'
|
||||
assert guard.feed(content) == content # armed but passthrough
|
||||
delta = _message_delta_event(180_000, cache_creation=5_000)
|
||||
out = guard.feed(delta)
|
||||
usage = json.loads(out.split(b"data: ")[1])["usage"]
|
||||
assert usage["input_tokens"] + usage["cache_creation_input_tokens"] == int(
|
||||
200_000 * REPORT_FRACTION
|
||||
)
|
||||
assert usage["output_tokens"] == 42
|
||||
# Inert afterwards.
|
||||
stop = b'event: message_stop\ndata: {"type":"message_stop"}\n\n'
|
||||
assert guard.feed(stop) == stop
|
||||
|
||||
def test_output_only_message_delta_untouched(self):
|
||||
guard = StreamUsageGuard(believed_limit=200_000, effective_limit=200_000)
|
||||
guard.feed(_message_start_event(185_000))
|
||||
delta = (
|
||||
b'event: message_delta\ndata: {"type":"message_delta","usage":{"output_tokens":9}}\n\n'
|
||||
)
|
||||
assert guard.feed(delta) == delta
|
||||
|
||||
def test_disarmed_guard_leaves_message_delta_alone(self):
|
||||
guard = StreamUsageGuard(believed_limit=200_000, effective_limit=200_000)
|
||||
guard.feed(_message_start_event(50_000)) # below trigger -> inert
|
||||
delta = _message_delta_event(50_000)
|
||||
assert guard.feed(delta) == delta
|
||||
|
||||
def test_message_delta_never_deflates(self):
|
||||
guard = StreamUsageGuard(believed_limit=200_000, effective_limit=200_000)
|
||||
guard.feed(_message_start_event(185_000))
|
||||
delta = _message_delta_event(199_000) # already above the 190k target
|
||||
assert guard.feed(delta) == delta
|
||||
|
||||
|
||||
class TestSseFraming:
|
||||
"""The scan reads the SSE envelope, never the payload text."""
|
||||
|
||||
def _armed_guard(self) -> StreamUsageGuard:
|
||||
guard = StreamUsageGuard(believed_limit=200_000, effective_limit=200_000)
|
||||
guard.feed(_message_start_event(190_000))
|
||||
return guard
|
||||
|
||||
def test_assistant_text_naming_an_event_does_not_disarm(self):
|
||||
# The model writing about message_delta -- explaining SSE, quoting a
|
||||
# log line -- used to end the scan, so the final cumulative usage went
|
||||
# out un-nudged and the client's gauge snapped back.
|
||||
guard = self._armed_guard()
|
||||
chatter = (
|
||||
b"event: content_block_delta\ndata: "
|
||||
+ json.dumps(
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 0,
|
||||
"delta": {
|
||||
"type": "text_delta",
|
||||
"text": "the final message_delta carries cumulative usage",
|
||||
},
|
||||
}
|
||||
).encode()
|
||||
+ b"\n\n"
|
||||
)
|
||||
assert guard.feed(chatter) == chatter
|
||||
nudged = guard.feed(_message_delta_event(190_000))
|
||||
assert json.loads(nudged.split(b"data: ", 1)[1])["usage"]["input_tokens"] == int(
|
||||
200_000 * REPORT_FRACTION
|
||||
)
|
||||
|
||||
def test_keepalive_frames_before_message_start_do_not_disarm(self):
|
||||
# An intermediary shim (nginx, a corporate egress proxy) injects SSE
|
||||
# comments and retry fields. They are not events, so they must not be
|
||||
# read as "a protocol we don't recognize" and switch the guard off.
|
||||
guard = StreamUsageGuard(believed_limit=200_000, effective_limit=200_000)
|
||||
for keepalive in (b": keepalive\n\n", b"retry: 3000\n\n", b":\n\n"):
|
||||
assert guard.feed(keepalive) == keepalive
|
||||
nudged = guard.feed(_message_start_event(190_000))
|
||||
assert _parse_usage(nudged)["input_tokens"] == int(200_000 * REPORT_FRACTION)
|
||||
|
||||
|
||||
class TestStreamResponseIntegration:
|
||||
"""The guard wired through _stream_response with a mocked upstream."""
|
||||
|
||||
def _create_mock_proxy(self):
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import httpx
|
||||
|
||||
from headroom.proxy.server import HeadroomProxy
|
||||
|
||||
proxy = object.__new__(HeadroomProxy)
|
||||
proxy.http_client = MagicMock(spec=httpx.AsyncClient)
|
||||
proxy.metrics = MagicMock()
|
||||
proxy.metrics.record_request = AsyncMock(return_value=None)
|
||||
proxy.metrics.record_failed = AsyncMock(return_value=None)
|
||||
proxy.cost_tracker = MagicMock()
|
||||
proxy.cost_tracker.estimate_cost.return_value = 0.001
|
||||
proxy.cost_tracker.record_request.return_value = None
|
||||
proxy.stats = {
|
||||
"requests_total": 0,
|
||||
"requests_optimized": 0,
|
||||
"tokens": {"original": 0, "optimized": 0, "saved": 0},
|
||||
"cost": {"total_usd": 0, "savings_usd": 0},
|
||||
"errors": 0,
|
||||
"active_requests": 0,
|
||||
"requests_per_model": {},
|
||||
}
|
||||
proxy.memory_manager = None
|
||||
proxy._config = MagicMock()
|
||||
proxy._config.memory_enabled = False
|
||||
proxy._config.ccr_inject_tool = False
|
||||
proxy._config.retry_max_attempts = 3
|
||||
proxy._config.retry_base_delay_ms = 0
|
||||
proxy._config.retry_max_delay_ms = 0
|
||||
proxy.config = proxy._config
|
||||
proxy._parse_sse_usage_from_buffer = MagicMock(return_value=None)
|
||||
proxy.memory_handler = None
|
||||
proxy.anthropic_provider = MagicMock()
|
||||
proxy.anthropic_provider.get_context_limit.return_value = 200_000
|
||||
return proxy
|
||||
|
||||
def _mock_upstream(self, sse_bytes: bytes, status_code: int = 200):
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import httpx
|
||||
|
||||
mock_response = AsyncMock()
|
||||
mock_response.headers = httpx.Headers({"content-type": "text/event-stream"})
|
||||
mock_response.status_code = status_code
|
||||
|
||||
async def aiter_bytes():
|
||||
yield sse_bytes
|
||||
|
||||
mock_response.aiter_bytes = aiter_bytes
|
||||
mock_response.aclose = AsyncMock()
|
||||
mock_response.aread = AsyncMock(return_value=sse_bytes)
|
||||
return mock_response
|
||||
|
||||
async def _run(self, proxy, mock_response, headers, **kwargs):
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
proxy.http_client.build_request = MagicMock(return_value=MagicMock())
|
||||
proxy.http_client.send = AsyncMock(return_value=mock_response)
|
||||
return await proxy._stream_response(
|
||||
url="https://api.anthropic.com/v1/messages",
|
||||
headers=headers,
|
||||
body={
|
||||
"model": "claude-sonnet-4-5",
|
||||
"max_tokens": 100,
|
||||
"stream": True,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
provider="anthropic",
|
||||
model="claude-sonnet-4-5",
|
||||
request_id="test-guard",
|
||||
original_tokens=10,
|
||||
optimized_tokens=10,
|
||||
tokens_saved=0,
|
||||
transforms_applied=[],
|
||||
tags={},
|
||||
optimization_latency=0.0,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_near_limit_message_start_is_nudged_on_the_wire(self):
|
||||
proxy = self._create_mock_proxy()
|
||||
# 185k = above the 90% trigger, below the 95% report target — the
|
||||
# assertions below can only pass if the rewrites actually happened.
|
||||
upstream = self._mock_upstream(
|
||||
_message_start_event(185_000)
|
||||
+ b'event: content_block_delta\ndata: {"type":"content_block_delta"}\n\n'
|
||||
+ _message_delta_event(185_000)
|
||||
+ b'event: message_stop\ndata: {"type":"message_stop"}\n\n'
|
||||
)
|
||||
result = await self._run(proxy, upstream, {"x-api-key": "sk-test"})
|
||||
client_bytes = b"".join([chunk async for chunk in result.body_iterator])
|
||||
events = client_bytes.split(b"\n\n")
|
||||
usage = _parse_usage(events[0] + b"\n\n")
|
||||
assert usage["input_tokens"] == int(200_000 * REPORT_FRACTION)
|
||||
# The final cumulative-usage message_delta is nudged too — clients
|
||||
# merge it over message_start, so it must agree.
|
||||
delta_usage = json.loads(events[2].split(b"data: ")[1])["usage"]
|
||||
assert delta_usage["input_tokens"] == int(200_000 * REPORT_FRACTION)
|
||||
assert delta_usage["output_tokens"] == 42
|
||||
# Everything else is untouched.
|
||||
assert (
|
||||
b'event: content_block_delta\ndata: {"type":"content_block_delta"}\n\n' in client_bytes
|
||||
)
|
||||
assert b'event: message_stop\ndata: {"type":"message_stop"}\n\n' in client_bytes
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_far_from_limit_stream_is_byte_identical(self):
|
||||
proxy = self._create_mock_proxy()
|
||||
sse = (
|
||||
_message_start_event(50_000) + b'event: message_stop\ndata: {"type":"message_stop"}\n\n'
|
||||
)
|
||||
result = await self._run(proxy, self._mock_upstream(sse), {"x-api-key": "sk-test"})
|
||||
client_bytes = b"".join([chunk async for chunk in result.body_iterator])
|
||||
assert client_bytes == sse
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_kill_switch_disables_nudge(self, monkeypatch):
|
||||
monkeypatch.setenv("HEADROOM_CONTEXT_GUARD", "0")
|
||||
proxy = self._create_mock_proxy()
|
||||
sse = _message_start_event(190_000)
|
||||
result = await self._run(proxy, self._mock_upstream(sse), {"x-api-key": "sk-test"})
|
||||
client_bytes = b"".join([chunk async for chunk in result.body_iterator])
|
||||
assert client_bytes == sse
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_streaming_400_learns_real_limit(self):
|
||||
proxy = self._create_mock_proxy()
|
||||
error_body = json.dumps(
|
||||
{
|
||||
"type": "error",
|
||||
"error": {
|
||||
"type": "invalid_request_error",
|
||||
"message": "prompt is too long: 213021 tokens > 200000 maximum",
|
||||
},
|
||||
}
|
||||
).encode()
|
||||
upstream = self._mock_upstream(error_body, status_code=400)
|
||||
upstream.headers = __import__("httpx").Headers({"content-type": "application/json"})
|
||||
result = await self._run(
|
||||
proxy, upstream, {"x-api-key": "sk-test", "anthropic-beta": "context-1m-2025-08-07"}
|
||||
)
|
||||
assert result.status_code == 400
|
||||
# Learned against the credential that hit the wall, not process-wide.
|
||||
scope = credential_scope("sk-test")
|
||||
assert (
|
||||
effective_context_limit(
|
||||
"claude-sonnet-4-5", 200_000, "context-1m-2025-08-07", scope=scope
|
||||
)
|
||||
== 200_000
|
||||
)
|
||||
# A different key on the same proxy keeps its own (optimistic) budget.
|
||||
assert (
|
||||
effective_context_limit(
|
||||
"claude-sonnet-4-5",
|
||||
200_000,
|
||||
"context-1m-2025-08-07",
|
||||
scope=credential_scope("sk-other"),
|
||||
)
|
||||
== 1_000_000
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sticky_1m_beta_does_not_inflate_a_200k_client_gauge(self):
|
||||
"""The client dropped context-1m; session-sticky merging kept it outbound.
|
||||
|
||||
The learned cap still keys on the forwarded beta, but the client's
|
||||
gauge is its own 200k window, so the nudge must target 95% of 200k.
|
||||
"""
|
||||
beta_1m = "context-1m-2025-08-07"
|
||||
note_prompt_too_long(
|
||||
"claude-sonnet-4-5",
|
||||
beta_1m,
|
||||
"prompt is too long: 213021 tokens > 200000 maximum",
|
||||
scope=credential_scope("sk-sticky"),
|
||||
)
|
||||
proxy = self._create_mock_proxy()
|
||||
sse = _message_start_event(185_000) + _message_delta_event(185_000)
|
||||
result = await self._run(
|
||||
proxy,
|
||||
self._mock_upstream(sse),
|
||||
{"x-api-key": "sk-sticky", "anthropic-beta": beta_1m},
|
||||
client_beta="",
|
||||
)
|
||||
client_bytes = b"".join([chunk async for chunk in result.body_iterator])
|
||||
events = client_bytes.split(b"\n\n")
|
||||
assert _parse_usage(events[0] + b"\n\n")["input_tokens"] == int(200_000 * REPORT_FRACTION)
|
||||
delta_usage = json.loads(events[1].split(b"data: ")[1])["usage"]
|
||||
assert delta_usage["input_tokens"] == int(200_000 * REPORT_FRACTION)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_guard_nudges_what_the_memory_filter_releases(self):
|
||||
"""The guard runs after the memory filter, on the client-bound frames.
|
||||
|
||||
The filter withholds the proxy's memory call and holds message_delta
|
||||
until the turn's outcome is known; that held delta is still the one
|
||||
the client merges over message_start, so it must come out nudged.
|
||||
"""
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
def frame(payload: dict) -> bytes:
|
||||
return f"event: {payload['type']}\ndata: {json.dumps(payload)}\n\n".encode()
|
||||
|
||||
def tool_block(index: int, tool_id: str, name: str) -> bytes:
|
||||
return (
|
||||
frame(
|
||||
{
|
||||
"type": "content_block_start",
|
||||
"index": index,
|
||||
"content_block": {"type": "tool_use", "id": tool_id, "name": name},
|
||||
}
|
||||
)
|
||||
+ frame(
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": index,
|
||||
"delta": {"type": "input_json_delta", "partial_json": "{}"},
|
||||
}
|
||||
)
|
||||
+ frame({"type": "content_block_stop", "index": index})
|
||||
)
|
||||
|
||||
delta = json.loads(_message_delta_event(185_000).split(b"data: ")[1])
|
||||
delta["delta"]["stop_reason"] = "tool_use"
|
||||
sse = (
|
||||
_message_start_event(185_000)
|
||||
+ tool_block(0, "toolu_mem", "memory_save")
|
||||
+ tool_block(1, "toolu_bash", "Bash")
|
||||
+ frame(delta)
|
||||
+ frame({"type": "message_stop"})
|
||||
)
|
||||
proxy = self._create_mock_proxy()
|
||||
proxy.memory_handler = MagicMock()
|
||||
proxy.memory_handler.handle_memory_tool_calls = AsyncMock(
|
||||
return_value=[{"type": "tool_result", "tool_use_id": "toolu_mem", "content": "ok"}]
|
||||
)
|
||||
result = await self._run(
|
||||
proxy,
|
||||
self._mock_upstream(sse),
|
||||
{"x-api-key": "sk-test"},
|
||||
memory_user_id="user-1",
|
||||
server_memory_tool_names=frozenset({"memory_save"}),
|
||||
)
|
||||
client_bytes = b"".join([chunk async for chunk in result.body_iterator])
|
||||
events = [
|
||||
json.loads(line[len(b"data: ") :])
|
||||
for line in client_bytes.split(b"\n")
|
||||
if line.startswith(b"data: ")
|
||||
]
|
||||
|
||||
assert b"memory_save" not in client_bytes
|
||||
assert events[0]["message"]["usage"]["input_tokens"] == int(200_000 * REPORT_FRACTION)
|
||||
final = [e for e in events if e["type"] == "message_delta"]
|
||||
assert len(final) == 1
|
||||
assert final[0]["delta"]["stop_reason"] == "tool_use"
|
||||
assert final[0]["usage"]["input_tokens"] == int(200_000 * REPORT_FRACTION)
|
||||
proxy.memory_handler.handle_memory_tool_calls.assert_awaited_once()
|
||||
Reference in New Issue
Block a user