Commit Graph
5 Commits
Author SHA1 Message Date
Tejas ChopraandJerrettDavis 62cc38a6b0 fix(backends): forward the standard service_tier param (litellm + anyllm) (#2468)
## Description

`service_tier` is a standard provider request field (OpenAI
`flex`/`default`/`priority`, Anthropic `auto`/`standard_only`), but the
SDK-translation backends rebuilt the upstream request from a fixed field
whitelist and **silently dropped** it — so a client that set
`service_tier` had it discarded on `--backend litellm`/`anyllm`. The
direct passthrough already forwards it; this makes the translation
backends do the same, alongside `temperature`/`top_p`/`tools`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `backends/litellm.py` (Anthropic path): forward `service_tier` when
present (the OpenAI path already forwards unconsumed fields via
`_build_openai_extra_body`).
- `backends/anyllm.py`: add `service_tier` to the forwarded params (both
the Anthropic whitelist and the OpenAI param list).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New test added

### Test Output

```text
$ python -m pytest tests/test_backend_anyllm.py -q
16 passed in 2.51s

$ ruff check headroom/backends/litellm.py headroom/backends/anyllm.py
All checks passed!

$ mypy headroom/backends/anyllm.py headroom/backends/litellm.py   # 0 new errors
```

## Real Behavior Proof

- Environment: local, FakeAnyLLM instance capturing `acompletion`
kwargs.
- Steps: `send_message({..., "service_tier": "standard_only"})`.
- Observed: `instance.calls[0]["service_tier"] == "standard_only"` — the
param reaches the SDK call (previously absent).
- Not tested: live provider round-trip (no keys in CI); relies on the
SDK forwarding standard params.

## Checklist

- [x] Self-reviewed
- [x] Tests added and passing
- [x] I did **not** edit `CHANGELOG.md`
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2468"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-03 22:29:09 -06:00
Ayush Kumar JhaandJD Davis 0c19244e38 feat(litellm): opt-in Bedrock prompt caching on the OpenAI-compatible path (#3565)
## Description

On `--backend bedrock`, the OpenAI-compatible route
(`send_openai_message` / `stream_openai_message`) forwards the client's
`messages` verbatim. A client that never sends `cache_control` —
OpenAI-compat gateways such as Bifrost strip it — therefore never gets a
Bedrock `cachePoint`, and `cache_read_input_tokens` never appears in
usage even for models litellm reports as caching-capable.

Note on the issue's stated root cause: there is no
`litellm.get_supported_openai_params` gate anywhere in headroom (checked
`main` and the v0.37.0 commit the reporter pinned), and litellm's
Converse transform honours `cache_control` unconditionally. The caching
half of #3554 is a missing behaviour, not a wrong API call, so this PR
adds it **opt-in** rather than changing the default. The
`max_completion_tokens` half is #3558.

Design:
- New rollout feature `bedrock_openai_prompt_caching` (stable channel,
**not** default-enabled): a breakpoint bills the system prompt at the
cache-write rate until it is read back, so a one-shot caller who did not
ask for it would pay more.
`HEADROOM_FEATURES=bedrock_openai_prompt_caching` turns it on; flipping
the default later is a one-line `default_enabled_in` change if you want
it.
- When enabled and `litellm.utils.supports_prompt_caching()` is true for
the **mapped** litellm model
(`bedrock/us.anthropic.claude-sonnet-5-v1:0`, not the client alias —
`claude-sonnet-5` alone is answered from litellm's Anthropic-direct
registry entry, not the Bedrock one), the first system message gets
`cache_control: {"type": "ephemeral"}`; litellm turns it into a Converse
`cachePoint` covering tools + system prefix. First (not last) so
volatile trailing system messages cannot turn every turn into a cache
write.
- Untouched: requests that already carry client-placed markers anywhere
(the client owns placement, as on the Anthropic route), models without
prompt caching (incl. ARN-mapped inference profiles, which litellm
cannot look up), non-Bedrock providers. The caller's `body["messages"]`
is never mutated (the proxy re-reads it for `POST_SEND`).

Refs #3554 (the `max_completion_tokens` half is #3558)

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/rollout.py`: register `bedrock_openai_prompt_caching`
(available in `stable`, opt-in, no legacy env alias).
- `headroom/backends/litellm.py`: import
`litellm.utils.supports_prompt_caching` alongside `acompletion`; add
pure helper `_place_system_cache_control()`; resolve the rollout
decision once in `LiteLLMBackend.__init__` (Bedrock only); gate both
OpenAI call sites on the decision + capability lookup (~20 µs/call, no
memoisation needed).
- `tests/test_backends/test_litellm_bedrock_prompt_caching.py`: 26 tests
— helper edge cases (empty/none/wrong-type content, list vs string
content, non-dict entries, first-of-several, client-placed markers
anywhere, input never mutated), gate off by default (identity), on for
buffered + streaming, skips non-caching models, non-Bedrock no-op
(lookup not even called), uses the mapped id, kill switch, and litellm's
real Converse transform emitting `cachePoint` from the produced
messages.
- `tests/test_rollout.py`: one assertion pinned a positional index into
the alphabetically sorted feature list; made it name-based so adding a
feature does not shift it.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_backends/test_litellm_bedrock_prompt_caching.py -q
26 passed in 4.11s

# same file on unpatched main (helper does not exist):
E   ImportError: cannot import name '_place_system_cache_control' from 'headroom.backends.litellm'

$ python -m pytest tests/test_backends tests/test_litellm_openai_passthrough.py tests/test_litellm_caller_key.py \
    tests/test_litellm_nonstream_cache_usage.py tests/test_litellm_optional.py tests/test_litellm_upstream_timeout.py \
    tests/test_rollout.py tests/test_bedrock_region.py tests/test_bedrock_streaming_input_tokens.py \
    tests/test_bedrock_tool_result_cache_and_streaming_stats.py tests/test_bedrock_prefix_tracker_wiring.py \
    tests/test_proxy/test_openai_backend_path.py tests/test_proxy/test_bedrock_passthrough.py -q
189 passed  (after the test_rollout.py assertion fix; 188 passed, 1 failed before it)

$ ruff check headroom/backends/litellm.py headroom/rollout.py tests/test_backends/test_litellm_bedrock_prompt_caching.py tests/test_rollout.py
All checks passed!
$ ruff format --check headroom/backends/litellm.py headroom/rollout.py tests/test_backends/test_litellm_bedrock_prompt_caching.py tests/test_rollout.py
4 files already formatted
$ mypy headroom/backends/litellm.py headroom/rollout.py
Success: no issues found in 2 source files

# Full local suite (12,650 collected; Windows, borrowed 0.35.0 headroom._core, known Windows-only
# CLI tests deselected) was still running when this PR was opened; result will be added below.
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.11.9, litellm 1.97.0 (issue reports
1.96.2; same `supports_prompt_caching` / Converse behaviour), boto3
1.43.93, headroom source checkout at this branch,
`LiteLLMBackend(provider="bedrock", region="us-east-1")`, model
`global.anthropic.claude-sonnet-5`. No live AWS account available to me,
so the request was driven through litellm's real pipeline (message
validation, Converse transformation, SigV4 signing) and the outgoing
`AsyncHTTPHandler.post` was intercepted to capture the exact wire body
and return a canned Bedrock response.
- Exact command / steps:
`HEADROOM_FEATURES=bedrock_openai_prompt_caching python wire_proof.py`
(script in the PR discussion) calling
`backend.send_openai_message({"model":
"global.anthropic.claude-sonnet-5", "max_tokens": 64, "messages":
[{"role": "system", "content": <~1800 tokens>}, {"role": "user",
"content": "hi"}]}, {})`; then the same command without the env var.
- Observed result: with the feature on, the captured POST to
`https://bedrock-runtime.us-east-1.amazonaws.com/model/global.anthropic.claude-sonnet-5/converse`
carries `"system": [{"text": ...}, {"cachePoint": {"type":
"default"}}]`, SigV4 `Authorization` header present, and the client's
`body["messages"]` still contains no `cache_control`. Without the env
var (default) the same request carries `"system": [{"text": ...}]` only
— byte-identical to today's behaviour. `headroom rollout status` lists
`bedrock_openai_prompt_caching: enabled=false decision=not_requested` by
default.
- Not tested: a live Bedrock round trip (no AWS credentials here), so
the returned `cache_creation_input_tokens` / `cache_read_input_tokens`
were not observed from Bedrock itself; Bifrost/Cline in front of the
proxy; Nova models (litellm reports `supports_prompt_caching=True` for
them, so they would be marked too); the Anthropic-format route is
untouched by this change.

## Runtime Rollout Safety

- Rollout-managed feature(s): `bedrock_openai_prompt_caching` (new entry
in `headroom/rollout.py` FEATURES).
- Minimum rollout channel: stable (explicit request required in every
channel; no `default_enabled_in`).
- Stable/default behavior changed: no. Without
`HEADROOM_FEATURES=bedrock_openai_prompt_caching` the backend forwards
the client's `messages` object untouched (pinned by an identity
assertion for buffered and streaming). Only `--backend bedrock` can ever
be affected.
- Kill switch / disable path: unset `HEADROOM_FEATURES`, or
`HEADROOM_DISABLE_FEATURES=bedrock_openai_prompt_caching` (explicit
disable wins over an explicit request; tested).
- Unsafe override required: no.
- Qualification impact: none; the feature is `not_requested` in the
default snapshot, so the snapshot digest of an unconfigured deployment
only changes by the registry digest (as for any new feature).
- Rollback path: revert the commit; no persisted state, schema, or
config format involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: not changed on purpose — there is no catalogue of
rollout features in `docs/`; the `FeatureSpec.description` is what
`headroom rollout status` prints. Happy to add a line to
`docs/content/docs/runtime-rollouts.mdx` if you want one.
- Open questions for the maintainer: (1) default-on for Bedrock instead
of opt-in? One-line change (`default_enabled_in=RolloutChannel.STABLE`),
but it changes billing for one-shot callers. (2) Prefer the *last*
system message (max cache coverage) over the first (robust to volatile
trailing system messages)? (3) ARN-mapped application inference profiles
(`HEADROOM_BEDROCK_MODEL_MAP`) get nothing because litellm cannot look
them up — acceptable, or gate on the underlying model family instead?
- Alternatives considered: default-on with no flag (rejected: silent
cost change, no kill switch); reusing
`AnthropicCacheOptimizer._insert_breakpoints` (Anthropic-format only,
mutates in place); doing it in the OpenAI handler (provider-agnostic,
lacks the mapped model id).
- Verification for someone with Bedrock access: run the proxy with
`--backend bedrock` and
`HEADROOM_FEATURES=bedrock_openai_prompt_caching`, send two identical
OpenAI chat completions with a >1024-token system prompt within 5
minutes; the second response's `usage` should carry non-zero
`cache_read_input_tokens` / `prompt_tokens_details.cached_tokens`.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-24 22:13:15 -05:00
Abhay SinghandJerrettDavis 12f9f58cb3 fix(backends/litellm): None-guard core token counts in OpenAI usage block (#2324)
## Description

`LiteLLMBackend.send_openai_message` builds the OpenAI-shape response
body. The core token counts are copied straight off LiteLLM's `Usage`
object with no guard, even though the cache fields immediately below
already use the defensive `int(getattr(..., 0) or 0)` form:

```python
usage_block: dict[str, Any] = {
    "prompt_tokens": response.usage.prompt_tokens,
    "completion_tokens": response.usage.completion_tokens,
    "total_tokens": response.usage.total_tokens,
}

# Defensive getattr right below:
cache_read = int(getattr(response.usage, "cache_read_input_tokens", 0) or 0)
cache_write = int(getattr(response.usage, "cache_creation_input_tokens", 0) or 0)
```

A provider can leave any of `prompt_tokens` / `completion_tokens` /
`total_tokens` as `None` on the `Usage` object. That `None` then lands
in `response.body["usage"]`, and the backend-routed OpenAI handler reads
it straight into arithmetic and the outcome ledger:

```python
usage = backend_response.body.get("usage", {})
output_tokens = usage.get("completion_tokens", 0)            # present key -> None, not the default
total_input_tokens = usage.get("prompt_tokens", optimized_tokens)  # present key -> None
...
uncached_input_tokens = max(0, total_input_tokens - cache_read_tokens - cache_write_tokens)  # None - int -> TypeError
...
RequestOutcome(..., output_tokens=output_tokens, ...)       # declared int; None crashes recording (e.g. prometheus += )
```

So an OpenAI-format request routed through a `--backend` (Bedrock /
Vertex / LiteLLM) whose provider returns a `None` count crashes on the
`max(0, None - ...)` subtraction, or later in outcome recording.
`.get(key, default)` does not help here because the key is present with
a `None` value, so the default never applies. This is the same class of
bug as the Anthropic-shape mapping and is fixed the same way.

## Fix

Coerce the three counts to `int` with the same defensive form already
used for the cache fields two lines down:

```python
usage_block: dict[str, Any] = {
    "prompt_tokens": int(getattr(response.usage, "prompt_tokens", 0) or 0),
    "completion_tokens": int(getattr(response.usage, "completion_tokens", 0) or 0),
    "total_tokens": int(getattr(response.usage, "total_tokens", 0) or 0),
}
```

No change for a normal integer usage; only a `None` (or absent) value
now becomes `0`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/backends/litellm.py`: `int`-coerce `prompt_tokens` /
`completion_tokens` / `total_tokens` in the `send_openai_message` usage
block.
- `tests/test_backends/test_litellm_cache_stats.py`: add
`test_none_core_counts_coerced_to_zero`, driving `send_openai_message`
with a `None`-count usage and asserting the block emits `int` `0`s.
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/backends/litellm.py tests/test_backends/test_litellm_cache_stats.py
All checks passed!
$ uvx ruff@0.15.17 format --check headroom/backends/litellm.py tests/test_backends/test_litellm_cache_stats.py
2 files already formatted
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/backends/litellm.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOMs this box (ML-stack import), so I
reproduced the field logic with a dependency-free script and left the
full pytest to CI.
- Exact command / steps: modeled the OLD (bare copy) and NEW
(`int(getattr(..., 0) or 0)`) field derivations for a `None` count, an
integer, and zero, then simulated the two downstream operations the
handler performs: `output_tokens += ...` and `max(0, prompt_tokens -
read - write)`.
- Observed result: OLD produced `None` and both downstream operations
raised `TypeError`; NEW produced `0` and both succeeded; an integer
count passed through unchanged. The added unit test drives
`send_openai_message` end to end (mocked `acompletion`) and asserts the
block emits `int` `0`s.
- Not tested: a live LiteLLM/Bedrock request that returns `None` counts;
the added test reuses the existing `_FakeUsage` / `_make_response` /
mocked-`acompletion` harness in `test_litellm_cache_stats.py`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" box is unchecked because a local pytest
run imports the ML stack and OOMs this box; the added test reuses the
existing mocked-`acompletion` harness in `test_litellm_cache_stats.py`
and runs under the normal CI pytest job, and the behavior is
corroborated by the standalone proof above.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-08-11 23:48:43 -05:00
Manmit SinghandJD Davis 54cfa361d3 fix(bedrock): fail fast when session-token auth lacks botocore (#1553)
## Description

With `--backend bedrock` and **temporary** AWS credentials
(`AWS_SESSION_TOKEN`, as produced by SSO / STS assume-role /
`credential_process`), every request fails. litellm self-signs Bedrock
requests without botocore for *static* IAM keys, but as soon as a
session token is present it takes the `_auth_with_aws_session_token`
path in `litellm/llms/bedrock/base_aws_llm.py`, which imports
`botocore`. botocore is an optional dependency — it ships only with
headroom's `bedrock` extra, and the default Docker image is built with
`HEADROOM_EXTRAS=proxy,code`, so botocore is absent. The failure
surfaces only at request time as a misleading `authentication_error: No
module named 'botocore'` (and as a bare `Invalid API key` in Claude
Code).

This PR makes the Bedrock backend **fail fast at startup** with an
actionable message when a session token is set but botocore is missing —
directly addressing the "clearer error message" the reporter asked for.
It mirrors the existing optional-dependency guard pattern already used
for boto3 in `backends/litellm.py`.

Scope note: this does not change what the published image ships —
whether to add botocore/`bedrock` to the default image extras is a
separate sizing decision I left to maintainers. Static-credential
Bedrock users (who never hit the botocore path) are unaffected.

Refs #1551

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/backends/litellm.py`: when initializing the Bedrock backend
with `AWS_SESSION_TOKEN` set and `botocore` not importable, raise an
`ImportError` pointing at `pip install 'headroom-ai[bedrock]'` instead
of letting the request fail later with a misleading auth error.
- `tests/test_backends/test_bedrock_botocore_preflight.py`: regression
tests — the guard raises an actionable error for the
session-token-without-botocore case, and stays quiet for the
static-credential case.
- `CHANGELOG.md`: note under Unreleased → Fixed.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`, `ruff format --check`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Regression test fails before the fix (no guard → no error raised),
passes after:

```text
# before fix (guard removed)
FAILED tests/test_backends/test_bedrock_botocore_preflight.py::test_bedrock_session_token_without_botocore_raises_actionable

# after fix
tests/test_backends/test_bedrock_botocore_preflight.py ..  [100%]
2 passed, 1 warning in 0.13s
```

`ruff check` / `ruff format --check` on the changed files: clean.

## Real Behavior Proof

- Environment: macOS (arm64), Python venv, editable install (`pip
install -e .`, no `bedrock` extra → botocore absent, matching the
reported slim-image condition), `pytest`.
- Exact command / steps: `python -m pytest
tests/test_backends/test_bedrock_botocore_preflight.py`. (1) Removed the
guard and ran the test → it failed because
`LiteLLMBackend(provider="bedrock")` with `AWS_SESSION_TOKEN` set and
botocore absent did NOT raise (reproducing the original "no early
signal" behavior). (2) Applied the guard. (3) Re-ran → both tests pass,
and the raised `ImportError` contains the `headroom-ai[bedrock]` install
hint.
- Observed result: with `AWS_SESSION_TOKEN` set and botocore not
importable, the backend now raises a clear, actionable `ImportError` at
construction time instead of deferring to litellm's later `No module
named 'botocore'` auth error. Without a session token the guard does not
fire, so static-credential users are unaffected.
- Not tested: I did not run a live Bedrock request against AWS with real
temporary credentials (no AWS account/STS access in this environment);
the reporter already confirmed that installing botocore makes the
identical request succeed, and this change surfaces that requirement at
startup.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-07-01 21:02:15 -05:00
chopratejas 20dc1f28f3 fix(proxy): Strands MCP bundle + backend path fixes + Codex fail-closed protection
Three logically-related sets of proxy changes ship in this branch:

1. Strands integration on the Bedrock path (HeadroomBundle + 4 OpenAI
   handler fixes + LiteLLM cache stats + dep pin)
2. /stats MCP aggregation (cross-process events log → proxy summary)
3. Codex compression-failure fail-closed (WS + HTTP /v1/responses)

== 1. Strands integration on the Bedrock path ==

* HeadroomBundle (headroom/integrations/strands/bundle.py): single-helper
  MCP wiring for a Strands Agent — Headroom MCP server (headroom_compress
  / headroom_retrieve / headroom_stats) plus optional Serena MCP and
  optional in-process compression hook. Constructor builds unstarted
  MCPClient instances per server; Strands' Agent owns the subprocess
  lifecycle. Default config: MCP enabled, Serena enabled, hook OFF
  (proxy is the single source of truth for compression). User-side
  integration is two lines in any Strands app.

* headroom/proxy/handlers/openai.py — backend path now:
  - calls PrefixCacheTracker.update_from_response (was direct-OpenAI only)
  - intercepts CCR headroom_retrieve tool_calls server-side, mirroring
    the Anthropic handler pattern; NO silent fallback, re-raises on
    CCR errors (per feedback_no_silent_fallbacks)
  - works for both non-streaming and streaming paths

* headroom/proxy/handlers/streaming.py: _stream_openai_via_backend now
  accepts prefix_tracker + optimized_messages, parses cache stats from
  the SSE final-usage frame (cache_creation_input_tokens added to the
  state machine), records CCR retrieve feedback via a new
  _record_ccr_feedback_from_openai_sse helper. Streaming CCR intercept
  is intentionally out of scope (mirrors Anthropic streaming behaviour).

* headroom/backends/litellm.py: send_openai_message response usage block
  now carries cache_read_input_tokens / cache_creation_input_tokens
  (Anthropic/Bedrock dialect) and prompt_tokens_details.cached_tokens
  (OpenAI dialect). Backwards-compatible — cold-start callers see the
  same 3-key shape; cache keys appear only when the underlying provider
  returns them. Pinned by test_no_cache_fields_means_no_cache_keys.

* headroom/proxy/auth_mode.py: ("strands-agents/", "strands") added to
  CLIENT_UA_MAP. Production callers should also set X-Client: strands
  since the default openai-python UA carries no Strands signal.

* pyproject.toml: huggingface-hub>=1.5.0,<2.0 pinned in [ml] so a sibling
  install (e.g. strands-agents) can't drag the version below the floor
  transformers 5.x requires (otherwise Kompress silently goes
  "unavailable").

== 2. /stats MCP aggregation ==

* headroom/proxy/cost.py: _aggregate_mcp_events() reads the cross-process
  shared events file the Headroom MCP server already writes to and
  surfaces summary.mcp with three new keys:
    - compressions       (count of headroom_compress invocations)
    - tokens_removed     (sum of input - output across those)
    - retrievals         (count of headroom_retrieve — the load-bearing
                          over-compression alarm; if it grows linearly
                          with turn count, lossy compressors are
                          dropping info the model actually needs)
  Defensive on every axis — missing MCP SDK, missing file, malformed
  events, read errors — never blocks /stats.

* examples/strands_bundle_demo.py: stats panel prints the new fields so
  the demo shows the full proxy-HTTP + MCP-tool story in one view.

== 3. Codex compression-failure fail-closed protection ==

Reported by Camille (2026-05-21): Codex threads were locking with
"ran out of room in the model's context window" after Headroom's
compression timed out on an oversized response.create frame and
forwarded the original ~1.7 MB frame to the upstream, which then
rejected it. Codex's auto-compact heuristic gates on the upstream-
reported total_usage_tokens (which Headroom had been shrinking on
earlier turns), so its compaction never fired and the thread locked.

Validated against open Codex issues (CLI + Desktop share codex-rs/core):
* #16068 — confirms compaction gates on total_usage_tokens,
  estimated_token_count is computed but only logged
* #19806 — confirms image token estimator unbounded, contributes to
  the same ContextManager.get_total_token_usage → auto-compaction chain

* headroom/proxy/helpers.py: decide_compression_failure_action() with a
  unit-tested decision matrix:
    - asyncio.TimeoutError                              → refuse, always
    - non-timeout failure + frame > 256 KiB (configurable) → refuse
    - non-timeout failure + small frame                 → forward (legacy)
  Operator escape hatches:
    - HEADROOM_WS_FAIL_OPEN_ON_COMPRESSION_FAILURE=1 restores legacy
    - HEADROOM_WS_COMPRESSION_FAIL_THRESHOLD_BYTES tunes the threshold

* headroom/proxy/handlers/openai.py (WS /v1/responses): consults the
  helper after compression failure. On refuse: close client websocket
  code 1009 with "headroom: compression <reason> — please compact
  context and retry" reason; set termination_cause for the outer
  lifecycle finally; return.

* headroom/proxy/handlers/openai.py (HTTP /v1/responses): same helper.
  On refuse: raise HTTPException(413) with a structured error body so
  FastAPI's HTTPException handler emits a clean 413. The existing
  `except HTTPException: raise` guard in this handler already ensures
  the 413 propagates without being swallowed by the 502 catch-all.

Anthropic /v1/messages NOT changed in this branch: no equivalent bug
report on Anthropic-protocol clients, Claude Code (Anthropic-owned)
handles context overflow via its own cache_control/ephemeral
primitives, and Cursor/Aider don't maintain the local-Y estimate the
Codex bug requires. Deferred until a real report lands; the patch is
a one-liner reusing the same helper.

== Tests + verification ==

* tests/test_backends/test_litellm_cache_stats.py — 3 tests pinning
  cache-stat surfacing across Anthropic/OpenAI dialects + backwards-
  compat for no-cache responses.
* tests/test_proxy/test_openai_backend_path.py — 5 tests (Bedrock cache
  fields, OpenAI fallback shape, CCR intercept with provider="openai",
  CCR re-raise on exception, streaming signature contract).
* tests/test_proxy/test_mcp_stats_aggregation.py — 5 tests pinning the
  aggregator across compress+retrieve mixes, empty events, unknown event
  types, missing token fields, and read failures.
* tests/test_proxy/test_compression_failure_action.py — 12 tests pinning
  the fail-closed decision matrix (timeout always refuses, small
  transient passes through, oversize refuses, env override variants,
  custom threshold, invalid threshold falls back, 0/negative ignored).

* examples/strands_bedrock_demo.py — model_id bumped from deprecated
  Claude 3 Haiku to Sonnet 4.5 (the deprecated model now errors on
  account access).
* examples/strands_via_proxy_demo.py — proxy + Bedrock cache + streaming
  smoke test.
* examples/strands_mcp_dispatch_test.py — pure MCP round-trip probe.
* examples/strands_bundle_demo.py — full Strands + HeadroomBundle E2E
  demo (this is the shape a real Strands user copies into their app).

Full pytest: 5327 passed, 178 skipped. The previously-failing
test_core_operations.py::TestAddBatch::test_add_batch_basic passes now
that the huggingface-hub pin in pyproject.toml unblocks transformers
imports.

E2E verified live against AWS Bedrock (Sonnet 4.5):
* cache_write=10,438 on turn A → cache_read=10,438 on turn B
* streaming SSE final usage frame carries cache_read_input_tokens
* 78.7% reduction on a 50 KB JSON tool_result via SmartCrusher (
  dispatched per-content-type by ContentRouter)
* Strands Agent + HeadroomBundle: model autonomously called
  headroom_compress + headroom_retrieve via MCP; CompressionStore
  round-trip succeeded; final answer correct.
2026-05-21 11:00:14 -07:00