Commit Graph
3 Commits
Author SHA1 Message Date
Mansurshakh JaparovandJerrettDavis 50bf775e8c fix(proxy): bypass compression for large admitted histories (#4142)
## Description

Closes #4139.

A history that is too large to compress should still reach the provider.
This keeps the 100 MiB hard admission limit and bypasses request-side
transforms above 16 MiB decoded size. The hard limit still applies
before forwarding.

Current head: `62ffe3defef1684f70d96f1640818d99445a43bb`, merging
current main `a2d4fe72` into `1902d8ef`. The
42-test/native/manual/pre-push results below qualify `1902d8ef`, not
this new merge. New-head shard 4 has a cached-prefix fixture failure
under investigation; coverage uploads are still incomplete.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement
- [x] Documentation update

## Changes Made

- Keep the startup override and both wire/decompressed hard bounds.
Large admitted histories skip normalization, image processing, memory
injection, tokenization and compression on Messages, Chat/Responses,
Gemini and Bedrock HTTP routes.
- Keep provider routing and backend translation, including Copilot Chat,
Codex `store:false` and Vertex route-forced streaming. Native forwarding
retains decoded bytes unless the provider's wire contract requires a
rewrite; translation retains the full message text.
- Report actual provider usage, with zero compression savings or
attempted tokens for the bypass. Keep `/v1/compress` and WebSocket
contracts unchanged.
- Add real-provider coverage for large Gemini JSON/SSE requests with
query-key auth, and OpenAI JSON/SSE through the installed LiteLLM
adapter. No admission-policy or CI changes in this follow-up.

## Testing

- [x] Unit tests pass (focused suite and normal critical pre-push suite)
- [x] Linting passes (normal hooks)
- [x] New tests added for admission, recovery and passthrough accounting
- [x] Manual testing performed against the real native build

### Test Output

```sh
python -m pytest -q tests/test_proxy_compression_admission.py tests/test_proxy/test_gemini_plugin_routed_streaming.py --cov=headroom --cov-branch
# 42 passed, 41 warnings
```

Normal pre-push `make ci-precheck` passed: 1668 Rust tests (3 ignored),
181 Python tests, lint/type checks and commitlint. No hook bypass. The
initial deep-module coverage command hit a Pydantic import-isolation
error; using the repo's `--cov=headroom` convention passed both complete
modules without disabling plugins or tests.

The earlier `96c5fe7e` qualification covered the other native protocols,
configured/per-request backends, hard-ceiling recovery and complete
local shards 2/3. Those runs are historical, not a new whole-suite
claim. New-head hosted success is not claimed.

## Real Behavior Proof

- Environment: Linux x86_64, Python 3.13.14, freshly built real
`headroom._core`, public Headroom ASGI routes, a real localhost HTTP
provider and installed LiteLLM. Synthetic credentials, offline mode and
`DO_NOT_TRACK=1`; no live provider calls.
- Exact command / steps: Start
`create_app(ProxyConfig(backend="litellm-openai", ...))` with all
upstream URLs set to the localhost peer and
`litellm.api_base=peer+"/v1"`. Leave the production 16 MiB soft/100 MiB
hard limits unchanged. POST `"history " * (16777216 // 8 + 1024) + "LAST
TURN"` as a user message to Chat Completions, then as a Gemini content
part to generateContent/streamGenerateContent, with streaming off/on.
The peer rejects a changed history, requires Gemini query-key auth and
one `alt=sse`, and returns protocol-valid JSON/SSE with 101 input/1
output tokens. The maintained four small-ceiling cases are in
`tests/test_proxy_compression_admission.py` and run with the command
above.
- Observed result: All four >16 MiB requests returned 200 with the full
history intact. JSON/SSE and Gemini auth survived; streamGenerateContent
forced SSE even with incoming `alt=json`. Each request recorded exactly
101 provider input/1 output tokens, zero compression attempts/savings
and no failures; `/livez` stayed 200. Earlier recorded qualification
also covered all five native HTTP protocols, real translating backends
and 413 recovery.
- Not tested: Live cloud accounts, the reporter's deployed 4 GB VPS,
screenshot/image quality or a zero-JSON-parse path. One envelope/model
JSON parse remains; no proxy-only RSS guarantee is claimed.

## Runtime Rollout Safety

- Rollout-managed feature(s): None; this is the admission/compatibility
fix.
- Minimum rollout channel: Stable/default.
- Stable/default behavior changed: Keep the 100 MiB hard ceiling; bypass
request-side transforms above 16 MiB decoded size on the named HTTP
routes.
- Kill switch / disable path: Set
`HEADROOM_REQUEST_BODY_MAX_BYTES=16777216` (or lower) and restart,
accepting local 413s instead of the large-body path.
- Unsafe override required: No.
- Qualification impact: Local native protocols, backend translation,
hard bounds and recovery exercised; live-provider/VPS qualification
remains external.
- Rollback path: Revert the admission revision and restart; no storage
migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] I have made corresponding changes to the documentation
- [x] I did not edit `CHANGELOG.md`

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-04 22:37:21 -06:00
Tejas ChopraandClaude Fable 5 4fa88026d9 feat(compress): session-aware /v1/compress (sidecar mode) + /v1/usage relay (#3270)
> Replaces #3262 (same changeset, squashed to one conventional commit —
the stacked branch's history could not pass commitlint after #3261's
squash-merge broke ancestry, and force-pushing the original branch was
not permitted). All review findings from the two max-effort reviews are
already incorporated; #3261 is merged.

## Why

Gateways that own routing (e.g. Kong as the upstream caller) can't use
Headroom's proxy path, and the stateless `/v1/compress` pushes all
byte-replay bookkeeping onto the caller. This PR moves that state into
the endpoint: **the caller sends the raw conversation + a session id
every turn, forwards the returned bytes verbatim, and gets a
byte-identical prefix — provider prompt cache preserved, no forwarding
through Headroom.**

## Design

- **Session pre-work** mirrors the proxy's Zone 1: content-addressed
swap of previously-computed compressed bytes, then freeze the **entire
locally-replayable prefix** (`compute_frozen_count`).
- **Freeze posture deliberately differs from the proxy's `min(tracker,
cache)`**: in sidecar mode, whatever this endpoint previously returned
*is* the provider's cache contract — recompressing an already-returned
message (even into a smaller form) is a bust. Over-freezing only forgoes
tail compression; it can never bust. (A test caught exactly this:
recompression drift produced a smaller form, and
`overlay_cached_prefix`'s non-inflation guard then couldn't repair it.)
- **`PrefixCacheTracker.record_returned()`** — the sidecar equivalent of
"last forwarded", captured at return time because whatever is returned
is what the caller forwards.
- **`POST /v1/usage`** (same loopback exposure policy): the caller
relays the provider's usage block; `update_from_response` makes freeze
decisions provider-confirmed. Optional — skipping it degrades freeze
precision, never correctness.
- Sessions are NUL-namespaced (`compress\x00<id>`, unspoofable via HTTP
headers); the registry's TTL/LRU lifecycle from #3261 applies
automatically. No session id ⇒ stateless contract byte-for-byte
unchanged.

## Hardening (from two max-effort code reviews, all applied)

- `compress_user_messages` + session_id → 400 (user-message rewrites are
not content-addressed → guaranteed later bust).
- Session-mode timeout / lock-busy → 503 `compression_timeout` /
`session_busy` with retry semantics, instead of failing open with raw
bytes (desync bust).
- Header-based session ids gated behind
`HEADROOM_COMPRESS_SESSION_FROM_HEADER` (default off).
- `/v1/usage` validation: unknown/expired session → 404; both cache
fields absent → 400; single-present-zero → `{"applied": false, "reason":
"no_cache_signal"}` (never wipes freeze state).
- Warm-turn savings recomputed from the raw payload (honest
`tokens_saved`), all CPU work in the executor under a per-session turn
lock.

## Caller contract (Kong)

1. Send raw history + `config.session_id` (or `x-headroom-session-id`
with the env gate on) every turn.
2. Forward the returned `messages` to the provider **verbatim**.
3. Optionally relay the provider's usage block to `/v1/usage`.

## Testing

20 cases in `tests/test_compress_session_mode.py`: stateless regression
+ no state leakage, invalid-id rejection, 2-turn and 3-turn whole-prefix
byte-stability, tracker-loss stability, spoof-resistance, header gating,
lock-busy 503s, usage validation and no-signal handling,
unknown/expired-session 404, TTL-eviction fail-open, explicit
`frozen_message_count` precedence. Plus the full local suite green (11k+
tests).

## Phase 2 (follow-up)

#3263 migrates the proxy request path onto this same session engine so
both modes share one compression/state codepath.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01EWKCmcH47hvvoQ35wftXhE

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-26 15:34:23 +05:30
Tejas ChopraandClaude Fable 5 826b600c9b feat(proxy): self-limiting session state for the compression-cache registry (#3261)
## Problem

The per-session `CompressionCache` registry (the map that replays
previously-compressed messages byte-identically so the provider prefix
cache stays warm) had no lifetime management:

- Idle/dead sessions lived forever until the hardcoded 500-session cap
was hit.
- At capacity, eviction dropped the oldest-**created** quarter — which
could wipe the busiest long-lived session (busting every one of its
prefixes at once) while dead sessions survived.
- Neither the cap nor any TTL was tunable, which blocks gateway
deployments (e.g. Kong sidecar/pool) fanning many concurrent sessions
into one process.

## Changes

- **Idle-TTL sweep**: sessions idle longer than
`HEADROOM_COMPRESSION_CACHE_TTL_SECONDS` (default 3900s) are evicted by
a lazy sweep, at most once per 60s, piggybacked on
`_get_compression_cache` — same pattern as
`PrefixCacheTrackerRegistry._maybe_cleanup`, no background task.
`last_seen` refreshes on **every** access, so an active session never
expires.
- **LRU capacity eviction**: the registry is now an access-ordered
`OrderedDict`; capacity pressure sheds the *idlest* quarter, never a
busy session.
- **Tunable cap**: `HEADROOM_COMPRESSION_CACHE_MAX_SESSIONS` (default
500, floor 1).

## Why 3900s

Eviction is bust-free only once the provider's own prompt cache has
lapsed. Providers don't expose their cache TTLs, and the risk is
one-sided (late eviction costs a few MB; early eviction *causes* the
bust this state exists to prevent), so the default is the upper bound of
documented lifetimes across providers — Anthropic's 1h extended
breakpoint, OpenAI's "up to an hour off-peak", Gemini's 60-min default —
plus 5m grace. A parse-time floor of 600s keeps the TTL from ever
dropping below the prefix tracker's session TTL: after the tracker
expires, the byte-identical swap is the only remaining protection for a
still-live provider prefix.

Read-hit signals are untouched: they govern the freeze boundary, never
eviction — `read_hits == 0` usually means cold start or TTL lapse, where
the map was just (re)written into the provider cache and deleting it
would guarantee a second bust.

## Behavior impact

- Steady state (any session active within the TTL): zero change — same
instances, same bytes, same freeze behavior.
- A session returning after >65 min idle now finds its map evicted — but
every provider had already forgotten its prefix by then, so that turn
was paying the cache-write price regardless (fail-open, no failed
requests).
- Capacity eviction now protects busy sessions instead of punishing
them.

## Testing

- New `tests/test_compression_cache_registry.py`: LRU-not-FIFO capacity
eviction, small-cap edge case, TTL sweep eviction,
access-refreshes-clock, sweep rate limiting.
- 386 tests pass across compression-cache, cache-stability (Anthropic +
OpenAI), prefix-overlay, cold-start, cache-mode, and Bedrock-tracker
suites; ruff check/format clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01EWKCmcH47hvvoQ35wftXhE

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-26 15:04:43 +05:30