mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 16:45:34 +08:00
feat/https-compose-caddy
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
50bf775e8c |
fix(proxy): bypass compression for large admitted histories (#4142)
## Description Closes #4139. A history that is too large to compress should still reach the provider. This keeps the 100 MiB hard admission limit and bypasses request-side transforms above 16 MiB decoded size. The hard limit still applies before forwarding. Current head: `62ffe3defef1684f70d96f1640818d99445a43bb`, merging current main `a2d4fe72` into `1902d8ef`. The 42-test/native/manual/pre-push results below qualify `1902d8ef`, not this new merge. New-head shard 4 has a cached-prefix fixture failure under investigation; coverage uploads are still incomplete. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement - [x] Documentation update ## Changes Made - Keep the startup override and both wire/decompressed hard bounds. Large admitted histories skip normalization, image processing, memory injection, tokenization and compression on Messages, Chat/Responses, Gemini and Bedrock HTTP routes. - Keep provider routing and backend translation, including Copilot Chat, Codex `store:false` and Vertex route-forced streaming. Native forwarding retains decoded bytes unless the provider's wire contract requires a rewrite; translation retains the full message text. - Report actual provider usage, with zero compression savings or attempted tokens for the bypass. Keep `/v1/compress` and WebSocket contracts unchanged. - Add real-provider coverage for large Gemini JSON/SSE requests with query-key auth, and OpenAI JSON/SSE through the installed LiteLLM adapter. No admission-policy or CI changes in this follow-up. ## Testing - [x] Unit tests pass (focused suite and normal critical pre-push suite) - [x] Linting passes (normal hooks) - [x] New tests added for admission, recovery and passthrough accounting - [x] Manual testing performed against the real native build ### Test Output ```sh python -m pytest -q tests/test_proxy_compression_admission.py tests/test_proxy/test_gemini_plugin_routed_streaming.py --cov=headroom --cov-branch # 42 passed, 41 warnings ``` Normal pre-push `make ci-precheck` passed: 1668 Rust tests (3 ignored), 181 Python tests, lint/type checks and commitlint. No hook bypass. The initial deep-module coverage command hit a Pydantic import-isolation error; using the repo's `--cov=headroom` convention passed both complete modules without disabling plugins or tests. The earlier `96c5fe7e` qualification covered the other native protocols, configured/per-request backends, hard-ceiling recovery and complete local shards 2/3. Those runs are historical, not a new whole-suite claim. New-head hosted success is not claimed. ## Real Behavior Proof - Environment: Linux x86_64, Python 3.13.14, freshly built real `headroom._core`, public Headroom ASGI routes, a real localhost HTTP provider and installed LiteLLM. Synthetic credentials, offline mode and `DO_NOT_TRACK=1`; no live provider calls. - Exact command / steps: Start `create_app(ProxyConfig(backend="litellm-openai", ...))` with all upstream URLs set to the localhost peer and `litellm.api_base=peer+"/v1"`. Leave the production 16 MiB soft/100 MiB hard limits unchanged. POST `"history " * (16777216 // 8 + 1024) + "LAST TURN"` as a user message to Chat Completions, then as a Gemini content part to generateContent/streamGenerateContent, with streaming off/on. The peer rejects a changed history, requires Gemini query-key auth and one `alt=sse`, and returns protocol-valid JSON/SSE with 101 input/1 output tokens. The maintained four small-ceiling cases are in `tests/test_proxy_compression_admission.py` and run with the command above. - Observed result: All four >16 MiB requests returned 200 with the full history intact. JSON/SSE and Gemini auth survived; streamGenerateContent forced SSE even with incoming `alt=json`. Each request recorded exactly 101 provider input/1 output tokens, zero compression attempts/savings and no failures; `/livez` stayed 200. Earlier recorded qualification also covered all five native HTTP protocols, real translating backends and 413 recovery. - Not tested: Live cloud accounts, the reporter's deployed 4 GB VPS, screenshot/image quality or a zero-JSON-parse path. One envelope/model JSON parse remains; no proxy-only RSS guarantee is claimed. ## Runtime Rollout Safety - Rollout-managed feature(s): None; this is the admission/compatibility fix. - Minimum rollout channel: Stable/default. - Stable/default behavior changed: Keep the 100 MiB hard ceiling; bypass request-side transforms above 16 MiB decoded size on the named HTTP routes. - Kill switch / disable path: Set `HEADROOM_REQUEST_BODY_MAX_BYTES=16777216` (or lower) and restart, accepting local 413s instead of the large-body path. - Unsafe override required: No. - Qualification impact: Local native protocols, backend translation, hard bounds and recovery exercised; live-provider/VPS qualification remains external. - Rollback path: Revert the admission revision and restart; no storage migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] I have made corresponding changes to the documentation - [x] I did not edit `CHANGELOG.md` --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
4fa88026d9 |
feat(compress): session-aware /v1/compress (sidecar mode) + /v1/usage relay (#3270)
> Replaces #3262 (same changeset, squashed to one conventional commit — the stacked branch's history could not pass commitlint after #3261's squash-merge broke ancestry, and force-pushing the original branch was not permitted). All review findings from the two max-effort reviews are already incorporated; #3261 is merged. ## Why Gateways that own routing (e.g. Kong as the upstream caller) can't use Headroom's proxy path, and the stateless `/v1/compress` pushes all byte-replay bookkeeping onto the caller. This PR moves that state into the endpoint: **the caller sends the raw conversation + a session id every turn, forwards the returned bytes verbatim, and gets a byte-identical prefix — provider prompt cache preserved, no forwarding through Headroom.** ## Design - **Session pre-work** mirrors the proxy's Zone 1: content-addressed swap of previously-computed compressed bytes, then freeze the **entire locally-replayable prefix** (`compute_frozen_count`). - **Freeze posture deliberately differs from the proxy's `min(tracker, cache)`**: in sidecar mode, whatever this endpoint previously returned *is* the provider's cache contract — recompressing an already-returned message (even into a smaller form) is a bust. Over-freezing only forgoes tail compression; it can never bust. (A test caught exactly this: recompression drift produced a smaller form, and `overlay_cached_prefix`'s non-inflation guard then couldn't repair it.) - **`PrefixCacheTracker.record_returned()`** — the sidecar equivalent of "last forwarded", captured at return time because whatever is returned is what the caller forwards. - **`POST /v1/usage`** (same loopback exposure policy): the caller relays the provider's usage block; `update_from_response` makes freeze decisions provider-confirmed. Optional — skipping it degrades freeze precision, never correctness. - Sessions are NUL-namespaced (`compress\x00<id>`, unspoofable via HTTP headers); the registry's TTL/LRU lifecycle from #3261 applies automatically. No session id ⇒ stateless contract byte-for-byte unchanged. ## Hardening (from two max-effort code reviews, all applied) - `compress_user_messages` + session_id → 400 (user-message rewrites are not content-addressed → guaranteed later bust). - Session-mode timeout / lock-busy → 503 `compression_timeout` / `session_busy` with retry semantics, instead of failing open with raw bytes (desync bust). - Header-based session ids gated behind `HEADROOM_COMPRESS_SESSION_FROM_HEADER` (default off). - `/v1/usage` validation: unknown/expired session → 404; both cache fields absent → 400; single-present-zero → `{"applied": false, "reason": "no_cache_signal"}` (never wipes freeze state). - Warm-turn savings recomputed from the raw payload (honest `tokens_saved`), all CPU work in the executor under a per-session turn lock. ## Caller contract (Kong) 1. Send raw history + `config.session_id` (or `x-headroom-session-id` with the env gate on) every turn. 2. Forward the returned `messages` to the provider **verbatim**. 3. Optionally relay the provider's usage block to `/v1/usage`. ## Testing 20 cases in `tests/test_compress_session_mode.py`: stateless regression + no state leakage, invalid-id rejection, 2-turn and 3-turn whole-prefix byte-stability, tracker-loss stability, spoof-resistance, header gating, lock-busy 503s, usage validation and no-signal handling, unknown/expired-session 404, TTL-eviction fail-open, explicit `frozen_message_count` precedence. Plus the full local suite green (11k+ tests). ## Phase 2 (follow-up) #3263 migrates the proxy request path onto this same session engine so both modes share one compression/state codepath. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01EWKCmcH47hvvoQ35wftXhE Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
826b600c9b |
feat(proxy): self-limiting session state for the compression-cache registry (#3261)
## Problem The per-session `CompressionCache` registry (the map that replays previously-compressed messages byte-identically so the provider prefix cache stays warm) had no lifetime management: - Idle/dead sessions lived forever until the hardcoded 500-session cap was hit. - At capacity, eviction dropped the oldest-**created** quarter — which could wipe the busiest long-lived session (busting every one of its prefixes at once) while dead sessions survived. - Neither the cap nor any TTL was tunable, which blocks gateway deployments (e.g. Kong sidecar/pool) fanning many concurrent sessions into one process. ## Changes - **Idle-TTL sweep**: sessions idle longer than `HEADROOM_COMPRESSION_CACHE_TTL_SECONDS` (default 3900s) are evicted by a lazy sweep, at most once per 60s, piggybacked on `_get_compression_cache` — same pattern as `PrefixCacheTrackerRegistry._maybe_cleanup`, no background task. `last_seen` refreshes on **every** access, so an active session never expires. - **LRU capacity eviction**: the registry is now an access-ordered `OrderedDict`; capacity pressure sheds the *idlest* quarter, never a busy session. - **Tunable cap**: `HEADROOM_COMPRESSION_CACHE_MAX_SESSIONS` (default 500, floor 1). ## Why 3900s Eviction is bust-free only once the provider's own prompt cache has lapsed. Providers don't expose their cache TTLs, and the risk is one-sided (late eviction costs a few MB; early eviction *causes* the bust this state exists to prevent), so the default is the upper bound of documented lifetimes across providers — Anthropic's 1h extended breakpoint, OpenAI's "up to an hour off-peak", Gemini's 60-min default — plus 5m grace. A parse-time floor of 600s keeps the TTL from ever dropping below the prefix tracker's session TTL: after the tracker expires, the byte-identical swap is the only remaining protection for a still-live provider prefix. Read-hit signals are untouched: they govern the freeze boundary, never eviction — `read_hits == 0` usually means cold start or TTL lapse, where the map was just (re)written into the provider cache and deleting it would guarantee a second bust. ## Behavior impact - Steady state (any session active within the TTL): zero change — same instances, same bytes, same freeze behavior. - A session returning after >65 min idle now finds its map evicted — but every provider had already forgotten its prefix by then, so that turn was paying the cache-write price regardless (fail-open, no failed requests). - Capacity eviction now protects busy sessions instead of punishing them. ## Testing - New `tests/test_compression_cache_registry.py`: LRU-not-FIFO capacity eviction, small-cap edge case, TTL sweep eviction, access-refreshes-clock, sweep rate limiting. - 386 tests pass across compression-cache, cache-stability (Anthropic + OpenAI), prefix-overlay, cold-start, cache-mode, and Bedrock-tracker suites; ruff check/format clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01EWKCmcH47hvvoQ35wftXhE --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |