Commit Graph
5 Commits
Author SHA1 Message Date
94a99ed20f feat(proxy): give turn hooks one conversation id per client conversation (#4496)
## Description

Plugins that pin wording per conversation (headroom-condense, and
through it skill-search and tool-search) cannot see a real conversation
id. Today:

- **Claude:** they hash the first typed prompt, so two sessions that
open with the same words share a key.
- **Codex:** they get no key at all, so every new skill or MCP
description waits out the one-hour hold.

This PR adds an optional `TurnContext.conversation` that core fills from
the client's own conversation id, so hooks can pin per real
conversation. It is item #4 of the compression-gaps plan; the plugin
work for #3, #6, #8 and #9 needs it to stay cache-safe.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **New field:** `TurnContext.conversation: str | None = None`. It is an
opaque key that stays the same for every turn and retry of one client
conversation, or `None` when the client names none. Hooks get a digest,
never the raw id.

  | Path | Source |
  | --- | --- |
| `/v1/messages` | sha256 of `X-Claude-Code-Session-Id` (Claude Code
sends it on every request), falling back to `X-Headroom-Session-Id`;
each source has its own namespace |
| `/v1/responses` (HTTP and WebSocket) | the new
`turn_conversation_key`, computed from the body before compression and
passed through `_compress_openai_responses_payload_in_executor` |
  | `/v1/chat/completions` | `X-Headroom-Session-Id` only |
| `/v1/compress` (gateway) | the gateway session id; `None` without one
|

- **Responses key.** `turn_conversation_key` uses the same identity and
digest as `savings_conversation_key` (`thread_id` before `session_id`,
then `client_metadata.*`, then the session header). Unlike the savings
key, it also applies on `previous_response_id` turns and on bodies that
name a server-side `conversation` (a string, or an object with an `id`),
so the key does not drop out after the first WebSocket frame.
`savings_conversation_key` keeps its existing exclusions.
- **Gateway seam.** `CompressTurn.prepare()` is a third-party seam, so
its signature is unchanged. Instead, the handler sets an optional
writable `conversation` attribute on the turn before calling `prepare`.
This is documented in `compress_turn.py`.
- **No new ids.** Nothing is generated. The per-connection WebSocket
uuid is not used, because a reconnect would change it.
- **One path left out.** The catch-all passthrough Responses route still
passes `None`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): 3 errors, all in files
this PR does not touch (embedders, model_metadata, langchain)
- [x] New tests added for new functionality
- [x] Manual testing performed

- **New tests:**
  - `tests/test_turn_context_conversation.py`:
- Anthropic: the same session gives one key, a different session with
the same prompt gives a different key, the fallback header gets its own
key, and no header gives `None`.
- Responses HTTP: the same thread gives one key, including on a
`previous_response_id` turn, and a different thread gives a different
key.
- Responses server-side `conversation`: string and object forms give one
key across turns; a different conversation gives a different key; the
savings key stays `None`.
- Responses WebSocket: first and later frames share one key; a different
thread gets a different key.
    - Chat Completions: covered the same way.
- `tests/gateway/test_contract_request_half.py`: per-session gateway
keys, and `None` without a session.
  - Every new test fails on `main`.
- **Updated test:** `tests/test_proxy_openai.py` now pins
`conversation=None` for a body with no ids.

### Test Output

```text
pytest (turn hooks, gateway, Codex WS lifecycle, proxy openai, conversation savings, passthrough, hook usage): 411 passed, 5 skipped
after the server-side conversation fix, on the synced head: 101 passed (conversation, gateway, Responses suites)
ruff@0.17.0 check / format --check: clean
mypy headroom --ignore-missing-imports: 3 errors, none in changed files
```

`tests/test_turn_hook_usage.py` needs
`LITELLM_LOCAL_MODEL_COST_MAP=True` in a fresh venv; that is an existing
litellm price-map fetch, and it fails the same way on `main`.

## Real Behavior Proof

- Environment:
  - macOS 27.0.1 (Darwin 27.0.0, arm64), Python 3.12.13.
- The proxy ran from this branch in a fresh `uv` venv (`uv venv -p 3.12
&& uv pip install -e ".[dev]"`) on 127.0.0.1:9100, with default
`ProxyConfig` (optimize on).
- Upstream was the real Anthropic API, with my own API key passed by
environment variable.
- Client was Claude Code 2.1.311 (`claude --version`) with `--model
claude-haiku-5-5`.
- Exact command / steps:
1. Start the proxy with a scratch-only launcher (not committed). It
calls `register_turn_hook()` with a hook whose `on_request` appends
`ctx.provider`, `ctx.model`, `ctx.conversation` and `len(ctx.messages)`
to a JSONL file, then calls `run_server(ProxyConfig(host="127.0.0.1",
port=9100))`. Run it with `PYTHONSAFEPATH=1 HEADROOM_TELEMETRY=off
LITELLM_LOCAL_MODEL_COST_MAP=True python run_proxy.py hooks.jsonl`, from
a directory outside the repo.
2. Point Claude Code at the proxy: `export
ANTHROPIC_BASE_URL=http://127.0.0.1:9100 CLAUDE_CONFIG_DIR=<fresh empty
dir>`.
3. Start two sessions with the same prompt: `claude -p "say hi" --model
claude-haiku-5-5 --output-format json`, run twice. Note each
`session_id`.
4. Add two more turns to session 1: `claude -p "now say bye" --resume
<session 1 id> --model claude-haiku-5-5`, then `claude -p "one more word
please" --resume <session 1 id> --model claude-haiku-5-5`.
5. Read `hooks.jsonl`, and compare each key with sha256(`claude-code\0`
+ session_id).
- Observed result (from `hooks.jsonl`, keys truncated):

  | Session | Turn | Messages | `ctx.conversation` |
  | --- | --- | --- | --- |
  | `305d64a9…` | first | 2 | `306306eeb9bc164b…` |
  | `4e50609f…` (same prompt) | first | 2 | `4a9e51bc7f8005f2…` |
  | `305d64a9…` | first `--resume` | 4 | `306306eeb9bc164b…` |
  | `305d64a9…` | second `--resume` | 6 | `306306eeb9bc164b…` |

Each key equals sha256(`claude-code\0<session id>`) of the session id
that Claude Code reported. All three runs answered normally, and the
proxy log had no errors.
- Not tested:
- Codex over Responses HTTP or WebSocket (unit and handler tests cover
these).
  - Chat Completions with `X-Headroom-Session-Id`.
  - The `/v1/compress` gateway path.
- The server-side `conversation` body form, live (covered by the handler
test).
  - The `X-Headroom-Session-Id` fallback on `/v1/messages`.
  - Claude Code `/compact` and subagents.
  - Bedrock and Vertex upstreams.
  - Linux and Windows.
- A real plugin consuming the key. That is wired in
headroomlabs-ai/headroom-condense#7; skill-search and tool-search
adoption comes next.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: no request or response bytes change.
Registered turn hooks receive one new optional field; hooks that do not
read it behave as before.
- Kill switch / disable path: none needed; a hook that should not use
the key ignores it (headroom-condense falls back to its previous key
when the field is `None` or absent).
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR. Plugins read the field with `getattr`,
so they keep working on cores without it.

Cache note: compare keys only within one endpoint, since the same header
gives different keys on `/v1/messages` and `/v1/responses`. Each key
comes from a client id that stays the same for the whole conversation
(Claude Code session ids survive retries, `--resume` and `/compact`;
Codex `thread_id` wins over `session_id`, so guardian and subagent
requests do not collide), and the Responses key is taken before
compression rewrites the body, so a pinning hook keeps the same pin on
every turn.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A (proxy-side change).

## Additional Notes

- "No new warnings" is unchecked because the run was not compared
warning by warning against `main`.
- Follow-ups: skill-search and tool-search switch to
`conversation_for(ctx)` once headroom-condense#7 is released.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


## Maintainer post-merge qualification
Published `819f82117a4737de621528e9c49aee6aeeb4a793` includes upstream
main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. 28
conversation/Responses/per-frame-memory/disconnect tests pass;
Ruff/format/diff pass. Clean main merge; hook conversation policy
unchanged, no new code finding. Author explicitly reserves unchecked
self-review/readiness attestations; existing hold remains. Contributor
correctness logic was not rewritten. Existing Changes Requested decision
retained without duplicate findings. Scoped Windows validation uses the
existing development native extension, not a fresh build. Hosted
qualification is handled by maintainers; nothing merged.


### Maintainer current-main requalification

Exact head `965ff21ffbea9b446e5e39269812bb0d8c43c108`, upstream main
`ecbaf86d78cf52d185fbffa306157b9a4cc1b906`.

Mechanically updated from upstream/main
ecbaf86d78 with no conflicts and no
contributor conversation/ownership policy changes. Re-reviewed current
discussion and the integration delta. All 38 focused
conversation-context, Responses, per-frame WebSocket memory, disconnect
and Gemini tests pass; Ruff0.17.3 lint/format and whitespace checks
pass. Existing runtime evidence remains scoped; local tests use the
existing native extension, not a fresh build.

No new source blocker was found. The existing readiness action remains:
the author has explicitly reserved and still not checked the self-review
and ready-for-human-review attestations. They are not asserted on the
author's behalf. Preserve the existing Changes Requested readiness hold
until the author completes that action. Maintainers handle exact-head
hosted checks; no contributor CI action requested. Nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `db69aa725d085d75b0881f0b2e1f2237480ee0c6`, upstream main
`8d3e9e5ca7cd645eb73084b848edbea5fc796e1a`.

Clean mechanical sync with upstream
main8d3e9e5ca7cd645eb73084b848edbea5fc796e1a. Re-reviewed
conversation-context integration and fresh author/Devin discussion
without changing contributor identity/ownership
policy.38conversation/Responses/per-frame-WebSocket/disconnect/Gemini
tests pass;10changed Python files pass Ruff0.17.3check/format and
whitespace checks. Existing runtime evidence remains scoped; copied
extension supports Python startup, not a fresh native build.

No new source blocker found. The current body still leaves author
self-review and ready-for-human-review attestations unchecked and
reserves those actions for the author. Preserve that existing readiness
hold; no boxes were ticked or attested on their behalf. This is not an
inherited-main CI defect. Maintainers handle exact-head hosted checks;
nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `7411de5607beabfcbbd63b134795b0053da0e681`, upstream main
`7592e681459f14a2041eb5f524a658cb6ad1dd92`.

Clean current-main
sync7592e681459f14a2041eb5f524a658cb6ad1dd92,including
latestmemoryfixes. Conversationidentity/ownership
policyunchanged.46conversation/Responses/perframeWS/disconnect/memory-regression/Gemini
tests pass;whitespace passes. PriorRuff0.17.3check/format applies to
unchangedPRfiles.
Freshbodystillreservesuncheckedself-review/ready-for-humanreviewboxesfortheauthor.

No
newsourceblockerfound;retainexistingauthor-readinessholdwithouttickingboxesonbehalf.
Existingruntimeevidence scoped,no freshnativebuildclaim.
Maintainershandlechecks;nothingmerged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `cdf48ae9561dd2075625a4085594ac429a421231`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Clean integration with upstream/main
2d3ee51846 after the external 0.43.1
release merge. Exact headroom/crates/tests comparison against previously
reviewed 7411de560 shows no changes. Re-reviewed generated metadata; 78
release/version/MCP tests pass on this tree and whitespace passes.
Earlier 46 conversation/Responses/WebSocket/disconnect/memory/Gemini
passes remain scoped to unchanged inputs.

No new source blocker found in reviewed scope. Existing author-owned
readiness declaration remains the hold; fresh body must be checked
before publication without checking author attestations on their behalf.
No contributor identity/ownership policy rewritten, no fresh
native-build/provider claim. Maintainers handle checks; nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `d343c187f73b6a9005f0268e280ec657f435f1df`, upstream main
`ea59ac3eef3a3dc02076e9fe4bb30d572ca9c1a9`.

Clean integration with upstream/main
ea59ac3eef after the Click/macOS setup
merge. Re-reviewed the OpenAI response replay transport-header overlap;
conversation and existing tool-name map parameters remain additive.14
conversation/context-compaction tests and3 targeted route/gateway cases
pass (71 unrelated cases deselected); Ruff and whitespace pass. No new
source blocker found. Fresh body still leaves author-owned
self-review/readiness attestations unchecked; preserve the existing hold
without attesting on their behalf. Existing wider runtime evidence
remains scoped; no fresh native-build or live-provider claim.
Maintainers handle hosted checks; nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `741582654b9d0817ca8318af6aaf2dd000d1ec81`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Final post-Dependabot requalification against upstream/main
c01cf07d94. All six Dependabot updates are
now merged. Relative to this branch's freshly tested prepared tree, the
final main sync changes only the three intended Cargo.lock records;
Python/source/test inputs are unchanged. Re-reviewed the integration and
confirmed complete native source/lock/toolchain objects match the tested
combined build and exact fresh native DLL used for this branch's local
tests.

Focused fresh-native qualification: 18 passed, 70 deselected, 1 warning
in 20.19s. Selected real dependency overlays and native DLL paths are
recorded in final4496-fresh-native-tests.log; the DLL provenance/input
hashes are in final-native-artifact-proof.json. No redundant rerun is
implied after this Cargo-only merge. Whitespace passes. Prior source
review and scoped real-behavior evidence remain applicable to unchanged
contributor inputs.

No new source blocker found. Fresh body still leaves author self-review
and ready-for-human-review attestations unchecked. Retain the
author-owned readiness hold without checking those boxes on their
behalf.

Maintainers handle hosted checks; no external-provider claim,
contributor semantic rewrite or non-Dependabot merge.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `cb5c34b6221880a811dd94d21c5e3f3e874ee565`, upstream main
`da6d9736cd82074988c7d0a2360b49d9d8b6c0a0`.

Current-main requalification against
da6d9736cd. All69 integration patches
exactly equal c01cf07-to-current-main delta after diff-location
normalization; conversation identity/ownership policy unchanged.88
conversation,Responses context-compaction,OpenAI route,gateway
request-contract tests pass;1 deprecation warning. Post-suite PyTorch
cache-stat logging reports closed-stream logging errors during process
exit; pytest exit0 and test results are retained without claiming full
runtime qualification.31 changed Python files pass Ruff lint/format;
offline lock and whitespace pass. Existing pr3982 native DLL verified
full input hashes/artifact checksum, no fresh build claimed.
Author/Devin discussion considered; previous substantive Devin findings
resolved. No new source blocker found. Author explicitly reserves
self-review/ready-for-human-review attestations and both remain
unchecked; retain existing readiness hold without attesting on their
behalf. Maintainers handle hosted checks; no contributor logic rewrite
or merge.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `951bdf5853aaf8757ba46253763c91b6601b5c6d`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration with upstream/main
0ddaa56c21. All 19 integration patches
exactly match the prior-reviewed-main da6d9736 to current-main delta
after diff-location normalization; conversation identity/ownership
policy is unchanged. Fresh conversation/turn-hook/Responses/gateway
request-contract, doctor and incoming Vertex/backend/CLI suite: 367
passed, 11 skipped, 1 warning. Fourteen changed Python files pass Ruff
lint/format; offline lock and whitespace checks pass.

Author discussion and Devin findings read: both historical Devin items
are marked resolved, including server-side conversation IDs. No new
source blocker found. Existing author-reserved
self-review/ready-for-human-review attestations remain a hold; this
maintainer does not attest on the author's behalf. Static source/test
scope and security inspection found no new workflow/dependency/setup
path; the inherited flagged workflow-validation job is excluded by this
exact PR's unchanged workflow paths. Conversation digests are not an
authentication boundary. Verified recorded native input-set/hashes and
artifact checksum before DLL reuse; no fresh build or new live-client
qualification claimed. Maintainers handle exact-head hosted checks; no
PR merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-09 14:56:34 -06:00
1d112cd36c fix(proxy): show turn-hook savings in /stats extension_savings, incl. Codex WS (#4233)
## Description

### The test we ran

A Daytona sandbox running Headroom Enterprise 0.4.3 (core 0.39.1), with
Codex CLI 0.162.1 and Claude Code connected to real GitHub, Context7 and
Fetch MCP servers, working through the workshop tasks.

### What we found

- The tool-search plugin's turn hook rewrote the Codex `/v1/responses`
input and saved about **8K tokens per request**. `run_request_hooks`
booked that saving through `ctx.record_savings(hook.savings_source,
tokens=...)`.
- After many such requests, `GET /stats` still showed
**`extension_savings: {}`**.

There were two causes:

1. **`extension_savings` has no writer in core.** The only thing that
fills it is `PrometheusMetrics.record_extension_savings`, and core never
calls it. Attribution rows from `ctx.record_savings` /
`record_scope_savings` go to `savings_by_source`, which `/stats` shows
as `savings.by_source`.
2. **The Codex WebSocket path drops attribution entirely.** Both WS
compression calls (first frame and later frames) called
`_compress_openai_responses_payload_in_executor` without `savings_tags`.
So `TurnContext.tags` was a throwaway `{}`, and the hook's row (along
with the built-in tool-search deferral row) never reached
`savings.by_source` either. Codex uses the WS transport by default. The
HTTP `/v1/responses` path already passed `savings_tags`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- **`/stats.extension_savings`** now reports measured savings per
source: the **realized** attribution rows, plus
`record_extension_savings` totals. If a source also calls
`record_extension_savings` (`headroom-hdr` reports through both APIs on
every request), it shows the larger of its two totals rather than their
sum, so hdr is not double-counted. Nothing ties an attribution row to a
`record_extension_savings` call, so a plugin that used the two APIs on
different turns would be under-reported, never over-reported.
Unrealized/estimated rows (tool-search deferral, `realized=False`) stay
out and remain visible in `savings.by_source`.
- **Codex WS:** each frame compresses into its own `savings_tags` dict.
Its rows are merged into the session's pending list on the event loop,
and only when that frame was actually rewritten and sent. Each per-turn
`RequestOutcome` carries the pending rows and then clears them. A turn
whose `response.completed` has no input usage leaves its rows pending,
the same gate as its headline `saved_delta`
(`_deferrable_savings_delta`). Rows no turn claimed go out with the
session-end residual outcome, and only if the residual has input usage.
A fresh dict per frame matters because a timed-out compression thread
keeps running (`_run_compression_in_executor` can't cancel it), and it
must not book rows for a frame that went out uncompressed.

**Realized vs estimated:** `savings.by_source` already keeps one row per
`(source, realized)`, so realized and unrealized savings are shown
separately. `extension_savings` now carries only the realized ones.

**Review fix (@JerrettDavis, bounded WS pending list):** the
session-wide pending list grew one row per rewritten frame, while
`savings_attribution.from_tags` read only the first `MAX_SOURCES` (32)
rows. After 33 frames that each booked a 100-token `tool_search` row,
the handoff cleared all 33 rows but booked only 3,200 of the 3,300
tokens. `savings_attribution.accumulate(rows, items)` replaces the
`extend` at both WS frame paths.

**Bounded rows, nothing dropped (follow-up to @JerrettDavis's two
reviews):** the 32-row cap that silently dropped rows is gone from
`record_savings`, `from_tags` and `accumulate`. Storage is bounded by
coalescing plus one overflow row:
- every writer goes through the one `accumulate` helper: the per-request
ledger (`record_savings`, so every handler) and the WS pending list.
Rows with the same `(source, realized, estimated)` key are summed into
one row (`tokens`, `usd`, `events`). `details` are kept only while every
summed row has the same `details`.
- **cardinality bound:** a ledger keeps at most `MAX_NAMED_ROWS` (256)
named rows. A new source past that bound is summed into an `other` row
with the same `(realized, estimated)`, and its `details` are dropped.
Tokens, dollars and events are conserved: nothing is dropped, only the
source name. `tool_search` never folds, because
`tool_schema_savings_policy` finds the deferral's row by name to lower
it when hooks un-defer tools.
- **finite sums:** a row's `usd` saturates at ±`MAX_ROW_USD` (1e15) and
its `tokens` stay within 0..`MAX_ROW_TOKENS` (10^15). A row that was
clipped, or summed from one that was, carries `saturated: true`. Every
total stays finite, so the Prometheus exposition and
`json.dumps(allow_nan=False)` never see `inf`/`nan`.
- each row carries `events`: how many recordings it stands for (1 on a
fresh row, summed on merge). `PrometheusMetrics.record_request`
(`headroom_savings_attribution_events_total`), the OTel
`headroom.savings.attribution.events` counter and the perf analyzer's
"Savings by Source" add that count instead of 1 per row, so 33 coalesced
frames are 33 events, not 1 (Devin's finding). A row without `events`
(older PERF logs, rows built by hand) counts as 1.
- a row that a reader edited in place (`reconcile_deferred_tokens`, the
tool-search plugin) is summed defensively, so a junk value counts as 0
instead of raising inside the handler. The core deferral and the native
deferral hook both book `tool_search` as estimated, so they now share
one row, and the reconcile debits that row; totals are unchanged.
- `MAX_SOURCES` is no longer used. No importer was found outside this
repo's own test, but `savings_attribution` is imported by plugins, so
the name is kept behind a module `__getattr__` that raises a
`DeprecationWarning`.
- **Scope of the bound:** it is per ledger (one request, or one WS
session's pending list). The process-wide
`PrometheusMetrics.savings_by_source` table has no source cap yet; that
is a separate PR.

### Before / after

| | origin/main | this PR |
|---|---|---|
| A turn hook books 8,000 tokens on 2 requests →
`/stats.extension_savings` | `{}` | `{"tool_search": 16000}` |
| An hdr-style plugin calls both APIs (300 tokens × 2) | `{"hdr": 600}`
| `{"hdr": 600}` (not 1200) |
| A Codex WS turn with a folding hook → attribution rows on the turn
outcome | `[]` | `[("ws_fold", realized=True)]` |
| 33 WS frames, each booking `tool_search` 100 tokens, then one billed
turn → booked `tool_search` tokens (helper-level, the reviewer's
scenario) | 3,200 of 3,300, 33 rows stored | 3,300, 1 row stored |
| Real WS handler, 33 frames each with a folding hook (`ws_fold` row)
plus a 100-token hook row → non-`ws_fold` tokens booked | 1,600 of 3,300
(66 rows stored) | 3,300 |
| Same, repeated source → rows stored / hook events in
`headroom_savings_attribution_events_total` | aecae31b: 2 rows / 1 event
| 2 rows / 33 events |
| Same, 33 distinct sources → rows stored / hook events / `other` row |
aecae31b: 29 rows / 28 events / yes (6 sources folded) | 34 rows (33 +
`ws_fold`) / 33 events / no |
| Same, 300 distinct sources → rows stored / tokens and USD booked |
rows past 32 dropped | 257 rows (256 named + `other`), every token and
dollar booked |
| One ledger, 2,000 distinct sources plus repeats | rows past 32 dropped
| 256 named rows + 2 `other` rows (realized, estimated); tokens, USD and
events conserved |
| Same source booked 1e308 USD twice, then 1.0 | `inf` in `/stats` and
Prometheus | `usd` = 1e15, `saturated: true`, exposition finite |
| One HTTP request, 40 distinct sources + 1 repeat, 100 tokens each
(`record_savings`, every handler) → booked tokens / events | 3,200 of
4,100 / 32 (8 rows dropped) | 4,100 / 41, 40 rows stored |

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

-
`tests/test_compression_observability.py::test_turn_hook_savings_surface_in_stats_extension_savings`:
fails on origin/main, passes here.
-
`tests/test_openai_codex_ws_lifecycle.py::test_ws_turn_hook_savings_reach_the_turn_outcome[billed|usage_less]`:
drives the real WS handler with a registered hook. `billed` fails on
origin/main (no rows). `usage_less` fails if the input-usage gate is
removed.
- Related suites pass: the WS lifecycle, per-frame memory, savings
deferral and compression scheduler tests, extension attribution,
responses context compaction, tool-search repair after hooks, and the
handler outcome tag invariant (136 passed).
- ruff check, ruff format --check and mypy pass.
-
`tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_32_rows_are_all_booked`
(review fix, renamed from `..._beyond_max_sources_...`), parametrized
over a repeated source × 33 frames, 33 distinct sources, and 300
distinct sources (folds into `other`, 257 rows, USD conserved): drives
the real WS handler through the rewritten frames. The upstream holds its
events until all 33 frames arrive, so every row is pending when the one
billed outcome claims it. The rows are booked through the real
`from_tags` → `PrometheusMetrics.record_request`. The test asserts that
every token is booked (3,300), that the hook rows count 33 events, that
there is no `other` row, and that the stored row count equals the
distinct keys (2 for a repeated source, 34 for 33 distinct sources plus
`ws_fold`).
-
`tests/test_extension_attribution.py::test_a_request_with_40_sources_books_every_token`:
one request's ledger with 40 distinct sources and one repeat books 4,100
tokens and 41 events through `PrometheusMetrics.record_request`.
-
`tests/test_extension_attribution.py::test_rows_coalesce_per_source_realized_and_estimated`:
same source with different `realized`/`estimated` stays in separate
rows; same key sums tokens and events.
-
`tests/test_observability_metrics.py::test_coalesced_savings_rows_count_every_event`:
the OTel events counter adds a row's `events` (33), and a row without
the field counts as 1.
-
`tests/test_cli_perf_format.py::test_coalesced_savings_rows_count_every_event_in_text_and_summary`:
both perf folds (`headroom perf` text report and the JSON `by_source`
summary) add a row's `events`.
-
`tests/test_extension_attribution.py::test_merging_into_a_junk_row_does_not_raise_inside_the_handler`:
a ledger row edited in place with junk (`tokens=None`) is summed as 0
instead of raising out of `record_savings`.
-
`tests/test_extension_attribution.py::test_distinct_sources_past_the_bound_fold_into_other_and_conserve_totals`:
2,000 sources plus estimated repeats give 258 rows, with tokens, USD and
events conserved.
-
`tests/test_extension_attribution.py::test_same_source_usd_saturates_instead_of_overflowing`,
`test_an_infinite_total_already_in_the_ledger_stays_saturated[±1.0]`,
`test_tokens_stay_within_zero_and_the_bound`: sums stay finite and are
flagged `saturated`.
-
`tests/test_extension_attribution.py::test_tool_search_keeps_its_own_row_past_the_bound`:
the deferral row is never folded, so the reconcile still finds it.
-
`tests/test_extension_attribution.py::test_max_sources_is_deprecated_not_removed`:
the old name still imports, with a `DeprecationWarning`.
-
`tests/test_turn_hooks.py::test_request_runner_attributes_savings_with_handler_counters`:
expected row now includes `events: 1`.

### Test Output

```text
# review fix, savings_attribution.py + openai.py reverted to the previous commit, new test:
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_max_sources_are_all_booked[repeated_source]   (assert 1600 == (100 * 33))
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_max_sources_are_all_booked[distinct_sources]  (assert 1600 == (100 * 33))
# helper-level, the reviewer's scenario (33 frames x tool_search 100):
append:     33 rows stored, 3200 booked
accumulate:  1 row stored,  3300 booked
40 distinct sources x 100: 29 rows stored, 4000 booked ("other" = 1200)
# no-cap follow-up, the 5 source files reverted to aecae31b, new/updated tests:
FAILED tests/test_extension_attribution.py::test_a_request_with_40_sources_books_every_token
FAILED tests/test_extension_attribution.py::test_rows_coalesce_per_source_realized_and_estimated
FAILED tests/test_observability_metrics.py::test_coalesced_savings_rows_count_every_event
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_32_rows_are_all_booked[repeated_source]
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_32_rows_are_all_booked[distinct_sources]
# with the no-cap follow-up: every test file that touches the savings ledger (21 files)
# plus gateway, perf format/per-port logs and /stats suites: 831 passed, 5 skipped (90d5e132)
# with the fix (aecae31b):
WS lifecycle, per-frame memory, savings deferral, compression scheduler, WS timings, pyo3 WS compression,
outcome tag invariant, extension attribution, compression observability, tool-search aggregation,
responses context compaction, proxy openai: 189 passed
ruff 0.17.0 check + format --check: clean on changed files
mypy headroom --ignore-missing-imports: no new errors (1 pre-existing local error in integrations/langchain/chat_model.py:380, untouched)
```

## Real Behavior Proof

- Environment: macOS 26.7, Python 3.12.13, this branch. The WS handler
is driven through the repo's fake-upstream harness
(`tests/test_openai_codex_ws_lifecycle.py`), which runs the real
`handle_openai_responses_ws`, `run_request_hooks` and outcome funnel
with only the upstream socket scripted; `/stats` runs on the real app
via `TestClient`; OTel uses the real `HeadroomOtelMetrics` with an
in-memory reader.
- Exact command / steps: `pytest tests/test_openai_codex_ws_lifecycle.py
tests/test_extension_attribution.py tests/test_observability_metrics.py
tests/test_compression_observability.py tests/test_turn_hooks.py` (plus
the other ledger suites listed under Testing), each run once on this
branch and once with the source files reverted.
- Observed result: with a registered folding hook, the hook really
rewrites the forwarded `input` (the upstream received only `"new"`) and
the turn outcome carries `[("ws_fold", realized=True, tokens>0)]`.
`/stats.extension_savings` reports `{"tool_search": 16000, "hdr": 600}`
for the two-request scenario above. 33 rewritten WS frames book 3,300
hook tokens and 33 events with no `other` row; 300 distinct sources fold
into 256 named rows plus `other` with every token and dollar booked; one
request with 40 distinct sources plus a repeat books 4,100 tokens and 41
events in 40 rows.
- Not tested: a real Codex CLI session against OpenAI on this branch, a
real Codex session with more than 32 rewritten frames before a billed
turn, and the real `headroom-tool-search` plugin were not re-run after
the patch. The sandbox numbers above (about 8K tokens/request,
`extension_savings: {}`) are from 0.39.1 before the fix.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: `/stats.extension_savings` now
includes realized attribution rows, and Codex WS turns now carry
attribution rows. Request bodies and headline totals are unchanged.
- Kill switch / disable path: none needed; the change only affects
reporting
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- plus the code-reviewer agent on each review fix. Its earlier SHOULD on
the collapsed `events` count is now fixed (rows carry `events`).
- [x] This PR is ready for human review

## Additional Notes

- `record_extension_savings` stays public and unchanged.
- The Prometheus exposition format is unchanged;
`headroom_savings_attribution_events_total` now counts recordings, which
is what it counted before rows were coalesced.
- Ledger rows now carry an `events` field. Readers that ignore unknown
keys are unaffected.
- A reader that finds "rows a hook added" by slicing the ledger at its
old length (`ledger[before:]`) can miss a hook's tokens when the hook
books a key that already had a row. Core has no such reader;
`headroom-kiro`'s handler does and should switch to a before/after sum.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-17 11:29:04 -07:00
4d27d024b0 fix(proxy): share one compression deadline across a Responses request (#3938)
## Description

#3693 made `HEADROOM_COMPRESSION_DEADLINE_MS` bound a request instead of
a block, but only inside `ContentRouter.apply()`. OpenAI Responses
requests (Codex HTTP, the Codex WebSocket path, the Responses
passthrough) never call `apply()`: the adapter compresses each text unit
and each small-unit batch with its own `ContentRouter.compress()` call,
several at a time on `_openai_responses_unit_executor()`. No deadline
origin exists on that path, so every Kompress call starts a fresh 20s
clock. A request with many tool outputs runs far past its compression
timeout; the awaiter gives up and forwards it uncompressed while the
worker keeps running inference, and that timeout debt quarantines
compression for every other request until it finishes.

This PR reuses #3693's mechanism on the Responses path: one deadline
origin per request, bound as the router's per-request
`kompress_deadline_started_at`, for every unit and batch, including the
ones on the unit pool.

Closes #3937

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `_compress_openai_responses_payload_in_executor` stamps one origin per
request (`_openai_responses_deadline_started_at`) before the job is
queued, because the caller's `timeout` counts queue wait too, and passes
it down through `_compress_openai_responses_payload` and
`_compress_openai_responses_live_text_units_with_router`.
`deadline_started_at` joins the existing list of optional keywords the
TypeError retry drops, so a narrow compressor override keeps working.
- The budget is `min(HEADROOM_COMPRESSION_DEADLINE_MS, 0.75 * timeout)`.
With the defaults (20s, 30s) that is the 20s deadline unchanged. On the
Codex WebSocket path, which awaits at most 5s, it is 3.75s; before, each
Kompress call allowed itself 20s there. Kompress checks the deadline
only at chunk boundaries, so the chunk in flight at expiry, the splice
and the output serialization need the remaining 25%. Kompress measures
`now - origin > deadline`, so the cap is expressed as a backdated origin
and Kompress itself is unchanged. `HEADROOM_COMPRESSION_DEADLINE_MS=0`
still disables the deadline (no origin).
- `ContentRouter.share_request_deadline(started_at)` binds a fresh
`_PerRequestRuntimeState(kompress_deadline_started_at=started_at)` in
the current Context, the same state `apply()` binds, so
`_try_ml_compressor` forwards the origin to Kompress unchanged. It
returns `False` and binds nothing once the deadline has passed.
- `compress_unit_with_router` and `compress_batch_with_router` take an
optional `deadline_started_at` and call it before entering the router. A
unit or batch that has not started by the deadline comes back unchanged
with reason `deadline_exceeded` (new unit reason category `deadline`),
without entering the router. Default `None` keeps every other caller as
before.
- The two unit-pool `submit` calls run each task in
`contextvars.copy_context()`. Each task binds fresh state (a shared
state object would race on the `_runtime_target_ratio` set/restore in
`compress_unit_with_router`), and the copy keeps that binding from
outliving the task on a long-lived pool thread.
- Finished units are kept, no thread is interrupted, and the unit result
cache and replay policy are untouched: the cache still stores exactly
what was forwarded, including a unit passed through because of the
deadline, so the next turn forwards the same bytes.
- Tests: `test_openai_responses_units_share_one_kompress_deadline`
(parallel fan-out, every Kompress call gets one non-`None` origin),
`test_openai_responses_units_after_the_deadline_pass_through_unchanged`
(finished unit kept, later units byte-identical, Kompress entered once),
`test_openai_responses_deadline_budget_stays_inside_the_timeout` (cap
and opt-out),
`test_batch_after_the_request_deadline_passes_through_without_routing`.
Three exact-kwargs assertions in `tests/test_proxy_openai.py` now also
ignore `deadline_started_at`, the same way they already ignore `timing`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Unit tests: targeted files only (the new tests plus the existing suites
for every module touched), not the whole suite. Lint and types: `ruff
check`, `ruff format --check` and `mypy` on the changed files, not the
whole tree.

### Test Output

New behavioural tests with only the `headroom/` changes reverted
(upstream `main` d5318ac2):

```text
$ python -m pytest "tests/test_openai_responses_compression_units.py::test_openai_responses_units_share_one_kompress_deadline" "tests/test_openai_responses_compression_units.py::test_openai_responses_units_after_the_deadline_pass_through_unchanged" -q -p no:cacheprovider
E   AssertionError: [None, None, None, None, None, None]
E   assert None not in [None, None, None, None, None, None]
E   assert 4 == 1
E    +  where 4 = len([None, None, None, None])
FAILED tests/test_openai_responses_compression_units.py::test_openai_responses_units_share_one_kompress_deadline
FAILED tests/test_openai_responses_compression_units.py::test_openai_responses_units_after_the_deadline_pass_through_unchanged
============================== 2 failed in 1.84s ===============================
```

With the fix:

```text
$ python -m pytest tests/test_openai_responses_compression_units.py tests/test_compression_batches.py tests/test_compression_units.py tests/test_transforms/test_kompress_deadline.py tests/test_content_router_concurrency.py tests/test_proxy_openai.py tests/test_openai_responses_additional_tools.py tests/test_codex_ws_compression_scheduler.py tests/test_codex_ws_per_frame_memory.py tests/test_openai_codex_ws_lifecycle.py tests/test_openai_codex_ws_timings.py tests/test_openai_codex_routing.py tests/test_responses_savings_denominator.py tests/test_compress_passthrough.py tests/test_openai_responses_read_protection.py tests/test_openai_responses_context_compaction.py tests/test_openai_responses_t3_replay_regression.py tests/test_compression_determinism.py tests/test_html_tool_reversibility.py tests/test_proxy_compression_executor.py tests/test_agent_savings.py -q -p no:cacheprovider
================== 334 passed, 2 skipped, 1 warning in 12.26s ==================

$ ruff check <7 changed files> && ruff format --check <7 changed files>   # ruff 0.16.3
All checks passed!
7 files already formatted

$ mypy headroom/proxy/handlers/openai.py headroom/transforms/compression_batches.py headroom/transforms/compression_units.py headroom/transforms/content_router.py
Success: no issues found in 4 source files
```

## Real Behavior Proof

- Environment: macOS 27.0.1 arm64, Python 3.11.13. Upstream `main`
d5318ac2 versus this branch, same venv.
- Exact command / steps: an offline probe (script below) builds the real
proxy with `create_app(ProxyConfig(...))` and calls
`HeadroomProxy._compress_openai_responses_payload_in_executor` with 60
Codex `function_call_output` items of 700 words. The path is all real:
`_run_compression_in_executor`, `_compress_openai_responses_payload`,
the unit fan-out on `_openai_responses_unit_executor` (parallelism 4),
`ContentRouter`, and the `KompressCompressor` chunk loop with its
execution slot and deadline checks. Only the ONNX model is replaced, by
a fake that sleeps per 350-word chunk, and the token counter is a word
counter. Network is blocked in-process. Times are scaled down 10x:
deadline 2s (default 20s); timeout 3s (HTTP default 30s) or 0.5s (Codex
WS 5s). After the request it waits for the worker to drain, then sends
an unrelated request. `PROBE_DEADLINE_MS=2000 PROBE_TIMEOUT=<t>
PROBE_DELAY_MS=<ms> PYTHONPATH=. python responses_budget_probe.py`
- Observed result: on main the request times out and the worker keeps
running inference for 6.5-10s, quarantining the next request; on this
branch the request returns inside its budget, nothing runs after it, and
nothing is quarantined (output below).

```text
=== main @ d5318ac2  PROBE_TIMEOUT=3.0 PROBE_DELAY_MS=100   (HTTP proportions)
request: TIMED OUT after 3002ms
inference calls: total=120, started after the 2000ms budget=101 (latest +10955.6ms), after the awaiter returned=93
worker busy until 13078ms (10075ms after the awaiter returned)
timeout debt: leaked_threads=1, quarantine_activations=1
next request right after a slow one (timed out): CompressionQuarantinedError: compression quarantined: 1 timed-out worker(s) still running

=== branch @ cfa775a1  PROBE_TIMEOUT=3.0 PROBE_DELAY_MS=100
request: completed after 2069ms
inference calls: total=19, started after the 2000ms budget=0, after the awaiter returned=0
worker busy until 2069ms (0ms after the awaiter returned)
units compressed=10/60, passed through byte-identical=50, tokens_saved=3175
timeout debt: leaked_threads=0, quarantine_activations=0
next request right after a slow one (completed): ran, not quarantined

=== main @ d5318ac2  PROBE_TIMEOUT=0.5 PROBE_DELAY_MS=50   (Codex WS proportions)
request: TIMED OUT after 501ms
inference calls: total=120, started after the 375ms budget=113 (latest +6570.6ms), after the awaiter returned=112
worker busy until 7019ms (6518ms after the awaiter returned)
timeout debt: leaked_threads=1, quarantine_activations=1
next request right after a slow one (timed out): CompressionQuarantinedError: compression quarantined: 1 timed-out worker(s) still running

=== branch @ cfa775a1  PROBE_TIMEOUT=0.5 PROBE_DELAY_MS=50
request: completed after 415ms
inference calls: total=7, started after the 375ms budget=0, after the awaiter returned=0
worker busy until 415ms (0ms after the awaiter returned)
units compressed=4/60, passed through byte-identical=56, tokens_saved=1165
timeout debt: leaked_threads=0, quarantine_activations=0
next request right after a slow one (completed): ran, not quarantined
```

On main the request is forwarded uncompressed at the timeout, the worker
then runs another 93 to 112 inference calls for 6.5 to 10s, and the next
request is quarantined. On the branch the request returns inside its
timeout with the units it finished, nothing runs after it returns, and
nothing is quarantined. The budget column is measured from just before
the call, so it slightly precedes the handler's own origin; in one of
three extra WS-proportion runs a single call started 9.4ms after that
clock's expiry. That call had passed Kompress's own pre-acquire deadline
check (#3693) just before. With 100ms chunks at the 0.5s timeout (a 1s
chunk at production scale, the worst case for the 25% margin) one branch
run still timed out, at 511ms; the worker had finished at that same
moment, so no debt was recorded and nothing was quarantined.

- Not tested: no live Codex or OpenAI traffic was measured, and no real
Kompress/ONNX model was run (none is downloaded on this machine and the
probe stays offline). `headroom proxy` was not booted as a process; the
probe drives the same handler method the HTTP and WebSocket routes call,
on an app from `create_app`. `RemoteKompressCompressor` does not take
`_deadline_started_at` (unchanged from #3693): with a remote endpoint
the deadline still stops new units, but not a unit already in flight.
Linux and Windows were not run.

<details>
<summary>responses_budget_probe.py</summary>

```python
"""Offline probe: does one Responses request stay inside its compression budget?

Drives the real handler path (create_app -> HeadroomProxy ->
_compress_openai_responses_payload_in_executor -> _compress_openai_responses_payload
-> unit fan-out -> real ContentRouter -> real KompressCompressor chunk loop).
Only the ONNX model is replaced, by a fake whose inference sleeps DELAY seconds
per 350-word chunk. No network, no model download, no provider call.

PROBE_DEADLINE_MS, PROBE_TIMEOUT (seconds) and PROBE_DELAY_MS (per chunk) scale
the production defaults down (deadline 20s, HTTP timeout 30s, Codex WS timeout 5s).
"""

import asyncio
import os
import socket
import sys
import threading
import time
from types import SimpleNamespace


def deny_network(*args, **kwargs):
    raise AssertionError("probe must stay offline")


socket.socket.connect = socket.socket.connect_ex = deny_network
socket.create_connection = socket.getaddrinfo = deny_network
os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
os.environ["HEADROOM_COMPRESSION_DEADLINE_MS"] = os.environ.get("PROBE_DEADLINE_MS", "400")
os.environ.setdefault("HEADROOM_TOOL_OUTPUT_COMPRESSION_PARALLELISM", "4")

from headroom.proxy.server import ProxyConfig, create_app  # noqa: E402
from headroom.transforms import kompress_compressor as kc  # noqa: E402
from headroom.transforms.content_router import ContentRouter  # noqa: E402

DELAY = float(os.environ.get("PROBE_DELAY_MS", "20")) / 1000
TIMEOUT = float(os.environ.get("PROBE_TIMEOUT", "0.6"))
UNITS = 60
WORDS = 700  # two 350-word Kompress chunks per unit

calls: list[tuple[float, float]] = []  # (start, end) perf_counter of each inference
calls_lock = threading.Lock()


class Enc(dict):
    def word_ids(self, batch_index=0):
        return self["_word_ids"]


class Tok:
    def __call__(self, chunk_words, **kw):
        n = len(chunk_words)
        return Enc(input_ids=[[0] * n], attention_mask=[[1] * n], _word_ids=list(range(n)))


class Model:
    def _infer(self, input_ids):
        started = time.perf_counter()
        time.sleep(DELAY)
        with calls_lock:
            calls.append((started, time.perf_counter()))
        return len(input_ids[0])

    def get_keep_mask(self, input_ids, attention_mask):
        n = self._infer(input_ids)
        return [[i < n // 2 for i in range(n)]]

    def get_scores(self, input_ids, attention_mask):
        n = self._infer(input_ids)
        return [[1.0 - i / n for i in range(n)]]


class TokenCounter:
    def count_text(self, text):
        return len(text.split())


def make_proxy():
    proxy = create_app(
        ProxyConfig(
            optimize=False,
            cache_enabled=False,
            rate_limit_enabled=False,
            cost_tracking_enabled=False,
            log_requests=False,
            ccr_inject_tool=False,
            ccr_handle_responses=False,
            ccr_context_tracking=False,
            image_optimize=False,
            compression_max_workers=4,
        )
    ).state.proxy
    compressor = kc.KompressCompressor()
    kc._kompress_cache[compressor.config.model_id] = (Model(), Tok(), "onnx")
    router = ContentRouter()
    router._get_kompress = lambda: compressor
    proxy.openai_pipeline = SimpleNamespace(transforms=[router])
    proxy.openai_provider = SimpleNamespace(get_token_counter=lambda _model: TokenCounter())
    return proxy


def payload(salt):
    return {
        "model": "gpt-5",
        "input": [
            {
                "type": "function_call_output",
                "call_id": f"call_{i}",
                "output": "\n".join(
                    " ".join(f"{salt}{i}line{line}word{j}" for j in range(10))
                    for line in range(WORDS // 10)
                ),
            }
            for i in range(UNITS)
        ],
    }


async def warm_up(proxy):
    # First-use imports and detector setup must not count against the request.
    small = {"model": "gpt-5", "input": payload("warm")["input"][:1]}
    await proxy._compress_openai_responses_payload_in_executor(
        small, model="gpt-5", request_id="warm-up", timeout=30
    )
    calls.clear()


async def main():
    proxy = make_proxy()
    await warm_up(proxy)
    deadline_s = float(os.environ["HEADROOM_COMPRESSION_DEADLINE_MS"]) / 1000
    budget_s = min(deadline_s, 0.75 * TIMEOUT)
    source = payload("a")
    started = time.perf_counter()
    try:
        result = await proxy._compress_openai_responses_payload_in_executor(
            source, model="gpt-5", request_id="probe-1", timeout=TIMEOUT
        )
        outcome = "completed"
    except asyncio.TimeoutError:
        result = None
        outcome = "TIMED OUT"
    returned = time.perf_counter()
    with calls_lock:
        at_return = len(calls)
    # Is the worker still doing inference after the awaiter gave up / returned?
    while proxy._compression_in_flight:
        await asyncio.sleep(0.01)
    worker_done = time.perf_counter()
    with calls_lock:
        snapshot = list(calls)
    expiry = started + budget_s
    print(f"deadline={deadline_s * 1000:.0f}ms outer_timeout={TIMEOUT * 1000:.0f}ms "
          f"units={UNITS} chunks/unit={WORDS // 350} inference={DELAY * 1000:.0f}ms/chunk")
    print(f"request: {outcome} after {(returned - started) * 1000:.0f}ms")
    late = [s - expiry for s, _ in snapshot if s > expiry]
    print(f"inference calls: total={len(snapshot)}, "
          f"started after the {budget_s * 1000:.0f}ms budget={len(late)}"
          + (f" (latest +{max(late) * 1000:.1f}ms)" if late else "")
          + f", after the awaiter returned={len(snapshot) - at_return}")
    print(f"worker busy until {(worker_done - started) * 1000:.0f}ms "
          f"({(worker_done - returned) * 1000:.0f}ms after the awaiter returned)")
    if result is not None:
        changed = sum(a != b for a, b in zip(source["input"], result[0]["input"]))
        print(f"units compressed={changed}/{UNITS}, passed through byte-identical={UNITS - changed}, "
              f"tokens_saved={result[2]}")
        if os.environ.get("PROBE_TIMINGS"):
            for name, ms in sorted(result[8].items(), key=lambda kv: -kv[1])[:8]:
                print(f"  timing {name}={ms:.1f}ms")
    print(f"timeout debt: leaked_threads={proxy._compression_leaked_threads}, "
          f"quarantine_activations={proxy._compression_quarantine_activations}")

    # An unrelated request that arrives while the first one's worker is running.
    proxy2 = make_proxy()
    await warm_up(proxy2)
    first = asyncio.ensure_future(
        proxy2._compress_openai_responses_payload_in_executor(
            payload("b"), model="gpt-5", request_id="probe-2", timeout=TIMEOUT
        )
    )
    try:
        await first
        first_outcome = "completed"
    except asyncio.TimeoutError:
        first_outcome = "timed out"
    small = {"model": "gpt-5", "input": payload("c")["input"][:1]}
    try:
        await proxy2._compress_openai_responses_payload_in_executor(
            small, model="gpt-5", request_id="probe-3", timeout=TIMEOUT
        )
        follow = "ran, not quarantined"
    except asyncio.TimeoutError as exc:
        follow = f"{type(exc).__name__}: {exc}"
    print(f"next request right after a slow one ({first_outcome}): {follow}")
    while proxy2._compression_in_flight:
        await asyncio.sleep(0.01)
    for p in (proxy, proxy2):
        p._compression_executor.shutdown(wait=True)


if __name__ == "__main__":
    print(f"python {sys.version.split()[0]}, headroom from {os.path.dirname(kc.__file__)}")
    asyncio.run(main())
```

</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, on the OpenAI Responses
compression path only. A request whose Kompress work exceeds
`min(HEADROOM_COMPRESSION_DEADLINE_MS, 0.75 * timeout)` in total now
stops starting new inference and forwards the units it had not started
unchanged. Before, each unit restarted the deadline, so such a request
either finished inside the timeout with more units compressed (fan-outs
that took between the budget and the timeout), or timed out, was
forwarded with no compression at all, and quarantined compression for
every request behind it. Codex WS requests now budget 3.75s of the 5s
they are awaited for, instead of 20s per Kompress call. Requests that
finish inside the budget are unchanged, byte for byte. The chat and
Anthropic paths (`apply()`) are not touched.
- Kill switch / disable path: `HEADROOM_COMPRESSION_DEADLINE_MS=0`
disables the deadline (the same switch #3693 has), and a higher value or
`HEADROOM_COMPRESSION_TIMEOUT_SECONDS` widens the budget. No new
environment variable.
- Unsafe override required: No.
- Qualification impact: Responses compression only, and only on requests
whose fan-out outruns the budget. Those that used to finish between the
budget and the timeout now compress fewer units (the rest pass through
unchanged); those that used to time out and go out with no compression
now keep the units finished in time, and no longer quarantine the
requests behind them. Requests inside the budget are byte-identical. No
provider cache or replay change.
- Rollback path: Revert this commit; Responses units go back to a fresh
deadline per Kompress call.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: no docs page describes
`HEADROOM_COMPRESSION_DEADLINE_MS` per path, so nothing to update.
- Why 0.75 and not #3721's 0.5: #3721 (open) caps cumulative ML time
inside `apply()` at half the outer timeout; this caps wall time from
enqueue on the Responses path, which already includes the non-ML work.
0.75 leaves the default HTTP budget at the existing 20s deadline and
only binds when the timeout is short (Codex WS) or lowered by the
operator. The probe above shows the remaining 25% covering the chunk in
flight plus splice and serialization. #3721 arms only `apply()` and
leaves direct `compress()` callers unbounded, so the two do not overlap;
if it merges, the Responses path still gets its bound from this change.
- Related: #2360 (Codex WS worker hang -> permanent quarantine; the
quarantine time cap from #2412 limits how long one leaked worker can
block, this removes the leak on this path), #3653 (a thread/fd per
quarantine event; fewer quarantine events here), #3711 (the chat-path
counterpart).
- Headroom Desktop has shipped an equivalent downstream patch since
2026-09-07. This PR re-implements it on the existing #3693 state instead
of a separate budget.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3938"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai>
2026-10-05 08:25:17 -05:00
Tejas ChopraandTejas Chopra 3145242645 Unify savings attribution across stats, perf, metrics, and dashboard (#2976)
## Summary

Adds a small provider-neutral savings attribution seam. Named sources
can attach realized or projected token/USD deltas to a request without
changing headline arithmetic or introducing private-package inventory
into OSS.

Also fixes the Anthropic buffered lifecycle so normal successful
responses run response hooks, applies stream-safety filtering, includes
tool savings in per-model perf totals, and surfaces the same breakdown
in request logs, `/stats`, `headroom perf`, Prometheus, OTEL, and the
dashboard.

## Why

Request-local savings were split between canonical token deltas,
process-global extension counters, and tool-only tags. This made correct
headline totals possible while losing attribution in perf, recent
requests, metrics, and the dashboard. Normal Anthropic responses also
skipped response hooks unless CCR ran.

## Validation

- 74 focused tests passed: turn hooks, OpenAI hook lifecycle, outcome
funnel, perf formats, and tool-search repair
- Ruff passes on all changed Python files
- Existing compression-observability suite: 11 passed; 2 tokenizer-cache
tests require network access to fetch the tiktoken vocabulary

## Compatibility

No named private packages or private inventory are encoded in OSS.
Existing hooks remain source-compatible because all new TurnContext
fields are optional.

---------

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
2026-08-13 17:13:23 -07:00
Rod BoevandJD Davis 54ea28d983 fix(openai): skip Responses tool-search deferral for clients that cannot execute it (#2696)
## Description

OpenCode rejects the proxy-injected Responses `tool_search` tool because
it resolves tool calls against its local registry. This PR now uses the
shared client policy from current `main` and leaves OpenCode tools
resident, alongside the existing Codex exclusion. Other clients retain
tool-search deferral.

Closes #2660.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] Documentation update

## Changes Made

- Add `opencode` to the shared exact-match unsupported-client set in
`headroom.proxy.helpers`.
- Carry the already-classified `client` through native HTTP, WebSocket,
and custom-base Responses paths.
- Preserve `main`'s compatibility loop, which retries only exact
unsupported `client` or `timing` keyword errors and re-raises internal
`TypeError`s.
- Add focused helper, compressor, HTTP, passthrough, and WebSocket
coverage.

## Testing

- [x] Unit tests pass
- [x] Ruff check and format pass
- [x] New tests added
- [ ] Live OpenCode session tested

```text
uv run --extra dev pytest tests/test_openai_tool_search_deferral.py tests/test_proxy_openai.py -q
57 passed

uv run --extra dev ruff check headroom/proxy/handlers/openai.py headroom/proxy/helpers.py tests/test_openai_tool_search_deferral.py tests/test_proxy_openai.py
All checks passed
```

## Real Behavior Proof

The focused route tests classify OpenCode from both `User-Agent` and
`X-Client`, verify its tools remain untouched, and verify the decision
reaches all three Responses ingresses. Supported clients continue to
receive deferral. Codex remains excluded by the policy already on
`main`.

Not tested: a live OpenCode instance; the incompatibility itself remains
based on the reporter's reproduction in #2660.

## Review Readiness

- [x] Updated from current upstream `main`
- [x] Merge conflicts resolved
- [x] Focused tests pass locally
- [x] Ready for human review

## Additional Notes

No user configuration or documentation change is required. Vercel
authorization failures are external integration noise, not a source
check.

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
2026-08-08 01:32:54 -05:00