Commit Graph
9 Commits
Author SHA1 Message Date
OmarandJerrettDavis bd164a2c3b fix(proxy): rename leaked_threads metric to timed_out_workers_total (#3908)
## Description

The counter called leaked_threads_total is cumulative. It counts workers
that finished after their deadline and never goes down, so it reads like
a growing leak even when no thread is stuck. This renames it to
timed_out_workers_total. The old key stays in /health with the same
value so existing dashboards keep working.

Closes #3653

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Renamed the counter to timed_out_workers_total and kept
leaked_threads_total as an alias with the same value
- Fixed comments that called it a gauge
- Rebased onto current main, keeping the upstream half-pool quarantine
threshold logic

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
tests/test_proxy_compression_executor.py ....................
tests/test_anthropic_pre_upstream_backpressure.py ......................
................
58 passed in 24.52s
```

Ruff check on the three changed files: all checks passed.

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, repo venv
- Exact command / steps: ran the two executor and backpressure suites
above
- Observed result: 58 passed, /health keeps both keys
- Not tested: mypy

### Maintainer executor runtime qualification (2026-10-08)

- Environment: actual Windows 11, Python 3.12.13, uvicorn/httpx over
loopback TCP, the real bounded compression executor with two workers.
This head contains current `main`; only the synchronization was added to
the contributor's metric rename.
- Exact steps: start the proxy, read `/health`, submit a real executor
worker blocked on a threading Event with a 30ms timeout, read `/health`
while it remains in flight, release it, and read `/health` until the
worker exits. No executor or HTTP route was mocked.
- Observed: cumulative/in-flight counts were **(0,0) before**, **(0,1)
while timed out but still running**, and **(1,0) after exit**. A later
read remained (1,0), and `leaked_threads_total` matched
`timed_out_workers_total` throughout. This confirms the new field is
cumulative, while `timed_out_workers` measures live timeout debt. **58
executor/backpressure tests passed** on the synchronized tree.
- Scope: the probe drives an actual worker through the proxy's executor
entrypoint and observes its published runtime metric over HTTP; no
provider inference is needed for a metric rename. Hosted checks at this
new head are under maintainer qualification; PR approval remains
pending.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: no, old key is kept
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Maintainer edits are enabled on my fork. Please let me know if you
prefer a different name for the metric. Thanks.

### Maintainer current-main requalification

Exact head `c825041c0fe5065451b01d7c6d618bb4f0a228b5`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Re-reviewed latest integration head 758f75453 and current discussion,
including author verification of branch history and the compatibility
alias. Clean integration with upstream/main
2d3ee51846. The cumulative
late-finishing-worker counter remains distinct from the live
timeout-debt gauge, and leaked_threads_total still aliases
timed_out_workers_total. No remaining PR-specific source blocker found
in scope.

All 136 executor/backpressure/release/version/MCP tests pass using the
existing native extension for Python startup; Ruff check/format and
whitespace pass. This is not a fresh native build. Earlier dependency
audit findings were against dependencies inherited from main, so they
are not contributor-code blockers. Maintainers handle fresh exact-head
qualification and separate inherited failures from PR regressions. No
contributor metric behavior rewritten; nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-05 20:17:17 -06:00
1fb4cbaea0 fix(proxy): astgrep disk verify truncation (#3192)
## Description

`_detect_truncation()` currently relies on client-specific truncation
banners in `tool_result` text. This works for clients that emit a
recognizable line-count banner, but not for clients such as Cursor
(byte-based, direction-reversed truncation) and OpenCode (placeholder +
external file pointer).

Because `headroom proxy` runs alongside the client it wraps, it can
optionally verify the returned content against the actual file on disk.
If the `tool_result` is a strict prefix of the file, we can identify
truncation without depending on client-specific output formatting.

This adds disk verification as an opt-in fallback: the existing
banner-based detection always runs first, and disk verification is only
attempted when no banner is found. All disk-verification failures or
ambiguous cases return `UNKNOWN` rather than guessing.

Following review, the verifier is also constrained to a trusted,
confined workspace root rather than trusting caller-supplied input
directly — see the last four bullets under Changes Made.

**Current scope:** the session-token workspace binding this fallback
requires is wired for `headroom wrap claude` and plugin-loaded `headroom
wrap opencode` only. Cursor remains unsupported/not tested — it has no
header-injection surface today to transmit an authoritative session
token, so disk verification stays `UNKNOWN` for it; extending Cursor
support needs a separate, safe transport design.

Related to #2430
Closes #2421

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made
- Added optional disk-based Read truncation verification behind
`HEADROOM_VERIFY_TRUNCATION_ON_DISK`.
- Disk verification compares the visible `tool_result` against the file
on disk and distinguishes `COMPLETE`, `TRUNCATED`, and `UNKNOWN`.
- Disk verification only runs when the existing `_detect_truncation()`
banner check finds no truncation marker, avoiding unnecessary file I/O
when the client already provides a signal.
- Added a `current_cwd` contextvar populated from `x-headroom-cwd` so
relative Read paths can be resolved against the client's working
directory.
- Registered `HEADROOM_VERIFY_TRUNCATION_ON_DISK` as a hot-reloadable
runtime setting.
- Gated verification on a new `is_current_request_trusted` contextvar,
set from `is_loopback_host(request.client.host)` — a server-observed
fact, not the caller-supplied header — so `x-headroom-cwd` is never
sufficient authority on its own.
- Confined all path resolution strictly under the canonicalized `cwd`
(`.resolve()` + `relative_to()`), rejecting `..` traversal,
absolute-path escapes, and post-resolution symlink escapes.
- Required the resolved target to be a regular file and under a byte cap
(new `HEADROOM_VERIFY_TRUNCATION_MAX_BYTES` knob, default 5MB), both
checked via `fstat` on the open fd before any read.
- Opened with `O_NONBLOCK` so a non-regular/special file can't park the
read indefinitely; disk I/O already runs off the event loop via the
existing compression thread pool.
- Added unit and integration tests covering successful verification,
missing/unreadable paths, relative paths, content races, feature gating,
banner precedence, and the adversarial cases below.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_tool_result_interceptors.py -v
51 passed in 2.21s
(includes TestVerifyReadAgainstDisk::* [14 cases, incl. absolute-path escape, `..`
traversal, symlink escape, sibling-prefix boundary, non-regular file, oversized
file, and untrusted-request-touches-no-disk], plus the end-to-end disk-verification
and banner-precedence cases)

$ pytest tests/test_proxy_loopback_gating.py -k trusted
16 passed
(includes test_record_headroom_stack_binds_trusted_true_for_loopback_client and
..._false_for_non_loopback_client — proves a spoofed x-headroom-cwd from a
non-loopback peer does not bind trust)

$ ruff check headroom/proxy/interceptors/astgrep.py headroom/proxy/project_context.py \
    headroom/proxy/server.py headroom/proxy/runtime_env.py \
    tests/test_tool_result_interceptors.py tests/test_proxy_loopback_gating.py
All checks passed!

$ mypy headroom/proxy/interceptors/astgrep.py headroom/proxy/project_context.py \
    headroom/proxy/server.py headroom/proxy/runtime_env.py
Success: no issues found in 4 source files
```

## Real Behavior Proof

- Environment: Mac OS local dev machine, no live proxy/network involved.
Exercised the real interceptor pipeline (`apply_to_messages`) directly
against an actual file on disk, using the same fixture and cwd-binding
path a live proxy request would use.
- Exact command / steps: wrote a 9-function Python fixture to a temp
file, built a client-truncated `tool_result` (first 8 functions, cut
mid-file, **no truncation banner text anywhere**), bound
`x-headroom-cwd` to the temp directory via `set_current_cwd()`, then
called `apply_to_messages()` once with
`HEADROOM_VERIFY_TRUNCATION_ON_DISK` unset and once with it set to `1`.
- Observed result: With disk verification disabled, the existing
ast-grep interceptor produced its normal header without indicating that
the input was truncated. With HEADROOM_VERIFY_TRUNCATION_ON_DISK=1, disk
verification detected that the 50-line tool_result was a strict prefix
of the 59-line file and produced:
 ```text
=== HEADROOM_VERIFY_TRUNCATION_ON_DISK unset (default) ===
[headroom: outlined by ast-grep — 8 definition(s); bodies elided.
Re-read the file with a line range to see a specific body.]

=== HEADROOM_VERIFY_TRUNCATION_ON_DISK=1, x-headroom-cwd bound to the
real directory ===
[headroom: outlined by ast-grep — 8 definition(s) in the visible
portion; input was truncated upstream (showing through line 50 of 59
total). Bodies elided. Re-read remaining lines to see more.]
```

**Trust-gate behavior:** same fixture and disk layout, `HEADROOM_VERIFY_TRUNCATION_ON_DISK=1`, but the request context's trust flag left at its default (`False`, simulating a non-loopback caller) instead of being marked trusted.

```text
=== trusted context (loopback) ===
[headroom: outlined by ast-grep — 8 definition(s) in the visible
portion; input was truncated upstream ...]

=== untrusted context (non-loopback) — same file, same cwd header, same
tool_result ===
[headroom: outlined by ast-grep — 8 definition(s); bodies elided.
Re-read the file with a line range to see a specific body.]
(no disk read attempted — verified via a call-recording wrapper around
os.open asserting zero invocations)
```
Also verified at the HTTP-middleware level with `TestClient(..., client=("100.90.0.5", 12345))`: a request carrying a spoofed `x-headroom-cwd` header from a non-loopback client IP does not bind trust, regardless of header content.

- Not tested: a live `headroom proxy` session with a real coding-agent client; the WebSocket/Codex responses-WS path.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. The feature is gated by `HEADROOM_VERIFY_TRUNCATION_ON_DISK`
- Minimum rollout channel: N/A. `--intercept-tool-results` must already be enabled for the ast-grep interceptor to run.
- Stable/default behavior changed: no
- Kill switch / disable path: Unset `HEADROOM_VERIFY_TRUNCATION_ON_DISK`; the setting is hot-reloadable and does not require a restart.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit, or unset the env var on a running proxy (takes effect immediately, no restart).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this)

## Additional Notes

this branch is based on `fix/astgrep-truncation-marker` #2430 not main


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3192" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-08 10:52:09 -05:00
be2b205c51 fix(proxy): quarantine compression only once half the pool is stuck (#3932)
## Description

The timeout-debt quarantine in
`HeadroomProxy._run_compression_in_executor` refused every new
compression call while even one timed-out worker was still running. On
an auto-sized pool (one worker per CPU, 8 on a typical laptop) a single
straggler forwarded every other request uncompressed for up to the 60s
cap while 7 of 8 workers sat idle. A common trigger is the cold-start
fast pass missing its deliberately short 10s fail-open budget.

The quarantine exists so timed-out work cannot saturate the executor
(#2292, #2360). That risk starts near saturation, not at the first
straggler, so the check now engages only once timed-out workers hold
half the pool (`max(1, max_workers // 2)`). Pools of 1-3 workers keep
today's behaviour.

Closes #3931

Related, not closed by this PR: #3711 (what makes one Kompress stage
overrun its budget; this PR narrows how far one overrun spreads, not the
overrun itself) and #3653 (thread/fd reclamation for stuck workers; see
Additional Notes for why this change does not add to it).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/server.py`: new `_compression_quarantine_threshold`,
defaulting to `max(1, compression_max_workers // 2)` and overridable
with `HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD` (read with the existing
`_get_env_int`, `min_value=1`; `=1` restores the old any-straggler
behaviour).
- The quarantine check and its 60s time-cap release both key on
`timed_out_in_flight >= threshold` instead of `> 0`. The cap is still
re-armed on every fresh timeout, as before.
- `quarantine_activations_total` (and the `activated` Prometheus event
and warning) now fire on the transition into the quarantined state, i.e.
when debt reaches the threshold, or when a fresh timeout re-arms it
after the time cap released it. Previously a re-arm after a release
while debt still stood was not counted.
- "Quarantine cleared" is logged when debt drops back below the
threshold, with the number of timed-out workers still running.
- `/health` `runtime.compression_executor.quarantine_active` now reports
the condition the check enforces (debt at or above the threshold and the
cap not lapsed). It used to report `timed_out_workers > 0`, which was
already wrong after a time-cap release.
- Tests: `test_quarantine_waits_for_half_the_pool_to_be_stuck` (8-worker
pool: 1 and 3 stuck still compress, 4 stuck refuses, `/health` matches,
re-arm after release counts as a second activation) and
`test_quarantine_threshold_is_half_the_pool_unless_overridden` (default
sizing and the env override). Two tests in
`tests/test_tokenizer_count_offload.py` that forced a quarantine with a
debt of 1 on an auto-sized pool now set the debt to the threshold.

## Testing

- [x] Unit tests pass (`pytest`) - targeted files only, listed below;
the full suite was not run
- [x] Linting passes (`ruff check .`) - on the changed files, plus `ruff
format --check`
- [x] Type checking passes (`mypy headroom`) - `mypy
headroom/proxy/server.py`
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Before the fix (new tests against unmodified `server.py`; the two
tokenizer tests fail only because they read the new attribute):

```text
$ python -m pytest tests/test_proxy_compression_executor.py tests/test_tokenizer_count_offload.py -q -p no:cacheprovider
FAILED tests/test_proxy_compression_executor.py::test_quarantine_waits_for_half_the_pool_to_be_stuck
FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[8-None-4]
FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[3-None-1]
FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[1-None-1]
FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[8-1-1]
FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[8-0-4]
FAILED tests/test_tokenizer_count_offload.py::test_count_tokens_offloaded_fails_open_on_executor_quarantine
FAILED tests/test_tokenizer_count_offload.py::test_count_tokens_offloaded_returns_count_text_capable_tokenizer
=================== 8 failed, 20 passed, 1 warning in 20.73s ===================
```

The behavioural failure on main is:

```text
tests/test_proxy_compression_executor.py:336: in test_quarantine_waits_for_half_the_pool_to_be_stuck
E   headroom.proxy.server.CompressionQuarantinedError: compression quarantined: 1 timed-out worker(s) still running
```

After the fix:

```text
$ python -m pytest tests/test_proxy_compression_executor.py tests/test_tokenizer_count_offload.py tests/test_platform_stabilization_functional.py tests/test_prometheus_obs_counters.py -q -p no:cacheprovider
======================== 51 passed, 1 warning in 28.02s ========================

$ ruff check headroom/proxy/server.py tests/test_proxy_compression_executor.py tests/test_tokenizer_count_offload.py
All checks passed!
$ ruff format --check headroom/proxy/server.py tests/test_proxy_compression_executor.py tests/test_tokenizer_count_offload.py
3 files already formatted
$ mypy headroom/proxy/server.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.11, upstream main `d5318ac2`
(before) and the same commit plus this branch (after). Proxy booted
through `headroom.proxy.server.run_server(ProxyConfig(port=...,
compression_max_workers=8, disable_kompress=True, cache_enabled=False,
rate_limit_enabled=False, log_requests=False))`, which is what `headroom
proxy` calls after argument parsing (the local venv lacked the
`mcp`/`magika`/`onnxruntime` extras that the CLI's dependency preflight
requires). Env: `HEADROOM_SKIP_UPSTREAM_CHECK=1
HEADROOM_COMPRESSION_TIMEOUT_SECONDS=3 HEADROOM_TELEMETRY=off`, a fresh
`HEADROOM_WORKSPACE_DIR` per run.
- Exact command / steps: over HTTP, one warm-up `/v1/compress` (200-row
JSON `tool_result`, 10,623 tokens), one oversized `/v1/compress` (40,000
rows, about 22s of work against the 3s budget, so its worker keeps
running after the request fails open), then three small `/v1/compress`
calls while that worker is still running, reading `/health`
`runtime.compression_executor` in between. A second run fires four
oversized calls concurrently to strand 4 of 8 workers, sends a small
call, waits until one stuck worker exits, and sends another.
- Observed result: on main one straggler refuses every following
request; on this branch the other 7 workers keep compressing, and half
the pool stuck still quarantines and clears again (output below).

Before (main), one straggler refuses everything:

```text
health after big timed out: {'max_workers': 8, 'running': 1, 'timed_out_workers': 1, 'quarantine_active': True, 'quarantine_activations_total': 1, 'quarantine_skips_total': 0}
small request 1: 0.02s tokens_before=0 tokens_saved=0 skipped=True reason=compression_timeout
small request 2: 0.01s tokens_before=0 tokens_saved=0 skipped=True reason=compression_timeout
small request 3: 0.01s tokens_before=0 tokens_saved=0 skipped=True reason=compression_timeout
health after small requests: {'max_workers': 8, 'running': 1, 'timed_out_workers': 1, 'quarantine_active': True, 'quarantine_activations_total': 1, 'quarantine_skips_total': 3}
```

After, the same straggler leaves the other 7 workers serving:

```text
health after big timed out: {'max_workers': 8, 'running': 1, 'timed_out_workers': 1, 'quarantine_active': False, 'quarantine_activations_total': 0, 'quarantine_skips_total': 0}
small request 1: 0.12s tokens_before=10623 tokens_saved=4411 skipped=False reason=None
small request 2: 0.12s tokens_before=10623 tokens_saved=4411 skipped=False reason=None
small request 3: 0.18s tokens_before=10623 tokens_saved=4411 skipped=False reason=None
health after small requests: {'max_workers': 8, 'running': 1, 'timed_out_workers': 1, 'quarantine_active': False, 'quarantine_activations_total': 0, 'quarantine_skips_total': 0}
```

After, half the pool stuck still quarantines, and it clears when debt
drops below half:

```text
health after 4 concurrent big requests timed out: {'max_workers': 8, 'running': 4, 'timed_out_workers': 4, 'quarantine_active': True, 'quarantine_activations_total': 1, 'quarantine_skips_total': 0}
small request while 4 of 8 stuck: 0.10s tokens_before=0 tokens_saved=0 skipped=True reason=compression_timeout
health once a stuck worker exited: {'max_workers': 8, 'running': 3, 'timed_out_workers': 3, 'quarantine_active': False, 'quarantine_activations_total': 1, 'quarantine_skips_total': 1}
small request with 3 stuck: 0.19s tokens_before=10623 tokens_saved=4411 skipped=False reason=None
```

```text
WARNING - Compression worker exceeded its request deadline and is still running; new compression is quarantined until fewer than 4 timed-out workers remain
INFO - Compression quarantine cleared; 3 timed-out worker(s) still running
```

- Not tested: the `headroom proxy` CLI entry point itself (booted via
`run_server`, see Environment); the `/v1/messages`, `/v1/responses` and
Codex WS paths end to end (they share the same
`_run_compression_in_executor`, but I only drove `/v1/compress`);
Windows and Linux (the field evidence covers them, my repro does not);
the 60s time-cap release over HTTP (covered by
`test_compression_quarantine_releases_after_time_cap` and the new re-arm
assertion, not by a live run); a long soak sampling OS thread and fd
counts.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. The quarantine is not in
`rollout.py` FEATURES.
- Minimum rollout channel: n/a (applies on every channel).
- Stable/default behavior changed: yes. On pools of 4 or more workers,
compression keeps running while fewer than half the workers are held by
timed-out jobs; previously the first timed-out job refused all
compression for up to 60s. Pools of 1-3 workers are unchanged.
- Kill switch / disable path:
`HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD=1` restores the previous
behaviour on any pool size.
- Unsafe override required: no.
- Qualification impact: more requests get compressed while a straggler
runs, so `tokens_saved` rises and the `skipped` quarantine counter falls
on affected installs; no change to what compression does to a request it
runs on, and no change to provider cache behaviour beyond the requests
that are now compressed instead of forwarded raw.
- Rollback path: set `HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD=1`, or
revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Pool capacity and #3653. The compression executor is a
`ThreadPoolExecutor` with a fixed `max_workers`; it never creates
threads beyond that cap, and a stuck worker keeps the thread it already
holds. Admitting work while a straggler runs only puts that work on the
pool's idle threads, which finish and release their resources normally,
so this change cannot raise the process's thread count above
`max_workers` or add a leaked thread/fd per quarantine event. What it
does change is how many timed-out jobs can stand before new work is
refused: up to `threshold - 1` (3 on an 8-worker pool) instead of 0.
That bound is no looser than today's in practice: the old rule never
prevented requests that were already admitted from timing out together
(it only refuses calls that arrive after a timeout is recorded), and the
60s cap already lets work back in while debt stands. At `threshold`
stuck workers the quarantine engages exactly as before, so at least half
the pool stays free for work that is not stuck.

Field evidence that motivated this (Headroom Desktop fleet canary,
2026-09-21 to 2026-09-23): 9 hosts on macOS, Windows and Linux had
between 2.1% and 38.9% of requests refused compression by the
quarantine. On one machine's proxy logs, 211 of 262 refusals happened
with ONE stuck worker and none with more than 3 of 8 stuck; on 0.37.0
that machine forwarded 42 of 180 requests (23%) uncompressed in a day
while `/health` showed `max_workers` 8, `queued_max` 1 and
`queue_wait_seconds_max` 0.23, i.e. an idle pool.

Headroom Desktop has shipped an equivalent downstream patch (standing
the quarantine down below half the pool) since 2026-09-23 (desktop
0.9.22-rc.1); this PR implements it in the check itself so it can be
dropped there.

Documentation: `HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD` follows the
existing undocumented `HEADROOM_COMPRESSION_QUARANTINE_MAX_SECONDS`, so
no docs page was updated; happy to add both to the proxy env reference
if wanted.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3932"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-04 13:56:54 -07:00
f90a56b038 fix(proxy): F3 — per-tenant TOIN learning key (#404)
## Description

This change isolates TOIN learning by tenant in shared-proxy
deployments. Each pattern key gains a tenant namespace derived from a
configurable sanitized header, a stable auth/token hash, or the
backwards-compatible global namespace. Legacy keys migrate to global and
every resolution path emits a structured event.

## Threat model

Pre-F3, every request's tool-call patterns aggregated into one global
pool keyed by `(auth_mode, model_family, sig_hash)`. That works for an
OSS single-tenant deploy, but in a multi-tenant SaaS / shared-proxy
deploy:

1. **Cross-tenant pattern leakage.** Tenant A's compression patterns
silently train recommendations the proxy then applies to tenant B's
requests, degrading both tenants' compression quality.
2. **Structural fingerprint exposure.** TOIN dumps include
`tool_signature_hash` per slice. With one shared pool, a user with read
access to a dump file learns the structural fingerprint of every other
tenant's tool surface.

F3 partitions: every TOIN store key gets a `tenant_key` prefix derived
from the inbound request. Two distinct tenants therefore train two
isolated pattern pools.

## Tenant key derivation rules

| Source | When | Key derivation |

|------------|--------------------------------------------------------------|-------------------------------------------------------------------|
| `header` | `HEADROOM_TENANT_KEY_HEADER` (default
`X-Headroom-Tenant-ID`) is present and survives sanitization | Sanitized
header value (ASCII alphanum + `-_`, max 64 chars) |
| `hash` | No header but bearer token + `auth_mode` (from F1) present |
`sha256(f"{auth_mode}:{api_key_prefix}")[:24]` (api_key_prefix = first 8
chars of bearer); same SHA-256[:24] idiom as PR #395 |
| `global` | Neither header nor bearer present (OSS single-tenant / dev)
| Literal `"global"` namespace; pre-F3 patterns migrate here on first
load |

Header path takes precedence over hash. Header values that sanitize to
empty (pure unicode / control chars only) fall through to the next rule.

## Configuration

Single env var: `HEADROOM_TENANT_KEY_HEADER`. Default
`X-Headroom-Tenant-ID`. The header NAME is configurable; the
sanitization rules and the `"global"` literal are NOT — they're
load-bearing for namespace isolation.

## Structured log fields

Every resolution path emits exactly one `tenant_key_resolved` log event
(no silent fallbacks per `feedback_no_silent_fallbacks.md`):

| Field | Always present | Notes |

|--------------|----------------|------------------------------------------------|
| `event` | yes | Always `"tenant_key_resolved"` |
| `tenant_key` | yes | Resolved key (or `"global"`) |
| `source` | yes | One of `header` / `hash` / `global` |
| `header` | header path | Header name that matched |
| `auth_mode` | hash path | `payg` / `oauth` / `subscription` |

Operators can alert on a sustained `source="global"` rate to spot
mis-configured upstream auth proxies.

## Implementation

- **NEW** `headroom/proxy/tenant_key.py`: pure resolver. Public surface
= `resolve_tenant_key(request) -> tuple[str, str]`. Plus
`set_request_tenant_key()` / `get_current_tenant_key()` ContextVar pair
so TOIN's deep-stack `record_compression` calls observe the right tenant
slice.
- `headroom/telemetry/toin.py`: extends `PatternKey` from a 3-tuple to a
4-tuple `(tenant_key, auth_mode, model_family, sig_hash)`.
`_deserialize_pattern_key` is backward-compat with both pre-F3 formats
(3-tuple from PR-B5, bare `sig_hash` from v1.0). `record_compression` /
`record_retrieval` / `get_pattern` accept an optional `tenant_key`
kwarg; when absent they read from the ContextVar.
- `headroom/cli/toin_publish.py`: each `[[recommendation]]` row now
carries `tenant_key`; rows sort by `(tenant_key, auth_mode,
model_family, sig_hash)` for clean diffs.
- `headroom/proxy/handlers/anthropic.py` +
`headroom/proxy/handlers/openai.py`: call `resolve_tenant_key(request)`
once at request entry, right after the F1 `classify_auth_mode` block.
Three call sites total (1 Anthropic, 2 OpenAI). Result attached to
`request.state.tenant_key` / `request.state.tenant_key_source` AND set
on the ContextVar.

## Realignment build constraints (per project memory)

- **configurable**: `HEADROOM_TENANT_KEY_HEADER` env var.
- **no hardcodes**: header name + `"global"` literal are the only
constants; both motivated and documented in code.
- **no regexes**: sanitization uses string ops +
`unicodedata.category()`.
- **no silent fallbacks**: every resolution emits a structured log
event.
- **comprehensive tests**: header / hash / global paths + sanitization
edges + tenant-isolation property test.
- **structured logs**: `event=tenant_key_resolved` with `tenant_key`,
`source`, optional `header` / `auth_mode`.

## Test plan

- [x] `pytest tests/test_tenant_key.py` (29 cases — header / hash /
global paths, structured-log emission, sanitization edges, ContextVar
round-trip)
- [x] `pytest tests/test_toin_observation_only.py` (6 new isolation
cases — two-tenant divergence, ContextVar read, explicit kwarg
precedence, default migration to `"global"`)
- [x] `pytest tests/test_toin*.py` (existing TOIN tests updated for the
4-tuple key shape)
- [x] `pytest tests/test_proxy_tenant_key_integration.py` (4 cases —
handler wires `request.state.tenant_key` + ContextVar through all three
resolution paths)
- [x] `pytest tests/test_critical_fixes.py tests/test_critical_gaps.py
tests/test_proxy_ccr.py` (legacy 3-tuple references updated)
- [x] `make ci-precheck-python` — 176 tests pass (passed via pre-push
hook)
- [x] `make ci-precheck-commitlint` — clean (passed via pre-push hook)

Refs: F1 (PR #399, auth_mode classifier), F2.1 (PR #400, per-auth-mode
CompressionPolicy gates).

## Type of Change

- [x] Bug fix / tenant-isolation hardening
- [ ] New feature
- [ ] Breaking change
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- Add a pure tenant-key resolver and request ContextVar.
- Extend TOIN pattern keys and published recommendations with a tenant
namespace while preserving legacy deserialization.
- Resolve and attach tenant identity at Anthropic and OpenAI request
entry.
- Add isolation, migration, sanitization, header, CCR, handler, and
executor coverage.

## Testing

- [x] Focused tenant/TOIN/proxy tests pass
- [x] Python CI precheck passes
- [x] Commitlint precheck passes

### Test Output

```text
176 focused/precheck tests passed
Current-main isolation, migration, header, CCR, and executor validation: green
```

## Real Behavior Proof

- Environment: current-main reconciled Python proxy/TOIN test
environment.
- Exact command / steps: ran tenant resolver, TOIN observation/publish,
handler integration, header-isolation, CCR, and compression-executor
suites.
- Observed result: distinct tenants train isolated pattern pools; legacy
patterns migrate to global; request handlers propagate the resolved
namespace through deep TOIN calls.
- Not tested: production multi-tenant traffic volume or an external TOIN
dump consumer.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 22:03:59 -05:00
Sebastian SchkudlaraandTejas Chopra 517bf992cf fix(proxy): quarantine compression while timed-out workers run (#2292)
## Description

A request-side `asyncio.wait_for()` timeout stops waiting, but it cannot
preempt an executor thread that already started. The proxy counted those
late workers and still admitted more compression, so repeated slow calls
could consume the whole compression pool and charge every request
another full timeout.

This change tracks running post-timeout workers as timeout debt and
quarantines request-path compression while that debt is non-zero. New
attempts raise `CompressionQuarantinedError` before executor admission,
using an `asyncio.TimeoutError` subclass so Python 3.10 handlers apply
the existing compression-failure policy. Quarantine clears automatically
after all known timed-out workers genuinely exit.

Mitigates #946 and #810. It does not attempt to kill the first running
thread; Python cannot safely preempt it.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Track started, finished, timed-out, and debt-recorded state under the
existing compression metrics lock.
- Reject new compression before enqueue while timed-out workers remain;
clear quarantine on the final worker exit.
- Preserve queued-timeout behavior: work cancelled before worker start
does not activate quarantine; a cancellation/start race is
conservatively tracked as running debt.
- Add `/health` and `/stats` runtime fields for quarantine state, worker
debt, activations, and skips.
- Add `headroom_compression_quarantine_total{event="activated|skipped"}`
Prometheus counters.
- Add regression, recovery, queue-race, runtime-payload, export, reset,
and Python 3.10 exception-class coverage.
- Update `CHANGELOG.md`; no dependency or lockfile changes.

## Reproduction

On base commit `718c8dc5`, I applied only the new regression test and
ran:

```bash
.venv/bin/pytest -q \
  tests/test_proxy_compression_executor.py::test_timeout_quarantines_new_work_until_timed_out_worker_finishes
```

The first worker timed out but remained blocked. The second callable
entered the executor instead of being rejected:

```text
FAILED: DID NOT RAISE TimeoutError
```

## Testing

- [x] Affected unit tests pass
- [x] Linting passes (`ruff check .`)
- [x] Changed-file type checking passes
- [x] New tests added for the fix
- [x] Manual testing performed

### Test Output

```text
Python 3.10.20

$ .venv/bin/pytest -q -m 'not slow' \
    tests/test_proxy_compression_executor.py \
    tests/test_prometheus_obs_counters.py \
    tests/test_proxy/test_compression_failure_action.py \
    tests/test_proxy/test_compression_timeout_config.py \
    tests/test_anthropic_pre_upstream_backpressure.py \
    tests/test_openai_codex_ws_lifecycle.py \
    tests/test_codex_ws_compression_scheduler.py \
    tests/test_gemini_compression_offload.py \
    tests/test_proxy_handlers_batch.py \
    tests/test_tokenizer_count_offload.py \
    tests/test_cold_start_fast_pass.py
125 passed, 1 skipped, 1 deselected, 1 warning in 11.06s

$ .venv/bin/ruff check .
All checks passed!

$ .venv/bin/ruff format --check .
1310 files already formatted

$ .venv/bin/mypy headroom/proxy/server.py headroom/proxy/prometheus_metrics.py
Success: no issues found in 2 source files

$ git diff --check
# no output
```

The warning is the existing Starlette `TestClient`/`httpx` deprecation
warning.

## Real Behavior Proof

- Environment: macOS 15.7.4 x86_64, Python 3.10.20,
`compression_max_workers=2`, direct proxy executor path, no external
provider/model.
- Exact command / steps: instantiate the proxy; run a blocking
compression callable with a 50 ms timeout; immediately attempt a second
callable and time the rejection; release the first worker; wait for debt
to reach zero; run the second callable again; export Prometheus metrics.
- Observed result: the first request timed out at 51.182 ms; the second
attempt was rejected in 0.014 ms and its callable never started; debt
was 1 while quarantined, returned to 0 after release, and compression
then resumed normally.

```json
{
  "after_release": {
    "activations_total": 1,
    "leaked_threads_total": 1,
    "quarantine_active": false,
    "skips_total": 1,
    "timed_out_workers": 0
  },
  "bypass_elapsed_ms": 0.014,
  "bypass_error": "compression quarantined: 1 timed-out worker(s) still running",
  "during_quarantine": {
    "quarantine_active": true,
    "timed_out_workers": 1
  },
  "first_timeout_elapsed_ms": 51.182,
  "prometheus": [
    "headroom_compression_quarantine_total{event=\"activated\"} 1",
    "headroom_compression_quarantine_total{event=\"skipped\"} 1"
  ],
  "resumed_result": "resumed",
  "second_callable_started_during_quarantine": false
}
```

- Not tested: a live external model/provider; forced termination of a
permanently wedged native worker; the marked slow native scheduler
benchmark. A broad non-slow run collected 9,859 selected tests but was
stopped at
`tests/test_adversarial_grid.py::TestRunGrid::test_grid_shape_and_schema`
after a macOS process sample showed the pre-existing native
`_core.abi3.so` semaphore stall (`_dispatch_semaphore_wait_slow` →
`semaphore_wait_trap`). The affected executor/handler slice above
completed cleanly.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented hard-to-understand concurrency paths
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove the fix is effective
- [x] New and affected existing unit tests pass locally
- [x] I have updated the CHANGELOG.md

---------

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-07-16 14:30:06 -07:00
Rod Boev 0a3851b240 perf(proxy): cap compression workers to CPU count (#1803)
## Description

The request-path compression executor currently uses asyncio-style I/O
sizing for CPU-bound Kompress work. When `compression_max_workers` is
unset, `HeadroomProxy.__init__` resolves the pool to `min(32, cpu * 4)`,
so an eight-core host can run 32 simultaneous compression workers that
all contend for real CPU.

This changes only the automatic request-path default to one worker per
reported CPU while preserving the existing explicit override path from
`--compression-max-workers` and `HEADROOM_COMPRESSION_MAX_WORKERS`.

Closes #1635

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Cap the automatic request-path compression executor default at `max(1,
os.cpu_count() or 1)`.
- Preserve explicit `compression_max_workers` values, including the
existing clamp to at least one worker.
- Keep CLI help, `ProxyConfig` comments, and nearby test documentation
aligned with the CPU-bound default.
- Update the focused compression executor regression so the default
contract documents CPU-bound sizing, and keep the existing Codex
compression stress guard stable when p50 rounds to zero.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_codex_ws_compression_scheduler.py
tests/test_proxy_compression_executor.py
tests/test_cli_proxy_improvements.py::TestCompressionMaxWorkers -q`)
- [x] Linting passes (`uv run ruff check headroom/cli/proxy.py
headroom/proxy/models.py headroom/proxy/server.py
tests/test_cli_proxy_improvements.py
tests/test_proxy_compression_executor.py
tests/test_codex_ws_compression_scheduler.py`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_codex_ws_compression_scheduler.py tests/test_proxy_compression_executor.py tests/test_cli_proxy_improvements.py::TestCompressionMaxWorkers -q
16 passed, 1 skipped, 1 warning in 6.13s

$ uv run ruff check headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/server.py tests/test_cli_proxy_improvements.py tests/test_proxy_compression_executor.py tests/test_codex_ws_compression_scheduler.py
All checks passed!
```

## Real Behavior Proof

- Environment: Windows, Python environment from `uv sync --extra dev`,
no provider credentials needed.
- Exact command / steps: construct `HeadroomProxy` with
`compression_max_workers=None`, inspect `proxy.compression_max_workers`
and `/health` `runtime.compression_executor`.
- Observed result: the automatic request-path pool resolves to reported
CPU count, while explicit overrides still resolve to the configured
value and report `source: explicit`.
- Not tested: multi-session wall-clock benchmark under live Kompress
load.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

No `CHANGELOG.md` edit: this repo generates changelog entries from
conventional commits. This intentionally does not touch the background
compression executor surface covered by #1633.
2026-07-05 14:01:23 -07:00
Rod Boev 15ac650d40 fix(proxy): fail open when kompress saturation would exhaust pre-upstream budget (#1430)
## Description

Concurrent Anthropic `/v1/messages` traffic can still exhaust Headroom's
pre-upstream budget because Kompress ONNX execution waits on the request
critical path. When Kompress saturates, requests eventually fail with
`503 pre-upstream queue saturated` even though compression can safely
degrade to passthrough.

This PR makes Kompress saturation fail open on the Anthropic hot path,
so requests continue uncompressed when compression capacity is under
pressure. It keeps the executor and stage-timing evidence intact, and it
preserves blocking model-load validation so runtime pressure does not
silently skip the validation path.

Closes #1025

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- add a bounded execution-slot acquire path so Anthropic requests fail
open to passthrough when Kompress saturation would consume the
pre-upstream budget
- preserve explicit execution-timeout counters and Anthropic
passthrough/stage-timing observability instead of hiding the pressure
path
- keep `_validate_pytorch_device()` on blocking acquire semantics so
model-load validation still waits for capacity instead of failing open
- make the blocking validation acquire explicit to `mypy` without
changing runtime behavior
- extend focused regressions for pre-upstream backpressure, Kompress
saturation, execution-skip observability, and validation waiting
- align the CLI timeout help text and `ProxyConfig` comment with the
fail-open runtime behavior
- update `CHANGELOG.md` for the proxy runtime fix

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_anthropic_pre_upstream_backpressure.py
tests/test_proxy_compression_executor.py
tests/test_kompress_request_nonblocking.py`)
- [x] Linting passes (`uv run ruff check
tests/test_anthropic_pre_upstream_backpressure.py` and `uv run ruff
format tests/test_anthropic_pre_upstream_backpressure.py --check`)
- [x] Type checking passes (`uv run mypy headroom
--ignore-missing-imports`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
Focused local validation passed:
- uv run pytest tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_compression_executor.py tests/test_kompress_request_nonblocking.py -x -v
  37 passed, 1 warning in 12.01s
- uv run ruff check headroom/proxy/handlers/anthropic.py headroom/transforms/kompress_compressor.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_compression_executor.py tests/test_kompress_request_nonblocking.py
  All checks passed!
- uv run ruff format headroom/proxy/handlers/anthropic.py headroom/transforms/kompress_compressor.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_compression_executor.py tests/test_kompress_request_nonblocking.py --check
  5 files already formatted
- uv run mypy headroom --ignore-missing-imports
  Success: no issues found in 398 source files

Base-branch proof on origin/main (fa05ebc849):
- test_acquire_timeout_degrades_to_passthrough fails because the handler still returns 503
- test_saturation_fail_open_does_not_hang_request fails because get_kompress_execution_stats() does not exist
- test_compression_executor_skip_signal_remains_visible passes on base too, so it stays as compatibility coverage rather than the failing-then-passing proof for this fix

Review-follow-up validation passed after aligning the timeout wording with fail-open behavior:
- uv run pytest tests/test_anthropic_pre_upstream_backpressure.py -x -v
  20 passed, 1 warning in 1.38s
- uv run ruff check headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/handlers/anthropic.py headroom/transforms/kompress_compressor.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_compression_executor.py tests/test_kompress_request_nonblocking.py
  All checks passed!
```

## Real Behavior Proof

- Environment: local Anthropic pre-upstream and Kompress execution
regression harnesses covering the `/v1/messages` hot path
- Exact command / steps: run the focused pytest command above on
`origin/main` and on this branch, including the semaphore-saturation
path in `test_saturation_fail_open_does_not_hang_request` and the
validation-slot hold in `test_validation_probe_waits_for_execution_slot`
- Observed result: the reviewed head no longer returns `503` on the
pre-upstream pressure path, request-thread Kompress saturation degrades
to passthrough while incrementing execution timeout stats, and
model-load validation still waits for capacity instead of failing open
- Not tested: wrap/install fallout mentioned in the original issue

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new type-check or lint warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Scoped to the runtime queue-pressure fault only; the issue's
wrap/unwrap and deployment complaints stay out of this PR.
- `test_compression_executor_skip_signal_remains_visible` remains in the
suite to prove the skip signal stays visible, but it is compatibility
coverage rather than the failing-then-passing regression for the bug
fix.
- Local validation included `uv run mypy headroom
--ignore-missing-imports` after the explicit validation-acquire
narrowing was added for CI parity.
- Attribution: the issue body isolated the hot-path ONNX compression
stall and the pre-upstream saturation symptom that this PR fixes.
2026-06-30 13:41:22 -05:00
Tejas Chopra e9cae0131b fix: expose compression latency bottlenecks
Add Codex WS unit-level timing and bounded parallel compression, clarify context-tool session savings, and avoid costly diff/log fallbacks to Kompress.
2026-05-12 13:34:08 -07:00
chopratejas ea78cf6252 fix(proxy): cache concurrency lock, multi-worker docs, bounded compression executor
Three audit follow-ups from issue #327's deep-dive review.

C1 — CompressionCache concurrency lock
======================================

`CompressionCache` instances are shared per `session_id` and accessed from
async-dispatched threadpool workers. Pre-fix, concurrent requests for the
same session raced on `_cache`, `_stable_hashes`, `_first_seen`, and
`_total_tokens_saved` with no synchronization. Observable failures:

* Lost-update on `_total_tokens_saved` (read-modify-write).
* `RuntimeError: OrderedDict mutated during iteration` from `apply_cached`
  when a concurrent `store_compressed` evicts during the walk.
* Lost stable-hash records — next-turn compute_frozen_count reads
  inconsistent state.

May also explain part of SvenMeyer's `_cache: 0 entries / 1003 misses`
observation: the cache was being clobbered concurrently.

Added `threading.RLock` guarding all mutating methods. `RLock` (not `Lock`)
so future code can call locked methods from inside another locked method
without self-deadlock. Also locked `HeadroomProxy._compression_caches`
dict-of-caches access via a separate `_compression_caches_lock` so two
concurrent calls for the same session_id can't each create distinct
CompressionCache objects (which would split the cache state between them).
The `/stats` endpoint snapshots the cache list under the dict lock before
iterating to avoid eviction-during-iteration.

C2 — Multi-worker CCR fragmentation: documented + startup warning
=================================================================

The in-memory `InMemoryCcrStore` (Rust), `_compression_caches` (Python),
`session_tracker_store` (Python), and TOIN learner state are ALL
per-process. Multi-worker uvicorn round-robins requests across workers,
so a session whose turn-1 lands on worker A may have turn-2 land on
worker B. Worker B has zero knowledge of A's CCR markers, replay cache,
or prefix-cache state. Result: `Retrieve original: hash=X` markers stay
in-context as opaque directives, every fresh tool_result is recompressed
from scratch, and `frozen_message_count=0` causes Anthropic prefix-cache
busts on every cross-worker turn.

Added a "Multi-worker deployment — CCR fragmentation" section in
`RUST_DEV.md` documenting the failure modes, the supported configuration
(`--workers 1`), and the sticky-session workaround for horizontal scale.
The proxy emits a `WARNING`-level log line on startup if `workers > 1` is
detected, pointing at the doc section.

C3 — Bounded compression executor with cancel-aware metrics
===========================================================

`asyncio.wait_for(asyncio.to_thread(pipeline.apply), timeout=...)`
cancellation does NOT propagate into the threadpool worker that's running
Rust code. Once the worker has picked up the task,
`concurrent.futures.Future.cancel()` returns False and the thread runs to
completion. Stuck threads accumulated invisibly on asyncio's default
executor, contending with unrelated `to_thread` callers (file IO, etc.).

Replaced all 7 `asyncio.to_thread` call sites for `pipeline.apply()`
across `proxy/handlers/anthropic.py` (3) and `proxy/handlers/openai.py` (4)
with a new `HeadroomProxy._run_compression_in_executor(fn, *, timeout)`
helper that:

  1. Submits to a dedicated bounded `ThreadPoolExecutor` named
     `headroom-compress` (configurable via
     `ProxyConfig.compression_max_workers`; defaults to
     `min(32, (cpu_count or 1) * 4)`).
  2. Increments `_compression_in_flight` (gauge) when work starts and
     decrements when work completes; tracks `_compression_in_flight_max`
     as a high-water mark.
  3. Detects "leaked threads" by comparing wall-clock elapsed against the
     timeout in the worker's `finally` block. Increments
     `_compression_leaked_threads` when a worker finishes after its
     asyncio future was cancelled. Operators can see the leaked-thread
     rate climbing in `/stats runtime.compression_executor` BEFORE the
     pool fills up.

Tests
=====

* `TestCompressionCacheConcurrency` (3 tests) — many threads
  store_compressed / apply_cached / update_from_result on a single
  CompressionCache; assert no exceptions, no lost updates, no partial
  state.
* `test_get_compression_cache_returns_same_instance_under_contention` —
  32 concurrent `_get_compression_cache(same_id)` calls return the
  identical instance (would split pre-lock).
* `test_proxy_compression_executor.py` (8 tests) — pool size respects
  config, in-flight gauge tracks running compressions, high-water mark
  is monotonic, timeout propagates to awaiter, leaked-thread counter
  increments on post-deadline completion, `/stats` surfaces all three
  gauges.

Verification
============

* All 123 targeted regression tests pass.
* `make ci-precheck` clean.
* No `Co-Authored-By` trailer; conventional `fix:` prefix; no
  `--no-verify`.
2026-05-01 15:25:18 -07:00