mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
fix/plugin-launcher-isolated-python
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
bd164a2c3b |
fix(proxy): rename leaked_threads metric to timed_out_workers_total (#3908)
## Description The counter called leaked_threads_total is cumulative. It counts workers that finished after their deadline and never goes down, so it reads like a growing leak even when no thread is stuck. This renames it to timed_out_workers_total. The old key stays in /health with the same value so existing dashboards keep working. Closes #3653 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Renamed the counter to timed_out_workers_total and kept leaked_threads_total as an alias with the same value - Fixed comments that called it a gauge - Rebased onto current main, keeping the upstream half-pool quarantine threshold logic ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text tests/test_proxy_compression_executor.py .................... tests/test_anthropic_pre_upstream_backpressure.py ...................... ................ 58 passed in 24.52s ``` Ruff check on the three changed files: all checks passed. ## Real Behavior Proof - Environment: Windows 11, Python 3.12, repo venv - Exact command / steps: ran the two executor and backpressure suites above - Observed result: 58 passed, /health keeps both keys - Not tested: mypy ### Maintainer executor runtime qualification (2026-10-08) - Environment: actual Windows 11, Python 3.12.13, uvicorn/httpx over loopback TCP, the real bounded compression executor with two workers. This head contains current `main`; only the synchronization was added to the contributor's metric rename. - Exact steps: start the proxy, read `/health`, submit a real executor worker blocked on a threading Event with a 30ms timeout, read `/health` while it remains in flight, release it, and read `/health` until the worker exits. No executor or HTTP route was mocked. - Observed: cumulative/in-flight counts were **(0,0) before**, **(0,1) while timed out but still running**, and **(1,0) after exit**. A later read remained (1,0), and `leaked_threads_total` matched `timed_out_workers_total` throughout. This confirms the new field is cumulative, while `timed_out_workers` measures live timeout debt. **58 executor/backpressure tests passed** on the synchronized tree. - Scope: the probe drives an actual worker through the proxy's executor entrypoint and observes its published runtime metric over HTTP; no provider inference is needed for a metric rename. Hosted checks at this new head are under maintainer qualification; PR approval remains pending. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: N/A - Stable/default behavior changed: no, old key is kept - Kill switch / disable path: N/A - Unsafe override required: no - Qualification impact: none - Rollback path: revert this PR ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Maintainer edits are enabled on my fork. Please let me know if you prefer a different name for the metric. Thanks. ### Maintainer current-main requalification Exact head `c825041c0fe5065451b01d7c6d618bb4f0a228b5`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Re-reviewed latest integration head |
||
|
|
1fb4cbaea0 |
fix(proxy): astgrep disk verify truncation (#3192)
## Description `_detect_truncation()` currently relies on client-specific truncation banners in `tool_result` text. This works for clients that emit a recognizable line-count banner, but not for clients such as Cursor (byte-based, direction-reversed truncation) and OpenCode (placeholder + external file pointer). Because `headroom proxy` runs alongside the client it wraps, it can optionally verify the returned content against the actual file on disk. If the `tool_result` is a strict prefix of the file, we can identify truncation without depending on client-specific output formatting. This adds disk verification as an opt-in fallback: the existing banner-based detection always runs first, and disk verification is only attempted when no banner is found. All disk-verification failures or ambiguous cases return `UNKNOWN` rather than guessing. Following review, the verifier is also constrained to a trusted, confined workspace root rather than trusting caller-supplied input directly — see the last four bullets under Changes Made. **Current scope:** the session-token workspace binding this fallback requires is wired for `headroom wrap claude` and plugin-loaded `headroom wrap opencode` only. Cursor remains unsupported/not tested — it has no header-injection surface today to transmit an authoritative session token, so disk verification stays `UNKNOWN` for it; extending Cursor support needs a separate, safe transport design. Related to #2430 Closes #2421 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added optional disk-based Read truncation verification behind `HEADROOM_VERIFY_TRUNCATION_ON_DISK`. - Disk verification compares the visible `tool_result` against the file on disk and distinguishes `COMPLETE`, `TRUNCATED`, and `UNKNOWN`. - Disk verification only runs when the existing `_detect_truncation()` banner check finds no truncation marker, avoiding unnecessary file I/O when the client already provides a signal. - Added a `current_cwd` contextvar populated from `x-headroom-cwd` so relative Read paths can be resolved against the client's working directory. - Registered `HEADROOM_VERIFY_TRUNCATION_ON_DISK` as a hot-reloadable runtime setting. - Gated verification on a new `is_current_request_trusted` contextvar, set from `is_loopback_host(request.client.host)` — a server-observed fact, not the caller-supplied header — so `x-headroom-cwd` is never sufficient authority on its own. - Confined all path resolution strictly under the canonicalized `cwd` (`.resolve()` + `relative_to()`), rejecting `..` traversal, absolute-path escapes, and post-resolution symlink escapes. - Required the resolved target to be a regular file and under a byte cap (new `HEADROOM_VERIFY_TRUNCATION_MAX_BYTES` knob, default 5MB), both checked via `fstat` on the open fd before any read. - Opened with `O_NONBLOCK` so a non-regular/special file can't park the read indefinitely; disk I/O already runs off the event loop via the existing compression thread pool. - Added unit and integration tests covering successful verification, missing/unreadable paths, relative paths, content races, feature gating, banner precedence, and the adversarial cases below. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_tool_result_interceptors.py -v 51 passed in 2.21s (includes TestVerifyReadAgainstDisk::* [14 cases, incl. absolute-path escape, `..` traversal, symlink escape, sibling-prefix boundary, non-regular file, oversized file, and untrusted-request-touches-no-disk], plus the end-to-end disk-verification and banner-precedence cases) $ pytest tests/test_proxy_loopback_gating.py -k trusted 16 passed (includes test_record_headroom_stack_binds_trusted_true_for_loopback_client and ..._false_for_non_loopback_client — proves a spoofed x-headroom-cwd from a non-loopback peer does not bind trust) $ ruff check headroom/proxy/interceptors/astgrep.py headroom/proxy/project_context.py \ headroom/proxy/server.py headroom/proxy/runtime_env.py \ tests/test_tool_result_interceptors.py tests/test_proxy_loopback_gating.py All checks passed! $ mypy headroom/proxy/interceptors/astgrep.py headroom/proxy/project_context.py \ headroom/proxy/server.py headroom/proxy/runtime_env.py Success: no issues found in 4 source files ``` ## Real Behavior Proof - Environment: Mac OS local dev machine, no live proxy/network involved. Exercised the real interceptor pipeline (`apply_to_messages`) directly against an actual file on disk, using the same fixture and cwd-binding path a live proxy request would use. - Exact command / steps: wrote a 9-function Python fixture to a temp file, built a client-truncated `tool_result` (first 8 functions, cut mid-file, **no truncation banner text anywhere**), bound `x-headroom-cwd` to the temp directory via `set_current_cwd()`, then called `apply_to_messages()` once with `HEADROOM_VERIFY_TRUNCATION_ON_DISK` unset and once with it set to `1`. - Observed result: With disk verification disabled, the existing ast-grep interceptor produced its normal header without indicating that the input was truncated. With HEADROOM_VERIFY_TRUNCATION_ON_DISK=1, disk verification detected that the 50-line tool_result was a strict prefix of the 59-line file and produced: ```text === HEADROOM_VERIFY_TRUNCATION_ON_DISK unset (default) === [headroom: outlined by ast-grep — 8 definition(s); bodies elided. Re-read the file with a line range to see a specific body.] === HEADROOM_VERIFY_TRUNCATION_ON_DISK=1, x-headroom-cwd bound to the real directory === [headroom: outlined by ast-grep — 8 definition(s) in the visible portion; input was truncated upstream (showing through line 50 of 59 total). Bodies elided. Re-read remaining lines to see more.] ``` **Trust-gate behavior:** same fixture and disk layout, `HEADROOM_VERIFY_TRUNCATION_ON_DISK=1`, but the request context's trust flag left at its default (`False`, simulating a non-loopback caller) instead of being marked trusted. ```text === trusted context (loopback) === [headroom: outlined by ast-grep — 8 definition(s) in the visible portion; input was truncated upstream ...] === untrusted context (non-loopback) — same file, same cwd header, same tool_result === [headroom: outlined by ast-grep — 8 definition(s); bodies elided. Re-read the file with a line range to see a specific body.] (no disk read attempted — verified via a call-recording wrapper around os.open asserting zero invocations) ``` Also verified at the HTTP-middleware level with `TestClient(..., client=("100.90.0.5", 12345))`: a request carrying a spoofed `x-headroom-cwd` header from a non-loopback client IP does not bind trust, regardless of header content. - Not tested: a live `headroom proxy` session with a real coding-agent client; the WebSocket/Codex responses-WS path. ## Runtime Rollout Safety - Rollout-managed feature(s): None. The feature is gated by `HEADROOM_VERIFY_TRUNCATION_ON_DISK` - Minimum rollout channel: N/A. `--intercept-tool-results` must already be enabled for the ast-grep interceptor to run. - Stable/default behavior changed: no - Kill switch / disable path: Unset `HEADROOM_VERIFY_TRUNCATION_ON_DISK`; the setting is hot-reloadable and does not require a restart. - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit, or unset the env var on a running proxy (takes effect immediately, no restart). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes this branch is based on `fix/astgrep-truncation-marker` #2430 not main <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3192" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
be2b205c51 |
fix(proxy): quarantine compression only once half the pool is stuck (#3932)
## Description The timeout-debt quarantine in `HeadroomProxy._run_compression_in_executor` refused every new compression call while even one timed-out worker was still running. On an auto-sized pool (one worker per CPU, 8 on a typical laptop) a single straggler forwarded every other request uncompressed for up to the 60s cap while 7 of 8 workers sat idle. A common trigger is the cold-start fast pass missing its deliberately short 10s fail-open budget. The quarantine exists so timed-out work cannot saturate the executor (#2292, #2360). That risk starts near saturation, not at the first straggler, so the check now engages only once timed-out workers hold half the pool (`max(1, max_workers // 2)`). Pools of 1-3 workers keep today's behaviour. Closes #3931 Related, not closed by this PR: #3711 (what makes one Kompress stage overrun its budget; this PR narrows how far one overrun spreads, not the overrun itself) and #3653 (thread/fd reclamation for stuck workers; see Additional Notes for why this change does not add to it). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/server.py`: new `_compression_quarantine_threshold`, defaulting to `max(1, compression_max_workers // 2)` and overridable with `HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD` (read with the existing `_get_env_int`, `min_value=1`; `=1` restores the old any-straggler behaviour). - The quarantine check and its 60s time-cap release both key on `timed_out_in_flight >= threshold` instead of `> 0`. The cap is still re-armed on every fresh timeout, as before. - `quarantine_activations_total` (and the `activated` Prometheus event and warning) now fire on the transition into the quarantined state, i.e. when debt reaches the threshold, or when a fresh timeout re-arms it after the time cap released it. Previously a re-arm after a release while debt still stood was not counted. - "Quarantine cleared" is logged when debt drops back below the threshold, with the number of timed-out workers still running. - `/health` `runtime.compression_executor.quarantine_active` now reports the condition the check enforces (debt at or above the threshold and the cap not lapsed). It used to report `timed_out_workers > 0`, which was already wrong after a time-cap release. - Tests: `test_quarantine_waits_for_half_the_pool_to_be_stuck` (8-worker pool: 1 and 3 stuck still compress, 4 stuck refuses, `/health` matches, re-arm after release counts as a second activation) and `test_quarantine_threshold_is_half_the_pool_unless_overridden` (default sizing and the env override). Two tests in `tests/test_tokenizer_count_offload.py` that forced a quarantine with a debt of 1 on an auto-sized pool now set the debt to the threshold. ## Testing - [x] Unit tests pass (`pytest`) - targeted files only, listed below; the full suite was not run - [x] Linting passes (`ruff check .`) - on the changed files, plus `ruff format --check` - [x] Type checking passes (`mypy headroom`) - `mypy headroom/proxy/server.py` - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output Before the fix (new tests against unmodified `server.py`; the two tokenizer tests fail only because they read the new attribute): ```text $ python -m pytest tests/test_proxy_compression_executor.py tests/test_tokenizer_count_offload.py -q -p no:cacheprovider FAILED tests/test_proxy_compression_executor.py::test_quarantine_waits_for_half_the_pool_to_be_stuck FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[8-None-4] FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[3-None-1] FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[1-None-1] FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[8-1-1] FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[8-0-4] FAILED tests/test_tokenizer_count_offload.py::test_count_tokens_offloaded_fails_open_on_executor_quarantine FAILED tests/test_tokenizer_count_offload.py::test_count_tokens_offloaded_returns_count_text_capable_tokenizer =================== 8 failed, 20 passed, 1 warning in 20.73s =================== ``` The behavioural failure on main is: ```text tests/test_proxy_compression_executor.py:336: in test_quarantine_waits_for_half_the_pool_to_be_stuck E headroom.proxy.server.CompressionQuarantinedError: compression quarantined: 1 timed-out worker(s) still running ``` After the fix: ```text $ python -m pytest tests/test_proxy_compression_executor.py tests/test_tokenizer_count_offload.py tests/test_platform_stabilization_functional.py tests/test_prometheus_obs_counters.py -q -p no:cacheprovider ======================== 51 passed, 1 warning in 28.02s ======================== $ ruff check headroom/proxy/server.py tests/test_proxy_compression_executor.py tests/test_tokenizer_count_offload.py All checks passed! $ ruff format --check headroom/proxy/server.py tests/test_proxy_compression_executor.py tests/test_tokenizer_count_offload.py 3 files already formatted $ mypy headroom/proxy/server.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.11, upstream main `d5318ac2` (before) and the same commit plus this branch (after). Proxy booted through `headroom.proxy.server.run_server(ProxyConfig(port=..., compression_max_workers=8, disable_kompress=True, cache_enabled=False, rate_limit_enabled=False, log_requests=False))`, which is what `headroom proxy` calls after argument parsing (the local venv lacked the `mcp`/`magika`/`onnxruntime` extras that the CLI's dependency preflight requires). Env: `HEADROOM_SKIP_UPSTREAM_CHECK=1 HEADROOM_COMPRESSION_TIMEOUT_SECONDS=3 HEADROOM_TELEMETRY=off`, a fresh `HEADROOM_WORKSPACE_DIR` per run. - Exact command / steps: over HTTP, one warm-up `/v1/compress` (200-row JSON `tool_result`, 10,623 tokens), one oversized `/v1/compress` (40,000 rows, about 22s of work against the 3s budget, so its worker keeps running after the request fails open), then three small `/v1/compress` calls while that worker is still running, reading `/health` `runtime.compression_executor` in between. A second run fires four oversized calls concurrently to strand 4 of 8 workers, sends a small call, waits until one stuck worker exits, and sends another. - Observed result: on main one straggler refuses every following request; on this branch the other 7 workers keep compressing, and half the pool stuck still quarantines and clears again (output below). Before (main), one straggler refuses everything: ```text health after big timed out: {'max_workers': 8, 'running': 1, 'timed_out_workers': 1, 'quarantine_active': True, 'quarantine_activations_total': 1, 'quarantine_skips_total': 0} small request 1: 0.02s tokens_before=0 tokens_saved=0 skipped=True reason=compression_timeout small request 2: 0.01s tokens_before=0 tokens_saved=0 skipped=True reason=compression_timeout small request 3: 0.01s tokens_before=0 tokens_saved=0 skipped=True reason=compression_timeout health after small requests: {'max_workers': 8, 'running': 1, 'timed_out_workers': 1, 'quarantine_active': True, 'quarantine_activations_total': 1, 'quarantine_skips_total': 3} ``` After, the same straggler leaves the other 7 workers serving: ```text health after big timed out: {'max_workers': 8, 'running': 1, 'timed_out_workers': 1, 'quarantine_active': False, 'quarantine_activations_total': 0, 'quarantine_skips_total': 0} small request 1: 0.12s tokens_before=10623 tokens_saved=4411 skipped=False reason=None small request 2: 0.12s tokens_before=10623 tokens_saved=4411 skipped=False reason=None small request 3: 0.18s tokens_before=10623 tokens_saved=4411 skipped=False reason=None health after small requests: {'max_workers': 8, 'running': 1, 'timed_out_workers': 1, 'quarantine_active': False, 'quarantine_activations_total': 0, 'quarantine_skips_total': 0} ``` After, half the pool stuck still quarantines, and it clears when debt drops below half: ```text health after 4 concurrent big requests timed out: {'max_workers': 8, 'running': 4, 'timed_out_workers': 4, 'quarantine_active': True, 'quarantine_activations_total': 1, 'quarantine_skips_total': 0} small request while 4 of 8 stuck: 0.10s tokens_before=0 tokens_saved=0 skipped=True reason=compression_timeout health once a stuck worker exited: {'max_workers': 8, 'running': 3, 'timed_out_workers': 3, 'quarantine_active': False, 'quarantine_activations_total': 1, 'quarantine_skips_total': 1} small request with 3 stuck: 0.19s tokens_before=10623 tokens_saved=4411 skipped=False reason=None ``` ```text WARNING - Compression worker exceeded its request deadline and is still running; new compression is quarantined until fewer than 4 timed-out workers remain INFO - Compression quarantine cleared; 3 timed-out worker(s) still running ``` - Not tested: the `headroom proxy` CLI entry point itself (booted via `run_server`, see Environment); the `/v1/messages`, `/v1/responses` and Codex WS paths end to end (they share the same `_run_compression_in_executor`, but I only drove `/v1/compress`); Windows and Linux (the field evidence covers them, my repro does not); the 60s time-cap release over HTTP (covered by `test_compression_quarantine_releases_after_time_cap` and the new re-arm assertion, not by a live run); a long soak sampling OS thread and fd counts. ## Runtime Rollout Safety - Rollout-managed feature(s): none. The quarantine is not in `rollout.py` FEATURES. - Minimum rollout channel: n/a (applies on every channel). - Stable/default behavior changed: yes. On pools of 4 or more workers, compression keeps running while fewer than half the workers are held by timed-out jobs; previously the first timed-out job refused all compression for up to 60s. Pools of 1-3 workers are unchanged. - Kill switch / disable path: `HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD=1` restores the previous behaviour on any pool size. - Unsafe override required: no. - Qualification impact: more requests get compressed while a straggler runs, so `tokens_saved` rises and the `skipped` quarantine counter falls on affected installs; no change to what compression does to a request it runs on, and no change to provider cache behaviour beyond the requests that are now compressed instead of forwarded raw. - Rollback path: set `HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD=1`, or revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` - it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Pool capacity and #3653. The compression executor is a `ThreadPoolExecutor` with a fixed `max_workers`; it never creates threads beyond that cap, and a stuck worker keeps the thread it already holds. Admitting work while a straggler runs only puts that work on the pool's idle threads, which finish and release their resources normally, so this change cannot raise the process's thread count above `max_workers` or add a leaked thread/fd per quarantine event. What it does change is how many timed-out jobs can stand before new work is refused: up to `threshold - 1` (3 on an 8-worker pool) instead of 0. That bound is no looser than today's in practice: the old rule never prevented requests that were already admitted from timing out together (it only refuses calls that arrive after a timeout is recorded), and the 60s cap already lets work back in while debt stands. At `threshold` stuck workers the quarantine engages exactly as before, so at least half the pool stays free for work that is not stuck. Field evidence that motivated this (Headroom Desktop fleet canary, 2026-09-21 to 2026-09-23): 9 hosts on macOS, Windows and Linux had between 2.1% and 38.9% of requests refused compression by the quarantine. On one machine's proxy logs, 211 of 262 refusals happened with ONE stuck worker and none with more than 3 of 8 stuck; on 0.37.0 that machine forwarded 42 of 180 requests (23%) uncompressed in a day while `/health` showed `max_workers` 8, `queued_max` 1 and `queue_wait_seconds_max` 0.23, i.e. an idle pool. Headroom Desktop has shipped an equivalent downstream patch (standing the quarantine down below half the pool) since 2026-09-23 (desktop 0.9.22-rc.1); this PR implements it in the check itself so it can be dropped there. Documentation: `HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD` follows the existing undocumented `HEADROOM_COMPRESSION_QUARANTINE_MAX_SECONDS`, so no docs page was updated; happy to add both to the proxy env reference if wanted. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3932" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
f90a56b038 |
fix(proxy): F3 — per-tenant TOIN learning key (#404)
## Description
This change isolates TOIN learning by tenant in shared-proxy
deployments. Each pattern key gains a tenant namespace derived from a
configurable sanitized header, a stable auth/token hash, or the
backwards-compatible global namespace. Legacy keys migrate to global and
every resolution path emits a structured event.
## Threat model
Pre-F3, every request's tool-call patterns aggregated into one global
pool keyed by `(auth_mode, model_family, sig_hash)`. That works for an
OSS single-tenant deploy, but in a multi-tenant SaaS / shared-proxy
deploy:
1. **Cross-tenant pattern leakage.** Tenant A's compression patterns
silently train recommendations the proxy then applies to tenant B's
requests, degrading both tenants' compression quality.
2. **Structural fingerprint exposure.** TOIN dumps include
`tool_signature_hash` per slice. With one shared pool, a user with read
access to a dump file learns the structural fingerprint of every other
tenant's tool surface.
F3 partitions: every TOIN store key gets a `tenant_key` prefix derived
from the inbound request. Two distinct tenants therefore train two
isolated pattern pools.
## Tenant key derivation rules
| Source | When | Key derivation |
|------------|--------------------------------------------------------------|-------------------------------------------------------------------|
| `header` | `HEADROOM_TENANT_KEY_HEADER` (default
`X-Headroom-Tenant-ID`) is present and survives sanitization | Sanitized
header value (ASCII alphanum + `-_`, max 64 chars) |
| `hash` | No header but bearer token + `auth_mode` (from F1) present |
`sha256(f"{auth_mode}:{api_key_prefix}")[:24]` (api_key_prefix = first 8
chars of bearer); same SHA-256[:24] idiom as PR #395 |
| `global` | Neither header nor bearer present (OSS single-tenant / dev)
| Literal `"global"` namespace; pre-F3 patterns migrate here on first
load |
Header path takes precedence over hash. Header values that sanitize to
empty (pure unicode / control chars only) fall through to the next rule.
## Configuration
Single env var: `HEADROOM_TENANT_KEY_HEADER`. Default
`X-Headroom-Tenant-ID`. The header NAME is configurable; the
sanitization rules and the `"global"` literal are NOT — they're
load-bearing for namespace isolation.
## Structured log fields
Every resolution path emits exactly one `tenant_key_resolved` log event
(no silent fallbacks per `feedback_no_silent_fallbacks.md`):
| Field | Always present | Notes |
|--------------|----------------|------------------------------------------------|
| `event` | yes | Always `"tenant_key_resolved"` |
| `tenant_key` | yes | Resolved key (or `"global"`) |
| `source` | yes | One of `header` / `hash` / `global` |
| `header` | header path | Header name that matched |
| `auth_mode` | hash path | `payg` / `oauth` / `subscription` |
Operators can alert on a sustained `source="global"` rate to spot
mis-configured upstream auth proxies.
## Implementation
- **NEW** `headroom/proxy/tenant_key.py`: pure resolver. Public surface
= `resolve_tenant_key(request) -> tuple[str, str]`. Plus
`set_request_tenant_key()` / `get_current_tenant_key()` ContextVar pair
so TOIN's deep-stack `record_compression` calls observe the right tenant
slice.
- `headroom/telemetry/toin.py`: extends `PatternKey` from a 3-tuple to a
4-tuple `(tenant_key, auth_mode, model_family, sig_hash)`.
`_deserialize_pattern_key` is backward-compat with both pre-F3 formats
(3-tuple from PR-B5, bare `sig_hash` from v1.0). `record_compression` /
`record_retrieval` / `get_pattern` accept an optional `tenant_key`
kwarg; when absent they read from the ContextVar.
- `headroom/cli/toin_publish.py`: each `[[recommendation]]` row now
carries `tenant_key`; rows sort by `(tenant_key, auth_mode,
model_family, sig_hash)` for clean diffs.
- `headroom/proxy/handlers/anthropic.py` +
`headroom/proxy/handlers/openai.py`: call `resolve_tenant_key(request)`
once at request entry, right after the F1 `classify_auth_mode` block.
Three call sites total (1 Anthropic, 2 OpenAI). Result attached to
`request.state.tenant_key` / `request.state.tenant_key_source` AND set
on the ContextVar.
## Realignment build constraints (per project memory)
- **configurable**: `HEADROOM_TENANT_KEY_HEADER` env var.
- **no hardcodes**: header name + `"global"` literal are the only
constants; both motivated and documented in code.
- **no regexes**: sanitization uses string ops +
`unicodedata.category()`.
- **no silent fallbacks**: every resolution emits a structured log
event.
- **comprehensive tests**: header / hash / global paths + sanitization
edges + tenant-isolation property test.
- **structured logs**: `event=tenant_key_resolved` with `tenant_key`,
`source`, optional `header` / `auth_mode`.
## Test plan
- [x] `pytest tests/test_tenant_key.py` (29 cases — header / hash /
global paths, structured-log emission, sanitization edges, ContextVar
round-trip)
- [x] `pytest tests/test_toin_observation_only.py` (6 new isolation
cases — two-tenant divergence, ContextVar read, explicit kwarg
precedence, default migration to `"global"`)
- [x] `pytest tests/test_toin*.py` (existing TOIN tests updated for the
4-tuple key shape)
- [x] `pytest tests/test_proxy_tenant_key_integration.py` (4 cases —
handler wires `request.state.tenant_key` + ContextVar through all three
resolution paths)
- [x] `pytest tests/test_critical_fixes.py tests/test_critical_gaps.py
tests/test_proxy_ccr.py` (legacy 3-tuple references updated)
- [x] `make ci-precheck-python` — 176 tests pass (passed via pre-push
hook)
- [x] `make ci-precheck-commitlint` — clean (passed via pre-push hook)
Refs: F1 (PR #399, auth_mode classifier), F2.1 (PR #400, per-auth-mode
CompressionPolicy gates).
## Type of Change
- [x] Bug fix / tenant-isolation hardening
- [ ] New feature
- [ ] Breaking change
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring
## Changes Made
- Add a pure tenant-key resolver and request ContextVar.
- Extend TOIN pattern keys and published recommendations with a tenant
namespace while preserving legacy deserialization.
- Resolve and attach tenant identity at Anthropic and OpenAI request
entry.
- Add isolation, migration, sanitization, header, CCR, handler, and
executor coverage.
## Testing
- [x] Focused tenant/TOIN/proxy tests pass
- [x] Python CI precheck passes
- [x] Commitlint precheck passes
### Test Output
```text
176 focused/precheck tests passed
Current-main isolation, migration, header, CCR, and executor validation: green
```
## Real Behavior Proof
- Environment: current-main reconciled Python proxy/TOIN test
environment.
- Exact command / steps: ran tenant resolver, TOIN observation/publish,
handler integration, header-isolation, CCR, and compression-executor
suites.
- Observed result: distinct tenants train isolated pattern pools; legacy
patterns migrate to global; request handlers propagate the resolved
namespace through deep TOIN calls.
- Not tested: production multi-tenant traffic volume or an external TOIN
dump consumer.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
---------
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
517bf992cf |
fix(proxy): quarantine compression while timed-out workers run (#2292)
## Description A request-side `asyncio.wait_for()` timeout stops waiting, but it cannot preempt an executor thread that already started. The proxy counted those late workers and still admitted more compression, so repeated slow calls could consume the whole compression pool and charge every request another full timeout. This change tracks running post-timeout workers as timeout debt and quarantines request-path compression while that debt is non-zero. New attempts raise `CompressionQuarantinedError` before executor admission, using an `asyncio.TimeoutError` subclass so Python 3.10 handlers apply the existing compression-failure policy. Quarantine clears automatically after all known timed-out workers genuinely exit. Mitigates #946 and #810. It does not attempt to kill the first running thread; Python cannot safely preempt it. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Track started, finished, timed-out, and debt-recorded state under the existing compression metrics lock. - Reject new compression before enqueue while timed-out workers remain; clear quarantine on the final worker exit. - Preserve queued-timeout behavior: work cancelled before worker start does not activate quarantine; a cancellation/start race is conservatively tracked as running debt. - Add `/health` and `/stats` runtime fields for quarantine state, worker debt, activations, and skips. - Add `headroom_compression_quarantine_total{event="activated|skipped"}` Prometheus counters. - Add regression, recovery, queue-race, runtime-payload, export, reset, and Python 3.10 exception-class coverage. - Update `CHANGELOG.md`; no dependency or lockfile changes. ## Reproduction On base commit `718c8dc5`, I applied only the new regression test and ran: ```bash .venv/bin/pytest -q \ tests/test_proxy_compression_executor.py::test_timeout_quarantines_new_work_until_timed_out_worker_finishes ``` The first worker timed out but remained blocked. The second callable entered the executor instead of being rejected: ```text FAILED: DID NOT RAISE TimeoutError ``` ## Testing - [x] Affected unit tests pass - [x] Linting passes (`ruff check .`) - [x] Changed-file type checking passes - [x] New tests added for the fix - [x] Manual testing performed ### Test Output ```text Python 3.10.20 $ .venv/bin/pytest -q -m 'not slow' \ tests/test_proxy_compression_executor.py \ tests/test_prometheus_obs_counters.py \ tests/test_proxy/test_compression_failure_action.py \ tests/test_proxy/test_compression_timeout_config.py \ tests/test_anthropic_pre_upstream_backpressure.py \ tests/test_openai_codex_ws_lifecycle.py \ tests/test_codex_ws_compression_scheduler.py \ tests/test_gemini_compression_offload.py \ tests/test_proxy_handlers_batch.py \ tests/test_tokenizer_count_offload.py \ tests/test_cold_start_fast_pass.py 125 passed, 1 skipped, 1 deselected, 1 warning in 11.06s $ .venv/bin/ruff check . All checks passed! $ .venv/bin/ruff format --check . 1310 files already formatted $ .venv/bin/mypy headroom/proxy/server.py headroom/proxy/prometheus_metrics.py Success: no issues found in 2 source files $ git diff --check # no output ``` The warning is the existing Starlette `TestClient`/`httpx` deprecation warning. ## Real Behavior Proof - Environment: macOS 15.7.4 x86_64, Python 3.10.20, `compression_max_workers=2`, direct proxy executor path, no external provider/model. - Exact command / steps: instantiate the proxy; run a blocking compression callable with a 50 ms timeout; immediately attempt a second callable and time the rejection; release the first worker; wait for debt to reach zero; run the second callable again; export Prometheus metrics. - Observed result: the first request timed out at 51.182 ms; the second attempt was rejected in 0.014 ms and its callable never started; debt was 1 while quarantined, returned to 0 after release, and compression then resumed normally. ```json { "after_release": { "activations_total": 1, "leaked_threads_total": 1, "quarantine_active": false, "skips_total": 1, "timed_out_workers": 0 }, "bypass_elapsed_ms": 0.014, "bypass_error": "compression quarantined: 1 timed-out worker(s) still running", "during_quarantine": { "quarantine_active": true, "timed_out_workers": 1 }, "first_timeout_elapsed_ms": 51.182, "prometheus": [ "headroom_compression_quarantine_total{event=\"activated\"} 1", "headroom_compression_quarantine_total{event=\"skipped\"} 1" ], "resumed_result": "resumed", "second_callable_started_during_quarantine": false } ``` - Not tested: a live external model/provider; forced termination of a permanently wedged native worker; the marked slow native scheduler benchmark. A broad non-slow run collected 9,859 selected tests but was stopped at `tests/test_adversarial_grid.py::TestRunGrid::test_grid_shape_and_schema` after a macOS process sample showed the pre-existing native `_core.abi3.so` semaphore stall (`_dispatch_semaphore_wait_slow` → `semaphore_wait_trap`). The affected executor/handler slice above completed cleanly. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented hard-to-understand concurrency paths - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove the fix is effective - [x] New and affected existing unit tests pass locally - [x] I have updated the CHANGELOG.md --------- Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
|
0a3851b240 |
perf(proxy): cap compression workers to CPU count (#1803)
## Description The request-path compression executor currently uses asyncio-style I/O sizing for CPU-bound Kompress work. When `compression_max_workers` is unset, `HeadroomProxy.__init__` resolves the pool to `min(32, cpu * 4)`, so an eight-core host can run 32 simultaneous compression workers that all contend for real CPU. This changes only the automatic request-path default to one worker per reported CPU while preserving the existing explicit override path from `--compression-max-workers` and `HEADROOM_COMPRESSION_MAX_WORKERS`. Closes #1635 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Cap the automatic request-path compression executor default at `max(1, os.cpu_count() or 1)`. - Preserve explicit `compression_max_workers` values, including the existing clamp to at least one worker. - Keep CLI help, `ProxyConfig` comments, and nearby test documentation aligned with the CPU-bound default. - Update the focused compression executor regression so the default contract documents CPU-bound sizing, and keep the existing Codex compression stress guard stable when p50 rounds to zero. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_codex_ws_compression_scheduler.py tests/test_proxy_compression_executor.py tests/test_cli_proxy_improvements.py::TestCompressionMaxWorkers -q`) - [x] Linting passes (`uv run ruff check headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/server.py tests/test_cli_proxy_improvements.py tests/test_proxy_compression_executor.py tests/test_codex_ws_compression_scheduler.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_codex_ws_compression_scheduler.py tests/test_proxy_compression_executor.py tests/test_cli_proxy_improvements.py::TestCompressionMaxWorkers -q 16 passed, 1 skipped, 1 warning in 6.13s $ uv run ruff check headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/server.py tests/test_cli_proxy_improvements.py tests/test_proxy_compression_executor.py tests/test_codex_ws_compression_scheduler.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows, Python environment from `uv sync --extra dev`, no provider credentials needed. - Exact command / steps: construct `HeadroomProxy` with `compression_max_workers=None`, inspect `proxy.compression_max_workers` and `/health` `runtime.compression_executor`. - Observed result: the automatic request-path pool resolves to reported CPU count, while explicit overrides still resolve to the configured value and report `source: explicit`. - Not tested: multi-session wall-clock benchmark under live Kompress load. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes No `CHANGELOG.md` edit: this repo generates changelog entries from conventional commits. This intentionally does not touch the background compression executor surface covered by #1633. |
||
|
|
15ac650d40 |
fix(proxy): fail open when kompress saturation would exhaust pre-upstream budget (#1430)
## Description
Concurrent Anthropic `/v1/messages` traffic can still exhaust Headroom's
pre-upstream budget because Kompress ONNX execution waits on the request
critical path. When Kompress saturates, requests eventually fail with
`503 pre-upstream queue saturated` even though compression can safely
degrade to passthrough.
This PR makes Kompress saturation fail open on the Anthropic hot path,
so requests continue uncompressed when compression capacity is under
pressure. It keeps the executor and stage-timing evidence intact, and it
preserves blocking model-load validation so runtime pressure does not
silently skip the validation path.
Closes #1025
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- add a bounded execution-slot acquire path so Anthropic requests fail
open to passthrough when Kompress saturation would consume the
pre-upstream budget
- preserve explicit execution-timeout counters and Anthropic
passthrough/stage-timing observability instead of hiding the pressure
path
- keep `_validate_pytorch_device()` on blocking acquire semantics so
model-load validation still waits for capacity instead of failing open
- make the blocking validation acquire explicit to `mypy` without
changing runtime behavior
- extend focused regressions for pre-upstream backpressure, Kompress
saturation, execution-skip observability, and validation waiting
- align the CLI timeout help text and `ProxyConfig` comment with the
fail-open runtime behavior
- update `CHANGELOG.md` for the proxy runtime fix
## Testing
- [x] Unit tests pass (`uv run pytest
tests/test_anthropic_pre_upstream_backpressure.py
tests/test_proxy_compression_executor.py
tests/test_kompress_request_nonblocking.py`)
- [x] Linting passes (`uv run ruff check
tests/test_anthropic_pre_upstream_backpressure.py` and `uv run ruff
format tests/test_anthropic_pre_upstream_backpressure.py --check`)
- [x] Type checking passes (`uv run mypy headroom
--ignore-missing-imports`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed
### Test Output
```text
Focused local validation passed:
- uv run pytest tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_compression_executor.py tests/test_kompress_request_nonblocking.py -x -v
37 passed, 1 warning in 12.01s
- uv run ruff check headroom/proxy/handlers/anthropic.py headroom/transforms/kompress_compressor.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_compression_executor.py tests/test_kompress_request_nonblocking.py
All checks passed!
- uv run ruff format headroom/proxy/handlers/anthropic.py headroom/transforms/kompress_compressor.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_compression_executor.py tests/test_kompress_request_nonblocking.py --check
5 files already formatted
- uv run mypy headroom --ignore-missing-imports
Success: no issues found in 398 source files
Base-branch proof on origin/main (
|
||
|
|
e9cae0131b |
fix: expose compression latency bottlenecks
Add Codex WS unit-level timing and bounded parallel compression, clarify context-tool session savings, and avoid costly diff/log fallbacks to Kompress. |
||
|
|
ea78cf6252 |
fix(proxy): cache concurrency lock, multi-worker docs, bounded compression executor
Three audit follow-ups from issue #327's deep-dive review. C1 — CompressionCache concurrency lock ====================================== `CompressionCache` instances are shared per `session_id` and accessed from async-dispatched threadpool workers. Pre-fix, concurrent requests for the same session raced on `_cache`, `_stable_hashes`, `_first_seen`, and `_total_tokens_saved` with no synchronization. Observable failures: * Lost-update on `_total_tokens_saved` (read-modify-write). * `RuntimeError: OrderedDict mutated during iteration` from `apply_cached` when a concurrent `store_compressed` evicts during the walk. * Lost stable-hash records — next-turn compute_frozen_count reads inconsistent state. May also explain part of SvenMeyer's `_cache: 0 entries / 1003 misses` observation: the cache was being clobbered concurrently. Added `threading.RLock` guarding all mutating methods. `RLock` (not `Lock`) so future code can call locked methods from inside another locked method without self-deadlock. Also locked `HeadroomProxy._compression_caches` dict-of-caches access via a separate `_compression_caches_lock` so two concurrent calls for the same session_id can't each create distinct CompressionCache objects (which would split the cache state between them). The `/stats` endpoint snapshots the cache list under the dict lock before iterating to avoid eviction-during-iteration. C2 — Multi-worker CCR fragmentation: documented + startup warning ================================================================= The in-memory `InMemoryCcrStore` (Rust), `_compression_caches` (Python), `session_tracker_store` (Python), and TOIN learner state are ALL per-process. Multi-worker uvicorn round-robins requests across workers, so a session whose turn-1 lands on worker A may have turn-2 land on worker B. Worker B has zero knowledge of A's CCR markers, replay cache, or prefix-cache state. Result: `Retrieve original: hash=X` markers stay in-context as opaque directives, every fresh tool_result is recompressed from scratch, and `frozen_message_count=0` causes Anthropic prefix-cache busts on every cross-worker turn. Added a "Multi-worker deployment — CCR fragmentation" section in `RUST_DEV.md` documenting the failure modes, the supported configuration (`--workers 1`), and the sticky-session workaround for horizontal scale. The proxy emits a `WARNING`-level log line on startup if `workers > 1` is detected, pointing at the doc section. C3 — Bounded compression executor with cancel-aware metrics =========================================================== `asyncio.wait_for(asyncio.to_thread(pipeline.apply), timeout=...)` cancellation does NOT propagate into the threadpool worker that's running Rust code. Once the worker has picked up the task, `concurrent.futures.Future.cancel()` returns False and the thread runs to completion. Stuck threads accumulated invisibly on asyncio's default executor, contending with unrelated `to_thread` callers (file IO, etc.). Replaced all 7 `asyncio.to_thread` call sites for `pipeline.apply()` across `proxy/handlers/anthropic.py` (3) and `proxy/handlers/openai.py` (4) with a new `HeadroomProxy._run_compression_in_executor(fn, *, timeout)` helper that: 1. Submits to a dedicated bounded `ThreadPoolExecutor` named `headroom-compress` (configurable via `ProxyConfig.compression_max_workers`; defaults to `min(32, (cpu_count or 1) * 4)`). 2. Increments `_compression_in_flight` (gauge) when work starts and decrements when work completes; tracks `_compression_in_flight_max` as a high-water mark. 3. Detects "leaked threads" by comparing wall-clock elapsed against the timeout in the worker's `finally` block. Increments `_compression_leaked_threads` when a worker finishes after its asyncio future was cancelled. Operators can see the leaked-thread rate climbing in `/stats runtime.compression_executor` BEFORE the pool fills up. Tests ===== * `TestCompressionCacheConcurrency` (3 tests) — many threads store_compressed / apply_cached / update_from_result on a single CompressionCache; assert no exceptions, no lost updates, no partial state. * `test_get_compression_cache_returns_same_instance_under_contention` — 32 concurrent `_get_compression_cache(same_id)` calls return the identical instance (would split pre-lock). * `test_proxy_compression_executor.py` (8 tests) — pool size respects config, in-flight gauge tracks running compressions, high-water mark is monotonic, timeout propagates to awaiter, leaked-thread counter increments on post-deadline completion, `/stats` surfaces all three gauges. Verification ============ * All 123 targeted regression tests pass. * `make ci-precheck` clean. * No `Co-Authored-By` trailer; conventional `fix:` prefix; no `--no-verify`. |