Commit Graph
5 Commits
Author SHA1 Message Date
97e7bc3333 test(proxy): make compression timeout regression deterministic (#4587)
## Description

Make the compression-timeout regression independent of host speed. The
old test required the entire HTTP request, including outcome
bookkeeping, to finish within 500 ms. Hosted runs show the unchanged
production path can exceed that limit after the simulated compression
timeout has already occurred.

## Type of Change

- [x] Test-only change; no production behavior change

## Changes Made

- Assert that the compression executor is awaited exactly once.
- Fail if timeout handling retries compression inline or recounts
tokens.
- Retain the fail-open response checks: HTTP 200, unchanged messages,
zero token counts/savings, unchanged compression ratio, and
compression-timeout metadata.
- Replace the unrelated whole-request wall-clock assertion with those
behavioral checks.
- Reword Copilot's generated initial-plan commit to satisfy Conventional
Commits; all file trees and contributor authorship were preserved.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)

- Local Windows/CPython 3.12: `G:/git/headroom/.venv/Scripts/python.exe
-m pytest tests/test_platform_stabilization_functional.py -q --tb=short`
— five passed, one existing Starlette deprecation warning, using the
existing native extension.
- Current head `14404162810068d2c4a82b8eeebaab1b313aa3da`: 32 successful
hosted checks and 12 skips; no failures or running checks at
qualification.
- The test source is byte-identical to the locally tested original
implementation commit; the subsequent changes only merge main and repair
commit metadata.

## Real Behavior Proof

- Environment: GitHub-hosted Linux CI plus the local Windows test
environment above; no provider credentials or paid inference.
- Exact command / steps: run the functional-test command in Testing,
then inspect CI run 41104791110 on the current head, including all four
Python shards.
- Observed result: five local functional tests passed; the current
head's hosted checks completed with 32 successes, 12 skips, and no
failures.
- Observed input to this fix: the prior timeout assertion failed at
approximately 1.0–1.2 seconds against a 0.5-second limit on several
otherwise unrelated PRs whose proxy implementation matched main.
- After-fix evidence: the current full hosted test run succeeds with the
deterministic timeout contract assertions in place. Local functional
tests also pass.
- Scope: this is an internal maintainer-coauthored test correction. The
evidence establishes the regression-test contract; it is not a new
live-provider or live-Claude session qualification. No production
timeout, deadline, or bookkeeping implementation changed.
- Not tested: live provider traffic or a fresh native rebuild.

## Runtime Rollout Safety

- Rollout-managed feature: none; tests only.
- Minimum rollout channel: not applicable; no runtime feature is added
or enabled.
- Stable/default behavior changed: no.
- Kill switch/rollback: revert this test change; production behavior is
unaffected.
- Rollback path: revert the test-only commit while retaining production
timeout handling.
- Unsafe override required: no.
- Qualification impact: replaces host-speed sensitivity with explicit
fail-open behavior checks.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Automated code qualification does not constitute final human merge
approval. Nothing has been merged, tagged, or published.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 08:41:55 +05:30
Abhay Singh c5a08d22e0 fix(proxy): time-cap the compression timeout-debt quarantine (#2360) (#2412)
## Description

Fixes #2360.

The proxy runs compression on a bounded thread-pool executor with a
per-request deadline. Because Python cannot preempt a worker after its
`asyncio.wait_for` times out, the code quarantines new compression while
a timed-out worker is still running (`_compression_timed_out_in_flight >
0`), to avoid piling more work onto a saturating executor.

The gap: that counter only decrements when the worker finally exits. A
worker that **never returns** — a hung or pathological compression of a
large frame — keeps the counter above zero forever, so the quarantine
stays open permanently and every subsequent compression raises
`CompressionQuarantinedError`. On Codex WS this is exactly what #2360
reports: one 5s timeout, then Token Savings pinned at ~0% with no
recovery, even though the machine is fine.

The "parity with direct upstream" nature of the accounting was correct;
the only missing piece is an upper bound on how long a single stuck
worker may hold the quarantine.

## Fix

Add a time cap on the quarantine:

- A deadline (`_compression_quarantine_deadline`) is (re)armed on every
fresh timeout, to `now + HEADROOM_COMPRESSION_QUARANTINE_MAX_SECONDS`
(default **60s**).
- The gate quarantines only while `timed_out_in_flight > 0` **and** `now
< deadline`. Once the deadline lapses with no new timeouts, the worker
is presumed leaked/abandoned and compression resumes. The release is
counted once (a `"released"` quarantine metric + a warning), and the
deadline is cleared so it is not re-counted on every later request.
- The bounded executor still caps thread growth, and any new timeout
re-arms the quarantine, so ongoing genuine slowness keeps quarantining
while a single hung worker cannot pin it forever.

This preserves the original protection (a burst of slow compressions
still quarantines) while guaranteeing recovery.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/server.py`: add `_compression_quarantine_deadline` /
`_compression_quarantine_max_seconds` (from
`HEADROOM_COMPRESSION_QUARANTINE_MAX_SECONDS`, default 60s) and
`_compression_quarantine_releases`; arm the deadline when timeout debt
is recorded; release the quarantine (once) in the gate when the deadline
lapses.
- `tests/test_platform_stabilization_functional.py`: add a test that a
standing timed-out worker quarantines within the cap and releases
(running compression again, counted once) past it.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/proxy/server.py tests/test_platform_stabilization_functional.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/server.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` here imports the ML stack and OOMs this
box, so I modeled the gate/deadline state machine with a dependency-free
script and left the added `create_app` test to CI.
- Exact command / steps: simulated a standing timed-out worker, then
exercised the gate at times within the cap, past the cap, and after a
fresh timeout, plus a normal worker exit.
- Observed result: within the cap the gate quarantines (raises); past
the cap it releases exactly once and then lets compression run; a new
timeout re-arms the quarantine; a normal worker exit clears the debt.
Matching the added handler test (`_run_compression_in_executor` raises
`CompressionQuarantinedError` within the cap and returns the callable's
result past it, with `_compression_quarantine_releases == 1`).
- Not tested: a live Codex WS session hanging a real worker; the added
test drives `_run_compression_in_executor` directly with the quarantine
state set.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The default cap (60s) is deliberately well above a normal
slow-but-completing compression so the original saturation protection is
unchanged in practice; it only ever fires for a worker that has run far
past its deadline. Tunable via
`HEADROOM_COMPRESSION_QUARANTINE_MAX_SECONDS`. The `"released"`
quarantine metric and a one-time warning make the recovery observable.
The "unit tests pass locally" box is unchecked because the added
`create_app` test imports the ML stack (OOM on this box); it runs under
the normal CI job, and the state machine is verified by the standalone
proof above.
2026-08-12 00:05:56 -05:00
Alex SunandJD Davis 685ebe457d fix(ccr): report embedded hashes from compress endpoint (#717)
## Description

Fixes `/v1/compress` so its `ccr_hashes` response includes retrievable
CCR hashes embedded in compressed message content, including row-drop
and recursive JSON markers that may not be present in
`TransformResult.markers_inserted`.

The original PR also changed query-based JSON row search. Current `main`
intentionally made CCR retrieval a hash-only, full-content lookup in
#1532, so that obsolete half is not restored. This reconciliation
preserves the reporting bug fix without reversing the current retrieval
contract.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that causes existing functionality
to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- extract 12–24 character CCR hashes from `Retrieve more`, `Retrieve
original`, and `<<ccr:...>>` markers
- scan both transform marker metadata and nested rendered message values
- preserve stable encounter order and deduplicate case-insensitively
- exclude non-retrieval transform metadata such as tool digests and
stable-prefix hashes
- return the normalized hashes from `/v1/compress`
- add helper-level and endpoint-level regression coverage

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual behavior inspection performed

### Test Output

```text
$ uv run --extra dev pytest tests/test_proxy_compress_endpoint.py -q
50 passed, 1 warning in 6.54s

$ uv run --extra dev ruff check headroom/proxy/handlers/openai.py tests/test_proxy_compress_endpoint.py
All checks passed!

$ uv run --extra dev ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy_compress_endpoint.py
2 files already formatted

$ git diff --check
# no output
```

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, current `main` at `7940c05e`,
project native extension built by `uv`
- Exact command / steps: ran the complete `/v1/compress` endpoint test
module, including a mocked pipeline response containing an embedded
row-drop marker but only unrelated tool-digest marker metadata
- Observed result: endpoint returned exactly the embedded retrievable
hash; helper coverage also proved nested markers, case normalization,
deduplication, stable ordering, and exclusion of unrelated metadata
- Not tested: full repository test and CI matrix; GitHub CI covers the
broader matrix

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented the non-obvious marker filtering behavior
- [x] Documentation is unchanged because the public response contract is
corrected, not expanded
- [x] My changes generate no new warnings
- [x] I have added tests that prove the fix is effective
- [x] New and existing endpoint tests pass locally
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from the Conventional Commit PR title

## Screenshots (if applicable)

Not applicable. This changes a JSON API response and tests, with no
graphical UI changes.

## Additional Notes

The query-based JSON row-search changes from the original branch were
made obsolete by #1532 and are deliberately excluded rather than
reviving a retired API behavior. The original contributor remains the
commit author for the reconciled fix.

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
2026-08-11 23:27:49 -05:00
Tejas Chopra f2c48e26c6 feat(compress): reach the lossless provider seam on the general path and default /v1/compress to marker-free output (#2691)
## Description

Two related changes to the compression seams, plus the review fixes for
both. Supersedes #2661 and #2662, which are closed in favour of this
branch — the fixes are inseparable from the code they fix, so reviewing
them together is cheaper than landing two PRs and patching them
afterwards.

**1. A registered lossless provider now competes on the general path.**
The `headroom.transforms.lossless_provider` seam was only ever consulted
from `_lossless_compact_excluded`, gated on `DEFAULT_EXCLUDE_TOOLS`
(`config.py:216` — `Read/Grep/Glob/Write/Edit/WebSearch/WebFetch`).
Gateway traffic carries the caller's own tool names — LiteLLM's
`headroom` guardrail (https://docs.litellm.ai/docs/proxy/headroom) posts
requests containing tools like `search_docs` / `run_ci` / `fetch_rows` —
so a registered provider was structurally unreachable for every
gateway/sidecar deployment. The seam existed; nothing could get to it.

**2. `POST /v1/compress` is marker-free by default.** A CCR marker is
only useful to a caller that also injects the `headroom_retrieve` tool
AND can reach `/v1/retrieve`. Neither holds here: tool injection lives
in the provider request handlers (`handlers/anthropic.py:1894`), never
in `handle_compress`; and every `/v1/retrieve*` route is
`Depends(_require_loopback)` (`server.py:4422, 4470, 4749, 4781`) with
no remote opt-in — `HEADROOM_COMPRESS_ALLOW_REMOTE` drops the loopback
dependency on `/v1/compress` only. So a gateway forwards a `Retrieve
more: hash=…` pointer the model cannot follow, and the proxy pays a CCR
store write nobody reads. `config.mode="ccr"` opts back in.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### Seam — `content_router.py`, `lossless_provider.py`

- `_lossless_first` (STAGE 0, every block on every path) consults
`get_lossless_provider()` and keeps whichever output is smaller.
**Strict no-op when no provider is registered**, which is the default;
and because it is best-of rather than authoritative, a provider can
never do worse than the built-in folds.
- Malformed provider output can no longer escape. Every shape check runs
inside the `try`: result must be `None`, or a 2-element tuple/list of
two `str`. Anything else is ignored at debug level. (Previously the
unpack sat outside the `try`, so a 3-tuple raised `ValueError` up
through `TransformPipeline.apply`, which re-raises.)
- Empty / whitespace-only candidates are rejected rather than silently
replacing block content.
- **Providers are never offered diff content.** Diff folding is
subtractive with no inverse check and a reflowed hunk breaks `git apply`
— the same reason the built-in `diff` fold is restricted at
`content_router.py:2481`.
- The third-party `kind` label is sanitised against `^[a-z0-9_]{1,32}$`
before reaching `transforms_applied` and the per-strategy metric dicts,
so a caller-controlled string cannot explode Prometheus label
cardinality. `fullmatch`, not `match`: `$` also matches before a
trailing newline, which would put a newline in a label.
- `set_lossless_provider(provider, *, verifier=None)` — in lossless-only
mode, where STAGE 0's output is final and there is no marker to recover
from, a registered verifier must confirm the fold or the candidate is
dropped. No verifier registered = today's behaviour. `provider=None`
clears both.
- The provider is invoked once per block, not twice
(`_has_lossless_fold` probes `_lossless_first` and discards the result,
then STAGE 0 recomputes). Bounded memo, wholesale clear on overflow, no
lock — a race costs one redundant fold. The memo keys on the provider
registration generation, so registering or clearing a provider after a
block was already folded takes effect.
- The seam docstring now records that the provider runs on the general
path and inside the parallel compression pool, so it must be thread-safe
as well as deterministic.

### Route — `handlers/openai.py`, `server.py`

- `_derived_compress_pipeline(key, **overrides)` replaces the
copy-pasted pipeline-derivation block; `_no_ccr_pipeline` (the new
default) and `_lossy_inline_pipeline` both use it.
- The default pipeline is **built at startup** and included in
`_eager_preload_transforms`, so a fresh pod does not pay ContentRouter
construction and compressor load on its first request, inside the
compression-executor budget.
- An unrecognised `config.mode` returns 400 naming the valid values
instead of silently falling back to the default.
- Claude-family model names resolve their context limit from the
Anthropic provider. Real divergence:
`bedrock/anthropic.claude-3-5-sonnet` is 200000 there and 128000 on the
OpenAI provider. The tokenizer still comes from the OpenAI pipeline's
provider — a separate, larger change, noted in a comment.
- Documents why the derived router deliberately does **not** share the
base router's compression cache: keys do not encode CCR-marker mode, so
sharing would leak marker-laden entries into the marker-free path.

### Behavior change

A `/v1/compress` caller that relied on default markers now gets none.
The only in-tree caller that can resolve them is the TypeScript SDK
(`sdk/typescript/src/client.ts:398 retrieve`, `:422 handleToolCall`); it
needs `config: {"mode": "ccr"}` to keep today's behaviour, and landing
that SDK default in the same release would leave only gateway callers —
for whom markers were never resolvable — seeing a difference.

No `HEADROOM_COMPRESS_DEFAULT_MODE` compat env deliberately: a flag
nobody sets becomes permanent debt, and the wire-level `mode` already
covers the one caller that needs it.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_lossless_first_dispatch.py tests/test_lossless_excluded_compaction.py \
    tests/test_lossless_mode.py tests/test_lossless_then_lossy.py tests/test_lossless_diff_fold_guard.py \
    tests/test_bash_search_lossless_fold.py tests/test_proxy_compress_endpoint.py \
    tests/test_ccr_row_drop_store_bridge.py tests/test_gateway_sidecar_ports.py tests/test_compress_api.py \
    tests/test_platform_stabilization_functional.py tests/test_proxy_eager_preload_bind.py \
    tests/test_proxy_warmup.py tests/test_router_registry_smartcrusher.py -q
189 passed in 32.04s

$ ruff check <all 9 changed files>
All checks passed!

$ ruff format --check <all 9 changed files>
9 files already formatted

$ mypy headroom/transforms/content_router.py headroom/transforms/lossless_provider.py \
      headroom/proxy/handlers/openai.py headroom/proxy/server.py
Success: no issues found in 4 source files
```

New tests cover, one concern each: every malformed provider shape; empty
and whitespace-only results; diff content never reaching a provider
(call-recording); `kind` sanitisation including the trailing-newline
case; the verifier accepting / rejecting / raising; clearing a provider
clearing its verifier; single provider invocation per block; memo
invalidation on re-registration; unknown and valid `mode` values; the
default pipeline existing before any request; and Claude vs OpenAI
context-limit resolution with `token_budget` precedence preserved.

## Real Behavior Proof

- **Environment:** macOS arm64, Python 3.12.6, proxy built from
`_proxy_config_from_env()` with the default `coding` savings profile;
Kompress both disabled and offloaded to a remote `kompress-v2-base`
endpoint; `HEADROOM_COMPRESSION_TIMEOUT_SECONDS=300`.
- **Steps:** `POST /v1/compress` over `TestClient` with OpenAI-shaped
payloads under non-excluded tool names (`run_ci`, `list_files`,
`code_search`, `fetch_rows`) — a CI log with ANSI escapes and repeated
lines, a 160-path listing, a 150-line grep dump, a 150-row JSON array;
plus a second payload with a RAG user blob, a 200-row JSON tool result
and a 300-line log. Ran with and without a provider registered via
`set_lossless_provider`.
- **Observed — seam reachability:**

  | Kompress | no provider registered | provider registered |
  |---|---|---|
  | off | 19,284 → 10,265 tokens (46.8%) | 19,284 → **7,879 (59.1%)** |
| on (remote) | 19,284 → 9,366 tokens (51.4%) | 19,284 → **7,146
(62.9%)** |

Before this change the right-hand column was identical to the left — the
registered provider was never called on this payload.

- **Observed — marker-free default costs nothing:**

  | Config | tokens | saved |
  |---|---|---|
  | markers on (previous default) | 37,791 → 24,415 | 35.4% |
| markers off (new default) | 37,791 → 24,415 | **35.4% — identical** |
  | `mode="lossy_inline"` | 37,791 → 25,129 | 33.5% |
  | `--lossless` | 37,791 → 35,100 | 7.1% |

- **Observed — memo staleness, before the fix:** registering a provider
that folds a grep block to 5 bytes left the block at its 1496-byte
built-in fold, and clearing a provider kept serving the provider's
output. Both correct after keying on the registration generation.
- **Observed — context limit:** `bedrock/anthropic.claude-3-5-sonnet`
resolves 200000 via the Anthropic provider, 128000 via the OpenAI
provider.
- **Not tested:** `tests/test_transforms_content_router.py` was not run
— it does not complete on this machine, wedging on its 5th test while
that test passes in 5.7s alone. Verified pre-existing before this work:
with the diff stashed, the clean tree stalled at the identical test, and
it stalled the same way under `HF_HUB_OFFLINE=1 HEADROOM_OFFLINE=1`. The
machine was also out of disk at the time, which may be the real cause
rather than the suspected native-detector deadlock (#575) — worth a
separate issue either way. CI should be the arbiter here.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Two pre-existing test files needed adjusting, both direct consequences
rather than scope creep:

-
`test_platform_stabilization_functional.py::test_v1_compress_success_reports_actual_metrics`
patches `openai_pipeline.apply`, which the default mode no longer routes
through. **This test was already failing on the marker-free-default
commit** before any of the fixes — my original test selection missed it.
It now patches the pipeline the route actually uses.
- `test_proxy_eager_preload_bind.py` substitutes fake pipelines to
control exactly what the preload walks; the eager build injected the
real derived router's statuses into an exact-equality assertion. Its
shared helper now clears the derived cache, preserving each test's
intent without weakening an assertion.

Docs unchecked — follow-ups worth doing in the same release: document
`config.mode` values in `docs/content/docs/litellm.mdx` and
`wiki/proxy.md`; the TS SDK `mode:"ccr"` default; and an operational
note that `COMPRESSION_TIMEOUT_SECONDS` defaults to 30
(`helpers.py:687`) while a remote ML endpoint makes one sequential call
per unit — on a large payload it trips the executor timeout and the
handler fails open, returning `compression_skipped: true` with
`tokens_before: 0`, which reads as "nothing to save" rather than "we
gave up". Those zeroed counters are misleading and worth a separate fix.
2026-07-31 12:31:38 -07:00
JD Davis 1d2b76e72e fix: harden persistent install startup (#1851)
## Description

Hardens persistent install startup and proxy compression behavior for
issue #1843. Repeated `headroom install start` / scheduled ensure calls
no longer spawn duplicate runtimes by default, and `/v1/compress` now
fails open on compression timeout instead of returning a 503. The PR
also adds a machine-readable platform feature matrix and app-level
stabilization tests for health, compression functionality, timeout
behavior, and matrix evidence.

Refs #1843

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Wrapped direct persistent deployment starts with the existing
profile-local runtime start lock.
- Made `headroom install start` idempotent when the deployment is
already healthy.
- Added wedged-runtime handling: if a PID is running but `/readyz` does
not recover inside the grace window, stop it before starting again.
- Kept `install agent ensure` inside the already-held lock while
delegating to the shared start helper.
- Changed `/v1/compress` timeout behavior from `503 compression_timeout`
to fail-open `200` with original messages, `compression_skipped: true`,
and `skip_reason: compression_timeout`.
- Added `tests/test_platform_stabilization_functional.py` covering real
FastAPI health/compression routes, successful compression metrics,
timeout fail-open speed, and a real JSON tool payload that reduces
tokens.
- Added `docs/platform-feature-matrix.json` and
`docs/platform-stabilization.md` for Linux/macOS/Windows hardening
coverage and known gaps.
- Strengthened matrix tests so cited local test/workflow paths must
exist.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# Clean source tree without copied Rust extension: functional module is skipped locally, as CI copies _core from the built wheel.
> python -m pytest tests/test_install tests/test_cli/test_install_cli.py tests/test_platform_feature_matrix.py tests/test_platform_stabilization_functional.py -q
collected 120 items / 1 skipped
119 passed, 2 skipped in 17.08s

> python -m ruff check headroom/proxy/handlers/openai.py tests/test_platform_stabilization_functional.py tests/test_platform_feature_matrix.py
All checks passed!

# Local Windows compiled-core proof:
> python -m maturin build --profile ci --out dist-local
Built wheel for abi3 Python >= 3.10 to dist-local\headroom_ai-0.29.0-cp310-abi3-win_amd64.whl

# Copied _core.pyd from the wheel into headroom/ for local route execution, then:
> python -m pytest tests/test_platform_stabilization_functional.py -q
collected 4 items
4 passed in 6.71s

> python -m pytest tests/test_install tests/test_cli/test_install_cli.py tests/test_platform_feature_matrix.py -q
collected 120 items
119 passed, 1 skipped in 17.16s

Commit hooks:
Sync plugin versions.....................................................Passed
check for merge conflicts................................................Passed
ruff.....................................................................Passed
ruff-format..............................................................Passed
mypy.....................................................................Passed
```

## Real Behavior Proof

- Environment: Windows 11, PowerShell, Python 3.13.13, worktree
`C:\git\headroom-stabilization` on branch
`jd/cross-platform-stabilization`.
- Exact command / steps: built the Windows wheel with `maturin`,
extracted `_core.pyd`, ran the new FastAPI route tests and
install/matrix tests listed above, then removed generated artifacts
before committing.
- Observed result: direct start paths now no-op when healthy, skip
spawning when the start lock is contended, and stop a wedged runtime
before restart. `/v1/compress` now returns original messages quickly on
timeout instead of a 503. The real JSON tool-payload smoke test returns
`tokens_before > tokens_after`, `tokens_saved > 0`, `compression_ratio <
1.0`, and non-empty transforms through the public route.
- Not tested: full native Windows persistent process e2e remains blocked
by the upstream CRT/wheel issue already documented in workflows and in
the matrix. No real OS service was installed locally; service manager
behavior is covered by argument-level unit tests.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

CHANGELOG is not updated because this is an unreleased
hardening/test/documentation pass. The platform matrix intentionally
records partial/blocked Windows/macOS e2e gaps instead of claiming full
coverage where the repo cannot currently run it.
2026-07-10 00:40:34 -04:00