mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
28
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
dc94c46a1f |
test: remove the developer's GitHub and Copilot tokens from every test's environment (#4790)
## Description Any test that enters the proxy lifespan starts the Copilot quota poller (`headroom/subscription/copilot_quota.py`), which reads `GITHUB_COPILOT_GITHUB_TOKEN`, `GITHUB_TOKEN`, `COPILOT_GITHUB_TOKEN` or `GITHUB_COPILOT_API_TOKEN` from the environment and sends it as a `Bearer` header to `api.github.com`. `tests/conftest.py` scrubs only `HEADROOM_*`, so a developer with `GITHUB_TOKEN` in the shell has that token used by the test run. `tests/test_subscription_poll_credential_policy.py:1-8` already treats using a credential nobody meant for this purpose as a defect. This PR adds an autouse fixture that removes the nine Copilot/GitHub token variables that `copilot_quota.py` and `copilot_auth.py` read. Tests that need one still set it with `monkeypatch.setenv`, which runs after the fixture. Part of #4789 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `tests/conftest.py`: `_COPILOT_TOKEN_ENV_VARS` and the autouse fixture `_scrub_developer_copilot_tokens`, placed before `_scrub_developer_headroom_env`. - `tests/test_conftest_scrubs_copilot_tokens.py`: five names must be absent from `os.environ`, `discover_github_token()` finds nothing, and a test can still set its own token. No product code changes. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed `mypy headroom` was not run (no change under `headroom/`). `ruff check` and `ruff format --check` (0.17.3) are clean on the two files. ### Test Output With `GITHUB_TOKEN` and `GH_TOKEN` set to dummy values in the shell, the new file: ```text this branch: 7 passed conftest from main: 3 failed, 4 passed (GITHUB_TOKEN and GH_TOKEN still in the environment, and discover_github_token() returns the dummy token) ``` The Copilot and GitHub-token related test files, with the same dummy variables set: ```text test_cli/test_wrap_copilot.py, test_cli/test_wrap_opencode.py, test_copilot_auth.py, test_copilot_quota.py, test_offline_egress_chokepoint.py this branch: 1 failed, 435 passed, 4 skipped conftest from main: 1 failed, 435 passed, 4 skipped test_release_workflows.py, test_subscription_poll_credential_policy.py, test_copilot_subscription_smoke.py this branch: 86 passed ``` The one failure is `tests/test_offline_egress_chokepoint.py::TestUpdateCheckOffline::test_explicit_update_check_never_constructs_request` (`AttributeError: module 'headroom.cli.main' has no attribute 'name'`). It fails the same way with `main`'s `conftest.py` and passes when run alone with either, so it depends on test order and is not caused by this change; I did not investigate it. ## Real Behavior Proof - Environment: Windows 11, Python 3.12, pytest 9.2.0, the compiled `headroom._core`, `main` at `3c66c91e`. - Exact command / steps: `GITHUB_TOKEN=ghp_dummy_not_real GH_TOKEN=dummy2 python -m pytest tests/test_conftest_scrubs_copilot_tokens.py`, once with this branch's `conftest.py` and once with `main`'s; then the two groups of files above with the same variables. - Observed result: as in Test Output. Only dummy values were used, never a real token (the runs of the other files with the dummy variables may have sent them to GitHub, where they are rejected). - Not tested: CI, Linux and macOS, the whole suite, and a network capture of the poller (the 53 lookups in #4789 come from an instrumented run, not from this PR's proof). ## Runtime Rollout Safety - Rollout-managed feature(s): none; test only. - Minimum rollout channel: n/a - Stable/default behavior changed: no. - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes (the files listed above; the whole suite was not run, and one order-dependent failure exists on `main`) - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Not in this PR: the other items of #4789 (credential readers in `test_copilot_auth.py`, writes to `~/.headroom`, tests that reach the network). - If a CI job relies on a token reaching a test, that test would now need `monkeypatch.setenv`; I found none in the files above. AI assistance: this change, its tests and the runs above were done with Claude (Sonnet 5.5), operated by me (HetCreep). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4790" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
50bf775e8c |
fix(proxy): bypass compression for large admitted histories (#4142)
## Description Closes #4139. A history that is too large to compress should still reach the provider. This keeps the 100 MiB hard admission limit and bypasses request-side transforms above 16 MiB decoded size. The hard limit still applies before forwarding. Current head: `62ffe3defef1684f70d96f1640818d99445a43bb`, merging current main `a2d4fe72` into `1902d8ef`. The 42-test/native/manual/pre-push results below qualify `1902d8ef`, not this new merge. New-head shard 4 has a cached-prefix fixture failure under investigation; coverage uploads are still incomplete. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement - [x] Documentation update ## Changes Made - Keep the startup override and both wire/decompressed hard bounds. Large admitted histories skip normalization, image processing, memory injection, tokenization and compression on Messages, Chat/Responses, Gemini and Bedrock HTTP routes. - Keep provider routing and backend translation, including Copilot Chat, Codex `store:false` and Vertex route-forced streaming. Native forwarding retains decoded bytes unless the provider's wire contract requires a rewrite; translation retains the full message text. - Report actual provider usage, with zero compression savings or attempted tokens for the bypass. Keep `/v1/compress` and WebSocket contracts unchanged. - Add real-provider coverage for large Gemini JSON/SSE requests with query-key auth, and OpenAI JSON/SSE through the installed LiteLLM adapter. No admission-policy or CI changes in this follow-up. ## Testing - [x] Unit tests pass (focused suite and normal critical pre-push suite) - [x] Linting passes (normal hooks) - [x] New tests added for admission, recovery and passthrough accounting - [x] Manual testing performed against the real native build ### Test Output ```sh python -m pytest -q tests/test_proxy_compression_admission.py tests/test_proxy/test_gemini_plugin_routed_streaming.py --cov=headroom --cov-branch # 42 passed, 41 warnings ``` Normal pre-push `make ci-precheck` passed: 1668 Rust tests (3 ignored), 181 Python tests, lint/type checks and commitlint. No hook bypass. The initial deep-module coverage command hit a Pydantic import-isolation error; using the repo's `--cov=headroom` convention passed both complete modules without disabling plugins or tests. The earlier `96c5fe7e` qualification covered the other native protocols, configured/per-request backends, hard-ceiling recovery and complete local shards 2/3. Those runs are historical, not a new whole-suite claim. New-head hosted success is not claimed. ## Real Behavior Proof - Environment: Linux x86_64, Python 3.13.14, freshly built real `headroom._core`, public Headroom ASGI routes, a real localhost HTTP provider and installed LiteLLM. Synthetic credentials, offline mode and `DO_NOT_TRACK=1`; no live provider calls. - Exact command / steps: Start `create_app(ProxyConfig(backend="litellm-openai", ...))` with all upstream URLs set to the localhost peer and `litellm.api_base=peer+"/v1"`. Leave the production 16 MiB soft/100 MiB hard limits unchanged. POST `"history " * (16777216 // 8 + 1024) + "LAST TURN"` as a user message to Chat Completions, then as a Gemini content part to generateContent/streamGenerateContent, with streaming off/on. The peer rejects a changed history, requires Gemini query-key auth and one `alt=sse`, and returns protocol-valid JSON/SSE with 101 input/1 output tokens. The maintained four small-ceiling cases are in `tests/test_proxy_compression_admission.py` and run with the command above. - Observed result: All four >16 MiB requests returned 200 with the full history intact. JSON/SSE and Gemini auth survived; streamGenerateContent forced SSE even with incoming `alt=json`. Each request recorded exactly 101 provider input/1 output tokens, zero compression attempts/savings and no failures; `/livez` stayed 200. Earlier recorded qualification also covered all five native HTTP protocols, real translating backends and 413 recovery. - Not tested: Live cloud accounts, the reporter's deployed 4 GB VPS, screenshot/image quality or a zero-JSON-parse path. One envelope/model JSON parse remains; no proxy-only RSS guarantee is claimed. ## Runtime Rollout Safety - Rollout-managed feature(s): None; this is the admission/compatibility fix. - Minimum rollout channel: Stable/default. - Stable/default behavior changed: Keep the 100 MiB hard ceiling; bypass request-side transforms above 16 MiB decoded size on the named HTTP routes. - Kill switch / disable path: Set `HEADROOM_REQUEST_BODY_MAX_BYTES=16777216` (or lower) and restart, accepting local 413s instead of the large-body path. - Unsafe override required: No. - Qualification impact: Local native protocols, backend translation, hard bounds and recovery exercised; live-provider/VPS qualification remains external. - Rollback path: Revert the admission revision and restart; no storage migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] I have made corresponding changes to the documentation - [x] I did not edit `CHANGELOG.md` --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
b196424e35 |
fix(cache): bound the router's compression cache, keep the rest in SQLite (#4357)
## Why The content router's result cache (`CompressionCache`, original → compressed) had no size limit, and its 30-min TTL was only checked when an entry was read again. Content that never came back stayed in memory for the life of the proxy. A 300,000-result soak grew past 550 MB. ## Change - **Memory budget:** results kept least-recently-used within `HEADROOM_COMPRESSION_CACHE_MB` (default 64, results only, sized with `sys.getsizeof`); skip set capped to match. The same soak now levels off (~+170 MB process growth, cache at 67 MB). - **Idle expiry:** a result expires 75 min after its *last use*. Nothing signals a session's end; once a session has been idle past the provider's longest prompt-cache TTL (1 h), there's no cached prefix left to protect, so recompressing on resume costs only the compression. Results with a CCR marker keep the CCR TTL from creation. - **Disk tier** (`headroom/cache/result_store.py`, proxy only): `workspace_dir()/compression_cache.db`, owner-only, WAL. An entry the budget evicted, or a restart lost, is read back instead of recompressed (a recompression could change the bytes of a cached prefix). - Namespaced by a digest of the router config + headroom version, so differently configured routers never share output. - Marker-free results only. - A SQLite error is a miss, never a failed transform. - Expired rows are deleted on open. - Off in stateless mode or with `HEADROOM_COMPRESSION_CACHE_DISK=0`; `HEADROOM_COMPRESSION_CACHE_PATH` moves it. - **Stable keys:** a blake2b digest (`content_cache_key`) instead of per-process `hash()`, so workers share the file and a shared store for replicas (Redis, for `/v1/compress` behind gateways) can follow. ## Platforms SQLite, blake2b and pathlib behave the same on macOS, Linux and Windows. File permissions go through `connect_private_sqlite` (owner-only on POSIX, inherited ACL on Windows, as for `ccr_store.db`). As with the CCR store, WAL needs a local filesystem. ## Tests `tests/test_router_result_cache.py`: - budget and LRU order - skip cap - evicted result comes back from disk unchanged - idle result dropped while a live one is kept - cross-config isolation - namespace and keys stable across processes - DB error is a miss - CCR-marker results stay off disk - stateless writes no file 731 router/proxy tests pass. ruff 0.17.0 and mypy clean on the changed files. `conftest` points the file at `tmp_path` so tests never write `~/.headroom`. ## Real behavior Environment: macOS 26, Python 3.12, this branch, default budget (64 MB). Steps: one process puts 300,000 distinct results of ~1.8 KB each (about 550 MB of text) through `CompressionCache`, measuring RSS after each 50,000; once memory-only, once with the SQLite tier. | results put | 50K | 100K | 150K | 200K | 250K | 300K | |---|---|---|---|---|---|---| | RSS growth, memory only (MB) | 89 | 91 | 168 | 168 | 168 | 168 | | RSS growth, with disk tier (MB) | 92 | 94 | 171 | 171 | 171 | 171 | The cache holds 67 MB at its budget. With the disk tier, the first result put (long evicted from memory) still reads back unchanged. Before this PR the same run grows without bound (no cap; TTL checked only on read). Not run: Windows (relies on CI's Windows job); a long-lived production proxy. ## Review fixes |
||
|
|
f3d240bcae |
fix(cli): say why the proxy failed to start (#4120)
## Description Fixes two proxy startup failures that hid their cause from the user. **1. `headroom proxy` on a port that is already in use.** It printed the full banner, ending in "Press Ctrl+C to stop.", and then about 10 INFO lines. The one line that mattered, uvicorn's `[Errno 48] ... address already in use`, came next. After it came `event=proxy_shutdown reason=signal` and a 10-line zero-valued session summary. The command now checks the port before printing the banner and exits with one clear line: ``` Error: Port 54351 on 127.0.0.1 is already in use by another process (Address already in use). Start this proxy on another port with --port N (env: HEADROOM_PORT). ``` When a Headroom proxy is the process holding the port, the message says so instead. The check asks the address that is actually busy (IPv6 included), so a Headroom proxy on 127.0.0.1:P is never reported as the owner of [::1]:P. When that address cannot be probed (a scoped IPv6 address, an unresolvable name), the message is the generic `already in use (<reason>)`. **2. `headroom wrap <tool>` when the proxy it starts dies during startup.** The error quoted the last 500 characters of the proxy stdio log. That was the wrong part of the log, for two reasons: - The proxy logs its shutdown summary after the error, so the tail was the summary box. - The log is appended to across runs, so the tail could come from an older run. The error also never said where the log was. It now reads only this run's output, at most its first 64 KiB and, when the head has no error, its last 64 KiB (so a runaway log is never read whole, and an error after a long burst of startup output is still found), quotes from the first `ERROR` / `CRITICAL` / `Traceback` line or line-leading Click `Error: ` (otherwise the end of the first non-blank window, so neither a blank head nor a blank tail hides useful startup text), and adds `Full log: <path>`. The timeout error now names the log too. Refs #4136 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/proxy.py`: new `_tcp_port_in_use` / `_refuse_busy_port`. They run after the bind-policy check and before the banner, and are skipped for `--uds`. The probe mirrors uvicorn's own bind, so it never refuses a bind that uvicorn would accept: - It sets `SO_REUSEADDR`, as both uvicorn paths do. A port in TIME_WAIT after a restart is therefore not reported as busy. - With one worker, it binds every address the host resolves to, with `IPV6_V6ONLY`, the same way asyncio `create_server` does. - With `--workers > 1`, it binds one AF_INET or AF_INET6 socket, as uvicorn's `Config.bind_socket` does. - On Windows, and for any error other than EADDRINUSE, it does nothing and leaves the case to uvicorn, exactly as today. - It returns the busy address. `_probe_host_for` maps it to the address to ask: wildcards go to loopback of the same family (`0.0.0.0` to `127.0.0.1`, `::` to `::1`), a hostname is resolved for IPv4, and a scoped IPv6 address returns None (generic message). - `headroom/cli/port_discovery.py`: `probe_headroom_proxy` takes an optional keyword `host` (default `127.0.0.1`, so existing callers are unchanged) and brackets IPv6 in the URL. - `headroom/cli/wrap.py`: - `_start_proxy` records the stdio log size before spawning the proxy. - New `_proxy_startup_failure_excerpt` picks the error line from this run's output, reading at most the first and last 64 KiB written after the recorded offset. - Both startup errors now end with `Full log: <path>`. - `tests/conftest.py` + `pyproject.toml`: an autouse guard, and its `proxy_port_preflight` marker, turn the new check off unless a test opts in. It follows the existing `proxy_dependency_gate` pattern. - Without it, the 16 test files that invoke `headroom proxy` with a stubbed `run_server` on the default port 8787 would fail whenever a developer has their own proxy running. - Tests: - New `tests/test_cli/test_proxy_port_preflight.py`, which uses real sockets: - busy port with workers 1 and 2; - port held by a Headroom proxy; - free port still starts; - `test_ipv6_conflict_is_identified_on_the_requested_host`: a real fake-Headroom `/livez` on 127.0.0.1:P and a plain listener on [::1]:P; `--host ::1` must say "another process"; - `test_unprobeable_host_gets_the_generic_message`. - New `test_startup_excerpt_reads_at_most_64_kib_of_this_run`. - New `test_start_proxy_exit_quotes_this_runs_error_not_summary_or_old_runs`. - The timeout test now also asserts the log path. - The existing stdio-tail test now writes its log output from the fake `Popen`, as the real child process does. - The new tests fail on the old code (3 preflight tests; the excerpt and timeout tests) and pass now. The three tests added for the review (IPv6 owner, unprobeable host, 64 KiB bound) fail on the PR's previous commit |
||
|
|
f78e66f7cc |
fix(proxy): stop headroom logger from suppressing propagation to stdout/stderr (#3096)
## Description `_setup_file_logging()` in `headroom/proxy/helpers.py` unconditionally set `headroom_logger.propagate = False`, on the theory that `wrap.py` redirects stderr into the same `proxy.log` file the rotating handler writes to, so propagation would double-write records. That premise is wrong: `wrap.py` redirects the proxy subprocess's stdout/stderr to a separate sibling file, `proxy-stdio.log` (see `_get_proxy_stdio_log_path()`), never `proxy.log`. Severing propagation bought no deduplication — it only stopped every `headroom.*` log record from reaching root's `logging.basicConfig()` handler (`proxy/server.py`), which is what writes to stderr/stdout. In containerized deployments (Docker/Kubernetes), stdout/stderr is the only channel the platform collects, so this silently dropped all application logs. Closes #3087 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Removed `headroom_logger.propagate = False` from `_setup_file_logging()` in `headroom/proxy/helpers.py`. - Updated the surrounding comment to explain why propagation is safe to leave on: `proxy.log` (file handler) and stderr/stdout (root's `basicConfig` handler) are two distinct destinations, so nothing is duplicated in either file. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check . All checks passed! $ pytest tests/test_corrupt_golden_bytes_recovery.py -q collected 9 items tests/test_corrupt_golden_bytes_recovery.py ......... [100%] 9 passed in 0.47s ``` `mypy headroom` was not run to a clean result: it fails before reaching this file with a pre-existing, unrelated error in the installed `numpy` stub (`Type statement is only supported in Python 3.12 and greater`) in this sandbox's dependency set. Not caused by this change (a 4-line pure-Python logging tweak). The broader `pytest tests/` collection also hits ~170 pre-existing, unrelated errors in this sandbox: `headroom._core`, the compiled Rust extension, isn't built here (no `cargo`/`maturin` available), so every module importing it fails at collection. This change doesn't touch Rust code or `_core` and isn't implicated. No new tests added: this restores an existing, already-tested code path (see `tests/conftest.py`'s autouse fixture and `tests/test_corrupt_golden_bytes_recovery.py`, both of which already assert/require `propagate=True` for `caplog` to work) to its intended default rather than adding new behavior. ## Real Behavior Proof - Environment: local sandbox, Python 3.12.3, unable to build/import `headroom.proxy.server` directly (missing compiled `headroom._core`). Verified with an isolated script that copies `_setup_file_logging()`'s exact logic byte-for-byte plus the same `logging.basicConfig()` call `proxy/server.py` makes at import time. - Exact command / steps: ran the fixed logic, then emitted `logging.getLogger("headroom.proxy").info("compression timeout: request abc123 exceeded 30s budget")`. - Observed result: the `headroom.proxy` record was written to stderr AND to `proxy.log` simultaneously (full output below); re-running the same script with the old `headroom_logger.propagate = False` line restored reproduces the bug — the record disappears from stderr entirely and only lands in `proxy.log`. ```text propagate flag on 'headroom' logger: True --- emitting a headroom.proxy log record, watch for it below on stderr --- 2026-08-17 21:17:39,545 - headroom.proxy - INFO - compression timeout: request abc123 exceeded 30s budget --- also present in proxy.log (file handler): OK --- ``` - Not tested: end-to-end through the actual compiled proxy binary/server process (blocked by missing `headroom._core` in this sandbox) and real container log collection (Docker/Kubernetes) — the isolated repro exercises the exact same logging configuration code the real process runs, so this is considered sufficient proxy proof for a pure logging-config change. ## Runtime Rollout Safety - Rollout-managed feature(s): None — this is unconditional startup logging configuration, not behind a rollout flag. - Minimum rollout channel: N/A - Stable/default behavior changed: Yes — `headroom.*` log records now also reach stderr/stdout (previously file-only). No change to request/response handling, compression, or any user-facing proxy behavior. - Kill switch / disable path: None needed; this restores default Python logging propagation. If a deployment truly wants file-only logging, `HEADROOM_LOG_TO_STDOUT`-style suppression could be added later, but no such request exists today. - Unsafe override required: No - Qualification impact: None - Rollback path: Revert this commit to restore `propagate = False`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — logging configuration change, no UI impact. ## Additional Notes - "New and existing unit tests pass locally" is left unchecked because the full `pytest tests/` collection can't complete in this sandbox (missing compiled `headroom._core` Rust extension, unrelated to this change) — see Test Output above for what was actually run and passed. - "I have added tests" is left unchecked: existing tests (`tests/conftest.py` autouse fixture, `tests/test_corrupt_golden_bytes_recovery.py`) already assert the code path this restores; happy to add a focused regression test asserting `logging.getLogger("headroom").propagate is True` after `_setup_file_logging()` if maintainers want one. --------- Co-authored-by: = <=> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
7d856b81f5 |
fix(proxy): keep the hard watchdog out of pytest and alive when sys.stderr is swapped (#3875)
## Description
CI test shards keep dying mid-file with `Process completed with exit
code 1` and no pytest summary, even after #3845. Examples on `main`
after #3845 merged: runs 36617366462 (shard 1), 36623491381 (shard 1),
36627341700 (shard 4), 36632696307 (shard 1). On PRs: 36646296979,
36648438161, 36671680550 (shards 1 and 4). From Sep 27 until #3180
merged there were none; the first was run 36520624898 (shard 1) about
ten hours after it.
Root cause, in two parts:
1. #3845 exports `HEADROOM_HARD_WATCHDOG_SECS=0` for the shards, but the
autouse `_scrub_developer_headroom_env` fixture in `tests/conftest.py`
deletes every `HEADROOM_*` variable before each test. About 100 test
files enter the proxy lifespan (`with TestClient(app)`), and each one
calls `start_hard_watchdog()`. That call now sees no variable and arms
the 90s production watchdog inside the pytest process.
2. An armed watchdog should still never fire, because the heartbeat
re-arms it every 30s. But `_arm` passed `sys.stderr` on every tick, and
`faulthandler` calls `.fileno()` on it. When a tick lands while
`sys.stderr` has no fileno (pytest `capsys`, click `CliRunner`,
`redirect_stderr(StringIO())`), `dump_traceback_later` raises and the
heartbeat thread dies. The timer from the previous tick then runs
`_exit(1)` on a healthy process. It writes the stack dump into the stale
stderr, which is pytest's capture file, so the job log shows nothing.
The last test file printed before the exit has nothing to do with it.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `tests/conftest.py`: re-pin `HEADROOM_HARD_WATCHDOG_SECS=0` after the
scrub. This makes #3845's opt-out actually reach the tests. The watchdog
tests still set the variable themselves.
- `headroom/proxy/hard_watchdog.py`: resolve the stderr fd once in
`start_hard_watchdog()` and re-arm onto that fd. The first arm writes to
the same place as before. Later ticks no longer depend on whatever
`sys.stderr` points at.
- `tests/test_hard_watchdog.py`: a subprocess that swaps `sys.stderr`
for a `StringIO` and outlives the 5s deadline must survive. A second
test checks that pytest resolves the watchdog to disabled.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_hard_watchdog.py scripts/tests/test_ci_workflow.py -q
9 passed in 29.73s
# with the fix reverted (new tests kept):
FAILED tests/test_hard_watchdog.py::test_swapped_stderr_does_not_kill_the_heartbeat (assert 1 == 0)
FAILED tests/test_hard_watchdog.py::test_pytest_never_arms_the_production_watchdog (assert 90.0 == 0.0)
$ uvx ruff check / ruff format --check (changed files): clean
$ uv run --frozen --extra dev mypy headroom/proxy/hard_watchdog.py: Success
```
## Real Behavior Proof
- Environment: macOS 15, Python 3.11, this branch and upstream main
|
||
|
|
5bf6612326 |
fix(cli): honor CODEX_HOME and align wrap/init/doctor with the live proxy port (#3855)
## Description Closes several CLI usability gaps where `wrap`, `init` and `doctor` disagreed about *where* things are: 1. **CODEX_HOME was ignored by `doctor` / `install` / `init -g codex` hooks.** `codex_config_path()` hardcoded `~/.codex/config.toml`, while `wrap` and the MCP registrar honored `CODEX_HOME`. With a relocated Codex home, `doctor` reported "not routed" for a correctly wrapped Codex (and `install apply` edited the wrong file). 2. **Hard-coded 8787.** `wrap <tool> --port` (~20 subcommands) and `init --port` ignored `HEADROOM_PORT` and never noticed a proxy already running on another port, so they would start a second proxy or point the agent at a dead port. 3. **`init hook ensure` output was fully suppressed**, so an agent whose `base_url` drifted from the managed proxy's port failed silently every session. 4. **`doctor`** only probed `/livez`, did not check the hooks `init` installs, and did not notice a live proxy on another port. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) ## Changes Made **CODEX_HOME (one helper, reused everywhere)** - `headroom.install.paths`: new `codex_home_dir()` (`CODEX_HOME`, blank-safe, else `~/.codex`), `codex_hooks_path()`, `codex_project_config_path()`; `codex_config_path()` now goes through it. - Reused by `wrap._codex_home_dir`, `CodexRegistrar`, `recover codex`, `init` (`-g` config **and** `hooks.json`), `install` (via `codex_config_path`), and `doctor`. - `doctor`'s `codex` check consults the project `.codex/config.toml` first (Codex layers it over the user config; `init codex` without `-g` writes there), then `$CODEX_HOME/config.toml`, and names the file that supplied the routing. The "no config" summary now names the real path instead of `~/.codex/config.toml`. **Port resolution (`headroom/cli/port_discovery.py`)** - Every `wrap` subcommand's `--port` / `--proxy-port` (shared `proxy_port_option`) and `init --port`: - explicit `--port` -> honored as-is, no discovery; - `HEADROOM_PORT` -> honored as-is, no discovery; - otherwise default 8787 **plus** cheap discovery: candidates are the requested port, `HEADROOM_PORT`, 8787, deployment-manifest ports and the project's wrap marker (max 8, de-duplicated), probed **concurrently** on loopback `/livez` with a 0.5 s timeout. A port counts only if `/livez` identifies `service: headroom-proxy` (unrelated local servers are ignored). No port-range scanning. - If the requested port has no Headroom proxy but another candidate does: - **TTY (stdin and stdout)**: prompt `Connect to the running proxy on N [c], keep port 8787 [k], or abort [a]?` (default `c`); - **non-interactive**: keep the requested port (nothing is adopted silently) and print to stderr the live port and the exact `--port N` / `HEADROOM_PORT=N` to use it. - `init hook ...` never runs discovery (hooks must stay silent/fast). - Kill switch: `HEADROOM_PORT_DISCOVERY=0`. - Before `wrap codex` / `init codex` rewrite an existing `[model_providers.headroom]` whose loopback `base_url` port differs, a warning names the old and new port and how to keep the old one. **`init hook ensure` alignment warning** - After the (still suppressed) ensure step, for Claude/Codex hooks on **SessionStart only** (event read from the hook's stdin JSON with a 0.5 s bound), compare the agent's configured base URL port -- Claude: `settings.local.json` > `settings.json` > `~/.claude/settings.json` > env; Codex: project `.codex/config.toml` > `$CODEX_HOME/config.toml` -- with the managed manifest's port. On mismatch it prints hook JSON: `systemMessage` (user) + `hookSpecificOutput.additionalContext` (agent), including the fix command (`headroom init [-g] --port N <agent>`; N = the configured port if a Headroom proxy is live there, else the managed port). PreToolUse / Copilot stay silent. Wrapped in a catch-all: never raises, never blocks. **`doctor`** - New `readiness` row: probes `/readyz` (3 s) when `/livez` answered; live-but-not-ready is a WARN. - New `hooks` row: for each `init` manifest in scope (`init-user`, or this project's profile), checks the Claude settings / Codex `hooks.json` (CODEX_HOME-aware) still contain the Headroom hook and that the hook's program still resolves; WARN with the exact `headroom init ...` re-install command. SKIP when no init hooks are expected. - When the probed port is down, discovery (same bounded candidates) names a live proxy on another port in the `proxy` row with `headroom doctor --port N` and the matching `wrap`/`init --port N` commands. Exit-code semantics unchanged (still 2 when the probed proxy is down). **`wrap selfheal`** - After clearing a dead wrap route, if a live Headroom proxy exists elsewhere it prints `headroom wrap claude --port N` to stderr. Deliberately **warn-only** (no automatic repoint): adopting another session's proxy could silently change backend/mode/owner. **Tests** - `tests/test_cli/test_port_alignment.py` (47 tests): CODEX_HOME helpers + consumers, doctor CODEX_HOME/project precedence, port candidates/bounds/identity/disable switch, TTY prompt c/k/a, non-TTY warning, click option source handling (default vs `--port` vs `HEADROOM_PORT`), all wrap subcommands wired to the shared option, init port resolution (incl. hook skip), Codex provider repoint warning, hook JSON on SessionStart / silent on PreToolUse / never raises, doctor readiness/hooks/live-elsewhere, selfheal warning. All homes/cwd/workspace under `tmp_path`, probes injected -- no real `~/.codex`, `~/.headroom`, or proxy. - `tests/conftest.py`: autouse sets `HEADROOM_PORT_DISCOVERY=0` and unsets `CODEX_HOME` so a developer's running proxy / Codex home can't leak into CLI tests (tests that need discovery re-enable it). ## Testing - [x] Unit tests pass (`pytest`) -- affected suites, see below - [x] Linting passes (`ruff check .`, `ruff format --check`) - [x] Type checking passes (`mypy headroom`) -- no new errors - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_cli/test_port_alignment.py -q 47 passed in 1.61s $ python -m pytest tests/test_cli tests/test_cli_doctor.py tests/test_install tests/test_mcp_registry tests/test_provider_codex_runtime.py -q 2 failed, 1314 passed, 8 skipped FAILED tests/test_cli/test_unwrap_claude.py::test_unwrap_claude_removes_mcp_purges_retired_hook_and_stops_proxy FAILED tests/test_cli/test_wrap_stale_marker.py::test_signal_handler_unwinds_so_the_restore_can_run # both fail identically on unmodified upstream/main on Windows (signal.SIGHUP missing; retired rtk hook cleanup) -- pre-existing, unrelated $ ruff check headroom tests/... && ruff format --check ... All checks passed! $ mypy headroom headroom/fileperms.py:71 (os.fchmod on Windows) and headroom/integrations/langchain/chat_model.py:394 -- both pre-existing, files untouched ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.13, worktree checkout. Fully sandboxed: `HOME`/`USERPROFILE`/`APPDATA`, `CODEX_HOME` and `HEADROOM_WORKSPACE_DIR` pointed at a scratch dir; a stand-in server answering only `/livez` with `service: headroom-proxy` on port 47311; a hand-written deployment manifest on 47311 and an `init-user` manifest on 8787; Claude settings and `$CODEX_HOME/config.toml` routed to 47311. - Exact command / steps: run each `$` command below in the sandboxed environment (doctor, init hook alignment, port reconcile, codex provider repoint). - Observed result: each command's output is shown directly beneath it; doctor names the live proxy on 47311 and honors CODEX_HOME, the hook warns on port mismatch, and non-interactive reconcile keeps the requested port. ```text $ headroom doctor --port 47399 --json (rows) proxy fail | not reachable at http://127.0.0.1:47399; a Headroom proxy IS running on port 47311 | re-run with: headroom doctor --port 47311 (or export HEADROOM_PORT=47311); to use port 47311 from agents, re-run `headroom wrap <agent> --port 47311` / `headroom init --port 47311 <agent>` codex warn | routed to port 47311, but doctor probed port 47399 (<scratch>\codex\config.toml) | re-run with: headroom doctor --port 47311 $ headroom doctor --port 47311 --json (rows) proxy pass | running at http://127.0.0.1:47311 (up, demo) readiness warn | proxy is live but /readyz is not ready (upstream unreachable or still starting) codex pass | routed via <scratch>\codex\config.toml <- CODEX_HOME honored hooks warn | claude (user): no Headroom hook in <scratch>\home\.claude\settings.json; codex (user): no Headroom hook in <scratch>\codex\hooks.json | re-install with: headroom init -g claude && headroom init -g codex $ echo '{"hook_event_name":"SessionStart"}' | <init hook alignment, marker headroom-init-claude, profile init-user> {"systemMessage": "Headroom: Claude Code is configured for http://127.0.0.1:47311 but the Headroom proxy managed by `headroom init` (init-user) uses port 8787; a Headroom proxy is running on 47311, not the managed one. Fix: headroom init -g --port 47311 claude", "hookSpecificOutput": {"hookEventName": "SessionStart", "additionalContext": "..."}} (same for codex via CODEX_HOME; PreToolUse -> no output) $ non-interactive default-port reconcile (requested 47398) headroom: no Headroom proxy on port 47398, but one is running on port 47311. headroom: continuing with port 47398. To use the running proxy, pass --port 47311 (or set HEADROOM_PORT=47311). $ codex provider repoint headroom: warning: cfg.toml routes Codex's [model_providers.headroom] to port 47311; repointing it to port 8787. Pass --port 47311 to keep the existing route. ``` - Not tested: the interactive TTY prompt end-to-end in a real terminal (covered by unit tests with a stubbed prompt); a full `wrap <agent>` launch against a real proxy; the hook JSON as rendered inside a live Claude Code / Codex session (Codex's acceptance of `hookSpecificOutput.additionalContext` for SessionStart is assumed to mirror Claude's schema); Linux/macOS. ## Runtime Rollout Safety - Rollout-managed feature(s): none (CLI only; proxy runtime untouched). - Minimum rollout channel: Stable. - Stable/default behavior changed: default `wrap`/`init` port now follows `HEADROOM_PORT`; discovery only warns or prompts, never switches ports non-interactively. - Kill switch / disable path: `HEADROOM_PORT_DISCOVERY=0`; explicit `--port` bypasses discovery. - Unsafe override required: No. - Qualification impact: None (CLI only; no proxy runtime or qualification-gated behavior touched). - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (CLI help text updated; no docs pages changed) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes (2 pre-existing Windows-only failures noted above) - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Design points for review: - `HEADROOM_PORT` is treated as an explicit choice (no discovery/prompt), same as `--port`. - The TTY prompt defaults to *connect* to the live proxy; non-TTY never adopts. - `unwrap` commands were intentionally left on their old `--port` default: discovering a port there could stop a proxy the user did not target. - `wrap` subcommand `--port` now also reads `HEADROOM_PORT` via Click's envvar, so an invalid `HEADROOM_PORT` is a usage error for `wrap` (init/hook ignore invalid values). - The hook reads stdin for `hook_event_name` in a daemon thread bounded to 0.5 s so a host that never closes stdin cannot stall session start. |
||
|
|
66258c40a4 |
fix: make Headroom work behind Zscaler and other TLS-inspecting networks (#3831)
## Description Customers behind **Zscaler** (and Netskope, Palo Alto, Cisco Secure Access, …) report that Headroom "cannot work": every upstream request fails with `CERTIFICATE_VERIFY_FAILED`, and the agent only sees a bare 502. These gateways inspect TLS by re-signing every certificate with a company root. IT installs that root in the **OS certificate store** (macOS Keychain, Windows cert store), which is why browsers, `curl` and Claude Code keep working. Headroom never read that store: - The httpx upstream client trusted certifi only. - On macOS, the stdlib default reads the static file `/etc/ssl/cert.pem`, not the Keychain. - The Rust proxy trusted Mozilla's webpki roots only. Only 3 of about 40 outbound call sites used the shared CA helper. We had been patching one path at a time (#741, #998, #1308, #3244, #3344, #3358, #810, #2029). This PR sets **one trust policy, based on the OS store, used everywhere.** Closes #3358 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **1. Verify upstream TLS through the OS store by default** (`headroom/proxy/ssl_context.py`) - New `HEADROOM_CERT_STORE`, default `system,bundled` (same shape as Claude Code's `CLAUDE_CODE_CERT_STORE`). It uses [`truststore`](https://truststore.readthedocs.io/), which pip has used by default since 24.2, plus certifi's public roots. - The same policy covers the HTTP upstream client, both WebSocket upstreams, and the `urlopen` paths (Copilot auth). - **The builders now always return a concrete, verifying `SSLContext`**, never a `bool`. The legacy default is built explicitly as httpx's `verify=True` equivalent (certifi). - OS-store contexts are cached per configuration. They set `VERIFY_X509_PARTIAL_CHAIN` so an exported intermediate still works as a trust anchor (matching `create_default_context()` on 3.13). On macOS and Windows the OS verifies the chain, so the Python 3.13 strict-mode rejection (#1308) no longer happens. - `HEADROOM_CA_BUNDLE` and `NODE_EXTRA_CA_CERTS` **add** roots. `SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE` keep their **replacement** meaning and turn the OS store off. - `HEADROOM_CERT_STORE=bundled` restores the old behavior. The aliases `bundle`, `certifi`, `os` and `native` are also accepted. - `truststore.inject_into_ssl()` (covers `huggingface_hub`, tiktoken and OTEL) runs only in processes the Headroom CLI owns. The CLI marks them with `HEADROOM_PROCESS_TRUST=1`, and proxy workers inherit it. Applications that embed Headroom keep their own process-wide TLS behavior. - `truststore` is added as a core dependency: pure Python, with no dependencies of its own. **2. Errors that say what's wrong** (`headroom/proxy/tls_diagnostics.py`) - When the certificate check fails, the Anthropic, streaming (SSE) and OpenAI passthrough errors now name who signed the certificate and give the fix. For example: *"…The certificate was issued by 'O=Zscaler Inc., CN=Zscaler Intermediate Root CA': Zscaler is inspecting this connection. Fix (pick one): …"* - The issuer comes from a separate handshake that inspects the certificate only: - It sends no data and requires **TLS ≥ 1.2**. - It has a 4-second timeout and is cached for 5 minutes per host. A burst of failures shares a single handshake. - It uses the same environment-variable proxies as the real client. - It targets the host that actually failed. - **Security:** on the request path it skips hosts that resolve to non-public addresses. It also connects to the exact address it checked, instead of looking the host up again. A per-request upstream therefore can't be used to read an internal service's certificate or to rebind DNS. **3. `headroom doctor`** - A new **tls trust** row reports the running proxy's policy, read from `/health`. - A **proxy env** warning appears when `HTTP(S)_PROXY` is set but `NO_PROXY` doesn't exempt loopback. - `--network` (plus a repeatable `--network-url`) probes the provider and model hosts. For each it shows the certificate issuer, whether Headroom trusts it, and whether a gateway block page came back. On failure it prints a paste-ready request for IT. **4. `headroom wrap`** - Merges `NO_PROXY` and `no_proxy`, appends `127.0.0.1,localhost,::1`, and writes the result to both, so a corporate proxy doesn't capture the agent's traffic to Headroom. - The notice goes to stderr, because the JSON that `--prepare-only` prints on stdout is parsed by `install.sh`. **5. Rust proxy** (`crates/headroom-proxy`) - reqwest now enables `rustls-tls-native-roots`, and `HEADROOM_CA_BUNDLE` / `NODE_EXTRA_CA_CERTS` add roots. - **Fixes a bug already on `main`: `wss://` upstreams panicked.** Both ring and aws-lc-rs are compiled into this binary, so `rustls::ClientConfig::builder()`, which `tokio_tungstenite::connect_async` calls, panics with "no process-level CryptoProvider available". WebSocket upstreams now use an explicit config: the ring provider, webpki + OS + additive roots, and ALPN `http/1.1`. **6. Docs** - New page **Corporate Networks**: trust settings, HTTP proxies, a domain table for IT, and notes on Caution/Isolate, SSE buffering, process bypass, tenant restrictions and `HF_ENDPOINT` mirrors. - README section updated, correcting a wrong claim about which trust store the Rust core uses. **7. `uv.lock`** - `main`'s lockfile plus `truststore` only. The diff is 11 lines: the new package entry and two lines in `headroom-ai`'s dependency lists. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_corporate_tls.py tests/test_ssl_context.py tests/test_copilot_auth.py tests/test_cli_doctor.py -q ============================= 278 passed in 2.88s ============================== $ pytest <117 related files: doctor, wrap, streaming, codex, copilot, ssl, upstream, websocket, passthrough> -q ====== 4 failed, 1876 passed, 25 skipped, 7 warnings in 132.42s (0:02:12) ====== # the 4 failures: tests/test_codex_live.py ordering issue, identical on main (see Additional Notes) $ cargo test -p headroom-proxy passed=478 failed=0 suites=37 $ cargo clippy -p headroom-proxy --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) $ ruff check . -> All checks passed! $ ruff format --check . -> 1660 files already formatted $ mypy headroom --ignore-missing-imports -> Success: no issues found in 544 source files $ uv export --frozen --no-dev --no-hashes | grep truststore -> truststore==0.10.4 ``` `tests/test_corporate_tls.py` runs a **fake Zscaler** HTTPS server. Its root is `O=Zscaler Inc.` with **non-critical basicConstraints**, like the real 2014 root. The tests check that: - certifi alone rejects it; - the default policy plus `HEADROOM_CA_BUNDLE` or `NODE_EXTRA_CA_CERTS` accepts it; - the probe names Zscaler, and the request path skips private hosts; - a streaming request surfaces the actionable message in a 502 SSE error; - a burst of failures shares one handshake; - the notices and JSON output from `wrap` go to the right streams; - process injection is gated as described. ## Real Behavior Proof - **Environment:** - macOS (Darwin 25.6, Apple Silicon), Python 3.12.13 venv - Python 3.13.15 via `uv run --python 3.13` - Linux `python:3.13-slim` in Docker - Rust proxy built from this branch - **Steps and results:** 1. **Standalone fake-Zscaler script** (same root as above) on Python 3.13: - old stdlib path, `ssl.create_default_context(cafile=root)` → `CERTIFICATE_VERIFY_FAILED: Missing Authority Key Identifier` - new `truststore` context + root → `200` - identical results on macOS and in `python:3.13-slim`. 2. **`headroom doctor --network`** from this branch on an uninspected network: every endpoint shows `reachable, certificate trusted` (api.anthropic.com 401, api.openai.com 401, api.githubcopilot.com 400, huggingface.co 200, openaipublic 404); the tls trust row shows `OS certificate store + certifi`. 3. **Rust `wss://` connection** (temporary test) to `wss://echo.websocket.org`: - `tokio_tungstenite::connect_async` as on `main` → **panic** (no process-level CryptoProvider) - `connect_async_tls_with_config` with the new config → `Ok(())` - **Not tested:** - A real Zscaler, Netskope or Palo Alto tenant. - Windows. `truststore` supports the Windows cert store, but no Windows run was done here. - A root installed into the OS store. That needs admin rights, so the additive-bundle path stood in for it; it goes through the same OS verifier. ## Runtime Rollout Safety - **Rollout-managed feature(s):** none. This is trust-store selection, not a rollout-gated feature. - **Minimum rollout channel:** n/a. - **Stable/default behavior changed:** yes. Upstream TLS now also trusts the OS certificate store; certifi is still loaded, so public endpoints behave the same. The Rust `wss://` upstream no longer panics. - **Kill switch / disable path:** `HEADROOM_CERT_STORE=bundled` restores certifi-only trust. An explicit `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` behaves exactly as before. - **Unsafe override required:** none. Nothing here disables certificate verification. - **Qualification impact:** none expected. Default behavior on uninspected networks is unchanged. - **Rollback path:** revert the PR, or set `HEADROOM_CERT_STORE=bundled`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) n/a (CLI and proxy behavior). ## Additional Notes **Review response** (@JerrettDavis): - **CodeQL:** the 5 alerts that were still open on `efe173c9` had two causes. - **Certificate-validation alerts:** CodeQL traced a `None` returned by the optional-context helpers through the `configured or default` fallback into `verify=`. All contexts are now built by a single `_build_context()`, and every branch returns a verifying `SSLContext`. `build_httpx_verify()` is typed `-> ssl.SSLContext`. - **TLS-version alerts:** every context the module creates now sets an explicit `minimum_version = TLSv1_2`. That includes the certificate-inspection probe, whose no-validation scope stays confined to that handshake behind the public-address / environment-proxy CONNECT guard. The test server also requires TLS 1.2, and the host assertion in the test is a regex. - **Verified locally with CodeQL 2.27.1** before pushing: - the same 5 alerts reproduce on `efe173c9`, and 0 remain on the fix commit; - `python-security-extended` over every Python file this PR changes finds the same 14 pre-existing alerts on `main`'s versions and on the branch's versions, so the PR adds none. - **`uv.lock`:** rebuilt as `main`'s lockfile plus `truststore` only (11 lines). Re-locking with current uv also rewrites markers across the file, because `main`'s lockfile predates the current `pyproject.toml`. That is left for a separate lock refresh. **Earlier review:** a high-effort code review of this branch produced 10 findings plus 3 minor notes. Commit `fix: address review findings…` resolves 9 of the 10 and two of the notes; the third note (lockfile markers) checked out as a false alarm. The remaining finding is the last follow-up below. **Follow-ups (not in this PR):** - SSE keep-alives and HTTP/1.1 fallback when a gateway buffers a stream. - A `wrap` pre-flight that steps aside when the upstream is unreachable. - The Rust core's fastembed/hf-hub embedding download, which hard-codes WebPki roots (needs an hf-hub patch; BM25 is the fallback, and `HF_ENDPOINT` is documented). - A CI job with a fake root installed in the OS store on macOS and Windows. - Map certificate failures once, in a shared layer, so batch, Gemini and the other handlers get the TLS hint too. **Known unrelated failures:** - 4 `tests/test_codex_live.py` tests fail when run after other files in this set, both on `main` and on this branch. They pass when run on their own. - After merging the latest `main` (thanks for the refresh), `tests/test_cli/test_wrap_helpers.py::test_start_proxy_strips_ambient_worker_configuration` and `::test_start_proxy_timeout_kills_failed_new_process` fail locally with `TypeError: <lambda>() takes 0 positional arguments but 1 was given`. They fail the same way on a clean checkout of current `main`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
21d166b77a |
test: stop three sources of cross-test pollution on main (#3774)
## What
Three tests fail on `main` in a full-suite run and pass when their file
is run alone. Each is state one test leaves behind that a later test
reads. This PR isolates all three. **No production code changes.**
| Failing test | Left behind by | Fix |
|---|---|---|
| `/readyz` readiness tests reporting kompress `ready: true` |
`test_kompress_failsafe.py::test_canary_disabled_via_env` calling
`compressor.preload()`, which writes the model into the module-level
`_kompress_cache` | autouse fixture snapshots/restores `_kompress_cache`
|
| `test_runtime_log_refuses_a_symlinked_path` |
`tests/gateway/test_compress_turn_seam.py` leaking an
`_OwnerOnlyRotatingFileHandler` on the `headroom` logger | autouse
fixture detaches + closes any handler named `_PROXY_LOG_HANDLER_NAME` |
| 2 × `test_dense_line_elider.py` | the optional `html` extra
(`trafilatura`) being installed, so `HTMLExtractor` wins routing instead
of the elider | both tests pin `enable_html_extractor=False` |
The third is not pollution between tests but pollution from the
environment: CI does not install the `html` extra, so CI has never seen
these two fail. Anyone with `pip install headroom[html]` in their venv
does.
## How to verify
Before:
```
pytest tests/test_kompress_failsafe.py tests/test_proxy_health.py # health tests fail
pytest tests/gateway/test_compress_turn_seam.py tests/test_install/test_runtime.py # symlink test fails
pytest tests/test_transforms/test_dense_line_elider.py # fails iff trafilatura installed
```
After: all pass, and each file still passes alone.
Slices run locally (venv `bin` on PATH):
- `-k "kompress or health or elider or breakpoint or transform"` → 1040
passed, 17 skipped
- `-k "install or savings or stats or memory or learn or router"` → 2960
passed, 59 skipped
## Behaviour change
None for users. For test authors: the kompress model cache and the proxy
log handler no longer survive a test, so a test that *relied* on a
previous test having preloaded the model would now have to preload it
itself. No test in the suite does.
## Two defects found while debugging this, deliberately not fixed here
1. **`HTMLExtractor` emits lossy output with no CCR retrieval marker.**
A 3232-char HTML tool result is reduced to `"hi there"` —
`<script>`/`<style>` stripped — and booked as 99.8% savings with no
marker, so the original cannot be retrieved. Present since
|
||
|
|
26a2c4939c |
fix(binaries): fail closed when a download carries no sha256 pin (#3724)
## A-7 (S2, supply chain) — binary verification failed open on an
unpinned URL
`headroom/binaries.py` skipped SHA-256 verification entirely, with only
a log line, when the registry carried **no pin** for a download URL — in
both `_verify_sha256()` (the `resolve()` fetch path) and
`verify_download_bytes()` (the in-memory installer path). An
off-registry version override, or an installer building a filename the
registry does not describe, therefore installed whatever the mirror
served, protected only by HTTPS transport trust.
`verify_download_bytes`'s own docstring already claimed it *"refuses an
unpinned URL"*. It did not — the documentation described behaviour the
code did not have.
**Both claims in the backlog verified against `origin/main` (
|
||
|
|
0d99e5595d |
fix(tests): isolate persisted settings from developer state (#3449)
## Description Clearing `HEADROOM_*` environment variables still leaves the developer's persisted settings active. Tests that construct the proxy or use the settings store can read `~/.headroom/settings.json`, and settings endpoint tests can write back to it. Give each test a temporary settings store through the existing `HEADROOM_SETTINGS_PATH` override. Explicit path-precedence tests can opt out or supply their own path. Extracts the settings-isolation portion of #2524, including the CLI precedence-test opt-outs, with focused regression coverage. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Set a fresh per-test settings path in `tests/conftest.py` after the shared fixture scrubs developer environment variables. - Keep both CLI settings-precedence tests in `tests/test_cli_proxy_env.py` pointed at their deliberately seeded temporary workspace. - Add `tests/test_settings_isolation.py`: a regression that reads and writes through the real settings store, verifies an empty initial state for each case, and preserves a simulated developer settings file byte-for-byte. ## Testing - [x] Unit tests pass (`pytest`) — 368 targeted cases across settings, paths, proxy endpoints, CLI configuration, and the related frozen-prefix regression - [x] Linting passes (`ruff check .`) — for the three changed files - [x] Type checking passes (`mypy headroom`) — via the full-package mypy commit hook - [x] New tests added for new functionality - [x] Manual testing performed — new regression observed failing before the fixture change ### Test Output ```text python -m pytest tests/test_settings_isolation.py tests/test_proxy_loopback_gating.py tests/test_proxy/test_settings_store.py tests/test_proxy_settings_endpoints.py tests/test_paths.py tests/test_proxy/test_settings_fresh_process_precedence.py -q --tb=short 233 passed python -m pytest tests/test_cli_proxy_env.py tests/test_cli_proxy_improvements.py tests/test_paths_backward_compat.py -k 'not TestMissingProxyDepsError' -q --tb=short 134 passed, 2 deselected python -m pytest tests/test_proxy_anthropic_cache_stability.py -k memory_context_avoids_system_mutation_when_prefix_frozen -q --tb=short 1 passed, 27 deselected python -m pytest tests/test_settings_isolation.py tests/test_proxy_loopback_gating.py::test_settings_post_trusted_gateway_client_same_origin_allowed tests/test_cli_proxy_env.py::TestSettingsFileToEnv -q --tb=short 6 passed ruff check tests/conftest.py tests/test_settings_isolation.py tests/test_cli_proxy_env.py All checks passed! ruff format --check tests/conftest.py tests/test_settings_isolation.py tests/test_cli_proxy_env.py 3 files already formatted git diff --check ``` The broader CLI run encountered two dependency-check failures in `TestMissingProxyDepsError`: those tests mock `builtins.__import__`, while the dependency check uses `importlib.import_module`. The missing-FastAPI case also fails in the unchanged checkout, so both are pre-existing and are excluded from the targeted passing run above. An existing Starlette TestClient deprecation warning remains. ## Real Behavior Proof - Environment: macOS, Python 3.12, existing native extension and installed test dependencies; clean worktree based on `e67b3c8a`. - Exact command / steps: seed a simulated developer home with `{"target_ratio": 0.9}`, run `python -m pytest tests/test_settings_isolation.py -q` against the unchanged fixture, then apply the `tests/conftest.py` change and rerun `python -m pytest tests/test_settings_isolation.py tests/test_proxy_loopback_gating.py::test_settings_post_trusted_gateway_client_same_origin_allowed tests/test_cli_proxy_env.py::TestSettingsFileToEnv -q --tb=short`, followed by the wider targeted runs listed under Test Output. - Observed result: before the fixture change, both regression cases fail because `settings_store.load()` returns the developer's persisted `target_ratio` instead of an empty mapping. After the change, both cases start empty, save and reload their own settings, and leave the simulated developer file byte-for-byte unchanged; the formerly failing settings endpoint test passes and the CLI precedence tests still read their own seeded workspace (6 passed). - Not tested: full repository suite, other operating systems, or live providers. Coverage is limited to persisted settings; tool-home and network isolation are separate follow-ups. ## Runtime Rollout Safety - Rollout-managed feature(s): None. Test-only change under `tests/`; no shipped code path is touched. - Minimum rollout channel: N/A — nothing ships to users from this change. - Stable/default behavior changed: No. Production behavior is unchanged; only the pytest fixture environment gains a per-test `HEADROOM_SETTINGS_PATH`. - Kill switch / disable path: N/A — no runtime feature to disable. Tests that need a specific path opt out of the fixture or set their own `HEADROOM_SETTINGS_PATH`. - Unsafe override required: No. - Qualification impact: None. No new qualification gates; the suite continues to run unchanged apart from stronger isolation. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] My changes generate no new warnings - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Test-only change: no documentation, dependency, or public configuration updates apply. The two `TestMissingProxyDepsError` failures noted above are pre-existing and unrelated to this fixture change. |
||
|
|
1f96dabc19 |
fix(security): address u9up assessment findings (WEB-01–07) (#2207)
Hardens client-selected upstreams, memory identity resolution, downloaded binary integrity, telemetry import, Docker defaults, Neo4j credentials, and archive extraction. Refreshes the branch against current main and preserves newer same-origin and loopback protections. |
||
|
|
b7f342c153 |
fix(wrap): verify proxy deps before mutating Codex config (#1628)
## Description \`headroom wrap codex\` now verifies that optional proxy dependencies (\`headroom-ai[proxy]\`) are installed before mutating Codex \`config.toml\`. If the check fails, the command exits with the same error message as \`headroom proxy\` and leaves Codex config untouched. Fixes #1614 (Bug 1: config mutated before proxy dependency check). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Extract \`ensure_proxy_dependencies()\` in \`headroom/cli/proxy.py\` (shared with \`headroom proxy\`) - Call it at the start of \`wrap codex\` when \`not no_proxy\`, before config snapshot/injection - Add regression tests for prepare-only abort, \`--no-proxy\` skip, and import failure messaging ## Testing - [x] Unit tests pass (\`pytest\`) - [x] Linting passes (\`ruff check .\`) - [ ] Type checking passes (\`mypy headroom\`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output \`\`\`bash pytest tests/test_cli/test_wrap_codex.py::test_wrap_codex_aborts_before_mutating_config_when_proxy_deps_missing \ tests/test_cli/test_wrap_codex.py::test_wrap_codex_skips_proxy_dependency_check_with_no_proxy \ tests/test_cli/test_wrap_codex.py::test_ensure_proxy_dependencies_exits_when_server_import_fails -q # 3 passed ruff check headroom/cli/wrap.py headroom/cli/proxy.py tests/test_cli/test_wrap_codex.py ruff format --check headroom/cli/wrap.py headroom/cli/proxy.py tests/test_cli/test_wrap_codex.py \`\`\` ## Real Behavior Proof Environment: Linux (Ubuntu), Python 3.12, local checkout with \`PYTHONPATH\` pointed at patched sources. Exact command / steps: 1. Created a temp \`~/.codex/config.toml\` with \`model_provider = "openai"\`. 2. Patched \`headroom.cli.wrap.ensure_proxy_dependencies\` to raise \`SystemExit(1)\` (simulating missing \`[proxy]\` extra). 3. Ran \`headroom wrap codex --prepare-only --no-serena --port 8787\`. Observed result: exit code 1; \`config.toml\` unchanged; no \`config.toml.headroom-backup\` created; no \`[mcp_servers.headroom]\` block written. Also verified: \`headroom wrap codex --prepare-only --no-proxy ...\` does not invoke the dependency check. Not tested: Windows-specific proxy selector behavior (covered separately in #1655). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did not edit CHANGELOG.md; release notes are generated automatically --------- Co-authored-by: syf2211 <syf2211@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
|
99f07e7bbd |
fix(proxy): cache litellm model resolution to stop repeated Provider List spam
## Description The proxy repeatedly prints LiteLLM's `Provider List: https://docs.litellm.ai/docs/providers` banner during normal operation, with no explanation or way to suppress it (#2851). Root cause: `_resolve_litellm_model()` in `headroom/proxy/savings_tracker.py` runs on every savings-tracking update (i.e. every request). For any model LiteLLM can't price (a custom/local/gateway model name — e.g. the reporter's local oMLX setup), the uncached fallback path calls `litellm.cost_per_token(...)` purely to probe resolvability. When that probe fails, LiteLLM prints the banner as an internal side effect before raising, and since the probe was never cached, it re-fires on every single request for the same unresolvable model. **Update:** review flagged that the first version of this fix cached into a plain, unbounded `dict` keyed by the (client-controlled) model name — a memory-retention path on a request-facing proxy, since a caller can grow it without limit by sending a new model string on every request. Replaced with a bounded `functools.lru_cache`; see Changes Made below. Closes #2851 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/savings_tracker.py`: `_resolve_litellm_model()` is now decorated with `@lru_cache(maxsize=256)` instead of backing onto a hand-rolled unbounded `dict`. An evicted model name simply re-probes LiteLLM on next use — never a correctness issue, only whether the noisy failure banner reruns for that specific name. - `tests/conftest.py`: added a global `autouse` fixture, `_reset_litellm_model_resolution_cache`, that clears the cache before and after every test. It's process-lifetime and module-global, and several existing tests monkeypatch `savings_tracker.litellm` with different behavior per test while reusing common model names like `"gpt-4o"` — without a reset, whichever test resolves a name first silently wins that cache slot for the rest of the run and later tests stop exercising their own fake. - `tests/test_savings_tracker_litellm_resolution_cache.py` (new): regression tests for the three properties that actually matter — repeated resolution of one unknown model only probes LiteLLM once, resolving far more distinct names than the bound never grows the cache past it, and an evicted name is transparently re-probed rather than reusing a slot it no longer owns. - No behavior change for models LiteLLM can already price (fast path via `model_cost` lookup) — only the noisy uncached probe path is memoized, same as before. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) — not run; `mypy` isn't installed in this environment - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python3 -m pytest tests/test_proxy_savings_history.py tests/test_savings_tracker_zero_price.py \ tests/test_savings_tracker_litellm_resolution_cache.py -q tests/test_proxy_savings_history.py .................................... [ 73%] ... [ 79%] tests/test_savings_tracker_zero_price.py ....... [ 93%] tests/test_savings_tracker_litellm_resolution_cache.py ... [100%] 49 passed, 1 warning in 1.26s # Re-run in reversed file order to check for the exact order-dependence the # review flagged — same 49 passed, no failures either direction: $ python3 -m pytest tests/test_savings_tracker_litellm_resolution_cache.py \ tests/test_savings_tracker_zero_price.py tests/test_proxy_savings_history.py -q 49 passed, 1 warning in 1.11s $ python3 -m ruff check headroom/proxy/savings_tracker.py tests/conftest.py \ tests/test_savings_tracker_litellm_resolution_cache.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS, Python 3.12.3, this repo checked out locally. - What changed since the last review pass: I got the compiled `headroom._core` Rust extension in hand (by installing the published `headroom-ai[all]` wheel into a separate venv and copying its `_core.abi3.so` next to this local source tree — same Python ABI, pure-Python edits in `savings_tracker.py` don't touch the compiled boundary). That unblocked the full test files this fix touches, including `tests/test_proxy_savings_history.py`, which was previously reported as untestable here. - Exact command / steps: three properties asserted directly against the real (now-bounded) cache in `tests/test_savings_tracker_litellm_resolution_cache.py`: 1. Resolve the same unresolvable model 5 times → assert the underlying `litellm.cost_per_token` probe fired exactly once. 2. Resolve `_MODEL_RESOLUTION_CACHE_MAXSIZE + 50` distinct model names → assert `_resolve_litellm_model.cache_info().currsize` stays at exactly `_MODEL_RESOLUTION_CACHE_MAXSIZE` (256), never higher — this is the actual memory-retention fix the review asked for. 3. Resolve one model, push exactly `maxsize` other distinct names through to evict it via LRU, then resolve it again → assert it re-probed (call count went 1 → 2), proving eviction is real and not just an untested cache_info number. - Observed result: all three pass; full affected-file suite (49 tests) passes in both forward and reversed run order, confirming the new `conftest.py` fixture actually fixes the cross-test leakage risk (verified by literally reordering the files, not just by inspection). - Not tested: a live HTTP request against a running `headroom proxy` process specifically re-exercising this bounded-cache commit — the earlier "20 simulated requests" proof against the previous (unbounded-dict) version of this fix was via a standalone script, not a real server; I have not repeated that specific live-server pass against this commit. The unit-level proof above exercises the exact same function (`_resolve_litellm_model`) the real proxy calls per-request from `headroom/proxy/server.py`, so I'm confident it generalizes, but flagging the gap rather than implying I re-ran it live. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas — the bound/eviction rationale is commented above `_resolve_litellm_model`, and the cross-test leakage rationale is commented above the new `conftest.py` fixture - [ ] I have made corresponding changes to the documentation — N/A, internal implementation detail with no user-facing API/doc surface - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - `mypy` still hasn't been run — not installed in this sandbox, and I didn't want to widen the PR further by installing/configuring it just for this. Flagging rather than silently skipping. - The earlier "Additional Notes" gap about `test_proxy_savings_history.py` being untestable in this environment is resolved (see Real Behavior Proof) — it now runs and passes, including the pre-existing `test_litellm_resolution_and_savings_estimation_fallbacks` test that exercises `_resolve_litellm_model` with a mutated `model_cost` dict across several assertions in one test. - Deliberately did not also bound `headroom/pricing/litellm_pricing.py`'s sibling `_resolved_model_cache` — same shape of cache, arguably the same exposure — since it's outside this PR's diff and touching it wasn't asked for. Flagging in case a maintainer wants it as a fast follow-up rather than silently leaving it unmentioned. --------- Co-authored-by: connectsudhindra-gif <connectsudhindra-gif@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
9cfb00838a |
fix(telemetry): anonymous compression stats — no prompts, no data (#2728)
## In one line
Headroom starts reporting **how well compression is working** — counters
and percentages only. **No prompts. No code. No file paths. Nothing
about what you're building.**
## Why
Right now nobody knows whether compression actually helps real users.
You can see your own numbers in `/stats`, but that's it — there's no way
to tell whether a given workload compresses well, or why it sometimes
doesn't. This closes that loop so we can make compression better for
everyone.
## Exactly what gets sent
One message per session, and every 5 minutes while you're active:
```json
{
"session": { "id": "random", "turns": 47, "duration_s": 4210, "seq": 3 },
"tokens": { "original": 890000, "attempted": 410000, "saved": 320000,
"tool_saved": 48000, "cache_read": 210000 },
"rates": { "saved_pct": 35.96, "eligible_pct": 46.07, "yield_pct": 78.05,
"cache_read_pct": 23.60, "overhead_pct": 1.96 },
"compression": { "transforms": {"crush": 47}, "passthrough_turns": 0 },
"skips": {},
"sources": { "proxy": 47 },
"providers": ["anthropic"],
"models": ["claude-sonnet-4-5-20250929"],
"failures": 2
}
```
Plus a random install ID, the Headroom version, and OS/architecture
(`darwin`, `arm64`).
That's the whole thing. A full example lives at
`deploy/beacon/sample-event.json`.
## What is never sent
- Your prompts or the model's responses
- Your code
- File paths, project names, repo names
- Tool names or MCP server names
- Hostname, username, or IP address
- Custom or fine-tuned model names (an id like `ft:gpt-4o:acme-corp:…`
contains a company name, so only models in a public registry are
reported)
**This is structural, not a pinky-swear.** Every value in the payload is
a number, a fixed word, or a random ID — there is no free-text field
anywhere for content to hide in. The receiver
(`deploy/beacon/worker.js`, in this repo so you can read it) drops
anything not on an explicit allowlist before storing.
## Turning it off
Any one of these:
```bash
HEADROOM_BEACON=off # or
DO_NOT_TRACK=1 # or
# offline mode
```
It's on by default, and Headroom says so at startup:
```
Telemetry: anonymous compression stats — never prompts, code, or file paths.
Helps us improve compression | Off: HEADROOM_BEACON=off
```
`HEADROOM_TELEMETRY` is a **separate** switch that still only affects
local stats. If you had turned that on, this change does not start
uploading anything — you answered a different question, and upgrading
should not change the answer.
## Why the percentages, not just "tokens saved"
"We saved 36%" hides the interesting part. In the example above only
**46% of tokens were eligible** for compression at all — the rest is
frozen cache prefix and system prompts we deliberately do not touch. Of
what we *could* touch, we removed **78%**.
Those are two separate problems. Raising eligibility is proxy work;
raising yield is compressor work. A single number cannot tell us which
to fix.
## Coverage
`emit_request_outcome` is a single chokepoint —
`handler.metrics.record_request` is called from exactly one place,
inside the funnel — so all 30 `RequestOutcome` construction sites are
covered: Anthropic, OpenAI, Gemini, Bedrock, batch, streaming, and the
long-lived Codex Responses-WS path.
The `headroom_compress` MCP path bypassed that funnel and is now wired
in separately. It has a different shape (no provider, no upstream
latency, and everything handed to the tool is eligible by construction),
so `sources` counts turns by origin — MCP turns always read
`eligible_pct: 100` and must not drag the proxy's real eligibility
ceiling upward.
**Subagents.** All subagent traffic through the proxy merges into one
session, which is correct for savings and retention but means `turns`
conflates fan-out with depth. Fan-out is still derivable —
`compression.latency_ms_total / session.duration_s` gives the
concurrency ratio (~1x serial, ~4x for four parallel agents), so no
extra field is needed. Verified no lost updates under 6-way concurrency
(1,200 turns).
**Known gap:** `--workers N` gives each process its own aggregator, so
one user session becomes up to N. Token totals and fleet rates stay
correct; session counts inflate. This matches the existing documented
limitation that TOIN state, CostTracker, and the prefix tracker are all
per-process.
## Notes for reviewers
- **Cumulative snapshots, not deltas.** Every report restates running
totals under one session ID, so the highest `seq` per `(install,
session)` is the complete session. Dedupe is a window function, and a
lost report costs nothing.
- **Never breaks the proxy.** Every path swallows its own exceptions;
uploads go out on a daemon thread so nothing blocks the request loop.
- **Explicit User-Agent is load-bearing.** urllib's default is blocked
by Cloudflare (error 1010). Combined with fire-and-forget error
handling, that would have failed every upload while looking perfectly
healthy.
- **The exit flush was broken and is fixed.** `atexit` handed the POST
to a daemon thread, and daemon threads are killed before they finish
during interpreter shutdown — so nothing was sent. That silently dropped
*every session shorter than the 5-minute heartbeat*, plus all
short-lived subagent MCP processes. The exit path now posts
synchronously with a 2s timeout.
- Receiver and query tooling are in `deploy/beacon/`.
## Testing
- `python -m headroom.telemetry.session` self-check: dedupe, cumulative
totals, dropped-report recovery, payload contains no model id or
prompt-derived string, allowlist coverage
- 175 telemetry/outcome tests pass; 6 new ones cover the opt-out notice
- Verified end to end against a live deployment: client → receiver →
storage → query
## Still to do before release
The default endpoint currently points at a temporary `workers.dev` URL.
It needs to move to a Headroom-owned hostname before this ships in a
tagged release — noted inline at `DEFAULT_ENDPOINT`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
759209cff3 |
fix(wrap/serena): stop creating serena_config.yml, unbricking Serena on fresh installs (#2676)
## Description
`_ensure_serena_dashboard_disabled()` wrote a one-key bootstrap config
(`web_dashboard_open_on_launch: false`) into
`~/.serena/serena_config.yml` when the file was absent, assuming Serena
fills in any key it omits.
Verified against **Serena 1.6.2.dev0**
(`serena/config/serena_config.py`), that holds for every field except
one. Serena autogenerates its own complete config **only when the path
does not exist**:
```python
if not os.path.exists(config_file_path):
cls._generate_config_file(config_file_path)
```
Once any file is present it validates instead. Every other field falls
back to a dataclass default via `get_value_or_default`, but a missing
`projects` key is fatal (~line 1064):
```
SerenaConfigError: `projects` key not found in Serena configuration.
```
So Headroom's own bootstrap file killed Serena on **every machine
without a pre-existing Serena config**. The MCP server exited during
handshake — surfacing as `connection closed: initialize response` on
Codex and a bare `MCP error -32000: Connection closed` on OpenCode
(#2674) — and `serena project index` failed identically.
Headroom now leaves that file to Serena. That is immune to Serena adding
required keys later; guessing the schema is what caused the outage. The
popup never needed the file anyway: `build_serena_spec` passes
`--open-web-dashboard False`, which Serena applies *after* loading the
config (`serena/mcp.py:361` — `config.web_dashboard_open_on_launch =
open_web_dashboard`), so the flag wins regardless of what is on disk.
An **existing** config is still edited in place — dashboard key flipped,
`projects: []` backfilled to repair machines an affected version already
wrote — preserving a populated `projects` list, other keys and comments.
Closes #2674
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- `_ensure_serena_dashboard_disabled()` never creates
`serena_config.yml`; it only edits an existing one, and backfills
`projects: []` there to repair already-broken machines.
- Dropped `_scope_serena_languages` + `_detect_repo_languages` +
`_EXT_TO_SERENA_LANGUAGE` (**−133 lines**). Dead weight: Serena
determines languages itself in `ProjectConfig.autogenerate`
(`_determine_project_language_servers`) and records them under
`language_servers` — `languages`, which Headroom wrote, is a legacy name
Serena migrates via `RENAMED_FIELDS`. Serena's generated file uses a
block-style list, so our single-line-flow regex never matched it: on any
Serena-generated `project.yml` the function was a **verified no-op**.
The only case where it acted was creating the file — the same
partial-config trap — which also skipped the `project.local.yml` sidecar
Serena writes alongside.
- **Test isolation:** the MCP install ledger defaults to
`~/.headroom/mcp_installs.json`, so any test registering a server wrote
into the developer's real ledger (observed adding a live `claude/serena`
entry during a local run). `conftest.py` now redirects it per-test.
- **Repo config:** `.serena/project.yml` carried a stale `project_name`
(`"feature-opencode-wrap"`) and listed only `typescript`, so Serena's
symbol index skipped 1331 Python and 194 Rust files for every
contributor.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_wrap_code_memory.py tests/test_cli/test_wrap_serena_boost.py \
tests/test_cli/test_serena_migrate.py tests/test_cli/test_serena_disable.py -q
35 passed, 1 skipped in 0.84s
$ SERENA_SRC=<serena checkout> pytest tests/test_wrap_code_memory.py -q
13 passed in 0.62s # the skipped test runs when a Serena source tree is available
$ ruff check headroom/ tests/ --exclude headroom/dashboard/templates
All checks passed!
$ mypy headroom/cli/wrap.py
Success: no issues found in 1 source file
```
New tests. The key one asserts the invariant rather than our own key
list, so it stays correct even if Serena adds a required key — a test
pinning `projects: []` would keep passing while users broke again:
- `test_serena_config_is_never_created_by_headroom` — Headroom must not
pre-empt Serena's bootstrap
- `test_serena_dashboard_disabled_repairs_config_missing_projects` —
heals a config an affected version wrote
- `test_serena_dashboard_disabled_preserves_registered_projects` — never
clobbers the real registry; comments kept, no duplicate key
- `test_serena_dashboard_disabled_is_idempotent`
- `test_serena_config_required_keys_match_serena_source` — reads
Serena's real source and pins the two facts this fix rests on
(bootstrap-only-when-absent, `projects` is the sole fatal omission).
Skipped unless `SERENA_SRC` is set; deliberately **not** named
`HEADROOM_*` because `conftest.py` scrubs that namespace, which would
make it silently always-skip.
## Real Behavior Proof
- **Environment:** macOS 15.4 (darwin 25.4.0), Python 3.12.6, Serena
1.6.2.dev0 via `uvx --from git+https://github.com/oraios/serena`, Codex
CLI 0.146.0.
- **Exact command / steps:** a probe doing a real JSON-RPC `initialize`
handshake against the exact command `headroom wrap` registers — i.e.
what Codex/OpenCode actually do — in a throwaway `HOME` per case. (A)
pre-seeded with the one-line config an affected version wrote; (B) no
config; (C) real `headroom wrap codex --prepare-only`, then handshake.
- **Observed result:**
```text
=== A. BROKEN: single-key config (Headroom 0.33.0) ===
MCP handshake: FAIL — no initialize response (exit=1). stderr tail:
File ".../serena/config/serena_config.py", line 1064, in from_config_file
raise SerenaConfigError("`projects` key not found in Serena configuration. ...")
serena.config.serena_config.SerenaConfigError: `projects` key not found ...
config after run: 1 lines, has 'projects': False
=== B. FIXED: no config, Serena bootstraps it ===
MCP handshake: PASS — initialize OK — serverInfo.name='Serena'
config after run: 213 lines, has 'projects': True
=== C. FULL FLOW: real `headroom wrap codex` then handshake ===
Serena: no serena_config.yml yet — letting Serena generate it
Serena MCP: registered (restart OpenAI Codex CLI if it was already running)
Serena: project pre-indexed (symbol cache warmed)
serena_config.yml: 213 lines, written by Serena (correct)
MCP handshake: PASS — initialize OK — serverInfo.name='Serena'
--- verdict ---
A (broken config) started: False <- expected False
B (fixed, no config) started: True <- expected True
C (after real wrap) started: True <- expected True
```
A second `wrap` in the same HOME flips the dashboard without damage:
`true` → `false`, `projects` intact, all 153 comment lines intact. The
writer was isolated against a pristine 213-line Serena config: **delta 0
newlines**.
- **Not tested:** Windows and Linux (macOS only); Serena versions other
than 1.6.2.dev0; the JetBrains language backend. The probe needs network
+ `uvx` (~2 min) so it is a manual verification tool, not wired into CI.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- **Docs:** N/A — no user-facing docs described the `serena_config.yml`
bootstrap or the language scoping.
- Also fixes the OpenCode report (#2674). The Codex-side report of the
same root cause quotes the `SerenaConfigError` verbatim; OpenCode only
surfaces the generic `-32000`, which is why it read as two different
bugs.
- Users already broken by an affected version are repaired automatically
on their next `headroom wrap` — no manual `serena_config.yml` edit
needed.
- A stacked PR removing the rtk/lean-ctx CLI context tools is based on
this branch; this one is deliberately small so it can land first.
|
||
|
|
a986d878b1 |
test: make copilot-flag fixture tolerate headroom not installed (#2407)
## Description
The autouse `_reset_copilot_routing_flag` fixture in `tests/conftest.py`
did an unconditional `from headroom.copilot_auth import
reset_request_routed_to_copilot` for **every** test. That import pulls
in the whole package (`headroom/__init__` → `compress.py` →
`observability` → `opentelemetry`).
The `macos-native-wrapper` and `windows-native-wrapper` CI jobs run
`tests/test_install/test_native_installers.py` with **only `pytest`
installed** (see `.github/workflows/ci.yml` — those jobs `pip install
pytest` and nothing else). Those tests drive the installer shell scripts
via `subprocess` and never import headroom, so the autouse fixture
errored at setup:
```
tests/conftest.py:40: in _reset_copilot_routing_flag
from headroom.copilot_auth import reset_request_routed_to_copilot
headroom/__init__.py:86: from .compress import ...
headroom/compress.py:65: from .observability import get_otel_metrics
headroom/observability/metrics.py:11: from opentelemetry import metrics
E ModuleNotFoundError: No module named 'opentelemetry'
```
Guard the import: when headroom isn't importable there is no routing
flag to reset, so the fixture is a no-op. No production code changes;
behavior is unchanged whenever headroom is installed (all other jobs).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `tests/conftest.py`: wrap the `_reset_copilot_routing_flag` fixture's
`headroom.copilot_auth` import in `try/except ModuleNotFoundError` →
yield-and-return when headroom is absent.
## Testing
- [x] Ran the exact CI command locally
- [x] Linting passes (`ruff check`)
### Test Output
```text
$ ruff check tests/conftest.py
All checks passed!
$ pytest tests/test_install/test_native_installers.py -q
collected 2 items
tests/test_install/test_native_installers.py ss [100%]
============================== 2 skipped in 0.11s ==============================
```
(2 skipped = Docker not available on the local box; the point is **no
more "ERROR at setup"**. Before this change the same run reported `1
error in 0.11s` with the `opentelemetry` traceback above.)
## Real Behavior Proof
- Environment: macOS, Python 3.12, headroom installed (normal path
exercised).
- Exact command / steps: `pytest
tests/test_install/test_native_installers.py -q`
- Observed result: no setup error; fixture takes the normal
(headroom-present) path — 2 tests skipped for lack of Docker.
- Not tested: the headroom-absent branch can't be reproduced locally
(headroom is installed here); it is exactly the CI job's environment,
which this PR's CI run will exercise.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Additional Notes
Scope is intentionally the native-wrapper failures only. The separate
`test-dashboard-ui` red X is unrelated (a stale UI-text assertion:
`element(s) not found — "Completed 128 Failed 0 Rate Limited 0 Cached
96"`) and is not addressed here. Checklist items about
docs/CHANGELOG/new-tests are N/A — this is a test-harness resilience
fix, not a behavior change.
|
||
|
|
d7a8cdbee1 |
feat(proxy): label GitHub Copilot traffic as "copilot" in the outcome… (#2377)
## Description Requests routed to the GitHub Copilot API travel on the OpenAI or Anthropic wire, so the proxy handlers stamp the *wire* provider (`openai` / `anthropic`) on the outcome. As a result, Copilot traffic is attributed to OpenAI/Claude in the dashboard's per-request provider stats, hiding the real upstream. (This is distinct from the existing **Copilot Quota** panel, which is separate from per-request provider attribution.) This labels Copilot traffic as `copilot` in the single outcome funnel. `build_copilot_upstream_url()` is already the one routing chokepoint every Copilot surface goes through (OpenAI `/chat/completions` + `/responses` and the Anthropic `/v1/messages` route all build their upstream URL there), so it flags the request via a task-local `ContextVar`; `emit_request_outcome()` reads the flag and relabels the provider. The relabel runs before the `>= 500` failed guard, so a failed Copilot request is attributed to `copilot` too. Non-Copilot traffic never sets the flag and is untouched. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `headroom/copilot_auth.py`: add a task-local `_request_routed_to_copilot` `ContextVar` with `mark_request_routed_to_copilot()` / `request_routed_to_copilot()` helpers; set the flag in `build_copilot_upstream_url()` whenever the base is a Copilot API URL (the existing `is_copilot_api_url` check). `/v1` path normalization is unchanged. - `headroom/proxy/outcome.py`: in `emit_request_outcome()`, when the request was routed to Copilot and the wire provider is `openai`/`anthropic`, relabel the outcome provider to `copilot` (before the 5xx guard). - `tests/test_copilot_provider_label.py`: new tests for the chokepoint marking and the outcome relabel. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) — ran on the changed files only (clean) - [ ] Type checking passes (`mypy headroom`) — ran on the changed files only (clean) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_copilot_provider_label.py tests/test_outcome_records_5xx_as_failed.py -q tests/test_copilot_provider_label.py ..... [ 71%] tests/test_outcome_records_5xx_as_failed.py .. [100%] 7 passed $ python -m pytest tests/test_copilot_auth.py -k "build_copilot_upstream_url or copilot_api_url" -q 8 passed, 58 deselected # existing /v1-stripping behavior preserved $ python -m ruff check headroom/copilot_auth.py headroom/proxy/outcome.py tests/test_copilot_provider_label.py All checks passed! $ python -m mypy headroom/copilot_auth.py headroom/proxy/outcome.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: Python 3.11, headroom installed with the `proxy` extra. - Exact command / steps: the unit tests above drive `build_copilot_upstream_url()` followed by `emit_request_outcome()` in an isolated context and assert the recorded provider. - Observed result: an `anthropic`/`openai` outcome for a request routed to `https://api.githubcopilot.com` is recorded as provider `copilot`; a request not routed to Copilot is recorded under its wire provider unchanged. - Not tested: end-to-end against a live Copilot subscription (no live seat in the test environment). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes - The flag is a `ContextVar` (task-local), so it cannot bleed across concurrent requests; each request that is not routed to Copilot simply reads the `False` default. - No `CHANGELOG.md` edits (release-please generates it from the Conventional Commit PR title). --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
718c8dc559 |
fix(proxy): repair main lint (ruff-format drift + mypy host_header) (#2268)
## Description
`main`'s `lint` CI job is currently **red** (latest main `eac49656` →
`lint: failure`), which blocks every open PR. Two causes, both from
recent merges that were green in isolation but combined into a red
`main`:
- **ruff-format drift** on 7 files — committed with formatter output
that ruff `0.15.17` (the CI-pinned version) rewrites.
- **mypy error** in `server.py`:
`_request_has_same_origin_or_no_provenance(request, host_header)` —
`host_header` is `request.headers.get("host")` (`str | None`) but the
function requires `str`.
These passed per-PR because each PR's checks ran against an older base;
the serialized `main` state is what went red — a logical-merge /
tool-version gap that per-PR CI doesn't catch without a strict merge
queue.
## Type of Change
- [x] Bug fix (CI/lint repair)
## Changes Made
- `ruff format` (0.15.17) the 7 drifted files — formatting only, no
logic changes: `cli/proxy.py`, `proxy/forwarded_headers.py`,
`proxy/savings_tracker.py`, `proxy/server.py`, `tests/conftest.py`,
`tests/test_persistent_metrics_persistence.py`,
`tests/test_proxy_loopback_gating.py`.
- Add `assert host_header is not None` after the
`is_ip_literal_host_header()` guard (which already rejects a missing
Host), narrowing the type for the same-origin check.
## Testing
- [x] `ruff check .` — clean
- [x] `ruff format --check .` — clean (tracked)
- [x] `mypy headroom --ignore-missing-imports` — clean
### Test Output
```text
$ mypy headroom --ignore-missing-imports → Success: no issues found in 504 source files
$ ruff check . → All checks passed (tracked)
$ ruff format --check . → clean (tracked)
```
## Real Behavior Proof
- Environment: branch off current `main` (`eac49656`), ruff 0.15.17 +
mypy 1.20.2 (CI-pinned).
- Confirmed `lint: failure` on main's latest CI run; after this change
all three lint steps pass locally.
- Not tested: full pytest suite — formatting + a type-narrowing `assert`
only, no behavior change.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines (this *is* the
style fix)
- [x] I have performed a self-review of my code
- [x] My changes generate no new warnings
- [ ] Tests added (N/A — no behavior change)
- [x] New and existing unit tests pass locally
- [ ] CHANGELOG (N/A)
## Additional Notes
The 7 files were touched by recent merges (#2198, #2247) whose local
ruff differed from the pinned `0.15.17`. Merging this unblocks the
`lint` gate for all open PRs (including #2207).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
|
e5b3a634df |
fix(proxy): dedupe Codex WS request logging for accurate mixed-provider dashboards (#2189)
## Description Running Claude Code (Anthropic) and Codex (OpenAI) against the **same** Headroom proxy instance on one port produced incorrect, unstable dashboard data. The proxy core is provider-isolated and multi-provider-safe by design; the defect was in the observability layer. The Codex `/v1/responses` **WebSocket** handler was the only path in the proxy that wrote to the request logger by hand instead of through the unified `emit_request_outcome` funnel, and it did so twice per session close: the per-turn funnel record plus an unconditional cumulative session-summary `RequestLog`. This PR removes the duplicate summary log so Codex WS emits exactly one request log per turn, matching the HTTP provider paths. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Dropped the duplicate cumulative session-summary `RequestLog` in the Codex WS handler while preserving the per-turn `emit_request_outcome` path. - Preserved gated `request_messages` and `turn_id` on residual outcomes so dashboard telemetry keeps the useful attribution without double-counting tokens. - Ensured explicit `--anyllm-provider` wins over a leaked `HEADROOM_ANYLLM_PROVIDER` environment variable. - Registered retry delay settings that had drifted out of the settings registry. - Hardened tests against developer-shell `HEADROOM_*` / `ANTHROPIC_CUSTOM_HEADERS` leakage and stabilized several focused proxy/wrap test fixtures. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest tests/ -q -p no:cacheprovider 8529 passed, 537 skipped, 5831 warnings in 276.25s (0:04:36) $ .venv/bin/ruff check <touched files> All checks passed! ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.4.0), Python 3.13.14, pytest 9.0.3, ruff via project venv, branch `fix/multi-provider-runtime`. - Exact command / steps: Ran the full test suite without pytest cache provider and Ruff on all touched files; used `git stash` to confirm the stale fake-config failures pre-existed this change. - Observed result: Full suite passed with no failures; Ruff passed; Codex WS now routes end-of-session logging through `emit_request_outcome`, emitting one request log per turn with the same accounting model as Anthropic HTTP turns. - Not tested: Live simultaneous Claude + Codex dashboard run. `mypy headroom` was not run to completion; a scoped run reported one pre-existing `settings_store.py:470` coercion error outside this diff. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A - server-side observability fix; no UI markup changed. ## Additional Notes - The proxy's multi-provider routing, header/auth isolation, and per-model cache keying are already correct and unchanged here; only the WS observability write path was double-counting. - Architectural assessment: `plans/reports/research-260714-0004-multi-provider-upstream-compression-report.md`; root-cause + resolution trail: `plans/reports/debug-assessment-260714-0011-dashboard-instability-mixed-claude-codex-report.md`. - No live simultaneous Claude + Codex dashboard run was performed; validation is from test coverage and code review of the WS logging path. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
fda4670ef8 |
fix(tests): reset whole headroom logger subtree so caplog stays deterministic (#1117)
## Description
Five `caplog`-based test assertions are order-dependent flakes: they
pass in isolation but fail in full-suite runs.
**Root cause** is a global logging-state leak.
`benchmarks.claude_session_mode_benchmark._disable_headroom_benchmark_logging()`
(exercised by `tests/test_claude_session_mode_benchmark.py`) sets
`propagate = False` + `CRITICAL` on the `headroom`, `headroom.proxy`,
`headroom.transforms` (and `headroom.cache`) loggers and never restores
them. pytest's `caplog` attaches its handler to the **root** logger, so
once any `headroom.*` child is left non-propagating, records from that
subtree silently never reach `caplog` for **every test that runs
afterwards** — which is exactly why these only fail in full-suite order.
The repo already ships a `_reset_headroom_logger_propagation` autouse
fixture for this hazard (its docstring documents the equivalent
`_setup_file_logging` leak), but it only reset the **top** `headroom`
logger, not children like `headroom.proxy`. A non-propagating child
still blocks the record before it reaches root. This PR extends the
existing fixture to reset the whole `headroom.*` subtree before each
test.
Scope is intentionally one file (`tests/conftest.py`) — test-harness
only, no production change.
> Design note: I extended the existing defensive fixture rather than
restoring state inside the benchmark, because (a) the fixture already
exists for exactly this and only needed completing, and (b) the same
`propagate=False` hazard also originates from production
`_setup_file_logging`, so a centralized per-test reset is the more
durable fix. Happy to instead make the benchmark restore its own logging
state if maintainers prefer fixing it at the source.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Extend the `_reset_headroom_logger_propagation` autouse fixture in
`tests/conftest.py` to reset `propagate = True` for **every** existing
`headroom.*` logger (previously only the top `headroom` logger), so
`caplog` capture is deterministic regardless of test execution order.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — N/A, change is under
`tests/`
- [ ] New tests added — N/A, this fixes existing tests; they are
themselves the proof
- [x] Manual testing performed
### Test Output
```text
# Causal proof: run the polluter first, then the 5 victims, in one process.
# BEFORE (fixture reset scoped to only "headroom"):
$ pytest tests/test_claude_session_mode_benchmark.py \
tests/test_corrupt_golden_bytes_recovery.py \
tests/test_forwarded_headers.py \
'tests/test_transforms/test_kompress_compressor.py::TestKompressBackendSelection::test_unrecognized_backend_warns_and_falls_back_to_auto'
5 failed, 54 passed
FAILED tests/test_corrupt_golden_bytes_recovery.py::TestCorruptMemoryGoldenBytes::test_corrupt_bytes_logs_error
FAILED tests/test_corrupt_golden_bytes_recovery.py::TestCorruptMemoryGoldenBytes::test_unicode_decode_error_handled
FAILED tests/test_corrupt_golden_bytes_recovery.py::TestCorruptCcrGoldenBytes::test_corrupt_bytes_logs_error
FAILED tests/test_forwarded_headers.py::test_non_allowlisted_peer_ignores_forwarded_and_logs
FAILED tests/test_transforms/test_kompress_compressor.py::...test_unrecognized_backend_warns_and_falls_back_to_auto
# AFTER (this PR — whole headroom.* subtree reset):
$ pytest <same selection>
59 passed
# Full suite (Rust core rebuilt locally):
$ pytest
6251 passed, 496 skipped
$ ruff check .
All checks passed!
$ ruff format --check tests/conftest.py
1 file already formatted
```
## Real Behavior Proof
- Environment: macOS, Python 3.13.3, branch off latest `main`, Rust
`_core` rebuilt locally (`uv pip install -e .`).
- Exact command / steps: ran the polluter
(`test_claude_session_mode_benchmark`) together with the 5 victim tests
in one process to reproduce the order-dependent failure, then toggled
**only** the fixture change to confirm causality; then ran the full
`pytest` suite and `ruff`.
- Observed result: scoping the reset to only `"headroom"` → 5 failed /
54 passed; extending it to the `headroom.*` subtree → 59 passed. Full
suite: 6251 passed, 496 skipped, 0 failed. Lint clean.
- Not tested: behavior under CI's sharded `test (N)` jobs specifically —
but the fix is order-independent (resets before *every* test), so
sharding cannot reintroduce the leak.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective — the 5
previously-flaky tests are the proof
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A (test-harness only)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
2cae13dd79 |
fix(ccr): skip Anthropic marker emission when tool injection is deferred (#1273)
## Description Anthropic request-side CCR can still compress a turn into retrieval markers after the frozen-prefix cache guard suppresses `headroom_retrieve` registration. That leaves the model with marker-only context it cannot redeem, so the proxy silently drops recoverable data on exactly the turns where cache preservation deferred tool injection. This change couples the Anthropic request-side CCR path to tool availability so a turn never emits retrieve-only markers without the retrieval tool, even when token mode or cache-mode prefix replay could otherwise reuse already-compressed marker text. Closes #1006 After a collaborator merged current `main` into this branch, CI also picked up unrelated offline-memory failures from the merged base. Those follow-up changes are test-only: they keep the offline Hugging Face cache lanes skipping cleanly instead of failing in memory tests that are outside the CCR runtime path. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Couple Anthropic request-side CCR compression to the same frozen-prefix guard that already defers `headroom_retrieve` registration. - Keep the existing cache-preservation behavior: frozen-prefix turns stop emitting CCR retrieval markers instead of forcing tool injection into the cached prefix. - Make the skip decision use the effective frozen prefix after token-mode reclamping, so turns that genuinely reclamp to zero still keep normal reversible CCR behavior. - Bypass cached marker reuse in both token mode and cache-mode prefix replay when tool injection is deferred. - Add focused regressions for the Anthropic request-path seams under this bug: - frozen-prefix turns do not emit marker-only payloads - unfrozen turns still keep normal reversible CCR behavior - token-mode reclamp back to zero still compresses normally - existing `headroom_retrieve` tools keep reversible CCR on frozen turns - cache-mode delta reuse and exact-prefix replay both forward original content when retrieval is unavailable - Add a `CHANGELOG.md` entry because the proxy's user-visible Anthropic CCR behavior changes. - Add a shared test skip helper for offline Hugging Face cache misses and apply it to the merged `main` memory tests that were failing only in the offline CI shards after the branch picked up current `main`. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py`) - [x] Linting passes (`uv run ruff check . && uv run ruff format . --check`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py 14 passed, 1 warning in 34.19s uv run pytest tests/test_memory/test_skip_helpers.py tests/test_memory/test_embedder_mps_serialization.py::test_cpu_uses_shared_executor tests/test_memory/test_hierarchical.py::TestLocalEmbedder::test_embed_single tests/test_memory_bridge.py::TestMemoryBridgeImport::test_import_claude_code_memory tests/test_memory_handler_concurrent_init.py::test_real_localbackend_initializes_via_public_entrypoint tests/test_memory_system.py::TestLocalBackend::test_save_memory_basic -q 4 passed, 5 skipped, 11 warnings in 7.65s uv run ruff check . All checks passed! uv run ruff format . --check 968 files already formatted ``` ## Real Behavior Proof - Environment: local FastAPI `TestClient` for the Anthropic request path, plus Windows Python 3.12 offline-memory repros with `TRANSFORMERS_OFFLINE=1` - Exact command / steps: Run the focused CCR regression command and the offline-memory repro subset below on the merged branch state. - `uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py` - `uv run pytest tests/test_memory/test_skip_helpers.py tests/test_memory/test_embedder_mps_serialization.py::test_cpu_uses_shared_executor tests/test_memory/test_hierarchical.py::TestLocalEmbedder::test_embed_single tests/test_memory_bridge.py::TestMemoryBridgeImport::test_import_claude_code_memory tests/test_memory_handler_concurrent_init.py::test_real_localbackend_initializes_via_public_entrypoint tests/test_memory_system.py::TestLocalBackend::test_save_memory_basic -q` - Scenario coverage from the CCR pytest command: - frozen prefix with deferred tool injection and cached marker text available in token mode - unfrozen turn with normal CCR marker emission - token mode where the tracked frozen prefix reclamps back to zero - frozen prefix where the client already supplied `headroom_retrieve` - cache-mode append-only delta reuse with a previously forwarded compressed prefix - cache-mode exact-prefix replay where the previous forwarded prefix already contained a marker - Scenario coverage from the offline-memory repro command: - direct local embedder startup on CPU with no cached HF model - hierarchical memory embedder startup with offline model cache missing - bridge import through `LocalBackend` - `MemoryHandler` public init warmup path - `LocalBackend` save path under the offline lane - Observed result: The CCR regression keeps marker-free forwarding on frozen turns without tool availability, and the merged-`main` offline-memory lanes now skip cleanly instead of failing unrelated CI shards. - frozen-prefix Anthropic turns without tool availability forward the original long transcript across both token-mode and cache-mode reuse paths, while unfrozen turns, reclamped token-mode turns, and frozen turns that already advertise `headroom_retrieve` keep the reversible CCR marker path - the merged-`main` offline-memory regressions now skip cleanly when the Hugging Face cache is unavailable instead of failing unrelated CI shards - Not tested: live Zed session cache-hit behavior, provider latency under real Anthropic upstreams, and online Hugging Face download lanes ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Runtime scope is still intentionally narrow to Anthropic request-side CCR. OpenAI, Gemini, streaming, and response-side CCR behavior are unchanged. - The only non-CCR diff is the test-only offline-memory follow-up required after current `main` was merged into the branch. - The focused proxy pytest run still emits the Windows-local `StarletteDeprecationWarning` from `fastapi.testclient`'s `httpx` bridge. The offline-memory repro command also emits existing datetime deprecation warnings and a pytest teardown warning around skipped offline lanes; none of those warnings were introduced by the CCR runtime change. |
||
|
|
b4571cc346 |
feat: headroom wrap opencode / unwrap opencode CLI (#1105)
## Summary This PR implements transparent `headroom wrap opencode` support without asking users to edit OpenCode provider URLs, choose an extra CLI flag, or maintain a static provider list. The wrapper now lives at the runtime transport boundary: OpenCode keeps its user/provider config, while Headroom intercepts outbound provider traffic in-process and routes it through the local Headroom proxy. ## What changed ### Transparent OpenCode wrapping - `headroom wrap opencode` injects the `headroom-opencode` plugin through `OPENCODE_CONFIG_CONTENT`. - Existing OpenCode provider URLs are preserved. We do not rewrite user config URLs to point at Headroom. - Existing `OPENAI_BASE_URL` and `ANTHROPIC_BASE_URL` env vars are preserved. - Local OpenCode traffic, localhost traffic, and Headroom proxy traffic bypass the shim to avoid loops. ### Runtime transport interception - Added an OpenCode plugin transport shim that wraps: - `globalThis.fetch` - `http.request` / `http.get` - `https.request` / `https.get` - External provider calls are routed to the local Headroom proxy. - The original upstream origin is passed through `x-headroom-base-url`, so the proxy can forward to the real provider without changing OpenCode config. - External `http2.connect` is blocked loudly instead of allowing direct provider traffic to leak outside Headroom. ### Live provider additions Provider coverage is no longer based on a static config scan. Because routing happens at outbound request time, providers added mid-session are routed through Headroom automatically as long as they use the covered Node transport paths. ### Subagent and child-process coverage - The parent OpenCode plugin sets a packaged Node preload shim through `NODE_OPTIONS=--import=.../hook-shim/handler.js`. - The transport shim patches `child_process.spawn`, `exec`, `execFile`, and `fork` so child Node processes receive the Headroom preload even when OpenCode passes a custom `env`. - The child-process shim fails closed if it loads without `HEADROOM_OPENCODE_TRANSPORT_PROXY_URL`. - This closes the subagent leak path where a child Node process could otherwise start without Headroom transport interception. ## Why this goes beyond PR #1089 PR #1089 improves OpenCode provider registration, but it still focuses on provider config shape. This PR moves the enforcement boundary to runtime transport interception. This PR goes further because: - No provider URL rewriting is required. - New providers added mid-session are covered automatically. - Subagents and child Node processes inherit the Headroom transport shim. - Direct external HTTP/2 paths fail loudly instead of leaking. - The wrap remains transparent to the user's OpenCode provider config. - The wrapper is fail-closed for unsupported child-process preload state. ## Additional robustness fixes While validating the change in Docker, the full Python suite exposed unrelated Linux/container robustness issues. These are fixed in this PR so the suite is green: - Binary cache handling now treats cache paths under a non-writable existing parent as unavailable, including when tests run as root in Docker. - `release_version.py` honors `MANUAL_VER` before git calls so direct script execution works outside a `.git` checkout. - Test logger isolation now resets relevant Headroom child loggers so proxy logging setup cannot poison later `caplog` tests. - The scanner missing-path test now uses a guaranteed missing `tmp_path` child instead of relying on `/nonexistent/path`. ## Validation All implementation validation was run inside Docker. - Full Python suite from a fresh Docker copy: `6605 passed, 523 skipped`. - Ruff on changed Python/OpenCode paths: passed. - OpenCode plugin typecheck: passed. - OpenCode plugin tests: `9 passed`. - OpenCode plugin build: passed. - Hook shim preload smoke test: passed. ## Notes This PR intentionally does not add a CLI option. `headroom wrap opencode` means full wrap. Either Headroom wraps OpenCode transparently, or the path fails loudly instead of silently leaking provider traffic. --------- Co-authored-by: Rudimar Ronsoni <6081613+rudironsoni@users.noreply.github.com> |
||
|
|
967b0db439 |
fix: B1 — retire ICM, RollingWindow, scoring, relevance + dependents
Phase B step 1 of the live-zone-only realignment. Removes ~10K LOC of
"drop messages from history" machinery that became unreachable after
PR-A1 made `/v1/messages` a passthrough on the proxy. Live-zone-only
compression (PR-B2..B7) operates on content blocks within messages;
message-list mutation no longer happens in the pipeline.
Python deletes:
- headroom/transforms/intelligent_context.py (1077 LOC)
- headroom/transforms/rolling_window.py (395 LOC)
- headroom/transforms/progressive_summarizer.py (508 LOC)
- headroom/transforms/scoring.py (459 LOC)
- headroom/transforms/tool_crusher.py (338 LOC)
- 5 corresponding tests/test_transforms/* and tests/test_proxy_intelligent_context.py
Rust deletes:
- crates/headroom-core/src/context/* (manager, config, workspace,
candidate, ccr_drop, strategy/, mod) + safety.rs replaced
- crates/headroom-core/src/scoring/* (mod, score, scorer, traits, weights)
- MessageScorerComparator from crates/headroom-parity (PR #338/#343
becomes deletable; sunk cost stays sunk)
- 13 message_scorer fixtures + record_message_scorer.py
Rust adds (move + rewrite):
- crates/headroom-core/src/transforms/safety.rs — `tool_pair_indices`
preserves the OpenAI/Anthropic tool_use ↔ tool_result pairing rule
the live-zone dispatcher (PR-B2) needs. No IcmConfig dependency.
Surface refactors:
- HeadroomConfig: drop `tool_crusher`, `rolling_window`,
`intelligent_context` fields; hoist `output_buffer_tokens` to top
level (used by client.py).
- ProxyConfig: drop `intelligent_context*` fields.
- `headroom wrap` proxy server: retire IntelligentContextManager
and RollingWindow imports + branch; pipeline is CacheAligner →
ContentRouter (smart_routing) or CacheAligner → SmartCrusher
(legacy).
- CLI: drop `--no-intelligent-context`, `--no-intelligent-scoring`,
`--no-compress-first` flags.
- LangChain memory integration: rename `_apply_rolling_window` →
`_apply_compression`, drop RollingWindowConfig dep. Threshold is
now advisory — B6 will rework the contract.
- TransformPipeline.create_pipeline now takes only cache_aligner_config.
- headroom/__init__.py + headroom/transforms/__init__.py: strip
exports of deleted symbols.
Bug fixes uncovered by full pytest sweep:
- providers/copilot/wrap.py: `environ or os.environ` collapsed
empty-dict to falsy → callers passing `environ={}` accidentally
pulled from os.environ. Use `environ if environ is not None else
os.environ`.
Test correctness fixes:
- _DummyAnthropicHandler._retry_request gains **_kwargs to match
the real handler signature post-A8.
- test_ws_http_fallback extracts JSON from `content=` (post-A3
byte-faithful) rather than the obsolete `json=` kwarg.
- test_ccr_response_handler_extra fixture joins SSE events with
`\n\n` per spec (post-A8 byte-buffer parser requirement).
- test_proxy_responses_phase_preservation: capture via direct
handler attached to the named logger, so the assertion is
order-independent (proxy `_setup_file_logging` flips
`headroom.propagate=False` once any earlier test triggers it).
- conftest.py autouse fixture resets `headroom.propagate=True`
before each test as a defensive measure for the same pollution.
- test_wrap_copilot_translated_backend_still_requires_byok:
monkeypatch.delenv every provider key so the BYOK error
actually fires.
- test_native_installers: skip when system bash < 4.3 (macOS ships 3.2).
- TestGeminiEmbedContent / TestGeminiBatchEmbedContents:
pytest.mark.skip — proxy currently has no :embedContent route;
feature gap, not regression.
Acceptance:
- cargo build --workspace + cargo clippy + cargo fmt --check: green.
- cargo test --workspace --exclude headroom-py: 777 passed.
- pytest: 4892 passed, 240 skipped, 0 failed.
- git grep returns only intentional comments referencing the deletion.
Per-PR-B1 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
|
fca68f0cbe |
Add global httpx.ReadTimeout handler for all tests
Move the network timeout skip handler to the main tests/conftest.py so it applies to all tests, not just tests/test_memory/*. Fixes flaky CI failures when HuggingFace model downloads timeout. |
||
|
|
c850ccc3b2 |
Replace legacy memory system with HierarchicalMemory
Major refactor of the memory module: - Add hierarchical scoping (user → session → agent → turn) - Add temporal versioning with supersession support - Add pluggable adapters (SQLite store, HNSW vectors, FTS5 text search) - Add protocol interfaces (ports) for all memory components - Update LRUMemoryCache to implement async MemoryCache protocol - Update wrapper.py to use HierarchicalMemory backend - Preserve with_memory() one-liner API with zero-latency inline extraction New files: - adapters/: sqlite.py, hnsw.py, fts5.py, cache.py, embedders.py - core.py: HierarchicalMemory orchestrator - models.py: Memory, MemoryCategory, ScopeLevel - ports.py: Protocol interfaces (MemoryStore, VectorIndex, etc.) - config.py: MemoryConfig with backend selection - factory.py: Component creation from config Removed legacy files: - store.py, fast_store.py, extractor.py, worker.py, fast_wrapper.py Breaking change: Removes legacy memory API (pre-0.3.0) |
||
|
|
e4a41faa33 |
Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file - Fix F401: Add noqa for availability check imports - Fix F402: Rename loop variables shadowing imports - Fix E722: Replace bare except with except Exception - Fix B904: Add exception chaining (from e) - Fix F811: Remove duplicate imports - Fix B027: Add noqa for empty close() method - Fix E741: Rename ambiguous variable l -> label - Fix I001: Import sorting issues - Apply ruff format to all 106 files All 902 tests pass. |
||
|
|
9c7d4512d6 |
Initial commit: Headroom SDK - LLM context optimization toolkit
A comprehensive SDK for optimizing LLM context windows, reducing token usage while preserving critical information for AI agents. Core Features: - SmartCrusher: Statistical compression of tool outputs (70-85% reduction) - CacheAligner: Prefix optimization for prompt cache hits - RollingWindow: Intelligent context window management - BM25/Hybrid relevance scoring for smart item selection Integrations: - OpenAI and Anthropic provider support - LangChain integration (ChatModel, Callbacks, Runnable) - MCP (Model Context Protocol) integration for tool compression Test Coverage: - 372 tests passing across all modules - 35 performance benchmarks - Real-world agent evaluations with 88% token savings Key Components: - headroom/transforms/: Core compression transforms - headroom/providers/: OpenAI and Anthropic support - headroom/integrations/: LangChain and MCP integrations - headroom/relevance/: BM25 and hybrid scoring - headroom/pricing/: Model pricing registry - benchmarks/: Performance benchmark suite - examples/: Usage examples and demos |