mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
2e4ff6f9c7 |
fix(proxy): keep upstream credentials out of manifests, env and TOIN (#4866)
## Description
Proxy secrets (upstream API keys, proxy token, licence) were stored or
passed in clear:
- Bench manifests and `ProxyDeploymentPlan.to_dict()` serialized the
full `ProxyConfig`, keys included.
- With `--workers N`, the whole config, keys included, went to each
worker in the `HEADROOM_PROXY_CONFIG_JSON` env var, which other
processes of the same user can read (`ps eww`, `/proc/<pid>/environ`).
- TOIN stored query field names in clear, although its module docstring
says field names are hashed; the text before a `:` can be a credential
(`<token>:x-oauth-basic`).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/models.py`: `PROXY_CONFIG_SECRET_FIELDS` (one list)
and `redact_proxy_config_secrets`.
- `headroom/testing/harness.py`, `ProxyDeploymentPlan.to_dict()`:
serialize the redacted config; the in-memory plan keeps real values.
- `headroom/proxy/server.py` (`--workers N`): secrets go to a 0600 temp
file named by `HEADROOM_PROXY_CONFIG_SECRETS_FILE`; workers read it, the
env JSON carries the config without them, and the file is removed when
the server stops. A worker that cannot read the file raises
`ConfigurationError` instead of falling back to env defaults (which
could drop the proxy token).
- `headroom/telemetry/toin.py`: query patterns hash field names; stored
patterns with raw field names are dropped on load and the store is
rewritten at once (a private `_import_patterns` reports whether anything
was dropped).
- Tests: `tests/test_proxy_config_secrets_at_rest.py` (manifest,
`to_dict`, worker env carry no secret; workers still get the real
values); `tests/test_toin_retention.py` expectations updated to hashed
field names.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest -q tests/test_proxy_config_secrets_at_rest.py tests/test_toin_retention.py tests/test_stateless_toin.py tests/test_toin_*.py ...
125 passed
# new tests against origin/main
tests/test_proxy_config_secrets_at_rest.py: 3 failed (+1 for the fail-closed worker, which fails on the first commit too)
test_legacy_raw_query_keys_are_removed_on_load: fails without the load-time rewrite
# every test file importing the changed modules (73 files), this branch
2 failed, 1504 passed: tests/test_read_maturation_handler_nobust.py, both also failing on origin/main
(an earlier run on the first commit also hit tests/test_proxy_health.py failures that fail the same way on origin/main,
and one tests/test_proxy_ccr.py timing failure that main showed in a sibling test; that file passes 3/3 alone on both)
ruff check / format --check: clean; mypy: no new errors (same 8 pre-existing as origin/main)
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, the real `headroom proxy --workers 2
--stateless --no-telemetry` CLI, isolated `HOME`/`TMPDIR`,
`--openai-api-url` pointed at a local HTTP server that prints the
`x-gateway-auth` header it receives, `--openai-extra-headers
'{"x-gateway-auth":"FAKE-GATEWAY-0123"}'`.
- Exact command / steps: start the proxy, run `ps eww` on each worker
process, list `$TMPDIR`, POST `/v1/chat/completions`, stop the proxy
with SIGTERM.
- Observed result:
- origin/main: both worker processes' environments contain
`FAKE-GATEWAY-0123` (via `HEADROOM_PROXY_CONFIG_JSON`).
- this branch: neither worker's environment contains it; `$TMPDIR` holds
one `headroom-proxy-config-*.json` with mode `-rw-------`; the upstream
still receives `x-gateway-auth: FAKE-GATEWAY-0123` (workers load it from
the file); after SIGTERM the file is gone.
- On both, the parent's own command line holds the value because the
test passed it as a CLI flag; that is outside this change.
- Not tested: SIGKILL of the parent (the file stays in `$TMPDIR`).
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: manifests and `to_dict()` show
secrets redacted; TOIN query-pattern keys are hashed and old raw-name
keys are dropped once on load (learned query statistics for those keys
restart).
- Kill switch / disable path: none.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
ee731d7ea8 |
fix(proxy): refuse token-less non-loopback binds, gate operator routes (#3852)
## Why Four related gaps in the proxy's network gate (findings 01-F1, 01-F2, 01-F5, 01-F9): - **Open bind.** A non-loopback bind with no `HEADROOM_PROXY_TOKEN` only logged a warning, then served the `/v1/*` relay, with the operator's upstream credentials, to every peer that could reach the port. - **Missing peer address.** `is_loopback_host(None)` returned `True`. A Unix-domain socket, an ASGI adapter that leaves `scope["client"]` unset, or a hand-built request therefore turned off the token gate, the `/admin` and `/debug` guards and the WebSocket gate. - **Operator data routes.** `/stats-history` (including CSV export), `/quota`, `/subscription-window` and `/metrics` were served to any network caller. - **WebSocket loopback exemption** checked only the peer IP, not the `Host` header, so it had no DNS-rebinding defence. ## What changes - New `headroom/proxy/bind_policy.py`. A non-loopback bind (`0.0.0.0`, `::`, a LAN address or a hostname) with no token is refused at three places: `create_app` (embedders and the multi-worker factory), `run_server` (one exit with code 2 before uvicorn forks) and the `headroom proxy` CLI (a clear error before the banner, exit code 1). An absent host means uvicorn's default `127.0.0.1`. - `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` acknowledges an open bind when the runtime already restricts reachability, such as a container published on `127.0.0.1`. The `proxy_open_bind` warning is still logged on that path, and the banner says the bind is open. - `is_loopback_host(None)` now returns `False` (fail closed), matching `identity.py`. - `/stats-history`, `/quota` and `/subscription-window` answer only: a non-loopback caller that passed the token gate, a loopback caller (peer and `Host` header both loopback), a trusted dashboard client, or, under an acknowledged token-less bind, the container's own default-gateway TCP peer with a loopback `Host` (the host browser reaching a `127.0.0.1`-published container). Everyone else gets 404, as with `/admin`. Settings writes keep their stricter rule with no token short-cut. - `/metrics` answers a non-loopback caller that passed the token gate, a loopback caller, a connecting peer in `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` (whatever it forwards), or a resolved client in the dashboard CIDRs. For CIDR peers a hostname `Host` is fine, so Prometheus scrapers work; a cross-origin browser request is still refused. - The token short-cut applies only to non-loopback requests. The gate exempts loopback peers without checking `Host`, so a loopback request has not proven it holds the token and must pass the loopback `Host` check. This blocks a DNS-rebound browser on the operator's machine. - The `/dashboard` shell stays ungated. It is a static template, like the already public `/dashboard/static` assets. Gating it would break the dashboard for containers published on host loopback, where the peer is the bridge gateway and `Host` is `localhost`. The data routes behind it are gated. - The WebSocket loopback exemption requires a loopback peer and a loopback `Host` header. A loopback peer with a foreign `Host` can still connect with the token. - Launchers that always publish on `127.0.0.1` set the acknowledgement themselves: `headroom install` (docker preset), `scripts/install.sh` and `scripts/install.ps1` (through one helper that adds the acknowledgement only together with the `127.0.0.1` publication), and `docker/docker-compose.native.yml`. - Docs: `proxy.mdx` documents the new variable and which routes the token also gates; `docker-install` (docs and wiki) and the `docker-compose.yml` comment show the acknowledgement in the loopback-published example. `e2e/docker-bind-security.sh` sets it and checks that the same launch without it refuses to start. ## Behaviour change (breaking) - `headroom proxy --host 0.0.0.0`, or `HEADROOM_HOST` set to any non-loopback address, without `HEADROOM_PROXY_TOKEN` now refuses to start. The error names both remedies. Set a token, bind loopback, or set `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` when the port is already confined. Anyone running the image by hand with `--host 0.0.0.0` and no token must add one of these. - Without a token, `/stats-history`, `/quota`, `/subscription-window` and `/metrics` return 404 to non-loopback callers outside a trusted CIDR. Prometheus scrapers on another host must be in `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` or send the token. - Code that relied on a request with no peer address being treated as loopback now gets the non-loopback path. - The default loopback run is unchanged. ## Tests - `tests/test_proxy_bind_policy.py`: the refuse/allow matrix (non-loopback, loopback, unset host, token, acknowledgement values), the warning on the acknowledged path, `create_app`, the `run_server` exit before uvicorn, and the CLI (refusal, `HEADROOM_HOST` from env, acknowledged start, offline banner, start with a token). - `tests/test_proxy_loopback_gating.py`: each data route against non-loopback, loopback, trusted-CIDR and token callers; the dashboard shell stays reachable; the host of an acknowledged, loopback-published container reads the data routes while another bridge peer, a forwarded gateway address and a foreign `Host` do not; CSV export is not served to network callers; `/metrics` across loopback, both CIDR lists, out-of-range peers, cross-origin browsers and token callers, including a trusted gateway relaying a scraper outside its own range. A regression test shows that a token on the proxy does not exempt a loopback peer from the `Host` check; it fails without the fix. - `tests/test_proxy_hardening.py`: the HTTP gate with no peer address, WebSocket `Host` cases, `require_loopback` and `/debug` with no peer. - Installer tests: `headroom install` adds the acknowledgement, and the shell and PowerShell wrappers add it only together with the loopback publication. Launcher-level tests start the proxy with the environment and `--host` that `build_runtime_command()` and `docker-compose.native.yml` produce and check that the host browser (bridge-gateway peer) gets `/stats-history` and another container does not. - Tests that bound `0.0.0.0` for unrelated reasons now set a token or the acknowledgement. Tests that read operator routes use an explicit loopback peer. On `main`, 14 of the gating and hardening tests fail and the bind-policy module does not exist. Manual runs: token-less `--host 0.0.0.0` exits with the refusal from both the CLI and `run_server`; with the acknowledgement, the data routes are 404 to a LAN peer and to a loopback peer with a foreign `Host`; with a token, a LAN peer gets 401 without it and 200 with it; the default loopback bind starts as before. Not tested: a real Unix-domain-socket deployment and a real Prometheus scrape (both covered by tests only). ## Merge order Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 → #3863. This PR goes first: it is the gate the others assume, and it is the release-note event. ## Not in this PR - Enforcing the token for loopback callers when one is configured. Revised 2026-10-02 after review: `/metrics` matches gateway CIDRs against the TCP peer, and the operator data routes serve the host of a loopback-published container (bridge-gateway peer under the launchers' bind acknowledgement) so the default Docker dashboard works. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3852" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
3077ac81e8 |
feat: add deterministic runtime rollout controls (#1490)
## Description Establish one centrally resolved, observable, deterministic, versioned runtime rollout-control mechanism for Headroom. Runtime rollout controls which behaviors an already-built artifact may expose; it does not select or qualify a Headroom release/version. ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Bug fix (non-breaking change that fixes rollout enforcement regressions) - [x] Documentation update - [x] Code refactoring (no functional changes) ## Changes Made - Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`, `--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared by Python configuration boundaries. - Added schema/policy versions, canonical registry and snapshot SHA-256 identities, per-feature decision reasons, disable precedence, unsafe qualification poisoning, strict CLI validation, and fail-closed environment handling. - Added `headroom rollout status --json`, Python `/stats.rollout`, and Rust `/rollout/status` runtime provenance. - Added equivalent Rust snapshot semantics and shared Python/Rust policy vectors while retaining language-specific feature registries. - Enforced rollout policy at alternate Python server composition roots so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate. - Preserved typed rollout snapshots across multi-worker serialization with schema, policy, registry, snapshot-digest, type, and feature-name validation. - Made loopback runtime output-shaper updates replace the immutable snapshot atomically for request readers, retain explicit request/disable provenance, preserve channel and kill-switch precedence, invalidate cached stats, and return the effective rollout decision. - Made `headroom learn --verbosity --apply` report a channel-blocked update instead of claiming the shaper is live. - Made explicit CLI feature flags fail loudly when their current channel blocks them. - Made persistent interceptor installation select canary automatically, or reject an explicitly insufficient channel unless the break-glass override is set. - Updated architecture, proxy, rollout, learn, and output-shaper documentation with required channels and hot-reload semantics. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check .` and `ruff format --check .`) - [x] Type checking passes (`mypy headroom --ignore-missing-imports`) - [x] New regression tests added for every corrected behavior - [x] Rust tests and production-target Clippy pass - [x] Documentation build passes ### Test Output ```text Focused rollout coverage suite 57 passed; headroom.rollout + rollout CLI: 98% coverage Affected proxy/rollout/transform/governance suites 222 passed; 0 failed Final changed regression suites 100 passed; 0 failed Cross-module hot-reload isolation regression 6 passed; 0 failed cargo test -p headroom-core -p headroom-proxy --quiet headroom-core: 924 passed; 1 ignored headroom-proxy and integration suites: all passed cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings cargo fmt --all -- --check ruff check . ruff format --check . mypy headroom --ignore-missing-imports git diff --check All passed cd docs && npm run build Compiled successfully; 164 static pages generated ``` The unsharded Windows-only CI selection exposed unrelated baseline failures, principally the existing `sqlite:///C:\\...` URL parser producing an invalid `\\C:\\...` path. At commit `8e793a80`, all 52 completed GitHub checks passed; the only other conclusions are expected skips and superseded governance jobs. ## Real Behavior Proof - **Environment:** Windows checkout on Python 3.13.3 and the current Rust workspace, based on upstream `main` at `93f2d7a2`. - **Exact command / steps:** Exercised canary and beta feature requests through CLI status, Python `/stats.rollout`, Rust `/rollout/status`, multi-worker payload round trips, loopback `/admin/runtime-env`, real proxy request shaping before/after hot reload, installer manifest generation, and shared Python/Rust policy vectors. - **Observed result:** Stable blocks unstable requests; disable wins over explicit/default/legacy/unsafe paths; unsafe state reports `qualification_eligible=false`; worker handoff rejects tampering; running output shaping changes only when the effective beta policy permits it; explicit blocked flags fail with actionable diagnostics. - **Not tested:** Live production traffic requiring provider credentials, or future artifact qualification/promotion automation (intentionally out of scope). ## Runtime Rollout Safety - **Rollout-managed features:** Python `tool_result_interceptors`, `proxy_output_shaper`, `read_maturation`; Rust `native_bedrock`, `openai_responses_streaming`, `canary_probe`. - **Minimum rollout channel:** Registry-defined per feature; process default is `stable`. - **Stable/default behavior changed:** No unstable feature becomes enabled by default. Explicit blocked CLI flags now fail instead of silently doing nothing. - **Kill switch / disable path:** `HEADROOM_DISABLE_FEATURES=<comma-separated feature names>`; explicit disable has highest precedence, including over the unsafe override. - **Unsafe override required:** No. `HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES=1` is break-glass only and makes qualification evidence ineligible. - **Qualification impact:** Adds machine-readable policy/snapshot identities and eligibility; does not implement qualification itself. - **Rollback path:** Set the named disable list for operational rollback, lower the channel, or revert this PR. ## Review Readiness - [x] I have performed a full diff review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented hard-to-understand areas - [x] I have made corresponding documentation changes - [x] My changes generate no new warnings - [x] I added tests that reproduce and prevent every regression fixed during review - [x] New and existing affected tests pass locally - [x] I did **not** edit `CHANGELOG.md`; release-please generates it from the Conventional Commit PR title ## Additional Notes Out of scope: artifact candidates, benchmark orchestration, qualification manifests/gates, promotion automation, release branches, publication guards, and release-risk classification. Those workflows can consume the rollout registry digest, runtime snapshot digest, decision reasons, and qualification eligibility through supported black-box interfaces. --------- Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> Co-authored-by: JD Davis <jd@JDH-AIR-00.local> |
||
|
|
e0eb0943f0 |
fix(proxy): support Windows selector loop on uvicorn < 0.36 (#1655)
## Description Fixes `headroom proxy` crashing on Windows with `KeyError: 'asyncio:SelectorEventLoop'` when the installed uvicorn version is older than 0.36. PR #1496 added `loop="asyncio:SelectorEventLoop"` to keep the Windows selector event loop and avoid ProactorEventLoop listener failures on transient AcceptEx errors. That string is valid on uvicorn >= 0.36 as a custom loop-factory import path, but uvicorn < 0.36 only accepts built-in loop names and raises during startup. Fixes #1650 Fixes #1621 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update ## Changes Made - Added `_configure_windows_uvicorn_loop()` to branch on uvicorn capability. - Keeps `loop="asyncio:SelectorEventLoop"` for uvicorn versions that support custom loop factories. - Uses `asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())` on older uvicorn versions without passing an unsupported `loop` kwarg. - Extended regression coverage to exercise both paths with mocks. - Added an Unreleased changelog entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Formatting check passes (`ruff format --check`) - [x] New tests added for new functionality - [x] Manual behavior proof supplied ### Test Output ```text Author reported: ruff check headroom/proxy/server.py tests/test_proxy_scalability.py ruff format --check headroom/proxy/server.py tests/test_proxy_scalability.py standalone uvicorn custom loop import-path verification passed on uvicorn 0.49 Reviewer previously attempted: python -m pytest tests/test_proxy_scalability.py -q Observed reviewer result: local run failed during import because this checkout did not have the native headroom._core extension built, before the PR-specific uvicorn assertions ran. ``` ## Real Behavior Proof - Environment: Linux CI agent, CPython 3.12, uvicorn 0.49.0, source checkout on `PYTHONPATH`; Windows 11 confirmation from a reporter using Headroom 0.29.0, Python 3.13, uvicorn 0.35.0. - Exact command / steps: `python3 -c "import asyncio, uvicorn; c=uvicorn.Config('app', loop='asyncio:SelectorEventLoop'); f=c.get_loop_factory(); loop=f(); print(type(loop).__name__); assert isinstance(loop, asyncio.SelectorEventLoop); loop.close()"` - Observed result: Printed `_UnixSelectorEventLoop`, confirming the uvicorn >= 0.36 import path resolves to a selector loop. A reporter confirmed uvicorn 0.35.0 lacks `Config.get_loop_factory`, hits the original `KeyError`, and starts cleanly with this PR's selector policy approach. - Not tested: A full live Windows proxy startup matrix across every uvicorn minor version; the older-uvicorn branch is covered by mocked regression tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: syf2211 <syf2211@users.noreply.github.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
ebe0a3bd7b |
feat(proxy): add provider-only HTTP proxy (#1807)
## Description Adds provider-only HTTP proxy configuration for upstream LLM calls without setting process-wide proxy environment variables. `--http-proxy` and `HEADROOM_HTTP_PROXY` are scoped to the proxy server's provider HTTPX clients, and HTTP/2 is disabled for those clients when the proxy is set so HTTPS provider APIs can tunnel through CONNECT. Using process env vars such as `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, or `NO_PROXY` would also affect HTTPX, but those vars are inherited by tool executions, so this keeps proxy routing out of the global environment. Closes: N/A ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `--http-proxy` with `HEADROOM_HTTP_PROXY` fallback. - Passed the proxy URL only into provider HTTPX clients. - Disabled provider HTTP/2 when the proxy is configured. - Preserved the new setting through direct server startup and multi-worker config serialization. - Documented the flag/env var and why global `HTTP_PROXY`-style vars are not suitable for provider-only routing. - Added an Unreleased changelog entry. - Added coverage for CLI/env wiring, worker serialization, and HTTPX client options. ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen pytest tests/test_cli_proxy_improvements.py tests/test_proxy_scalability.py ============================== 72 passed in 5.95s ============================== $ uv run --frozen ruff check headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/server.py tests/test_cli_proxy_improvements.py tests/test_proxy_scalability.py All checks passed! $ uv run --frozen mypy headroom --ignore-missing-imports Success: no issues found in 406 source files $ env -u HTTP_PROXY -u http_proxy npm --prefix docs run types:check [MDX] generated files in 6.351916000000074ms Generating route types... [MDX] generated files in 5.813166999999794ms ✓ Types generated successfully $ git diff --check # no output ``` ## Real Behavior Proof - Environment: local provider setup that requires outbound LLM traffic through an HTTP proxy - Exact command / steps: ran focused pytest, Ruff, mypy, docs `types:check`, and `git diff --check` after rebasing the branch onto `origin/main`; reviewed the docs and changelog diffs; actively used the new proxy setting locally for a provider that requires proxied egress - Observed result: CLI/env/config tests passed; static checks passed; docs type generation passed; local provider traffic can be routed through the provider-only proxy setting without exporting global proxy variables to tool executions - Not tested: broad provider matrix across every supported upstream ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. CLI/backend/docs update only. ## Additional Notes The branch keeps implementation, docs, changelog, and formatting changes in separate commits. |
||
|
|
17c7347402 |
fix(proxy): use selector loop on Windows (#1496)
## Description Fixes the Windows proxy listener failure where `headroom proxy` can keep running while `127.0.0.1:8787` stops accepting connections after a transient `WinError 64` / AcceptEx failure. On Windows, uvicorn's default single-process asyncio loop is ProactorEventLoop. If a keep-alive client resets a connection during accept, the Proactor accept path can close the listening socket and never re-arm accept. Passing uvicorn `loop="asyncio:SelectorEventLoop"` on Windows keeps accept failures scoped to the individual connection and leaves the listener registered. Closes #1116 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Force `uvicorn.run(...)` to use `loop="asyncio:SelectorEventLoop"` when `sys.platform == "win32"`. - Leave non-Windows uvicorn loop selection unchanged. - Add regression tests that assert Windows receives the selector-loop kwarg and non-Windows does not. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $env:PYTHONPATH = (Get-Location).Path python -m pytest tests/test_proxy_scalability.py::TestWorkerConfiguration -q ============================= test session starts ============================= platform win32 -- Python 3.13.1, pytest-9.1.1, pluggy-1.6.0 rootdir: E:\work\code\headroom configfile: pyproject.toml plugins: anyio-4.14.0 collected 5 items tests\test_proxy_scalability.py ..... [100%] ======================== 5 passed, 1 warning in 0.79s ========================= $env:PYTHONPATH = (Get-Location).Path python -m ruff check headroom/proxy/server.py tests/test_proxy_scalability.py All checks passed! $env:PYTHONPATH = (Get-Location).Path python -m ruff format --check headroom/proxy/server.py tests/test_proxy_scalability.py 2 files already formatted ``` ### CI Validation GitHub Actions is green for this PR, including: ```text build pass build-wheel pass commitlint pass lint pass # ruff check ., ruff format --check ., mypy headroom test (1) pass test (2) pass test (3) pass test (4) pass test-extras pass test-agno pass test-dashboard-ui pass windows-native-wrapper pass macos-native-wrapper pass docker-native-e2e pass docker-init-e2e pass docker-wrap-e2e pass ``` ### Local Full-Suite Attempt I also completed a local full Python test run on Windows after building the native extension locally and prefetching the HuggingFace model cache: ```text maturin develop -m crates/headroom-py/Cargo.toml --features extension-module -v $env:PYTHONPATH = (Get-Location).Path $env:PYTHONUTF8 = '1' $env:HF_HUB_OFFLINE = '1' $env:TRANSFORMERS_OFFLINE = '1' $env:HF_HUB_DISABLE_TELEMETRY = '1' .\.venv\Scripts\python.exe -m pytest tests scripts/tests --tb=short -q --timeout=90 --timeout-method=thread ``` Result: ```text 50 failed, 7166 passed, 518 skipped, 5807 warnings, 131 errors in 283.41s ``` The local failures are outside this proxy event-loop change and are concentrated in existing Windows/local-environment issues: - SQLite temp database cleanup errors on Windows (`PermissionError: [WinError 32] ... .db`) across memory, graph, and vector-index tests. - Windows URI/path parsing for `sqlite:///C:/...` and `jsonl:///C:/...`, producing invalid `\\C:\...` paths in storage/cache integration tests. - Missing/non-portable local external tooling such as `difftastic`. - Windows-local process/runtime assumptions in a few installer, RTK, lock, and default-storage-path tests. The PR-specific regression tests still pass locally, and the full GitHub Actions suite for this PR is green with the freshly built extension. ## Real Behavior Proof - Environment: Windows 10.0.19045, CPython 3.13.1, uvicorn 0.49.0, Headroom 0.27.0 tool environment, local checkout on `PYTHONPATH`. - Exact command / steps: ran `python -c "import asyncio, uvicorn; c=uvicorn.Config('headroom.proxy.server:create_app_from_env', loop='asyncio:SelectorEventLoop', factory=True); f=c.get_loop_factory(); loop=f(); print(type(loop).__name__); assert isinstance(loop, asyncio.SelectorEventLoop); loop.close()"` with `PYTHONPATH` pointed at this checkout. - Observed result: command printed `_WindowsSelectorEventLoop`, proving uvicorn 0.49 resolves the configured loop string to the Windows selector event loop. - Not tested: I did not run a long live Claude/Codex session against this source checkout because the checkout was not fully installed from source on this machine. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes - Documentation: N/A; this is an internal event-loop selection fix with no user-facing API change. - CHANGELOG: not updated because this branch's `CHANGELOG.md` currently contains pre-existing conflict markers on `main`, and this PR intentionally avoids touching unrelated release-note state. |
||
|
|
b7be3814f1 |
feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits (#818)
## Description
A data-driven push for better compression savings without accuracy loss,
in four parts: expose and tune the Rust compressor knobs, harden the CCR
retrieval store, add traffic-audit tooling that sizes opportunities from
real transcripts, and introduce **read maturation** — a new,
live-validated mechanism that compresses Read outputs *before* they ever
enter the provider prefix cache.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### 1. Rust compressor extraction
- Expose `lossless_min_savings_ratio` end-to-end and lower the default
0.30 → 0.15 (lockstep across Rust, PyO3, and both Python config classes)
so the lossless Table/CSV compaction path wins more often.
- Expose the `CompactConfig` heuristics (core-field fraction,
heterogeneity ratio, flatten cap, bucket bounds) through PyO3 + Python.
- `SearchCompressor` grouped-by-file output (`rg --heading` style — path
once per file instead of per match). Library default off; the proxy
enables it in token mode.
- Complete `factor_out_constants`: constant fields now emit once in a
`_constant_fields` sentinel with slim rows (defensive per-item value
match; default off).
- `ContentRouter` accepts a SmartCrusher config override and the
search-grouping knob.
### 2. CCR store hardening
- Session-scale TTL: 300s → 1800s (CCRConfig, CompressionEntry,
CompressionStore, Rust `DEFAULT_TTL` — lockstep).
- **SQLite is the default CCR backend** (`~/.headroom/ccr_store.db`,
WAL): survives proxy restarts and is shared across workers.
`HEADROOM_CCR_BACKEND=memory` opts out.
- Multi-worker safety: `busy_timeout`, and corruption detection narrowed
so transient `SQLITE_BUSY` errors can never trigger database deletion.
- Data-at-rest hygiene: `chmod 600` on db + sidecars, expired rows swept
at open.
- Retrieval-miss messages are actionable (re-read the file / re-run the
command).
### 3. Traffic audit tooling (measure before tuning)
- `headroom audit-reads`: sizes Read opportunities from local Claude
Code transcripts (read share, stale %, line-number overhead, context
residency, cache-death windows).
- `--simulate-maturation`: Mechanism B risk sizing (re-read rates,
never-touched-again share, quiesce coverage, at-risk edits).
- `--codex`: shell-read classifier for Codex transcripts (rtk-wrapper
aware, workdir resolution).
- Findings that shaped this PR (81 sessions): Reads are 67% of tool
bytes; median Read lingers 118 turns (~13x lifetime cost); a prototyped
repeat-Read dedup measured 0.1% and was **removed** rather than shipped
as dead code.
### 4. Read maturation (Mechanism B) — experimental, default OFF
- Activity-based: a fresh large Read is held **out** of the provider
cache (trailing breakpoint relocated before it), stays verbatim while
its file is active, and matures into a CCR-backed marker once the file
is quiet for `quiesce_turns` (default 5; `max_hold_turns` bounds busy
files).
- Only the final compressed form ever enters the cache — **no cached
byte is ever mutated**; matured markers replay byte-identically.
- Wired into the Anthropic handler behind `--read-maturation` /
`HEADROOM_READ_MATURATION=1`; session state rides on the prefix tracker;
advisory (can never fail a request).
- Live-validated against the Anthropic API: held content excluded from
cache_creation; after maturation the prior cached prefix still served —
the no-bust invariant holds end-to-end.
### 5. Rebase / CI fixups (this update)
- Rebased onto latest `main` (was 28 commits behind): picks up `ci: pass
CODECOV_TOKEN to coverage uploads (#968)`, which is what was turning the
4 test shards red — the tests themselves passed (1528) but the post-test
codecov upload exited non-zero on a protected branch.
- Resolved the duplicate `lossless_min_savings_ratio` that two
independent main/branch additions left in `SmartCrusherConfig` and the
Rust-config kwarg (import-time `SyntaxError` + mypy `no-redef`).
- Aligned CCR tests with the new defaults (SQLite backend, 1800s TTL)
across `test_ccr`, `test_adapter_hooks`, `test_compression_store`,
`test_proxy_ccr`, and the lossy row-drop bridge test.
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy_ccr.py tests/test_ccr.py tests/test_compression_store.py tests/test_adapter_hooks.py tests/test_ccr_row_drop_store_bridge.py -q
170 passed, 4 warnings in 42.49s
$ python -m pytest tests/test_audit_reads.py tests/test_audit_codex.py tests/test_read_maturation.py tests/test_transforms_content_router.py tests/test_smart_crusher_toin_attachment.py -q
83 passed
$ mypy headroom/
Success: no issues found in 365 source files
$ python -m compileall headroom/ -q
COMPILE-OK
# CI (run 27488990477, pre-rebase head): all 4 shards ran to completion —
# "1528 passed, 120 skipped, 4922 deselected"
# The red shards were the codecov upload step, not test failures; fixed by
# the #968 rebase above.
```
## Real Behavior Proof
- Environment: macOS (darwin), Python 3.12 venv; branch
`feat/compression-extraction` rebased onto `origin/main` (head
|
||
|
|
e2d95614c2 | fix(proxy): support multi-worker Docker env startup | ||
|
|
f4160f9e47 |
Fix Codex WebSocket HTTP 500: ChatGPT auth routing, correct beta header, HTTP fallback (#71)
Root causes: - ChatGPT session auth tokens sent to api.openai.com instead of chatgpt.com - Fallback beta header was responses-api=v1 instead of responses_websockets=2026-02-06 Fixes: - Detect ChatGPT-Account-ID header and route WS/HTTP to chatgpt.com/backend-api/codex/responses - Update beta header fallback to match what Codex actually sends - Add HTTP POST streaming fallback when upstream WS fails (relay SSE over client WS) - Unwrap response.create envelope in HTTP fallback for correct POST body - Initialize body before JSON parse to prevent NameError in fallback path - Fix async test failures: convert asyncio.get_event_loop() to asyncio.run() |
||
|
|
a187d80d7c |
fix: resolve CI failures
- Format feature_extractor.py to pass ruff format check - Skip uvicorn tests when uvicorn not installed in CI environment - Exclude experiments/ from pre-commit ruff checks |
||
|
|
7c6d713ae0 |
feat(proxy): add connection pooling, HTTP/2, and multi-worker support
Improve proxy scalability for high-concurrency scenarios with multiple agents: - Add connection pool configuration (max_connections=500, max_keepalive=100) - Enable HTTP/2 multiplexing by default for better throughput - Add multi-worker support via --workers flag for multi-core scaling - Add --limit-concurrency flag for backpressure control - Reuse httpx client for CCR continuations instead of creating new clients - Add httpx[http2] dependency for HTTP/2 support - Add comprehensive test suite for scalability features New CLI flags: --max-connections Max connections to upstream APIs (default: 500) --max-keepalive Max keepalive connections (default: 100) --no-http2 Disable HTTP/2 (enabled by default) --workers Number of worker processes (default: 1) --limit-concurrency Max concurrent connections before 503 (default: 1000) |