Commit Graph
11 Commits
Author SHA1 Message Date
2e4ff6f9c7 fix(proxy): keep upstream credentials out of manifests, env and TOIN (#4866)
## Description

Proxy secrets (upstream API keys, proxy token, licence) were stored or
passed in clear:

- Bench manifests and `ProxyDeploymentPlan.to_dict()` serialized the
full `ProxyConfig`, keys included.
- With `--workers N`, the whole config, keys included, went to each
worker in the `HEADROOM_PROXY_CONFIG_JSON` env var, which other
processes of the same user can read (`ps eww`, `/proc/<pid>/environ`).
- TOIN stored query field names in clear, although its module docstring
says field names are hashed; the text before a `:` can be a credential
(`<token>:x-oauth-basic`).

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/models.py`: `PROXY_CONFIG_SECRET_FIELDS` (one list)
and `redact_proxy_config_secrets`.
- `headroom/testing/harness.py`, `ProxyDeploymentPlan.to_dict()`:
serialize the redacted config; the in-memory plan keeps real values.
- `headroom/proxy/server.py` (`--workers N`): secrets go to a 0600 temp
file named by `HEADROOM_PROXY_CONFIG_SECRETS_FILE`; workers read it, the
env JSON carries the config without them, and the file is removed when
the server stops. A worker that cannot read the file raises
`ConfigurationError` instead of falling back to env defaults (which
could drop the proxy token).
- `headroom/telemetry/toin.py`: query patterns hash field names; stored
patterns with raw field names are dropped on load and the store is
rewritten at once (a private `_import_patterns` reports whether anything
was dropped).
- Tests: `tests/test_proxy_config_secrets_at_rest.py` (manifest,
`to_dict`, worker env carry no secret; workers still get the real
values); `tests/test_toin_retention.py` expectations updated to hashed
field names.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_proxy_config_secrets_at_rest.py tests/test_toin_retention.py tests/test_stateless_toin.py tests/test_toin_*.py ...
125 passed

# new tests against origin/main
tests/test_proxy_config_secrets_at_rest.py: 3 failed (+1 for the fail-closed worker, which fails on the first commit too)
test_legacy_raw_query_keys_are_removed_on_load: fails without the load-time rewrite

# every test file importing the changed modules (73 files), this branch
2 failed, 1504 passed: tests/test_read_maturation_handler_nobust.py, both also failing on origin/main
(an earlier run on the first commit also hit tests/test_proxy_health.py failures that fail the same way on origin/main,
and one tests/test_proxy_ccr.py timing failure that main showed in a sibling test; that file passes 3/3 alone on both)

ruff check / format --check: clean; mypy: no new errors (same 8 pre-existing as origin/main)
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, the real `headroom proxy --workers 2
--stateless --no-telemetry` CLI, isolated `HOME`/`TMPDIR`,
`--openai-api-url` pointed at a local HTTP server that prints the
`x-gateway-auth` header it receives, `--openai-extra-headers
'{"x-gateway-auth":"FAKE-GATEWAY-0123"}'`.
- Exact command / steps: start the proxy, run `ps eww` on each worker
process, list `$TMPDIR`, POST `/v1/chat/completions`, stop the proxy
with SIGTERM.
- Observed result:
- origin/main: both worker processes' environments contain
`FAKE-GATEWAY-0123` (via `HEADROOM_PROXY_CONFIG_JSON`).
- this branch: neither worker's environment contains it; `$TMPDIR` holds
one `headroom-proxy-config-*.json` with mode `-rw-------`; the upstream
still receives `x-gateway-auth: FAKE-GATEWAY-0123` (workers load it from
the file); after SIGTERM the file is gone.
- On both, the parent's own command line holds the value because the
test passed it as a CLI flag; that is outside this change.
- Not tested: SIGKILL of the parent (the file stays in `$TMPDIR`).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: manifests and `to_dict()` show
secrets redacted; TOIN query-pattern keys are hashed and old raw-name
keys are dropped once on load (learned query statistics for those keys
restart).
- Kill switch / disable path: none.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-14 10:46:55 +05:30
ee731d7ea8 fix(proxy): refuse token-less non-loopback binds, gate operator routes (#3852)
## Why

Four related gaps in the proxy's network gate (findings 01-F1, 01-F2,
01-F5, 01-F9):

- **Open bind.** A non-loopback bind with no `HEADROOM_PROXY_TOKEN` only
logged a warning, then served the `/v1/*` relay, with the operator's
upstream credentials, to every peer that could reach the port.
- **Missing peer address.** `is_loopback_host(None)` returned `True`. A
Unix-domain socket, an ASGI adapter that leaves `scope["client"]` unset,
or a hand-built request therefore turned off the token gate, the
`/admin` and `/debug` guards and the WebSocket gate.
- **Operator data routes.** `/stats-history` (including CSV export),
`/quota`, `/subscription-window` and `/metrics` were served to any
network caller.
- **WebSocket loopback exemption** checked only the peer IP, not the
`Host` header, so it had no DNS-rebinding defence.

## What changes

- New `headroom/proxy/bind_policy.py`. A non-loopback bind (`0.0.0.0`,
`::`, a LAN address or a hostname) with no token is refused at three
places: `create_app` (embedders and the multi-worker factory),
`run_server` (one exit with code 2 before uvicorn forks) and the
`headroom proxy` CLI (a clear error before the banner, exit code 1). An
absent host means uvicorn's default `127.0.0.1`.
- `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` acknowledges an open bind when
the runtime already restricts reachability, such as a container
published on `127.0.0.1`. The `proxy_open_bind` warning is still logged
on that path, and the banner says the bind is open.
- `is_loopback_host(None)` now returns `False` (fail closed), matching
`identity.py`.
- `/stats-history`, `/quota` and `/subscription-window` answer only: a
non-loopback caller that passed the token gate, a loopback caller (peer
and `Host` header both loopback), a trusted dashboard client, or, under
an acknowledged token-less bind, the container's own default-gateway TCP
peer with a loopback `Host` (the host browser reaching a
`127.0.0.1`-published container). Everyone else gets 404, as with
`/admin`. Settings writes keep their stricter rule with no token
short-cut.
- `/metrics` answers a non-loopback caller that passed the token gate, a
loopback caller, a connecting peer in
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` (whatever it forwards), or a
resolved client in the dashboard CIDRs. For CIDR peers a hostname `Host`
is fine, so Prometheus scrapers work; a cross-origin browser request is
still refused.
- The token short-cut applies only to non-loopback requests. The gate
exempts loopback peers without checking `Host`, so a loopback request
has not proven it holds the token and must pass the loopback `Host`
check. This blocks a DNS-rebound browser on the operator's machine.
- The `/dashboard` shell stays ungated. It is a static template, like
the already public `/dashboard/static` assets. Gating it would break the
dashboard for containers published on host loopback, where the peer is
the bridge gateway and `Host` is `localhost`. The data routes behind it
are gated.
- The WebSocket loopback exemption requires a loopback peer and a
loopback `Host` header. A loopback peer with a foreign `Host` can still
connect with the token.
- Launchers that always publish on `127.0.0.1` set the acknowledgement
themselves: `headroom install` (docker preset), `scripts/install.sh` and
`scripts/install.ps1` (through one helper that adds the acknowledgement
only together with the `127.0.0.1` publication), and
`docker/docker-compose.native.yml`.
- Docs: `proxy.mdx` documents the new variable and which routes the
token also gates; `docker-install` (docs and wiki) and the
`docker-compose.yml` comment show the acknowledgement in the
loopback-published example. `e2e/docker-bind-security.sh` sets it and
checks that the same launch without it refuses to start.

## Behaviour change (breaking)

- `headroom proxy --host 0.0.0.0`, or `HEADROOM_HOST` set to any
non-loopback address, without `HEADROOM_PROXY_TOKEN` now refuses to
start. The error names both remedies. Set a token, bind loopback, or set
`HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` when the port is already
confined. Anyone running the image by hand with `--host 0.0.0.0` and no
token must add one of these.
- Without a token, `/stats-history`, `/quota`, `/subscription-window`
and `/metrics` return 404 to non-loopback callers outside a trusted
CIDR. Prometheus scrapers on another host must be in
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` or send the token.
- Code that relied on a request with no peer address being treated as
loopback now gets the non-loopback path.
- The default loopback run is unchanged.

## Tests

- `tests/test_proxy_bind_policy.py`: the refuse/allow matrix
(non-loopback, loopback, unset host, token, acknowledgement values), the
warning on the acknowledged path, `create_app`, the `run_server` exit
before uvicorn, and the CLI (refusal, `HEADROOM_HOST` from env,
acknowledged start, offline banner, start with a token).
- `tests/test_proxy_loopback_gating.py`: each data route against
non-loopback, loopback, trusted-CIDR and token callers; the dashboard
shell stays reachable; the host of an acknowledged, loopback-published
container reads the data routes while another bridge peer, a forwarded
gateway address and a foreign `Host` do not; CSV export is not served to
network callers; `/metrics` across loopback, both CIDR lists,
out-of-range peers, cross-origin browsers and token callers, including a
trusted gateway relaying a scraper outside its own range. A regression
test shows that a token on the proxy does not exempt a loopback peer
from the `Host` check; it fails without the fix.
- `tests/test_proxy_hardening.py`: the HTTP gate with no peer address,
WebSocket `Host` cases, `require_loopback` and `/debug` with no peer.
- Installer tests: `headroom install` adds the acknowledgement, and the
shell and PowerShell wrappers add it only together with the loopback
publication. Launcher-level tests start the proxy with the environment
and `--host` that `build_runtime_command()` and
`docker-compose.native.yml` produce and check that the host browser
(bridge-gateway peer) gets `/stats-history` and another container does
not.
- Tests that bound `0.0.0.0` for unrelated reasons now set a token or
the acknowledgement. Tests that read operator routes use an explicit
loopback peer.

On `main`, 14 of the gating and hardening tests fail and the bind-policy
module does not exist. Manual runs: token-less `--host 0.0.0.0` exits
with the refusal from both the CLI and `run_server`; with the
acknowledgement, the data routes are 404 to a LAN peer and to a loopback
peer with a foreign `Host`; with a token, a LAN peer gets 401 without it
and 200 with it; the default loopback bind starts as before.

Not tested: a real Unix-domain-socket deployment and a real Prometheus
scrape (both covered by tests only).

## Merge order

Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 →
#3863. This PR goes first: it is the gate the others assume, and it is
the release-note event.

## Not in this PR

- Enforcing the token for loopback callers when one is configured.

Revised 2026-10-02 after review: `/metrics` matches gateway CIDRs
against the TCP peer, and the operator data routes serve the host of a
loopback-published container (bridge-gateway peer under the launchers'
bind acknowledgement) so the default Docker dashboard works.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3852"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 08:27:35 -05:00
3077ac81e8 feat: add deterministic runtime rollout controls (#1490)
## Description

Establish one centrally resolved, observable, deterministic, versioned
runtime rollout-control mechanism for Headroom. Runtime rollout controls
which behaviors an already-built artifact may expose; it does not select
or qualify a Headroom release/version.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Bug fix (non-breaking change that fixes rollout enforcement
regressions)
- [x] Documentation update
- [x] Code refactoring (no functional changes)

## Changes Made

- Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`,
`--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared
by Python configuration boundaries.
- Added schema/policy versions, canonical registry and snapshot SHA-256
identities, per-feature decision reasons, disable precedence, unsafe
qualification poisoning, strict CLI validation, and fail-closed
environment handling.
- Added `headroom rollout status --json`, Python `/stats.rollout`, and
Rust `/rollout/status` runtime provenance.
- Added equivalent Rust snapshot semantics and shared Python/Rust policy
vectors while retaining language-specific feature registries.
- Enforced rollout policy at alternate Python server composition roots
so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate.
- Preserved typed rollout snapshots across multi-worker serialization
with schema, policy, registry, snapshot-digest, type, and feature-name
validation.
- Made loopback runtime output-shaper updates replace the immutable
snapshot atomically for request readers, retain explicit request/disable
provenance, preserve channel and kill-switch precedence, invalidate
cached stats, and return the effective rollout decision.
- Made `headroom learn --verbosity --apply` report a channel-blocked
update instead of claiming the shaper is live.
- Made explicit CLI feature flags fail loudly when their current channel
blocks them.
- Made persistent interceptor installation select canary automatically,
or reject an explicitly insufficient channel unless the break-glass
override is set.
- Updated architecture, proxy, rollout, learn, and output-shaper
documentation with required channels and hot-reload semantics.

## Testing

- [x] Unit tests pass
- [x] Linting passes (`ruff check .` and `ruff format --check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New regression tests added for every corrected behavior
- [x] Rust tests and production-target Clippy pass
- [x] Documentation build passes

### Test Output

```text
Focused rollout coverage suite
57 passed; headroom.rollout + rollout CLI: 98% coverage

Affected proxy/rollout/transform/governance suites
222 passed; 0 failed

Final changed regression suites
100 passed; 0 failed

Cross-module hot-reload isolation regression
6 passed; 0 failed

cargo test -p headroom-core -p headroom-proxy --quiet
headroom-core: 924 passed; 1 ignored
headroom-proxy and integration suites: all passed

cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings
cargo fmt --all -- --check
ruff check .
ruff format --check .
mypy headroom --ignore-missing-imports
git diff --check
All passed

cd docs && npm run build
Compiled successfully; 164 static pages generated
```

The unsharded Windows-only CI selection exposed unrelated baseline
failures, principally the existing `sqlite:///C:\\...` URL parser
producing an invalid `\\C:\\...` path. At commit `8e793a80`, all 52
completed GitHub checks passed; the only other conclusions are expected
skips and superseded governance jobs.

## Real Behavior Proof

- **Environment:** Windows checkout on Python 3.13.3 and the current
Rust workspace, based on upstream `main` at `93f2d7a2`.
- **Exact command / steps:** Exercised canary and beta feature requests
through CLI status, Python `/stats.rollout`, Rust `/rollout/status`,
multi-worker payload round trips, loopback `/admin/runtime-env`, real
proxy request shaping before/after hot reload, installer manifest
generation, and shared Python/Rust policy vectors.
- **Observed result:** Stable blocks unstable requests; disable wins
over explicit/default/legacy/unsafe paths; unsafe state reports
`qualification_eligible=false`; worker handoff rejects tampering;
running output shaping changes only when the effective beta policy
permits it; explicit blocked flags fail with actionable diagnostics.
- **Not tested:** Live production traffic requiring provider
credentials, or future artifact qualification/promotion automation
(intentionally out of scope).

## Runtime Rollout Safety

- **Rollout-managed features:** Python `tool_result_interceptors`,
`proxy_output_shaper`, `read_maturation`; Rust `native_bedrock`,
`openai_responses_streaming`, `canary_probe`.
- **Minimum rollout channel:** Registry-defined per feature; process
default is `stable`.
- **Stable/default behavior changed:** No unstable feature becomes
enabled by default. Explicit blocked CLI flags now fail instead of
silently doing nothing.
- **Kill switch / disable path:**
`HEADROOM_DISABLE_FEATURES=<comma-separated feature names>`; explicit
disable has highest precedence, including over the unsafe override.
- **Unsafe override required:** No.
`HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES=1` is break-glass only and
makes qualification evidence ineligible.
- **Qualification impact:** Adds machine-readable policy/snapshot
identities and eligibility; does not implement qualification itself.
- **Rollback path:** Set the named disable list for operational
rollback, lower the channel, or revert this PR.

## Review Readiness

- [x] I have performed a full diff review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented hard-to-understand areas
- [x] I have made corresponding documentation changes
- [x] My changes generate no new warnings
- [x] I added tests that reproduce and prevent every regression fixed
during review
- [x] New and existing affected tests pass locally
- [x] I did **not** edit `CHANGELOG.md`; release-please generates it
from the Conventional Commit PR title

## Additional Notes

Out of scope: artifact candidates, benchmark orchestration,
qualification manifests/gates, promotion automation, release branches,
publication guards, and release-risk classification. Those workflows can
consume the rollout registry digest, runtime snapshot digest, decision
reasons, and qualification eligibility through supported black-box
interfaces.

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JD Davis <jd@JDH-AIR-00.local>
2026-08-12 23:16:54 -05:00
e0eb0943f0 fix(proxy): support Windows selector loop on uvicorn < 0.36 (#1655)
## Description

Fixes `headroom proxy` crashing on Windows with `KeyError:
'asyncio:SelectorEventLoop'` when the installed uvicorn version is older
than 0.36.

PR #1496 added `loop="asyncio:SelectorEventLoop"` to keep the Windows
selector event loop and avoid ProactorEventLoop listener failures on
transient AcceptEx errors. That string is valid on uvicorn >= 0.36 as a
custom loop-factory import path, but uvicorn < 0.36 only accepts
built-in loop names and raises during startup.

Fixes #1650
Fixes #1621

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update

## Changes Made

- Added `_configure_windows_uvicorn_loop()` to branch on uvicorn
capability.
- Keeps `loop="asyncio:SelectorEventLoop"` for uvicorn versions that
support custom loop factories.
- Uses `asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())`
on older uvicorn versions without passing an unsupported `loop` kwarg.
- Extended regression coverage to exercise both paths with mocks.
- Added an Unreleased changelog entry.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Formatting check passes (`ruff format --check`)
- [x] New tests added for new functionality
- [x] Manual behavior proof supplied

### Test Output

```text
Author reported:
ruff check headroom/proxy/server.py tests/test_proxy_scalability.py
ruff format --check headroom/proxy/server.py tests/test_proxy_scalability.py
standalone uvicorn custom loop import-path verification passed on uvicorn 0.49

Reviewer previously attempted:
python -m pytest tests/test_proxy_scalability.py -q

Observed reviewer result: local run failed during import because this checkout did not have the native headroom._core extension built, before the PR-specific uvicorn assertions ran.
```

## Real Behavior Proof

- Environment: Linux CI agent, CPython 3.12, uvicorn 0.49.0, source
checkout on `PYTHONPATH`; Windows 11 confirmation from a reporter using
Headroom 0.29.0, Python 3.13, uvicorn 0.35.0.
- Exact command / steps: `python3 -c "import asyncio, uvicorn;
c=uvicorn.Config('app', loop='asyncio:SelectorEventLoop');
f=c.get_loop_factory(); loop=f(); print(type(loop).__name__); assert
isinstance(loop, asyncio.SelectorEventLoop); loop.close()"`
- Observed result: Printed `_UnixSelectorEventLoop`, confirming the
uvicorn >= 0.36 import path resolves to a selector loop. A reporter
confirmed uvicorn 0.35.0 lacks `Config.get_loop_factory`, hits the
original `KeyError`, and starts cleanly with this PR's selector policy
approach.
- Not tested: A full live Windows proxy startup matrix across every
uvicorn minor version; the older-uvicorn branch is covered by mocked
regression tests.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: syf2211 <syf2211@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-07-14 01:42:25 -04:00
Andrew McFague ebe0a3bd7b feat(proxy): add provider-only HTTP proxy (#1807)
## Description

Adds provider-only HTTP proxy configuration for upstream LLM calls
without setting process-wide proxy environment variables.

`--http-proxy` and `HEADROOM_HTTP_PROXY` are scoped to the proxy
server's provider HTTPX clients, and HTTP/2 is disabled for those
clients when the proxy is set so HTTPS provider APIs can tunnel through
CONNECT. Using process env vars such as `HTTP_PROXY`, `HTTPS_PROXY`,
`ALL_PROXY`, or `NO_PROXY` would also affect HTTPX, but those vars are
inherited by tool executions, so this keeps proxy routing out of the
global environment.

Closes: N/A

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `--http-proxy` with `HEADROOM_HTTP_PROXY` fallback.
- Passed the proxy URL only into provider HTTPX clients.
- Disabled provider HTTP/2 when the proxy is configured.
- Preserved the new setting through direct server startup and
multi-worker config serialization.
- Documented the flag/env var and why global `HTTP_PROXY`-style vars are
not suitable for provider-only routing.
- Added an Unreleased changelog entry.
- Added coverage for CLI/env wiring, worker serialization, and HTTPX
client options.

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen pytest tests/test_cli_proxy_improvements.py tests/test_proxy_scalability.py
============================== 72 passed in 5.95s ==============================

$ uv run --frozen ruff check headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/server.py tests/test_cli_proxy_improvements.py tests/test_proxy_scalability.py
All checks passed!

$ uv run --frozen mypy headroom --ignore-missing-imports
Success: no issues found in 406 source files

$ env -u HTTP_PROXY -u http_proxy npm --prefix docs run types:check
[MDX] generated files in 6.351916000000074ms
Generating route types...
[MDX] generated files in 5.813166999999794ms
✓ Types generated successfully

$ git diff --check
# no output
```

## Real Behavior Proof

- Environment: local provider setup that requires outbound LLM traffic
through an HTTP proxy
- Exact command / steps: ran focused pytest, Ruff, mypy, docs
`types:check`, and `git diff --check` after rebasing the branch onto
`origin/main`; reviewed the docs and changelog diffs; actively used the
new proxy setting locally for a provider that requires proxied egress
- Observed result: CLI/env/config tests passed; static checks passed;
docs type generation passed; local provider traffic can be routed
through the provider-only proxy setting without exporting global proxy
variables to tool executions
- Not tested: broad provider matrix across every supported upstream

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A. CLI/backend/docs update only.

## Additional Notes

The branch keeps implementation, docs, changelog, and formatting changes
in separate commits.
2026-07-05 15:56:59 -07:00
weijie_chen 17c7347402 fix(proxy): use selector loop on Windows (#1496)
## Description

Fixes the Windows proxy listener failure where `headroom proxy` can keep
running while `127.0.0.1:8787` stops accepting connections after a
transient `WinError 64` / AcceptEx failure.

On Windows, uvicorn's default single-process asyncio loop is
ProactorEventLoop. If a keep-alive client resets a connection during
accept, the Proactor accept path can close the listening socket and
never re-arm accept. Passing uvicorn `loop="asyncio:SelectorEventLoop"`
on Windows keeps accept failures scoped to the individual connection and
leaves the listener registered.

Closes #1116

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Force `uvicorn.run(...)` to use `loop="asyncio:SelectorEventLoop"`
when `sys.platform == "win32"`.
- Leave non-Windows uvicorn loop selection unchanged.
- Add regression tests that assert Windows receives the selector-loop
kwarg and non-Windows does not.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$env:PYTHONPATH = (Get-Location).Path
python -m pytest tests/test_proxy_scalability.py::TestWorkerConfiguration -q

============================= test session starts =============================
platform win32 -- Python 3.13.1, pytest-9.1.1, pluggy-1.6.0
rootdir: E:\work\code\headroom
configfile: pyproject.toml
plugins: anyio-4.14.0
collected 5 items

tests\test_proxy_scalability.py .....                                    [100%]

======================== 5 passed, 1 warning in 0.79s =========================

$env:PYTHONPATH = (Get-Location).Path
python -m ruff check headroom/proxy/server.py tests/test_proxy_scalability.py
All checks passed!

$env:PYTHONPATH = (Get-Location).Path
python -m ruff format --check headroom/proxy/server.py tests/test_proxy_scalability.py
2 files already formatted
```

### CI Validation

GitHub Actions is green for this PR, including:

```text
build                         pass
build-wheel                   pass
commitlint                    pass
lint                          pass  # ruff check ., ruff format --check ., mypy headroom
test (1)                      pass
test (2)                      pass
test (3)                      pass
test (4)                      pass
test-extras                   pass
test-agno                     pass
test-dashboard-ui             pass
windows-native-wrapper        pass
macos-native-wrapper          pass
docker-native-e2e             pass
docker-init-e2e               pass
docker-wrap-e2e               pass
```

### Local Full-Suite Attempt

I also completed a local full Python test run on Windows after building
the native extension locally and prefetching the HuggingFace model
cache:

```text
maturin develop -m crates/headroom-py/Cargo.toml --features extension-module -v
$env:PYTHONPATH = (Get-Location).Path
$env:PYTHONUTF8 = '1'
$env:HF_HUB_OFFLINE = '1'
$env:TRANSFORMERS_OFFLINE = '1'
$env:HF_HUB_DISABLE_TELEMETRY = '1'
.\.venv\Scripts\python.exe -m pytest tests scripts/tests --tb=short -q --timeout=90 --timeout-method=thread
```

Result:

```text
50 failed, 7166 passed, 518 skipped, 5807 warnings, 131 errors in 283.41s
```

The local failures are outside this proxy event-loop change and are
concentrated in existing Windows/local-environment issues:

- SQLite temp database cleanup errors on Windows (`PermissionError:
[WinError 32] ... .db`) across memory, graph, and vector-index tests.
- Windows URI/path parsing for `sqlite:///C:/...` and `jsonl:///C:/...`,
producing invalid `\\C:\...` paths in storage/cache integration tests.
- Missing/non-portable local external tooling such as `difftastic`.
- Windows-local process/runtime assumptions in a few installer, RTK,
lock, and default-storage-path tests.

The PR-specific regression tests still pass locally, and the full GitHub
Actions suite for this PR is green with the freshly built extension.

## Real Behavior Proof

- Environment: Windows 10.0.19045, CPython 3.13.1, uvicorn 0.49.0,
Headroom 0.27.0 tool environment, local checkout on `PYTHONPATH`.
- Exact command / steps: ran `python -c "import asyncio, uvicorn;
c=uvicorn.Config('headroom.proxy.server:create_app_from_env',
loop='asyncio:SelectorEventLoop', factory=True); f=c.get_loop_factory();
loop=f(); print(type(loop).__name__); assert isinstance(loop,
asyncio.SelectorEventLoop); loop.close()"` with `PYTHONPATH` pointed at
this checkout.
- Observed result: command printed `_WindowsSelectorEventLoop`, proving
uvicorn 0.49 resolves the configured loop string to the Windows selector
event loop.
- Not tested: I did not run a long live Claude/Codex session against
this source checkout because the checkout was not fully installed from
source on this machine.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A.

## Additional Notes

- Documentation: N/A; this is an internal event-loop selection fix with
no user-facing API change.
- CHANGELOG: not updated because this branch's `CHANGELOG.md` currently
contains pre-existing conflict markers on `main`, and this PR
intentionally avoids touching unrelated release-note state.
2026-06-28 13:19:40 -07:00
Tejas Chopra b7be3814f1 feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits (#818)
## Description

A data-driven push for better compression savings without accuracy loss,
in four parts: expose and tune the Rust compressor knobs, harden the CCR
retrieval store, add traffic-audit tooling that sizes opportunities from
real transcripts, and introduce **read maturation** — a new,
live-validated mechanism that compresses Read outputs *before* they ever
enter the provider prefix cache.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### 1. Rust compressor extraction

- Expose `lossless_min_savings_ratio` end-to-end and lower the default
0.30 → 0.15 (lockstep across Rust, PyO3, and both Python config classes)
so the lossless Table/CSV compaction path wins more often.
- Expose the `CompactConfig` heuristics (core-field fraction,
heterogeneity ratio, flatten cap, bucket bounds) through PyO3 + Python.
- `SearchCompressor` grouped-by-file output (`rg --heading` style — path
once per file instead of per match). Library default off; the proxy
enables it in token mode.
- Complete `factor_out_constants`: constant fields now emit once in a
`_constant_fields` sentinel with slim rows (defensive per-item value
match; default off).
- `ContentRouter` accepts a SmartCrusher config override and the
search-grouping knob.

### 2. CCR store hardening

- Session-scale TTL: 300s → 1800s (CCRConfig, CompressionEntry,
CompressionStore, Rust `DEFAULT_TTL` — lockstep).
- **SQLite is the default CCR backend** (`~/.headroom/ccr_store.db`,
WAL): survives proxy restarts and is shared across workers.
`HEADROOM_CCR_BACKEND=memory` opts out.
- Multi-worker safety: `busy_timeout`, and corruption detection narrowed
so transient `SQLITE_BUSY` errors can never trigger database deletion.
- Data-at-rest hygiene: `chmod 600` on db + sidecars, expired rows swept
at open.
- Retrieval-miss messages are actionable (re-read the file / re-run the
command).

### 3. Traffic audit tooling (measure before tuning)

- `headroom audit-reads`: sizes Read opportunities from local Claude
Code transcripts (read share, stale %, line-number overhead, context
residency, cache-death windows).
- `--simulate-maturation`: Mechanism B risk sizing (re-read rates,
never-touched-again share, quiesce coverage, at-risk edits).
- `--codex`: shell-read classifier for Codex transcripts (rtk-wrapper
aware, workdir resolution).
- Findings that shaped this PR (81 sessions): Reads are 67% of tool
bytes; median Read lingers 118 turns (~13x lifetime cost); a prototyped
repeat-Read dedup measured 0.1% and was **removed** rather than shipped
as dead code.

### 4. Read maturation (Mechanism B) — experimental, default OFF

- Activity-based: a fresh large Read is held **out** of the provider
cache (trailing breakpoint relocated before it), stays verbatim while
its file is active, and matures into a CCR-backed marker once the file
is quiet for `quiesce_turns` (default 5; `max_hold_turns` bounds busy
files).
- Only the final compressed form ever enters the cache — **no cached
byte is ever mutated**; matured markers replay byte-identically.
- Wired into the Anthropic handler behind `--read-maturation` /
`HEADROOM_READ_MATURATION=1`; session state rides on the prefix tracker;
advisory (can never fail a request).
- Live-validated against the Anthropic API: held content excluded from
cache_creation; after maturation the prior cached prefix still served —
the no-bust invariant holds end-to-end.

### 5. Rebase / CI fixups (this update)

- Rebased onto latest `main` (was 28 commits behind): picks up `ci: pass
CODECOV_TOKEN to coverage uploads (#968)`, which is what was turning the
4 test shards red — the tests themselves passed (1528) but the post-test
codecov upload exited non-zero on a protected branch.
- Resolved the duplicate `lossless_min_savings_ratio` that two
independent main/branch additions left in `SmartCrusherConfig` and the
Rust-config kwarg (import-time `SyntaxError` + mypy `no-redef`).
- Aligned CCR tests with the new defaults (SQLite backend, 1800s TTL)
across `test_ccr`, `test_adapter_hooks`, `test_compression_store`,
`test_proxy_ccr`, and the lossy row-drop bridge test.

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_proxy_ccr.py tests/test_ccr.py tests/test_compression_store.py tests/test_adapter_hooks.py tests/test_ccr_row_drop_store_bridge.py -q
170 passed, 4 warnings in 42.49s

$ python -m pytest tests/test_audit_reads.py tests/test_audit_codex.py tests/test_read_maturation.py tests/test_transforms_content_router.py tests/test_smart_crusher_toin_attachment.py -q
83 passed

$ mypy headroom/
Success: no issues found in 365 source files

$ python -m compileall headroom/ -q
COMPILE-OK

# CI (run 27488990477, pre-rebase head): all 4 shards ran to completion —
#   "1528 passed, 120 skipped, 4922 deselected"
# The red shards were the codecov upload step, not test failures; fixed by
# the #968 rebase above.
```

## Real Behavior Proof

- Environment: macOS (darwin), Python 3.12 venv; branch
`feat/compression-extraction` rebased onto `origin/main` (head
7cb0f43b); GitHub Actions CI run 27488990477 for the test shards
- Exact command / steps: rebased onto latest main (clean, 13 commits
replayed, 0 conflicts); ran the pytest suites and mypy above locally;
inspected CI shard logs to confirm the failure was the codecov upload,
not the test phase
- Observed result: 253 targeted tests pass locally; mypy clean on 365
files; CI test phase reports `1528 passed, 120 skipped`; the only red
step (codecov `upload-coverage` → "Token required because branch is
protected") is resolved by the rebased-in #968 CODECOV_TOKEN fix
- Not tested: the read-maturation live-API no-bust validation
(`tests/test_live/`) was not re-run in this rebase pass (requires
provider keys); it was validated when the feature first landed, and no
maturation code changed in the rebase — only CCR-default test assertions
and the duplicate-field resolution

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

CHANGELOG is generated by release-please from the conventional commits,
so the CHANGELOG box is intentionally left unchecked. "Manual testing
performed" is unchecked deliberately — see `Real Behavior Proof` → `Not
tested` for the exact boundary (the live-API maturation validation was
not re-run in this rebase pass).

### Follow-ups (tracked, not in this PR)

- Mechanism B provider extensions: OpenAI-family wiring (no breakpoint
hold — bounded near-tail bust) and the Codex runtime read-detector (the
audit classifier is the prototype).
- Pilot enablement playbook: run `audit-reads --simulate-maturation` on
target traffic → pick `quiesce_turns` → enable via env → watch cache hit
rate + `read_maturation:N` transform tags.
2026-06-16 20:21:13 -07:00
Kayzo e2d95614c2 fix(proxy): support multi-worker Docker env startup 2026-04-26 12:25:33 +00:00
chopratejas f4160f9e47 Fix Codex WebSocket HTTP 500: ChatGPT auth routing, correct beta header, HTTP fallback (#71)
Root causes:
- ChatGPT session auth tokens sent to api.openai.com instead of chatgpt.com
- Fallback beta header was responses-api=v1 instead of responses_websockets=2026-02-06

Fixes:
- Detect ChatGPT-Account-ID header and route WS/HTTP to chatgpt.com/backend-api/codex/responses
- Update beta header fallback to match what Codex actually sends
- Add HTTP POST streaming fallback when upstream WS fails (relay SSE over client WS)
- Unwrap response.create envelope in HTTP fallback for correct POST body
- Initialize body before JSON parse to prevent NameError in fallback path
- Fix async test failures: convert asyncio.get_event_loop() to asyncio.run()
2026-04-06 12:48:46 -07:00
chopratejas a187d80d7c fix: resolve CI failures
- Format feature_extractor.py to pass ruff format check
- Skip uvicorn tests when uvicorn not installed in CI environment
- Exclude experiments/ from pre-commit ruff checks
2026-01-27 16:17:24 -08:00
chopratejas 7c6d713ae0 feat(proxy): add connection pooling, HTTP/2, and multi-worker support
Improve proxy scalability for high-concurrency scenarios with multiple agents:

- Add connection pool configuration (max_connections=500, max_keepalive=100)
- Enable HTTP/2 multiplexing by default for better throughput
- Add multi-worker support via --workers flag for multi-core scaling
- Add --limit-concurrency flag for backpressure control
- Reuse httpx client for CCR continuations instead of creating new clients
- Add httpx[http2] dependency for HTTP/2 support
- Add comprehensive test suite for scalability features

New CLI flags:
  --max-connections    Max connections to upstream APIs (default: 500)
  --max-keepalive      Max keepalive connections (default: 100)
  --no-http2           Disable HTTP/2 (enabled by default)
  --workers            Number of worker processes (default: 1)
  --limit-concurrency  Max concurrent connections before 503 (default: 1000)
2026-01-27 16:08:36 -08:00