Commit Graph
11 Commits
Author SHA1 Message Date
a9815bbb5f fix(proxy): record the provider's billed input count on streamed turns (#4052)
## Description

Every input-volume figure Headroom reports for streamed Claude traffic
was Headroom's own tokenizer estimate, not Anthropic's billed count.
That covers "input after Headroom", `/stats` `tokens.input`, the
dashboard's per-model table and the licence usage report's
`tokens_after`. Since Claude Code always streams, this was effectively
all Claude Code traffic, so customers comparing our numbers with the
Anthropic Console could not reconcile them.

**Root cause.** `emit_request_outcome` records `provider_input_tokens or
optimized_tokens`. The streaming finalizer never set
`provider_input_tokens` for Anthropic (#3946 fixed only the uncached
sub-count), so every streamed Claude turn fell back to
`optimized_tokens`. That value is:
- an `o200k_base` count standing in for Claude's private tokenizer,
measured against `count_tokens` on 6 real Claude Code transcripts at
15–25% under Claude 4.x and 35–38% under Claude 5.x;
- a count of the forwarded **messages only**, which never sees the
system prompt or tool definitions.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`headroom/proxy/provider_usage.py` (new).** One place that knows
each provider's usage semantics:
- Anthropic, Bedrock and Vertex: `input_tokens` is the uncached tail,
and the billed total is `input + cache_read + cache_write`.
- OpenAI `prompt_tokens`/`input_tokens` and Gemini `promptTokenCount`
are already inclusive of cached tokens.
- **Streaming finalizer.** Sets `provider_input_tokens` from the
provider's usage on the Anthropic/OpenAI/Gemini SSE path, the Bedrock
stream path and the OpenAI-via-backend stream path.
`RequestOutcome.from_stream` gains the parameter.
- **OpenAI pass-through.** On both streaming and non-streaming paths,
Anthropic-shape usage was read as if `input_tokens` were the total. It
now uses the billed total and is marked provider-reported.
- **Gemini non-streaming and OpenAI Responses WebSocket.** These already
used the provider's count as `optimized_tokens` but never declared it,
so they now also set `provider_input_tokens`, but only when the provider
actually sent a count.
- **No estimate is ever labelled as billed.** The Bedrock #1132
`message_start` backfill and Gemini's fallback for a missing
`promptTokenCount` both stay estimated. The pass-through usage dialect
is chosen by provider, not inferred from which keys are present.
- **Provenance counters.** Every input total now carries how much of it
is billed versus estimated:
- `/stats` `tokens.input_provider_reported`, `input_estimated`,
`input_provider_reported_percent`, `requests_input_*`.
- Licence usage report: `tokens_after_provider_reported`,
`tokens_after_estimated`, `requests_provider_reported`, plus the billed
`cache_read_tokens`, `cache_write_tokens` and `uncached_input_tokens`,
so the cloud side can reconcile with the console (input + cache read +
cache write). The report also states `tokens_saved_basis:
"headroom_tokenizer"`.
- A one-time `event=input_tokens_estimated` warning per provider/model
when a successful request carried no provider usage.
- **No delta math changes.** `optimized_tokens` stays on the local
scale, and `tokens_saved`, ratios and the ledger delta are untouched.
The provider count is never subtracted from a local count.

## Testing

- [ ] Unit tests pass (`pytest`): targeted suites only, listed below.
The full suite is left to CI (see Additional Notes).
- [x] Linting passes (`ruff check .`): run on all changed files.
- [x] Type checking passes (`mypy headroom`): run on the changed
modules.
- [x] New tests added for new functionality
- [x] Manual testing performed

`tests/test_provider_billed_input.py` adds 15 tests, plus regression
tests for the review fixes in
`tests/test_bedrock_streaming_input_tokens.py` (the #1132 backfill is
reported as estimated) and `tests/test_gemini_ccr_continuation_usage.py`
(a missing `promptTokenCount` is not provider-reported):
- **Recorded fixture.**
`tests/fixtures/anthropic/stream_cached_turn.sse` is a **real recorded
Anthropic stream**: a second call on a cached ~8.9K-token system prompt,
billed as 19 uncached + 8,857 cache-read. It is replayed in 97-byte
chunks through the proxy's own SSE usage parser and
`_finalize_stream_response`, then through `emit_request_outcome` into a
real `PrometheusMetrics`, `CostTracker` and `UsageReporter`. The test
asserts that `/stats` input and the licence `tokens_after` equal
Anthropic's 8,876.
- Other cases:
  - an OpenAI stream records the inclusive `prompt_tokens`;
  - a stream without usage stays estimated;
  - each provider shape is covered;
  - the pass-through normaliser returns the total.

Against the unpatched handlers, the 5 behaviour tests fail. The 8 that
pass exercise only the new helper.

### Test Output

```text
$ pytest tests/test_provider_billed_input.py tests/test_proxy_handler_helpers.py tests/test_request_outcome.py \
    tests/test_proxy_streaming_request_logger.py tests/test_proxy_cache_ttl_metrics.py \
    tests/test_usage_reporter_snapshot.py tests/test_5xx_accounting_all_providers.py tests/test_copilot_provider_label.py \
    tests/test_bedrock_streaming_input_tokens.py tests/test_gemini_ccr_continuation_usage.py -q
181 passed in 6.56s

# same new tests against the unpatched handlers
FAILED tests/test_provider_billed_input.py::test_passthrough_normaliser_returns_anthropic_total_not_uncached_tail
FAILED tests/test_provider_billed_input.py::test_recorded_anthropic_stream_records_anthropics_billed_input
FAILED tests/test_provider_billed_input.py::test_openai_stream_records_inclusive_prompt_tokens
FAILED tests/test_provider_billed_input.py::test_streamed_claude_turn_reaches_dashboard_and_licence_report_as_billed
FAILED tests/test_provider_billed_input.py::test_estimated_input_is_reported_as_estimated
5 failed, 8 passed

$ ruff check <changed files>        -> All checks passed!
$ mypy <changed proxy/telemetry modules> -> Success: no issues found in 6 source files
```

## Real Behavior Proof

- Environment: local proxy from this branch (`headroom proxy --port 8931
--stateless --no-telemetry --no-cache`) against the real Anthropic API.
For the baseline, the same command ran from a `main` worktree on port
8932.
- Exact command / steps: 3 streamed `claude-sonnet-4-6` turns through
the proxy, with a cached ~20K-token system prompt, one tool and a
300-item JSON tool result. Anthropic's `message_start` usage (input +
cache read + cache write) was summed from each stream and compared with
`GET /stats` → `tokens`.
- Observed result: the author-provided live Anthropic comparison below
reports exact billed-input agreement at 61,999 tokens; the baseline
undercounts by 37%. This external run was not independently repeated
during the current-head review.

  | | Anthropic billed input | Headroom `/stats` `tokens.input` |
  |---|---|---|
  | `main` | 62,001 | **38,835** (−37%) |
| this PR | 61,999 | **61,999** (exact),
`input_provider_reported_percent: 100.0` |

The two runs differ by 2 tokens because they are separate API calls.
`proxy_compression_saved` was 29,313 on both, so savings behaviour is
unchanged.
- Not tested: live Bedrock, Vertex and Gemini streams (unit-tested on
provider shapes only), and the licence cloud endpoint receiving the new
fields.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is accounting only, and request
and response bytes are unchanged.
- Minimum rollout channel: N/A
- Stable/default behavior changed: reported input volume now uses the
provider's billed count where the provider reports one. New fields are
added to `/stats` and the licence usage payload; no existing field is
removed or renamed.
- Kill switch / disable path: none needed. With no usage frame, the
fallback is the previous estimate, now labelled `input_estimated`.
- Unsafe override required: no
- Qualification impact: dashboards and licence reports will show larger,
correct Claude input totals once installs upgrade, so historical and new
figures are not directly comparable.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes: targeted
suites only; full suite on CI
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- **Full suite.** Locally, the full run on the pre-rebase base reached
2,147 passed before stopping on
`tests/test_cli/test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`.
That test fails identically on unmodified `main` (unrelated MCP install
args). The full local run was then abandoned because several
wrap/install tests launch a real `serena project index` (rust-analyzer,
tsserver, rustc), which should be stubbed in a separate PR.
- **Docs.** The new `/stats` and licence fields are not yet documented.
- Follow-ups not in this PR:
- **Raw Bedrock invoke pass-through (`handlers/bedrock.py`)** forwards
bytes without parsing them, so it still records an estimate. It is now
counted as `input_estimated`, so it's visible rather than silent.
- **Batch create** has no usage until results come back, so it stays
estimated as above.
- **`tokens_saved` is still on the `o200k_base` scale**, so it
under-reports Claude savings. Calibrating it to provider tokens is a
separate PR.
  - **The licence cloud endpoint** needs to read the new fields.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


## Current maintainer verification (October 8)

Exact head: `81527148ed9777e1b9ca921bbf6bb457d49354db`, containing main
`01d9194854233ef6ed04005cf0a5b8c32bf65d6d` through a clean merge. No
contributor logic was modified. 63 focused tests pass with the existing
compatible native extension on Windows/Python 3.12. Three
backend-boundary probes preserve zero client input and exact cache-read,
cache-write, and combined provider input totals. This is local
mocked-backend/TestClient evidence; the author-provided
external-provider evidence above was not independently rerun.
Current-head hosted snapshot: 31 successful checks, 12 skips, no failed
or live checks.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-09 14:16:49 -07:00
Rod BoevandJerrettDavis ef7605fdbb fix(plugin): resolve hook CLI through plugin-root launcher (#3053)
## Description

The `headroom` plugin invokes `headroom init hook ensure` through
non-login `/bin/sh`, but both `SessionStart` and `PreToolUse` entries
use a bare executable name. Agent hosts whose PATH omits the default
uv/pipx user bin exit 127 on every matching event. Tool calls continue
because the hook is non-blocking, while Headroom compression remains
inactive and the failure banner repeats.

Both hooks now invoke one launcher bundled under `${CLAUDE_PLUGIN_ROOT}`
through `sh`. The launcher honors `HEADROOM_BIN`, preserves
PATH-resolved installations, covers standard uv and pipx prefixes,
records the Homebrew candidates for macOS, and falls back to an
importable Python module. On the launcher-routed path, an absent
Headroom CLI emits one actionable diagnostic and exits 0. Hosts that do
not expose a readable plugin root retain the existing bare-command tail,
preserving the shared GitHub Copilot CLI manifest behavior.

Reported by [@donnydongchen](https://github.com/donnydongchen), whose
exact non-login `env -i` reproduction, Apple Silicon path detail, and
plugin-root suggestion shaped the fix.

Closes #3039

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Route both plugin hook entries through a shared
`${CLAUDE_PLUGIN_ROOT}`-anchored `sh` launcher while preserving
matchers, arguments, and 60-second timeouts.
- Require a non-empty plugin root before probing the launcher, so unset
or empty roots always reach the existing bare-command tail.
- Resolve `HEADROOM_BIN`, PATH, standard uv/pipx/Homebrew locations, and
an importable `python -m headroom.cli` fallback in a deterministic
order.
- Preserve the current bare command for hosts without a readable plugin
root, including the shared Copilot CLI manifest path.
- On the launcher-routed path, replace the absent-CLI status-127 banner
with one actionable diagnostic and a non-blocking exit.
- Add real-shell regression coverage for the reported environment,
resolution precedence, supported prefixes, Python fallback, and negative
boundaries.

## Testing

- [x] Unit tests pass (`uv run --extra dev python -m pytest
tests/test_plugin_manifests.py tests/test_plugin_hook_launcher.py
tests/test_cli/test_init_cli.py -q`, `98 passed`)
- [x] Reproduction override passes (`HEADROOM_REPRO_ARTIFACT` can select
the issue fixture, `uv run --extra dev python -m pytest
tests/test_plugin_hook_launcher.py::test_manifest_command_recovers_reported_non_login_environment
-q`, `1 passed`)
- [x] Linting passes (Ruff check and format validation on the changed
Python test files)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
98 passed
All checks passed!
2 files already formatted
```

## Real Behavior Proof

- Environment: POSIX `sh` with the reporter's sanitized non-login PATH
and a recording Headroom executable only at `$HOME/.local/bin/headroom`
- Exact command / steps: `uv run --extra dev python -m pytest
tests/test_plugin_hook_launcher.py::test_manifest_command_recovers_reported_non_login_environment
-q`; the test preserves the reported PATH and error evidence, executes
both manifest commands with a recorder at `$HOME/.local/bin/headroom`,
and asserts both `home-local init hook ensure` receipts.
- Observed result: `1 passed in 0.41s`; both hooks recovered under the
reported non-login PATH and emitted no command-not-found banner.
- Not tested: live Claude Code plugin-hook execution on a macOS tmux
host

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: plugin hooks resolve Headroom through
the bundled launcher when the plugin root is readable; invalid launcher
roots retain the existing bare-command fallback.
- Kill switch / disable path: none; the existing bare-command fallback
remains available when the plugin root is absent or unreadable.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert the implementation commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes

## Screenshots (if applicable)

Not applicable; this PR changes plugin hook startup behavior without a
rendered UI.

## Additional Notes

The launcher does not source shell profiles, install packages, access
the network, change deployment/profile selection, or modify plugin
versions.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3053"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-05 08:35:35 -05:00
Devanshi Vyas 12c157966f fix(proxy): preserve Claude Code auto-mode protocol (#3784)
## Description

Fix Claude Code server-side auto-mode compatibility when requests pass
through
Headroom’s native Anthropic Messages endpoint.

Previously, auto-mode capability headers, safeguard payloads, query
parameters,
unknown response fields, or opaque SSE frames could be rewritten or lost
while
proxying and handling CCR continuations. This could trigger Claude
Code’s
client-side classifier fallback and its associated warning.

This change adds a request-local Anthropic wire envelope, preserves
client-owned
capability data and opaque response attachments, prevents classifier
payloads
from entering Headroom-owned observations, and documents that
schema-translating
Bedrock, Vertex, and LiteLLM routes do not currently support the
server-side
auto-mode protocol.


Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Preserve native Anthropic request paths and raw query parameters.
- Preserve client-owned `anthropic-version`, `anthropic-beta`, and
safeguard data.
- Avoid sticky-beta, memory, or turn-hook rewrites on classifier-bearing
turns.
- Preserve unknown Anthropic response fields across CCR continuations.
- Preserve opaque SSE frames through request-local parse/render
envelopes.
- Reject unknown or malformed deltas during strict buffered
reconstruction.
- Redact safeguard payloads from diagnostics and pipeline observations.
- Consolidate Anthropic SSE parsing and rendering into the wire module.
- Document auto-mode limitations for Bedrock, Vertex, and LiteLLM
translation paths.
- Add request, response, streaming, CCR, malformed-byte, and passthrough
regression tests.

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output


```text
$ python -m pytest -p no:cacheprovider \
    tests/test_anthropic_auto_mode_passthrough.py \
    tests/test_anthropic_buffered_sse.py \
    tests/test_ccr_response_handler_extra.py \
    tests/test_ccr_buffered_stream_signed_thinking.py \
    tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py \
    tests/test_buffered_ccr_salvage.py \
    tests/test_buffered_ccr_accept_header.py \
    tests/test_proxy/test_anthropic_copilot_upstream_auth.py \
    tests/test_proxy_streaming_request_logger.py \
    tests/test_proxy_streaming_resilience.py -q

125 passed, 1 warning in 13.70s

$ ruff check --no-cache .
All checks passed!

$ ruff format --check --no-cache .
1559 files already formatted

$ python -m mypy \
    headroom/proxy/anthropic_wire.py \
    headroom/ccr/response_handler.py \
    headroom/proxy/handlers/anthropic.py \
    headroom/proxy/handlers/streaming.py

Success: no issues found in 4 source files

$ npx --no-install commitlint \
    --from origin/main --to HEAD \
    --config .commitlintrc.json

Exited successfully.
```

## Real Behavior Proof

- Environment: Pending. Record OS, Python version, Claude Code version,
provider/model, and Headroom revision.
- Exact command / steps: Pending. Start Headroom with `headroom proxy
--port 8787`, then run Claude Code with
`ANTHROPIC_BASE_URL=http://127.0.0.1:8787` and exercise server-side auto
mode.
- Observed result: Pending. Attach redacted output or logs showing the
request completed through Headroom and whether Claude Code used its
client-side fallback.
- Not tested: Live Claude Code auto mode against Anthropic; Bedrock,
Vertex, and LiteLLM translation routes.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Native Anthropic safeguard-capable
requests.
- Kill switch / disable path: Set `CLAUDE_CODE_AUTO_MODE_SERVER=0` to
use Claude Code’s client-side fallback.
- Unsafe override required: No.
- Qualification impact: Live qualification is pending. Translation
routes remain documented as unsupported for server-side auto mode.
- Rollback path: Revert this PR or deploy the previous Headroom
revision.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code where needed
- [x] I have made corresponding documentation changes
- [x] No new warnings were observed in focused tests
- [x] I have added tests for the fix
- [ ] Full existing and new unit test suite passes locally
- [x] I did **not** edit `CHANGELOG.md`

## Screenshots (if applicable)

N/A. This change affects proxy wire behavior.
2026-09-25 09:53:27 -07:00
Rod BoevandJerrettDavis b0c19a25f0 fix(security): reject unauthenticated public proxy binds (#3316)
## Description

Headroom's published image and maintained Docker launchers could expose
the
proxy on every host interface without a proxy token. This change moves
the
security boundary to the deployment artifacts: published images default
to a
container-loopback listener, supported local launchers publish only on
host
loopback, and deliberate public deployments retain explicit token
protection.
The application factory remains transport-neutral.

Closes #3283

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Set both published image stages to a container-loopback default while
keeping explicit host overrides available.
- Changed native Compose and the maintained Bash and PowerShell
launchers to use an internal `0.0.0.0` listener with host-loopback
publication.
- Preserved the top-level Compose token requirement and the existing
HTTP and WebSocket credential behavior for deliberate public binds.
- Scoped differential-capture token injection to the internal
`headroom-proxy` destination and kept sensitive headers redacted in
captured records.
- Added focused launch-contract, native-core, capture, and Docker
publication checks, plus a base/head harness that tolerates a base
already containing the fix.
- Updated Docker, installer, capture, and mirrored documentation,
including an unauthenticated local loopback quickstart and an
authenticated deliberate-public example.

## Testing

- [x] Focused unit and integration tests pass
- [x] Linting and formatting checks pass
- [ ] Type checking passes
- [x] New tests added for the changed deployment and capture behavior
- [ ] Manual testing performed

### Test Output

```text
uv run python -m pytest tests/test_proxy_hardening.py tests/test_install/test_native_installers.py tests/test_docker_compose_persistence.py tests/test_network_diff_capture.py tests/test_rust_core_smoke.py -q
76 passed, 1 skipped, 1 warning in 23.45s

uv run ruff check tests/test_install/test_native_installers.py tests/test_install/test_runtime.py
All checks passed!

bash -n e2e/docker-bind-security.sh
passed

bash e2e/docker-bind-security.sh --base-image headroom-t3283-base --head-image headroom-t3283-head --fixture tests/fixtures/docker-bind-exposure.json --base-ref origin/main
base_ref=origin/main ... runtime_slim_config=loopback rust_core=loaded local_completion=200 attacker_loopback=000 ws_status=403
docker bind security: fixture forwarding, loopback isolation, built-image health, slim-image loopback config, and token controls passed
```

## Real Behavior Proof

- Environment: Windows, Docker Desktop 29.6.2, current `origin/main`
base, head image, and a real Docker bridge peer.
- Exact command / steps: run the focused security/runtime tests, Compose
configuration checks, then `bash e2e/docker-bind-security.sh` against
the current base and head image definitions.
- Observed result: `76 passed, 1 skipped`; the built-image harness
passed base/head health, local completion, non-loopback isolation,
authenticated public HTTP/WebSocket controls, loopback runtime-slim
configuration, and direct Rust-core loading.
- Not tested: runtime-slim proxy service health on this host, because
its unrelated ONNXRuntime import path exits with code 139; the full
hosted matrix and a future published GHCR image remain CI-owned.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Unsafe override required: no
- Minimum rollout channel: next release containing the Docker and
launcher changes
- Stable/default behavior changed: published images and maintained local
Docker launchers now use loopback-safe publication defaults
- Kill switch / disable path: revert the deployment-artifact changes
before release; deliberate public deployments remain explicitly
operator-controlled
- Qualification impact: Docker users relying on all-interface default
publication must opt into public publication and configure the existing
proxy token
- Deliberate public deployment: use an explicit public host publication,
set `HEADROOM_PROXY_TOKEN`, and send the matching bearer or
`X-Headroom-Proxy-Token` credential
- Local deployment: use the maintained loopback publication and no
mandatory token
- Rollback path: revert the deployment-artifact and documentation
changes together

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove the behavior is effective
- [x] Focused tests pass locally with these changes
- [x] I did **not** edit `CHANGELOG.md`; release-please generates
release notes from the conventional PR title

## Additional Notes

The issue fixture remains provenance for the unauthenticated
publication. Native
`headroom proxy` defaults and the application-level authentication
owners are
unchanged.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-24 23:58:46 -05:00
Ayush Kumar Jha 35b11564c6 fix(mcp): report a plugin-provided Serena that reconcile cannot see (#3578)
## Description

**Problem.** With the `serena` plugin from the `claude-plugins-official`
marketplace installed, Claude Code runs **two** Serena MCP servers: the
`mcpServers` entry Headroom manages and the plugin's own
(`plugin:serena:serena` in `claude mcp list`). `headroom mcp reconcile`
reports the Headroom entry as consistent and never mentions the second
one. The plugin's spec is the worse of the two — `uvx --from
git+https://github.com/oraios/serena serena start-mcp-server` with no
`--project-from-cwd`, no `--context claude-code`, and no
`--open-web-dashboard False` (so the #1003 popup returns), tracking
`main` instead of a PyPI release.

**Cause.** `ClaudeRegistrar.get_server` only reads the `mcpServers`
object in `~/.claude.json` / `~/.claude/mcp.json`. A plugin-provided
server is declared in the plugin's own `.mcp.json` under
`<claude_dir>/plugins/cache/...`, indexed by
`<claude_dir>/plugins/installed_plugins.json` and gated by
`enabledPlugins` in Claude's settings files. Nothing in Headroom reads
any of that, so `reconcile` structurally cannot see the duplicate.

**Approach.** Add a read-only
`ClaudeRegistrar.get_plugin_servers(server_name)` that walks
`installed_plugins.json` (v2 shape) and reads each plugin's `.mcp.json`
in both shapes the official marketplace uses (flat `{name: entry}` and
`{"mcpServers": {...}}`). Whether Claude will actually launch a plugin
is resolved the way Claude Code itself does it — verified probe by probe
against Claude Code 2.1.238 with real `claude plugin install/disable`
runs:

- `enabledPlugins` is merged from the user `settings.json`, then
`<cwd>/.claude/settings.json`, then `<cwd>/.claude/settings.local.json`;
the most specific scope wins.
- The plugin must be explicitly `true`; an installed plugin with no
entry anywhere is **not** launched.
- The install record's `scope` / `projectPath` do not gate anything: a
subdirectory with its own `.claude` settings launches it, one without
does not.

So a `--scope project` install is reported inside its project and silent
elsewhere, an installed-but-never-enabled plugin does not warn, and
`claude plugin disable <id>` (which writes `false` to whichever of those
three files matches the scope and keeps the install record) reliably
clears the warning. `headroom mcp reconcile` prints the plugin entry and
that disable command. Per the issue, **nothing is removed** — Headroom
does not mutate third-party plugins. Plugin state and the user
`settings.json` follow `CLAUDE_CONFIG_DIR` the same way the modern
config already does in this registrar.

**Alternatives considered.**
- Read the plugin files directly in `cli/mcp.py` next to
`find_headroom_registration` — shorter, but that module's path constants
ignore `CLAUDE_CONFIG_DIR`, so it would duplicate (and get wrong) the
resolution the registrar already has.
- Fold the plugin entry into `get_server` / `register_server` `MISMATCH`
— would change `headroom wrap` behaviour for every user and could steer
`--adopt`/force paths at a plugin; the issue explicitly asks for no
automatic removal.
- Gate project-scoped records by comparing the resolved cwd against
`projectPath` (exact or containment). Measured against Claude: it is
neither necessary nor sufficient — `proj/sub` does *not* launch a plugin
installed at `proj` unless `proj/sub` has its own `.claude` settings,
and a project-level `false` silences a *user-scope* install. Reading the
three settings files is both simpler and exact.
- Also warn from `mcp status`, `doctor`, or at wrap time — left for
follow-ups to keep this diff to the command the issue reproduces with
(`reconcile` is the Serena command; `status` reports Headroom's own
server).

Closes #3570

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/mcp_registry/claude.py`: new
`ClaudeRegistrar.get_plugin_servers(server_name) ->
list[tuple[plugin_id, ServerSpec]]`, `_enabled_plugins(project_dir,
user_settings)` (merges the three `enabledPlugins` sources in Claude's
precedence) and `_read_plugin_mcp_entry` (both `.mcp.json` shapes).
Resolves the user `settings.json` / `plugins/installed_plugins.json`
under `CLAUDE_CONFIG_DIR` when set, else `~/.claude`. Reuses the
existing read-only `_read_json` (any unreadable/malformed file → no
match). No change to `get_server`, `register_server`, or `wrap`.
- `headroom/cli/mcp.py`: `mcp reconcile` (inspect path) prints `plugin:
<id> also provides a Serena MCP server (<command>)` and `action: ...
claude plugin disable <id>` for each plugin bundling Serena that Claude
would launch from the current directory.
- `tests/fixtures/headroom-issue-3570.json`: the exact on-disk shapes
Claude Code 2.1.238 wrote for `claude plugin install
serena@claude-plugins-official` at user and project scope
(`installed_plugins.json` v2 with per-scope record list incl.
`projectPath`, `enabledPlugins`, the plugin's flat `.mcp.json`) — this
layout is not in the Claude Code docs.
- `tests/test_mcp_registry/test_claude_registrar.py`: 19 tests / 43
cases for `get_plugin_servers` — flat and `mcpServers` shapes; disabled
plugin skipped; absent / `{}` / `null` / unparseable `enabledPlugins` →
not launched; project-scope install reported inside its project and
quiet in `$HOME`, an unrelated dir and a subdirectory; project `false`
silences a user install; project `true` overrides user `false`;
`settings.local.json` overrides `settings.json` both ways; one report
per plugin; multiple plugins in registry order; name filter; explicit
`config_dir` and `CLAUDE_CONFIG_DIR` env; 9 malformed-registry inputs; 6
malformed `.mcp.json` inputs; missing install dir. Every test pins cwd
to an isolated directory.
- `tests/test_cli/test_mcp_reconcile.py`: regression
`test_issue_fixture_plugin_serena_is_reported_base_fail_head_pass`
(fails on `main`, passes here; also asserts the Claude-owned plugin
files keep identical bytes and mtimes), quiet-once-disabled, `--adopt`
leaves plugin files untouched, and
`test_reconcile_reports_project_scoped_plugin_only_inside_its_project`
(in-project positive, `$HOME` and subdirectory negatives, remedy clears
it).
- `docs/content/docs/mcp.mdx`: one sentence on the new warning.
`CHANGELOG.md` not touched.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# Regression test on unpatched main (aa4739e5) — source stashed, tests kept
$ pytest tests/test_cli/test_mcp_reconcile.py::test_issue_fixture_plugin_serena_is_reported_base_fail_head_pass -q
E   AssertionError: assert 'plugin: serena@claude-plugins-official also provides a Serena MCP server (uvx --from git+https://github.com/oraios/serena serena start-mcp-server)' in 'Serena reconciliation for Claude\n  observed: present\n  recommendation: uvx --from serena-agent serena start-mcp-server --project-from-cwd --context claude-code --open-web-dashboard False\n'
FAILED tests/test_cli/test_mcp_reconcile.py::test_issue_fixture_plugin_serena_is_reported_base_fail_head_pass
1 failed in 0.91s

# Second commit (d07ba880) — its 10 new scope/precedence cases fail against the first commit's source (a81f3dc4)
$ pytest tests/test_cli/test_mcp_reconcile.py tests/test_mcp_registry/test_claude_registrar.py -k plugin -q
FAILED ...::test_reconcile_reports_project_scoped_plugin_only_inside_its_project
FAILED ...::test_get_plugin_servers_requires_an_explicit_enable[None|{}|{"enabledPlugins": null}|{"enabledPlugins": {}}|not json]
FAILED ...::test_get_plugin_servers_project_install_reports_inside_its_project
FAILED ...::test_get_plugin_servers_project_disable_silences_user_install
FAILED ...::test_get_plugin_servers_project_enable_overrides_user_disable
FAILED ...::test_get_plugin_servers_local_settings_override_project[False]
10 failed, 30 passed, 69 deselected in 1.53s

# Patched (d07ba880)
$ pytest tests/test_cli/test_mcp_reconcile.py tests/test_mcp_registry/ tests/test_cli/test_mcp.py tests/test_cli/test_mcp_status.py tests/test_cli/test_serena_reconcile.py tests/test_cli/test_serena_migrate.py tests/test_cli/test_serena_disable.py -q
228 passed in 9.08s

$ ruff check .
All checks passed!
$ ruff format --check .
1580 files already formatted
$ mypy headroom --ignore-missing-imports
Success: no issues found in 531 source files

$ coverage report --include=headroom/mcp_registry/claude.py,headroom/cli/mcp.py --show-missing
headroom\mcp_registry\claude.py     243     20    102     14    90%   73, 136-137, 147, 184, 217, 249-250, 258-259, 266-267, 275-276, 315->317, 398, 417->419, 420, 429, 432->434, 456, 458, 459->461, 462
# none of the missing lines fall in the changed ranges (mcp.py 263-274; claude.py 67-71, 96-126, 340-376) => changed lines 100% covered

# Full suite, Windows 11 / Python 3.11.9, first commit (deselecting the pre-existing Windows-only failures in
# test_recover_codex.py, test_unwrap_claude.py, test_wrap_stale_marker.py::...SIGHUP, and files that
# need the built Rust core / langchain_core, none of which this PR touches)
$ pytest tests -q   # a81f3dc4
197 failed, 10927 passed, 792 skipped, 46 deselected, 19 warnings, 192 errors in 3216.17s (0:53:36)
#   0 of the 389 failing IDs are in tests/test_cli/test_mcp*.py, test_serena*.py or tests/test_mcp_registry/.
#   364 of them fail identically on unpatched upstream/main (aa4739e5) in a clean worktree with the same
#   local env (52 affected files: 182 failed, 741 passed, 183 errors) - missing sentence-transformers /
#   playwright / matching Rust core, i.e. environment, not this change.
#   The other 24 (dashboard playwright, test_proxy_health, test_read_maturation_handler_nobust,
#   test_savings_ledger_offload) pass on this branch when their 8 files run alone: 52 passed in 59.13s.
# CI on a81f3dc4: test (1)..(4) shards, lint (ruff + mypy), template, commitlint all green; codecov:
#   "All modified and coverable lines are covered by tests." CI on d07ba880: see the checks on this PR.
```

## Real Behavior Proof

- Environment: Windows 11 Home 10.0.26200, Python 3.11.9, Claude Code
CLI 2.1.238, checkout at d07ba880 run via `PYTHONPATH=$(pwd) python -m
headroom.cli`, isolated `CLAUDE_CONFIG_DIR` (no real config touched)
- Exact command / steps: `claude plugin marketplace add
anthropics/claude-plugins-official` → `claude plugin install
serena@claude-plugins-official` → `claude mcp add serena -s user -- uvx
--from serena-agent serena start-mcp-server --project-from-cwd --context
claude-code --open-web-dashboard False` (Headroom's spec) → `claude mcp
list | grep -i serena` → `headroom mcp reconcile` on unpatched main → on
this branch → `claude plugin disable serena@claude-plugins-official`
(the printed remedy) → `headroom mcp reconcile` again. Then, to derive
the launch rule, each row of the matrix below was set up by editing the
three settings files / reinstalling at `--scope project|local`, and
checked with both `claude mcp list | grep -ic serena` (what Claude does)
and `headroom mcp reconcile | grep -c "plugin: serena@"` (what this
branch reports), from `proj`, `proj/sub` and outside.
- Observed result: `claude mcp list` shows two lines,
`plugin:serena:serena: uvx --from git+https://github.com/oraios/serena
serena start-mcp-server` and `serena: uvx --from serena-agent ...`
(issue step 3 confirmed). Unpatched main printed only `observed:
present` + `recommendation: ...` (issue step 4 confirmed). Patched
printed the same two lines plus `plugin: serena@claude-plugins-official
also provides a Serena MCP server (uvx --from
git+https://github.com/oraios/serena serena start-mcp-server)` and
`action: disable it so only one Serena runs: claude plugin disable
serena@claude-plugins-official`, exit 0. After the disable command the
warning disappeared and `claude mcp list` showed one Serena. Launch-rule
matrix, Claude vs this branch (identical in every row): user `true` →
1/1 in and out of proj; no `enabledPlugins` entry anywhere → 0/0; user
`true` + proj `false` → 0/0 in proj, 1/1 outside; proj `true` + local
`false` → 0/0; user `false` + proj `true` → 1/1 in proj, 0/0 outside;
`proj/sub` without own settings → 0/0; `proj/sub` with its own
`.claude/settings.json` `true` → 1/1. `--scope project` install: `claude
mcp list` 1 inside / 0 outside, this branch warns inside / silent
outside; `claude plugin disable serena@claude-plugins-official` run
inside wrote `false` to `proj/.claude/settings.json` (user file
untouched) and both Claude and this branch went to 0. `--scope local`
behaves the same via `.claude/settings.local.json`.
`installed_plugins.json` records for user and project scope were
captured verbatim into the fixture.
- Not tested: macOS/Linux paths (only via CI); a plugin that inlines
`mcpServers` in `.claude-plugin/plugin.json` instead of `.mcp.json` (0
of 39 official-marketplace manifests do; not read); enterprise managed
settings (`managed-settings.json`) as a fourth `enabledPlugins` source
(not consulted); Claude Code releases other than 2.1.238 (an older
`installed_plugins.json` shape, if it differs, yields no warning rather
than an error).

## Runtime Rollout Safety

- Rollout-managed feature(s): none — this is a read-only CLI diagnostic
in `headroom mcp reconcile`, not a proxy/runtime path
- Minimum rollout channel: n/a
- Stable/default behavior changed: `headroom mcp reconcile` (inspect
path) prints up to two extra lines per plugin bundling a `serena` server
that Claude would launch from the current directory; no change when no
such plugin is installed or it is not enabled there. `--adopt`, `mcp
install`, `mcp status`, `wrap`, `get_server` and `register_server` are
unchanged.
- Kill switch / disable path: `claude plugin disable <id>` (the printed
remedy) or `claude plugin uninstall <id>` removes the lines; no Headroom
flag needed
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commits; no persisted state, files, or
config are written by this change

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — terminal output is quoted verbatim above.

## Additional Notes

- Second commit `d07ba880` (after the first approval) replaces the
"user-scope records only" simplification with Claude's actual
per-directory `enabledPlugins` resolution, per the project-scope point
raised on #3576. It is a strict superset: every previously covered case
behaves the same, plus project/local-scope installs are reported where
Claude launches them, and an installed-but-never-enabled plugin no
longer produces a false positive.
- Deliberately out of scope, happy to follow up if wanted: the same
warning in `mcp status` / `doctor` / at `wrap` time; managed settings as
an `enabledPlugins` source; the inline `plugin.json` `mcpServers` form.
- `--adopt` intentionally keeps its one-line confirmation; the warning
is on the inspect path the issue reproduces with. If you would rather
have it on both, it is a two-line change.
- The detector fails open: any missing or malformed plugin or settings
file yields no warning rather than an error, matching how `get_server`
treats `~/.claude.json`.
2026-09-15 21:20:06 -05:00
Rod Boev 7550efb68f fix(mcp): add explicit Serena reconciliation (#3222)
## Description

Headroom repeatedly warns about user-managed Serena drift but has no
scoped remediation command. Add a Claude-only read-only mcp reconcile
command with explicit --adopt consent, using the canonical Serena spec
and existing Claude registrar. Adoption validates every relevant ledger
and Claude config root before mutation, writes only the Serena entry,
and records ownership after the config write succeeds. Automatic wrap
migration and ordinary install remain unchanged. Closes #3054

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (feature that would cause existing behavior to
change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add Claude-only `headroom mcp reconcile`, read-only by default, with
`--adopt` as its only mutation action.
- Reuse the shared `CLAUDE_SERENA_CONTEXT` and canonical Claude Serena
spec builder.
- Fail closed on malformed or unreadable ledger/config state before
adoption.
- Preserve automatic wrap recovery, user-managed warnings, ordinary `mcp
install --force`, unrelated Claude config, and corrupt-ledger tolerance
outside explicit adoption.
- Record Headroom ownership only after a successful registrar write.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_cli/test_mcp_reconcile.py
tests/test_cli/test_serena_reconcile.py
tests/test_mcp_registry/test_ledger.py`)
- [x] Linting passes (`uv run ruff check .`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed through the file-backed Claude registrar

### Test Output

```text
uv run pytest tests/test_cli/test_mcp_reconcile.py tests/test_mcp_registry/test_ledger.py tests/test_cli/test_serena_reconcile.py tests/test_mcp_registry/test_claude_registrar.py tests/test_mcp_registry/test_install.py -q
102 passed in 0.70s
uv run ruff check headroom/mcp_registry/ledger.py headroom/cli/wrap.py tests/test_mcp_registry/test_ledger.py tests/test_cli/test_serena_reconcile.py tests/test_cli/test_mcp_reconcile.py
All checks passed!
uv run ruff format --check headroom/mcp_registry/ledger.py headroom/cli/wrap.py tests/test_mcp_registry/test_ledger.py tests/test_cli/test_serena_reconcile.py tests/test_cli/test_mcp_reconcile.py
5 files already formatted
git diff --check
```

## Real Behavior Proof

- Environment: Windows, file-backed Claude configuration and isolated
MCP ledger.
- Exact command / steps: run the stale user-managed Serena fixture from
`tests/fixtures/headroom-issue-3054.json`; run read-only reconcile; run
`mcp reconcile --adopt`; rerun wrap and ordinary `mcp install --force`;
exercise malformed JSON, non-dict `mcpServers`, null ledger agents, and
unreadable-ledger adoption.
- Observed result: read-only reconciliation leaves config and ledger
bytes unchanged; adoption updates only Claude Serena and records
ownership after a successful write; automatic wrap remains lenient;
unsafe adoption inputs leave all files unchanged; ordinary install does
not adopt Serena.
- Not tested: live Claude CLI acceptance and Serena stdio handshake

## Runtime Rollout Safety

- Rollout-managed feature(s): None; explicit `mcp reconcile --adopt` is
the only mutation path.
- Minimum rollout channel: Stable; no staged rollout mechanism exists
for this command.
- Stable/default behavior changed: No, read-only reconcile is the
default and automatic wrap plus ordinary install remain unchanged.
- Kill switch / disable path: Do not invoke `--adopt` or revert the
release commit.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert the release commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The changelog is generated by the release pipeline. This change is
limited to Claude Serena reconciliation and does not add a new
persistent acknowledgement state or a multi-provider adoption route.
2026-08-23 11:52:50 -07:00
3077ac81e8 feat: add deterministic runtime rollout controls (#1490)
## Description

Establish one centrally resolved, observable, deterministic, versioned
runtime rollout-control mechanism for Headroom. Runtime rollout controls
which behaviors an already-built artifact may expose; it does not select
or qualify a Headroom release/version.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Bug fix (non-breaking change that fixes rollout enforcement
regressions)
- [x] Documentation update
- [x] Code refactoring (no functional changes)

## Changes Made

- Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`,
`--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared
by Python configuration boundaries.
- Added schema/policy versions, canonical registry and snapshot SHA-256
identities, per-feature decision reasons, disable precedence, unsafe
qualification poisoning, strict CLI validation, and fail-closed
environment handling.
- Added `headroom rollout status --json`, Python `/stats.rollout`, and
Rust `/rollout/status` runtime provenance.
- Added equivalent Rust snapshot semantics and shared Python/Rust policy
vectors while retaining language-specific feature registries.
- Enforced rollout policy at alternate Python server composition roots
so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate.
- Preserved typed rollout snapshots across multi-worker serialization
with schema, policy, registry, snapshot-digest, type, and feature-name
validation.
- Made loopback runtime output-shaper updates replace the immutable
snapshot atomically for request readers, retain explicit request/disable
provenance, preserve channel and kill-switch precedence, invalidate
cached stats, and return the effective rollout decision.
- Made `headroom learn --verbosity --apply` report a channel-blocked
update instead of claiming the shaper is live.
- Made explicit CLI feature flags fail loudly when their current channel
blocks them.
- Made persistent interceptor installation select canary automatically,
or reject an explicitly insufficient channel unless the break-glass
override is set.
- Updated architecture, proxy, rollout, learn, and output-shaper
documentation with required channels and hot-reload semantics.

## Testing

- [x] Unit tests pass
- [x] Linting passes (`ruff check .` and `ruff format --check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New regression tests added for every corrected behavior
- [x] Rust tests and production-target Clippy pass
- [x] Documentation build passes

### Test Output

```text
Focused rollout coverage suite
57 passed; headroom.rollout + rollout CLI: 98% coverage

Affected proxy/rollout/transform/governance suites
222 passed; 0 failed

Final changed regression suites
100 passed; 0 failed

Cross-module hot-reload isolation regression
6 passed; 0 failed

cargo test -p headroom-core -p headroom-proxy --quiet
headroom-core: 924 passed; 1 ignored
headroom-proxy and integration suites: all passed

cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings
cargo fmt --all -- --check
ruff check .
ruff format --check .
mypy headroom --ignore-missing-imports
git diff --check
All passed

cd docs && npm run build
Compiled successfully; 164 static pages generated
```

The unsharded Windows-only CI selection exposed unrelated baseline
failures, principally the existing `sqlite:///C:\\...` URL parser
producing an invalid `\\C:\\...` path. At commit `8e793a80`, all 52
completed GitHub checks passed; the only other conclusions are expected
skips and superseded governance jobs.

## Real Behavior Proof

- **Environment:** Windows checkout on Python 3.13.3 and the current
Rust workspace, based on upstream `main` at `93f2d7a2`.
- **Exact command / steps:** Exercised canary and beta feature requests
through CLI status, Python `/stats.rollout`, Rust `/rollout/status`,
multi-worker payload round trips, loopback `/admin/runtime-env`, real
proxy request shaping before/after hot reload, installer manifest
generation, and shared Python/Rust policy vectors.
- **Observed result:** Stable blocks unstable requests; disable wins
over explicit/default/legacy/unsafe paths; unsafe state reports
`qualification_eligible=false`; worker handoff rejects tampering;
running output shaping changes only when the effective beta policy
permits it; explicit blocked flags fail with actionable diagnostics.
- **Not tested:** Live production traffic requiring provider
credentials, or future artifact qualification/promotion automation
(intentionally out of scope).

## Runtime Rollout Safety

- **Rollout-managed features:** Python `tool_result_interceptors`,
`proxy_output_shaper`, `read_maturation`; Rust `native_bedrock`,
`openai_responses_streaming`, `canary_probe`.
- **Minimum rollout channel:** Registry-defined per feature; process
default is `stable`.
- **Stable/default behavior changed:** No unstable feature becomes
enabled by default. Explicit blocked CLI flags now fail instead of
silently doing nothing.
- **Kill switch / disable path:**
`HEADROOM_DISABLE_FEATURES=<comma-separated feature names>`; explicit
disable has highest precedence, including over the unsafe override.
- **Unsafe override required:** No.
`HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES=1` is break-glass only and
makes qualification evidence ineligible.
- **Qualification impact:** Adds machine-readable policy/snapshot
identities and eligibility; does not implement qualification itself.
- **Rollback path:** Set the named disable list for operational
rollback, lower the channel, or revert this PR.

## Review Readiness

- [x] I have performed a full diff review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented hard-to-understand areas
- [x] I have made corresponding documentation changes
- [x] My changes generate no new warnings
- [x] I added tests that reproduce and prevent every regression fixed
during review
- [x] New and existing affected tests pass locally
- [x] I did **not** edit `CHANGELOG.md`; release-please generates it
from the Conventional Commit PR title

## Additional Notes

Out of scope: artifact candidates, benchmark orchestration,
qualification manifests/gates, promotion automation, release branches,
publication guards, and release-risk classification. Those workflows can
consume the rollout registry digest, runtime snapshot digest, decision
reasons, and qualification eligibility through supported black-box
interfaces.

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JD Davis <jd@JDH-AIR-00.local>
2026-08-12 23:16:54 -05:00
Rod BoevandTejas Chopra 38479fcda1 fix(codex): rerun memory lookup on every response.create WS frame (#2113)
## Description

Long-lived Codex `/v1/responses` WebSocket sessions only ran memory
decision, query construction, and context injection on the first
`response.create` frame because `handle_openai_responses_ws` kept that
logic outside the relay loop. This change extracts that path into a
local helper reused before compression for every eligible
`response.create`, while keeping sticky memory tools deduplicated by the
existing session helper.

Closes #2059

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Extracted the first-frame memory preparation path into a local async
helper inside `handle_openai_responses_ws`.
- Reused that helper for the initial frame and every later eligible
`response.create` before compression and shaping.
- Added focused two-turn WebSocket regressions for per-frame lookup,
bypass and disabled-memory handling, list-shaped later inputs,
memory-handler fail-open recovery, and sticky-tool replay.
- Raised the locked production floors for `click` and `pillow` to clear
the current `pip-audit` findings that now fail external PR merge
snapshots.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_codex_ws_per_frame_memory.py
tests/test_openai_codex_ws_lifecycle.py -q`)
- [x] Linting passes (`uv run ruff check
headroom/proxy/handlers/openai.py
tests/test_codex_ws_per_frame_memory.py
tests/test_openai_codex_ws_lifecycle.py`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run pytest tests/test_codex_ws_per_frame_memory.py tests/test_openai_codex_ws_lifecycle.py -q

============================= test session starts =============================
platform win32 -- Python 3.12.13, pytest-9.0.3, pluggy-1.6.0
rootdir: D:\Repos\headroom-pr-2059-codex-ws-memory-lookup
configfile: pyproject.toml
plugins: anyio-4.12.1, langsmith-0.9.3, asyncio-1.3.0, cov-7.0.0
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 35 items

tests\test_codex_ws_per_frame_memory.py .........                        [ 25%]
tests\test_openai_codex_ws_lifecycle.py ..........................       [100%]

============================= 35 passed in 1.89s ==============================

uv run ruff check headroom/proxy/handlers/openai.py tests/test_codex_ws_per_frame_memory.py tests/test_openai_codex_ws_lifecycle.py
All checks passed!

uv run ruff format headroom/proxy/handlers/openai.py tests/test_codex_ws_per_frame_memory.py tests/test_openai_codex_ws_lifecycle.py --check
3 files already formatted
```

## Real Behavior Proof

- Environment: Windows, Python 3.12.13, in-process WebSocket harness
with a fake memory handler
- Exact command / steps: `uv run pytest
tests/test_codex_ws_per_frame_memory.py
tests/test_openai_codex_ws_lifecycle.py -q`, which opens one connection,
sends distinct `response.create` frames, and records each memory lookup
plus the forwarded tool set
- Observed result: the handler performs one lookup per eligible frame,
later-frame compression sees current-turn memory-prepared input, and
both forwarded turns carry the same deduplicated `memory_search` and
`memory_save` definitions
- Not tested: live Codex subscription WebSocket

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Security CI currently flags `click 8.3.1` and `pillow 12.2.0`, so this
PR carries the narrow floor bump to `click>=8.3.3` and `pillow>=12.3.0`
as a supply-chain unblock for the same final merge snapshot.
- `CHANGELOG.md` remains untouched because Headroom generates release
notes from conventional commits.
- This PR is scoped to the OpenAI Responses WebSocket relay lifecycle;
it does not change the HTTP `/v1/responses` path or the Anthropic
handler.

---------

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-07-13 14:42:27 -04:00
AshishandClaude Opus 4.8 23d73ae070 test(evals): add offline fidelity regression gate (recall-based, zero-model) (#1187)
## Description

Headroom's lossy compression drops rows/lines using statistical
heuristics but **never checks that meaning survived** — a dropped `OOM
killed worker 3` line can silently flip a model's answer with no signal
that compression caused it. The repo already ships a quality-metric
toolkit (`headroom/evals/metrics.py`) and a `weekly-suite` eval job, but
neither gates the compression path on a PR.

This adds a **per-PR fidelity regression gate**: compress vendored
golden tool-outputs through SmartCrusher's lossy path and assert the
evidence that answers each case's question survives. It is the first of
a planned trio (this is the "offline gate" half of the fidelity work);
query-aware retention and a hard token-budget API are documented
follow-ups.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Blocking gate** (`tests/test_compression_fidelity_regression.py`):
compresses each golden case via `smart_crush_tool_output(...,
with_compaction=False)` and scores with `compute_information_recall`.
Two assertions:
- **Per-case critical recall == 1.0** — every `answer_evidence` string
(placed in error/anomaly rows, the documented SmartCrusher retention
guarantee) must survive.
- **Aggregate recall ≥ committed baseline** (`baseline.json`, tol 0.02)
— catches softer regressions.
- **Vendored fixtures** (`tests/fixtures/fidelity_golden/`):
deterministic `_generate.py` emits `cases.json` (4 cases: OOM crash,
payment exception, latency anomaly, CI failure) + `baseline.json`.
- **Non-blocking weekly report** (`.github/workflows/eval.yml`): one
step in the existing `weekly-suite` job (schedule/manual only) reuses
the existing `evaluate_information_retention` runner for a recall report
on the production routing path.
- **Pure reuse**: scoring (`evals/metrics.py`), compressor
(`smart_crush_tool_output`), and the weekly runner
(`evaluate_information_retention`) all already existed.

### Design notes

- **Zero new CI setup.** The blocking gate runs in the existing `[dev]`
test shard — no new workflow, no new deps, **no model, no network, no
secrets** (verified under `HF_HUB_OFFLINE=1`). It deliberately uses
small hand-made structured fixtures rather than the repo's HuggingFace
dataset loaders, which would require a network download + ModernBERT and
don't belong in a fast PR gate.
- **Scope:** structured JSON tool-output (the dominant, deterministic,
model-free case). Real-dataset (HotpotQA/BFCL) recall — which needs
`[all]` + a local model — is a **documented follow-up PR**, and the
`weekly-suite` job (which genuinely runs every Monday) is its natural
home.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ HF_HUB_OFFLINE=1 python -m pytest tests/test_compression_fidelity_regression.py -v
tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[logs_oom] PASSED
tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[payment_exception] PASSED
tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[latency_anomaly] PASSED
tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[ci_test_failures] PASSED
tests/test_compression_fidelity_regression.py::test_aggregate_recall_not_regressed PASSED
============================== 5 passed in 0.18s ===============================
```

## Real Behavior Proof

- **Environment:** local checkout of `feat/fidelity-regression-gate`,
`pip install -e ".[dev]"`, `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1`
(proves no model/network).
- **Exact command / steps:** `HF_HUB_OFFLINE=1 python -m pytest
tests/test_compression_fidelity_regression.py -q` → `5 passed in 0.14s`.
- **Negative control (proves the gate has teeth):** compressing
`logs_oom` and probing for a benign row that compression legitimately
drops returns `recall = 0.00, lost = ['heartbeat ping 25']` — i.e. the
gate fires when critical evidence is dropped, so it is not trivially
green.
- **Weekly (non-blocking) step verified locally:**
  ```text
  Information retention: 50/50 cases >=0.9 recall, avg compression 65.7%
  ```
- **Not tested:** real-dataset (HotpotQA/BFCL) recall and
prose/ModernBERT compression — intentionally deferred to a follow-up PR
targeting the weekly job.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- CHANGELOG/version intentionally untouched: repo uses
**release-please**.
- **Follow-up PR (planned):** wire the real HotpotQA/BFCL loaders
(`headroom/evals/datasets.py`) into the `weekly-suite` job for genuine
benchmark-scale recall coverage (model-allowed, non-blocking). Further
follow-ups from the same design: a live per-request fidelity guardrail,
query-aware lossy retention, and a hard `target_tokens` budget API.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 22:53:59 -05:00
chopratejas c62d45eea8 fix(memory): expose memory IDs in auto-tail + memory_list tool + ID-usage guidance
Pre-this-PR the auto-injected memory block rendered rows as `1. <content>`
with no addressable handle. To UPDATE or DELETE a row the model first had
to call memory_search to discover its ID — two round trips, against the
model-as-judge architecture.

This PR adds three tightly-coupled affordances so the model can act on
memory directly:

1. Auto-tail rows now carry the memory ID:
     `1. [mem_alpha_001] User prefers Python`
   The bracketed token is the canonical ID — same identifier accepted by
   memory_update and memory_delete.

2. New `memory_list` tool — chronological browse (vs `memory_search`'s
   semantic lookup). Returns recent memories with their IDs. Backend
   dispatches to `Backend.list_memories` if available, else falls back
   to an empty-query `search_memories`. Caps at 100 entries.

3. ID-usage guidance text appended to the auto-tail block. Tells the
   model that bracketed IDs can go straight to memory_update /
   memory_delete with no intervening search. The guidance lives in the
   user-message tail (never system) — preserves cache-prefix byte
   stability (invariant I2).

`memory_update` and `memory_delete` tool descriptions also point at the
[id] block as a valid ID source — keeps tool docs consistent with the
new affordance.

Verification:
- 10/10 tests pass in tests/test_memory_auto_tail.py (incl. 2 new
  guidance tests + 2 new ID-format tests)
- 31/31 tests pass in tests/test_memory_handler_native_ops.py (incl. 4
  new memory_list dispatch tests + existing assertions updated for the
  [id] format change)
- Golden fixtures regenerated for the tool-description copy changes
  (tests/fixtures/memory_tool_definitions/{anthropic,openai}.json)
- Live end-to-end test against real Anthropic API
  (tests/test_proxy_memory_integration.py::TestMemoryIdAutoTailAndUpdate):
  seeded memory → auto-tail → Claude → memory_update with exact ID.
  PASSED.
2026-05-19 22:10:42 -05:00
chopratejas 8dcd474aca fix: A7 — memory tool injection session-sticky for both Anthropic and OpenAI
Closes the second half of P0-6: once memory injects memory_save / memory_search
into body["tools"] for a session, every subsequent turn injects the byte-equal
same definitions — even if memory is disabled mid-session. Toggling tool list
mid-session busts Anthropic prefix cache per guide §6.3 #2.

Adds in headroom/proxy/helpers.py:

  * SessionToolTracker — bounded LRU keyed by (provider, session_id) storing
    GOLDEN tool-definition bytes from the first injection. Tracker is
    provider-aware so the same session_id under Anthropic and OpenAI keeps
    independent state. Reentrant lock for concurrent access; LRU eviction at
    HEADROOM_TOOL_TRACKER_MAX_SESSIONS (default 1000).
  * apply_session_sticky_memory_tools — single coordination point with three
    paths: first-time inject (record golden bytes), sticky replay (always
    inject golden bytes regardless of inject_this_turn), and skip. Honors
    HEADROOM_TOOL_INJECTION_STICKY=disabled as a loud operator opt-in for
    rollback (NOT a fallback).
  * serialize_tool_definition_canonical — deterministic byte serialization
    via the same separators=(",",":")/ensure_ascii=False rules as
    serialize_body_canonical.
  * log_tool_injection_decision — structured per-decision log line; never
    logs the tool definition contents.

Wires the helper into all four memory tool injection sites:
  * handlers/anthropic.py — /v1/messages
  * handlers/openai.py — /v1/chat/completions
  * handlers/openai.py — /v1/responses
  * handlers/openai.py — Codex WS path

memory_handler.MemoryHandler gains compute_memory_tool_definitions(provider) —
a pure builder that returns the tool definitions without mutating a tools
list, so the proxy can route through the sticky tracker. The legacy
inject_tools(...) is preserved for callers without a session_id.

Tests: tests/test_memory_tool_session_sticky.py — 29 unit + integration
cases covering: turn-1→turn-2 byte-equality (Anthropic + OpenAI), sticky
replay after memory disabled, golden-fixture pin, LRU eviction, provider
isolation under shared session_id, thread-safe concurrent access, env-var
contract, disabled-mode passthrough, dedupe with client tools.

Golden fixtures pin canonical bytes:
  * tests/fixtures/memory_tool_definitions/anthropic.json
  * tests/fixtures/memory_tool_definitions/openai.json

No regex. No hardcodes (env-configurable: HEADROOM_TOOL_INJECTION_STICKY,
HEADROOM_TOOL_TRACKER_MAX_SESSIONS). No silent fallbacks. Per-decision
structured logging. Realignment build constraints satisfied.
2026-05-02 10:11:27 -07:00