mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
a9815bbb5f |
fix(proxy): record the provider's billed input count on streamed turns (#4052)
## Description Every input-volume figure Headroom reports for streamed Claude traffic was Headroom's own tokenizer estimate, not Anthropic's billed count. That covers "input after Headroom", `/stats` `tokens.input`, the dashboard's per-model table and the licence usage report's `tokens_after`. Since Claude Code always streams, this was effectively all Claude Code traffic, so customers comparing our numbers with the Anthropic Console could not reconcile them. **Root cause.** `emit_request_outcome` records `provider_input_tokens or optimized_tokens`. The streaming finalizer never set `provider_input_tokens` for Anthropic (#3946 fixed only the uncached sub-count), so every streamed Claude turn fell back to `optimized_tokens`. That value is: - an `o200k_base` count standing in for Claude's private tokenizer, measured against `count_tokens` on 6 real Claude Code transcripts at 15–25% under Claude 4.x and 35–38% under Claude 5.x; - a count of the forwarded **messages only**, which never sees the system prompt or tool definitions. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`headroom/proxy/provider_usage.py` (new).** One place that knows each provider's usage semantics: - Anthropic, Bedrock and Vertex: `input_tokens` is the uncached tail, and the billed total is `input + cache_read + cache_write`. - OpenAI `prompt_tokens`/`input_tokens` and Gemini `promptTokenCount` are already inclusive of cached tokens. - **Streaming finalizer.** Sets `provider_input_tokens` from the provider's usage on the Anthropic/OpenAI/Gemini SSE path, the Bedrock stream path and the OpenAI-via-backend stream path. `RequestOutcome.from_stream` gains the parameter. - **OpenAI pass-through.** On both streaming and non-streaming paths, Anthropic-shape usage was read as if `input_tokens` were the total. It now uses the billed total and is marked provider-reported. - **Gemini non-streaming and OpenAI Responses WebSocket.** These already used the provider's count as `optimized_tokens` but never declared it, so they now also set `provider_input_tokens`, but only when the provider actually sent a count. - **No estimate is ever labelled as billed.** The Bedrock #1132 `message_start` backfill and Gemini's fallback for a missing `promptTokenCount` both stay estimated. The pass-through usage dialect is chosen by provider, not inferred from which keys are present. - **Provenance counters.** Every input total now carries how much of it is billed versus estimated: - `/stats` `tokens.input_provider_reported`, `input_estimated`, `input_provider_reported_percent`, `requests_input_*`. - Licence usage report: `tokens_after_provider_reported`, `tokens_after_estimated`, `requests_provider_reported`, plus the billed `cache_read_tokens`, `cache_write_tokens` and `uncached_input_tokens`, so the cloud side can reconcile with the console (input + cache read + cache write). The report also states `tokens_saved_basis: "headroom_tokenizer"`. - A one-time `event=input_tokens_estimated` warning per provider/model when a successful request carried no provider usage. - **No delta math changes.** `optimized_tokens` stays on the local scale, and `tokens_saved`, ratios and the ledger delta are untouched. The provider count is never subtracted from a local count. ## Testing - [ ] Unit tests pass (`pytest`): targeted suites only, listed below. The full suite is left to CI (see Additional Notes). - [x] Linting passes (`ruff check .`): run on all changed files. - [x] Type checking passes (`mypy headroom`): run on the changed modules. - [x] New tests added for new functionality - [x] Manual testing performed `tests/test_provider_billed_input.py` adds 15 tests, plus regression tests for the review fixes in `tests/test_bedrock_streaming_input_tokens.py` (the #1132 backfill is reported as estimated) and `tests/test_gemini_ccr_continuation_usage.py` (a missing `promptTokenCount` is not provider-reported): - **Recorded fixture.** `tests/fixtures/anthropic/stream_cached_turn.sse` is a **real recorded Anthropic stream**: a second call on a cached ~8.9K-token system prompt, billed as 19 uncached + 8,857 cache-read. It is replayed in 97-byte chunks through the proxy's own SSE usage parser and `_finalize_stream_response`, then through `emit_request_outcome` into a real `PrometheusMetrics`, `CostTracker` and `UsageReporter`. The test asserts that `/stats` input and the licence `tokens_after` equal Anthropic's 8,876. - Other cases: - an OpenAI stream records the inclusive `prompt_tokens`; - a stream without usage stays estimated; - each provider shape is covered; - the pass-through normaliser returns the total. Against the unpatched handlers, the 5 behaviour tests fail. The 8 that pass exercise only the new helper. ### Test Output ```text $ pytest tests/test_provider_billed_input.py tests/test_proxy_handler_helpers.py tests/test_request_outcome.py \ tests/test_proxy_streaming_request_logger.py tests/test_proxy_cache_ttl_metrics.py \ tests/test_usage_reporter_snapshot.py tests/test_5xx_accounting_all_providers.py tests/test_copilot_provider_label.py \ tests/test_bedrock_streaming_input_tokens.py tests/test_gemini_ccr_continuation_usage.py -q 181 passed in 6.56s # same new tests against the unpatched handlers FAILED tests/test_provider_billed_input.py::test_passthrough_normaliser_returns_anthropic_total_not_uncached_tail FAILED tests/test_provider_billed_input.py::test_recorded_anthropic_stream_records_anthropics_billed_input FAILED tests/test_provider_billed_input.py::test_openai_stream_records_inclusive_prompt_tokens FAILED tests/test_provider_billed_input.py::test_streamed_claude_turn_reaches_dashboard_and_licence_report_as_billed FAILED tests/test_provider_billed_input.py::test_estimated_input_is_reported_as_estimated 5 failed, 8 passed $ ruff check <changed files> -> All checks passed! $ mypy <changed proxy/telemetry modules> -> Success: no issues found in 6 source files ``` ## Real Behavior Proof - Environment: local proxy from this branch (`headroom proxy --port 8931 --stateless --no-telemetry --no-cache`) against the real Anthropic API. For the baseline, the same command ran from a `main` worktree on port 8932. - Exact command / steps: 3 streamed `claude-sonnet-4-6` turns through the proxy, with a cached ~20K-token system prompt, one tool and a 300-item JSON tool result. Anthropic's `message_start` usage (input + cache read + cache write) was summed from each stream and compared with `GET /stats` → `tokens`. - Observed result: the author-provided live Anthropic comparison below reports exact billed-input agreement at 61,999 tokens; the baseline undercounts by 37%. This external run was not independently repeated during the current-head review. | | Anthropic billed input | Headroom `/stats` `tokens.input` | |---|---|---| | `main` | 62,001 | **38,835** (−37%) | | this PR | 61,999 | **61,999** (exact), `input_provider_reported_percent: 100.0` | The two runs differ by 2 tokens because they are separate API calls. `proxy_compression_saved` was 29,313 on both, so savings behaviour is unchanged. - Not tested: live Bedrock, Vertex and Gemini streams (unit-tested on provider shapes only), and the licence cloud endpoint receiving the new fields. ## Runtime Rollout Safety - Rollout-managed feature(s): none. This is accounting only, and request and response bytes are unchanged. - Minimum rollout channel: N/A - Stable/default behavior changed: reported input volume now uses the provider's billed count where the provider reports one. New fields are added to `/stats` and the licence usage payload; no existing field is removed or renamed. - Kill switch / disable path: none needed. With no usage frame, the fallback is the previous estimate, now labelled `input_estimated`. - Unsafe override required: no - Qualification impact: dashboards and licence reports will show larger, correct Claude input totals once installs upgrade, so historical and new figures are not directly comparable. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes: targeted suites only; full suite on CI - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - **Full suite.** Locally, the full run on the pre-rebase base reached 2,147 passed before stopping on `tests/test_cli/test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`. That test fails identically on unmodified `main` (unrelated MCP install args). The full local run was then abandoned because several wrap/install tests launch a real `serena project index` (rust-analyzer, tsserver, rustc), which should be stubbed in a separate PR. - **Docs.** The new `/stats` and licence fields are not yet documented. - Follow-ups not in this PR: - **Raw Bedrock invoke pass-through (`handlers/bedrock.py`)** forwards bytes without parsing them, so it still records an estimate. It is now counted as `input_estimated`, so it's visible rather than silent. - **Batch create** has no usage until results come back, so it stays estimated as above. - **`tokens_saved` is still on the `o200k_base` scale**, so it under-reports Claude savings. Calibrating it to provider tokens is a separate PR. - **The licence cloud endpoint** needs to read the new fields. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Current maintainer verification (October 8) Exact head: `81527148ed9777e1b9ca921bbf6bb457d49354db`, containing main `01d9194854233ef6ed04005cf0a5b8c32bf65d6d` through a clean merge. No contributor logic was modified. 63 focused tests pass with the existing compatible native extension on Windows/Python 3.12. Three backend-boundary probes preserve zero client input and exact cache-read, cache-write, and combined provider input totals. This is local mocked-backend/TestClient evidence; the author-provided external-provider evidence above was not independently rerun. Current-head hosted snapshot: 31 successful checks, 12 skips, no failed or live checks. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
ef7605fdbb |
fix(plugin): resolve hook CLI through plugin-root launcher (#3053)
## Description
The `headroom` plugin invokes `headroom init hook ensure` through
non-login `/bin/sh`, but both `SessionStart` and `PreToolUse` entries
use a bare executable name. Agent hosts whose PATH omits the default
uv/pipx user bin exit 127 on every matching event. Tool calls continue
because the hook is non-blocking, while Headroom compression remains
inactive and the failure banner repeats.
Both hooks now invoke one launcher bundled under `${CLAUDE_PLUGIN_ROOT}`
through `sh`. The launcher honors `HEADROOM_BIN`, preserves
PATH-resolved installations, covers standard uv and pipx prefixes,
records the Homebrew candidates for macOS, and falls back to an
importable Python module. On the launcher-routed path, an absent
Headroom CLI emits one actionable diagnostic and exits 0. Hosts that do
not expose a readable plugin root retain the existing bare-command tail,
preserving the shared GitHub Copilot CLI manifest behavior.
Reported by [@donnydongchen](https://github.com/donnydongchen), whose
exact non-login `env -i` reproduction, Apple Silicon path detail, and
plugin-root suggestion shaped the fix.
Closes #3039
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Route both plugin hook entries through a shared
`${CLAUDE_PLUGIN_ROOT}`-anchored `sh` launcher while preserving
matchers, arguments, and 60-second timeouts.
- Require a non-empty plugin root before probing the launcher, so unset
or empty roots always reach the existing bare-command tail.
- Resolve `HEADROOM_BIN`, PATH, standard uv/pipx/Homebrew locations, and
an importable `python -m headroom.cli` fallback in a deterministic
order.
- Preserve the current bare command for hosts without a readable plugin
root, including the shared Copilot CLI manifest path.
- On the launcher-routed path, replace the absent-CLI status-127 banner
with one actionable diagnostic and a non-blocking exit.
- Add real-shell regression coverage for the reported environment,
resolution precedence, supported prefixes, Python fallback, and negative
boundaries.
## Testing
- [x] Unit tests pass (`uv run --extra dev python -m pytest
tests/test_plugin_manifests.py tests/test_plugin_hook_launcher.py
tests/test_cli/test_init_cli.py -q`, `98 passed`)
- [x] Reproduction override passes (`HEADROOM_REPRO_ARTIFACT` can select
the issue fixture, `uv run --extra dev python -m pytest
tests/test_plugin_hook_launcher.py::test_manifest_command_recovers_reported_non_login_environment
-q`, `1 passed`)
- [x] Linting passes (Ruff check and format validation on the changed
Python test files)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed
### Test Output
```text
98 passed
All checks passed!
2 files already formatted
```
## Real Behavior Proof
- Environment: POSIX `sh` with the reporter's sanitized non-login PATH
and a recording Headroom executable only at `$HOME/.local/bin/headroom`
- Exact command / steps: `uv run --extra dev python -m pytest
tests/test_plugin_hook_launcher.py::test_manifest_command_recovers_reported_non_login_environment
-q`; the test preserves the reported PATH and error evidence, executes
both manifest commands with a recorder at `$HOME/.local/bin/headroom`,
and asserts both `home-local init hook ensure` receipts.
- Observed result: `1 passed in 0.41s`; both hooks recovered under the
reported non-login PATH and emitted no command-not-found banner.
- Not tested: live Claude Code plugin-hook execution on a macOS tmux
host
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: plugin hooks resolve Headroom through
the bundled launcher when the plugin root is readable; invalid launcher
roots retain the existing bare-command fallback.
- Kill switch / disable path: none; the existing bare-command fallback
remains available when the plugin root is absent or unreadable.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert the implementation commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
## Screenshots (if applicable)
Not applicable; this PR changes plugin hook startup behavior without a
rendered UI.
## Additional Notes
The launcher does not source shell profiles, install packages, access
the network, change deployment/profile selection, or modify plugin
versions.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3053"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
12c157966f |
fix(proxy): preserve Claude Code auto-mode protocol (#3784)
## Description
Fix Claude Code server-side auto-mode compatibility when requests pass
through
Headroom’s native Anthropic Messages endpoint.
Previously, auto-mode capability headers, safeguard payloads, query
parameters,
unknown response fields, or opaque SSE frames could be rewritten or lost
while
proxying and handling CCR continuations. This could trigger Claude
Code’s
client-side classifier fallback and its associated warning.
This change adds a request-local Anthropic wire envelope, preserves
client-owned
capability data and opaque response attachments, prevents classifier
payloads
from entering Headroom-owned observations, and documents that
schema-translating
Bedrock, Vertex, and LiteLLM routes do not currently support the
server-side
auto-mode protocol.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Preserve native Anthropic request paths and raw query parameters.
- Preserve client-owned `anthropic-version`, `anthropic-beta`, and
safeguard data.
- Avoid sticky-beta, memory, or turn-hook rewrites on classifier-bearing
turns.
- Preserve unknown Anthropic response fields across CCR continuations.
- Preserve opaque SSE frames through request-local parse/render
envelopes.
- Reject unknown or malformed deltas during strict buffered
reconstruction.
- Redact safeguard payloads from diagnostics and pipeline observations.
- Consolidate Anthropic SSE parsing and rendering into the wire module.
- Document auto-mode limitations for Bedrock, Vertex, and LiteLLM
translation paths.
- Add request, response, streaming, CCR, malformed-byte, and passthrough
regression tests.
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest -p no:cacheprovider \
tests/test_anthropic_auto_mode_passthrough.py \
tests/test_anthropic_buffered_sse.py \
tests/test_ccr_response_handler_extra.py \
tests/test_ccr_buffered_stream_signed_thinking.py \
tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py \
tests/test_buffered_ccr_salvage.py \
tests/test_buffered_ccr_accept_header.py \
tests/test_proxy/test_anthropic_copilot_upstream_auth.py \
tests/test_proxy_streaming_request_logger.py \
tests/test_proxy_streaming_resilience.py -q
125 passed, 1 warning in 13.70s
$ ruff check --no-cache .
All checks passed!
$ ruff format --check --no-cache .
1559 files already formatted
$ python -m mypy \
headroom/proxy/anthropic_wire.py \
headroom/ccr/response_handler.py \
headroom/proxy/handlers/anthropic.py \
headroom/proxy/handlers/streaming.py
Success: no issues found in 4 source files
$ npx --no-install commitlint \
--from origin/main --to HEAD \
--config .commitlintrc.json
Exited successfully.
```
## Real Behavior Proof
- Environment: Pending. Record OS, Python version, Claude Code version,
provider/model, and Headroom revision.
- Exact command / steps: Pending. Start Headroom with `headroom proxy
--port 8787`, then run Claude Code with
`ANTHROPIC_BASE_URL=http://127.0.0.1:8787` and exercise server-side auto
mode.
- Observed result: Pending. Attach redacted output or logs showing the
request completed through Headroom and whether Claude Code used its
client-side fallback.
- Not tested: Live Claude Code auto mode against Anthropic; Bedrock,
Vertex, and LiteLLM translation routes.
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Native Anthropic safeguard-capable
requests.
- Kill switch / disable path: Set `CLAUDE_CODE_AUTO_MODE_SERVER=0` to
use Claude Code’s client-side fallback.
- Unsafe override required: No.
- Qualification impact: Live qualification is pending. Translation
routes remain documented as unsupported for server-side auto mode.
- Rollback path: Revert this PR or deploy the previous Headroom
revision.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code where needed
- [x] I have made corresponding documentation changes
- [x] No new warnings were observed in focused tests
- [x] I have added tests for the fix
- [ ] Full existing and new unit test suite passes locally
- [x] I did **not** edit `CHANGELOG.md`
## Screenshots (if applicable)
N/A. This change affects proxy wire behavior.
|
||
|
|
b0c19a25f0 |
fix(security): reject unauthenticated public proxy binds (#3316)
## Description Headroom's published image and maintained Docker launchers could expose the proxy on every host interface without a proxy token. This change moves the security boundary to the deployment artifacts: published images default to a container-loopback listener, supported local launchers publish only on host loopback, and deliberate public deployments retain explicit token protection. The application factory remains transport-neutral. Closes #3283 ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Set both published image stages to a container-loopback default while keeping explicit host overrides available. - Changed native Compose and the maintained Bash and PowerShell launchers to use an internal `0.0.0.0` listener with host-loopback publication. - Preserved the top-level Compose token requirement and the existing HTTP and WebSocket credential behavior for deliberate public binds. - Scoped differential-capture token injection to the internal `headroom-proxy` destination and kept sensitive headers redacted in captured records. - Added focused launch-contract, native-core, capture, and Docker publication checks, plus a base/head harness that tolerates a base already containing the fix. - Updated Docker, installer, capture, and mirrored documentation, including an unauthenticated local loopback quickstart and an authenticated deliberate-public example. ## Testing - [x] Focused unit and integration tests pass - [x] Linting and formatting checks pass - [ ] Type checking passes - [x] New tests added for the changed deployment and capture behavior - [ ] Manual testing performed ### Test Output ```text uv run python -m pytest tests/test_proxy_hardening.py tests/test_install/test_native_installers.py tests/test_docker_compose_persistence.py tests/test_network_diff_capture.py tests/test_rust_core_smoke.py -q 76 passed, 1 skipped, 1 warning in 23.45s uv run ruff check tests/test_install/test_native_installers.py tests/test_install/test_runtime.py All checks passed! bash -n e2e/docker-bind-security.sh passed bash e2e/docker-bind-security.sh --base-image headroom-t3283-base --head-image headroom-t3283-head --fixture tests/fixtures/docker-bind-exposure.json --base-ref origin/main base_ref=origin/main ... runtime_slim_config=loopback rust_core=loaded local_completion=200 attacker_loopback=000 ws_status=403 docker bind security: fixture forwarding, loopback isolation, built-image health, slim-image loopback config, and token controls passed ``` ## Real Behavior Proof - Environment: Windows, Docker Desktop 29.6.2, current `origin/main` base, head image, and a real Docker bridge peer. - Exact command / steps: run the focused security/runtime tests, Compose configuration checks, then `bash e2e/docker-bind-security.sh` against the current base and head image definitions. - Observed result: `76 passed, 1 skipped`; the built-image harness passed base/head health, local completion, non-loopback isolation, authenticated public HTTP/WebSocket controls, loopback runtime-slim configuration, and direct Rust-core loading. - Not tested: runtime-slim proxy service health on this host, because its unrelated ONNXRuntime import path exits with code 139; the full hosted matrix and a future published GHCR image remain CI-owned. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Unsafe override required: no - Minimum rollout channel: next release containing the Docker and launcher changes - Stable/default behavior changed: published images and maintained local Docker launchers now use loopback-safe publication defaults - Kill switch / disable path: revert the deployment-artifact changes before release; deliberate public deployments remain explicitly operator-controlled - Qualification impact: Docker users relying on all-interface default publication must opt into public publication and configure the existing proxy token - Deliberate public deployment: use an explicit public host publication, set `HEADROOM_PROXY_TOKEN`, and send the matching bearer or `X-Headroom-Proxy-Token` credential - Local deployment: use the maintained loopback publication and no mandatory token - Rollback path: revert the deployment-artifact and documentation changes together ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove the behavior is effective - [x] Focused tests pass locally with these changes - [x] I did **not** edit `CHANGELOG.md`; release-please generates release notes from the conventional PR title ## Additional Notes The issue fixture remains provenance for the unauthenticated publication. Native `headroom proxy` defaults and the application-level authentication owners are unchanged. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
35b11564c6 |
fix(mcp): report a plugin-provided Serena that reconcile cannot see (#3578)
## Description **Problem.** With the `serena` plugin from the `claude-plugins-official` marketplace installed, Claude Code runs **two** Serena MCP servers: the `mcpServers` entry Headroom manages and the plugin's own (`plugin:serena:serena` in `claude mcp list`). `headroom mcp reconcile` reports the Headroom entry as consistent and never mentions the second one. The plugin's spec is the worse of the two — `uvx --from git+https://github.com/oraios/serena serena start-mcp-server` with no `--project-from-cwd`, no `--context claude-code`, and no `--open-web-dashboard False` (so the #1003 popup returns), tracking `main` instead of a PyPI release. **Cause.** `ClaudeRegistrar.get_server` only reads the `mcpServers` object in `~/.claude.json` / `~/.claude/mcp.json`. A plugin-provided server is declared in the plugin's own `.mcp.json` under `<claude_dir>/plugins/cache/...`, indexed by `<claude_dir>/plugins/installed_plugins.json` and gated by `enabledPlugins` in Claude's settings files. Nothing in Headroom reads any of that, so `reconcile` structurally cannot see the duplicate. **Approach.** Add a read-only `ClaudeRegistrar.get_plugin_servers(server_name)` that walks `installed_plugins.json` (v2 shape) and reads each plugin's `.mcp.json` in both shapes the official marketplace uses (flat `{name: entry}` and `{"mcpServers": {...}}`). Whether Claude will actually launch a plugin is resolved the way Claude Code itself does it — verified probe by probe against Claude Code 2.1.238 with real `claude plugin install/disable` runs: - `enabledPlugins` is merged from the user `settings.json`, then `<cwd>/.claude/settings.json`, then `<cwd>/.claude/settings.local.json`; the most specific scope wins. - The plugin must be explicitly `true`; an installed plugin with no entry anywhere is **not** launched. - The install record's `scope` / `projectPath` do not gate anything: a subdirectory with its own `.claude` settings launches it, one without does not. So a `--scope project` install is reported inside its project and silent elsewhere, an installed-but-never-enabled plugin does not warn, and `claude plugin disable <id>` (which writes `false` to whichever of those three files matches the scope and keeps the install record) reliably clears the warning. `headroom mcp reconcile` prints the plugin entry and that disable command. Per the issue, **nothing is removed** — Headroom does not mutate third-party plugins. Plugin state and the user `settings.json` follow `CLAUDE_CONFIG_DIR` the same way the modern config already does in this registrar. **Alternatives considered.** - Read the plugin files directly in `cli/mcp.py` next to `find_headroom_registration` — shorter, but that module's path constants ignore `CLAUDE_CONFIG_DIR`, so it would duplicate (and get wrong) the resolution the registrar already has. - Fold the plugin entry into `get_server` / `register_server` `MISMATCH` — would change `headroom wrap` behaviour for every user and could steer `--adopt`/force paths at a plugin; the issue explicitly asks for no automatic removal. - Gate project-scoped records by comparing the resolved cwd against `projectPath` (exact or containment). Measured against Claude: it is neither necessary nor sufficient — `proj/sub` does *not* launch a plugin installed at `proj` unless `proj/sub` has its own `.claude` settings, and a project-level `false` silences a *user-scope* install. Reading the three settings files is both simpler and exact. - Also warn from `mcp status`, `doctor`, or at wrap time — left for follow-ups to keep this diff to the command the issue reproduces with (`reconcile` is the Serena command; `status` reports Headroom's own server). Closes #3570 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/mcp_registry/claude.py`: new `ClaudeRegistrar.get_plugin_servers(server_name) -> list[tuple[plugin_id, ServerSpec]]`, `_enabled_plugins(project_dir, user_settings)` (merges the three `enabledPlugins` sources in Claude's precedence) and `_read_plugin_mcp_entry` (both `.mcp.json` shapes). Resolves the user `settings.json` / `plugins/installed_plugins.json` under `CLAUDE_CONFIG_DIR` when set, else `~/.claude`. Reuses the existing read-only `_read_json` (any unreadable/malformed file → no match). No change to `get_server`, `register_server`, or `wrap`. - `headroom/cli/mcp.py`: `mcp reconcile` (inspect path) prints `plugin: <id> also provides a Serena MCP server (<command>)` and `action: ... claude plugin disable <id>` for each plugin bundling Serena that Claude would launch from the current directory. - `tests/fixtures/headroom-issue-3570.json`: the exact on-disk shapes Claude Code 2.1.238 wrote for `claude plugin install serena@claude-plugins-official` at user and project scope (`installed_plugins.json` v2 with per-scope record list incl. `projectPath`, `enabledPlugins`, the plugin's flat `.mcp.json`) — this layout is not in the Claude Code docs. - `tests/test_mcp_registry/test_claude_registrar.py`: 19 tests / 43 cases for `get_plugin_servers` — flat and `mcpServers` shapes; disabled plugin skipped; absent / `{}` / `null` / unparseable `enabledPlugins` → not launched; project-scope install reported inside its project and quiet in `$HOME`, an unrelated dir and a subdirectory; project `false` silences a user install; project `true` overrides user `false`; `settings.local.json` overrides `settings.json` both ways; one report per plugin; multiple plugins in registry order; name filter; explicit `config_dir` and `CLAUDE_CONFIG_DIR` env; 9 malformed-registry inputs; 6 malformed `.mcp.json` inputs; missing install dir. Every test pins cwd to an isolated directory. - `tests/test_cli/test_mcp_reconcile.py`: regression `test_issue_fixture_plugin_serena_is_reported_base_fail_head_pass` (fails on `main`, passes here; also asserts the Claude-owned plugin files keep identical bytes and mtimes), quiet-once-disabled, `--adopt` leaves plugin files untouched, and `test_reconcile_reports_project_scoped_plugin_only_inside_its_project` (in-project positive, `$HOME` and subdirectory negatives, remedy clears it). - `docs/content/docs/mcp.mdx`: one sentence on the new warning. `CHANGELOG.md` not touched. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Regression test on unpatched main ( |
||
|
|
7550efb68f |
fix(mcp): add explicit Serena reconciliation (#3222)
## Description Headroom repeatedly warns about user-managed Serena drift but has no scoped remediation command. Add a Claude-only read-only mcp reconcile command with explicit --adopt consent, using the canonical Serena spec and existing Claude registrar. Adoption validates every relevant ledger and Claude config root before mutation, writes only the Serena entry, and records ownership after the config write succeeds. Automatic wrap migration and ordinary install remain unchanged. Closes #3054 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (feature that would cause existing behavior to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add Claude-only `headroom mcp reconcile`, read-only by default, with `--adopt` as its only mutation action. - Reuse the shared `CLAUDE_SERENA_CONTEXT` and canonical Claude Serena spec builder. - Fail closed on malformed or unreadable ledger/config state before adoption. - Preserve automatic wrap recovery, user-managed warnings, ordinary `mcp install --force`, unrelated Claude config, and corrupt-ledger tolerance outside explicit adoption. - Record Headroom ownership only after a successful registrar write. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_cli/test_mcp_reconcile.py tests/test_cli/test_serena_reconcile.py tests/test_mcp_registry/test_ledger.py`) - [x] Linting passes (`uv run ruff check .`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed through the file-backed Claude registrar ### Test Output ```text uv run pytest tests/test_cli/test_mcp_reconcile.py tests/test_mcp_registry/test_ledger.py tests/test_cli/test_serena_reconcile.py tests/test_mcp_registry/test_claude_registrar.py tests/test_mcp_registry/test_install.py -q 102 passed in 0.70s uv run ruff check headroom/mcp_registry/ledger.py headroom/cli/wrap.py tests/test_mcp_registry/test_ledger.py tests/test_cli/test_serena_reconcile.py tests/test_cli/test_mcp_reconcile.py All checks passed! uv run ruff format --check headroom/mcp_registry/ledger.py headroom/cli/wrap.py tests/test_mcp_registry/test_ledger.py tests/test_cli/test_serena_reconcile.py tests/test_cli/test_mcp_reconcile.py 5 files already formatted git diff --check ``` ## Real Behavior Proof - Environment: Windows, file-backed Claude configuration and isolated MCP ledger. - Exact command / steps: run the stale user-managed Serena fixture from `tests/fixtures/headroom-issue-3054.json`; run read-only reconcile; run `mcp reconcile --adopt`; rerun wrap and ordinary `mcp install --force`; exercise malformed JSON, non-dict `mcpServers`, null ledger agents, and unreadable-ledger adoption. - Observed result: read-only reconciliation leaves config and ledger bytes unchanged; adoption updates only Claude Serena and records ownership after a successful write; automatic wrap remains lenient; unsafe adoption inputs leave all files unchanged; ordinary install does not adopt Serena. - Not tested: live Claude CLI acceptance and Serena stdio handshake ## Runtime Rollout Safety - Rollout-managed feature(s): None; explicit `mcp reconcile --adopt` is the only mutation path. - Minimum rollout channel: Stable; no staged rollout mechanism exists for this command. - Stable/default behavior changed: No, read-only reconcile is the default and automatic wrap plus ordinary install remain unchanged. - Kill switch / disable path: Do not invoke `--adopt` or revert the release commit. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert the release commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The changelog is generated by the release pipeline. This change is limited to Claude Serena reconciliation and does not add a new persistent acknowledgement state or a multi-provider adoption route. |
||
|
|
3077ac81e8 |
feat: add deterministic runtime rollout controls (#1490)
## Description Establish one centrally resolved, observable, deterministic, versioned runtime rollout-control mechanism for Headroom. Runtime rollout controls which behaviors an already-built artifact may expose; it does not select or qualify a Headroom release/version. ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Bug fix (non-breaking change that fixes rollout enforcement regressions) - [x] Documentation update - [x] Code refactoring (no functional changes) ## Changes Made - Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`, `--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared by Python configuration boundaries. - Added schema/policy versions, canonical registry and snapshot SHA-256 identities, per-feature decision reasons, disable precedence, unsafe qualification poisoning, strict CLI validation, and fail-closed environment handling. - Added `headroom rollout status --json`, Python `/stats.rollout`, and Rust `/rollout/status` runtime provenance. - Added equivalent Rust snapshot semantics and shared Python/Rust policy vectors while retaining language-specific feature registries. - Enforced rollout policy at alternate Python server composition roots so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate. - Preserved typed rollout snapshots across multi-worker serialization with schema, policy, registry, snapshot-digest, type, and feature-name validation. - Made loopback runtime output-shaper updates replace the immutable snapshot atomically for request readers, retain explicit request/disable provenance, preserve channel and kill-switch precedence, invalidate cached stats, and return the effective rollout decision. - Made `headroom learn --verbosity --apply` report a channel-blocked update instead of claiming the shaper is live. - Made explicit CLI feature flags fail loudly when their current channel blocks them. - Made persistent interceptor installation select canary automatically, or reject an explicitly insufficient channel unless the break-glass override is set. - Updated architecture, proxy, rollout, learn, and output-shaper documentation with required channels and hot-reload semantics. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check .` and `ruff format --check .`) - [x] Type checking passes (`mypy headroom --ignore-missing-imports`) - [x] New regression tests added for every corrected behavior - [x] Rust tests and production-target Clippy pass - [x] Documentation build passes ### Test Output ```text Focused rollout coverage suite 57 passed; headroom.rollout + rollout CLI: 98% coverage Affected proxy/rollout/transform/governance suites 222 passed; 0 failed Final changed regression suites 100 passed; 0 failed Cross-module hot-reload isolation regression 6 passed; 0 failed cargo test -p headroom-core -p headroom-proxy --quiet headroom-core: 924 passed; 1 ignored headroom-proxy and integration suites: all passed cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings cargo fmt --all -- --check ruff check . ruff format --check . mypy headroom --ignore-missing-imports git diff --check All passed cd docs && npm run build Compiled successfully; 164 static pages generated ``` The unsharded Windows-only CI selection exposed unrelated baseline failures, principally the existing `sqlite:///C:\\...` URL parser producing an invalid `\\C:\\...` path. At commit `8e793a80`, all 52 completed GitHub checks passed; the only other conclusions are expected skips and superseded governance jobs. ## Real Behavior Proof - **Environment:** Windows checkout on Python 3.13.3 and the current Rust workspace, based on upstream `main` at `93f2d7a2`. - **Exact command / steps:** Exercised canary and beta feature requests through CLI status, Python `/stats.rollout`, Rust `/rollout/status`, multi-worker payload round trips, loopback `/admin/runtime-env`, real proxy request shaping before/after hot reload, installer manifest generation, and shared Python/Rust policy vectors. - **Observed result:** Stable blocks unstable requests; disable wins over explicit/default/legacy/unsafe paths; unsafe state reports `qualification_eligible=false`; worker handoff rejects tampering; running output shaping changes only when the effective beta policy permits it; explicit blocked flags fail with actionable diagnostics. - **Not tested:** Live production traffic requiring provider credentials, or future artifact qualification/promotion automation (intentionally out of scope). ## Runtime Rollout Safety - **Rollout-managed features:** Python `tool_result_interceptors`, `proxy_output_shaper`, `read_maturation`; Rust `native_bedrock`, `openai_responses_streaming`, `canary_probe`. - **Minimum rollout channel:** Registry-defined per feature; process default is `stable`. - **Stable/default behavior changed:** No unstable feature becomes enabled by default. Explicit blocked CLI flags now fail instead of silently doing nothing. - **Kill switch / disable path:** `HEADROOM_DISABLE_FEATURES=<comma-separated feature names>`; explicit disable has highest precedence, including over the unsafe override. - **Unsafe override required:** No. `HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES=1` is break-glass only and makes qualification evidence ineligible. - **Qualification impact:** Adds machine-readable policy/snapshot identities and eligibility; does not implement qualification itself. - **Rollback path:** Set the named disable list for operational rollback, lower the channel, or revert this PR. ## Review Readiness - [x] I have performed a full diff review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented hard-to-understand areas - [x] I have made corresponding documentation changes - [x] My changes generate no new warnings - [x] I added tests that reproduce and prevent every regression fixed during review - [x] New and existing affected tests pass locally - [x] I did **not** edit `CHANGELOG.md`; release-please generates it from the Conventional Commit PR title ## Additional Notes Out of scope: artifact candidates, benchmark orchestration, qualification manifests/gates, promotion automation, release branches, publication guards, and release-risk classification. Those workflows can consume the rollout registry digest, runtime snapshot digest, decision reasons, and qualification eligibility through supported black-box interfaces. --------- Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> Co-authored-by: JD Davis <jd@JDH-AIR-00.local> |
||
|
|
38479fcda1 |
fix(codex): rerun memory lookup on every response.create WS frame (#2113)
## Description Long-lived Codex `/v1/responses` WebSocket sessions only ran memory decision, query construction, and context injection on the first `response.create` frame because `handle_openai_responses_ws` kept that logic outside the relay loop. This change extracts that path into a local helper reused before compression for every eligible `response.create`, while keeping sticky memory tools deduplicated by the existing session helper. Closes #2059 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Extracted the first-frame memory preparation path into a local async helper inside `handle_openai_responses_ws`. - Reused that helper for the initial frame and every later eligible `response.create` before compression and shaping. - Added focused two-turn WebSocket regressions for per-frame lookup, bypass and disabled-memory handling, list-shaped later inputs, memory-handler fail-open recovery, and sticky-tool replay. - Raised the locked production floors for `click` and `pillow` to clear the current `pip-audit` findings that now fail external PR merge snapshots. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_codex_ws_per_frame_memory.py tests/test_openai_codex_ws_lifecycle.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_codex_ws_per_frame_memory.py tests/test_openai_codex_ws_lifecycle.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run pytest tests/test_codex_ws_per_frame_memory.py tests/test_openai_codex_ws_lifecycle.py -q ============================= test session starts ============================= platform win32 -- Python 3.12.13, pytest-9.0.3, pluggy-1.6.0 rootdir: D:\Repos\headroom-pr-2059-codex-ws-memory-lookup configfile: pyproject.toml plugins: anyio-4.12.1, langsmith-0.9.3, asyncio-1.3.0, cov-7.0.0 asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 35 items tests\test_codex_ws_per_frame_memory.py ......... [ 25%] tests\test_openai_codex_ws_lifecycle.py .......................... [100%] ============================= 35 passed in 1.89s ============================== uv run ruff check headroom/proxy/handlers/openai.py tests/test_codex_ws_per_frame_memory.py tests/test_openai_codex_ws_lifecycle.py All checks passed! uv run ruff format headroom/proxy/handlers/openai.py tests/test_codex_ws_per_frame_memory.py tests/test_openai_codex_ws_lifecycle.py --check 3 files already formatted ``` ## Real Behavior Proof - Environment: Windows, Python 3.12.13, in-process WebSocket harness with a fake memory handler - Exact command / steps: `uv run pytest tests/test_codex_ws_per_frame_memory.py tests/test_openai_codex_ws_lifecycle.py -q`, which opens one connection, sends distinct `response.create` frames, and records each memory lookup plus the forwarded tool set - Observed result: the handler performs one lookup per eligible frame, later-frame compression sees current-turn memory-prepared input, and both forwarded turns carry the same deduplicated `memory_search` and `memory_save` definitions - Not tested: live Codex subscription WebSocket ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - Security CI currently flags `click 8.3.1` and `pillow 12.2.0`, so this PR carries the narrow floor bump to `click>=8.3.3` and `pillow>=12.3.0` as a supply-chain unblock for the same final merge snapshot. - `CHANGELOG.md` remains untouched because Headroom generates release notes from conventional commits. - This PR is scoped to the OpenAI Responses WebSocket relay lifecycle; it does not change the HTTP `/v1/responses` path or the Anthropic handler. --------- Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
|
23d73ae070 |
test(evals): add offline fidelity regression gate (recall-based, zero-model) (#1187)
## Description Headroom's lossy compression drops rows/lines using statistical heuristics but **never checks that meaning survived** — a dropped `OOM killed worker 3` line can silently flip a model's answer with no signal that compression caused it. The repo already ships a quality-metric toolkit (`headroom/evals/metrics.py`) and a `weekly-suite` eval job, but neither gates the compression path on a PR. This adds a **per-PR fidelity regression gate**: compress vendored golden tool-outputs through SmartCrusher's lossy path and assert the evidence that answers each case's question survives. It is the first of a planned trio (this is the "offline gate" half of the fidelity work); query-aware retention and a hard token-budget API are documented follow-ups. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Blocking gate** (`tests/test_compression_fidelity_regression.py`): compresses each golden case via `smart_crush_tool_output(..., with_compaction=False)` and scores with `compute_information_recall`. Two assertions: - **Per-case critical recall == 1.0** — every `answer_evidence` string (placed in error/anomaly rows, the documented SmartCrusher retention guarantee) must survive. - **Aggregate recall ≥ committed baseline** (`baseline.json`, tol 0.02) — catches softer regressions. - **Vendored fixtures** (`tests/fixtures/fidelity_golden/`): deterministic `_generate.py` emits `cases.json` (4 cases: OOM crash, payment exception, latency anomaly, CI failure) + `baseline.json`. - **Non-blocking weekly report** (`.github/workflows/eval.yml`): one step in the existing `weekly-suite` job (schedule/manual only) reuses the existing `evaluate_information_retention` runner for a recall report on the production routing path. - **Pure reuse**: scoring (`evals/metrics.py`), compressor (`smart_crush_tool_output`), and the weekly runner (`evaluate_information_retention`) all already existed. ### Design notes - **Zero new CI setup.** The blocking gate runs in the existing `[dev]` test shard — no new workflow, no new deps, **no model, no network, no secrets** (verified under `HF_HUB_OFFLINE=1`). It deliberately uses small hand-made structured fixtures rather than the repo's HuggingFace dataset loaders, which would require a network download + ModernBERT and don't belong in a fast PR gate. - **Scope:** structured JSON tool-output (the dominant, deterministic, model-free case). Real-dataset (HotpotQA/BFCL) recall — which needs `[all]` + a local model — is a **documented follow-up PR**, and the `weekly-suite` job (which genuinely runs every Monday) is its natural home. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ HF_HUB_OFFLINE=1 python -m pytest tests/test_compression_fidelity_regression.py -v tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[logs_oom] PASSED tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[payment_exception] PASSED tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[latency_anomaly] PASSED tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[ci_test_failures] PASSED tests/test_compression_fidelity_regression.py::test_aggregate_recall_not_regressed PASSED ============================== 5 passed in 0.18s =============================== ``` ## Real Behavior Proof - **Environment:** local checkout of `feat/fidelity-regression-gate`, `pip install -e ".[dev]"`, `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1` (proves no model/network). - **Exact command / steps:** `HF_HUB_OFFLINE=1 python -m pytest tests/test_compression_fidelity_regression.py -q` → `5 passed in 0.14s`. - **Negative control (proves the gate has teeth):** compressing `logs_oom` and probing for a benign row that compression legitimately drops returns `recall = 0.00, lost = ['heartbeat ping 25']` — i.e. the gate fires when critical evidence is dropped, so it is not trivially green. - **Weekly (non-blocking) step verified locally:** ```text Information retention: 50/50 cases >=0.9 recall, avg compression 65.7% ``` - **Not tested:** real-dataset (HotpotQA/BFCL) recall and prose/ModernBERT compression — intentionally deferred to a follow-up PR targeting the weekly job. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG/version intentionally untouched: repo uses **release-please**. - **Follow-up PR (planned):** wire the real HotpotQA/BFCL loaders (`headroom/evals/datasets.py`) into the `weekly-suite` job for genuine benchmark-scale recall coverage (model-allowed, non-blocking). Further follow-ups from the same design: a live per-request fidelity guardrail, query-aware lossy retention, and a hard `target_tokens` budget API. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
c62d45eea8 |
fix(memory): expose memory IDs in auto-tail + memory_list tool + ID-usage guidance
Pre-this-PR the auto-injected memory block rendered rows as `1. <content>`
with no addressable handle. To UPDATE or DELETE a row the model first had
to call memory_search to discover its ID — two round trips, against the
model-as-judge architecture.
This PR adds three tightly-coupled affordances so the model can act on
memory directly:
1. Auto-tail rows now carry the memory ID:
`1. [mem_alpha_001] User prefers Python`
The bracketed token is the canonical ID — same identifier accepted by
memory_update and memory_delete.
2. New `memory_list` tool — chronological browse (vs `memory_search`'s
semantic lookup). Returns recent memories with their IDs. Backend
dispatches to `Backend.list_memories` if available, else falls back
to an empty-query `search_memories`. Caps at 100 entries.
3. ID-usage guidance text appended to the auto-tail block. Tells the
model that bracketed IDs can go straight to memory_update /
memory_delete with no intervening search. The guidance lives in the
user-message tail (never system) — preserves cache-prefix byte
stability (invariant I2).
`memory_update` and `memory_delete` tool descriptions also point at the
[id] block as a valid ID source — keeps tool docs consistent with the
new affordance.
Verification:
- 10/10 tests pass in tests/test_memory_auto_tail.py (incl. 2 new
guidance tests + 2 new ID-format tests)
- 31/31 tests pass in tests/test_memory_handler_native_ops.py (incl. 4
new memory_list dispatch tests + existing assertions updated for the
[id] format change)
- Golden fixtures regenerated for the tool-description copy changes
(tests/fixtures/memory_tool_definitions/{anthropic,openai}.json)
- Live end-to-end test against real Anthropic API
(tests/test_proxy_memory_integration.py::TestMemoryIdAutoTailAndUpdate):
seeded memory → auto-tail → Claude → memory_update with exact ID.
PASSED.
|
||
|
|
8dcd474aca |
fix: A7 — memory tool injection session-sticky for both Anthropic and OpenAI
Closes the second half of P0-6: once memory injects memory_save / memory_search into body["tools"] for a session, every subsequent turn injects the byte-equal same definitions — even if memory is disabled mid-session. Toggling tool list mid-session busts Anthropic prefix cache per guide §6.3 #2. Adds in headroom/proxy/helpers.py: * SessionToolTracker — bounded LRU keyed by (provider, session_id) storing GOLDEN tool-definition bytes from the first injection. Tracker is provider-aware so the same session_id under Anthropic and OpenAI keeps independent state. Reentrant lock for concurrent access; LRU eviction at HEADROOM_TOOL_TRACKER_MAX_SESSIONS (default 1000). * apply_session_sticky_memory_tools — single coordination point with three paths: first-time inject (record golden bytes), sticky replay (always inject golden bytes regardless of inject_this_turn), and skip. Honors HEADROOM_TOOL_INJECTION_STICKY=disabled as a loud operator opt-in for rollback (NOT a fallback). * serialize_tool_definition_canonical — deterministic byte serialization via the same separators=(",",":")/ensure_ascii=False rules as serialize_body_canonical. * log_tool_injection_decision — structured per-decision log line; never logs the tool definition contents. Wires the helper into all four memory tool injection sites: * handlers/anthropic.py — /v1/messages * handlers/openai.py — /v1/chat/completions * handlers/openai.py — /v1/responses * handlers/openai.py — Codex WS path memory_handler.MemoryHandler gains compute_memory_tool_definitions(provider) — a pure builder that returns the tool definitions without mutating a tools list, so the proxy can route through the sticky tracker. The legacy inject_tools(...) is preserved for callers without a session_id. Tests: tests/test_memory_tool_session_sticky.py — 29 unit + integration cases covering: turn-1→turn-2 byte-equality (Anthropic + OpenAI), sticky replay after memory disabled, golden-fixture pin, LRU eviction, provider isolation under shared session_id, thread-safe concurrent access, env-var contract, disabled-mode passthrough, dedupe with client tools. Golden fixtures pin canonical bytes: * tests/fixtures/memory_tool_definitions/anthropic.json * tests/fixtures/memory_tool_definitions/openai.json No regex. No hardcodes (env-configurable: HEADROOM_TOOL_INJECTION_STICKY, HEADROOM_TOOL_TRACKER_MAX_SESSIONS). No silent fallbacks. Per-decision structured logging. Realignment build constraints satisfied. |