## Description
`headroom/graph/installer.py` had its own downloader for the
codebase-memory-mcp binary: `download_cbm` and `ensure_cbm`, plus
platform, asset-name and zip helpers. `headroom wrap` stopped calling
`ensure_cbm()` in dca9853e (#1230), when tokensave and Serena replaced
codebase-memory-mcp in wrap. Since then the only product user of the
module is the `--code-graph` watcher (`graph/watcher.py:121`). It calls
only `get_cbm_path()`. The downloader duplicated `headroom.binaries`
(the same `tools.json` pin, the same verification), and only its tests
kept it alive.
This PR keeps `get_cbm_path`, `_installed_name`, `CBM_BIN_DIR` and
`CBM_BIN_NAME`, and removes the rest.
Evidence:
- `grep -rnw` for `download_cbm`, `ensure_cbm`, `_detect_platform`,
`_asset_filename`, `_extract_from_zip`, `CBM_VERSION`, `CBM_REPO` and
`GITHUB_RELEASE_URL` found only `installer.py` and tests. The search
covered `headroom crates plugins sdk docs wiki scripts benchmarks
examples e2e .github` and the consumer repos.
- `git show dca9853e` shows the removed `from headroom.graph.installer
import ensure_cbm, get_cbm_path` in `cli/wrap.py`.
- `installer.py` has no `__all__`. `headroom/graph/__init__.py`
re-exports nothing. Neither is documented.
**`binaries.resolve` vs `get_cbm_path` check (requested before
deleting).**
- `binaries.resolve("codebase-memory-mcp")` installs to
`binaries.cache_dir()/codebase-memory-mcp-<version>-<platform>/codebase-memory-mcp`.
The same goes for `headroom tools install` and `ensure_tools()` at proxy
startup with `--intercept-tool-results`. On macOS that is
`~/Library/Caches/headroom/bin/...`; elsewhere it is the platform cache
dir or `HEADROOM_BINARIES_CACHE`.
- `get_cbm_path()` looks on `PATH` and then in `~/.local/bin` only.
- So a binary installed through `headroom.binaries` is **not** found by
the watcher unless that cache dir is on `PATH`. This gap already exists
on main and this PR does not change it.
- The removed `download_cbm` did install into `~/.local/bin`, but
nothing has called it since #1230.
- A follow-up could make `get_cbm_path()` also try
`binaries.which("codebase-memory-mcp")`. I left that out because it is a
behaviour change.
Refs #4136
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- `headroom/graph/installer.py`:
- Remove `download_cbm`, `ensure_cbm`, `_detect_platform`,
`_asset_filename` and `_extract_from_zip`.
- Remove the constants `CBM_VERSION`, `CBM_REPO` and
`GITHUB_RELEASE_URL`, and the imports only they used.
- The module docstring now says what the module does: it locates the
binary.
- `headroom/offline.py`: the docstring lists codebase-memory-mcp under
release-binary downloads.
- Tests:
- `tests/test_graph.py`: drop the platform, download and ensure tests.
- `tests/test_graph_installer.py`: rewrite the real-PATH-lookup test so
it creates the binary directly instead of through `download_cbm`, and
keep the off-PATH test.
- `tests/test_binaries_refusal_reaches_callers.py`: drop the two
`ensure_cbm` tests.
- `tests/test_offline_egress_chokepoint.py`: drop
`test_cbm_download_opens_no_socket`.
- The watcher tests are unchanged.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest -q -p no:cacheprovider tests/test_graph.py tests/test_graph_installer.py tests/test_binaries_refusal_reaches_callers.py tests/test_offline_egress_chokepoint.py tests/test_litellm_optional.py tests/test_binaries.py
FAILED tests/test_offline_egress_chokepoint.py::TestHeadroomCloudCompressionOffline::test_asgi_middleware_opens_no_socket
FAILED tests/test_offline_egress_chokepoint.py::TestHeadroomCloudCompressionOffline::test_litellm_callback_opens_no_socket
======================== 2 failed, 188 passed in 16.14s ========================
# The same two fail on a clean origin/main (2 failed, 141 passed for that file), so they are pre-existing and unrelated.
$ mypy headroom/graph/installer.py headroom/graph/watcher.py --ignore-missing-imports
Success: no issues found in 2 source files
$ uvx ruff@0.16.8 check <changed files>
All checks passed!
```
## Real Behavior Proof
- Environment: macOS, Python 3.12.13, repo .venv, branch checkout at
0cbd67d6 (base a9815bbb), temp HOME / HEADROOM_WORKSPACE_DIR /
CODEX_HOME, offline.
- Exact command / steps:
- Ran the shared smoke harness with `--code-graph`. It starts a real
proxy from this checkout against a local fake upstream and sends one
Anthropic and one OpenAI request with a compressible tool result.
- Ran a direct script with a stub `codebase-memory-mcp` in the temp
`~/.local/bin`. It calls `get_cbm_path` and starts the real
`CodeGraphWatcher`, then repeats both with the binary removed.
- Ran the graph, installer, binaries and offline-egress tests.
- Observed result:
- With `--code-graph`, the proxy starts and both requests return HTTP
200 with savings. The temp HOME has no binary, so the watcher stays off,
which is unchanged and logged at DEBUG only.
- With the stub present, `get_cbm_path` finds it and the watcher starts.
Without it, both return None/False.
- 188 tests pass. The 2 failures are pre-existing on main.
- Not tested: Indexing with a real codebase-memory-mcp binary.
`get_cbm_path` and the watcher are unchanged.
```text
$ bash smoke.sh <worktree> --code-graph
branch: cleanup/memory-cache-graph-downloader @ 0cbd67d6
health: 200
POST /v1/messages -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines:
2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
$ HOME=$TMP/home HEADROOM_WORKSPACE_DIR=$TMP/ws CODEX_HOME=$TMP/codex python -u -c "<get_cbm_path(); CodeGraphWatcher(project_dir).start(); remove stub; repeat>"
get_cbm_path() -> $TMP/home/.local/bin/codebase-memory-mcp
watcher.start() -> True
without binary: get_cbm_path() -> None ; start() -> False
```
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; dead-code removal.
- Stable/default behavior changed: No. Nothing called the removed
downloader.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A.
## Additional Notes
Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.
Follow-ups, out of scope here:
- the install-location gap described above;
- `binaries.verify_download_bytes`, whose only product caller was the
removed `download_cbm`. It is now used only by `tests/test_binaries.py`,
and its docstring still names codebase-memory-mcp.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Description
On Windows, `download_cbm()` asked for
`codebase-memory-mcp-windows-amd64.tar.gz` and then opened the download
with `tarfile`. Upstream publishes the Windows build as
`codebase-memory-mcp-windows-amd64.zip`, which is also the asset
`headroom/tools.json` pins. So on Windows the request went to an asset
that does not exist and has no pin, and extraction could not have read
the zip anyway.
This takes the asset name from the `tools.json` entry for the platform
and extracts `.zip` assets with `zipfile`, as the issue suggests. Only
the binary member is read, and it is written to the same target path as
before. There is no `extractall` and no path from the archive is used.
The `.tar.gz` branch is the same code, re-indented under an `else`.
Closes#3725
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/graph/installer.py`: `_asset_filename(plat)` returns the
`codebase-memory-mcp-<plat>.tar.gz` or `.zip` name that the registry
lists for the platform, falling back to `.tar.gz`. `download_cbm()` uses
it and extracts `.zip` downloads with the new `_extract_from_zip()`,
which accepts a member named `codebase-memory-mcp` or
`codebase-memory-mcp.exe`. `zipfile.BadZipFile` is reported like
`tarfile.TarError`, as "Failed to extract archive".
- `tests/test_graph_installer.py` (new file, kept apart from
`tests/test_graph.py`, which #3724 edits): every platform
`_detect_platform()` returns maps to an asset URL listed in
`tools.json`, with `.zip` for `windows-amd64`. A simulated Windows
download requests the `.zip` URL and installs the binary for both member
names. A zip without the binary, or bytes that are not a zip, raise the
existing errors.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_graph_installer.py tests/test_graph.py tests/test_binaries.py -q
48 passed in 7.31s
# the new tests against the old installer.py:
$ pytest tests/test_graph_installer.py -q
8 failed (5x AttributeError: no attribute '_asset_filename';
2x RuntimeError: Failed to extract archive: not a gzip file;
1x the missing-binary case gets "not a gzip file" instead)
$ ruff check . All checks passed!
$ ruff format --check . 1622 files already formatted
$ mypy headroom --ignore-missing-imports Success: no issues found in 538 source files (mypy 1.20.2)
```
With the #3724 diff applied on top of this branch, the same three test
files also pass (55 passed).
## Real Behavior Proof
- Environment: Windows 11 (AMD64), Python 3.12.13, this branch and
`main` at 5ff4ea1e.
- Exact command / steps: a script calls the real `download_cbm()` with
`CBM_BIN_DIR` set to a temp dir. Only `urlopen` is replaced: it records
the URL and returns a zip containing `codebase-memory-mcp.exe`, the
member name from the `tools.json` comment.
`HEADROOM_BINARIES_ALLOW_UNVERIFIED=1` is set so the fake bytes are not
checked against the real pin.
- Observed result on `main`: requested
`.../v0.8.1/codebase-memory-mcp-windows-amd64.tar.gz` (not in
`tools.json`), then `RuntimeError: Failed to extract archive: not a gzip
file`.
- Observed result on this branch: requested
`.../v0.8.1/codebase-memory-mcp-windows-amd64.zip` (the pinned URL) and
installed the binary as `codebase-memory-mcp.exe`. With only the install
dir on `PATH`, the real `shutil.which("codebase-memory-mcp")` and
`get_cbm_path()` both resolve it, and the `--version` probe runs that
path.
- A HEAD request for the real release URLs returns 404 for the `.tar.gz`
and 302 for the `.zip`.
- Not tested: a real download and extraction of the published Windows
zip. The member name `codebase-memory-mcp.exe` comes from the
`tools.json` comment ("not yet extracted on a windows host"), so both
`codebase-memory-mcp` and `codebase-memory-mcp.exe` are accepted. The
full test suite was not run locally. Also note that no production path
calls `download_cbm()` today (the proxy installs the binary through
`binaries.ensure_tools()`), so this fixes the helper and its callers,
not the proxy hot path.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only on Windows, where
`download_cbm()` now fetches the `.zip`. Other platforms request the
same `.tar.gz` URL as before.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Documentation: no user-facing docs mention `download_cbm()`, so none
changed.
---------
Co-authored-by: JD Davis <mxjerrett@gmail.com>
## A-7 (S2, supply chain) — binary verification failed open on an
unpinned URL
`headroom/binaries.py` skipped SHA-256 verification entirely, with only
a log line, when the registry carried **no pin** for a download URL — in
both `_verify_sha256()` (the `resolve()` fetch path) and
`verify_download_bytes()` (the in-memory installer path). An
off-registry version override, or an installer building a filename the
registry does not describe, therefore installed whatever the mirror
served, protected only by HTTPS transport trust.
`verify_download_bytes`'s own docstring already claimed it *"refuses an
unpinned URL"*. It did not — the documentation described behaviour the
code did not have.
**Both claims in the backlog verified against `origin/main` (5ff4ea1e);
line numbers had not drifted from the 0.37.0 review.**
### What changed
- New `UnpinnedDownload(BinaryError)`. Both verification paths now
**raise** on a missing pin instead of falling through — the partial
download is deleted first, so nothing unverified is left in the cache
for a later call to pick up.
- `HEADROOM_BINARIES_ALLOW_UNVERIFIED` is kept as the escape hatch but
is now **loud**: a shared `_allow_unverified()` helper logs a warning
*and* prints to stderr. The logger is routinely unconfigured at proxy
startup, which is exactly when this fires, so the log-only warning was
invisible in practice.
- The env var is now documented in the module docstring (it was missing
from the `Env vars:` block entirely) with an explicit "never set this in
CI or production; pin the asset instead".
- `verify_download_bytes`'s docstring corrected to describe what the
code does, and its stale caller list trimmed (see below).
- `ensure_tools()` catches `UnpinnedDownload` alongside the other fetch
failures, so proxy startup still degrades to "tool missing" rather than
crashing.
### ⚠️ This changes install-time behaviour
A download with no `tools.json` pin now **fails** instead of installing.
No shipped path is affected (see the caller trace), but anyone relying
on the old silent fall-through — a local off-registry version override,
a fork with a hand-edited registry — will now get a hard error. The fix
is to pin the asset (`scripts/refresh_tool_hashes.py`), not to set the
escape hatch.
### Callers traced
`_verify_sha256` has exactly one caller and `verify_download_bytes`
exactly one; everything else reaches them transitively.
| Path | Pinned today? | Breaks? |
|---|---|---|
| `resolve()` → `_verify_sha256` (binaries.py) | Yes — pin comes from
`tools.json`, **all 21 assets across difft/scc/codebase-memory-mcp are
pinned** | No |
| `ensure_tools()` → `resolve()` (proxy startup, `cli/proxy.py:1216`) |
Yes | No — and a refusal is caught, so startup still succeeds with the
tool skipped |
| `cli/tools.py:43,230` → `resolve()` (`headroom tools install` /
`doctor`) | Yes | No |
| `proxy/interceptors/astgrep.py:122` → `resolve("ast-grep")` | n/a —
ast-grep is PyPI-only, never downloaded here | No |
| `graph/installer.py:82` `download_cbm()` → `verify_download_bytes` |
Yes for the default `CBM_VERSION = v0.8.1` on linux-amd64/arm64 and
darwin-amd64/arm64 | No |
| …same, with a `version=` override | **No** — off-registry URL | **Yes,
by design.** No production caller passes `version`; `ensure_cbm()` calls
`download_cbm()` with no args. Only tests did. |
| …same, on **Windows** | **No** — see "things the brief did not know"
below | Already broken today; now fails earlier with a clearer message |
| ONNX / embedding models (`headroom/onnx_runtime.py`) | n/a —
`hf_hub_download` at a pinned HF revision, never touches this module |
No |
| Rust / SmartCrusher | n/a — built by cargo (`ort-download-binaries` is
a cargo feature), not fetched through `binaries.py` | No |
| rtk / lean-ctx | n/a — those download paths were **removed** (see
`context_tool_cleanup.py`) | No |
The registry really does pin everything the product ships:
`.github/workflows/tools-hash-refresh.yml` runs
`scripts/refresh_tool_hashes.py --check`, which exits 1 if any pin is
*missing or stale*, on every `tools.json` PR and weekly. A new unit test
(`test_every_shipped_asset_is_pinned`) asserts the same thing so an
unpinned entry fails fast in unit tests too — that gate is what makes
failing closed safe.
### Fail-before / pass-after
Method: commit the work, `git checkout origin/main --
headroom/binaries.py` (keeping the new tests), run, restore with `git
checkout HEAD -- headroom/binaries.py`.
**Before** (new tests against `origin/main`'s `binaries.py`):
```
tests/test_binaries.py ...................FFFF....... [ 63%]
tests/test_graph.py ..........F...... [100%]
______________ test_unpinned_asset_is_refused_and_partial_deleted ______________
E Failed: DID NOT RAISE <class 'headroom.binaries.BinaryError'>
_________ test_unpinned_asset_allowed_by_escape_hatch_warns_on_stderr __________
E AssertionError: assert 'WITHOUT sha256 verification' in ''
_______________ test_verify_download_bytes_refuses_unpinned_url ________________
E Failed: DID NOT RAISE <class 'headroom.binaries.BinaryError'>
___________ test_verify_download_bytes_escape_hatch_warns_on_stderr ____________
E AssertionError: assert 'WITHOUT sha256 verification' in ''
______________ test_download_cbm_refuses_an_off_registry_version _______________
E Failed: DID NOT RAISE <class 'headroom.binaries.BinaryError'>
========================= 5 failed, 42 passed in 5.21s =========================
```
The three `DID NOT RAISE` failures are the defect itself: on `main`, an
unpinned download is accepted and installed.
**After**:
```
tests/test_binaries.py .............................. [ 63%]
tests/test_graph.py ................. [100%]
============================== 47 passed in 3.74s ==============================
```
Wider sweep (every test module that touches `binaries`) — `test_binaries
test_graph test_cli_tools test_release_workflows
test_context_tool_cleanup test_memory/test_traffic_learner`: **294
passed**.
Lint: `uvx ruff@0.16.7 check .` → clean; `format --check` → clean.
### Existing tests that had to be adjusted
`tests/test_binaries.py`'s autouse fixture nulls every shipped pin so
mechanics tests can serve tiny mock archives. That made *every* such
fetch an unpinned one, so four fetch/extract-mechanics tests now opt
into the escape hatch via a new `allow_unverified` fixture, and the
autouse fixture `delenv`s the variable so a developer's exported value
cannot silently mask the guard. Two `download_cbm` tests in
`tests/test_graph.py` needed the same, for the same reason — they drive
off-registry URLs on purpose.
### Things the brief got wrong / found along the way (not fixed here)
1. **`verify_download_bytes`'s caller list was stale.** It named rtk and
lean-ctx; both downloaders were removed
(`headroom/context_tool_cleanup.py` exists precisely to clean up after
them). Only codebase-memory-mcp remains. Docstring corrected.
2. **Pre-existing, unrelated bug: `download_cbm` is broken on Windows.**
`graph/installer.py` builds `codebase-memory-mcp-windows-amd64.tar.gz`,
but the registry pins `...-windows-amd64.zip`, and the extractor calls
`tarfile.open(mode="r:gz")` unconditionally. So the Windows path already
fails (404, or a tar error on a zip) before this change. This PR makes
it fail one step earlier, at verification, with a message that names the
cause. Worth a separate issue — I did not widen scope to fix it.
3. **`ensure_cbm()` catches only `RuntimeError`**, so a `Sha256Mismatch`
— a genuine tamper signal — already escapes it today, and
`UnpinnedDownload` would too. Since no production caller passes a
`version` override, the shipped path cannot reach it, so I left it alone
rather than converting a loud integrity failure into a silent `None`.
Flagging it as a judgement call.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but
74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings,
violating that contract. Every macOS/Linux clone reports these files as
"modified" on fresh checkout because git's diff engine sees the stored
bytes don't match the attribute contract, even though the working tree
and index match byte-for-byte.
Running `git add --renormalize .` rewrites each affected blob so the
stored form matches the attribute declaration. No semantic changes —
every affected file's diff is "N insertions, N deletions" with inserts
and deletes being the same lines modulo line endings.
Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub
blame skip this mechanical commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>