mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(graph): fetch and extract the zip codebase-memory-mcp asset on Windows (#3730)
## Description
On Windows, `download_cbm()` asked for
`codebase-memory-mcp-windows-amd64.tar.gz` and then opened the download
with `tarfile`. Upstream publishes the Windows build as
`codebase-memory-mcp-windows-amd64.zip`, which is also the asset
`headroom/tools.json` pins. So on Windows the request went to an asset
that does not exist and has no pin, and extraction could not have read
the zip anyway.
This takes the asset name from the `tools.json` entry for the platform
and extracts `.zip` assets with `zipfile`, as the issue suggests. Only
the binary member is read, and it is written to the same target path as
before. There is no `extractall` and no path from the archive is used.
The `.tar.gz` branch is the same code, re-indented under an `else`.
Closes #3725
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/graph/installer.py`: `_asset_filename(plat)` returns the
`codebase-memory-mcp-<plat>.tar.gz` or `.zip` name that the registry
lists for the platform, falling back to `.tar.gz`. `download_cbm()` uses
it and extracts `.zip` downloads with the new `_extract_from_zip()`,
which accepts a member named `codebase-memory-mcp` or
`codebase-memory-mcp.exe`. `zipfile.BadZipFile` is reported like
`tarfile.TarError`, as "Failed to extract archive".
- `tests/test_graph_installer.py` (new file, kept apart from
`tests/test_graph.py`, which #3724 edits): every platform
`_detect_platform()` returns maps to an asset URL listed in
`tools.json`, with `.zip` for `windows-amd64`. A simulated Windows
download requests the `.zip` URL and installs the binary for both member
names. A zip without the binary, or bytes that are not a zip, raise the
existing errors.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_graph_installer.py tests/test_graph.py tests/test_binaries.py -q
48 passed in 7.31s
# the new tests against the old installer.py:
$ pytest tests/test_graph_installer.py -q
8 failed (5x AttributeError: no attribute '_asset_filename';
2x RuntimeError: Failed to extract archive: not a gzip file;
1x the missing-binary case gets "not a gzip file" instead)
$ ruff check . All checks passed!
$ ruff format --check . 1622 files already formatted
$ mypy headroom --ignore-missing-imports Success: no issues found in 538 source files (mypy 1.20.2)
```
With the #3724 diff applied on top of this branch, the same three test
files also pass (55 passed).
## Real Behavior Proof
- Environment: Windows 11 (AMD64), Python 3.12.13, this branch and
`main` at 5ff4ea1e.
- Exact command / steps: a script calls the real `download_cbm()` with
`CBM_BIN_DIR` set to a temp dir. Only `urlopen` is replaced: it records
the URL and returns a zip containing `codebase-memory-mcp.exe`, the
member name from the `tools.json` comment.
`HEADROOM_BINARIES_ALLOW_UNVERIFIED=1` is set so the fake bytes are not
checked against the real pin.
- Observed result on `main`: requested
`.../v0.8.1/codebase-memory-mcp-windows-amd64.tar.gz` (not in
`tools.json`), then `RuntimeError: Failed to extract archive: not a gzip
file`.
- Observed result on this branch: requested
`.../v0.8.1/codebase-memory-mcp-windows-amd64.zip` (the pinned URL) and
installed the binary as `codebase-memory-mcp.exe`. With only the install
dir on `PATH`, the real `shutil.which("codebase-memory-mcp")` and
`get_cbm_path()` both resolve it, and the `--version` probe runs that
path.
- A HEAD request for the real release URLs returns 404 for the `.tar.gz`
and 302 for the `.zip`.
- Not tested: a real download and extraction of the published Windows
zip. The member name `codebase-memory-mcp.exe` comes from the
`tools.json` comment ("not yet extracted on a windows host"), so both
`codebase-memory-mcp` and `codebase-memory-mcp.exe` are accepted. The
full test suite was not run locally. Also note that no production path
calls `download_cbm()` today (the proxy installs the binary through
`binaries.ensure_tools()`), so this fixes the helper and its callers,
not the proxy hot path.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only on Windows, where
`download_cbm()` now fetches the `.zip`. Other platforms request the
same `.tar.gz` URL as before.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Documentation: no user-facing docs mention `download_cbm()`, so none
changed.
---------
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JD Davis
parent
994a89f169
commit
b776983128
+52
-13
@@ -8,6 +8,7 @@ import platform
|
||||
import shutil
|
||||
import stat
|
||||
import tarfile
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from urllib.request import urlopen
|
||||
|
||||
@@ -38,6 +39,41 @@ def _detect_platform() -> str:
|
||||
raise RuntimeError(f"Unsupported platform: {system} {machine}")
|
||||
|
||||
|
||||
def _asset_filename(plat: str) -> str:
|
||||
"""Return the release asset name for ``plat``, with the extension tools.json pins.
|
||||
|
||||
Upstream publishes a .zip for Windows and a .tar.gz for every other platform.
|
||||
"""
|
||||
from headroom.binaries import _tool_entry
|
||||
|
||||
stem = f"codebase-memory-mcp-{plat}"
|
||||
for asset in _tool_entry("codebase-memory-mcp").get("assets", {}).values():
|
||||
name = str(asset.get("url", "")).rsplit("/", 1)[-1]
|
||||
if name in (f"{stem}.tar.gz", f"{stem}.zip"):
|
||||
return name
|
||||
return f"{stem}.tar.gz"
|
||||
|
||||
|
||||
def _extract_from_zip(data: bytes, target_path: Path) -> None:
|
||||
"""Write the binary member of a .zip release archive to ``target_path``."""
|
||||
wanted = (CBM_BIN_NAME, f"{CBM_BIN_NAME}.exe")
|
||||
with zipfile.ZipFile(io.BytesIO(data)) as archive:
|
||||
for info in archive.infolist():
|
||||
if not info.is_dir() and info.filename.rsplit("/", 1)[-1] in wanted:
|
||||
target_path.write_bytes(archive.read(info))
|
||||
return
|
||||
raise RuntimeError("codebase-memory-mcp binary not found in archive")
|
||||
|
||||
|
||||
def _installed_name(plat: str) -> str:
|
||||
"""Return the file name to install the binary under for ``plat``.
|
||||
|
||||
On Windows, ``shutil.which`` (and so a PATH lookup) only finds a binary
|
||||
whose name ends in a PATHEXT extension such as ``.exe``.
|
||||
"""
|
||||
return f"{CBM_BIN_NAME}.exe" if plat.startswith("windows") else CBM_BIN_NAME
|
||||
|
||||
|
||||
def get_cbm_path() -> Path | None:
|
||||
"""Find codebase-memory-mcp binary, return path or None."""
|
||||
# Check PATH first
|
||||
@@ -46,7 +82,7 @@ def get_cbm_path() -> Path | None:
|
||||
return Path(found)
|
||||
|
||||
# Check our install location
|
||||
installed = CBM_BIN_DIR / CBM_BIN_NAME
|
||||
installed = CBM_BIN_DIR / _installed_name(platform.system().lower())
|
||||
if installed.exists() and installed.is_file():
|
||||
return installed
|
||||
|
||||
@@ -60,11 +96,11 @@ def download_cbm(version: str | None = None) -> Path:
|
||||
"""
|
||||
version = version or CBM_VERSION
|
||||
plat = _detect_platform()
|
||||
filename = f"codebase-memory-mcp-{plat}.tar.gz"
|
||||
filename = _asset_filename(plat)
|
||||
url = f"{GITHUB_RELEASE_URL}/{version}/{filename}"
|
||||
|
||||
CBM_BIN_DIR.mkdir(parents=True, exist_ok=True)
|
||||
target_path = CBM_BIN_DIR / CBM_BIN_NAME
|
||||
target_path = CBM_BIN_DIR / _installed_name(plat)
|
||||
|
||||
logger.info("Downloading codebase-memory-mcp %s for %s ...", version, plat)
|
||||
|
||||
@@ -81,17 +117,20 @@ def download_cbm(version: str | None = None) -> Path:
|
||||
|
||||
verify_download_bytes(data, url=url, name="codebase-memory-mcp")
|
||||
|
||||
# Extract binary from tar.gz
|
||||
# Extract binary from the .zip (Windows) or .tar.gz (everything else)
|
||||
try:
|
||||
with tarfile.open(fileobj=io.BytesIO(data), mode="r:gz") as tar:
|
||||
for member in tar.getmembers():
|
||||
if member.name.endswith(CBM_BIN_NAME) or member.name == CBM_BIN_NAME:
|
||||
member.name = target_path.name
|
||||
tar.extract(member, CBM_BIN_DIR, filter="data")
|
||||
break
|
||||
else:
|
||||
raise RuntimeError("codebase-memory-mcp binary not found in archive")
|
||||
except tarfile.TarError as e:
|
||||
if filename.endswith(".zip"):
|
||||
_extract_from_zip(data, target_path)
|
||||
else:
|
||||
with tarfile.open(fileobj=io.BytesIO(data), mode="r:gz") as tar:
|
||||
for member in tar.getmembers():
|
||||
if member.name.endswith(CBM_BIN_NAME) or member.name == CBM_BIN_NAME:
|
||||
member.name = target_path.name
|
||||
tar.extract(member, CBM_BIN_DIR, filter="data")
|
||||
break
|
||||
else:
|
||||
raise RuntimeError("codebase-memory-mcp binary not found in archive")
|
||||
except (tarfile.TarError, zipfile.BadZipFile) as e:
|
||||
raise RuntimeError(f"Failed to extract archive: {e}") from e
|
||||
|
||||
# Make executable
|
||||
|
||||
@@ -65,6 +65,7 @@ def test_get_cbm_path_prefers_path_then_install_dir(monkeypatch, tmp_path: Path)
|
||||
installed = tmp_path / installer.CBM_BIN_NAME
|
||||
installed.write_text("bin")
|
||||
monkeypatch.setattr(installer, "CBM_BIN_DIR", tmp_path)
|
||||
monkeypatch.setattr(installer.platform, "system", lambda: "Linux")
|
||||
monkeypatch.setattr(installer.shutil, "which", lambda name: str(on_path))
|
||||
assert installer.get_cbm_path() == on_path
|
||||
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
"""Release asset selection and extraction for the codebase-memory-mcp installer."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import shutil
|
||||
import tarfile
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom import binaries
|
||||
from headroom.graph import installer
|
||||
|
||||
|
||||
class FakeResponse(io.BytesIO):
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, tb) -> None:
|
||||
return None
|
||||
|
||||
|
||||
def _zip_archive(member_name: str, data: bytes = b"MZ fake binary") -> bytes:
|
||||
payload = io.BytesIO()
|
||||
with zipfile.ZipFile(payload, "w") as archive:
|
||||
archive.writestr(member_name, data)
|
||||
return payload.getvalue()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("plat", "extension"),
|
||||
[
|
||||
("darwin-arm64", ".tar.gz"),
|
||||
("darwin-amd64", ".tar.gz"),
|
||||
("linux-arm64", ".tar.gz"),
|
||||
("linux-amd64", ".tar.gz"),
|
||||
("windows-amd64", ".zip"),
|
||||
],
|
||||
)
|
||||
def test_asset_filename_matches_the_registry_asset(plat: str, extension: str) -> None:
|
||||
filename = installer._asset_filename(plat)
|
||||
|
||||
assert filename == f"codebase-memory-mcp-{plat}{extension}"
|
||||
pinned_urls = {
|
||||
asset["url"] for asset in binaries._tool_entry("codebase-memory-mcp")["assets"].values()
|
||||
}
|
||||
assert f"{installer.GITHUB_RELEASE_URL}/{installer.CBM_VERSION}/{filename}" in pinned_urls
|
||||
|
||||
|
||||
@pytest.mark.parametrize("member_name", ["codebase-memory-mcp.exe", "codebase-memory-mcp"])
|
||||
def test_download_cbm_on_windows_fetches_and_extracts_the_zip(
|
||||
monkeypatch, tmp_path: Path, member_name: str
|
||||
) -> None:
|
||||
monkeypatch.setenv("HEADROOM_BINARIES_ALLOW_UNVERIFIED", "1")
|
||||
monkeypatch.setattr(installer, "CBM_BIN_DIR", tmp_path)
|
||||
monkeypatch.setattr(installer, "_detect_platform", lambda: "windows-amd64")
|
||||
requested: list[str] = []
|
||||
|
||||
def fake_urlopen(url: str, timeout: int = 60) -> FakeResponse:
|
||||
requested.append(url)
|
||||
return FakeResponse(_zip_archive(member_name))
|
||||
|
||||
monkeypatch.setattr(installer, "urlopen", fake_urlopen)
|
||||
probes: list[list[str]] = []
|
||||
|
||||
def fake_run(command: list[str], **kwargs) -> SimpleNamespace:
|
||||
probes.append(command)
|
||||
return SimpleNamespace(returncode=1, stdout="")
|
||||
|
||||
monkeypatch.setattr("headroom._subprocess.run", fake_run)
|
||||
|
||||
path = installer.download_cbm()
|
||||
|
||||
assert requested == [
|
||||
f"{installer.GITHUB_RELEASE_URL}/{installer.CBM_VERSION}/"
|
||||
"codebase-memory-mcp-windows-amd64.zip"
|
||||
]
|
||||
# A PATH lookup on Windows only finds the binary through its .exe extension.
|
||||
assert path == tmp_path / "codebase-memory-mcp.exe"
|
||||
assert path.read_bytes() == b"MZ fake binary"
|
||||
assert probes == [[str(path), "--version"]]
|
||||
|
||||
|
||||
def _host_release_archive(filename: str) -> bytes:
|
||||
"""Build a release archive in the format ``download_cbm`` requests on this host."""
|
||||
if filename.endswith(".zip"):
|
||||
return _zip_archive("codebase-memory-mcp.exe")
|
||||
payload = io.BytesIO()
|
||||
with tarfile.open(fileobj=payload, mode="w:gz") as tar:
|
||||
data = b"#!/bin/sh\necho codebase-memory-mcp test\n"
|
||||
info = tarfile.TarInfo(name="codebase-memory-mcp")
|
||||
info.size = len(data)
|
||||
tar.addfile(info, io.BytesIO(data))
|
||||
return payload.getvalue()
|
||||
|
||||
|
||||
def test_installed_binary_is_found_by_a_real_path_lookup(monkeypatch, tmp_path: Path) -> None:
|
||||
"""The installed file must be what ``shutil.which`` resolves on this host.
|
||||
|
||||
Nothing about the file name is mocked: on Windows this needs the .exe
|
||||
extension, elsewhere the extensionless name with its executable bit.
|
||||
"""
|
||||
bin_dir = tmp_path / "bin"
|
||||
monkeypatch.setenv("HEADROOM_BINARIES_ALLOW_UNVERIFIED", "1")
|
||||
monkeypatch.setattr(installer, "CBM_BIN_DIR", bin_dir)
|
||||
monkeypatch.setattr(
|
||||
installer,
|
||||
"urlopen",
|
||||
lambda url, timeout=60: FakeResponse(_host_release_archive(url.rsplit("/", 1)[-1])),
|
||||
)
|
||||
probes: list[list[str]] = []
|
||||
|
||||
def fake_run(command: list[str], **kwargs) -> SimpleNamespace:
|
||||
probes.append(command)
|
||||
return SimpleNamespace(returncode=1, stdout="")
|
||||
|
||||
monkeypatch.setattr("headroom._subprocess.run", fake_run)
|
||||
|
||||
path = installer.download_cbm()
|
||||
|
||||
assert probes == [[str(path), "--version"]]
|
||||
monkeypatch.setenv("PATH", str(bin_dir))
|
||||
found = shutil.which("codebase-memory-mcp")
|
||||
assert found is not None
|
||||
assert Path(found).samefile(path)
|
||||
assert Path(probes[0][0]).samefile(found)
|
||||
assert installer.get_cbm_path() == Path(found)
|
||||
|
||||
|
||||
def test_get_cbm_path_finds_the_installed_binary_off_path(monkeypatch, tmp_path: Path) -> None:
|
||||
empty_dir = tmp_path / "empty"
|
||||
empty_dir.mkdir()
|
||||
bin_dir = tmp_path / "bin"
|
||||
bin_dir.mkdir()
|
||||
monkeypatch.setenv("PATH", str(empty_dir))
|
||||
monkeypatch.setattr(installer, "CBM_BIN_DIR", bin_dir)
|
||||
monkeypatch.setattr(installer.platform, "system", lambda: "Windows")
|
||||
(bin_dir / "codebase-memory-mcp.exe").write_bytes(b"MZ fake binary")
|
||||
|
||||
assert installer.get_cbm_path() == bin_dir / "codebase-memory-mcp.exe"
|
||||
|
||||
monkeypatch.setattr(installer.platform, "system", lambda: "Linux")
|
||||
assert installer.get_cbm_path() is None
|
||||
(bin_dir / "codebase-memory-mcp").write_bytes(b"#!/bin/sh\n")
|
||||
assert installer.get_cbm_path() == bin_dir / "codebase-memory-mcp"
|
||||
|
||||
|
||||
def test_download_cbm_zip_errors(monkeypatch, tmp_path: Path) -> None:
|
||||
monkeypatch.setenv("HEADROOM_BINARIES_ALLOW_UNVERIFIED", "1")
|
||||
monkeypatch.setattr(installer, "CBM_BIN_DIR", tmp_path)
|
||||
monkeypatch.setattr(installer, "_detect_platform", lambda: "windows-amd64")
|
||||
|
||||
monkeypatch.setattr(
|
||||
installer, "urlopen", lambda url, timeout=60: FakeResponse(_zip_archive("README.md"))
|
||||
)
|
||||
with pytest.raises(RuntimeError, match="binary not found in archive"):
|
||||
installer.download_cbm()
|
||||
|
||||
monkeypatch.setattr(installer, "urlopen", lambda url, timeout=60: FakeResponse(b"not a zip"))
|
||||
with pytest.raises(RuntimeError, match="Failed to extract archive"):
|
||||
installer.download_cbm()
|
||||
Reference in New Issue
Block a user