mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
feat/https-compose-caddy
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
7a0f15f563 |
fix(binaries): ignore tool binaries found via the current directory (#4838)
## Description
`headroom.binaries._path_lookup()` (used by `which()`, `resolve()` and
`ensure_tools()`) and `status()` returned `shutil.which(name)` before
looking at the SHA-256-pinned cache. On Windows `shutil.which` searches
the current directory before PATH (unless
`NoDefaultCurrentDirectoryInExePath` is set), and on any OS a relative
PATH entry (`.`, an empty entry, `bin`) resolves against the cwd. The
cwd is usually the project being worked on, so a file there named
`ast-grep`, `difft`, `scc`, ... was run in place of the real tool, and
the cache's pin never applied to it.
A PATH hit is now ignored when it comes from a relative PATH entry or
lives in the current directory; lookup then falls back to the
interpreter's Scripts/bin dir and the verified cache (or a pinned fetch
in `resolve()`). `status()` (`headroom tools doctor`) uses the same rule
so it no longer reports such a file as `on-path`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/binaries.py`: new `_which_outside_cwd()`; `_path_lookup()`
and `status()` use it. A warning names the ignored file.
- `tests/test_binaries.py`: real `shutil.which` with `PATH=.` and a tool
in the cwd for `which()`, `resolve()` and `status()`, plus a test that a
normal PATH hit outside the cwd is still used.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# new tests on origin/main (fix stashed)
$ pytest -q tests/test_binaries.py
FAILED tests/test_binaries.py::test_which_ignores_a_tool_in_the_current_directory
FAILED tests/test_binaries.py::test_resolve_ignores_a_tool_in_the_current_directory
FAILED tests/test_binaries.py::test_status_does_not_report_a_cwd_tool_as_on_path
========================= 3 failed, 45 passed in 3.19s =========================
# with the fix
$ pytest -q tests/test_binaries.py tests/test_binaries_refusal_reaches_callers.py tests/test_bundled_tools_savings.py tests/test_cli_tools.py tests/test_offline_egress_chokepoint.py tests/test_ssl_context.py
================== 286 passed, 4 skipped in 156.19s (0:02:36) ==================
$ ruff check / ruff format --check (0.17.3) on changed files
All checks passed!
2 files already formatted
$ mypy headroom/binaries.py --ignore-missing-imports
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS, Python 3.12 venv; empty `HEADROOM_BINARIES_CACHE`,
`HEADROOM_BINARIES_OFFLINE=1`.
- Exact command / steps: in a directory containing an executable
`difft`, run with `PATH=.`: `python -c 'from headroom import binaries as
b; print(b.which("difft")); print([(r["state"], r["path"]) for r in
b.status() if r["tool"]=="difft"])'`.
- Observed result:
- main: `which: difft`, `status: [('on-path', './difft')]`
- this branch: `Ignoring difft found via the current directory
(.../proj/difft); using the interpreter's Scripts dir or the Headroom
cache instead`, `which: None`, `status: [('missing', None)]`
- Not tested: on a Windows host (the Windows case is the same code path:
`shutil.which` returns a relative `.\name` hit there).
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: a tool binary that is only reachable
through the current directory or a relative PATH entry is no longer
used; Headroom uses its verified cached copy instead. Running Headroom
from inside a directory that is itself listed on PATH also falls back to
the cache for tools in that directory.
- Kill switch / disable path: put the tool in a directory listed on PATH
by absolute path.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
No docs change needed. Fix adapted from a proposal by HetCreep (this
version also ignores hits from relative PATH entries other than `.`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4838"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
da6d9736cd |
fix(binaries): fail closed on HEADROOM_BINARIES_ALLOW_UNVERIFIED=0/false (#3218)
## Description
The sha256-verification bypass for fetched tool binaries is a
supply-chain
integrity control, but both call sites gated it on a bare presence
check:
```python
if os.environ.get("HEADROOM_BINARIES_ALLOW_UNVERIFIED"):
...skip sha256 verification...
```
Any non-empty value is truthy — including `0`, `false`, `no`, `off`. An
operator who set `HEADROOM_BINARIES_ALLOW_UNVERIFIED=0` (the natural way
to
say "keep verification on") therefore **silently disabled** sha256
verification of every fetched binary (`difft`, `scc`, and the in-memory
`rtk` / `lean-ctx` / `codebase-memory-mcp` installer path via
`verify_download_bytes`). The failure direction is backwards for a
security
control: it fails **open**.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Parse `HEADROOM_BINARIES_ALLOW_UNVERIFIED` as an affirmative flag
(`1`/`true`/`yes`/`on`) via a shared `_allow_unverified_binaries()`
helper,
used by both `_verify_sha256` and `verify_download_bytes`.
- Every non-affirmative value — and an unset or empty variable — now
**enforces** verification (fail closed). Matches the truthy-set parsing
already used across the codebase (`onnx_runtime`, `cold_prefix`,
`ttl_observations`, `qdrant_env`).
- Added regression tests over the env-value matrix on both call sites.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_binaries.py -q
35 passed, 1 warning in 1.15s
# New tests fail before the fix (0/false/no/off/whitespace + in-memory path):
$ git stash push -- headroom/binaries.py && \
python -m pytest tests/test_binaries.py -q -k "allow_unverified or verify_download_bytes_zero"
FAILED ...::test_allow_unverified_non_affirmative_still_verifies[0]
FAILED ...::test_allow_unverified_non_affirmative_still_verifies[false]
FAILED ...::test_allow_unverified_non_affirmative_still_verifies[no]
FAILED ...::test_allow_unverified_non_affirmative_still_verifies[off]
FAILED ...::test_allow_unverified_non_affirmative_still_verifies[ ]
FAILED ...::test_verify_download_bytes_zero_still_verifies
6 failed, 7 passed, 22 deselected
$ python -m ruff check headroom/binaries.py tests/test_binaries.py && mypy headroom --ignore-missing-imports
All checks passed!
Success: no issues found in 527 source files
```
## Real Behavior Proof
- Environment: macOS (Darwin arm64), Python 3.13. Operator sets
`HEADROOM_BINARIES_ALLOW_UNVERIFIED=0` (intending to keep verification
enabled). A tampered binary whose bytes do not match the registry-pinned
sha256 is passed to `binaries._verify_sha256`.
- Exact command / steps: write a file containing
`b"TAMPERED-BINARY-CONTENT"`, then call `binaries._verify_sha256(path,
<wrong pin>)` under `HEADROOM_BINARIES_ALLOW_UNVERIFIED=0`, once with
the fix stashed (before) and once applied (after).
- Observed result: BEFORE the fix the mismatched binary is accepted
(verification skipped); AFTER the fix it is rejected with
`Sha256Mismatch`.
```text
### HEADROOM_BINARIES_ALLOW_UNVERIFIED=0 (operator intent: keep
verifying)
--- BEFORE fix ---
skipping sha256 verification for difft
(HEADROOM_BINARIES_ALLOW_UNVERIFIED=1)
RESULT: verification SKIPPED — tampered binary ACCEPTED (fail-open)
--- AFTER fix ----
RESULT: Sha256Mismatch raised — tampered binary REJECTED (fail-closed)
```
- Not tested: a live end-to-end download of difft/scc from GitHub
releases (network-dependent); the verification function that gates every
such download is exercised directly above with a real byte payload.
## Runtime Rollout Safety
- Rollout-managed feature(s): none — unconditional hardening of an
existing security control.
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for non-affirmative env values
that previously bypassed verification; `=1`/`true`/`yes`/`on` still
bypass exactly as before, and the default (unset) still verifies.
- Kill switch / disable path: `HEADROOM_BINARIES_ALLOW_UNVERIFIED=1`
still bypasses verification for operators who genuinely need it.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [] I have made corresponding changes to the documentation (N/A —
behavior now matches the existing `=1` docstrings)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3218"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
### Maintainer current-main requalification
Exact head `8c981e8248ff7dca7fd36bc55805a096d3943473`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.
Clean mechanical current-main integration with upstream
|
||
|
|
22d0905a73 |
fix(cli): pass configured SSL context to stdlib urlopen calls (#4011)
## Description Fixes #3976. On `uv tool` installs and other environments where certifi is not wired into the stdlib OpenSSL trust path, `headroom update` and bundled-binary downloads can fail behind corporate TLS inspection. These urllib calls now use the configured Headroom trust context, matching the proxy's TLS policy. The lazy helper import still falls back to urllib's default when the optional trust module cannot be imported. Errors while constructing an explicitly configured context now propagate instead of silently replacing the requested trust roots with stdlib defaults. The update checker retains its outer error-handling behavior and returns no update on failure. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Changes Made - Pass `build_urlopen_context()` to the PyPI version check and release-binary download paths. - Catch only `ImportError` around the lazy helper import; let trust-context construction errors reach the caller. - Add valid and invalid replacement-bundle tests for both helpers and verify the update check still handles the error through its existing outer handler. - Update mocked `urlopen` functions to accept the `context` keyword used by the production call path. ## Testing - [x] Targeted unit tests pass - [ ] Full test suite run - [x] Ruff lint and format checks pass for changed files - [x] New tests added for the requested behavior - [ ] Type checking run - [ ] Manual live-network testing performed ### Test Output ```text python -m pytest --disable-plugin-autoload -p pytest_asyncio.plugin tests/test_ssl_context.py tests/test_binaries.py tests/test_update_check.py 95 passed in 5.10s python -m ruff check headroom/binaries.py headroom/update_check.py tests/test_ssl_context.py tests/test_binaries.py All checks passed! python -m ruff format --check headroom/binaries.py headroom/update_check.py tests/test_ssl_context.py tests/test_binaries.py 4 files already formatted git diff --check passed ``` ## Real Behavior Proof **Original reproduction:** On macOS behind a corporate TLS-inspecting network, `headroom update` failed with `CERTIFICATE_VERIFY_FAILED` when the inspection root was present in the OS trust store but absent from Python's stdlib OpenSSL bundle. The same machine's proxy reached its upstream through the configured root. The original fix routed the direct urllib calls through `build_urlopen_context()`. **Review regression checks:** On Windows with Python 3.13.13, tests using a valid `REQUESTS_CA_BUNDLE` confirm both helpers return a verifying `SSLContext`. With an existing file containing invalid PEM, both helpers now raise `ssl.SSLError`; the update-check caller still returns `None` through its outer error handler. All 95 tests in the affected TLS, binary-download, and update-check suites pass. **Not tested:** A live corporate TLS-inspection connection, Windows `uv tool` end-to-end behavior behind such a network, the full repository test suite, and the separate eval dataset download path. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Not applicable. - Stable/default behavior changed: An invalid explicitly configured trust bundle now fails visibly rather than silently falling back to stdlib roots; unconfigured default behavior is unchanged. - Kill switch / disable path: No new switch; existing TLS environment settings remain available. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this PR's changes. ## Review Readiness - [x] I have performed a self-review. - [x] This PR is ready for human review. ## Checklist - [x] My code follows the project's style guidelines. - [x] I have performed a self-review of my code. - [ ] I have added comments where the code needs additional explanation. - [ ] I have made corresponding documentation changes (none needed for this internal error-handling change). - [x] The changed files pass Ruff checks and the focused tests add no warnings. - [x] I have added tests that prove the requested behavior. - [ ] The full repository unit-test suite was run. - [x] I did not edit `CHANGELOG.md`. --------- Co-authored-by: dajiaohuang <dajiaohuang@users.noreply.github.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
26a2c4939c |
fix(binaries): fail closed when a download carries no sha256 pin (#3724)
## A-7 (S2, supply chain) — binary verification failed open on an
unpinned URL
`headroom/binaries.py` skipped SHA-256 verification entirely, with only
a log line, when the registry carried **no pin** for a download URL — in
both `_verify_sha256()` (the `resolve()` fetch path) and
`verify_download_bytes()` (the in-memory installer path). An
off-registry version override, or an installer building a filename the
registry does not describe, therefore installed whatever the mirror
served, protected only by HTTPS transport trust.
`verify_download_bytes`'s own docstring already claimed it *"refuses an
unpinned URL"*. It did not — the documentation described behaviour the
code did not have.
**Both claims in the backlog verified against `origin/main` (
|
||
|
|
d62a3c4ad0 |
fix(tools): key codebase-memory-mcp assets by platform, not upstream naming (#3432)
## Description
`_asset_for_platform()` resolves a tool's download with a plain
`assets.get(plat.key())`, and `PlatformKey.key()` normalizes arch to
`aarch64`/`x86_64`. The `codebase-memory-mcp` entry in `tools.json` was
keyed in the upstream project's own naming (`darwin-arm64`,
`linux-amd64`, …), so the lookup missed on **every** platform: the tool
ships binaries for four platforms and resolved on none, reporting
`unsupported-platform` everywhere.
`difft` and `scc` already transliterate upstream names into the
registry's scheme (including duplicating one asset across `-gnu`/`-musl`
with a `_comment`), so this fixes the data to match that convention
rather than adding an alias layer in the lookup.
This went unnoticed because `headroom/graph/installer.py` fetches the
same tool over its own code path, building the URL from upstream naming
directly — so the graph feature works and only the registry view
(`headroom tools doctor`) reports the tool as unavailable.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Refile the four existing `codebase-memory-mcp` assets under the keys
`PlatformKey.key()` actually emits. URLs and sha256 pins are unchanged,
and all four match the release's published `checksums.txt`.
- Add the assets the entry was missing: `linux-{x86_64,aarch64}-musl`
pointing at upstream's `-portable` builds, and `windows-x86_64`. The
entry carried `"_comment": "No Windows build published for v0.8.1."`,
but v0.8.1 publishes `codebase-memory-mcp-windows-amd64.zip`; that stale
comment is replaced.
- Add `test_every_registry_asset_key_is_reachable` — every asset key
must be one `PlatformKey.key()` can produce. This is the guard that
would have caught the bug.
- Add `test_asset_lookup_matches_declared_platforms` — a tool resolves
iff it declares that platform. It builds the `PlatformKey` from the
platform axes rather than by parsing the registry key, so a mis-keyed
asset cannot make it pass by round-tripping, and it forbids "fixing" a
future mis-key with a silent alias fallback instead of correcting the
table.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_binaries.py -q
24 passed, 1 warning in 1.84s
$ python -m pytest tests/test_binaries.py tests/test_graph.py tests/test_cli_tools.py \
tests/test_bundled_tools_savings.py -q
53 passed, 2 skipped, 1 warning in 2.07s
$ ruff check .
All checks passed!
$ ruff format --check tests/test_binaries.py
1 file already formatted
$ mypy headroom
Success: no issues found in 530 source files
```
Test fails before the fix, passes after — re-introducing the original
key in-process:
```text
$ python -c "import tests.test_binaries as t; from headroom import binaries
a = binaries._registry()['tools']['codebase-memory-mcp']['assets']
a['darwin-arm64'] = a.pop('darwin-aarch64')
t.test_every_registry_asset_key_is_reachable()"
AssertionError: codebase-memory-mcp: asset key 'darwin-arm64' is unreachable;
expected one of ['darwin-aarch64', 'darwin-x86_64', 'linux-aarch64-gnu',
'linux-aarch64-musl', 'linux-x86_64-gnu', 'linux-x86_64-musl', 'windows-x86_64']
```
## Real Behavior Proof
- Environment: macOS 26 (Darwin 27.0.0), Apple silicon →
`darwin-aarch64`; Python 3.13.13; branch = `upstream/main` + this
commit. "Before" numbers come from headroom-ai 0.37.0 installed from
PyPI in a separate site-packages, unpatched.
- Exact command / steps: ran `headroom tools doctor` and
`binaries.status()` against the unpatched installed 0.37.0, then
resolved all seven declared platforms against the patched tree; full
transcript below.
Before (unpatched 0.37.0, imported from outside the checkout):
```text
$ python -c "import headroom.binaries as b; print([r for r in b.status()
if r['tool']=='codebase-memory-mcp'])"
codebase-memory-mcp -> unsupported-platform
$ headroom tools doctor
│ codebase-memo… │ unsupported-pl… │ v0.8.1 │ darwin-aarch64 │ - │
codebase-memory-mcp: no prebuilt binary for darwin-aarch64;
supported: ['darwin-amd64', 'darwin-arm64', 'linux-amd64',
'linux-arm64']
```
After (patched tree), resolving every platform the registry now
declares:
```text
$ python -c "from headroom.binaries import PlatformKey,
_asset_for_platform; ..."
darwin-aarch64 -> codebase-memory-mcp-darwin-arm64.tar.gz
darwin-x86_64 -> codebase-memory-mcp-darwin-amd64.tar.gz
linux-x86_64-gnu -> codebase-memory-mcp-linux-amd64.tar.gz
linux-x86_64-musl -> codebase-memory-mcp-linux-amd64-portable.tar.gz
linux-aarch64-gnu -> codebase-memory-mcp-linux-arm64.tar.gz
linux-aarch64-musl -> codebase-memory-mcp-linux-arm64-portable.tar.gz
windows-x86_64 -> codebase-memory-mcp-windows-amd64.zip
```
- Observed result: the tool resolves on all seven declared platforms
instead of none, and no longer reports `unsupported-platform` on this
host. On my machine `status()` now reports `cached` — note that the
cached binary was already present from an earlier run, so `cached`
reflects the existing cache, not a download performed by this patch.
- Not tested: two gaps, detailed below.
- I did not download the newly added `-portable` or `windows-amd64`
archives locally; their pins were taken from the release's published
`checksums.txt`. CI's `tools-hash-refresh / verify-pins` has since
fetched and re-hashed every pin in this PR and passed, so the URLs and
digests are confirmed. What remains unverified is the `member` name
*inside* those three archives, which CI does not extract; they follow
the existing convention (`codebase-memory-mcp`,
`codebase-memory-mcp.exe`). A wrong `member` fails loudly and safely —
the pin is verified before extraction and `_extract_member_from_tar/zip`
raises `archive did not contain expected member`. Happy to drop those
hunks and land only the key fix if you would rather not take them.
- No musl or Windows host was available. Mapping the `-portable` build
to the `-musl` keys is inferred from upstream shipping it as a separate
variant; the release notes do not state its linking. This is recorded in
a `_comment` on those assets.
## Runtime Rollout Safety
- Rollout-managed feature(s): none — registry data plus tests.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: yes, in the sense that a tool that
previously never resolved now resolves. No proxy behavior, compression
path, or default is affected.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit; the registry returns to its prior
state.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
- No linked issue: `CONTRIBUTING.md` routes a bug fix with a repro and a
test straight to a PR.
- No other consumer regresses: `headroom/graph/installer.py` fetches
this tool over its own path and resolves its pin by URL via
`sha256_for_url()`, and no pre-existing URL changes here, so that lookup
still resolves (checked on this host).
- Docs checkbox is N/A — the platform key scheme is not user-facing
documentation; the reasoning is recorded in a `_comment` on the registry
entry instead.
- The two added tests are registry-wide, not specific to this tool, so
any future asset added under an upstream naming scheme fails at test
time rather than silently on every machine.
|
||
|
|
1f96dabc19 |
fix(security): address u9up assessment findings (WEB-01–07) (#2207)
Hardens client-selected upstreams, memory identity resolution, downloaded binary integrity, telemetry import, Docker defaults, Neo4j credentials, and archive extraction. Refreshes the branch against current main and preserves newer same-origin and loopback protections. |
||
|
|
93f2d7a2da |
chore: release main (#2792)
🤖 I have created a release *beep* *boop* --- <details><summary>0.35.0</summary> ## [0.35.0](https://github.com/headroomlabs-ai/headroom/compare/v0.34.0...v0.35.0) (2026-08-12) ### Features * **beacon:** allowlist the routing summary key ([#2818](https://github.com/headroomlabs-ai/headroom/issues/2818)) ([ |
||
|
|
78d935fcf9 |
fix: docker install fails with PermissionError in readonly cache dir
Two bugs collided to break the Docker-native install CI: 1. `ensure_tools()` ran unconditionally at every proxy startup, even when `--intercept-tool-results` was not passed. The feature is opt-in, so there's no reason to pay the binary-fetch cost (or risk a failure) when nothing will use them. 2. The fetch loop caught PlatformNotSupported / OfflineError / BinaryFetchError / Sha256Mismatch but not `PermissionError`. In containerized environments where the home dir / cache dir isn't writable, `binary_path.parent.mkdir()` raises PermissionError (subclass of OSError), which propagated out of ensure_tools() and crashed proxy startup. Fixes: - Move ensure_tools() inside the `if intercept_tool_results:` branch in cli/proxy.py so the base case never triggers a fetch. - Catch OSError (covers PermissionError, ENOSPC, etc.) in ensure_tools() so sandboxed / readonly filesystems degrade to no-op instead of crashing. Interceptors fall back to pass-through when their tool isn't resolvable. Adds regression test `test_ensure_tools_survives_readonly_cache_dir` that points the cache at a chmod-0500 parent and asserts ensure_tools() returns without raising. |
||
|
|
e165a579f8 |
fix(review): address PR #210 feedback — race, query-params, SHA logging, frozen prefix, etc
Addresses all 24 inline comments across the two review passes. **CRITICAL fixes:** - binaries.py: PID-scoped partial-file name prevents concurrent `headroom proxy` starts from clobbering each other's downloads. - binaries.py: strip URL query params before computing the download filename (was breaking archive-type detection for mirror URLs with `?token=...`). - cli/tools.py: `--force` cleanup now logs failures and bumps exit_code instead of silently swallowing exceptions. **HIGH fixes:** - binaries.py: log at INFO when SHA256 is unpinned; expose `sha_pinned` in doctor's status output. - proxy/interceptors/base.py: `_FAILURES` counter + `interceptor_failure_counts()` getter; incremented on every `matches()`/`transform()`/`key()` exception so dashboards can distinguish "nothing eligible" from "everything crashing". - cli/proxy.py: validate critical tools resolved when `--intercept-tool-results` is set; warn (don't fail) if a dependency is missing. - proxy/interceptors/base.py: compute `tokens_before` from the original messages via `count_messages()` instead of back-calculating from `tokens_after + sum(saved)` (which double-counted message-level overhead). - proxy/interceptors/astgrep.py: write untrusted tool_output into a private mode-0700 `tempfile.mkdtemp()` directory, not directly into shared `/tmp`. - proxy/interceptors/base.py: `ToolResultInterceptorTransform.apply()` now honors `frozen_message_count` — leading cached-prefix messages are passed through untouched to preserve provider prefix caches. **MEDIUM fixes:** - proxy/interceptors/base.py: pre-built O(1) tool_use index replaces the O(n²) per-tool-result linear scan. - proxy/interceptors/base.py: broken `progressive_disclosure_key()` now skips the interceptor entirely rather than firing without key protection. - proxy/interceptors/astgrep.py: distinguish ast-grep rc=1 (no matches) from rc>=2 (real errors — bad syntax, missing grammar, corrupt binary). - proxy/interceptors/astgrep.py: count JSON parse failures; warn when all lines fail to parse (indicates version mismatch). - binaries.py: musl detection falls back to checking `/lib/ld-musl-*.so.1` when `ldd` is absent (Alpine). - proxy/interceptors/astgrep.py: use `tempfile.mkdtemp()` + `shutil.rmtree` instead of `NamedTemporaryFile(delete=False)`; cleans up on Windows. - binaries.py: chmod failures on POSIX now log a warning (only swallow on Windows where .exe is implicitly executable). - tests/test_binaries.py: `test_mirror_substitution` now uses `monkeypatch.setenv()` instead of raw `os.environ` manipulation. - tools.json: add `linux-x86_64-musl` and `linux-aarch64-musl` entries for `difft`; document the shared-asset strategy for both tools. - proxy/interceptors/astgrep.py: log a debug line when `progressive_disclosure_key()` returns None for a tool whose tool_input shape we don't recognize. - proxy/interceptors/base.py: moved `import json` to module top (was inside `_find_tool_use` hot loop). - binaries.py: fix bare `.gz` detection — now explicitly excludes `.tar.gz`/`.tgz` instead of relying on a brittle "no dots" heuristic. - proxy/interceptors/astgrep.py: provenance comment on each `_RANGE_KEYS` entry so future maintainers know which tool defined which key. - cli/tools.py: comment explaining os.execv's lack of Python finalizer cleanup. - proxy/interceptors/base.py: `InterceptionResult` now `frozen=True`. **Test gaps closed:** - Interceptor failure isolation (transform() raises → request survives, counter increments). - Broken key() skips interceptor entirely. - Refuse-to-enlarge guard (rewrite larger than original → pass through). - Orphaned tool_result (no matching tool_use) doesn't crash. - ToolResultInterceptorTransform.apply() happy path + frozen_message_count. - ensure_tools() partial failure (one tool fetch fails, others succeed, proxy still starts). - Mirror URL with query params doesn't leak into download filename. 44 tests total; ruff + mypy clean. |
||
|
|
7cd67ef06d |
fix: bundle ast-grep/difftastic/scc + generic tool_result interceptor framework
What this does, in plain terms: Headroom's proxy now ships with three CLI tools (ast-grep, difftastic, scc) that it can use to shrink tool_result payloads before they reach the model. The goal is simple: when Claude Code (or Codex, Aider, etc.) asks the model to reason about a big file or diff, we swap the verbose output for a compact, same-meaning version. Fewer tokens per turn, same answers, lower bill. Today a single interceptor is wired: ast-grep on Read. When an agent reads a large code file, the proxy replaces the file body with an outline of its top-level functions/classes plus docstrings. In live tests that cut prompt tokens 74–76% on both OpenAI and Anthropic, same answer either way. How it works: - `pip install headroom-ai` now installs ast-grep via a PyPI wheel (core dep). difftastic and scc are fetched once at proxy startup from pinned upstream GitHub releases and cached per-user. - A generic registry (`headroom/proxy/interceptors/`) lets us add more tool-aware rewrites in one file each: declare `matches()` and `transform()`, call `register()`, done. No proxy or metrics plumbing per tool. - Safety rails built in: pass-through when a Read specifies a line range; second Read of the same file in a conversation returns full content (progressive disclosure); any failing interceptor logs and skips, never crashes a request. Opt-in for now: - Off by default while this ships. Turn on with `headroom proxy --intercept-tool-results` or `HEADROOM_INTERCEPT_ENABLED=1`, so we can measure before flipping defaults. What users see after turning it on: - First `headroom wrap claude` boot is ~5s longer (binaries fetched). Every subsequent run is cache-only. - Existing `transforms_applied` field in metrics gets entries like `interceptor:ast-grep`, so savings show up in current dashboards and HTML reports with no UI change. Other housekeeping in this PR: - uv.lock moved to .gitignore — regenerated locally per environment. - 35 unit + integration tests, ruff + mypy clean. - Dead-code audit done: removed `binaries.run()`, `needs_filesystem` plumbing, unused `_kind` tuple elements, unused `tool_output` parameter, and the never-set HEADROOM_SKIP_TOOLS_BOOTSTRAP env. |