Commit Graph
19 Commits
Author SHA1 Message Date
lorenzozaneandJerrettDavis a8905214d9 fix(packaging)!: keep PyTorch extras out of all (#4067)
<!-- maintainer-current-main -->
## Maintainer latest-main checkpoint
Published `fb146c0663b30488da0a49cfb33624ee82862074` contains current
upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. 22
dependency/Gemini tests and lock check passed. Earlier live behavior
proofs below retain their commit/platform scope; contributor runtime
logic is unchanged.

The common shard-2 failure was independently confirmed on main's hosted
job 127069149772 and locally (four failures, six passes): the Gemini
memory fixture mocked an unused body reader and provided no request
stream. The maintainer test-only follow-up now constructs a real
FastAPI/ASGI Request carrying JSON bytes and exercises the actual body
reader. All six test-function ASTs/assertions are unchanged, and all 10
parameterized Gemini cases pass. Touched-file Ruff/format and whitespace
checks pass. This fixes the inherited fixture instead of asking
contributors to run CI or weakening the memory/content assertions. Fresh
exact-head hosted qualification is requested. Nothing merged.
<!-- /maintainer-current-main -->

## Description



On Linux, installing `headroom-ai[all]` resolved PyTorch and NVIDIA CUDA
packages even for users who only needed the CLI/proxy profile. The
locked dependency graph included those packages through `ml`, `memory`,
`evals`, and `voice`.



This keeps those PyTorch-backed extras available for explicit
installation while removing them from `[all]`. Image support and the
other CLI/proxy dependencies remain included; dependency metadata cannot
detect installed GPU hardware.



Closes #3339



## Type of Change



- [ ] Bug fix (non-breaking change that fixes an issue)

- [ ] New feature (non-breaking change that adds functionality)

- [x] Breaking change (fix or feature that would cause existing
functionality to change)

- [ ] Documentation update

- [ ] Performance improvement

- [ ] Code refactoring (no functional changes)



## Changes Made



- Remove `ml`, `memory`, `evals`, and `voice` from the `[all]` extra
while keeping each separately installable.

- Retain `image` and the other default CLI/proxy capabilities.

- Update README and installation docs to describe the opt-in extras and
CUDA package implications.

- Add a dependency contract assertion for the `[all]` extra.



## Testing



- [ ] Unit tests pass (`pytest`)

- [ ] Linting passes (`ruff check .`)

- [ ] Type checking passes (`mypy headroom`)

- [ ] New tests added for new functionality

- [x] Manual testing performed



### Test Output



```text

Focused dependency contract tests: 3 passed, 5 deselected.

Python 3.13 locked `uv export` exited 0; the fixed export omits torch, CUDA, NVIDIA, sentence-transformers, and datasets. The baseline export contained them.

`uv lock --check`, scoped Ruff check/format, and `git diff --check` exited 0.

The broader dependency test module had 3 passed and 5 import failures because `opentelemetry` is absent from the local environment.

```



## Real Behavior Proof



- Environment: Linux, Python 3.13, locked uv dependency resolution.

- Exact command / steps: `UV_PYTHON_DOWNLOADS=never uv export --python
3.13 --extra all --no-dev --locked --no-hashes --no-emit-project`.

- Observed result: exit 0; the resolved export no longer contains torch,
CUDA, NVIDIA, sentence-transformers, or datasets, all of which were
present in the baseline export.

- Not tested: a full environment install and GPU runtime behavior.



## Runtime Rollout Safety



- Rollout-managed feature(s): None.

- Minimum rollout channel: Stable package release.

- Stable/default behavior changed: Yes; `[all]` no longer installs the
four PyTorch-backed extras.

- Kill switch / disable path: Not applicable; install the desired extras
explicitly.

- Unsafe override required: No.

- Qualification impact: Users of those capabilities must add the
corresponding extras.

- Rollback path: Restore the extras to `[all]` and regenerate `uv.lock`.



## Review Readiness



- [x] I have performed a self-review

- [x] This PR is ready for human review



## Checklist



- [x] My code follows the project's style guidelines

- [x] I have performed a self-review of my code

- [ ] I have commented my code, particularly in hard-to-understand areas

- [x] I have made corresponding changes to the documentation

- [ ] My changes generate no new warnings

- [x] I have added tests that prove my fix is effective or that my
feature works

- [ ] New and existing unit tests pass locally with my changes

- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)



## Screenshots (if applicable)



Not applicable to this dependency-resolution change.



## Additional Notes



The `[all]` install profile intentionally no longer includes
PyTorch-backed optional capabilities; each remains available through its
named extra. The focused regression and locked resolver proof passed,
but the full dependency test module and a full install were not verified
locally.





## Maintainer integration qualification

Published head `3229690b613810d496f015cedd086ca1d7421843` includes
current upstream main `31dcaf41`. The merge preserves main's `lean`
dependency profile and deprecated `sandbox` alias alongside the
contributor's opt-in `[all]` policy. No optional-feature runtime logic
was changed.



- Reconciled the existing macOS dependency test with the declared
policy, retaining the transitive exclusion check and independently
verifying explicit `ml`/`voice` opt-ins on Intel macOS and Linux. All 12
tests across `tests/test_onnx_dependency_contract.py` and
`tests/test_optional_dependencies.py` pass on Windows/Python 3.12.13.

- Evaluation workflow installs now explicitly request `evals`, and the
model-allowed weekly job also requests `ml`, preserving its previous
intended dependencies. The evaluation guide and AI install index
identify those opt-ins. These address the existing Devin workflow/docs
findings.

- `uv lock --check`, touched Python Ruff/format and whitespace checks
pass. All 253 locked package versions are unchanged; only the root
`[all]` dependency metadata/flattened list changes.

- Real resolver commands from this checkout: `uv export --extra all
--no-dev --locked --no-hashes --no-emit-project` and `uv export --extra
all --extra ml --extra voice --extra evals --no-dev --locked --no-hashes
--no-emit-project`. Evaluating their actual requirement markers for
Python 3.12/3.13/3.14 on Linux x86_64, Windows AMD64, macOS arm64 and
macOS x86_64 shows `[all]` excludes torch, sentence-transformers,
datasets and CUDA/NVIDIA packages in all 12 environments. Explicit
opt-ins retain datasets and torch where supported; Intel macOS still
excludes torch.

- Not tested: a fresh full environment install, GPU runtime, live paid
evaluations or a fresh native build. Hosted CI must qualify the
published final head separately.



### Maintainer current-main requalification

Exact head `f1ed1e0019037e7b53ff9ef548843449c3c7a16f`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Current-main integration against upstream
c01cf07d94. The sole conflict was the
inherited Gemini fixture: adopted main's streamed-body helper after
verifying all six test-function ASTs unchanged. No contributor
dependency policy rewritten. The declared breaking change keeps
Torch-backed ml/memory/evals/voice opt-in while preserving the default
CLI/proxy bundle, image capability and main's lean profile. Evaluation
workflows and guides explicitly request removed extras; current
author/Devin discussion considered.

Fresh verification: 25 dependency/Gemini tests pass; offline uv lock
check resolves 253 packages; changed Python lint/format and
main-relative whitespace pass. Actual locked exports evaluated over 12
OS/Python marker environments (Linux/Windows/macOS, Python 3.10–3.13):
all excludes Torch/NVIDIA dependencies, explicit
all+ml+memory+evals+voice retains Torch. This is resolver/export proof,
not a fresh full installation or GPU runtime claim. No new source
blocker found; formal approval awaits exact-head hosted checks. Nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-06 09:23:03 -06:00
2973ca593d ci: pin every GitHub Action to a commit SHA (#4671)
## Description

Every third-party action in `.github/` ran from a version tag like
`actions/checkout@v7`. A tag can be moved to other code at any time.
That is how `tj-actions/changed-files` leaked CI secrets from ~23,000
repos in March 2025.

Here a moved tag would run with the release job's PyPI OIDC token
(`id-token: write`) and the Docker job's `packages: write` token.
Pinning to a commit SHA closes that, and a new test keeps it closed.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- All 195 third-party `uses:` refs in 25 files now name the commit their
tag points to today, with the version as a comment:
`actions/checkout@3d3c42e5... # v7.0.1`. No action version changes.
- Dependabot's existing `github-actions` config updates SHA pins and
their version comments, so the pins stay current.
- Pin dtolnay/rust-toolchain stable and 1.95.0 branches to their
existing commit heads. The stable action retains its runtime toolchain
input/default; the 1.95.0 action embeds toolchain: 1.95.0. This
preserves each call site's existing toolchain selection.
- New `test_every_action_is_pinned_to_a_commit_sha` in
`scripts/tests/test_ci_workflow.py` fails on any tag-pinned action and
lists each one.
- Three tests in `tests/test_release_workflows.py` matched the old tag
text (`setup-buildx-action@v4`, `gh-action-pypi-publish@vX.Y.Z`,
`release-please-action@v4/v5`). They now match the SHA pin and its
version comment.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest -q scripts/tests/test_ci_workflow.py scripts/tests/test_pr_health_workflow.py scripts/tests/test_normalize_release_pr_bodies.py tests/test_release_workflows.py
87 passed in 0.87s

$ uv run --frozen --extra dev pytest -q scripts/tests/test_pr_health_labels.py scripts/tests/test_version_sync.py tests/test_cli/test_wrap_copilot.py tests/test_learn/test_subagent_scanning.py tests/test_plugin_manifests.py tests/test_pr208_changes.py
114 passed in 1.32s

$ uvx ruff check scripts/tests/test_ci_workflow.py tests/test_release_workflows.py && uvx ruff format --check <same>
All checks passed!
```

## Real Behavior Proof

- Environment: macOS, Python 3.12 via `uv run --frozen --extra dev`,
branch cut from `main` at c01cf07d.
- Exact command / steps: resolved each tag with `git ls-remote --tags
--heads https://github.com/<owner>/<repo>` (peeled for annotated tags),
rewrote only the `uses:` lines, then ran the test files above and a
`yaml.safe_load` over every file in `.github/`.
- Observed result: 195 lines changed, 195 removed, no other line touched
(mixed LF/CRLF in `pr-health.yml` kept). All 201 tests pass. On `main`
the new test fails and lists the unpinned refs.
- Not tested: `scripts/validate-workflows.sh` (actionlint plus act dry
runs) was not run locally. The `workflow-validation` job runs it on this
PR.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. CI workflow files and their tests
only.
- Minimum rollout channel: N/A, no runtime code changes.
- Stable/default behavior changed: No. Each SHA is the commit its tag
points to today.
- Kill switch / disable path: N/A, revert the commit.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A, no UI change.

## Additional Notes

- mypy and docs are N/A: no `headroom/` code or documented behavior
changes.
- Dependabot does not bump branch pins, so the two
`dtolnay/rust-toolchain` pins stay on today's branch head until someone
moves them. That action rarely changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4671"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `57b7edd8d05b6dc5e3bc4eac1dff55f21519afe8`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Reviewed the27-file delta and existing discussion/Devin no-issues result
against current upstream main c01cf07d94,
already included in the contributor head. Independent full-workflow
comparison proves only uses refs/comments change across the25
workflow/action files. All195 replaced refs resolve exactly to their
original upstream tag/branch commits (29 unique pins across26
repositories), with annotated tags peeled; no mismatches. Pinned dtolnay
action source independently confirms stable runtime default versus
embedded1.95.0 branch behavior, preserving toolchain selection.87
workflow/release tests and113 adjacent tests pass,1skip; changed Python
lint/format passes. Maintainer fix only removes CRLF endings on two
edited pr-health lines to clear whitespace; provenance reverified
afterward. No contributor workflow logic altered. Description is
corrected to explain embedded1.95.0 semantics. No publish/release
workflow execution claimed; full hosted qualification must run on this
new mechanical-fix head. Maintainers handle CI. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-06 09:16:15 -06:00
r3bb1tandJD Davis 87a4394d49 ci: run eval smoke test on Rust compressor changes (#2948)
## Description

`eval.yml`'s `pull_request` trigger watches only
`headroom/transforms/**`, `headroom/evals/**`, and
`headroom/compress.py`. But the eval smoke test installs the package
with `pip install -e ".[all]"`, which compiles the Rust workspace into
`headroom._core` and imports `SmartCrusher` from it — so the eval suite
exercises Rust compressor code, yet a PR that changes only `crates/**`
never runs it. This PR adds `crates/**`, `Cargo.toml`, and `Cargo.lock`
to the paths filter so Rust compressor changes get the same zero-cost
eval gate as their Python callers.

Related to #2949.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
- [x] CI/workflow change (no runtime code affected)

## Changes Made

- `.github/workflows/eval.yml`: added `crates/**`, `Cargo.toml`,
`Cargo.lock` to `pull_request.paths` (+3 lines; nothing else touched).

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -c "import yaml; yaml.safe_load(open('.github/workflows/eval.yml')); print('YAML is valid')"
YAML is valid

Trigger proof (fork): crates-only PR creates the "Evaluation Suite" check run
with this patch applied — https://github.com/r3bb1t/headroom/actions/runs/31614601951
```

## Real Behavior Proof

- Environment: Fork `r3bb1t/headroom` with GitHub Actions enabled;
Windows 11 host for git operations; upstream base `main`.
- Exact command / steps: Pushed this branch to the fork; created a
scratch branch on top adding one comment line to
`crates/headroom-core/src/lib.rs`; opened a draft PR on the fork
(crates-only diff) and observed which workflows were created.
- Observed result: The crates-only PR created the `Evaluation Suite`
check run (run 31614601951, link in Test Output) alongside the `rust`
workflow; under the pre-change filter the identical diff creates no
Evaluation Suite run.
- Not tested: The `weekly-suite` job (schedule/dispatch-only; path
filters do not affect it) and the live-eval steps requiring
`OPENAI_API_KEY` (skipped with a warning on forks without the secret, by
design).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — workflows-only change.

## Additional Notes

Unchecked checklist items are N/A for a 3-line workflows-only diff:
there is no code to comment, no docs impact, and no unit-testable
surface — the fork trigger-proof above is the behavioral test.
Branch-protection caveat (same one `rust.yml` documents in its header):
`eval.yml` uses a workflow-level `paths:` filter, so on non-matching PRs
the check run is never created (rather than created-then-skipped). If
`Evaluation Suite / smoke-test` is ever made a required check, the
filter should move to a `dorny/paths-filter` job-level gate like
`rust.yml`'s — out of scope here; happy to follow up if wanted.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2948"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 22:30:53 -06:00
Tejas ChopraandClaude Opus 5.5 a88e2d5145 fix(security): resolve 23 open CodeQL alerts (#4045)
## Description

CodeQL had 43 open alerts on `headroom` (19 high, 24 medium). We need 0
for SOC 2 (risk R-10: developers introduce vulnerabilities). This PR
fixes the 23 that are real problems in code we ship or run. The 16 in
`tests/` are handled by #4047. 11 were dismissed in GitHub with written
reasons: 7 stale alerts from the retired default-setup scan, and 4 false
positives (#21, #105, #109, #135).

**In plain words:**
1. **Error replies no longer include internal error text.** When a
request body couldn't be read, the proxy sent back Python's raw error
message. It now sends a fixed message that still says what kind of
problem it was.
2. **A benchmark no longer logs a possible API key.** `--model_args` is
shown as `<redacted>` in the log, but `lm_eval` still receives the real
value.
3. **A test no longer prints part of the Gemini API key.**
4. **Five workflows declare read-only permissions**, and CodeQL scans
GitHub Actions workflows again.
5. **The npm release scripts no longer pass arguments through Windows
`cmd.exe`**, which reinterprets `&` and `|` inside arguments, file paths
included.

Closes CodeQL alerts #20, #23, #24, #25, #26, #27, #28, #29, #31, #33,
#193 (stack-trace exposure), #177, #51, #145, #146, #147. The
workflow-permission fixes correspond to the dismissed #2, #7, #67, #85,
#86, #87, #125.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/helpers.py`:
- New `invalid_request_body_message(exc)` returns a fixed string per
error type and logs the type:
    - `…: malformed JSON`
    - `…: not valid UTF-8 (possibly compressed?)`
    - `…: must be a JSON object`
    - `…: too large`
    - plain `Invalid request body` for anything else
- New `RequestBodyNotObject(ValueError)` replaces a bare `ValueError`,
so every existing `except ValueError` still catches it.
- `headroom/proxy/handlers/{anthropic,openai,gemini,batch}.py`:
  - All 10 `f"Invalid request body: {e!s}"` replies now use the helper.
- The Google batch forwarder's 500 reply drops the exception text; it
was already logged on the line above.
- `headroom/evals/comprehensive_benchmark.py`: `--model_args` is
appended after the command is logged, and the log line shows it as
`<redacted>`.
- `tests/test_google_multimodal_e2e.py`: stops printing the first 10
characters of the API key.
-
`.github/workflows/{devcontainers,eval,init-e2e,init-native-e2e,wrap-e2e}.yml`:
top-level `permissions: contents: read`. None of these jobs write to the
repo; caching and artifact upload don't use the repo token.
- `.github/workflows/security.yml`: CodeQL matrix adds `actions`.
- `scripts/{build,verify}_npm_release_assets.mjs`:
- On Windows, npm runs as `node <npm-cli.js> …` instead of `cmd.exe /c
npm.cmd …`.
- `npm-cli.js` is found via `npm_execpath`, then next to node, then
beside each `npm.cmd` on PATH. This handles node and npm in different
directories, per Devin's review.
  - If none is found, it fails listing every path searched.
- New tests:
  - `tests/test_invalid_request_body_message.py`
- `tests/test_evals/test_comprehensive_benchmark_log_redaction.py`, per
Devanshi's review.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): one error remains, the
same one `main` has, in an untouched file. See notes.
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_release_workflows.py tests/test_invalid_request_body_message.py \
    tests/test_evals/test_comprehensive_benchmark_log_redaction.py
============================== 62 passed in 3.10s ==============================

# The redaction test against the pre-fix comprehensive_benchmark.py (origin/main):
========================= 1 failed, 1 passed in 0.50s ==========================
# With the fix:
============================== 2 passed in 0.57s ===============================

$ pytest -q (proxy/handler/batch/eval/request-body files, run one file at a time)
187 files: 182 OK; the 5 below fail or hang identically on unchanged origin/main:
  tests/test_proxy_uds.py                          11 failed  (macOS 104-byte sun_path limit)
  tests/test_anthropic_buffered_ccr_usage.py        6 failed
  tests/test_proxy_copilot_auth_hooks.py            2 failed
  tests/gateway/test_parity_proxy_vs_gateway.py     2 failed
  tests/test_cli_proxy_improvements.py             hangs >300s

$ ruff check . -q                      -> clean
$ ruff format --check headroom tests   -> clean
$ mypy headroom
headroom/integrations/langchain/chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context  [truthy-function]
Found 1 error in 1 file (checked 566 source files)   # identical on origin/main

$ node --check scripts/build_npm_release_assets.mjs && node --check scripts/verify_npm_release_assets.mjs  -> ok
```

CI: the `test (3)` failures (`tests/test_onnx_dependency_contract.py`, 5
tests) and `test (4)` failures (`tests/test_xai_cache_economics.py`, 5
tests) also fail on `main`'s latest CI run (run 37806990929), so they're
unrelated to this PR.

## Real Behavior Proof

- Environment: macOS 26 (arm64), Python 3.12.13, Node 24, npm 11.19.0.
Worktree of `origin/main` at 87e05b68 plus this branch.
- Exact command / steps: the three checks below.
1. **Error replies:** `invalid_request_body_message` was exercised with
each exception type, including a `JSONDecodeError` whose document is
`"secret-doc"`. Each case asserts the reply names the category and
contains neither the exception text nor the document.
2. **Benchmark:**
`run_lm_eval(model_args="model=gpt-4o-mini,api_key=sk-test-FAKE-…")` was
called with the subprocess call mocked. The log shows `--model_args
<redacted>` and not the fake key, while the mocked call received
`model=gpt-4o-mini,api_key=sk-test-FAKE-…,base_url=http://localhost:8787/v1`.
3. **Windows npm lookup:** `findNpmCli()` was extracted from each script
and run against a temporary layout reproducing Devin's case: node shim
in `dev/bin`, `npm.cmd` plus `node_modules/npm/bin/npm-cli.js` in
`Program Files/nodejs`, both on PATH.
- Observed result: all three Windows npm lookup cases pass in both
scripts (output below); checks 1 and 2 pass as shown in Test Output.
  ```text
  == scripts/build_npm_release_assets.mjs
  separate node/npm dirs -> found npm-cli.js beside npm.cmd on PATH
  npm_execpath set       -> uses npm_execpath
  missing npm            -> clear error: true
  == scripts/verify_npm_release_assets.mjs
  separate node/npm dirs -> found npm-cli.js beside npm.cmd on PATH
  npm_execpath set       -> uses npm_execpath
  missing npm            -> clear error: true
  ```
- Not tested: the two items below.
- A real Windows run of the release scripts. No Windows machine was
available, and CI runs these scripts on Ubuntu, where the code path is
unchanged (`npm` spawned directly, no shell).
- The new CodeQL `actions` scan. It runs on this PR's CI and on `main`
after merge.

### Maintainer verification at `128f271f` (2026-10-08)

- Environment: actual Windows 11, Python 3.12.13, Node 22.22.0, npm
10.9.4. Contributor logic unchanged; this head contains current `main`.
- Commands: the focused `test_invalid_request_body_message.py`,
`test_comprehensive_benchmark_log_redaction.py`, and
`test_release_workflows.py` suites; the existing Windows npm lookup
probe against both checked-in scripts; and a two-process-boundary
loopback uvicorn/httpx malformed-body probe against this head.
- Observed: **62 focused tests passed**. Both npm scripts invoked the
real npm CLI under Node with `shell:false`, including Node copied into a
separate `G:/tmp/headroom-review/resume49-node shim & literal` directory
while npm remained under `C:/Program Files/nodejs`; `npm_execpath` was
unset, so PATH lookup found the separate npm installation. npm reported
10.9.4 and a cache argument containing spaces and `&` was preserved
literally. **36 actual TCP requests** (four malformed-body categories
across nine endpoints) returned the expected fixed 400 messages, with no
private marker disclosed and no upstream requests. Only outbound
provider requests were replaced with a fail-if-called boundary; HTTP
parsing and proxy handlers were real.
- Hosted qualification at this exact head: **67 passed, 11 skipped**,
including all three CodeQL languages. Evidence is retained in the
maintainer review artifacts. The earlier simulated Windows-layout proof
remains below as historical author evidence.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, the text of 400 replies for
unreadable request bodies. It was `Invalid request body: <python
exception text>` and is now a fixed category message. Status codes,
error `type`/`code` fields and response shape are unchanged. Clients
that parse the free-text message, rather than the status and type, would
see different text.
- Kill switch / disable path: none needed; this is message text only.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (none
needed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- Review feedback addressed in 921de19b: 
- Devin: Windows npm lookup when node and npm are in different
directories.
  - Devanshi: benchmark redaction test.
- **#21 (`security_block` reply in `anthropic.py`) is left as is.** It
returns the security plugin's deliberate, user-facing reason for
blocking a request. It was dismissed as a false positive with that
reason.
- The `mypy` error is pre-existing: the same `truthy-function` error
appears on `origin/main`, in a file this PR doesn't touch.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 22:40:04 -05:00
JD Davis 04cdf79ab0 fix: stabilize release checks and consolidate dependency updates (#3531)
## Description

Consolidates the open dependency updates into one draft and fixes the
remaining release 0.38.0 test failures. Release packaging already
includes the merged Node 24 fix from #3516. The concurrency test now
proves request overlap with a barrier, and the release workflow tests
verify registry-range consistency and publication failure gating without
hard-coding obsolete dependency versions.

Updates npm, Cargo, Python, and GitHub Actions dependencies. Adds
recurring audits of all five npm lockfiles at every severity. Upgrades
CrewAI to remove its vulnerable json-repair 0.25.2 pin, and replaces
yanked chacha20 and pypdfium2 releases.

This remains a draft. All 67 hosted checks pass on 59854000c, including
CI, release dry-run, security scans, and end-to-end tests. Unpatched
optional ChromaDB/Accelerate vulnerabilities still prevent claiming that
all dependency security issues are fixed. No alerts are dismissed and no
integration is removed.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Upgrade OpenAI SDK / AI SDK development dependencies, Fumadocs
Twoslash, docs TypeScript, OpenCode Vitest, grouped npm dependencies,
and the wrap CLI pin.
- Upgrade Cargo's grouped dependencies, Redis to locked 1.7.0,
tree-sitter to 0.26.12, and chacha20 to 0.10.2.
- Upgrade Ruff to 0.16.4, Sentence Transformers to locked 6.0.1, CrewAI
to >=1.15.21 / json-repair 0.60.1, and pypdfium2 to 5.13.0.
- Consolidate checkout v7 and the Rust toolchain / PyPI publishing
action updates. Use Node 24 for OpenCode's Vitest 5 checks.
- Scope TypeScript 7 exceptions to the SDK and plugins whose tsup
declaration builds still require its legacy compiler API. Docs uses
TypeScript 7 successfully. Retain the Python tree-sitter-language-pack
1.x compatibility exception documented in #1216.
- Ignore only the reviewed unpatched ChromaDB/Accelerate update ranges,
leaving later releases eligible. Document all five distinct upstream
advisories in SECURITY.md (four currently have open repository
Dependabot alerts).

## Dependabot PR disposition

The dispositions below describe what this branch will supersede after
successful validation and merge. They do not authorize closing the PRs
before then. Future releases and newly disclosed advisories must remain
eligible for updates.

| PRs | Disposition |
| --- | --- |
| #3530, #3524 | @ai-sdk/openai 4.0.60 in SDK and docs |
| #3529, #3526, #3297 | openai 7.10.0 in SDK and docs |
| #3525 | fumadocs-twoslash 4.0.0 |
| #2278 | docs TypeScript 7.0.2 |
| #3528, #3527, #2282 | Bounded TypeScript 7 exception for tsup
consumers; TypeScript 7 declaration failure reproduced |
| #3523 | Grouped npm updates included |
| #3518 | Cargo grouped updates included |
| #3515 | Superseded secure wrap tree: OpenClaw 2026.9.3, Hono 4.13.7,
tar 7.5.22 |
| #3497 | OpenCode Vitest 5.0.0 |
| #3420 | TOML 4.3.0 already present |
| #3303 | All remaining checkout actions moved to v7 |
| #3299 | PyPI publish action 1.14.2; Rust uses @stable with explicit
1.95.0 input matching rust-toolchain.toml (1.100.0 downloads return 404,
and compiler versions are no longer action refs for Dependabot to
update) |
| #3292 | Sentence Transformers <7 constraint, locked 6.0.1 |
| #3291 | Bounded language-pack 1.x exception; incompatible parser API
documented in #1216 |
| #3290 | Ruff 0.16.4 in pyproject, lockfile, and pre-commit |
| #3159 | Rust tree-sitter 0.26.12, grammar versions unchanged |
| #3148 | Redis 1.x supported and locked at 1.7.0 |

## Testing

- [x] Unit tests pass (`pytest`) for the changed/tested areas below
- [x] Manual testing performed

### Test Output

- All five npm locks audit clean; changed npm trees re-audited after
major upgrades.
- SDK: typecheck, build, 294 tests passed / 33 external integration
tests skipped.
- OpenCode: typecheck, build, 17 tests passed; both rebuilt standalone
artifacts match the committed wheel bundles.
- OpenClaw: typecheck and build passed. Wrap CLIs installed and version
checks passed.
- Docs: fresh-container npm ci, typecheck, and production build passed
with TypeScript 7 and Twoslash 4 (164 pages), excluding all generated
caches. Updated Twoslash compiler options to its native string format
after hosted CI exposed the old numeric/filename configuration.
- Rust: core check with Redis enabled passed; 14 CCR backend tests
passed against a live isolated Redis, including round-trip and TTL
tests. All 30 code-compression parity fixtures matched. Other parity
categories passed or reported their existing unavailable
comparators/models.
- Cargo audit: zero vulnerabilities and warnings under the existing
repository policy; its existing unmaintained-paste exception is
unchanged.
- Python: all 50 release workflow tests plus embedder tests passed (62
passed, 3 MPS-only skips); all 12 CrewAI integration tests passed
against dependencies exported from the revised lockfile.
- Real Sentence Transformers 6.0.1 CPU embedding produced a (2, 384)
array; PDFium 5.13.0 rendered a 100x100 page.
- PyPI vulnerability metadata checked for all 288 registry
package/version pairs in uv.lock. Only ChromaDB and Accelerate remain
affected. The production pip-audit export also passed after the final
CrewAI-related lock refresh.
- Ruff 0.16.4, actionlint, uv lock --check, Dependabot directory
uniqueness, and git diff --check passed.
- Final combined release/concurrency suite: 76 passed. Strict
workspace/all-target Rust clippy with Redis enabled passed with -D
warnings.
- Independent read-only review found no important actionable issues
before pushing e5c542f57. Hosted CI then exposed unavailable Rust
1.100.0 downloads and obsolete Twoslash compiler options; both were
corrected in 59854000c. All 67 hosted checks passed on final commit
59854000c: CI run 34506787966 and release dry-run 34506788244 both
succeeded. All four Python shards passed; shard 1 reported 3,037 passed
/ 141 skipped. The docs build, Rust tests/parity/audit, all wheel import
checks, security scans, devcontainers, and Docker/native end-to-end
checks also passed.

## Real Behavior Proof

- Environment: local Windows/Python 3.12, Linux Node 24 containers, and
isolated Redis 7 container.
- Exact command / steps: npm package scripts; cargo test --locked -p
headroom-core --features redis --test ccr_backends with
HEADROOM_TEST_REDIS_URL set; cargo run --locked -p headroom-parity --
run --fixtures tests/parity/fixtures; pytest
tests/test_release_workflows.py and relevant embedder/CrewAI tests.
- Observed result: tests and builds above pass. Temporarily serializing
the overlap test causes TimeoutError; restoring unbounded mode passes
all 26 tests in that module.
- Not performed: publication or merge. Final hosted CI and release
dry-run both passed. MPS-only and external-service SDK tests were
skipped locally.

## Runtime Rollout Safety

- Rollout-managed feature(s): no new feature flags; dependency and test
changes.
- Minimum rollout channel: existing policy unchanged.
- Stable/default behavior changed: dependency versions updated; no
integration removed.
- Kill switch / disable path: existing feature controls unchanged.
- Unsafe override required: no.
- Qualification impact: hosted release, security, and end-to-end checks
passed on final head 59854000c. Unpatched optional-extra advisories
remain a security qualification blocker.
- Rollback path: revert the applicable commits.

## Review Readiness

- [x] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Unresolved upstream vulnerabilities: ChromaDB GHSA-f4j7-r4q5-qw2c,
GHSA-2wm9-hf6c-p5cr, GHSA-36p7-vc44-83pf, GHSA-xph7-9rjv-w5fr;
Accelerate GHSA-4j2p-28q2-5m79. Existing exposure restrictions are
mitigations, not fixes. Dependabot ignore rules cannot make these
dependencies vulnerability-free. Keep this draft open; do not merge
automatically.
2026-09-10 12:34:31 -05:00
dependabot[bot] 920a93bc2a ci: bump actions/setup-python from 5 to 7 (#3300)
Bumps [actions/setup-python](https://github.com/actions/setup-python)
from 5 to 7.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/setup-python/releases">actions/setup-python's
releases</a>.</em></p>
<blockquote>
<h2>v7.0.0</h2>
<h2>What's Changed</h2>
<h3>Enhancements</h3>
<ul>
<li>Migrate to ESM and upgrade dependencies by <a
href="https://github.com/priyagupta108"><code>@​priyagupta108</code></a>
in <a
href="https://redirect.github.com/actions/setup-python/pull/1330">actions/setup-python#1330</a></li>
<li>Pin SHA commits and update docs with latest versions by <a
href="https://github.com/HarithaVattikuti"><code>@​HarithaVattikuti</code></a>
in <a
href="https://redirect.github.com/actions/setup-python/pull/1338">actions/setup-python#1338</a></li>
<li>Remove the pip-install input by <a
href="https://github.com/gowridurgad"><code>@​gowridurgad</code></a> in
<a
href="https://redirect.github.com/actions/setup-python/pull/1336">actions/setup-python#1336</a></li>
</ul>
<h3>Bug Fix</h3>
<ul>
<li>Fix to Classify stderr warning messages as warnings instead of
errors in annotations by <a
href="https://github.com/lmvysakh"><code>@​lmvysakh</code></a> in <a
href="https://redirect.github.com/actions/setup-python/pull/1335">actions/setup-python#1335</a></li>
<li>Validate and retry manifest fetch to prevent silent failures by <a
href="https://github.com/priyagupta108"><code>@​priyagupta108</code></a>
in <a
href="https://redirect.github.com/actions/setup-python/pull/1332">actions/setup-python#1332</a></li>
</ul>
<h3>Dependency Upgrade</h3>
<ul>
<li>Bump certifi from 2020.6.20 to 2024.7.4 in
/<strong>tests</strong>/data by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a> in <a
href="https://redirect.github.com/actions/setup-python/pull/1328">actions/setup-python#1328</a></li>
<li>Remove EOL Python versions and Bumps numpy text fixture by <a
href="https://github.com/priya-kinthali"><code>@​priya-kinthali</code></a>
in <a
href="https://redirect.github.com/actions/setup-python/pull/1333">actions/setup-python#1333</a></li>
<li>Upgrade <code>@​actions/cache</code> to 6.2.0 by <a
href="https://github.com/philip-gai"><code>@​philip-gai</code></a> in <a
href="https://redirect.github.com/actions/setup-python/pull/1337">actions/setup-python#1337</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/lmvysakh"><code>@​lmvysakh</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/setup-python/pull/1335">actions/setup-python#1335</a></li>
<li><a
href="https://github.com/philip-gai"><code>@​philip-gai</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/setup-python/pull/1337">actions/setup-python#1337</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/setup-python/compare/v6...v7.0.0">https://github.com/actions/setup-python/compare/v6...v7.0.0</a></p>
<h2>v6.3.0</h2>
<h2>What's Changed</h2>
<h3>Enhancement</h3>
<ul>
<li>Add RHEL support and include Linux distro in cache keys by <a
href="https://github.com/priyagupta108"><code>@​priyagupta108</code></a>
in <a
href="https://redirect.github.com/actions/setup-python/pull/1323">actions/setup-python#1323</a></li>
<li>Fix pip cache error handling on Windows by <a
href="https://github.com/priyagupta108"><code>@​priyagupta108</code></a>
in <a
href="https://redirect.github.com/actions/setup-python/pull/1040">actions/setup-python#1040</a></li>
</ul>
<h3>Dependency update</h3>
<ul>
<li>Upgrade minimatch from 3.1.2 to 3.1.5 by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a> in <a
href="https://redirect.github.com/actions/setup-python/pull/1281">actions/setup-python#1281</a></li>
<li>Upgrade actions dependencies by <a
href="https://github.com/gowridurgad"><code>@​gowridurgad</code></a>
with <a href="https://github.com/Copilot"><code>@​Copilot</code></a> in
<a
href="https://redirect.github.com/actions/setup-python/pull/1303">actions/setup-python#1303</a></li>
<li>Upgrade <code>@​actions/cache</code> to 5.1.0, log cache write
denied by <a
href="https://github.com/jasongin"><code>@​jasongin</code></a> in <a
href="https://redirect.github.com/actions/setup-python/pull/1324">actions/setup-python#1324</a></li>
<li>Upgrade dependency versions and test workflow configuration by <a
href="https://github.com/HarithaVattikuti"><code>@​HarithaVattikuti</code></a>
in <a
href="https://redirect.github.com/actions/setup-python/pull/1322">actions/setup-python#1322</a></li>
</ul>
<h3>Documentation</h3>
<ul>
<li>Update advanced-usage.md by <a
href="https://github.com/Dunky-Z"><code>@​Dunky-Z</code></a> in <a
href="https://redirect.github.com/actions/setup-python/pull/811">actions/setup-python#811</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/gowridurgad"><code>@​gowridurgad</code></a>
with <a href="https://github.com/Copilot"><code>@​Copilot</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/setup-python/pull/1303">actions/setup-python#1303</a></li>
<li><a href="https://github.com/jasongin"><code>@​jasongin</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/setup-python/pull/1324">actions/setup-python#1324</a></li>
<li><a href="https://github.com/Dunky-Z"><code>@​Dunky-Z</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/setup-python/pull/811">actions/setup-python#811</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/setup-python/compare/v6.2.0...v6.3.0">https://github.com/actions/setup-python/compare/v6.2.0...v6.3.0</a></p>
<h2>v6.2.0</h2>
<h2>What's Changed</h2>
<h3>Dependency Upgrades</h3>
<ul>
<li>Upgrade dependencies to Node 24 compatible versions by <a
href="https://github.com/salmanmkc"><code>@​salmanmkc</code></a> in <a
href="https://redirect.github.com/actions/setup-python/pull/1259">actions/setup-python#1259</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/actions/setup-python/commit/5fda3b95a4ea91299a34e894583c3862153e4b97"><code>5fda3b9</code></a>
Pin SHA commits and update docs with latest versions (<a
href="https://redirect.github.com/actions/setup-python/issues/1338">#1338</a>)</li>
<li><a
href="https://github.com/actions/setup-python/commit/4ab7e95f05e168b4356aebde89dd84f59c283d8e"><code>4ab7e95</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/setup-python/issues/1337">#1337</a>
from actions/philip-gai/bump-actions-cache-6-2-0</li>
<li><a
href="https://github.com/actions/setup-python/commit/0f3a009f475dbea83c0371cd85d099690fee8c5c"><code>0f3a009</code></a>
Remove the pip-install input (<a
href="https://redirect.github.com/actions/setup-python/issues/1336">#1336</a>)</li>
<li><a
href="https://github.com/actions/setup-python/commit/f8cf4291c8b8e273ddd26e569454615c7315d932"><code>f8cf429</code></a>
Migrate to ESM and upgrade dependencies (<a
href="https://redirect.github.com/actions/setup-python/issues/1330">#1330</a>)</li>
<li><a
href="https://github.com/actions/setup-python/commit/54baeea5b34417d10a7479663a23cca53ea209b5"><code>54baeea</code></a>
Validate and retry manifest fetch to prevent silent failures (<a
href="https://redirect.github.com/actions/setup-python/issues/1332">#1332</a>)</li>
<li><a
href="https://github.com/actions/setup-python/commit/c7092773a316760f4ecfe498e4af668a4dafeac5"><code>c709277</code></a>
Annotation code fix (<a
href="https://redirect.github.com/actions/setup-python/issues/1335">#1335</a>)</li>
<li><a
href="https://github.com/actions/setup-python/commit/6849080452e69b330395e8a6d23cf90f56d76a1a"><code>6849080</code></a>
remove EOL Python versions and Bumps numpy text fixture (<a
href="https://redirect.github.com/actions/setup-python/issues/1333">#1333</a>)</li>
<li><a
href="https://github.com/actions/setup-python/commit/0903b469fbf4441aadfe4f4b249dc5b1fba3a73e"><code>0903b46</code></a>
Bump certifi from 2020.6.20 to 2024.7.4 in /<strong>tests</strong>/data
(<a
href="https://redirect.github.com/actions/setup-python/issues/1328">#1328</a>)</li>
<li><a
href="https://github.com/actions/setup-python/commit/ece7cb06caefa5fff74198d8649806c4678c61a1"><code>ece7cb0</code></a>
Fix pip cache error handling on Windows. (<a
href="https://redirect.github.com/actions/setup-python/issues/1040">#1040</a>)</li>
<li><a
href="https://github.com/actions/setup-python/commit/1d18d7af5f767c1259ede05a0a5bcc30f3dcf1cf"><code>1d18d7a</code></a>
Update advanced-usage.md (<a
href="https://redirect.github.com/actions/setup-python/issues/811">#811</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/actions/setup-python/compare/v5...v7">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/setup-python&package-manager=github_actions&previous-version=5&new-version=7)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-27 12:40:33 -05:00
46d4378cf7 feat(evals): weekly HotpotQA answer-recall report on the prose path (#1188)
## Description

Follow-up to **#1187** (the offline fidelity gate). That gate is
hermetic and **structured-only** (JSON tool outputs via Rust
compressors) so it can block every PR with zero setup. This PR adds the
genuinely-uncovered piece: **prose answer-recall on a real dataset
(HotpotQA)** in the **model-allowed weekly job**, where compression
routes through Kompress (ModernBERT).

> **Stacked on #1187.** Until that merges, this PR's diff shows its
commit too; it reduces to just `c71cc0cb` once #1187 lands. Please
review/merge #1187 first.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`CompressionOnlyRunner.evaluate_dataset_recall(suite)`**: for each
QA case, compress the supporting `context` via the production routing
path (`ContentRouter`) and check the `ground_truth` answer survives
(`compute_information_recall`). Counts only **probeable** cases — answer
literally present in the context and non-trivial (skips `yes/no`,
too-short) — so the aggregate is meaningful rather than inflated by
un-measurable cases.
- **`.github/workflows/eval.yml`**: a non-blocking step in the existing
`weekly-suite` job (schedule/manual only) drives it with
`load_hotpotqa(n=50)`. Defensive: a dataset download or model failure
emits `::warning::` and `|| true`, never failing the job.
- **Hermetic unit test** (`tests/test_dataset_recall_runner.py`):
exercises the method with synthetic JSON-array contexts (SmartCrusher /
Rust — no model, no network), so it runs in the standard `[dev]` shard.

### Scope notes

- **Prose path only.** BFCL / tool-schema integrity is already covered
by the existing `evaluate_tool_schema_compaction` eval (which runs in
the PR smoke-test), so this targets the previously-uncovered prose
recall path. NQ is an easy further extension using the same method +
`load_natural_questions`.
- **Why weekly, not per-PR.** Real datasets need a network download +
the ModernBERT model. The `weekly-suite` job already installs `[all]`
and genuinely runs every Monday (verified: 5 consecutive successful
scheduled runs), so it's the correct home — keeping PR CI fast and
hermetic.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ HF_HUB_OFFLINE=1 python -m pytest tests/test_dataset_recall_runner.py -q
..                                                                       [100%]
2 passed in 0.20s
```

## Real Behavior Proof

- Environment: local checkout of `feat/weekly-dataset-recall`, `pip
install -e ".[dev]"`, `HF_HUB_OFFLINE=1` (proves the unit tests need no
model/network)
- Exact command / steps: `HF_HUB_OFFLINE=1 python -m pytest
tests/test_dataset_recall_runner.py -q` -> `6 passed in 0.36s`; coverage
JSON confirms the runner's per-case exception handler and both
`warm_kompress_model` outcomes are exercised
- Observed result: with a synthetic suite of 3 cases (one probeable
answer in an error row, one trivial `yes`, one absent answer),
`evaluate_dataset_recall` counts only the 1 probeable case (`passed=1`,
`accuracy_rate=1.0`, `benchmark="dataset_recall:synthetic"`); a
monkeypatched compressor crash records the error and counts the case
failed instead of aborting; the new weekly-suite YAML step parses via
`yaml.safe_load` and sits under the `schedule || workflow_dispatch`
guard
- Not tested: the live HotpotQA download + ModernBERT compression --
exercised only by the weekly job (or `workflow_dispatch`), by design

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- CHANGELOG/version intentionally untouched: repo uses
**release-please**.
- The weekly job can be triggered on demand via **workflow_dispatch** to
see the HotpotQA recall numbers without waiting for Monday.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-07-15 21:40:55 +00:00
dependabot[bot] ebd23152d5 ci: bump actions/cache from 5 to 6 (#1413)
Bumps [actions/cache](https://github.com/actions/cache) from 5 to 6.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/cache/releases">actions/cache's
releases</a>.</em></p>
<blockquote>
<h2>v6.0.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Update packages, migrate to ESM by <a
href="https://github.com/Samirat"><code>@​Samirat</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1760">actions/cache#1760</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/cache/compare/v5...v6.0.0">https://github.com/actions/cache/compare/v5...v6.0.0</a></p>
<h2>v5.1.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Bump <code>@​actions/cache</code> to v5.1.0 - handle read-only cache
access by <a
href="https://github.com/jasongin"><code>@​jasongin</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1775">actions/cache#1775</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/cache/compare/v5...v5.1.0">https://github.com/actions/cache/compare/v5...v5.1.0</a></p>
<h2>v5.0.5</h2>
<h2>What's Changed</h2>
<ul>
<li>Update ts-http-runtime dependency by <a
href="https://github.com/yacaovsnc"><code>@​yacaovsnc</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1747">actions/cache#1747</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/cache/compare/v5...v5.0.5">https://github.com/actions/cache/compare/v5...v5.0.5</a></p>
<h2>v5.0.4</h2>
<h2>What's Changed</h2>
<ul>
<li>Add release instructions and update maintainer docs by <a
href="https://github.com/Link"><code>@​Link</code></a>- in <a
href="https://redirect.github.com/actions/cache/pull/1696">actions/cache#1696</a></li>
<li>Potential fix for code scanning alert no. 52: Workflow does not
contain permissions by <a
href="https://github.com/Link"><code>@​Link</code></a>- in <a
href="https://redirect.github.com/actions/cache/pull/1697">actions/cache#1697</a></li>
<li>Fix workflow permissions and cleanup workflow names / formatting by
<a href="https://github.com/Link"><code>@​Link</code></a>- in <a
href="https://redirect.github.com/actions/cache/pull/1699">actions/cache#1699</a></li>
<li>docs: Update examples to use the latest version by <a
href="https://github.com/XZTDean"><code>@​XZTDean</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1690">actions/cache#1690</a></li>
<li>Fix proxy integration tests by <a
href="https://github.com/Link"><code>@​Link</code></a>- in <a
href="https://redirect.github.com/actions/cache/pull/1701">actions/cache#1701</a></li>
<li>Fix cache key in examples.md for bun.lock by <a
href="https://github.com/RyPeck"><code>@​RyPeck</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1722">actions/cache#1722</a></li>
<li>Update dependencies &amp; patch security vulnerabilities by <a
href="https://github.com/Link"><code>@​Link</code></a>- in <a
href="https://redirect.github.com/actions/cache/pull/1738">actions/cache#1738</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/XZTDean"><code>@​XZTDean</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1690">actions/cache#1690</a></li>
<li><a href="https://github.com/RyPeck"><code>@​RyPeck</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1722">actions/cache#1722</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/cache/compare/v5...v5.0.4">https://github.com/actions/cache/compare/v5...v5.0.4</a></p>
<h2>v5.0.3</h2>
<h2>What's Changed</h2>
<ul>
<li>Bump <code>@actions/cache</code> to v5.0.5 (Resolves: <a
href="https://github.com/actions/cache/security/dependabot/33">https://github.com/actions/cache/security/dependabot/33</a>)</li>
<li>Bump <code>@actions/core</code> to v2.0.3</li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/cache/compare/v5...v5.0.3">https://github.com/actions/cache/compare/v5...v5.0.3</a></p>
<h2>v.5.0.2</h2>
<h1>v5.0.2</h1>
<h2>What's Changed</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/actions/cache/blob/main/RELEASES.md">actions/cache's
changelog</a>.</em></p>
<blockquote>
<h1>Releases</h1>
<h2>How to prepare a release</h2>
<blockquote>
<p>[!NOTE]
Relevant for maintainers with write access only.</p>
</blockquote>
<ol>
<li>Switch to a new branch from <code>main</code>.</li>
<li>Run <code>npm test</code> to ensure all tests are passing.</li>
<li>Update the version in <a
href="https://github.com/actions/cache/blob/main/package.json"><code>https://github.com/actions/cache/blob/main/package.json</code></a>.</li>
<li>Run <code>npm run build</code> to update the compiled files.</li>
<li>Update this <a
href="https://github.com/actions/cache/blob/main/RELEASES.md"><code>https://github.com/actions/cache/blob/main/RELEASES.md</code></a>
with the new version and changes in the <code>## Changelog</code>
section.</li>
<li>Run <code>licensed cache</code> to update the license report.</li>
<li>Run <code>licensed status</code> and resolve any warnings by
updating the <a
href="https://github.com/actions/cache/blob/main/.licensed.yml"><code>https://github.com/actions/cache/blob/main/.licensed.yml</code></a>
file with the exceptions.</li>
<li>Commit your changes and push your branch upstream.</li>
<li>Open a pull request against <code>main</code> and get it reviewed
and merged.</li>
<li>Draft a new release <a
href="https://github.com/actions/cache/releases">https://github.com/actions/cache/releases</a>
use the same version number used in <code>package.json</code>
<ol>
<li>Create a new tag with the version number.</li>
<li>Auto generate release notes and update them to match the changes you
made in <code>RELEASES.md</code>.</li>
<li>Toggle the set as the latest release option.</li>
<li>Publish the release.</li>
</ol>
</li>
<li>Navigate to <a
href="https://github.com/actions/cache/actions/workflows/release-new-action-version.yml">https://github.com/actions/cache/actions/workflows/release-new-action-version.yml</a>
<ol>
<li>There should be a workflow run queued with the same version
number.</li>
<li>Approve the run to publish the new version and update the major tags
for this action.</li>
</ol>
</li>
</ol>
<h2>Changelog</h2>
<h3>6.1.0</h3>
<ul>
<li>Bump <code>@actions/cache</code> to v6.1.0 to pick up <a
href="https://redirect.github.com/actions/toolkit/pull/2435">actions/toolkit#2435
Handle cache write error due to read-only token</a></li>
<li>Switch redundant &quot;Cache save failed&quot; warning to debug log
in save-only</li>
</ul>
<h3>6.0.0</h3>
<ul>
<li>Updated <code>@actions/cache</code> to ^6.0.1,
<code>@actions/core</code> to ^3.0.1, <code>@actions/exec</code> to
^3.0.0, <code>@actions/io</code> to ^3.0.2</li>
<li>Migrated to ESM module system</li>
<li>Upgraded Jest to v30 and test infrastructure to be ESM
compatible</li>
</ul>
<h3>5.0.4</h3>
<ul>
<li>Bump <code>minimatch</code> to v3.1.5 (fixes ReDoS via globstar
patterns)</li>
<li>Bump <code>undici</code> to v6.24.1 (WebSocket decompression bomb
protection, header validation fixes)</li>
<li>Bump <code>fast-xml-parser</code> to v5.5.6</li>
</ul>
<h3>5.0.3</h3>
<ul>
<li>Bump <code>@actions/cache</code> to v5.0.5 (Resolves: <a
href="https://github.com/actions/cache/security/dependabot/33">https://github.com/actions/cache/security/dependabot/33</a>)</li>
<li>Bump <code>@actions/core</code> to v2.0.3</li>
</ul>
<h3>5.0.2</h3>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/actions/cache/commit/55cc8345863c7cc4c66a329aec7e433d2d1c52a9"><code>55cc834</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/cache/issues/1768">#1768</a>
from jasongin/readonly-cache</li>
<li><a
href="https://github.com/actions/cache/commit/d8cd72f230726cdf4457ebb61ec1b593a8d12337"><code>d8cd72f</code></a>
Bump <code>@​actions/cache</code> to v6.1.0 - handle cache write error
due to RO token</li>
<li><a
href="https://github.com/actions/cache/commit/2c8a9bd7457de244a408f35966fab2fb45fda9c8"><code>2c8a9bd</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/cache/issues/1760">#1760</a>
from actions/samirat/esm_migration_and_package_update</li>
<li><a
href="https://github.com/actions/cache/commit/e9b91fdc3fea7d79165fceb79042ef45c2d51023"><code>e9b91fd</code></a>
Prettier fixes</li>
<li><a
href="https://github.com/actions/cache/commit/e4884b8ff7f92ef6b52c79eda480bbc86e685adb"><code>e4884b8</code></a>
Rebuild dist</li>
<li><a
href="https://github.com/actions/cache/commit/10baf0191a3c426ea0fa4a3253a5c04233b6e18f"><code>10baf01</code></a>
Fixed licenses</li>
<li><a
href="https://github.com/actions/cache/commit/e39b386c9004d72a15d864ade8c0b3a702d47a37"><code>e39b386</code></a>
Fix test mock return order</li>
<li><a
href="https://github.com/actions/cache/commit/b6928203372a8571ff984c0c883ef3a1adfb0c06"><code>b692820</code></a>
PR feedback</li>
<li><a
href="https://github.com/actions/cache/commit/60749128a44d25d3c520a489e576380cf00ff3f1"><code>6074912</code></a>
Rebuild dist bundles as ESM to match type:module</li>
<li><a
href="https://github.com/actions/cache/commit/5a912e8b4af820fa082a0e75cfd2c782f8fbfe0e"><code>5a912e8</code></a>
Fix lint and jest issues</li>
<li>Additional commits viewable in <a
href="https://github.com/actions/cache/compare/v5...v6">compare
view</a></li>
</ul>
</details>
<br />

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-09 13:32:20 -05:00
dependabot[bot] bfe4b4d21f ci: bump actions/checkout from 4 to 7 (#1414)
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to
7.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/checkout/releases">actions/checkout's
releases</a>.</em></p>
<blockquote>
<h2>v7.0.0</h2>
<h2>What's Changed</h2>
<ul>
<li>block checking out fork pr for pull_request_target and workflow_run
by <a href="https://github.com/aiqiaoy"><code>@​aiqiaoy</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2454">actions/checkout#2454</a></li>
<li>Bump actions/publish-immutable-action from 0.0.3 to 0.0.4 in the
minor-actions-dependencies group across 1 directory by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/checkout/pull/2458">actions/checkout#2458</a></li>
<li>Bump flatted from 3.3.1 to 3.4.2 by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/checkout/pull/2460">actions/checkout#2460</a></li>
<li>Bump js-yaml from 4.1.0 to 4.2.0 by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/checkout/pull/2461">actions/checkout#2461</a></li>
<li>Bump <code>@​actions/core</code> and
<code>@​actions/tool-cache</code> and Remove uuid by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/checkout/pull/2459">actions/checkout#2459</a></li>
<li>upgrade module to esm and update dependencies by <a
href="https://github.com/aiqiaoy"><code>@​aiqiaoy</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2463">actions/checkout#2463</a></li>
<li>Bump the minor-npm-dependencies group across 1 directory with 3
updates by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/checkout/pull/2462">actions/checkout#2462</a></li>
<li>getting ready for checkout v7 release by <a
href="https://github.com/aiqiaoy"><code>@​aiqiaoy</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2464">actions/checkout#2464</a></li>
<li>update error wording by <a
href="https://github.com/aiqiaoy"><code>@​aiqiaoy</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2467">actions/checkout#2467</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/aiqiaoy"><code>@​aiqiaoy</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/checkout/pull/2454">actions/checkout#2454</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/checkout/compare/v6.0.3...v7.0.0">https://github.com/actions/checkout/compare/v6.0.3...v7.0.0</a></p>
<h2>v6.0.3</h2>
<h2>What's Changed</h2>
<ul>
<li>Update changelog by <a
href="https://github.com/ericsciple"><code>@​ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2357">actions/checkout#2357</a></li>
<li>fix: expand merge commit SHA regex and add SHA-256 test cases by <a
href="https://github.com/yaananth"><code>@​yaananth</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2414">actions/checkout#2414</a></li>
<li>Fix checkout init for SHA-256 repositories by <a
href="https://github.com/yaananth"><code>@​yaananth</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2439">actions/checkout#2439</a></li>
<li>Update changelog for v6.0.3 by <a
href="https://github.com/yaananth"><code>@​yaananth</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2446">actions/checkout#2446</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/yaananth"><code>@​yaananth</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/checkout/pull/2414">actions/checkout#2414</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/checkout/compare/v6...v6.0.3">https://github.com/actions/checkout/compare/v6...v6.0.3</a></p>
<h2>v6.0.2</h2>
<h2>What's Changed</h2>
<ul>
<li>Add orchestration_id to git user-agent when ACTIONS_ORCHESTRATION_ID
is set by <a
href="https://github.com/TingluoHuang"><code>@​TingluoHuang</code></a>
in <a
href="https://redirect.github.com/actions/checkout/pull/2355">actions/checkout#2355</a></li>
<li>Fix tag handling: preserve annotations and explicit fetch-tags by <a
href="https://github.com/ericsciple"><code>@​ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2356">actions/checkout#2356</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/checkout/compare/v6.0.1...v6.0.2">https://github.com/actions/checkout/compare/v6.0.1...v6.0.2</a></p>
<h2>v6.0.1</h2>
<h2>What's Changed</h2>
<ul>
<li>Update all references from v5 and v4 to v6 by <a
href="https://github.com/ericsciple"><code>@​ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2314">actions/checkout#2314</a></li>
<li>Add worktree support for persist-credentials includeIf by <a
href="https://github.com/ericsciple"><code>@​ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2327">actions/checkout#2327</a></li>
<li>Clarify v6 README by <a
href="https://github.com/ericsciple"><code>@​ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2328">actions/checkout#2328</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/checkout/compare/v6...v6.0.1">https://github.com/actions/checkout/compare/v6...v6.0.1</a></p>
<h2>v6.0.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Update README to include Node.js 24 support details and requirements
by <a href="https://github.com/salmanmkc"><code>@​salmanmkc</code></a>
in <a
href="https://redirect.github.com/actions/checkout/pull/2248">actions/checkout#2248</a></li>
<li>Persist creds to a separate file by <a
href="https://github.com/ericsciple"><code>@​ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2286">actions/checkout#2286</a></li>
<li>v6-beta by <a
href="https://github.com/ericsciple"><code>@​ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2298">actions/checkout#2298</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/actions/checkout/blob/main/CHANGELOG.md">actions/checkout's
changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<h2>v7.0.0</h2>
<ul>
<li>Block checking out fork PR for pull_request_target and workflow_run
by <a href="https://github.com/aiqiaoy"><code>@​aiqiaoy</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2454">actions/checkout#2454</a></li>
<li>Bump actions/publish-immutable-action from 0.0.3 to 0.0.4 in the
minor-actions-dependencies group across 1 directory by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/checkout/pull/2458">actions/checkout#2458</a></li>
<li>Bump flatted from 3.3.1 to 3.4.2 by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/checkout/pull/2460">actions/checkout#2460</a></li>
<li>Bump js-yaml from 4.1.0 to 4.2.0 by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/checkout/pull/2461">actions/checkout#2461</a></li>
<li>Bump <code>@​actions/core</code> and
<code>@​actions/tool-cache</code> and Remove uuid by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/checkout/pull/2459">actions/checkout#2459</a></li>
<li>upgrade module to esm and update dependencies by <a
href="https://github.com/aiqiaoy"><code>@​aiqiaoy</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2463">actions/checkout#2463</a></li>
<li>Bump the minor-npm-dependencies group across 1 directory with 3
updates by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/checkout/pull/2462">actions/checkout#2462</a></li>
</ul>
<h2>v6.0.3</h2>
<ul>
<li>Fix checkout init for SHA-256 repositories by <a
href="https://github.com/yaananth"><code>@​yaananth</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2439">actions/checkout#2439</a></li>
<li>fix: expand merge commit SHA regex and add SHA-256 test cases by <a
href="https://github.com/yaananth"><code>@​yaananth</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2414">actions/checkout#2414</a></li>
</ul>
<h2>v6.0.2</h2>
<ul>
<li>Fix tag handling: preserve annotations and explicit fetch-tags by <a
href="https://github.com/ericsciple"><code>@​ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2356">actions/checkout#2356</a></li>
</ul>
<h2>v6.0.1</h2>
<ul>
<li>Add worktree support for persist-credentials includeIf by <a
href="https://github.com/ericsciple"><code>@​ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2327">actions/checkout#2327</a></li>
</ul>
<h2>v6.0.0</h2>
<ul>
<li>Persist creds to a separate file by <a
href="https://github.com/ericsciple"><code>@​ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2286">actions/checkout#2286</a></li>
<li>Update README to include Node.js 24 support details and requirements
by <a href="https://github.com/salmanmkc"><code>@​salmanmkc</code></a>
in <a
href="https://redirect.github.com/actions/checkout/pull/2248">actions/checkout#2248</a></li>
</ul>
<h2>v5.0.1</h2>
<ul>
<li>Port v6 cleanup to v5 by <a
href="https://github.com/ericsciple"><code>@​ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2301">actions/checkout#2301</a></li>
</ul>
<h2>v5.0.0</h2>
<ul>
<li>Update actions checkout to use node 24 by <a
href="https://github.com/salmanmkc"><code>@​salmanmkc</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2226">actions/checkout#2226</a></li>
</ul>
<h2>v4.3.1</h2>
<ul>
<li>Port v6 cleanup to v4 by <a
href="https://github.com/ericsciple"><code>@​ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2305">actions/checkout#2305</a></li>
</ul>
<h2>v4.3.0</h2>
<ul>
<li>docs: update README.md by <a
href="https://github.com/motss"><code>@​motss</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1971">actions/checkout#1971</a></li>
<li>Add internal repos for checking out multiple repositories by <a
href="https://github.com/mouismail"><code>@​mouismail</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1977">actions/checkout#1977</a></li>
<li>Documentation update - add recommended permissions to Readme by <a
href="https://github.com/benwells"><code>@​benwells</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2043">actions/checkout#2043</a></li>
<li>Adjust positioning of user email note and permissions heading by <a
href="https://github.com/joshmgross"><code>@​joshmgross</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2044">actions/checkout#2044</a></li>
<li>Update README.md by <a
href="https://github.com/nebuk89"><code>@​nebuk89</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2194">actions/checkout#2194</a></li>
<li>Update CODEOWNERS for actions by <a
href="https://github.com/TingluoHuang"><code>@​TingluoHuang</code></a>
in <a
href="https://redirect.github.com/actions/checkout/pull/2224">actions/checkout#2224</a></li>
<li>Update package dependencies by <a
href="https://github.com/salmanmkc"><code>@​salmanmkc</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2236">actions/checkout#2236</a></li>
</ul>
<h2>v4.2.2</h2>
<ul>
<li><code>url-helper.ts</code> now leverages well-known environment
variables by <a href="https://github.com/jww3"><code>@​jww3</code></a>
in <a
href="https://redirect.github.com/actions/checkout/pull/1941">actions/checkout#1941</a></li>
<li>Expand unit test coverage for <code>isGhes</code> by <a
href="https://github.com/jww3"><code>@​jww3</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1946">actions/checkout#1946</a></li>
</ul>
<h2>v4.2.1</h2>
<ul>
<li>Check out other refs/* by commit if provided, fall back to ref by <a
href="https://github.com/orhantoy"><code>@​orhantoy</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1924">actions/checkout#1924</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/actions/checkout/commit/9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0"><code>9c091bb</code></a>
update error wording (<a
href="https://redirect.github.com/actions/checkout/issues/2467">#2467</a>)</li>
<li><a
href="https://github.com/actions/checkout/commit/1044a6dea927916f2c38ba5aeffbc0a847b1221a"><code>1044a6d</code></a>
getting ready for checkout v7 release (<a
href="https://redirect.github.com/actions/checkout/issues/2464">#2464</a>)</li>
<li><a
href="https://github.com/actions/checkout/commit/f0282184c7ce73ab54c7e4ab5a617122602e575f"><code>f028218</code></a>
Bump the minor-npm-dependencies group across 1 directory with 3 updates
(<a
href="https://redirect.github.com/actions/checkout/issues/2462">#2462</a>)</li>
<li><a
href="https://github.com/actions/checkout/commit/d914b262ffc244530a203ab40decab34c3abf34d"><code>d914b26</code></a>
upgrade module to esm and update dependencies (<a
href="https://redirect.github.com/actions/checkout/issues/2463">#2463</a>)</li>
<li><a
href="https://github.com/actions/checkout/commit/537c7ef99cef6e5ddb5e7ff5d16d14510503801d"><code>537c7ef</code></a>
Bump <code>@​actions/core</code> and <code>@​actions/tool-cache</code>
and Remove uuid (<a
href="https://redirect.github.com/actions/checkout/issues/2459">#2459</a>)</li>
<li><a
href="https://github.com/actions/checkout/commit/130a169078a413d3a5246a393625e8e742f387f6"><code>130a169</code></a>
Bump js-yaml from 4.1.0 to 4.2.0 (<a
href="https://redirect.github.com/actions/checkout/issues/2461">#2461</a>)</li>
<li><a
href="https://github.com/actions/checkout/commit/7d09575332117a40b46e5e020664df234cd416f3"><code>7d09575</code></a>
Bump flatted from 3.3.1 to 3.4.2 (<a
href="https://redirect.github.com/actions/checkout/issues/2460">#2460</a>)</li>
<li><a
href="https://github.com/actions/checkout/commit/0f9f3aa320cb53abeb534aeb54048075d9697a0e"><code>0f9f3aa</code></a>
Bump actions/publish-immutable-action (<a
href="https://redirect.github.com/actions/checkout/issues/2458">#2458</a>)</li>
<li><a
href="https://github.com/actions/checkout/commit/f9e715a95fcd1f9253f77dd28f11e88d2d6460c7"><code>f9e715a</code></a>
block checking out fork pr for pull_request_target and workflow_run (<a
href="https://redirect.github.com/actions/checkout/issues/2454">#2454</a>)</li>
<li><a
href="https://github.com/actions/checkout/commit/df4cb1c069e1874edd31b4311f1884172cec0e10"><code>df4cb1c</code></a>
Update changelog for v6.0.3 (<a
href="https://redirect.github.com/actions/checkout/issues/2446">#2446</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/actions/checkout/compare/v4...v7">compare
view</a></li>
</ul>
</details>
<br />

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-09 11:09:14 -05:00
AshishandClaude Opus 4.8 23d73ae070 test(evals): add offline fidelity regression gate (recall-based, zero-model) (#1187)
## Description

Headroom's lossy compression drops rows/lines using statistical
heuristics but **never checks that meaning survived** — a dropped `OOM
killed worker 3` line can silently flip a model's answer with no signal
that compression caused it. The repo already ships a quality-metric
toolkit (`headroom/evals/metrics.py`) and a `weekly-suite` eval job, but
neither gates the compression path on a PR.

This adds a **per-PR fidelity regression gate**: compress vendored
golden tool-outputs through SmartCrusher's lossy path and assert the
evidence that answers each case's question survives. It is the first of
a planned trio (this is the "offline gate" half of the fidelity work);
query-aware retention and a hard token-budget API are documented
follow-ups.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Blocking gate** (`tests/test_compression_fidelity_regression.py`):
compresses each golden case via `smart_crush_tool_output(...,
with_compaction=False)` and scores with `compute_information_recall`.
Two assertions:
- **Per-case critical recall == 1.0** — every `answer_evidence` string
(placed in error/anomaly rows, the documented SmartCrusher retention
guarantee) must survive.
- **Aggregate recall ≥ committed baseline** (`baseline.json`, tol 0.02)
— catches softer regressions.
- **Vendored fixtures** (`tests/fixtures/fidelity_golden/`):
deterministic `_generate.py` emits `cases.json` (4 cases: OOM crash,
payment exception, latency anomaly, CI failure) + `baseline.json`.
- **Non-blocking weekly report** (`.github/workflows/eval.yml`): one
step in the existing `weekly-suite` job (schedule/manual only) reuses
the existing `evaluate_information_retention` runner for a recall report
on the production routing path.
- **Pure reuse**: scoring (`evals/metrics.py`), compressor
(`smart_crush_tool_output`), and the weekly runner
(`evaluate_information_retention`) all already existed.

### Design notes

- **Zero new CI setup.** The blocking gate runs in the existing `[dev]`
test shard — no new workflow, no new deps, **no model, no network, no
secrets** (verified under `HF_HUB_OFFLINE=1`). It deliberately uses
small hand-made structured fixtures rather than the repo's HuggingFace
dataset loaders, which would require a network download + ModernBERT and
don't belong in a fast PR gate.
- **Scope:** structured JSON tool-output (the dominant, deterministic,
model-free case). Real-dataset (HotpotQA/BFCL) recall — which needs
`[all]` + a local model — is a **documented follow-up PR**, and the
`weekly-suite` job (which genuinely runs every Monday) is its natural
home.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ HF_HUB_OFFLINE=1 python -m pytest tests/test_compression_fidelity_regression.py -v
tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[logs_oom] PASSED
tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[payment_exception] PASSED
tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[latency_anomaly] PASSED
tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[ci_test_failures] PASSED
tests/test_compression_fidelity_regression.py::test_aggregate_recall_not_regressed PASSED
============================== 5 passed in 0.18s ===============================
```

## Real Behavior Proof

- **Environment:** local checkout of `feat/fidelity-regression-gate`,
`pip install -e ".[dev]"`, `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1`
(proves no model/network).
- **Exact command / steps:** `HF_HUB_OFFLINE=1 python -m pytest
tests/test_compression_fidelity_regression.py -q` → `5 passed in 0.14s`.
- **Negative control (proves the gate has teeth):** compressing
`logs_oom` and probing for a benign row that compression legitimately
drops returns `recall = 0.00, lost = ['heartbeat ping 25']` — i.e. the
gate fires when critical evidence is dropped, so it is not trivially
green.
- **Weekly (non-blocking) step verified locally:**
  ```text
  Information retention: 50/50 cases >=0.9 recall, avg compression 65.7%
  ```
- **Not tested:** real-dataset (HotpotQA/BFCL) recall and
prose/ModernBERT compression — intentionally deferred to a follow-up PR
targeting the weekly job.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- CHANGELOG/version intentionally untouched: repo uses
**release-please**.
- **Follow-up PR (planned):** wire the real HotpotQA/BFCL loaders
(`headroom/evals/datasets.py`) into the `weekly-suite` job for genuine
benchmark-scale recall coverage (model-allowed, non-blocking). Further
follow-ups from the same design: a live per-request fidelity guardrail,
query-aware lossy retention, and a hard `target_tokens` budget API.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 22:53:59 -05:00
dependabot[bot]andJerrettDavis e408012c2b ci: bump dtolnay/rust-toolchain from 1.95.0 to 1.100.0 in the actions-minor-patch group (#849)
Bumps the actions-minor-patch group with 1 update:
[dtolnay/rust-toolchain](https://github.com/dtolnay/rust-toolchain).

Updates `dtolnay/rust-toolchain` from 1.95.0 to 1.100.0
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/dtolnay/rust-toolchain/commit/4a76a4951e2b37e999824e644cd130c67037ee2b"><code>4a76a49</code></a>
toolchain: 1.100.0</li>
<li>See full diff in <a
href="https://github.com/dtolnay/rust-toolchain/compare/1.95.0...1.100.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=dtolnay/rust-toolchain&package-manager=github_actions&previous-version=1.95.0&new-version=1.100.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-06-11 19:44:15 -05:00
AshishandClaude Sonnet 4.6 53a08c63bf feat(evals): add zero-cost tool schema compaction integrity eval (#817)
## Summary

- Adds `evaluate_tool_schema_compaction()` and
`generate_tool_schema_cases()` to `CompressionOnlyRunner`
- Four built-in cases cover the property-name vs annotation-key
distinction: `title`, `deprecated`, `readOnly`, and all four at once
- Each case asserts: byte count shrinks (annotations stripped), all
`must_preserve` property names survive in `properties`, no `required`
entry points to a stripped key, root-level schema annotations
(`$schema`, `title`) are dropped
- Wires the new eval into `.github/workflows/eval.yml` alongside the
existing CCR round-trip smoke step — runs on every PR touching
`headroom/transforms/**`, `headroom/evals/**`, or
`headroom/compress.py`, at zero API cost

## Motivation

PR #785 fixed a bug where the compaction pass stripped property *names*
that happened to match DROP_KEYS (e.g. a field literally called
`title`). This eval encodes the invariant that fix established so future
changes to the compaction logic can't silently regress it.

## Test plan

- [ ] `pytest
tests/test_evals_metrics.py::test_tool_schema_compaction_integrity` —
all 4 cases pass, `total_tokens_saved > 0`
- [ ] CI smoke step "Run tool schema compaction integrity eval (zero
cost)" passes with no API key required

## Real behavior proof

```
$ pytest tests/test_evals_metrics.py::test_tool_schema_compaction_integrity -v
PASSED [100%]
1 passed in 0.53s
```

Zero API calls, zero cost. Runs in under 1 second.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-10 16:03:42 -05:00
Patrick A 53d2342291 ci: speed up GitHub Actions — path filters, caching, timeouts, version upgrades (#620)
* ci: speed up GitHub Actions - path filters, caching, timeouts, version upgrades

Performance improvements:
- init-e2e.yml, wrap-e2e.yml: add path filters so e2e Docker builds only run when
  e2e-related files change (saves ~10 min per irrelevant PR push)
- init-e2e.yml, wrap-e2e.yml: add concurrency groups to cancel superseded PR runs
- ci.yml: add pip caching to lint and build jobs
- ci.yml: cache actionlint + act binaries in workflow-validation (skip curl on hits)
- eval.yml: add pip caching to smoke-test and weekly-suite jobs
- docs.yml: add pip caching for mkdocs-material install
- rust.yml: replace cargo install --locked cargo-audit/deny with taiki-e/install-action
  (prebuilt binaries; saves 2-5 min per audit run)

Bug fixes:
- docker.yml: fix actions/checkout@v6 -> @v4 (v6 does not exist; would break all
  Docker builds on every release/PR touching docker paths)

Version upgrades:
- wagoid/commitlint-github-action: @v5 -> @v6
- devcontainers.yml: docker/setup-buildx-action@v3 -> @v4 (align with docker.yml)

Safety improvements:
- ci.yml: add timeout-minutes to all 13 jobs (changes, lint, build-wheel,
  prefetch-model, test x4, test-extras, test-agno, commitlint, build,
  workflow-validation, docker-native-e2e, windows-native-wrapper, macos-native-wrapper)
- docker.yml: add timeout-minutes to docker-build (75m), docker-manifest (20m),
  promote-latest (10m)
- eval.yml: add timeout-minutes to smoke-test (30m); bump weekly-suite 60->90m
- rust.yml: add timeout-minutes to test (30m), wheels (45m), audit (20m)

Observed wall-clock impact on recent PRs:
- Init E2E and Wrap E2E were running on every single PR push regardless of content
- CI workflow was taking 12-17 min; path filters reduce unnecessary e2e runs to 0

* fix(ci): bust actionlint+act cache when workflow file changes

Static cache key 'ci-tools-actionlint-act-v1' never invalidated on
tool version updates. Switched to hashFiles('.github/workflows/ci.yml')
so the cache busts automatically whenever the download scripts are
updated to point at a newer release.

Flagged by adversarial review (Architecture + Testing/Reliability personas).

* fix(ci): add missing Dockerfile COPY paths to e2e path filters

e2e/init/Dockerfile and e2e/wrap/Dockerfile COPY files not covered
by the initial path filter set:

  init-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
            .claude-plugin, .github/plugin/**, plugins/headroom-agent-hooks/**
  wrap-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
            sdk/typescript/**, plugins/openclaw/**

Without these, a Rust toolchain bump or SDK change on a PR would
skip the e2e gate entirely, only catching it on the merge to main.

Flagged by adversarial review (Domain/Correctness persona).

* fix(devcontainer): upgrade uv to >=0.7.0 to parse uv.lock revision=3

* fix(devcontainer): set UV_SKIP_WHEEL_FILENAME_CHECK=1 in post-create.sh for gitpython wheel

* ci: bump actions/checkout and actions/setup-node to v5 (Node.js 20 EOL Jun 16)

* fix(devcontainer): export UV_SKIP_WHEEL_FILENAME_CHECK so uv run also skips wheel check

* ci: bump all GitHub Actions to latest versions (Node.js 24)

* fix(test): accept release-please-action v4 or v5 in workflow assertion

* fix(format): ruff format test_release_workflows.py
2026-06-05 14:32:53 -08:00
JerrettDavis 2fd48260f6 ci: skip live evals without credentials 2026-05-11 13:35:24 -05:00
chopratejas a281de6bb0 fix(ci): skip smoke OpenAI eval cleanly when OPENAI_API_KEY secret unset
The smoke-test job in .github/workflows/eval.yml has been silently
broken on every PR that touches headroom/transforms/**, evals/**, or
compress.py. Root cause: the public chopratejas/headroom repo has zero
Actions secrets configured (`gh api repos/chopratejas/headroom/actions/secrets`
returns `{"total_count":0,"secrets":[]}`), so OPENAI_API_KEY resolves
to the empty string and `openai.OpenAI()` raises OpenAIError at client
init before any compression code runs.

Fix: gate the live OpenAI eval step on a non-empty key and emit a
GitHub `::warning::` annotation when skipped, so the skip is loud in
the run summary (per the project's "no silent fallbacks" rule). The
CCR round-trip step above remains the mandatory gate — it tests real
compression logic with zero external dependencies.

Operators who DO wire OPENAI_API_KEY as a repo secret get the live
eval as before.

Test plan:
- yaml syntax validated locally
- Will re-run on PR #400 push; expect smoke-test to pass with the
  ::warning:: annotation visible in the run summary.

Refs: F2.1 (unblocks merge of #400)
2026-05-05 18:07:15 -07:00
chopratejas 2a91cbb4b4 refactor: single-wheel maturin build backend (fixes #355)
Eliminates the dual-package architecture that was the root cause of #355.
`pip install headroom-ai` now produces ONE wheel containing both the Python
source (headroom/*.py) and the compiled Rust extension (headroom/_core.so).
No more separate `headroom-core-py` package, no more chicken-and-egg with
PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action
plumbing in CI.

This is the canonical pattern used by cryptography, polars, ruff,
pydantic-core, and other Rust-as-core Python packages. Honors the
"Rust as core engine" direction.

## What changed

- pyproject.toml: `[build-system]` swapped from hatchling to maturin.
  `[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at
  `crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."`
  picks up the root `headroom/` package directly (dashboard HTML
  templates and other non-Python files included automatically).
- crates/headroom-py/pyproject.toml: deleted. The crate is no longer a
  separate published package; its Cargo.toml stays as the cdylib build
  target invoked via `[tool.maturin] manifest-path`.
- crates/headroom-py/python/: deleted (placeholder layout for the old
  separate package).

## CI updates

- ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust
  toolchain set up before `pip install -e .` (which now invokes maturin
  via build-system). Removed the "build wheel + symlink .so" dance.
  `build` job swapped from `python -m build` (hatch) to
  `maturin build` + `maturin sdist`.
- release.yml: collapsed dual-package matrix into one. New `build-wheels`
  matrix produces cross-platform wheels for cp310/11/12/13 ×
  {linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New
  `collect-dist` aggregator merges artifacts. publish-pypi consumes the
  merged dist.
- init-native-e2e.yml: dropped windows-latest from the matrix —
  upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting
  MSVC C runtime libraries, so the Rust extension cannot build for
  win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS.
- headroom-e2e-setup: composite action now sets up Rust toolchain +
  Swatinem/rust-cache before `pip install -e .[proxy]`.
- eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before
  install. rust.yml's wheels job builds from root pyproject.toml (no
  more `-m crates/headroom-py/Cargo.toml`).
- e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in
  the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the
  install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false`
  from wrap-e2e — the image now ships the full Rust core.
- Dockerfile (main): simplified — no more Layer 2/3 dance with
  `headroom-core-py` install + symlink. Single `uv pip install` builds
  + installs everything.
- .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin
  added so `uv sync` builds the extension inside the devcontainer.

## Lockfile + script

- uv.lock: regenerated. No `headroom-core-py` entries remain.
- scripts/build_rust_extension.sh: simplified from a symlink-into-tree
  workaround to a thin wrapper around `pip install -e .`. The maturin
  build-backend handles placement automatically.

## Local validation (all green on macOS aarch64)

1. Clean venv `pip install -e .` → `from headroom._core import …` works.
2. `maturin build --release` → 13.8 MB wheel, 336 files including
   `headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and
   `headroom/dashboard/templates/dashboard.html`.
3. `pip install <wheel>` in fresh venv → import works.
4. Wheel contents verified via `unzip -l`.
5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed.
6. `pytest tests/test_relevance.py` — 30 passed.
7. `cargo build --workspace` + `cargo test --workspace` — all green.
8. `make ci-precheck` — 176 Python tests + Rust + commitlint green.

## Migration notes

Users on `pip install headroom-ai` get the Rust core automatically
(linux + macos wheels). sdist installs require rust toolchain available
locally — pip will build via maturin.

Closes #355
Supersedes #357 (workarounds-based fix abandoned in favor of
architectural fix)
2026-05-03 13:16:41 -07:00
chopratejas beec0789ed fix(ci): pin dtolnay/rust-toolchain to 1.95.0 to match rust-toolchain.toml
The action was set to @stable, which installs whatever the latest
stable is (1.95.0 right now). Then maturin invokes cargo, which reads
rust-toolchain.toml and re-resolves to "1.95.0 + clippy + rustfmt".
rustup treats stable and 1.95.0 as distinct toolchain identities and
refuses the second install with:

  failed to install component 'clippy-preview-x86_64-unknown-linux-gnu',
  detected conflict: 'bin/cargo-clippy'

This was intermittent across the matrix (only test (3.10) tripped on
the most recent run; others got lucky on cache state). Pinning the
action ref to 1.95.0 makes both sides ask for the exact same toolchain
identity, so the second install is a no-op and the conflict can't fire.

Bump procedure stays the same: when rust-toolchain.toml's channel
changes, update these refs in lock-step.

Plugin manifests auto-bumped 0.11.0 -> 0.13.2 by sync-plugin-versions
hook (unrelated to the workflow fix).
2026-04-27 19:44:39 -07:00
chopratejas d6a00ee89c ci: fix smart_crusher branch CI failures + add make ci-precheck pre-push gate
Five gates broke on the 2026-04-27 push of the smart_crusher branch.
Each is fixed below; the second half adds a `make ci-precheck` target
(plus an installable git pre-push hook) so the same dance never happens
again.

Failures fixed:

1. cargo fmt — 22 files had formatting drift introduced over the
   stage 3c.1 work. `cargo fmt --all` reformatted them; no semantic
   changes. `cargo test --workspace` still green (388 + supporting).

2. wheels job (macOS x86_64) — `fastembed -> ort -> ort-sys` does not
   publish prebuilt ONNX Runtime binaries for `x86_64-apple-darwin`.
   Removed that target from `.github/workflows/rust.yml`'s wheels
   matrix. Apple Silicon (`aarch64-apple-darwin`) covers macOS
   distribution; Intel macOS users can build from source. The matrix
   now has 2 targets: linux x86_64 + macOS aarch64.

3. test-extras (relevance.py) — `tests/test_relevance.py::TestSmartCrusherIntegration`
   constructs a `SmartCrusher`, which hard-imports `headroom._core`
   since the python implementation was retired in stage 3c.1b. The
   test-extras job didn't build the rust extension. Added the same
   `maturin build + symlink` block the main `test` job uses.

4. smoke-test (eval.yml) — same root cause:
   `compression_only.evaluate_ccr_lossless` instantiates a SmartCrusher.
   Same fix: build the rust extension before the smoke test runs.

5. commitlint — three rules tripped:
   - `subject-case` rejects PascalCase identifiers in subjects, but
     the project deliberately names classes (SmartCrusher, HfTokenizer,
     ContentRouter, DiffCompressor) in commit subjects. Disabled.
   - `footer-leading-blank` is a warning that the wagoid action turns
     into a CI failure; lines like `Module: foo.rs` in our bodies
     match the conventional footer pattern and trip it. Disabled.
   - `type-enum` doesn't include `parity`, but the project ships
     parity-test infrastructure as its own concern (separate from
     `test:`); added `parity` to the allowed types.

Pre-push verification — the prevention half:

`make ci-precheck` runs all of the above CI gates locally:
- `ci-precheck-rust`: cargo fmt --check + clippy + test --workspace.
- `ci-precheck-python`: builds the rust extension via maturin, then
  runs the smart_crusher-affected python test files (185 tests across
  test_transforms/, test_relevance*, test_ccr, test_acceptance,
  test_critical_fixes, test_quality_retention).
- `ci-precheck-commitlint`: `npx commitlint --from origin/main --to
  HEAD` against the same config CI uses. Skipped silently if npx is
  not on PATH (install Node 18+ to enable).

`make install-git-hooks` (or `scripts/install-git-hooks.sh`) installs
a git pre-push hook that runs `make ci-precheck` automatically.
Bypass with `--no-verify` only when truly needed.

When new CI gates land in `.github/workflows/`, mirror them into a
`make ci-precheck-*` target. The Makefile is the local mirror of the
CI configuration; keeping them in sync is a load-bearing invariant.

Verification: `make ci-precheck` runs green on this commit.
2026-04-27 11:13:47 -07:00
chopratejas a2b2ec5463 Add OSS evaluation suite, universal JSON crush, latency benchmarks
Evaluation Suite:
- Tiered eval framework (Tier 1 ~$3/15min, Tier 2/3 for extended coverage)
- 16 benchmarks across 3 tiers: GSM8K, TruthfulQA, MMLU, ARC, HumanEval,
  SQuAD v2, BFCL, Tool Outputs, CCR needle retention, HotpotQA, and more
- Before/After runner with full proxy support (compression + CCR retrieval)
- LLM-as-judge for ground-truth comparison (BFCL function calling)
- Zero-cost compression-only runner (CCR needle retention, info retention)
- Cost tracker with per-model pricing and budget enforcement
- Report card generator (Markdown, JSON, HTML)
- Suite CLI: python -m headroom.evals suite --tier 1
- Fix BFCL dataset loader for current HuggingFace schema
- CI workflow: PR smoke test + weekly full Tier 1

Results: SQuAD 97%, BFCL 97%, Tool Outputs 100%, CCR 100%

SmartCrusher:
- Universal JSON crush for heterogeneous arrays
- Fix mypy redefinition warning in _crush_string_array

Other:
- Latency benchmark suite with docs
- Known limitations doc
- Prompt comparison evaluator
- Config updates for new features
2026-02-23 19:08:54 -08:00