Files
headroom/.github/workflows/eval.yml
lorenzozaneandJerrettDavis a8905214d9 fix(packaging)!: keep PyTorch extras out of all (#4067)
<!-- maintainer-current-main -->
## Maintainer latest-main checkpoint
Published `fb146c0663b30488da0a49cfb33624ee82862074` contains current
upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. 22
dependency/Gemini tests and lock check passed. Earlier live behavior
proofs below retain their commit/platform scope; contributor runtime
logic is unchanged.

The common shard-2 failure was independently confirmed on main's hosted
job 127069149772 and locally (four failures, six passes): the Gemini
memory fixture mocked an unused body reader and provided no request
stream. The maintainer test-only follow-up now constructs a real
FastAPI/ASGI Request carrying JSON bytes and exercises the actual body
reader. All six test-function ASTs/assertions are unchanged, and all 10
parameterized Gemini cases pass. Touched-file Ruff/format and whitespace
checks pass. This fixes the inherited fixture instead of asking
contributors to run CI or weakening the memory/content assertions. Fresh
exact-head hosted qualification is requested. Nothing merged.
<!-- /maintainer-current-main -->

## Description



On Linux, installing `headroom-ai[all]` resolved PyTorch and NVIDIA CUDA
packages even for users who only needed the CLI/proxy profile. The
locked dependency graph included those packages through `ml`, `memory`,
`evals`, and `voice`.



This keeps those PyTorch-backed extras available for explicit
installation while removing them from `[all]`. Image support and the
other CLI/proxy dependencies remain included; dependency metadata cannot
detect installed GPU hardware.



Closes #3339



## Type of Change



- [ ] Bug fix (non-breaking change that fixes an issue)

- [ ] New feature (non-breaking change that adds functionality)

- [x] Breaking change (fix or feature that would cause existing
functionality to change)

- [ ] Documentation update

- [ ] Performance improvement

- [ ] Code refactoring (no functional changes)



## Changes Made



- Remove `ml`, `memory`, `evals`, and `voice` from the `[all]` extra
while keeping each separately installable.

- Retain `image` and the other default CLI/proxy capabilities.

- Update README and installation docs to describe the opt-in extras and
CUDA package implications.

- Add a dependency contract assertion for the `[all]` extra.



## Testing



- [ ] Unit tests pass (`pytest`)

- [ ] Linting passes (`ruff check .`)

- [ ] Type checking passes (`mypy headroom`)

- [ ] New tests added for new functionality

- [x] Manual testing performed



### Test Output



```text

Focused dependency contract tests: 3 passed, 5 deselected.

Python 3.13 locked `uv export` exited 0; the fixed export omits torch, CUDA, NVIDIA, sentence-transformers, and datasets. The baseline export contained them.

`uv lock --check`, scoped Ruff check/format, and `git diff --check` exited 0.

The broader dependency test module had 3 passed and 5 import failures because `opentelemetry` is absent from the local environment.

```



## Real Behavior Proof



- Environment: Linux, Python 3.13, locked uv dependency resolution.

- Exact command / steps: `UV_PYTHON_DOWNLOADS=never uv export --python
3.13 --extra all --no-dev --locked --no-hashes --no-emit-project`.

- Observed result: exit 0; the resolved export no longer contains torch,
CUDA, NVIDIA, sentence-transformers, or datasets, all of which were
present in the baseline export.

- Not tested: a full environment install and GPU runtime behavior.



## Runtime Rollout Safety



- Rollout-managed feature(s): None.

- Minimum rollout channel: Stable package release.

- Stable/default behavior changed: Yes; `[all]` no longer installs the
four PyTorch-backed extras.

- Kill switch / disable path: Not applicable; install the desired extras
explicitly.

- Unsafe override required: No.

- Qualification impact: Users of those capabilities must add the
corresponding extras.

- Rollback path: Restore the extras to `[all]` and regenerate `uv.lock`.



## Review Readiness



- [x] I have performed a self-review

- [x] This PR is ready for human review



## Checklist



- [x] My code follows the project's style guidelines

- [x] I have performed a self-review of my code

- [ ] I have commented my code, particularly in hard-to-understand areas

- [x] I have made corresponding changes to the documentation

- [ ] My changes generate no new warnings

- [x] I have added tests that prove my fix is effective or that my
feature works

- [ ] New and existing unit tests pass locally with my changes

- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)



## Screenshots (if applicable)



Not applicable to this dependency-resolution change.



## Additional Notes



The `[all]` install profile intentionally no longer includes
PyTorch-backed optional capabilities; each remains available through its
named extra. The focused regression and locked resolver proof passed,
but the full dependency test module and a full install were not verified
locally.





## Maintainer integration qualification

Published head `3229690b613810d496f015cedd086ca1d7421843` includes
current upstream main `31dcaf41`. The merge preserves main's `lean`
dependency profile and deprecated `sandbox` alias alongside the
contributor's opt-in `[all]` policy. No optional-feature runtime logic
was changed.



- Reconciled the existing macOS dependency test with the declared
policy, retaining the transitive exclusion check and independently
verifying explicit `ml`/`voice` opt-ins on Intel macOS and Linux. All 12
tests across `tests/test_onnx_dependency_contract.py` and
`tests/test_optional_dependencies.py` pass on Windows/Python 3.12.13.

- Evaluation workflow installs now explicitly request `evals`, and the
model-allowed weekly job also requests `ml`, preserving its previous
intended dependencies. The evaluation guide and AI install index
identify those opt-ins. These address the existing Devin workflow/docs
findings.

- `uv lock --check`, touched Python Ruff/format and whitespace checks
pass. All 253 locked package versions are unchanged; only the root
`[all]` dependency metadata/flattened list changes.

- Real resolver commands from this checkout: `uv export --extra all
--no-dev --locked --no-hashes --no-emit-project` and `uv export --extra
all --extra ml --extra voice --extra evals --no-dev --locked --no-hashes
--no-emit-project`. Evaluating their actual requirement markers for
Python 3.12/3.13/3.14 on Linux x86_64, Windows AMD64, macOS arm64 and
macOS x86_64 shows `[all]` excludes torch, sentence-transformers,
datasets and CUDA/NVIDIA packages in all 12 environments. Explicit
opt-ins retain datasets and torch where supported; Intel macOS still
excludes torch.

- Not tested: a fresh full environment install, GPU runtime, live paid
evaluations or a fresh native build. Hosted CI must qualify the
published final head separately.



### Maintainer current-main requalification

Exact head `f1ed1e0019037e7b53ff9ef548843449c3c7a16f`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Current-main integration against upstream
c01cf07d94. The sole conflict was the
inherited Gemini fixture: adopted main's streamed-body helper after
verifying all six test-function ASTs unchanged. No contributor
dependency policy rewritten. The declared breaking change keeps
Torch-backed ml/memory/evals/voice opt-in while preserving the default
CLI/proxy bundle, image capability and main's lean profile. Evaluation
workflows and guides explicitly request removed extras; current
author/Devin discussion considered.

Fresh verification: 25 dependency/Gemini tests pass; offline uv lock
check resolves 253 packages; changed Python lint/format and
main-relative whitespace pass. Actual locked exports evaluated over 12
OS/Python marker environments (Linux/Windows/macOS, Python 3.10–3.13):
all excludes Torch/NVIDIA dependencies, explicit
all+ml+memory+evals+voice retains Torch. This is resolver/export proof,
not a fresh full installation or GPU runtime claim. No new source
blocker found; formal approval awaits exact-head hosted checks. Nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-06 09:23:03 -06:00

189 lines
8.9 KiB
YAML

name: Evaluation Suite
on:
schedule:
- cron: '0 6 * * 1' # Weekly on Monday 6am UTC
workflow_dispatch: # Manual trigger
pull_request:
paths:
- 'headroom/transforms/**'
- 'headroom/evals/**'
- 'headroom/compress.py'
- 'crates/**' # Rust compressors are compiled into headroom._core
- 'Cargo.toml' # and exercised by the eval smoke test (SmartCrusher
- 'Cargo.lock' # import in the smoke-test job); Rust-only PRs currently skip it
permissions:
contents: read
jobs:
# Fast smoke test on PRs touching compression code (~$0.05, ~2 min)
smoke-test:
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.11"
- name: Cache pip
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-eval-${{ hashFiles('pyproject.toml') }}
restore-keys: ${{ runner.os }}-pip-eval-
# `pip install -e .` invokes maturin (declared in pyproject.toml's
# build-system) which calls cargo to compile the Rust extension.
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@b725eda525362e7cf34af9a7243cac1e3f01a7ec # stable
with:
toolchain: 1.95.0 # Keep in sync with rust-toolchain.toml.
- name: Cache cargo registry + build
uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
workspaces: ". -> target"
- name: Install dependencies (builds Rust extension via maturin)
run: |
pip install -e ".[all,evals]"
python -c "from headroom._core import SmartCrusher; print('headroom._core OK:', SmartCrusher)"
- name: Run CCR round-trip (zero cost)
run: |
python -c "
from headroom.evals.runners.compression_only import CompressionOnlyRunner
runner = CompressionOnlyRunner()
cases = runner.generate_ccr_test_cases(n=50)
result = runner.evaluate_ccr_lossless(cases)
print(f'CCR Round-trip: {result.passed_cases}/{result.total_cases} passed')
assert result.passed, f'CCR failures: {result.errors}'
"
- name: Run tool schema compaction integrity eval (zero cost)
run: |
python -c "
from headroom.evals.runners.compression_only import CompressionOnlyRunner
runner = CompressionOnlyRunner()
result = runner.evaluate_tool_schema_compaction()
print(f'Tool schema compaction: {result.passed_cases}/{result.total_cases} passed, {result.total_tokens_saved} annotation tokens stripped')
assert result.passed, f'Schema compaction failures: {result.errors}'
"
# OPENAI_API_KEY is intentionally not set in the public OSS repo
# (the secret list is empty). The CCR round-trip step above is the
# mandatory gate; this step only runs when an operator has wired
# OPENAI_API_KEY as a repo secret (e.g. on a downstream fork). When
# missing, emit a loud GitHub `::warning::` annotation so the skip
# is visible in the run summary — never a silent pass.
- name: Run built-in tool output eval (skipped when OPENAI_API_KEY unset)
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
if [ -z "${OPENAI_API_KEY}" ]; then
echo "::warning title=Smoke eval skipped::OPENAI_API_KEY is not configured for this repo; only the CCR round-trip gate ran. Wire the secret to enable the live OpenAI eval."
exit 0
fi
python -m headroom.evals quick -n 8 --provider openai --model gpt-4o-mini
# Full Tier 1 suite, weekly or manual (~$3-5, ~30-45 min)
weekly-suite:
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
timeout-minutes: 90
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.11"
- name: Cache pip
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-eval-${{ hashFiles('pyproject.toml') }}
restore-keys: ${{ runner.os }}-pip-eval-
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@b725eda525362e7cf34af9a7243cac1e3f01a7ec # stable
with:
toolchain: 1.95.0 # Keep in sync with rust-toolchain.toml.
- name: Cache cargo registry + build
uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
workspaces: ". -> target"
- name: Install dependencies (builds Rust extension via maturin)
run: |
pip install -e ".[all,evals,ml]"
python -c "from headroom._core import SmartCrusher; print('headroom._core OK')"
- name: Run Tier 1 evaluation suite
run: |
if [ -z "${OPENAI_API_KEY}" ]; then
echo "::warning title=Weekly eval skipped::OPENAI_API_KEY is not configured for this repo; skipping the live Tier 1 suite."
mkdir -p eval_results
printf '%s\n\n%s\n' \
'# Weekly Evaluation Skipped' \
'OPENAI_API_KEY is not configured for this repository, so the live Tier 1 evaluation suite was skipped.' \
> eval_results/skipped.md
exit 0
fi
python -m headroom.evals suite --tier 1 --ci -o eval_results/
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
# Recall-based fidelity report on the production routing path. Zero cost
# (synthetic structured cases -> Rust compressors; no model, no API, no
# secrets). Non-blocking: surfaces recall trends weekly without gating.
# The blocking per-PR fidelity gate lives in
# tests/test_compression_fidelity_regression.py (runs in the [dev] shard).
- name: Information-retention recall report (zero cost, non-blocking)
run: |
python -c "
from headroom.evals.runners.compression_only import CompressionOnlyRunner
runner = CompressionOnlyRunner()
cases = runner.generate_info_retention_cases(n=50)
result = runner.evaluate_information_retention(cases)
print(f'Information retention: {result.passed_cases}/{result.total_cases} cases >=0.9 recall, avg compression {result.avg_compression_ratio:.1%}')
if not result.passed:
print(f'::warning title=Fidelity recall::{result.failed_cases} case(s) fell below 0.9 recall: {result.errors[:3]}')
"
# Real-dataset recall on the prose path (HotpotQA): does the ground-truth
# answer survive compressing the supporting context? Uses the production
# routing path, so prose flows through Kompress (ModernBERT) — allowed here
# because the weekly job explicitly installs [ml]. Non-blocking and defensive: a
# dataset download or model failure warns rather than fails the job.
- name: Dataset recall report — HotpotQA (model-allowed, non-blocking)
run: |
python -c "
try:
from headroom.transforms.kompress_compressor import warm_kompress_model
from headroom.evals.datasets import load_hotpotqa
from headroom.evals.runners.compression_only import CompressionOnlyRunner
# Block until the Kompress model is loaded; otherwise prose passes
# through uncompressed and the recall number is meaningless.
warmed = warm_kompress_model()
suite = load_hotpotqa(n=50)
result = CompressionOnlyRunner().evaluate_dataset_recall(suite)
print(f'HotpotQA answer recall: {result.passed_cases}/{result.total_cases} probeable cases >=0.9, avg compression {result.avg_compression_ratio:.1%} (model_warmed={warmed})')
if result.avg_compression_ratio < 0.01:
print('::warning title=Dataset recall::compression did not engage (~0%); recall is not a meaningful fidelity signal — check Kompress model availability')
elif result.failed_cases:
print(f'::warning title=Dataset recall::{result.failed_cases} HotpotQA case(s) lost the answer under compression')
except Exception as e:
print(f'::warning title=Dataset recall::skipped (dataset/model unavailable): {e}')
" || true
- name: Upload results
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: eval-results-${{ github.run_number }}
path: eval_results/