mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
<!-- maintainer-current-main -->
## Maintainer latest-main checkpoint
Published `fb146c0663b30488da0a49cfb33624ee82862074` contains current
upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. 22
dependency/Gemini tests and lock check passed. Earlier live behavior
proofs below retain their commit/platform scope; contributor runtime
logic is unchanged.
The common shard-2 failure was independently confirmed on main's hosted
job 127069149772 and locally (four failures, six passes): the Gemini
memory fixture mocked an unused body reader and provided no request
stream. The maintainer test-only follow-up now constructs a real
FastAPI/ASGI Request carrying JSON bytes and exercises the actual body
reader. All six test-function ASTs/assertions are unchanged, and all 10
parameterized Gemini cases pass. Touched-file Ruff/format and whitespace
checks pass. This fixes the inherited fixture instead of asking
contributors to run CI or weakening the memory/content assertions. Fresh
exact-head hosted qualification is requested. Nothing merged.
<!-- /maintainer-current-main -->
## Description
On Linux, installing `headroom-ai[all]` resolved PyTorch and NVIDIA CUDA
packages even for users who only needed the CLI/proxy profile. The
locked dependency graph included those packages through `ml`, `memory`,
`evals`, and `voice`.
This keeps those PyTorch-backed extras available for explicit
installation while removing them from `[all]`. Image support and the
other CLI/proxy dependencies remain included; dependency metadata cannot
detect installed GPU hardware.
Closes #3339
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Remove `ml`, `memory`, `evals`, and `voice` from the `[all]` extra
while keeping each separately installable.
- Retain `image` and the other default CLI/proxy capabilities.
- Update README and installation docs to describe the opt-in extras and
CUDA package implications.
- Add a dependency contract assertion for the `[all]` extra.
## Testing
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
Focused dependency contract tests: 3 passed, 5 deselected.
Python 3.13 locked `uv export` exited 0; the fixed export omits torch, CUDA, NVIDIA, sentence-transformers, and datasets. The baseline export contained them.
`uv lock --check`, scoped Ruff check/format, and `git diff --check` exited 0.
The broader dependency test module had 3 passed and 5 import failures because `opentelemetry` is absent from the local environment.
```
## Real Behavior Proof
- Environment: Linux, Python 3.13, locked uv dependency resolution.
- Exact command / steps: `UV_PYTHON_DOWNLOADS=never uv export --python
3.13 --extra all --no-dev --locked --no-hashes --no-emit-project`.
- Observed result: exit 0; the resolved export no longer contains torch,
CUDA, NVIDIA, sentence-transformers, or datasets, all of which were
present in the baseline export.
- Not tested: a full environment install and GPU runtime behavior.
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable package release.
- Stable/default behavior changed: Yes; `[all]` no longer installs the
four PyTorch-backed extras.
- Kill switch / disable path: Not applicable; install the desired extras
explicitly.
- Unsafe override required: No.
- Qualification impact: Users of those capabilities must add the
corresponding extras.
- Rollback path: Restore the extras to `[all]` and regenerate `uv.lock`.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
Not applicable to this dependency-resolution change.
## Additional Notes
The `[all]` install profile intentionally no longer includes
PyTorch-backed optional capabilities; each remains available through its
named extra. The focused regression and locked resolver proof passed,
but the full dependency test module and a full install were not verified
locally.
## Maintainer integration qualification
Published head `3229690b613810d496f015cedd086ca1d7421843` includes
current upstream main `31dcaf41`. The merge preserves main's `lean`
dependency profile and deprecated `sandbox` alias alongside the
contributor's opt-in `[all]` policy. No optional-feature runtime logic
was changed.
- Reconciled the existing macOS dependency test with the declared
policy, retaining the transitive exclusion check and independently
verifying explicit `ml`/`voice` opt-ins on Intel macOS and Linux. All 12
tests across `tests/test_onnx_dependency_contract.py` and
`tests/test_optional_dependencies.py` pass on Windows/Python 3.12.13.
- Evaluation workflow installs now explicitly request `evals`, and the
model-allowed weekly job also requests `ml`, preserving its previous
intended dependencies. The evaluation guide and AI install index
identify those opt-ins. These address the existing Devin workflow/docs
findings.
- `uv lock --check`, touched Python Ruff/format and whitespace checks
pass. All 253 locked package versions are unchanged; only the root
`[all]` dependency metadata/flattened list changes.
- Real resolver commands from this checkout: `uv export --extra all
--no-dev --locked --no-hashes --no-emit-project` and `uv export --extra
all --extra ml --extra voice --extra evals --no-dev --locked --no-hashes
--no-emit-project`. Evaluating their actual requirement markers for
Python 3.12/3.13/3.14 on Linux x86_64, Windows AMD64, macOS arm64 and
macOS x86_64 shows `[all]` excludes torch, sentence-transformers,
datasets and CUDA/NVIDIA packages in all 12 environments. Explicit
opt-ins retain datasets and torch where supported; Intel macOS still
excludes torch.
- Not tested: a fresh full environment install, GPU runtime, live paid
evaluations or a fresh native build. Hosted CI must qualify the
published final head separately.
### Maintainer current-main requalification
Exact head `f1ed1e0019037e7b53ff9ef548843449c3c7a16f`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.
Current-main integration against upstream
c01cf07d94. The sole conflict was the
inherited Gemini fixture: adopted main's streamed-body helper after
verifying all six test-function ASTs unchanged. No contributor
dependency policy rewritten. The declared breaking change keeps
Torch-backed ml/memory/evals/voice opt-in while preserving the default
CLI/proxy bundle, image capability and main's lean profile. Evaluation
workflows and guides explicitly request removed extras; current
author/Devin discussion considered.
Fresh verification: 25 dependency/Gemini tests pass; offline uv lock
check resolves 253 packages; changed Python lint/format and
main-relative whitespace pass. Actual locked exports evaluated over 12
OS/Python marker environments (Linux/Windows/macOS, Python 3.10–3.13):
all excludes Torch/NVIDIA dependencies, explicit
all+ml+memory+evals+voice retains Torch. This is resolver/export proof,
not a fresh full installation or GPU runtime claim. No new source
blocker found; formal approval awaits exact-head hosted checks. Nothing
merged.
Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
189 lines
8.9 KiB
YAML
189 lines
8.9 KiB
YAML
name: Evaluation Suite
|
|
|
|
on:
|
|
schedule:
|
|
- cron: '0 6 * * 1' # Weekly on Monday 6am UTC
|
|
workflow_dispatch: # Manual trigger
|
|
pull_request:
|
|
paths:
|
|
- 'headroom/transforms/**'
|
|
- 'headroom/evals/**'
|
|
- 'headroom/compress.py'
|
|
- 'crates/**' # Rust compressors are compiled into headroom._core
|
|
- 'Cargo.toml' # and exercised by the eval smoke test (SmartCrusher
|
|
- 'Cargo.lock' # import in the smoke-test job); Rust-only PRs currently skip it
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
# Fast smoke test on PRs touching compression code (~$0.05, ~2 min)
|
|
smoke-test:
|
|
if: github.event_name == 'pull_request'
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 30
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
|
|
with:
|
|
python-version: "3.11"
|
|
|
|
- name: Cache pip
|
|
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
with:
|
|
path: ~/.cache/pip
|
|
key: ${{ runner.os }}-pip-eval-${{ hashFiles('pyproject.toml') }}
|
|
restore-keys: ${{ runner.os }}-pip-eval-
|
|
|
|
# `pip install -e .` invokes maturin (declared in pyproject.toml's
|
|
# build-system) which calls cargo to compile the Rust extension.
|
|
- name: Install Rust toolchain
|
|
uses: dtolnay/rust-toolchain@b725eda525362e7cf34af9a7243cac1e3f01a7ec # stable
|
|
with:
|
|
toolchain: 1.95.0 # Keep in sync with rust-toolchain.toml.
|
|
|
|
- name: Cache cargo registry + build
|
|
uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
|
|
with:
|
|
workspaces: ". -> target"
|
|
|
|
- name: Install dependencies (builds Rust extension via maturin)
|
|
run: |
|
|
pip install -e ".[all,evals]"
|
|
python -c "from headroom._core import SmartCrusher; print('headroom._core OK:', SmartCrusher)"
|
|
|
|
- name: Run CCR round-trip (zero cost)
|
|
run: |
|
|
python -c "
|
|
from headroom.evals.runners.compression_only import CompressionOnlyRunner
|
|
runner = CompressionOnlyRunner()
|
|
cases = runner.generate_ccr_test_cases(n=50)
|
|
result = runner.evaluate_ccr_lossless(cases)
|
|
print(f'CCR Round-trip: {result.passed_cases}/{result.total_cases} passed')
|
|
assert result.passed, f'CCR failures: {result.errors}'
|
|
"
|
|
|
|
- name: Run tool schema compaction integrity eval (zero cost)
|
|
run: |
|
|
python -c "
|
|
from headroom.evals.runners.compression_only import CompressionOnlyRunner
|
|
runner = CompressionOnlyRunner()
|
|
result = runner.evaluate_tool_schema_compaction()
|
|
print(f'Tool schema compaction: {result.passed_cases}/{result.total_cases} passed, {result.total_tokens_saved} annotation tokens stripped')
|
|
assert result.passed, f'Schema compaction failures: {result.errors}'
|
|
"
|
|
|
|
# OPENAI_API_KEY is intentionally not set in the public OSS repo
|
|
# (the secret list is empty). The CCR round-trip step above is the
|
|
# mandatory gate; this step only runs when an operator has wired
|
|
# OPENAI_API_KEY as a repo secret (e.g. on a downstream fork). When
|
|
# missing, emit a loud GitHub `::warning::` annotation so the skip
|
|
# is visible in the run summary — never a silent pass.
|
|
- name: Run built-in tool output eval (skipped when OPENAI_API_KEY unset)
|
|
env:
|
|
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
|
run: |
|
|
if [ -z "${OPENAI_API_KEY}" ]; then
|
|
echo "::warning title=Smoke eval skipped::OPENAI_API_KEY is not configured for this repo; only the CCR round-trip gate ran. Wire the secret to enable the live OpenAI eval."
|
|
exit 0
|
|
fi
|
|
python -m headroom.evals quick -n 8 --provider openai --model gpt-4o-mini
|
|
|
|
# Full Tier 1 suite, weekly or manual (~$3-5, ~30-45 min)
|
|
weekly-suite:
|
|
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 90
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
|
|
with:
|
|
python-version: "3.11"
|
|
|
|
- name: Cache pip
|
|
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
with:
|
|
path: ~/.cache/pip
|
|
key: ${{ runner.os }}-pip-eval-${{ hashFiles('pyproject.toml') }}
|
|
restore-keys: ${{ runner.os }}-pip-eval-
|
|
|
|
- name: Install Rust toolchain
|
|
uses: dtolnay/rust-toolchain@b725eda525362e7cf34af9a7243cac1e3f01a7ec # stable
|
|
with:
|
|
toolchain: 1.95.0 # Keep in sync with rust-toolchain.toml.
|
|
|
|
- name: Cache cargo registry + build
|
|
uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
|
|
with:
|
|
workspaces: ". -> target"
|
|
|
|
- name: Install dependencies (builds Rust extension via maturin)
|
|
run: |
|
|
pip install -e ".[all,evals,ml]"
|
|
python -c "from headroom._core import SmartCrusher; print('headroom._core OK')"
|
|
- name: Run Tier 1 evaluation suite
|
|
run: |
|
|
if [ -z "${OPENAI_API_KEY}" ]; then
|
|
echo "::warning title=Weekly eval skipped::OPENAI_API_KEY is not configured for this repo; skipping the live Tier 1 suite."
|
|
mkdir -p eval_results
|
|
printf '%s\n\n%s\n' \
|
|
'# Weekly Evaluation Skipped' \
|
|
'OPENAI_API_KEY is not configured for this repository, so the live Tier 1 evaluation suite was skipped.' \
|
|
> eval_results/skipped.md
|
|
exit 0
|
|
fi
|
|
python -m headroom.evals suite --tier 1 --ci -o eval_results/
|
|
env:
|
|
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
|
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
|
|
|
# Recall-based fidelity report on the production routing path. Zero cost
|
|
# (synthetic structured cases -> Rust compressors; no model, no API, no
|
|
# secrets). Non-blocking: surfaces recall trends weekly without gating.
|
|
# The blocking per-PR fidelity gate lives in
|
|
# tests/test_compression_fidelity_regression.py (runs in the [dev] shard).
|
|
- name: Information-retention recall report (zero cost, non-blocking)
|
|
run: |
|
|
python -c "
|
|
from headroom.evals.runners.compression_only import CompressionOnlyRunner
|
|
runner = CompressionOnlyRunner()
|
|
cases = runner.generate_info_retention_cases(n=50)
|
|
result = runner.evaluate_information_retention(cases)
|
|
print(f'Information retention: {result.passed_cases}/{result.total_cases} cases >=0.9 recall, avg compression {result.avg_compression_ratio:.1%}')
|
|
if not result.passed:
|
|
print(f'::warning title=Fidelity recall::{result.failed_cases} case(s) fell below 0.9 recall: {result.errors[:3]}')
|
|
"
|
|
|
|
# Real-dataset recall on the prose path (HotpotQA): does the ground-truth
|
|
# answer survive compressing the supporting context? Uses the production
|
|
# routing path, so prose flows through Kompress (ModernBERT) — allowed here
|
|
# because the weekly job explicitly installs [ml]. Non-blocking and defensive: a
|
|
# dataset download or model failure warns rather than fails the job.
|
|
- name: Dataset recall report — HotpotQA (model-allowed, non-blocking)
|
|
run: |
|
|
python -c "
|
|
try:
|
|
from headroom.transforms.kompress_compressor import warm_kompress_model
|
|
from headroom.evals.datasets import load_hotpotqa
|
|
from headroom.evals.runners.compression_only import CompressionOnlyRunner
|
|
# Block until the Kompress model is loaded; otherwise prose passes
|
|
# through uncompressed and the recall number is meaningless.
|
|
warmed = warm_kompress_model()
|
|
suite = load_hotpotqa(n=50)
|
|
result = CompressionOnlyRunner().evaluate_dataset_recall(suite)
|
|
print(f'HotpotQA answer recall: {result.passed_cases}/{result.total_cases} probeable cases >=0.9, avg compression {result.avg_compression_ratio:.1%} (model_warmed={warmed})')
|
|
if result.avg_compression_ratio < 0.01:
|
|
print('::warning title=Dataset recall::compression did not engage (~0%); recall is not a meaningful fidelity signal — check Kompress model availability')
|
|
elif result.failed_cases:
|
|
print(f'::warning title=Dataset recall::{result.failed_cases} HotpotQA case(s) lost the answer under compression')
|
|
except Exception as e:
|
|
print(f'::warning title=Dataset recall::skipped (dataset/model unavailable): {e}')
|
|
" || true
|
|
|
|
- name: Upload results
|
|
if: always()
|
|
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
|
with:
|
|
name: eval-results-${{ github.run_number }}
|
|
path: eval_results/
|