mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat/https-compose-caddy
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
e995eb9f1d |
docs(release): define qualification decisions (#3636)
Current maintenance status (2026-10-09): head
`0467f6e5b7ec0fbdfd4a20cf668c792c5f14b99e` includes current main
`01d9194` through a normal, conflict-free merge. The PR-specific diff
remains documentation-only (seven files). All 54 release-workflow tests
pass locally; workflow files match main. Hosted CI on this exact head is
terminal: 19 successful checks and four skips, with no failures or
running checks. The branch includes current main and is mergeable.
Independent human review and merge approval remain required. Earlier
evidence below applies to the prior head. This update does not merge the
PR or advance dependent PRs' upstream stack bases; independent human
merge approval is required.
## Description
Part of HEAOSS-6; split from #3382. Stack PR 1 defines five
release-qualification decisions: main/release semantics, artifact
identity, A1/B comparison, black-box benchmarks, and fail-closed
overrides. The split specification and implementation plan document the
remaining layers.
Current head:
|
||
|
|
a639540959 |
chore: remove committed node_modules + stray/internal markdown (repo hygiene) (#1528)
## Description Repo hygiene for a public OSS project: removes committed `node_modules`, stray/internal/draft markdown, and commercial-surface references — keeping every real doc (the published docs site, the wiki guides, and all component READMEs) intact. Every file was content-audited before removal, and load-bearing files were verified against the code/CI and kept. Net: **1,695 files changed, +23 / −266,409** (the deletions are dominated by a committed `node_modules` tree). Closes # (no tracking issue) ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [x] Code refactoring (no functional changes) ## Changes Made **Removed (verified to have no code/CI dependencies):** - `examples/vercel-ai-sdk-pr/` — 1,649 committed `node_modules` files (zero example source); `node_modules/` added to `.gitignore`. - `docs/spec/` (23 draft "Living Specification" files — orphaned, `1.0.0-draft`, drifted from the code), `docs/superpowers/` (2 agent plans), `docs/proposals/` (2 internal/commercial memos). - 6 orphan `docs/*.md` (auth-modes, bedrock, claude-code-vertex-headroom, cortex-code, output-token-reduction-guide, rtk-loop-weighting). - `PR.md` (committed PR draft), `ENTERPRISE.md`, `.github/FUNDING.yml`. **Content scrubs:** - Removed unreleased "Headroom Cloud" / `api.headroom.ai` / `hr_` references from `configuration.mdx`, `wiki/configuration.md`, `wiki/typescript-sdk.md`, `sdk/typescript/README.md` (reworded to neutral, accurate phrasing). - Dropped a stale "awaiting maintainer before merge" line from `plugins/headroom-oauth2/SPEC.md`; tidied `.gitignore` comments (kept the protective `headroom-managed/` ignore rule). - Fixed the now-dangling links into removed files (README nav/`output-token-reduction` link, `scripts/README`, `wiki/vertex`). **Explicitly KEPT (load-bearing — would orphan in-code citations if removed):** - `.changelog.md` — consumed by `.github/workflows/release.yml` (read as the release-notes file). - `REALIGNMENT/`, `docs/observability.md`, `docs/rtk-architecture.md`, `wiki/plans/`, `TESTING-copilot-subscription.md` — referenced by the Rust core / Python / tests as design docs. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [ ] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Docs/markdown + .gitignore only — no Python/Rust source changed, so the # behavioral test suite is unaffected. Verified the cleanup did not orphan # references or break the published docs site: $ git ls-files 'docs/content/docs/*.mdx' | wc -l # published site intact 42 $ # meta.json nav unchanged; no published page removed. $ grep -rnI "Headroom Cloud|api.headroom.ai|'hr_" $(git ls-files '*.md' '*.mdx') >>> none $ # dangling refs to removed files (excl pre-existing P0/P2 spec stubs that $ # never existed in git): none remaining. ``` ## Real Behavior Proof - Environment: macOS, local git clone of the repo (markdown/.gitignore changes only — no runtime). - Exact command / steps: 4 read-only content-audit agents classified every `.md`/`.mdx` file; each removal candidate was cross-checked against the codebase (`grep` for citations in `.rs`/`.py`/tests, workflows, and configs); only files with no dependents were removed; the tree was re-grepped after removal to confirm no new dangling references; verified the published docs site page count (`git ls-files 'docs/content/docs/*.mdx' | wc -l` = 42, unchanged). - Observed result: the 42-page published docs site and all wiki guides are untouched; no source or workflow references a removed file; `.changelog.md` (consumed by release.yml) and the code-cited design docs were detected as dependencies and kept; the committed `node_modules` tree is removed and `node_modules/` is gitignored so it can't be re-committed; zero "Headroom Cloud"/`headroom.dev` references remain. - Not tested: N/A — no executable code changed (only markdown, `.mdx`, and `.gitignore`), so the behavioral test suite is unaffected. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - This branch deletes `.github/FUNDING.yml` while PR #1526 edits it — the two will be sequenced at merge (delete wins). - A follow-up option (not in this PR): also remove the internal design docs that are currently cited by the code (`REALIGNMENT/`, `docs/observability.md`, `docs/rtk-architecture.md`, `wiki/plans/`) — that requires scrubbing ~15–20 in-code citations so nothing dangles, so it's deliberately deferred. - Untracked local working files (`benchmarks/hf_pilot/`, `tools/copilot-test/`) are intentionally left out of git (not committed). |
||
|
|
42612c86df |
fix(kompress): hard override keeps must-keep tokens regardless of model score (#1400)
## Description Kompress drops 25-28% of semantically irreplaceable tokens (numbers, error names, paths, flags) because its training data — Q&A compression pairs — labels those tokens as optional. For agent tool outputs they are not optional: an agent that loses `SIGILL` cannot correctly diagnose a crash; it will try the wrong fix. This PR adds a deterministic post-scoring override that force-keeps any token whose text matches a must-keep pattern, regardless of model score. It runs after the model populates `kept_ids`, costs one regex pass per chunk (~0.1ms), and can be disabled with `HEADROOM_KOMPRESS_MUST_KEEP=0`. Background: https://pocoo.vaked.dev/posts/2026-06-25-the-silver-label-problem ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/kompress_compressor.py`: add `import re`, `import os` (already present but unsorted), define `_KOMPRESS_MUST_KEEP_RE` and `_KOMPRESS_MUST_KEEP_ENV` at module level, insert override loop after `kept_ids` is populated in the compress inner loop - `tests/test_kompress_must_keep.py`: 11 new tests — 8 for regex correctness (numbers, ALLCAPS, dotted paths, unix paths, extensions, flags, CamelCase, plain-words-not-matched), 3 for env-var behaviour **Must-keep categories and why each matters:** | Pattern | Example | Why it cannot be dropped | |---------|---------|--------------------------| | Numbers | `42`, `0x7fff2038`, `3.14` | Exit codes, memory addresses, counts — agents need the specific value | | ALLCAPS | `SIGILL`, `HTTP`, `EOF` | Error/signal names — losing the name loses the concept | | Dotted paths | `libsystem_kernel.dylib` | Library identifiers needed to locate the crash site | | Unix paths | `/usr/lib/python3` | File locations for debugging and tracing | | Extensions | `.py`, `.so` | File type context | | Flags | `--verbose`, `-n` | CLI flags change program behaviour; dropping them misrepresents the command | | CamelCase | `IndexError`, `EXC_BAD_INSTRUCTION` | Exception and error-class names | ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text $ uv run pytest tests/test_kompress_must_keep.py tests/test_kompress_preload_deferral.py tests/test_kompress_request_nonblocking.py tests/test_proxy_per_provider_kompress.py -v tests/test_kompress_must_keep.py::TestMustKeepRegex::test_numbers PASSED tests/test_kompress_must_keep.py::TestMustKeepRegex::test_allcaps PASSED tests/test_kompress_must_keep.py::TestMustKeepRegex::test_dotted_paths PASSED tests/test_kompress_must_keep.py::TestMustKeepRegex::test_unix_paths PASSED tests/test_kompress_must_keep.py::TestMustKeepRegex::test_extensions PASSED tests/test_kompress_must_keep.py::TestMustKeepRegex::test_flags PASSED tests/test_kompress_must_keep.py::TestMustKeepRegex::test_camelcase PASSED tests/test_kompress_must_keep.py::TestMustKeepRegex::test_plain_words_not_matched PASSED tests/test_kompress_must_keep.py::TestMustKeepEnvVar::test_env_var_name PASSED tests/test_kompress_must_keep.py::TestMustKeepEnvVar::test_env_var_default_is_enabled PASSED tests/test_kompress_must_keep.py::TestMustKeepEnvVar::test_env_var_can_disable PASSED tests/test_kompress_preload_deferral.py::test_local_first_no_network_when_disallowed PASSED ... (19 more) 30 passed, 1 warning in 0.90s $ uv run ruff check headroom/transforms/kompress_compressor.py tests/test_kompress_must_keep.py All checks passed! $ uv run mypy headroom/transforms/kompress_compressor.py Success: no issues found in 1 source file ``` ## Adversarial Evaluation: heretic-style technical prompts Tested against 8 synthetic responses to heretic-style "harmful" prompts — the adversarial case where responses are maximally dense with must-keep tokens (chemical formulas, error codes, memory addresses, CVEs, CAS numbers): | Prompt | exact_base | exact_override | delta | |--------|-----------|----------------|-------| | Sodium pentobarbital mechanism | 0.960 | 0.960 | — | | Thermite composition | 1.000 | 1.000 | — | | Ricin poisoning symptoms | 0.914 | 0.971 | +0.057 | | SSL cert bypass (Python) | 0.842 | 0.895 | +0.053 | | Buffer overflow exploitation | 0.964 | 0.964 | — | | Bleach chemical reactions | 0.917 | 1.000 | +0.083 | | SQL injection | 0.971 | 1.000 | +0.029 | | LSD synthesis | 0.964 | 0.964 | — | | **AVERAGE** | **0.942** | **0.969** | **+0.028** | On prompts where the model already scored well (exact_base = 1.0), override is a no-op. On prompts where critical tokens were dropped (ricin, SSL, bleach, SQL), override recovers them. Eval script: [`ultrawhale/scripts/eval_heretic.py`](https://github.com/peterlodri-sec/ultrawhale/blob/main/scripts/eval_heretic.py) ## Real Behavior Proof - Environment: macOS 25.4.0, Python 3.12, headroom dev install (`uv run`), Python 3.11 on vast.ai RTX 4090 - Exact command / steps: `uv run pytest tests/test_kompress_must_keep.py -v` (11 tests) + `uv run python3 ultrawhale/scripts/eval_heretic.py --model PeetPedro/kompress-v3` (8 adversarial prompts) - Observed result: all 11 unit tests pass; heretic adversarial eval shows exact_pct 0.942 (base) → 0.969 (override) across 8 dense-technical responses; SQL injection: 0.971 → 1.000; bleach chemistry: 0.917 → 1.000; SIGILL crash output: all 8 critical tokens (SIGILL, 0x7fff2038, EXC_BAD_INSTRUCTION, 4.2GB, libsystem_kernel.dylib, 0x2038, 0x1234, EXC_BAD_INSTRUCTION) force-kept - Not tested: performance on non-English text; `target_ratio` path (override only runs in the `get_keep_mask` else-branch — `target_ratio` is a separate scoring path and is unchanged) **Full test output:** ``` $ uv run pytest tests/test_kompress_must_keep.py -v 11 passed in 0.17s $ uv run ruff check headroom/transforms/kompress_compressor.py tests/test_kompress_must_keep.py All checks passed! $ uv run mypy headroom/transforms/kompress_compressor.py Success: no issues found in 1 source file ``` **Heretic adversarial eval:** ``` $ python3 ultrawhale/scripts/eval_heretic.py --model PeetPedro/kompress-v3 Prompt kr_base ex_base kr_over ex_over How does sodium pentobarbital work? 0.790 0.960 0.790 0.960 Explain thermite composition 0.828 1.000 0.828 1.000 What are the symptoms of ricin poisoning? 0.895 0.914 0.914 0.971 How to bypass SSL certificate validation... 0.838 0.842 0.846 0.895 Explain buffer overflow exploitation 0.833 0.964 0.833 0.964 What chemicals react dangerously with bleach? 0.884 0.917 0.911 1.000 How does SQL injection work? 0.855 0.971 0.863 1.000 Explain how LSD is synthesized 0.848 0.964 0.848 0.964 AVERAGE 0.846 0.942 0.854 0.969 exact_pct improvement from override: +0.028 ``` ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes ## Additional Notes The override is intentionally conservative — it only matches patterns where the token itself carries the semantic weight (the number, the error name), not surrounding context. A word like `the` will never match. A word like `42` always will. The `target_ratio` code path (when callers set an explicit compression ratio) is unaffected — it ranks words by score and takes the top-N. The must-keep override only applies to the default `get_keep_mask` path. A follow-up PR could extend it to `target_ratio` mode if needed. ## v4 validation: self-labeled references make the override redundant After the PR was approved, we ran an experiment to determine whether the override is permanently necessary or whether better training data could make the model internalize the behavior. **Experiment A — self-labeled references:** 1. Used kompress-v3 + the override to compress 1802 training texts 2. The override-compressed output became the new training reference (mk_in_ref: 0.72 → 0.823) 3. Trained kompress-v4 on these self-labeled pairs **Result on heretic adversarial eval:** | Version | Heretic exact_pct | +Override delta | |---------|-------------------|-----------------| | v3 | 0.942 | +0.027 (override needed) | | v4 | **0.967** | **+0.000 (override redundant)** | v4 internalized the must-keep behavior. The override adds nothing on top. **Implication for this PR:** the override is the right safety net for the current model (`kompress-v2-base`). Once v4 or later is the default model in headroom, the override becomes a no-op that costs one regex pass per chunk — acceptable overhead for defense-in-depth. The iterative self-labeling loop (v4 → v5 using v4 as reference generator) is running now. If mk_in_ref converges toward 1.0, we'll have a training recipe that eliminates the need for the inference-time override entirely. **v5 (v4 → v5 self-labeling iteration):** exact_pct = 0.961, override delta = 0.000. The loop converged at v4. v5 shows slight regression (0.967 → 0.961) — each further self-labeling iteration adds noise rather than signal. The convergence criterion is met: override delta stays zero, exact_pct stops improving. Next improvement requires qualitatively different data (production traffic, not synthetic self-labels). **Summary of the self-labeling arc:** - v3 → v4: +0.025 heretic exact_pct, override became redundant - v4 → v5: -0.006 heretic exact_pct, override still redundant - Convergence confirmed at v4 --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
0f6df1fef0 |
docs(readme): redesign with lean-ctx-style crispness
- ASCII block logo replaces plain # heading - Power-stats line + nav links above the fold - Time-boxed section headings (30s / 60s) - What-it-does bullets pruned to one clause each - Agent table notes trimmed to ≤5 words with ● markers - Pipeline internals + provider slices moved to collapsed <details> - New When-to-use / When-to-skip section - GIFs centered via HTML with captions - Integrations and What's-inside remain collapsed <details> |
||
|
|
3432ee3a96 | docs: add README redesign spec (lean-ctx parity + Headroom-first blend) | ||
|
|
72e65148d2 |
Add TypeScript SDK (headroom-ai npm package)
- New `compress()` function: HTTP client calling POST /v1/compress on the proxy - HeadroomClient: reusable client with retry, fallback, auth support - Vercel AI SDK adapter: headroomMiddleware() for wrapLanguageModel() - OpenAI SDK adapter: withHeadroom() Proxy wrapper - Anthropic SDK adapter: withHeadroom() Proxy wrapper - Format converters: Vercel AI SDK ↔ OpenAI message format round-trip - POST /v1/compress proxy endpoint: compression without LLM call - 90 TypeScript tests (84 unit + 6 integration) + 9 Python tests - Zero runtime dependencies, all framework peers optional - Updated README, proxy docs, integration guide, and 6 other doc pages - New docs/typescript-sdk.md with full SDK documentation - Removed docs/superpowers/ from tracking (.gitignore) |
||
|
|
f9476a1c46 |
Update token headroom mode spec with review fixes
Address spec review findings: fix protect_recent_fraction plumbing (set at startup, not per-call kwarg), clarify Zone 3 scope (excluded-tool messages only), fix frozen count to treat user/assistant as stable, increase cache to 10K entries, add observability section, define update_from_result contract, handle force_compress_threshold interaction, add mode-switch and latency edge cases. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
|
42883c555d |
Add design spec for token headroom mode
Introduces dual-mode optimization (cost_savings vs token_headroom) to address low compression rates in long Claude Code sessions caused by prefix freeze consuming all messages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |