Commit Graph
8 Commits
Author SHA1 Message Date
JD Davis e995eb9f1d docs(release): define qualification decisions (#3636)
Current maintenance status (2026-10-09): head
`0467f6e5b7ec0fbdfd4a20cf668c792c5f14b99e` includes current main
`01d9194` through a normal, conflict-free merge. The PR-specific diff
remains documentation-only (seven files). All 54 release-workflow tests
pass locally; workflow files match main. Hosted CI on this exact head is
terminal: 19 successful checks and four skips, with no failures or
running checks. The branch includes current main and is mergeable.
Independent human review and merge approval remain required. Earlier
evidence below applies to the prior head. This update does not merge the
PR or advance dependent PRs' upstream stack bases; independent human
merge approval is required.
## Description

Part of HEAOSS-6; split from #3382. Stack PR 1 defines five
release-qualification decisions: main/release semantics, artifact
identity, A1/B comparison, black-box benchmarks, and fail-closed
overrides. The split specification and implementation plan document the
remaining layers.

Current head: f4790a5359. Next PR: #3637.

## Type of Change

- [x] Documentation update
- [x] Bug fix (the restore command stages ADRs; the plan must inspect
the index)

## Changes Made

- Add the five ADRs and the specification/plan for the focused release
stack.
- Require both unstaged and cached whitespace checks after restoring
ADRs, stopping on either failure.
- Scope the placeholder scan to this slice's seven documents and
distinguish no matches from a failed scan.

## Testing

- [x] Actual disposable Git replay reproduces the staged-whitespace gap
and verifies the cached check catches it.
- [x] Both checks pass after correcting and restaging the document.
- [x] Seven actual changed documents are readable UTF-8, nonempty, and
contain no unresolved TBD/TODO markers.
- [x] Unstaged/staged whitespace checks pass on the published change.

```text
PASS reproduction: unstaged-only diff check exits 0; staged check exits 2 and reports trailing whitespace
PASS corrected staged document: both whitespace checks exit 0
PASS seven actual changed documents: readable UTF-8, nonempty, no unresolved TBD/TODO markers
PASS original source ADR untouched
```

## Real Behavior Proof

- Environment: Windows/Python 3.12.13/Git, disposable repository on G;
real ADR bytes copied from this checkout.
- Exact command / steps: `python
G:/git/headroom/.remember/tmp/pr3636-resume40-doc-replay.py`. Commit the
actual ADR as a baseline, append trailing whitespace, stage it, and
execute both Git checks. Correct/restage and rerun both checks; inspect
the seven real changed documents and verify the original ADR's digest is
unchanged.
- Observed result: The old unstaged-only check falsely succeeds while
the cached check exits 2 and identifies the staged violation. Both exit
0 after correction; the source ADR remains untouched and the changed
documentation has no unresolved placeholders.
- Not tested: Full six-PR reconstruction, release controller behavior,
candidate dispatch, production publishing, or documentation
publication/rendering. This is documentation-integrity proof.
- Evidence: pr3636-resume40-doc-replay.py/.log under
G:/git/headroom/.remember/tmp; owned replay repository
G:/tmp/headroom-review/resume40-3636-doc-replay.

## Runtime Rollout Safety

- Rollout-managed feature(s): Release qualification design;
documentation only.
- Minimum rollout channel: No runtime channel change.
- Stable/default behavior changed: The implementation plan checks staged
documents correctly; no executable release behavior is introduced here.
- Kill switch / disable path: Do not execute the plan; revert the
documentation.
- Unsafe override required: No.
- Qualification impact: Documents the intended fail-closed evidence
model; implementation/hosted qualification belongs to later stack PRs.
- Rollback path: Revert the ADR/plan slice; existing publisher remains
operational.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Independent review remains required. No PR merge, self-approval or
release publication performed.
2026-10-21 15:02:54 -05:00
Tejas Chopra a639540959 chore: remove committed node_modules + stray/internal markdown (repo hygiene) (#1528)
## Description

Repo hygiene for a public OSS project: removes committed `node_modules`,
stray/internal/draft markdown, and commercial-surface references —
keeping every real doc (the published docs site, the wiki guides, and
all component READMEs) intact. Every file was content-audited before
removal, and load-bearing files were verified against the code/CI and
kept.

Net: **1,695 files changed, +23 / −266,409** (the deletions are
dominated by a committed `node_modules` tree).

Closes # (no tracking issue)

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [x] Code refactoring (no functional changes)

## Changes Made

**Removed (verified to have no code/CI dependencies):**
- `examples/vercel-ai-sdk-pr/` — 1,649 committed `node_modules` files
(zero example source); `node_modules/` added to `.gitignore`.
- `docs/spec/` (23 draft "Living Specification" files — orphaned,
`1.0.0-draft`, drifted from the code), `docs/superpowers/` (2 agent
plans), `docs/proposals/` (2 internal/commercial memos).
- 6 orphan `docs/*.md` (auth-modes, bedrock,
claude-code-vertex-headroom, cortex-code, output-token-reduction-guide,
rtk-loop-weighting).
- `PR.md` (committed PR draft), `ENTERPRISE.md`, `.github/FUNDING.yml`.

**Content scrubs:**
- Removed unreleased "Headroom Cloud" / `api.headroom.ai` / `hr_`
references from `configuration.mdx`, `wiki/configuration.md`,
`wiki/typescript-sdk.md`, `sdk/typescript/README.md` (reworded to
neutral, accurate phrasing).
- Dropped a stale "awaiting maintainer before merge" line from
`plugins/headroom-oauth2/SPEC.md`; tidied `.gitignore` comments (kept
the protective `headroom-managed/` ignore rule).
- Fixed the now-dangling links into removed files (README
nav/`output-token-reduction` link, `scripts/README`, `wiki/vertex`).

**Explicitly KEPT (load-bearing — would orphan in-code citations if
removed):**
- `.changelog.md` — consumed by `.github/workflows/release.yml` (read as
the release-notes file).
- `REALIGNMENT/`, `docs/observability.md`, `docs/rtk-architecture.md`,
`wiki/plans/`, `TESTING-copilot-subscription.md` — referenced by the
Rust core / Python / tests as design docs.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [ ] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# Docs/markdown + .gitignore only — no Python/Rust source changed, so the
# behavioral test suite is unaffected. Verified the cleanup did not orphan
# references or break the published docs site:

$ git ls-files 'docs/content/docs/*.mdx' | wc -l      # published site intact
42
$ # meta.json nav unchanged; no published page removed.

$ grep -rnI "Headroom Cloud|api.headroom.ai|'hr_" $(git ls-files '*.md' '*.mdx')
>>> none

$ # dangling refs to removed files (excl pre-existing P0/P2 spec stubs that
$ # never existed in git): none remaining.
```

## Real Behavior Proof

- Environment: macOS, local git clone of the repo (markdown/.gitignore
changes only — no runtime).
- Exact command / steps: 4 read-only content-audit agents classified
every `.md`/`.mdx` file; each removal candidate was cross-checked
against the codebase (`grep` for citations in `.rs`/`.py`/tests,
workflows, and configs); only files with no dependents were removed; the
tree was re-grepped after removal to confirm no new dangling references;
verified the published docs site page count (`git ls-files
'docs/content/docs/*.mdx' | wc -l` = 42, unchanged).
- Observed result: the 42-page published docs site and all wiki guides
are untouched; no source or workflow references a removed file;
`.changelog.md` (consumed by release.yml) and the code-cited design docs
were detected as dependencies and kept; the committed `node_modules`
tree is removed and `node_modules/` is gitignored so it can't be
re-committed; zero "Headroom Cloud"/`headroom.dev` references remain.
- Not tested: N/A — no executable code changed (only markdown, `.mdx`,
and `.gitignore`), so the behavioral test suite is unaffected.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- This branch deletes `.github/FUNDING.yml` while PR #1526 edits it —
the two will be sequenced at merge (delete wins).
- A follow-up option (not in this PR): also remove the internal design
docs that are currently cited by the code (`REALIGNMENT/`,
`docs/observability.md`, `docs/rtk-architecture.md`, `wiki/plans/`) —
that requires scrubbing ~15–20 in-code citations so nothing dangles, so
it's deliberately deferred.
- Untracked local working files (`benchmarks/hf_pilot/`,
`tools/copilot-test/`) are intentionally left out of git (not
committed).
2026-06-27 23:32:54 -07:00
Peter LodriandJerrettDavis 42612c86df fix(kompress): hard override keeps must-keep tokens regardless of model score (#1400)
## Description

Kompress drops 25-28% of semantically irreplaceable tokens (numbers,
error names, paths, flags) because its training data — Q&A compression
pairs — labels those tokens as optional. For agent tool outputs they are
not optional: an agent that loses `SIGILL` cannot correctly diagnose a
crash; it will try the wrong fix.

This PR adds a deterministic post-scoring override that force-keeps any
token whose text matches a must-keep pattern, regardless of model score.
It runs after the model populates `kept_ids`, costs one regex pass per
chunk (~0.1ms), and can be disabled with
`HEADROOM_KOMPRESS_MUST_KEEP=0`.

Background:
https://pocoo.vaked.dev/posts/2026-06-25-the-silver-label-problem

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/transforms/kompress_compressor.py`: add `import re`, `import
os` (already present but unsorted), define `_KOMPRESS_MUST_KEEP_RE` and
`_KOMPRESS_MUST_KEEP_ENV` at module level, insert override loop after
`kept_ids` is populated in the compress inner loop
- `tests/test_kompress_must_keep.py`: 11 new tests — 8 for regex
correctness (numbers, ALLCAPS, dotted paths, unix paths, extensions,
flags, CamelCase, plain-words-not-matched), 3 for env-var behaviour

**Must-keep categories and why each matters:**

| Pattern | Example | Why it cannot be dropped |
|---------|---------|--------------------------|
| Numbers | `42`, `0x7fff2038`, `3.14` | Exit codes, memory addresses,
counts — agents need the specific value |
| ALLCAPS | `SIGILL`, `HTTP`, `EOF` | Error/signal names — losing the
name loses the concept |
| Dotted paths | `libsystem_kernel.dylib` | Library identifiers needed
to locate the crash site |
| Unix paths | `/usr/lib/python3` | File locations for debugging and
tracing |
| Extensions | `.py`, `.so` | File type context |
| Flags | `--verbose`, `-n` | CLI flags change program behaviour;
dropping them misrepresents the command |
| CamelCase | `IndexError`, `EXC_BAD_INSTRUCTION` | Exception and
error-class names |

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
$ uv run pytest tests/test_kompress_must_keep.py tests/test_kompress_preload_deferral.py tests/test_kompress_request_nonblocking.py tests/test_proxy_per_provider_kompress.py -v

tests/test_kompress_must_keep.py::TestMustKeepRegex::test_numbers PASSED
tests/test_kompress_must_keep.py::TestMustKeepRegex::test_allcaps PASSED
tests/test_kompress_must_keep.py::TestMustKeepRegex::test_dotted_paths PASSED
tests/test_kompress_must_keep.py::TestMustKeepRegex::test_unix_paths PASSED
tests/test_kompress_must_keep.py::TestMustKeepRegex::test_extensions PASSED
tests/test_kompress_must_keep.py::TestMustKeepRegex::test_flags PASSED
tests/test_kompress_must_keep.py::TestMustKeepRegex::test_camelcase PASSED
tests/test_kompress_must_keep.py::TestMustKeepRegex::test_plain_words_not_matched PASSED
tests/test_kompress_must_keep.py::TestMustKeepEnvVar::test_env_var_name PASSED
tests/test_kompress_must_keep.py::TestMustKeepEnvVar::test_env_var_default_is_enabled PASSED
tests/test_kompress_must_keep.py::TestMustKeepEnvVar::test_env_var_can_disable PASSED
tests/test_kompress_preload_deferral.py::test_local_first_no_network_when_disallowed PASSED
... (19 more)

30 passed, 1 warning in 0.90s

$ uv run ruff check headroom/transforms/kompress_compressor.py tests/test_kompress_must_keep.py
All checks passed!

$ uv run mypy headroom/transforms/kompress_compressor.py
Success: no issues found in 1 source file
```



## Adversarial Evaluation: heretic-style technical prompts

Tested against 8 synthetic responses to heretic-style "harmful" prompts
— the adversarial case where responses are maximally dense with
must-keep tokens (chemical formulas, error codes, memory addresses,
CVEs, CAS numbers):

| Prompt | exact_base | exact_override | delta |
|--------|-----------|----------------|-------|
| Sodium pentobarbital mechanism | 0.960 | 0.960 | — |
| Thermite composition | 1.000 | 1.000 | — |
| Ricin poisoning symptoms | 0.914 | 0.971 | +0.057 |
| SSL cert bypass (Python) | 0.842 | 0.895 | +0.053 |
| Buffer overflow exploitation | 0.964 | 0.964 | — |
| Bleach chemical reactions | 0.917 | 1.000 | +0.083 |
| SQL injection | 0.971 | 1.000 | +0.029 |
| LSD synthesis | 0.964 | 0.964 | — |
| **AVERAGE** | **0.942** | **0.969** | **+0.028** |

On prompts where the model already scored well (exact_base = 1.0),
override is a no-op. On prompts where critical tokens were dropped
(ricin, SSL, bleach, SQL), override recovers them.

Eval script:
[`ultrawhale/scripts/eval_heretic.py`](https://github.com/peterlodri-sec/ultrawhale/blob/main/scripts/eval_heretic.py)

## Real Behavior Proof

- Environment: macOS 25.4.0, Python 3.12, headroom dev install (`uv
run`), Python 3.11 on vast.ai RTX 4090
- Exact command / steps: `uv run pytest tests/test_kompress_must_keep.py
-v` (11 tests) + `uv run python3 ultrawhale/scripts/eval_heretic.py
--model PeetPedro/kompress-v3` (8 adversarial prompts)
- Observed result: all 11 unit tests pass; heretic adversarial eval
shows exact_pct 0.942 (base) → 0.969 (override) across 8 dense-technical
responses; SQL injection: 0.971 → 1.000; bleach chemistry: 0.917 →
1.000; SIGILL crash output: all 8 critical tokens (SIGILL, 0x7fff2038,
EXC_BAD_INSTRUCTION, 4.2GB, libsystem_kernel.dylib, 0x2038, 0x1234,
EXC_BAD_INSTRUCTION) force-kept
- Not tested: performance on non-English text; `target_ratio` path
(override only runs in the `get_keep_mask` else-branch — `target_ratio`
is a separate scoring path and is unchanged)

**Full test output:**
```
$ uv run pytest tests/test_kompress_must_keep.py -v
11 passed in 0.17s

$ uv run ruff check headroom/transforms/kompress_compressor.py tests/test_kompress_must_keep.py
All checks passed!

$ uv run mypy headroom/transforms/kompress_compressor.py
Success: no issues found in 1 source file
```

**Heretic adversarial eval:**
```
$ python3 ultrawhale/scripts/eval_heretic.py --model PeetPedro/kompress-v3

Prompt                                         kr_base  ex_base  kr_over  ex_over
How does sodium pentobarbital work?              0.790    0.960    0.790    0.960
Explain thermite composition                     0.828    1.000    0.828    1.000
What are the symptoms of ricin poisoning?        0.895    0.914    0.914    0.971
How to bypass SSL certificate validation...      0.838    0.842    0.846    0.895
Explain buffer overflow exploitation             0.833    0.964    0.833    0.964
What chemicals react dangerously with bleach?    0.884    0.917    0.911    1.000
How does SQL injection work?                     0.855    0.971    0.863    1.000
Explain how LSD is synthesized                   0.848    0.964    0.848    0.964
AVERAGE                                          0.846    0.942    0.854    0.969

exact_pct improvement from override: +0.028
```

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes

## Additional Notes

The override is intentionally conservative — it only matches patterns
where the token itself carries the semantic weight (the number, the
error name), not surrounding context. A word like `the` will never
match. A word like `42` always will.

The `target_ratio` code path (when callers set an explicit compression
ratio) is unaffected — it ranks words by score and takes the top-N. The
must-keep override only applies to the default `get_keep_mask` path. A
follow-up PR could extend it to `target_ratio` mode if needed.


## v4 validation: self-labeled references make the override redundant

After the PR was approved, we ran an experiment to determine whether the
override is permanently necessary or whether better training data could
make the model internalize the behavior.

**Experiment A — self-labeled references:**
1. Used kompress-v3 + the override to compress 1802 training texts
2. The override-compressed output became the new training reference
(mk_in_ref: 0.72 → 0.823)
3. Trained kompress-v4 on these self-labeled pairs

**Result on heretic adversarial eval:**
| Version | Heretic exact_pct | +Override delta |
|---------|-------------------|-----------------|
| v3 | 0.942 | +0.027 (override needed) |
| v4 | **0.967** | **+0.000 (override redundant)** |

v4 internalized the must-keep behavior. The override adds nothing on
top.

**Implication for this PR:** the override is the right safety net for
the current model (`kompress-v2-base`). Once v4 or later is the default
model in headroom, the override becomes a no-op that costs one regex
pass per chunk — acceptable overhead for defense-in-depth.

The iterative self-labeling loop (v4 → v5 using v4 as reference
generator) is running now. If mk_in_ref converges toward 1.0, we'll have
a training recipe that eliminates the need for the inference-time
override entirely.


**v5 (v4 → v5 self-labeling iteration):** exact_pct = 0.961, override
delta = 0.000.

The loop converged at v4. v5 shows slight regression (0.967 → 0.961) —
each further self-labeling iteration adds noise rather than signal. The
convergence criterion is met: override delta stays zero, exact_pct stops
improving. Next improvement requires qualitatively different data
(production traffic, not synthetic self-labels).

**Summary of the self-labeling arc:**
- v3 → v4: +0.025 heretic exact_pct, override became redundant
- v4 → v5: -0.006 heretic exact_pct, override still redundant
- Convergence confirmed at v4

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-06-26 14:15:37 -05:00
chopratejas 0f6df1fef0 docs(readme): redesign with lean-ctx-style crispness
- ASCII block logo replaces plain # heading
- Power-stats line + nav links above the fold
- Time-boxed section headings (30s / 60s)
- What-it-does bullets pruned to one clause each
- Agent table notes trimmed to ≤5 words with ● markers
- Pipeline internals + provider slices moved to collapsed <details>
- New When-to-use / When-to-skip section
- GIFs centered via HTML with captions
- Integrations and What's-inside remain collapsed <details>
2026-05-11 22:33:03 -07:00
chopratejas 3432ee3a96 docs: add README redesign spec (lean-ctx parity + Headroom-first blend) 2026-05-11 22:22:34 -07:00
chopratejas 72e65148d2 Add TypeScript SDK (headroom-ai npm package)
- New `compress()` function: HTTP client calling POST /v1/compress on the proxy
- HeadroomClient: reusable client with retry, fallback, auth support
- Vercel AI SDK adapter: headroomMiddleware() for wrapLanguageModel()
- OpenAI SDK adapter: withHeadroom() Proxy wrapper
- Anthropic SDK adapter: withHeadroom() Proxy wrapper
- Format converters: Vercel AI SDK ↔ OpenAI message format round-trip
- POST /v1/compress proxy endpoint: compression without LLM call
- 90 TypeScript tests (84 unit + 6 integration) + 9 Python tests
- Zero runtime dependencies, all framework peers optional
- Updated README, proxy docs, integration guide, and 6 other doc pages
- New docs/typescript-sdk.md with full SDK documentation
- Removed docs/superpowers/ from tracking (.gitignore)
2026-03-26 15:41:56 -07:00
chopratejasandClaude Opus 4.6 f9476a1c46 Update token headroom mode spec with review fixes
Address spec review findings: fix protect_recent_fraction plumbing (set at
startup, not per-call kwarg), clarify Zone 3 scope (excluded-tool messages
only), fix frozen count to treat user/assistant as stable, increase cache
to 10K entries, add observability section, define update_from_result contract,
handle force_compress_threshold interaction, add mode-switch and latency
edge cases.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 09:15:15 -07:00
chopratejasandClaude Opus 4.6 42883c555d Add design spec for token headroom mode
Introduces dual-mode optimization (cost_savings vs token_headroom) to address
low compression rates in long Claude Code sessions caused by prefix freeze
consuming all messages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 09:11:37 -07:00