Commit Graph
6 Commits
Author SHA1 Message Date
71663b44e2 fix(learn): pin the learned verbosity level on a cache-mode proxy (#4033)
## Description

`headroom learn --verbosity --apply` hot-enables the output shaper on a
running proxy and prints `level N is live now`. On a proxy in **cache
mode**, which is the default, that claim is false. The proxy never reads
`verbosity.json` there: `resolve_verbosity_level` returns the startup
level (`cache_mode_default`, L2) unless `HEADROOM_VERBOSITY_LEVEL` pins
one. A level that can change mid-conversation would bust the prefix
cache (#3773). So after `--apply` the shaper is on, but at L2 instead of
the learned level, while the CLI reports the learned level as live.

This PR makes `--apply` either do what it says or say what it did:

- **Cache mode, no pin:** the learned level is pinned through
`HEADROOM_VERBOSITY_LEVEL` in the same `/admin/runtime-env` update. That
is exactly the stable-pin path #3773 allows in cache mode.
- **An existing `HEADROOM_VERBOSITY_LEVEL`:** left alone, and the CLI
says which level stays in effect. The one exception is a pin this CLI
provably set. When the proxy confirms a pin (its `applied` map),
`--apply` records `{port, pid, started_at, level}` in
`verbosity_pin.json`. `started_at` is `timestamp - uptime_seconds` from
`/health`, because pids are reused. A later `--apply` replaces the pin
only while that same proxy process still holds that exact value. Hot
overrides die with the process, so a restarted proxy's pin, a reused
pid, or a value changed since stays the operator's.
- **Mode unreadable:** the shaper is still enabled, but the CLI no
longer claims the level is live.
- **Token mode:** unchanged. The proxy reads `verbosity.json` per
request, so no pin is added (a pin would also shadow the verbosity
autotune controller).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/learn.py`:
- New `_query_proxy_verbosity()` reads `config.mode` and
`config.runtime_env.HEADROOM_VERBOSITY_LEVEL` from the local proxy's
`/health`. The proxy includes that config block for loopback callers.
The level is read the way `OutputShaperSettings.from_env` reads it
(parse, default on garbage, clamp to 0–4), so `9` or `04` compare and
print as the level the proxy steers at.
- `_proxy_process()`, `_record_pin()` and `_pin_is_ours()` keep the
ownership record described above. Recording is best-effort: an `OSError`
no longer aborts `--apply`; the CLI reports the activation and notes the
pin wasn't recorded. The record is written with `newline="\n"`.
- `_activate_output_shaper()` returns `(status, port, applied)`, so a
pin the proxy applied while reporting the shaper "blocked" is still
recorded.
- `_activate_output_shaper()` takes an optional `verbosity_level` and
sends it in the same runtime-env request.
- The `--apply` messages distinguish the four cases above. The restart
hints now include `HEADROOM_VERBOSITY_LEVEL=<level>` for cache mode.
- The "learned level is used automatically" wording on the blocked and
no-proxy paths no longer claims cache mode reads the level.
- Those hints no longer prescribe `HEADROOM_ROLLOUT_CHANNEL=beta`:
`proxy_output_shaper` is available on every channel, and beta would also
enable unrelated features. "Blocked" now points at
`HEADROOM_DISABLE_FEATURES`, the only thing that refuses the hot enable.
- `tests/test_cli_learn.py`:
- CLI tests run `learn --verbosity --apply` against a stubbed local
proxy: cache-mode pin, token mode, an operator pin, an unreadable mode,
recording and then replacing this CLI's own pin, and keeping pins it
cannot prove it set. That last case covers an operator pin equal to the
previously learned level, a record from a restarted proxy, and a value
changed since. Also covered: out-of-range pins, re-pinning the same
level, and a proxy that disables the shaper. Each asserts both the
runtime-env body sent and the message printed.
- Further cases: a reused pid, a record without a start time, a pin
applied while the shaper is blocked, a failed record write, and a proxy
without a pid.
- Fallback branches: a non-numeric pin (read as the default level), a
`/health` reply without a config object, no usable proxy start time
(zero or missing uptime, missing or bad timestamp), and a record that
isn't a JSON object. Every new line and branch in
`headroom/cli/learn.py` is covered (`pytest --cov=headroom.cli.learn
--cov-branch`).
- A `windows_newline` test asserts that the ownership record is written
with `newline="\n"`.
- `tests/test_offline_egress_chokepoint.py`: `cli/learn.py` now has two
loopback egress sites (`/health` and `/admin/runtime-env`), recorded in
the allowlist with the same wording as `cli/wrap.py`.
- The existing `--all --apply` test made a real POST to whatever
listened on `127.0.0.1:8787`, which on a developer machine is their own
proxy. It now runs against a stubbed `urlopen`.
- `README.md`, `wiki/learn.md`: state that cache mode does not read the
learned level, and what `--apply` does there. `wiki/learn.md` no longer
says the shaper needs the beta channel.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# Before: origin/main's headroom/cli/learn.py with the new tests
$ pytest tests/test_cli_learn.py -k verbosity_apply -q
FAILED tests/test_cli_learn.py::test_verbosity_apply_pins_the_level_on_a_cache_mode_proxy
FAILED tests/test_cli_learn.py::test_verbosity_apply_leaves_a_token_mode_proxy_reading_the_profile
FAILED tests/test_cli_learn.py::test_verbosity_apply_does_not_override_an_explicit_level
FAILED tests/test_cli_learn.py::test_verbosity_apply_does_not_claim_live_when_the_mode_is_unknown
4 failed, 22 deselected in 1.04s

# Review round 3: the previous head (e10141c3) with this round's tests
$ pytest tests/test_cli_learn.py -q
FAILED tests/test_cli_learn.py::test_activate_output_shaper_reports_effective_rollout_decision[True-live]
FAILED tests/test_cli_learn.py::test_activate_output_shaper_reports_effective_rollout_decision[False-blocked]
FAILED tests/test_cli_learn.py::test_activate_output_shaper_handles_malformed_response
FAILED tests/test_cli_learn.py::test_verbosity_apply_records_the_pin_it_sets
FAILED tests/test_cli_learn.py::test_verbosity_apply_replaces_the_pin_an_earlier_apply_set
FAILED tests/test_cli_learn.py::test_verbosity_apply_keeps_a_pin_it_cannot_prove_it_set[no-start-time]
FAILED tests/test_cli_learn.py::test_verbosity_pin_record_is_written_with_lf
FAILED tests/test_cli_learn.py::test_verbosity_apply_records_a_pin_the_proxy_applied_while_the_shaper_is_disabled
FAILED tests/test_cli_learn.py::test_verbosity_apply_reports_activation_when_the_pin_cannot_be_recorded
FAILED tests/test_cli_learn.py::test_verbosity_apply_does_not_record_a_pin_without_a_process_identity
10 failed, 31 passed in 0.74s

# After: this branch
$ pytest tests/test_cli_learn.py -q
50 passed in 0.71s

# Offline-egress coverage (failed on e10141c3 in CI shard 3)
$ pytest tests/test_offline_egress_chokepoint.py -k test_every_egress_site_is_guarded_or_allowlisted -q
1 passed, 142 deselected in 2.63s

# Related suites (learn, verbosity, output shaper, runtime-env), with branch coverage of the changed file
$ pytest tests/test_*learn*.py tests/test_*verbosity*.py tests/test_*output_shaper*.py tests/test_*runtime_env*.py -q --cov=headroom.cli.learn --cov-branch
207 passed, 3 warnings in 7.83s
# lines/branches added by this PR in headroom/cli/learn.py that are not covered: none

$ ruff check headroom/cli/learn.py tests/test_cli_learn.py
All checks passed!
$ ruff format --check headroom/cli/learn.py tests/test_cli_learn.py
2 files already formatted
```

## Real Behavior Proof

- Environment: macOS (Apple Silicon), Python 3.11. A real `headroom
proxy --port 8799` in default cache mode from this checkout, with an
isolated `HEADROOM_WORKSPACE_DIR` and `--anthropic-api-url` pointed at a
closed local port, so no request left the machine. `headroom learn
--verbosity --apply` ran over real Claude Code transcripts and learned
level 3. Before = `origin/main`'s `headroom/cli/learn.py`; after = this
branch. A fresh proxy was used for each run.
- Exact command / steps: Start the proxy. Run `HEADROOM_PORT=8799
headroom learn --verbosity --apply --project <repo>`. Send one
`/v1/messages` request through the proxy. Read `/health`
`config.runtime_env` and the proxy's `OutputShaper(...)` log line.
- Observed result: before, the CLI says level 3 is live while the proxy
steers at L2. After, the level is pinned and the proxy steers at L3:

  ```text
  ===== BEFORE (origin/main learn.py), proxy in default cache mode
  >> Recommended verbosity level: 3 (confidence: high)
✓ Output shaper enabled on the running proxy (port 8799); level 3 is
live now (while HEADROOM_VERBOSITY_LEVEL is unset).
To keep it on across restarts: export HEADROOM_ROLLOUT_CHANNEL=beta and
HEADROOM_OUTPUT_SHAPER=1 before `headroom wrap ...`.
proxy /health: {"mode": "cache", "HEADROOM_OUTPUT_SHAPER": "1",
"HEADROOM_VERBOSITY_LEVEL": null}
  proxy log: OutputShaper(L2/cache_mode_default)

  ===== AFTER (this branch), proxy in default cache mode
  >> Recommended verbosity level: 3 (confidence: high)
✓ Output shaper enabled on the running proxy (port 8799); level 3 is
live now (pinned with HEADROOM_VERBOSITY_LEVEL: cache mode does not read
the learned level).
Conversations already open re-cache their prompt once, because the
steering text changed.
To keep it on across restarts: export HEADROOM_OUTPUT_SHAPER=1
HEADROOM_VERBOSITY_LEVEL=3 before `headroom wrap ...` or `headroom
proxy`.
  --- verbosity_pin.json: {"port": 8799, "pid": 25345, "level": 3}
  --- second --apply against the same proxy:
✓ Output shaper enabled on the running proxy (port 8799); level 3 is
live now (pinned with HEADROOM_VERBOSITY_LEVEL: cache mode does not read
the learned level).
proxy /health: {"mode": "cache", "HEADROOM_OUTPUT_SHAPER": "1",
"HEADROOM_VERBOSITY_LEVEL": "3"}
  proxy log: OutputShaper(L3/env_pinned)
  ```
- Not tested: a real token-mode proxy, one with an operator-set
`HEADROOM_VERBOSITY_LEVEL`, and a second `--apply` that learns a
*different* level (all covered by the CLI tests against a stubbed proxy;
the real run above re-learns the same level 3); multi-worker proxies,
which reject `/admin/runtime-env` before and after this change; Windows
(the record write is covered by a `windows_newline` test).

## Runtime Rollout Safety

- Rollout-managed feature(s): `proxy_output_shaper` (unchanged). This PR
only changes what the `headroom learn` CLI sends to and prints about a
running proxy.
- Minimum rollout channel: stable.
- Stable/default behavior changed: only `learn --verbosity --apply`
against a running cache-mode proxy with no `HEADROOM_VERBOSITY_LEVEL`.
It now also pins that variable to the learned level, which is what the
CLI already claimed. The proxy itself is unchanged.
- Kill switch / disable path: run `learn --verbosity` without `--apply`;
or POST `{"HEADROOM_VERBOSITY_LEVEL": ""}` / restart the proxy to drop
the hot override.
- Unsafe override required: no.
- Qualification impact: none expected.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Type checking was not run locally.
- Pin ownership: `/health` cannot say whether `HEADROOM_VERBOSITY_LEVEL`
came from startup env or a hot override, so `--apply` keeps its own
record. The process start time is derived from `/health` rather than
adding an instance id to the proxy, which keeps this PR CLI-only. The
one remaining ambiguity is an operator hot-POSTing the *same* value to
the *same* proxy process after `--apply` pinned it. That pin would be
treated as ours, but its value is the one `--apply` set anyway.
- Locally, 5 OTLP/Langfuse tests in `test_offline_egress_chokepoint.py`
fail on both `origin/main` and this branch because the dev venv lacks
`opentelemetry-exporter`; they pass in CI.


## Maintainer follow-up at 8c3eba8919

Ownership now compares the exact raw `HEADROOM_VERBOSITY_LEVEL` value
from the same health snapshot as process identity. A changed operator
value is preserved even when it clamps to the same effective level as a
previous CLI pin (for example, CLI4 followed by operator8). The display
still reports the effective clamped level. The new regression fails on
the preceding head and passes here; existing CLI ownership/restart/error
behavior remains covered.

Fresh validation:140 CLI/verbosity/output-shaper/runtime-env tests pass
with one existing warning; Ruff lint/format and Linux-platform mypy
over566files pass. Actual proxy/CLI proof additionally covers learning3,
re-learning1, preserving an operator8 pin, and the
changed-raw/same-clamped-level ownership case. The last case fails
before this follow-up and passes afterward.

Exact commands and full runnable Windows/Python3.12 script: [maintainer
evidence](https://github.com/headroomlabs-ai/headroom/pull/4033#issuecomment-6063120923).
Save the script as `verify_learn_pin.py` in the checkout root and run
`python verify_learn_pin.py` with proxy dependencies installed.
Synthetic Claude transcripts are processed by the real analyzer; no live
LLM judge or provider request is made. Hosted CI and fresh review must
qualify this new head before approval.

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-08 10:48:59 -05:00
TrîggeredandJD Davis 4ba231ab3d feat(learn): add agy (Antigravity CLI) as an analysis backend (#3939)
## Description

Adds the Antigravity CLI (`agy`) as a keyless LLM backend for `headroom
learn`, next to `claude`, `gemini` and `codex`. Antigravity subscription
users with no raw API key can now run the analysis through `agy`.

`agy` cannot use the existing stdin path as is. `agy -p` takes the
prompt as a flag argument and never reads stdin, so a large digest would
hit OS argument-length limits. The only stdin path is `--input-format
stream-json`, which requires `--output-format stream-json`. The backend
therefore sends the prompt as a single NDJSON user event on stdin and
reads the answer from `result.response` in the final `result` event.
Everything else (wall-clock timeout, Windows shim fallback, failure
detail, fenced-JSON parsing) reuses the existing `run()` path that
`gemini` and `codex` use.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/learn/analyzer.py`: add an `agy` / `agy-cli` entry to
`_CLI_BACKENDS` (`agy --input-format stream-json --output-format
stream-json`). `agy` is never auto-detected, and it only runs when
`HEADROOM_LEARN_ALLOW_UNSAFE_AGY=1` is set: `agy` has no zero-tool mode,
so transcript text could drive its tools. Without the opt-in,
`HEADROOM_LEARN_CLI=agy` and `--model agy-cli` fail before any
subprocess starts; with it, a warning is logged on every run.
- `headroom/learn/analyzer.py`: in `_call_cli_llm`, wrap the prompt as
`{"event": "user", "message": {"content": ...}}` for `agy-cli`, and
extract `result.response` from the stream with a new
`_agy_result_text()` helper before the existing `_strip_fenced_json()`
parsing. A missing `result` event surfaces as the existing "unparseable
output" error, with both ends of stdout.
- `tests/test_learn/test_analyzer.py`: tests that auto-detection never
selects `agy` (even when it is the only CLI on PATH and the opt-in is
set), that `HEADROOM_LEARN_CLI=agy` and `_call_cli_llm(..., "agy-cli")`
fail without the opt-in (no subprocess call), `HEADROOM_LEARN_CLI=agy`
with the opt-in, existing CLIs still taking precedence, the exact
command and stdin payload, result-event parsing (including a fenced
response and a literal U+2028), and a missing `result` event.
- `headroom/cli/learn.py`: check the agy opt-in during model resolution,
so `HEADROOM_LEARN_CLI=agy` and `--model agy-cli` without it exit 1 with
an `Error:` line before any sessions are scanned.
- `tests/test_cli_learn.py`: a CLI test for both paths without the
opt-in.
- `wiki/learn.md`: list `agy` among valid `HEADROOM_LEARN_CLI` values
and document the `HEADROOM_LEARN_ALLOW_UNSAFE_AGY=1` opt-in and its
risk.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_learn/ tests/test_cli_learn.py -q
5 failed, 345 passed, 4 skipped in 10.78s
# The same 5 fail on unmodified upstream/main on this Windows machine (5 failed, 340 passed),
# all path/git related and unrelated to this change:
#   test_analyzer.py::TestDigestBuilder::test_includes_project_info
#   test_scanner.py::TestDecodeProjectPath::test_home_dir_username_stays_single_component
#   test_writer.py::TestClaudeLocalMdStaysOutOfGit::{test_apply_adds_exclude_entry,
#     test_second_run_does_not_duplicate_the_entry, test_tracked_file_warns_instead_of_excluding}

$ uv run --frozen --extra dev pytest tests/test_learn/test_analyzer.py -q -k agy   # analyzer.py reverted
FAILED TestDetectDefaultModel::test_env_var_selects_agy_with_unsafe_opt_in
FAILED TestDetectDefaultModel::test_env_var_agy_without_unsafe_opt_in_raises
FAILED TestCallCliLlm::test_agy_cli_sends_stream_json_and_parses_result_event
FAILED TestCallCliLlm::test_agy_cli_result_with_unicode_line_separator
FAILED TestCallCliLlm::test_agy_cli_without_unsafe_opt_in_does_not_run
FAILED TestCallCliLlm::test_agy_cli_missing_result_event_raises
6 failed, 2 passed, 97 deselected
# All 8 pass with the change.

$ uvx ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ uvx ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/learn/analyzer.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, `agy` 1.2.14 signed in via subscription, no
API keys set.
- Exact command / steps: `HEADROOM_LEARN_CLI=agy
HEADROOM_LEARN_ALLOW_UNSAFE_AGY=1 ANTHROPIC_API_KEY= OPENAI_API_KEY=
GEMINI_API_KEY= uv run --frozen --extra dev headroom learn` in this
repository (dry run).
- Observed result: `Analyzing with agy-cli...`, then `Sessions: 3 |
Calls: 109 | Failures: 19 (17.4%)`, `Recommendations: 8`, with `[WOULD
WRITE]` blocks for `CLAUDE.local.md` and `MEMORY.md`; exit 0. Before
wiring the backend I confirmed the protocol by hand: `agy -p -` answers
the literal prompt `-` and ignores stdin, while the stream-json
invocation returns the answer in `{"event": "result", "result":
{"status": "SUCCESS", "response": "..."}}`.
- Not tested: macOS/Linux `agy` installs; `agy` under a permission mode
that prompts for tool approval (my config is `always-proceed`), where a
tool request in print mode would be bounded by the existing
`HEADROOM_LEARN_CLI_TIMEOUT_SECS` wall-clock cap.

## Runtime Rollout Safety

- Rollout-managed feature(s): None (offline `headroom learn` CLI)
- Minimum rollout channel: dev
- Stable/default behavior changed: No. `agy` is never auto-detected; it
is only used when selected with `HEADROOM_LEARN_CLI=agy` / `--model
agy-cli` together with `HEADROOM_LEARN_ALLOW_UNSAFE_AGY=1`.
- Kill switch / disable path: Set an API key, `HEADROOM_LEARN_CLI`, or
`--model` to pick another backend.
- Unsafe override required: None
- Qualification impact: None
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A (CLI change).

## Additional Notes

- No tracking issue exists for an `agy` analysis backend, so `Closes #`
is left empty. Related Antigravity work (#3428 `headroom init -g agy`,
#3610 Antigravity transcripts in `headroom learn`) is independent of
this change.
- Unlike `claude-cli` (#3892), `agy` has no flag to start a session with
zero tools. I deliberately did not add `--dangerously-skip-permissions`:
the digest contains untrusted transcript content, and the backend should
run under the user's own permission settings.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3939"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-04 13:56:26 -07:00
Tejas ChopraandClaude Opus 5 27bdc2a899 docs(wiki): correct the unpublished tree, and remove invented telemetry (#3390)
`wiki/` **is not published anywhere.** `.github/workflows/docs.yml` says
so in its own header, no mkdocs or sphinx config exists, and `wiki/**`
sits in `paths-ignore` for CI — changes there don't even run a build. It
survives as pending-migration reference material for contributors.

That's why this is separate from the published-site work in #3388:
nothing here reaches a user through `docs.headroomlabs.ai`. It still
matters, because a contributor migrating a page would carry its errors
across, and because the repo is public.

## The finding that justifies this PR on its own

`wiki/benchmarks.md` carried an entire **invented "Production Telemetry"
section**, presented as real-world fleet data:

> real-world production telemetry from 250+ active proxy instances …
**52ms median overhead** … **1.4 billion tokens saved** across 249
instances

Plus P90/P99 timings, ~$4,000 saved, an OS split, a version-share
breakdown, and a dated collection window of March 30 – April 2, 2026.

A repo-wide grep for those figures **matches only that one file.** No
script, test or fixture produces them. The published benchmarks page
never carried the section, so the site owner had already excised it.

Removed rather than re-estimated. The per-category ranges hanging off it
(4.8%, 40–80%, 83–94%, 85–94%, 60–76%) went with it and now point at the
concrete compression table on the same page.

The same page also described `HEADROOM_TELEMETRY` as the opt-in switch
governing fleet-wide numbers. Fleet data would come from
`HEADROOM_BEACON` — a **separate switch that defaults ON**. That note is
gone with the section it footnoted; the accurate description of both
switches shipped to the live site in #3388.

## Other corrections, each verified against source

- **`wiki/agno.md`** constructed `HeadroomAgnoModel(config=...)`; the
field is `headroom_config` (`integrations/agno/model.py:135`). The
sibling `HeadroomPreHook` genuinely does take `config`, which is how
they got conflated.
- **`wiki/ccr.md`** documented CLI flags that don't exist, and a dead
demo script.
- **`wiki/image-compression.md`** had two broken CLI examples.
- **`wiki/getting-started.md`** used `plan.transforms_applied`;
`SimulationResult` exposes `transforms`. (`CompressResult` really does
have `transforms_applied` — two different result types, easy to cross.)
- **`wiki/cli.md`** documents 15 of ~27 top-level commands; an audit
note now says so rather than implying completeness.

## Disposition, for whoever picks up the migration

`wiki/cli.md` is the highest-value page to port — no site page is a full
CLI reference, and `docs.yml` names it as the reason `wiki/` still
exists. `wiki/integration-guide.md` and `wiki/configuration.md` also
cover ground the site doesn't.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01HRaEPjQcCSAwD2xbQMNENu

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-02 15:53:02 +05:30
3077ac81e8 feat: add deterministic runtime rollout controls (#1490)
## Description

Establish one centrally resolved, observable, deterministic, versioned
runtime rollout-control mechanism for Headroom. Runtime rollout controls
which behaviors an already-built artifact may expose; it does not select
or qualify a Headroom release/version.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Bug fix (non-breaking change that fixes rollout enforcement
regressions)
- [x] Documentation update
- [x] Code refactoring (no functional changes)

## Changes Made

- Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`,
`--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared
by Python configuration boundaries.
- Added schema/policy versions, canonical registry and snapshot SHA-256
identities, per-feature decision reasons, disable precedence, unsafe
qualification poisoning, strict CLI validation, and fail-closed
environment handling.
- Added `headroom rollout status --json`, Python `/stats.rollout`, and
Rust `/rollout/status` runtime provenance.
- Added equivalent Rust snapshot semantics and shared Python/Rust policy
vectors while retaining language-specific feature registries.
- Enforced rollout policy at alternate Python server composition roots
so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate.
- Preserved typed rollout snapshots across multi-worker serialization
with schema, policy, registry, snapshot-digest, type, and feature-name
validation.
- Made loopback runtime output-shaper updates replace the immutable
snapshot atomically for request readers, retain explicit request/disable
provenance, preserve channel and kill-switch precedence, invalidate
cached stats, and return the effective rollout decision.
- Made `headroom learn --verbosity --apply` report a channel-blocked
update instead of claiming the shaper is live.
- Made explicit CLI feature flags fail loudly when their current channel
blocks them.
- Made persistent interceptor installation select canary automatically,
or reject an explicitly insufficient channel unless the break-glass
override is set.
- Updated architecture, proxy, rollout, learn, and output-shaper
documentation with required channels and hot-reload semantics.

## Testing

- [x] Unit tests pass
- [x] Linting passes (`ruff check .` and `ruff format --check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New regression tests added for every corrected behavior
- [x] Rust tests and production-target Clippy pass
- [x] Documentation build passes

### Test Output

```text
Focused rollout coverage suite
57 passed; headroom.rollout + rollout CLI: 98% coverage

Affected proxy/rollout/transform/governance suites
222 passed; 0 failed

Final changed regression suites
100 passed; 0 failed

Cross-module hot-reload isolation regression
6 passed; 0 failed

cargo test -p headroom-core -p headroom-proxy --quiet
headroom-core: 924 passed; 1 ignored
headroom-proxy and integration suites: all passed

cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings
cargo fmt --all -- --check
ruff check .
ruff format --check .
mypy headroom --ignore-missing-imports
git diff --check
All passed

cd docs && npm run build
Compiled successfully; 164 static pages generated
```

The unsharded Windows-only CI selection exposed unrelated baseline
failures, principally the existing `sqlite:///C:\\...` URL parser
producing an invalid `\\C:\\...` path. At commit `8e793a80`, all 52
completed GitHub checks passed; the only other conclusions are expected
skips and superseded governance jobs.

## Real Behavior Proof

- **Environment:** Windows checkout on Python 3.13.3 and the current
Rust workspace, based on upstream `main` at `93f2d7a2`.
- **Exact command / steps:** Exercised canary and beta feature requests
through CLI status, Python `/stats.rollout`, Rust `/rollout/status`,
multi-worker payload round trips, loopback `/admin/runtime-env`, real
proxy request shaping before/after hot reload, installer manifest
generation, and shared Python/Rust policy vectors.
- **Observed result:** Stable blocks unstable requests; disable wins
over explicit/default/legacy/unsafe paths; unsafe state reports
`qualification_eligible=false`; worker handoff rejects tampering;
running output shaping changes only when the effective beta policy
permits it; explicit blocked flags fail with actionable diagnostics.
- **Not tested:** Live production traffic requiring provider
credentials, or future artifact qualification/promotion automation
(intentionally out of scope).

## Runtime Rollout Safety

- **Rollout-managed features:** Python `tool_result_interceptors`,
`proxy_output_shaper`, `read_maturation`; Rust `native_bedrock`,
`openai_responses_streaming`, `canary_probe`.
- **Minimum rollout channel:** Registry-defined per feature; process
default is `stable`.
- **Stable/default behavior changed:** No unstable feature becomes
enabled by default. Explicit blocked CLI flags now fail instead of
silently doing nothing.
- **Kill switch / disable path:**
`HEADROOM_DISABLE_FEATURES=<comma-separated feature names>`; explicit
disable has highest precedence, including over the unsafe override.
- **Unsafe override required:** No.
`HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES=1` is break-glass only and
makes qualification evidence ineligible.
- **Qualification impact:** Adds machine-readable policy/snapshot
identities and eligibility; does not implement qualification itself.
- **Rollback path:** Set the named disable list for operational
rollback, lower the channel, or revert this PR.

## Review Readiness

- [x] I have performed a full diff review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented hard-to-understand areas
- [x] I have made corresponding documentation changes
- [x] My changes generate no new warnings
- [x] I added tests that reproduce and prevent every regression fixed
during review
- [x] New and existing affected tests pass locally
- [x] I did **not** edit `CHANGELOG.md`; release-please generates it
from the Conventional Commit PR title

## Additional Notes

Out of scope: artifact candidates, benchmark orchestration,
qualification manifests/gates, promotion automation, release branches,
publication guards, and release-risk classification. Those workflows can
consume the rollout registry digest, runtime snapshot digest, decision
reasons, and qualification eligibility through supported black-box
interfaces.

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JD Davis <jd@JDH-AIR-00.local>
2026-08-12 23:16:54 -05:00
Tejas Chopra bd76235f5c fix(cli): harden all CLI surfaces + fix docs accuracy (#1491)
## Summary

Full CLI audit + documentation accuracy pass. All 5 commits on this
branch:

### CLI Hardening (4 commits)
- **Clean errors instead of tracebacks**: corrupt manifests, missing
Docker, malformed JSONL, bad `--profile`, invalid env-var values all now
raise `click.ClickException` with helpful messages
- **Range validation**: ~25 numeric flags across 10 files now use
`click.IntRange`/`FloatRange` — `--port 0`, `--hours -1`, `--limit 0`
etc. produce clean usage errors instead of silent wrong behavior
- **Flag combination warnings**: conflicting combos (`--no-rate-limit` +
`--rpm`, `--no-optimize` + `--target-ratio`, `--telemetry` +
`--no-telemetry`) emit yellow warnings on stderr
- **`memory --db-path` default fixed**: was resolving to
`headroom_memory.db` (wrong bare file); now uses project store
`./.headroom/memory.db` if present, else `~/.headroom/memory.db`
- **`memory list --search` + filters**: `--scope`/`--session`/`--since`
were silently ignored when `--search` was also set; now filters are
applied to search results
- **`learn --verbosity --apply` now works**: the output shaper is off by
default (`HEADROOM_OUTPUT_SHAPER`); `--apply` now hot-enables it via
`POST /admin/runtime-env` on a running proxy, or prints explicit `export
HEADROOM_OUTPUT_SHAPER=1` instructions when no proxy is running
- **`perf --hours` overflow**: `1e9` hours no longer raises
`OverflowError`; treated as "all data"
- **`evals memory --categories` invalid input**: `abc,1,2` now raises
`BadParameter` instead of a raw `ValueError` traceback

### Documentation (1 commit, 20 files)

Corrected factual errors found by 3 parallel audit agents across root
docs, wiki, and the published Fumadocs site:

**Critical (caused runtime errors or wrong behavior if followed):**
- `simulation.mdx`: `plan.transforms_applied` -> `plan.transforms`;
`plan.savings_percent` -> computed from available fields (both raised
`AttributeError`)
- `shared-context.mdx`: `import { SharedContext } from "headroom"` ->
`"headroom-ai"` (5x `ImportError`)
- `claude-code-azure-foundry.mdx`: `pip install headroom` -> `pip
install headroom-ai`
- `api-reference.mdx` + `configuration.mdx`: `from headroom import
GoogleProvider` -> `from headroom.providers import GoogleProvider`
- `ccr.mdx`: CCR TTL default 300s -> 1800s (30 min)

**Fabricated flags removed:**
- `wiki/proxy.md` + `wiki/cli.md`: `--no-intelligent-context`,
`--no-intelligent-scoring`, `--no-compress-first` (none exist); replaced
with real CCR flags
- `wiki/configuration.md`: `--no-ccr-responses`, `--no-ccr-expansion`
(none exist); replaced with real flags
- `wiki/troubleshooting.md`, `wiki/metrics.md`,
`docs/troubleshooting.mdx`: `headroom proxy --log-level debug` (flag
doesn't exist)

**Stale content corrected:**
- `llms.txt`: telemetry stated as enabled-by-default (it's opt-in); wrap
list had 5 tools (now 11)
- `README.md`: compatibility matrix added 5 missing `wrap` targets;
`unwrap`, `doctor`, `init`/`install`, savings-analytics now mentioned
- `SECURITY.md`: supported version table showed 0.2.x (current: 0.27.x)
- `wiki/learn.md`: 5 missing flags added; verbosity shaper-off behavior
documented
- `wiki/quickstart.md`: "Configuration Reference" linked to `api.md`
(wrong) -> `configuration.md`
- `CacheAlignerConfig.enabled` default corrected: `True` -> `False`
- `opencode.mdx`: `--port` default wrong ("random") -> 8787; `openai`
backend removed
- `CONTRIBUTING.md`: broken Markdown table cell fixed
- `docs/meta.json`: `claude-code-azure-foundry` added to nav (was
unreachable orphan page)
- `configuration.mdx`: SDK modes vs proxy `--mode` now clearly
distinguished

## Test plan

- [x] `python -m pytest tests/ -x -q` — 857 passed, 0 failures
- [x] 41-combination CLI smoke test (all flag combos across 8 commands)
— 0 tracebacks
- [x] `ruff check` on all modified Python files — clean
- [x] Docs changes are removals/corrections of fabricated or stale
content; no new claims introduced
2026-06-27 14:48:43 -07:00
Adib Mohsin 911eb85a44 new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00