Commit Graph
5 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5.5 310c108c24 refactor: remove test-only compatibility helpers in rollout and agent_savings (#4089)
## Description

This PR removes four compatibility helpers whose only callers were their
own tests:

| Name | What it was |
|---|---|
| `headroom.rollout.current_rollout(environ)` | A wrapper that only
called `resolve_rollout(environ)`. |
| `headroom.rollout.feature_enabled(feature, *, explicit, environ)` | A
wrapper around `resolve_rollout(environ,
requested=...).is_enabled(feature)`. |
| `headroom.rollout.Rollout` | The alias `Rollout = RolloutSnapshot`.
Its own comment called it an undocumented in-branch shim for callers
that were migrating. Those callers have migrated. |
| `headroom.agent_savings.with_target_savings(profile, target_savings)`
| A `dataclasses.replace` helper that nothing called. |

Why this is safe to delete rather than deprecate:
- Neither module has an `__all__`, and these names are not re-exported
anywhere.
- None of the names is documented in `docs/content/docs/`.
`configuration.mdx` documents the rollout environment variables, not
these functions.
- None of the org's consumer repos imports `headroom.rollout` or
`headroom.agent_savings`.

Evidence: a word-boundary search across `headroom`, `crates`, `plugins`,
`sdk`, `examples`, `benchmarks`, `scripts`, `e2e`, `docs`, `wiki`,
`.github`, `pyproject.toml`, `tests` and the consumer repos finds only
the definitions and their tests. The `feature_enabled` hits in
`tests/test_backends/test_litellm_bedrock_prompt_caching.py` are an
unrelated pytest fixture with the same name.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `headroom/rollout.py`: delete `current_rollout`, `feature_enabled` and
the `Rollout` alias.
- `headroom/agent_savings.py`: delete `with_target_savings`, and the
`replace` import it alone used.
- `tests/test_rollout.py`:
`test_snapshot_query_and_compatibility_helpers` becomes
`test_snapshot_query_helpers`. It makes the same assertions through
`resolve_rollout(...)` directly, which is what the helpers wrapped.
- `tests/test_agent_savings.py`: delete
`test_with_target_savings_recomputes_target_ratio`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/test_rollout.py tests/test_agent_savings.py tests/test_backends/test_litellm_bedrock_prompt_caching.py tests/test_config.py
146 passed in 6.83s
$ uvx ruff@0.16.8 check headroom/rollout.py headroom/agent_savings.py tests/test_rollout.py tests/test_agent_savings.py
All checks passed!
$ python -m mypy headroom/rollout.py headroom/agent_savings.py --ignore-missing-imports
Success: no issues found
$ rg -n 'current_rollout|feature_enabled\(|with_target_savings' headroom tests docs wiki
tests/test_backends/test_litellm_bedrock_prompt_caching.py:102:def feature_enabled(monkeypatch: pytest.MonkeyPatch) -> None:   # unrelated fixture
```

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, this branch checked out in a
worktree, repo .venv; temp HOME and HEADROOM_WORKSPACE_DIR, offline.
- Exact command / steps: (1) The shared smoke harness `smoke.sh
<worktree>`: it starts a real proxy from this checkout against a local
fake upstream and sends one Anthropic and one OpenAI request with a
compressible tool result. (2) A direct command on this PR's own path:
`HOME=$TMP/home HEADROOM_WORKSPACE_DIR=$TMP/ws
HEADROOM_ROLLOUT_CHANNEL=canary
HEADROOM_FEATURES=tool_result_interceptors python -m headroom.cli
rollout status`.
- Observed result: Both requests return 200 and `/stats` shows savings,
with only the expected offline-mode warnings in proxy.log. The direct
command's output is below.
- Not tested: An outside caller of the removed undocumented helpers;
none exists in the repo or the org's consumer repos.

```text
$ bash smoke.sh <worktree>
branch: cleanup/core-misc-test-only-helpers @ af71b61a
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE_TMP/ws/logs/proxy-50198.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   2 headroom.transforms.kompress_compressor - WARNING - Kompress: 'onnx/kompress-int8-wo.onnx' for chopratejas/kompress-v2-base is not cached and HEADROOM_OFFLINE forbids fetching it (HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.). Compression falls back to the non-ML path.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE_TMP)

$ HOME=$TMP/home HEADROOM_WORKSPACE_DIR=$TMP/ws HEADROOM_ROLLOUT_CHANNEL=canary HEADROOM_FEATURES=tool_result_interceptors python -m headroom.cli rollout status
Rollout channel: canary
Policy: 1 (sha256:7752c57ebd846a5cf5f1f3d56c1a1b3e84b69bef54b5e67da6353f51e5dded2f)
Snapshot: sha256:d0a3cdbe4b506547100168eeaba3ac4c1b086876ca1b97209962080146897b2f
Qualification eligible: true
  bedrock_openai_prompt_caching: enabled=false decision=not_requested
  proxy_output_shaper: enabled=false decision=not_requested
  read_maturation: enabled=false decision=not_requested
  tool_result_interceptors: enabled=true decision=explicit
exit=0
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None changed. `resolve_rollout`,
`RolloutSnapshot` and `FEATURES` are untouched.
- Minimum rollout channel: N/A; dead-code removal.
- Stable/default behavior changed: No.
- Kill switch / disable path: Unchanged (`HEADROOM_DISABLE_FEATURES`).
- Unsafe override required: No.
- Qualification impact: None. The registry digest and snapshot format
are unchanged.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

The parser helpers that were first planned for this PR,
`find_tool_units` and `get_message_content_text`, live in a
consumer-imported module. They are deprecated in a separate PR instead
of being deleted.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 08:57:49 -07:00
Sabee Ur Rehman KhanandJerrettDavis 58b1454282 fix(settings): report live proxy configuration (#3177)
## Description

The settings schema re-reads process environment values after the CLI
has already resolved the live proxy configuration. Savings-profile
defaults seeded later in startup can therefore make `/settings/schema`
report values that are not actually active. For example, `--lossless`
can produce `ProxyConfig.lossless=True` while the schema reports the
seeded `HEADROOM_LOSSLESS=0`.

This change makes the live `ProxyConfig` the highest-precedence settings
source, preserves provenance for profile-generated environment defaults,
and adds the missing `HEADROOM_MODE` registry field.

Closes #3090

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `HEADROOM_MODE` with canonical `token` and `cache` choices.
- Made live `ProxyConfig` values take precedence in `/settings/schema`.
- Added `runtime_override` metadata for live values that differ from
lower tiers.
- Prevented savings-profile defaults from being labeled as explicit
environment overrides.
- Preserved provenance through multi-worker configuration serialization.
- Added regression tests for precedence, provenance, endpoint reporting,
and worker round-tripping.

## Testing

- [ ] Unit tests pass (`pytest`) — focused settings tests pass; full
suite not run
- [x] Linting passes (`ruff check .`) — changed files checked
- [ ] Type checking passes (`mypy headroom`) — not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ focused settings-store test run
36 passed, 1 deselected in 0.09s

$ direct savings-profile provenance check
seed provenance check passed (18 generated keys)

$ ruff check <8 changed files>
All checks passed!

$ ruff format --check <8 changed files>
8 files already formatted

$ python -m compileall -q <8 changed files>
# No output

$ git diff --check
# No output
```

## Real Behavior Proof

- Environment: Linux x86_64, Python 3.12.13, source based on upstream
`5e0ce242` (v0.36.2).
- Exact command / steps: Set profile-seeded `HEADROOM_MODE=cache` and
`HEADROOM_LOSSLESS=0`, then generated the schema with live values
`mode=token` and `lossless=True`, excluding those generated keys from
explicit-environment provenance.
- Observed result: The schema reports `mode=token` and `lossless=true`;
both generated variables report `env_override=false`, and both
differences report `runtime_override=true`.
- Not tested: Complete repository and FastAPI endpoint suites. This
environment lacks the compiled `headroom._core` extension and Rust
toolchain, and dependency downloads are unavailable.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Proxy compression behavior is
unchanged; only settings registry/schema reporting changes.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: The dashboard receives the actual live proxy
values.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A; no
user documentation change required
- [ ] My changes generate no new warnings — focused checks pass; full
suite not run
- [x] I have added tests that prove my fix is effective
- [ ] New and existing unit tests pass locally — focused tests pass;
full suite not run
- [x] I did **not** edit `CHANGELOG.md` — generated by release-please

## Screenshots (if applicable)

N/A — settings API behavior only.

## Additional Notes

This is intentionally narrow for #3090. PR #2346 overlaps by adding
`HEADROOM_MODE`, but currently includes unrelated dashboard/runtime
work; this PR isolates the live-config precedence and provenance
correction.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 12:25:26 -05:00
Ayush Kumar JhaandJD Davis 0c19244e38 feat(litellm): opt-in Bedrock prompt caching on the OpenAI-compatible path (#3565)
## Description

On `--backend bedrock`, the OpenAI-compatible route
(`send_openai_message` / `stream_openai_message`) forwards the client's
`messages` verbatim. A client that never sends `cache_control` —
OpenAI-compat gateways such as Bifrost strip it — therefore never gets a
Bedrock `cachePoint`, and `cache_read_input_tokens` never appears in
usage even for models litellm reports as caching-capable.

Note on the issue's stated root cause: there is no
`litellm.get_supported_openai_params` gate anywhere in headroom (checked
`main` and the v0.37.0 commit the reporter pinned), and litellm's
Converse transform honours `cache_control` unconditionally. The caching
half of #3554 is a missing behaviour, not a wrong API call, so this PR
adds it **opt-in** rather than changing the default. The
`max_completion_tokens` half is #3558.

Design:
- New rollout feature `bedrock_openai_prompt_caching` (stable channel,
**not** default-enabled): a breakpoint bills the system prompt at the
cache-write rate until it is read back, so a one-shot caller who did not
ask for it would pay more.
`HEADROOM_FEATURES=bedrock_openai_prompt_caching` turns it on; flipping
the default later is a one-line `default_enabled_in` change if you want
it.
- When enabled and `litellm.utils.supports_prompt_caching()` is true for
the **mapped** litellm model
(`bedrock/us.anthropic.claude-sonnet-5-v1:0`, not the client alias —
`claude-sonnet-5` alone is answered from litellm's Anthropic-direct
registry entry, not the Bedrock one), the first system message gets
`cache_control: {"type": "ephemeral"}`; litellm turns it into a Converse
`cachePoint` covering tools + system prefix. First (not last) so
volatile trailing system messages cannot turn every turn into a cache
write.
- Untouched: requests that already carry client-placed markers anywhere
(the client owns placement, as on the Anthropic route), models without
prompt caching (incl. ARN-mapped inference profiles, which litellm
cannot look up), non-Bedrock providers. The caller's `body["messages"]`
is never mutated (the proxy re-reads it for `POST_SEND`).

Refs #3554 (the `max_completion_tokens` half is #3558)

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/rollout.py`: register `bedrock_openai_prompt_caching`
(available in `stable`, opt-in, no legacy env alias).
- `headroom/backends/litellm.py`: import
`litellm.utils.supports_prompt_caching` alongside `acompletion`; add
pure helper `_place_system_cache_control()`; resolve the rollout
decision once in `LiteLLMBackend.__init__` (Bedrock only); gate both
OpenAI call sites on the decision + capability lookup (~20 µs/call, no
memoisation needed).
- `tests/test_backends/test_litellm_bedrock_prompt_caching.py`: 26 tests
— helper edge cases (empty/none/wrong-type content, list vs string
content, non-dict entries, first-of-several, client-placed markers
anywhere, input never mutated), gate off by default (identity), on for
buffered + streaming, skips non-caching models, non-Bedrock no-op
(lookup not even called), uses the mapped id, kill switch, and litellm's
real Converse transform emitting `cachePoint` from the produced
messages.
- `tests/test_rollout.py`: one assertion pinned a positional index into
the alphabetically sorted feature list; made it name-based so adding a
feature does not shift it.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_backends/test_litellm_bedrock_prompt_caching.py -q
26 passed in 4.11s

# same file on unpatched main (helper does not exist):
E   ImportError: cannot import name '_place_system_cache_control' from 'headroom.backends.litellm'

$ python -m pytest tests/test_backends tests/test_litellm_openai_passthrough.py tests/test_litellm_caller_key.py \
    tests/test_litellm_nonstream_cache_usage.py tests/test_litellm_optional.py tests/test_litellm_upstream_timeout.py \
    tests/test_rollout.py tests/test_bedrock_region.py tests/test_bedrock_streaming_input_tokens.py \
    tests/test_bedrock_tool_result_cache_and_streaming_stats.py tests/test_bedrock_prefix_tracker_wiring.py \
    tests/test_proxy/test_openai_backend_path.py tests/test_proxy/test_bedrock_passthrough.py -q
189 passed  (after the test_rollout.py assertion fix; 188 passed, 1 failed before it)

$ ruff check headroom/backends/litellm.py headroom/rollout.py tests/test_backends/test_litellm_bedrock_prompt_caching.py tests/test_rollout.py
All checks passed!
$ ruff format --check headroom/backends/litellm.py headroom/rollout.py tests/test_backends/test_litellm_bedrock_prompt_caching.py tests/test_rollout.py
4 files already formatted
$ mypy headroom/backends/litellm.py headroom/rollout.py
Success: no issues found in 2 source files

# Full local suite (12,650 collected; Windows, borrowed 0.35.0 headroom._core, known Windows-only
# CLI tests deselected) was still running when this PR was opened; result will be added below.
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.11.9, litellm 1.97.0 (issue reports
1.96.2; same `supports_prompt_caching` / Converse behaviour), boto3
1.43.93, headroom source checkout at this branch,
`LiteLLMBackend(provider="bedrock", region="us-east-1")`, model
`global.anthropic.claude-sonnet-5`. No live AWS account available to me,
so the request was driven through litellm's real pipeline (message
validation, Converse transformation, SigV4 signing) and the outgoing
`AsyncHTTPHandler.post` was intercepted to capture the exact wire body
and return a canned Bedrock response.
- Exact command / steps:
`HEADROOM_FEATURES=bedrock_openai_prompt_caching python wire_proof.py`
(script in the PR discussion) calling
`backend.send_openai_message({"model":
"global.anthropic.claude-sonnet-5", "max_tokens": 64, "messages":
[{"role": "system", "content": <~1800 tokens>}, {"role": "user",
"content": "hi"}]}, {})`; then the same command without the env var.
- Observed result: with the feature on, the captured POST to
`https://bedrock-runtime.us-east-1.amazonaws.com/model/global.anthropic.claude-sonnet-5/converse`
carries `"system": [{"text": ...}, {"cachePoint": {"type":
"default"}}]`, SigV4 `Authorization` header present, and the client's
`body["messages"]` still contains no `cache_control`. Without the env
var (default) the same request carries `"system": [{"text": ...}]` only
— byte-identical to today's behaviour. `headroom rollout status` lists
`bedrock_openai_prompt_caching: enabled=false decision=not_requested` by
default.
- Not tested: a live Bedrock round trip (no AWS credentials here), so
the returned `cache_creation_input_tokens` / `cache_read_input_tokens`
were not observed from Bedrock itself; Bifrost/Cline in front of the
proxy; Nova models (litellm reports `supports_prompt_caching=True` for
them, so they would be marked too); the Anthropic-format route is
untouched by this change.

## Runtime Rollout Safety

- Rollout-managed feature(s): `bedrock_openai_prompt_caching` (new entry
in `headroom/rollout.py` FEATURES).
- Minimum rollout channel: stable (explicit request required in every
channel; no `default_enabled_in`).
- Stable/default behavior changed: no. Without
`HEADROOM_FEATURES=bedrock_openai_prompt_caching` the backend forwards
the client's `messages` object untouched (pinned by an identity
assertion for buffered and streaming). Only `--backend bedrock` can ever
be affected.
- Kill switch / disable path: unset `HEADROOM_FEATURES`, or
`HEADROOM_DISABLE_FEATURES=bedrock_openai_prompt_caching` (explicit
disable wins over an explicit request; tested).
- Unsafe override required: no.
- Qualification impact: none; the feature is `not_requested` in the
default snapshot, so the snapshot digest of an unconfigured deployment
only changes by the registry digest (as for any new feature).
- Rollback path: revert the commit; no persisted state, schema, or
config format involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: not changed on purpose — there is no catalogue of
rollout features in `docs/`; the `FeatureSpec.description` is what
`headroom rollout status` prints. Happy to add a line to
`docs/content/docs/runtime-rollouts.mdx` if you want one.
- Open questions for the maintainer: (1) default-on for Bedrock instead
of opt-in? One-line change (`default_enabled_in=RolloutChannel.STABLE`),
but it changes billing for one-shot callers. (2) Prefer the *last*
system message (max cache coverage) over the first (robust to volatile
trailing system messages)? (3) ARN-mapped application inference profiles
(`HEADROOM_BEDROCK_MODEL_MAP`) get nothing because litellm cannot look
them up — acceptable, or gate on the underlying model family instead?
- Alternatives considered: default-on with no flag (rejected: silent
cost change, no kill switch); reusing
`AnthropicCacheOptimizer._insert_breakpoints` (Anthropic-format only,
mutates in place); doing it in the OpenAI handler (provider-agnostic,
lacks the mapped model id).
- Verification for someone with Bedrock access: run the proxy with
`--backend bedrock` and
`HEADROOM_FEATURES=bedrock_openai_prompt_caching`, send two identical
OpenAI chat completions with a >1024-token system prompt within 5
minutes; the second response's `usage` should carry non-zero
`cache_read_input_tokens` / `prompt_tokens_details.cached_tokens`.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-24 22:13:15 -05:00
Tejas Chopra 1390d89715 fix(shaper): measure output-token steering, and stop claiming savings we did not measure (#3377)
## What this does

Verbosity steering already shipped (BETA, L2). This branch measures it
properly, fixes two ways it was misbehaving, and changes what Headroom
is willing to *claim* about it.

Net effect for a user: steering saves more than it did, on more models,
and the dashboard stops showing a savings percentage that was never
measured on their traffic.

## Three changes worth reviewing separately

### 1. The L3 steering text had two defects, both measured

**"Reproduce code, error strings, numbers, units, and API or CLI names
exactly"** read as a licence to emit code. Against the same SWE-bench
turn it left gpt-5.2 unchanged but inflated gpt-5.1 by 58% (1631 → 2573
output tokens), cutting L3's saving on that model from 51% to 23%.
Replaced with a completeness floor that forbids dropping task-critical
content — negations included — without instructing the model to
reproduce anything. The whole OpenAI family went from *negative* to
+31.2%.

**"reference by path and line"** was too soft. Told to stop restating
code, models wrote "change the check in the mixin" and the reader had to
ask which file. Measured with a blind judge playing the engineer who
must apply the fix: of four blocker categories only "under-specified
location" moved between L0 and L3 (0 → 3 across 9 paired tasks). Notably
the *predicted* failure — "omit rationale" → "but why?" — scored zero at
both levels. Making the citation obligatory took that category to 0 and
turn inflation from +23.1% to 0.0%.

Suppressing content is safe. Suppressing the pointer to it is not: the
reader buys it back at the cost of a whole turn, which at agentic
context sizes is far more expensive than the tokens saved.

### 2. The savings table now ships empty

`MODELLED_REDUCTION` previously carried built-in factors. It now ships
`{}` plus a `register_modelled_factors()` seam.

A credible reduction factor is not a constant. It depends on the model
family, the shape of the turn, and the exact steering text — and
producing one means running a paired benchmark across models and paying
for both arms. A constant compiled into the source is a number measured
on somebody else's traffic.

So `estimate_from_model` returns `None` for every level and the
dashboard renders its "not measured" fallback. That is the correct
rendering of *not measured*; a number nobody measured is not. Both real
paths are unchanged and documented at the table: run a holdout (which
outranks this tier entirely), or register factors from a benchmark.

The old built-in figure is a good argument for this: it claimed `(0.336,
0.738)` from five synthetic prompts, measured against a version of the
L3 string that has since changed twice. It over-reported by roughly 2×
at the optimistic end — the direction that matters for a number on a
dashboard.

### 3. Everything else

Thinking tokens split from visible output (`thinking_tokens.py`), the
estimator that was wired but never ran now runs, and the
`OUTCOME_OBSERVED` pipeline stage.

An earlier commit in this range adds a `headroom-output` plugin and a
writable `PRE_SEND_PARAMS` body hook; a later one removes both. Live
measurement showed per-turn effort routing ~15× underwater on mechanical
turns, so the hypothesis it was built on did not survive. The net diff
has neither, and a test now pins the hook as deliberately absent — an
unused writable-body seam is a liability, not an extension point.

## How it was measured

SWE-bench Verified, 12 instances carrying the real source files each
gold patch touches (6k–48k chars), stratified by the dataset's
difficulty label, run at L0 and L3 across six models and paired on
instance. Truncated pairs excluded — a capped response's length is the
ceiling, not the model's choice.

| | pairs | reduction |
|---|---|---|
| Anthropic (opus, sonnet, haiku) | 33 | 18.2% |
| OpenAI (terra, luna, sol) | 28 | 31.2% |

61 usable pairs, median +31.8%, mean +23.0% (SE 5.3, t=4.4), all six
positive in sign, 13% of individual pairs negative.

**On power, since it is easy to over-read the per-model figures:** only
luna (t=6.0), terra (t=3.0) and sonnet (t=2.7) are individually
distinguishable from zero. sol, haiku and opus are not (t≈1.3, 1.3,
0.7). They are single samples per instance against a model whose
*unsteered* output varies by a median 33% run to run — measured
directly, by re-running the L0 arm on identical prompts. The aggregate
effect is solid; a single-model number from this run is not.

## Risk

- Steering is the one lever that mutates the provider prefix-cache key,
so the text is byte-stable and applied once per conversation;
`steering_allowed_for` suppresses it entirely in `mode="cache"`.
- The feature stays opt-in (`available_in=STABLE`, no
`default_enabled_in`). L3 is a visible change to every response and
should be a choice.
- `shaper_enabled_for` keeps byte-faithful forwarding intact: default-on
plus `optimize=False` means don't shape, because creating a `system`
field where the client sent none breaks the contract.

## Testing

612 targeted tests pass; full suite green apart from
`test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`,
which fails identically on `main`.

The final commit fixes a UP037 lint error that arrived with the
savings-rendering work on main. It is unrelated to this branch but fails
CI lint for anything opened on top of it.
2026-09-01 15:28:09 +05:30
3077ac81e8 feat: add deterministic runtime rollout controls (#1490)
## Description

Establish one centrally resolved, observable, deterministic, versioned
runtime rollout-control mechanism for Headroom. Runtime rollout controls
which behaviors an already-built artifact may expose; it does not select
or qualify a Headroom release/version.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Bug fix (non-breaking change that fixes rollout enforcement
regressions)
- [x] Documentation update
- [x] Code refactoring (no functional changes)

## Changes Made

- Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`,
`--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared
by Python configuration boundaries.
- Added schema/policy versions, canonical registry and snapshot SHA-256
identities, per-feature decision reasons, disable precedence, unsafe
qualification poisoning, strict CLI validation, and fail-closed
environment handling.
- Added `headroom rollout status --json`, Python `/stats.rollout`, and
Rust `/rollout/status` runtime provenance.
- Added equivalent Rust snapshot semantics and shared Python/Rust policy
vectors while retaining language-specific feature registries.
- Enforced rollout policy at alternate Python server composition roots
so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate.
- Preserved typed rollout snapshots across multi-worker serialization
with schema, policy, registry, snapshot-digest, type, and feature-name
validation.
- Made loopback runtime output-shaper updates replace the immutable
snapshot atomically for request readers, retain explicit request/disable
provenance, preserve channel and kill-switch precedence, invalidate
cached stats, and return the effective rollout decision.
- Made `headroom learn --verbosity --apply` report a channel-blocked
update instead of claiming the shaper is live.
- Made explicit CLI feature flags fail loudly when their current channel
blocks them.
- Made persistent interceptor installation select canary automatically,
or reject an explicitly insufficient channel unless the break-glass
override is set.
- Updated architecture, proxy, rollout, learn, and output-shaper
documentation with required channels and hot-reload semantics.

## Testing

- [x] Unit tests pass
- [x] Linting passes (`ruff check .` and `ruff format --check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New regression tests added for every corrected behavior
- [x] Rust tests and production-target Clippy pass
- [x] Documentation build passes

### Test Output

```text
Focused rollout coverage suite
57 passed; headroom.rollout + rollout CLI: 98% coverage

Affected proxy/rollout/transform/governance suites
222 passed; 0 failed

Final changed regression suites
100 passed; 0 failed

Cross-module hot-reload isolation regression
6 passed; 0 failed

cargo test -p headroom-core -p headroom-proxy --quiet
headroom-core: 924 passed; 1 ignored
headroom-proxy and integration suites: all passed

cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings
cargo fmt --all -- --check
ruff check .
ruff format --check .
mypy headroom --ignore-missing-imports
git diff --check
All passed

cd docs && npm run build
Compiled successfully; 164 static pages generated
```

The unsharded Windows-only CI selection exposed unrelated baseline
failures, principally the existing `sqlite:///C:\\...` URL parser
producing an invalid `\\C:\\...` path. At commit `8e793a80`, all 52
completed GitHub checks passed; the only other conclusions are expected
skips and superseded governance jobs.

## Real Behavior Proof

- **Environment:** Windows checkout on Python 3.13.3 and the current
Rust workspace, based on upstream `main` at `93f2d7a2`.
- **Exact command / steps:** Exercised canary and beta feature requests
through CLI status, Python `/stats.rollout`, Rust `/rollout/status`,
multi-worker payload round trips, loopback `/admin/runtime-env`, real
proxy request shaping before/after hot reload, installer manifest
generation, and shared Python/Rust policy vectors.
- **Observed result:** Stable blocks unstable requests; disable wins
over explicit/default/legacy/unsafe paths; unsafe state reports
`qualification_eligible=false`; worker handoff rejects tampering;
running output shaping changes only when the effective beta policy
permits it; explicit blocked flags fail with actionable diagnostics.
- **Not tested:** Live production traffic requiring provider
credentials, or future artifact qualification/promotion automation
(intentionally out of scope).

## Runtime Rollout Safety

- **Rollout-managed features:** Python `tool_result_interceptors`,
`proxy_output_shaper`, `read_maturation`; Rust `native_bedrock`,
`openai_responses_streaming`, `canary_probe`.
- **Minimum rollout channel:** Registry-defined per feature; process
default is `stable`.
- **Stable/default behavior changed:** No unstable feature becomes
enabled by default. Explicit blocked CLI flags now fail instead of
silently doing nothing.
- **Kill switch / disable path:**
`HEADROOM_DISABLE_FEATURES=<comma-separated feature names>`; explicit
disable has highest precedence, including over the unsafe override.
- **Unsafe override required:** No.
`HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES=1` is break-glass only and
makes qualification evidence ineligible.
- **Qualification impact:** Adds machine-readable policy/snapshot
identities and eligibility; does not implement qualification itself.
- **Rollback path:** Set the named disable list for operational
rollback, lower the channel, or revert this PR.

## Review Readiness

- [x] I have performed a full diff review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented hard-to-understand areas
- [x] I have made corresponding documentation changes
- [x] My changes generate no new warnings
- [x] I added tests that reproduce and prevent every regression fixed
during review
- [x] New and existing affected tests pass locally
- [x] I did **not** edit `CHANGELOG.md`; release-please generates it
from the Conventional Commit PR title

## Additional Notes

Out of scope: artifact candidates, benchmark orchestration,
qualification manifests/gates, promotion automation, release branches,
publication guards, and release-risk classification. Those workflows can
consume the rollout registry digest, runtime snapshot digest, decision
reasons, and qualification eligibility through supported black-box
interfaces.

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JD Davis <jd@JDH-AIR-00.local>
2026-08-12 23:16:54 -05:00