Commit Graph
2 Commits
Author SHA1 Message Date
danieldutuandJerrettDavis 3a38d0b93b docs: drop effort-routing text that #3377 removed (#4449)
## Description

<!-- Briefly explain the change and why it is needed. -->

`python scripts/eval_output_shaper.py B` raised `AssertionError: []` at
`scripts/eval_output_shaper.py:201`. Scenario B asserted
`body["output_config"]["effort"] == "low"` after `shape_request`, and
`shape_request` returned no labels.

Root cause: #3377 (`1390d897`) deleted `output_effort_policy.py` and the
`effort_router` config, so the shaper no longer changes
`output_config.effort`, `reasoning.effort` or `thinking.budget_tokens`.
The text around it was not updated. 17 lines mentioning effort routing
remained in 9 files: `README.md:186-192`,
`headroom/proxy/output_shaper.py:12-22,103-104,196`,
`headroom/proxy/thinking_tokens.py:4`,
`headroom/proxy/outcome.py:127,253`,
`headroom/proxy/handlers/anthropic.py:124,3651`,
`headroom/proxy/handlers/openai.py:834`,
`headroom/agent_savings.py:176`, `headroom/proxy/server.py:6511` and the
eval script. @HetCreep reported the stale text in #4421 and listed most
of these locations. I started from that list and found
`agent_savings.py:176`, `server.py:6511` and `anthropic.py:124` by grep.

Fix: I removed the effort-routing bullet and the `reasoning_effort` /
`output_config.effort` paragraph from the README, and docstring item 2
plus the `output_config.effort` safety rule from `output_shaper.py`. I
deleted two cache-mode sentences that said effort routing stays on in
cache mode, and reworded the other comments. I dropped Scenario B from
the eval script (66 lines).

Boundary: no runtime code changed and `CHANGELOG.md` is untouched. I did
not re-implement effort routing; #3377 removed it on purpose and the
issue asks for docs only. I deleted Scenario B instead of rewriting it
to assert that effort stays `xhigh`, because
`tests/test_output_shaper.py:185` already asserts that. Question for
maintainers: do you want that assertion kept in the script as well? I
will split the script change into its own PR if you prefer.

Closes #4421

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `README.md`: removed the effort-routing bullet and the
`reasoning_effort` / `output_config.effort` paragraph.
- `headroom/proxy/output_shaper.py`: removed docstring item 2, the
`output_config.effort` safety rule and two cache-mode comments.
- `headroom/proxy/thinking_tokens.py`, `outcome.py`,
`handlers/anthropic.py`, `handlers/openai.py`, `agent_savings.py`,
`server.py`: comment and docstring wording only.
- `scripts/eval_output_shaper.py`: removed Scenario B.

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run
- [ ] New tests added for new functionality — none added, no behavior
change
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest -q tests/test_output_shaper.py tests/test_thinking_tokens.py tests/test_outcome_token_scale.py tests/test_output_shaper_responses.py tests/test_openai_responses_output_shaper.py
117 passed, 1 warning in 21.78s
$ uv run ruff check .
All checks passed!
$ uv run ruff format --check .
1812 files already formatted
$ ANTHROPIC_API_KEY=dummy uv run python scripts/eval_output_shaper.py B
exit=0
```

The pytest run covers the five test files that touch the output shaper.
I did not run the full suite. The one warning is a Starlette `httpx`
deprecation from `fastapi/testclient.py` in site-packages.

## Real Behavior Proof

- Environment: macOS 27.0.1, Python 3.12.13, `uv sync --extra dev
--extra proxy`, `ANTHROPIC_API_KEY=dummy` (no network call)
- Exact command / steps: before the patch (`e995eb9f`)
`ANTHROPIC_API_KEY=dummy python scripts/eval_output_shaper.py B` in a
venv with `anthropic`; after the patch `ANTHROPIC_API_KEY=dummy uv run
python scripts/eval_output_shaper.py B`; after the patch `grep -rniE
"effort.rout|effort router|effort-router" README.md headroom scripts`
- Observed result: before, `AssertionError: []` at
`scripts/eval_output_shaper.py:201`, exit 1; after, exit 0 with the
header `model=claude-opus-4-8 trials=2` and an empty table, because `B`
now matches no scenario and the script makes no API call; the grep
returned 0 lines after the patch and 17 lines at `e995eb9f`
- Not tested: a live Anthropic run of Scenario A (no API key), the full
pytest suite, `mypy headroom`, Windows and Linux

## Runtime Rollout Safety

- Rollout-managed feature(s): none, no runtime code changed
- Minimum rollout channel: n/a, no runtime change
- Stable/default behavior changed: no, comments, docstrings, README and
one eval script only
- Kill switch / disable path: n/a, no behavior added
- Unsafe override required: no
- Qualification impact: none, docs and comments only
- Rollback path: `git revert` of the single commit

## Review Readiness

- [x] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
— no code added
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works — none added, no behavior change
- [ ] New and existing unit tests pass locally with my changes — 5 test
files ran, full suite not run
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

None. No UI change.

## Additional Notes

The README now states that the shaper applies verbosity steering to
`/v1/messages`, `/v1/chat/completions` and `/v1/responses` with the same
`output_shaper:*` labels. I kept that sentence from the removed
paragraph because it still holds.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:44:57 -07:00
Tejas Chopra a99dc61424 feat: output-token reduction — verbosity shaper, per-user learning, counterfactual savings (#965)
## Description

Adds the first levers that reduce the tokens the model **writes back**
(output), complementing Headroom's existing input compression. Output
costs 5× input on Opus-class models and is full of waste (ceremony,
restated code, deep "thinking" on routine steps). Two phases in one
self-contained PR off `main`: the request-side output shaper, then
per-user verbosity learning plus an honest counterfactual savings
estimator and dashboard surfacing.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Output shaper** (`output_shaper.py`, opt-in
`HEADROOM_OUTPUT_SHAPER=1`): cache-safe verbosity steering appended to
the system-prompt tail (5 levels); effort routing that lowers
`output_config.effort` on mechanical tool-result continuations; legacy
`thinking.budget_tokens` clamp. Never injects effort where absent, never
toggles `thinking.type`.
- **`headroom learn --verbosity`**: mines Claude Code transcripts for
behavioral signals (interrupts, length-adaptive fast-skips, echo ratio),
recommends a verbosity level (heuristic + optional `--llm-judge`), and
seeds the savings baseline.
- **Counterfactual estimator** (`output_savings.py`): per-stratum
synthetic-control (estimated) + A/B holdout (measured) with a propagated
95% CI; conversation-stable arm assignment for A/B validity and
prefix-cache safety.
- **AIMD verbosity controller** (`verbosity_controller.py`):
additive-increase / fast-back-off state machine; live signal emission
gated off by default.
- **Wiring + surfaces**: shaper resolves the learned level; recording
rides the existing `transforms_applied` channel through the outcome
funnel (no `RequestOutcome` changes); `headroom output-savings` CLI;
dashboard "Output Tokens Saved" card.
- **Docs**: simple-words user guide + design doc with the counterfactual
methodology.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_output_savings.py tests/test_output_savings_cli.py \
        tests/test_verbosity_learn.py tests/test_verbosity_controller.py \
        tests/test_output_shaper.py -q
94 passed in 0.54s

$ pytest tests/test_request_outcome.py tests/test_handler_outcome_tag_invariant.py \
        tests/test_proxy_dashboard_stats_cache.py -q
44 passed

$ ruff format --check .
831 files already formatted

$ mypy headroom --ignore-missing-imports
Success: no issues found in 361 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12 (`.venv`), `anthropic` 0.76, live API
model `claude-opus-4-8`.
- Exact command / steps: `HEADROOM_OUTPUT_SHAPER=1`; `headroom learn
--verbosity --apply` (seeds level + baseline); `python
scripts/eval_output_shaper.py A` (live before/after); simulate holdout
traffic then `headroom output-savings`.
- Observed result: code-review ask — baseline 1,750 output tokens → L2
1,354 (−22.7%) → L3 599 (−65.8%), same bugs found. `learn --verbosity`
on 24 real sessions → 11% interrupt / 26% fast-skip → L3 (high
confidence). Measured A/B path → 31.7% reduction (95% CI 27.7%–35.7%).
94 new tests + 44 existing outcome/dashboard tests green; ruff + mypy
clean.
- Not tested: live streaming-path recording exercised only via unit
tests (the `transforms_applied` funnel is shared across paths); runtime
AIMD signal emission is gated off by default and not exercised live.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

Output savings are counterfactual (we never observe what the model
*would* have written), so the estimator separates **estimated** (vs a
learned baseline) from **measured** (A/B holdout via
`HEADROOM_OUTPUT_HOLDOUT`) and always reports a confidence band — never
a single made-up number. CHANGELOG left unchecked (release-please
manages it). Runtime AIMD self-tuning is intentionally a TODO
(controller built/tested; live signal emission gated behind
`HEADROOM_VERBOSITY_AUTOTUNE`).
2026-06-16 21:06:43 -07:00