mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat/https-compose-caddy
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
3a38d0b93b |
docs: drop effort-routing text that #3377 removed (#4449)
## Description <!-- Briefly explain the change and why it is needed. --> `python scripts/eval_output_shaper.py B` raised `AssertionError: []` at `scripts/eval_output_shaper.py:201`. Scenario B asserted `body["output_config"]["effort"] == "low"` after `shape_request`, and `shape_request` returned no labels. Root cause: #3377 (`1390d897`) deleted `output_effort_policy.py` and the `effort_router` config, so the shaper no longer changes `output_config.effort`, `reasoning.effort` or `thinking.budget_tokens`. The text around it was not updated. 17 lines mentioning effort routing remained in 9 files: `README.md:186-192`, `headroom/proxy/output_shaper.py:12-22,103-104,196`, `headroom/proxy/thinking_tokens.py:4`, `headroom/proxy/outcome.py:127,253`, `headroom/proxy/handlers/anthropic.py:124,3651`, `headroom/proxy/handlers/openai.py:834`, `headroom/agent_savings.py:176`, `headroom/proxy/server.py:6511` and the eval script. @HetCreep reported the stale text in #4421 and listed most of these locations. I started from that list and found `agent_savings.py:176`, `server.py:6511` and `anthropic.py:124` by grep. Fix: I removed the effort-routing bullet and the `reasoning_effort` / `output_config.effort` paragraph from the README, and docstring item 2 plus the `output_config.effort` safety rule from `output_shaper.py`. I deleted two cache-mode sentences that said effort routing stays on in cache mode, and reworded the other comments. I dropped Scenario B from the eval script (66 lines). Boundary: no runtime code changed and `CHANGELOG.md` is untouched. I did not re-implement effort routing; #3377 removed it on purpose and the issue asks for docs only. I deleted Scenario B instead of rewriting it to assert that effort stays `xhigh`, because `tests/test_output_shaper.py:185` already asserts that. Question for maintainers: do you want that assertion kept in the script as well? I will split the script change into its own PR if you prefer. Closes #4421 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `README.md`: removed the effort-routing bullet and the `reasoning_effort` / `output_config.effort` paragraph. - `headroom/proxy/output_shaper.py`: removed docstring item 2, the `output_config.effort` safety rule and two cache-mode comments. - `headroom/proxy/thinking_tokens.py`, `outcome.py`, `handlers/anthropic.py`, `handlers/openai.py`, `agent_savings.py`, `server.py`: comment and docstring wording only. - `scripts/eval_output_shaper.py`: removed Scenario B. ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) — not run - [ ] New tests added for new functionality — none added, no behavior change - [x] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_output_shaper.py tests/test_thinking_tokens.py tests/test_outcome_token_scale.py tests/test_output_shaper_responses.py tests/test_openai_responses_output_shaper.py 117 passed, 1 warning in 21.78s $ uv run ruff check . All checks passed! $ uv run ruff format --check . 1812 files already formatted $ ANTHROPIC_API_KEY=dummy uv run python scripts/eval_output_shaper.py B exit=0 ``` The pytest run covers the five test files that touch the output shaper. I did not run the full suite. The one warning is a Starlette `httpx` deprecation from `fastapi/testclient.py` in site-packages. ## Real Behavior Proof - Environment: macOS 27.0.1, Python 3.12.13, `uv sync --extra dev --extra proxy`, `ANTHROPIC_API_KEY=dummy` (no network call) - Exact command / steps: before the patch (`e995eb9f`) `ANTHROPIC_API_KEY=dummy python scripts/eval_output_shaper.py B` in a venv with `anthropic`; after the patch `ANTHROPIC_API_KEY=dummy uv run python scripts/eval_output_shaper.py B`; after the patch `grep -rniE "effort.rout|effort router|effort-router" README.md headroom scripts` - Observed result: before, `AssertionError: []` at `scripts/eval_output_shaper.py:201`, exit 1; after, exit 0 with the header `model=claude-opus-4-8 trials=2` and an empty table, because `B` now matches no scenario and the script makes no API call; the grep returned 0 lines after the patch and 17 lines at `e995eb9f` - Not tested: a live Anthropic run of Scenario A (no API key), the full pytest suite, `mypy headroom`, Windows and Linux ## Runtime Rollout Safety - Rollout-managed feature(s): none, no runtime code changed - Minimum rollout channel: n/a, no runtime change - Stable/default behavior changed: no, comments, docstrings, README and one eval script only - Kill switch / disable path: n/a, no behavior added - Unsafe override required: no - Qualification impact: none, docs and comments only - Rollback path: `git revert` of the single commit ## Review Readiness - [x] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas — no code added - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works — none added, no behavior change - [ ] New and existing unit tests pass locally with my changes — 5 test files ran, full suite not run - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) None. No UI change. ## Additional Notes The README now states that the shaper applies verbosity steering to `/v1/messages`, `/v1/chat/completions` and `/v1/responses` with the same `output_shaper:*` labels. I kept that sentence from the removed paragraph because it still holds. Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
a99dc61424 |
feat: output-token reduction — verbosity shaper, per-user learning, counterfactual savings (#965)
## Description
Adds the first levers that reduce the tokens the model **writes back**
(output), complementing Headroom's existing input compression. Output
costs 5× input on Opus-class models and is full of waste (ceremony,
restated code, deep "thinking" on routine steps). Two phases in one
self-contained PR off `main`: the request-side output shaper, then
per-user verbosity learning plus an honest counterfactual savings
estimator and dashboard surfacing.
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **Output shaper** (`output_shaper.py`, opt-in
`HEADROOM_OUTPUT_SHAPER=1`): cache-safe verbosity steering appended to
the system-prompt tail (5 levels); effort routing that lowers
`output_config.effort` on mechanical tool-result continuations; legacy
`thinking.budget_tokens` clamp. Never injects effort where absent, never
toggles `thinking.type`.
- **`headroom learn --verbosity`**: mines Claude Code transcripts for
behavioral signals (interrupts, length-adaptive fast-skips, echo ratio),
recommends a verbosity level (heuristic + optional `--llm-judge`), and
seeds the savings baseline.
- **Counterfactual estimator** (`output_savings.py`): per-stratum
synthetic-control (estimated) + A/B holdout (measured) with a propagated
95% CI; conversation-stable arm assignment for A/B validity and
prefix-cache safety.
- **AIMD verbosity controller** (`verbosity_controller.py`):
additive-increase / fast-back-off state machine; live signal emission
gated off by default.
- **Wiring + surfaces**: shaper resolves the learned level; recording
rides the existing `transforms_applied` channel through the outcome
funnel (no `RequestOutcome` changes); `headroom output-savings` CLI;
dashboard "Output Tokens Saved" card.
- **Docs**: simple-words user guide + design doc with the counterfactual
methodology.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_output_savings.py tests/test_output_savings_cli.py \
tests/test_verbosity_learn.py tests/test_verbosity_controller.py \
tests/test_output_shaper.py -q
94 passed in 0.54s
$ pytest tests/test_request_outcome.py tests/test_handler_outcome_tag_invariant.py \
tests/test_proxy_dashboard_stats_cache.py -q
44 passed
$ ruff format --check .
831 files already formatted
$ mypy headroom --ignore-missing-imports
Success: no issues found in 361 source files
```
## Real Behavior Proof
- Environment: macOS, Python 3.12 (`.venv`), `anthropic` 0.76, live API
model `claude-opus-4-8`.
- Exact command / steps: `HEADROOM_OUTPUT_SHAPER=1`; `headroom learn
--verbosity --apply` (seeds level + baseline); `python
scripts/eval_output_shaper.py A` (live before/after); simulate holdout
traffic then `headroom output-savings`.
- Observed result: code-review ask — baseline 1,750 output tokens → L2
1,354 (−22.7%) → L3 599 (−65.8%), same bugs found. `learn --verbosity`
on 24 real sessions → 11% interrupt / 26% fast-skip → L3 (high
confidence). Measured A/B path → 31.7% reduction (95% CI 27.7%–35.7%).
94 new tests + 44 existing outcome/dashboard tests green; ruff + mypy
clean.
- Not tested: live streaming-path recording exercised only via unit
tests (the `transforms_applied` funnel is shared across paths); runtime
AIMD signal emission is gated off by default and not exercised live.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
Output savings are counterfactual (we never observe what the model
*would* have written), so the estimator separates **estimated** (vs a
learned baseline) from **measured** (A/B holdout via
`HEADROOM_OUTPUT_HOLDOUT`) and always reports a confidence band — never
a single made-up number. CHANGELOG left unchecked (release-please
manages it). Runtime AIMD self-tuning is intentionally a TODO
(controller built/tested; live signal emission gated behind
`HEADROOM_VERBOSITY_AUTOTUNE`).
|