Files
headroom/scripts/eval_output_shaper.py
danieldutuandJerrettDavis 3a38d0b93b docs: drop effort-routing text that #3377 removed (#4449)
## Description

<!-- Briefly explain the change and why it is needed. -->

`python scripts/eval_output_shaper.py B` raised `AssertionError: []` at
`scripts/eval_output_shaper.py:201`. Scenario B asserted
`body["output_config"]["effort"] == "low"` after `shape_request`, and
`shape_request` returned no labels.

Root cause: #3377 (`1390d897`) deleted `output_effort_policy.py` and the
`effort_router` config, so the shaper no longer changes
`output_config.effort`, `reasoning.effort` or `thinking.budget_tokens`.
The text around it was not updated. 17 lines mentioning effort routing
remained in 9 files: `README.md:186-192`,
`headroom/proxy/output_shaper.py:12-22,103-104,196`,
`headroom/proxy/thinking_tokens.py:4`,
`headroom/proxy/outcome.py:127,253`,
`headroom/proxy/handlers/anthropic.py:124,3651`,
`headroom/proxy/handlers/openai.py:834`,
`headroom/agent_savings.py:176`, `headroom/proxy/server.py:6511` and the
eval script. @HetCreep reported the stale text in #4421 and listed most
of these locations. I started from that list and found
`agent_savings.py:176`, `server.py:6511` and `anthropic.py:124` by grep.

Fix: I removed the effort-routing bullet and the `reasoning_effort` /
`output_config.effort` paragraph from the README, and docstring item 2
plus the `output_config.effort` safety rule from `output_shaper.py`. I
deleted two cache-mode sentences that said effort routing stays on in
cache mode, and reworded the other comments. I dropped Scenario B from
the eval script (66 lines).

Boundary: no runtime code changed and `CHANGELOG.md` is untouched. I did
not re-implement effort routing; #3377 removed it on purpose and the
issue asks for docs only. I deleted Scenario B instead of rewriting it
to assert that effort stays `xhigh`, because
`tests/test_output_shaper.py:185` already asserts that. Question for
maintainers: do you want that assertion kept in the script as well? I
will split the script change into its own PR if you prefer.

Closes #4421

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `README.md`: removed the effort-routing bullet and the
`reasoning_effort` / `output_config.effort` paragraph.
- `headroom/proxy/output_shaper.py`: removed docstring item 2, the
`output_config.effort` safety rule and two cache-mode comments.
- `headroom/proxy/thinking_tokens.py`, `outcome.py`,
`handlers/anthropic.py`, `handlers/openai.py`, `agent_savings.py`,
`server.py`: comment and docstring wording only.
- `scripts/eval_output_shaper.py`: removed Scenario B.

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run
- [ ] New tests added for new functionality — none added, no behavior
change
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest -q tests/test_output_shaper.py tests/test_thinking_tokens.py tests/test_outcome_token_scale.py tests/test_output_shaper_responses.py tests/test_openai_responses_output_shaper.py
117 passed, 1 warning in 21.78s
$ uv run ruff check .
All checks passed!
$ uv run ruff format --check .
1812 files already formatted
$ ANTHROPIC_API_KEY=dummy uv run python scripts/eval_output_shaper.py B
exit=0
```

The pytest run covers the five test files that touch the output shaper.
I did not run the full suite. The one warning is a Starlette `httpx`
deprecation from `fastapi/testclient.py` in site-packages.

## Real Behavior Proof

- Environment: macOS 27.0.1, Python 3.12.13, `uv sync --extra dev
--extra proxy`, `ANTHROPIC_API_KEY=dummy` (no network call)
- Exact command / steps: before the patch (`e995eb9f`)
`ANTHROPIC_API_KEY=dummy python scripts/eval_output_shaper.py B` in a
venv with `anthropic`; after the patch `ANTHROPIC_API_KEY=dummy uv run
python scripts/eval_output_shaper.py B`; after the patch `grep -rniE
"effort.rout|effort router|effort-router" README.md headroom scripts`
- Observed result: before, `AssertionError: []` at
`scripts/eval_output_shaper.py:201`, exit 1; after, exit 0 with the
header `model=claude-opus-4-8 trials=2` and an empty table, because `B`
now matches no scenario and the script makes no API call; the grep
returned 0 lines after the patch and 17 lines at `e995eb9f`
- Not tested: a live Anthropic run of Scenario A (no API key), the full
pytest suite, `mypy headroom`, Windows and Linux

## Runtime Rollout Safety

- Rollout-managed feature(s): none, no runtime code changed
- Minimum rollout channel: n/a, no runtime change
- Stable/default behavior changed: no, comments, docstrings, README and
one eval script only
- Kill switch / disable path: n/a, no behavior added
- Unsafe override required: no
- Qualification impact: none, docs and comments only
- Rollback path: `git revert` of the single commit

## Review Readiness

- [x] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
— no code added
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works — none added, no behavior change
- [ ] New and existing unit tests pass locally with my changes — 5 test
files ran, full suite not run
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

None. No UI change.

## Additional Notes

The README now states that the shaper applies verbosity steering to
`/v1/messages`, `/v1/chat/completions` and `/v1/responses` with the same
`output_shaper:*` labels. I kept that sentence from the removed
paragraph because it still holds.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:44:57 -07:00

170 lines
5.6 KiB
Python

"""Live before/after eval for the output shaper.
Sends the SAME request to the Anthropic API twice — once as a client would
send it (baseline) and once after `shape_request` rewrites it (exactly what
the proxy forwards upstream) — and compares `usage.output_tokens`, which
includes thinking tokens.
Scenario A (verbosity steering): a complex code-review ask. Baseline vs
verbosity levels 2 and 3.
Usage:
source .venv/bin/activate && python scripts/eval_output_shaper.py
Requires ANTHROPIC_API_KEY in the environment or in ./.env.
"""
from __future__ import annotations
import copy
import os
import statistics
import sys
from pathlib import Path
from typing import Any
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
import anthropic # noqa: E402
from headroom.proxy.output_shaper import OutputShaperSettings, shape_request # noqa: E402
MODEL = "claude-opus-4-8"
TRIALS = 2
BUGGY_CODE = '''\
import threading
from collections import OrderedDict
class TTLCache:
"""LRU cache with per-entry TTL."""
def __init__(self, max_size=128, ttl=300):
self.max_size = max_size
self.ttl = ttl
self._store = OrderedDict()
self._lock = threading.Lock()
def get(self, key, now):
entry = self._store.get(key)
if entry is None:
return None
value, expires_at = entry
if now > expires_at:
del self._store[key]
return None
self._store.move_to_end(key)
return value
def put(self, key, value, now):
with self._lock:
if key in self._store:
self._store.move_to_end(key)
self._store[key] = (value, now + self.ttl)
if len(self._store) > self.max_size:
self._store.popitem(last=True)
def cleanup(self, now):
for key, (_, expires_at) in self._store.items():
if now > expires_at:
del self._store[key]
'''
def load_env() -> None:
env_path = Path(__file__).resolve().parent.parent / ".env"
if not env_path.exists() or os.environ.get("ANTHROPIC_API_KEY"):
return
for line in env_path.read_text().splitlines():
line = line.strip()
if line and not line.startswith("#") and "=" in line:
key, _, value = line.partition("=")
value = value.strip().strip("'\"")
os.environ.setdefault(key.strip(), value)
def scenario_a_body() -> dict[str, Any]:
"""Complex single-turn ask — exercises verbosity steering."""
return {
"model": MODEL,
"max_tokens": 8000,
"system": "You are a senior Python engineer doing code review.",
"messages": [
{
"role": "user",
"content": (
"Review this cache implementation. Identify every bug and "
"thread-safety issue, then show how to fix each one:\n\n"
f"```python\n{BUGGY_CODE}```"
),
}
],
}
def run(client: anthropic.Anthropic, body: dict[str, Any]) -> dict[str, int]:
# The installed SDK may predate output_config as a typed kwarg; the API
# accepts it either way, so pass it through extra_body.
body = dict(body)
extra_body = None
if "output_config" in body:
extra_body = {"output_config": body.pop("output_config")}
response = client.messages.create(**body, extra_body=extra_body)
if response.stop_reason == "refusal":
raise RuntimeError("request was refused by safety classifiers")
return {
"input_tokens": response.usage.input_tokens,
"output_tokens": response.usage.output_tokens,
}
def main() -> int:
load_env()
if not os.environ.get("ANTHROPIC_API_KEY"):
print("ANTHROPIC_API_KEY not found (env or .env)", file=sys.stderr)
return 1
client = anthropic.Anthropic()
which = sys.argv[1].upper() if len(sys.argv) > 1 else "ALL"
conditions: list[tuple[str, str, dict[str, Any]]] = []
if which in ("A", "ALL"):
# Scenario A: baseline vs steered.
conditions.append(("A:verbosity", "baseline", scenario_a_body()))
for level in (2, 3):
body = scenario_a_body()
shape_request(body, OutputShaperSettings(enabled=True, verbosity_level=level))
conditions.append(("A:verbosity", f"shaped L{level}", body))
print(f"model={MODEL} trials={TRIALS}\n")
print(f"{'scenario':<18} {'condition':<16} {'trial':<6} {'in_tok':>7} {'out_tok':>8}")
print("-" * 60)
results: dict[tuple[str, str], list[int]] = {}
for scenario, condition, body in conditions:
for trial in range(1, TRIALS + 1):
usage = run(client, copy.deepcopy(body))
results.setdefault((scenario, condition), []).append(usage["output_tokens"])
print(
f"{scenario:<18} {condition:<16} {trial:<6} "
f"{usage['input_tokens']:>7} {usage['output_tokens']:>8}"
)
print("\n=== Summary (mean output tokens, reduction vs baseline) ===")
baselines: dict[str, float] = {}
for (scenario, condition), outs in results.items():
if condition.startswith("baseline"):
baselines[scenario] = statistics.mean(outs)
for (scenario, condition), outs in results.items():
mean = statistics.mean(outs)
base = baselines.get(scenario, 0)
if condition.startswith("baseline") or not base:
print(f"{scenario:<18} {condition:<16} {mean:>8.0f} (baseline)")
else:
pct = (base - mean) / base * 100
print(f"{scenario:<18} {condition:<16} {mean:>8.0f} ({pct:+.1f}% vs baseline)")
return 0
if __name__ == "__main__":
sys.exit(main())