mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description <!-- Briefly explain the change and why it is needed. --> `python scripts/eval_output_shaper.py B` raised `AssertionError: []` at `scripts/eval_output_shaper.py:201`. Scenario B asserted `body["output_config"]["effort"] == "low"` after `shape_request`, and `shape_request` returned no labels. Root cause: #3377 (`1390d897`) deleted `output_effort_policy.py` and the `effort_router` config, so the shaper no longer changes `output_config.effort`, `reasoning.effort` or `thinking.budget_tokens`. The text around it was not updated. 17 lines mentioning effort routing remained in 9 files: `README.md:186-192`, `headroom/proxy/output_shaper.py:12-22,103-104,196`, `headroom/proxy/thinking_tokens.py:4`, `headroom/proxy/outcome.py:127,253`, `headroom/proxy/handlers/anthropic.py:124,3651`, `headroom/proxy/handlers/openai.py:834`, `headroom/agent_savings.py:176`, `headroom/proxy/server.py:6511` and the eval script. @HetCreep reported the stale text in #4421 and listed most of these locations. I started from that list and found `agent_savings.py:176`, `server.py:6511` and `anthropic.py:124` by grep. Fix: I removed the effort-routing bullet and the `reasoning_effort` / `output_config.effort` paragraph from the README, and docstring item 2 plus the `output_config.effort` safety rule from `output_shaper.py`. I deleted two cache-mode sentences that said effort routing stays on in cache mode, and reworded the other comments. I dropped Scenario B from the eval script (66 lines). Boundary: no runtime code changed and `CHANGELOG.md` is untouched. I did not re-implement effort routing; #3377 removed it on purpose and the issue asks for docs only. I deleted Scenario B instead of rewriting it to assert that effort stays `xhigh`, because `tests/test_output_shaper.py:185` already asserts that. Question for maintainers: do you want that assertion kept in the script as well? I will split the script change into its own PR if you prefer. Closes #4421 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `README.md`: removed the effort-routing bullet and the `reasoning_effort` / `output_config.effort` paragraph. - `headroom/proxy/output_shaper.py`: removed docstring item 2, the `output_config.effort` safety rule and two cache-mode comments. - `headroom/proxy/thinking_tokens.py`, `outcome.py`, `handlers/anthropic.py`, `handlers/openai.py`, `agent_savings.py`, `server.py`: comment and docstring wording only. - `scripts/eval_output_shaper.py`: removed Scenario B. ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) — not run - [ ] New tests added for new functionality — none added, no behavior change - [x] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_output_shaper.py tests/test_thinking_tokens.py tests/test_outcome_token_scale.py tests/test_output_shaper_responses.py tests/test_openai_responses_output_shaper.py 117 passed, 1 warning in 21.78s $ uv run ruff check . All checks passed! $ uv run ruff format --check . 1812 files already formatted $ ANTHROPIC_API_KEY=dummy uv run python scripts/eval_output_shaper.py B exit=0 ``` The pytest run covers the five test files that touch the output shaper. I did not run the full suite. The one warning is a Starlette `httpx` deprecation from `fastapi/testclient.py` in site-packages. ## Real Behavior Proof - Environment: macOS 27.0.1, Python 3.12.13, `uv sync --extra dev --extra proxy`, `ANTHROPIC_API_KEY=dummy` (no network call) - Exact command / steps: before the patch (`e995eb9f`) `ANTHROPIC_API_KEY=dummy python scripts/eval_output_shaper.py B` in a venv with `anthropic`; after the patch `ANTHROPIC_API_KEY=dummy uv run python scripts/eval_output_shaper.py B`; after the patch `grep -rniE "effort.rout|effort router|effort-router" README.md headroom scripts` - Observed result: before, `AssertionError: []` at `scripts/eval_output_shaper.py:201`, exit 1; after, exit 0 with the header `model=claude-opus-4-8 trials=2` and an empty table, because `B` now matches no scenario and the script makes no API call; the grep returned 0 lines after the patch and 17 lines at `e995eb9f` - Not tested: a live Anthropic run of Scenario A (no API key), the full pytest suite, `mypy headroom`, Windows and Linux ## Runtime Rollout Safety - Rollout-managed feature(s): none, no runtime code changed - Minimum rollout channel: n/a, no runtime change - Stable/default behavior changed: no, comments, docstrings, README and one eval script only - Kill switch / disable path: n/a, no behavior added - Unsafe override required: no - Qualification impact: none, docs and comments only - Rollback path: `git revert` of the single commit ## Review Readiness - [x] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas — no code added - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works — none added, no behavior change - [ ] New and existing unit tests pass locally with my changes — 5 test files ran, full suite not run - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) None. No UI change. ## Additional Notes The README now states that the shaper applies verbosity steering to `/v1/messages`, `/v1/chat/completions` and `/v1/responses` with the same `output_shaper:*` labels. I kept that sentence from the removed paragraph because it still holds. Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
170 lines
5.6 KiB
Python
170 lines
5.6 KiB
Python
"""Live before/after eval for the output shaper.
|
|
|
|
Sends the SAME request to the Anthropic API twice — once as a client would
|
|
send it (baseline) and once after `shape_request` rewrites it (exactly what
|
|
the proxy forwards upstream) — and compares `usage.output_tokens`, which
|
|
includes thinking tokens.
|
|
|
|
Scenario A (verbosity steering): a complex code-review ask. Baseline vs
|
|
verbosity levels 2 and 3.
|
|
|
|
Usage:
|
|
source .venv/bin/activate && python scripts/eval_output_shaper.py
|
|
Requires ANTHROPIC_API_KEY in the environment or in ./.env.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import copy
|
|
import os
|
|
import statistics
|
|
import sys
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
|
|
|
import anthropic # noqa: E402
|
|
|
|
from headroom.proxy.output_shaper import OutputShaperSettings, shape_request # noqa: E402
|
|
|
|
MODEL = "claude-opus-4-8"
|
|
TRIALS = 2
|
|
|
|
BUGGY_CODE = '''\
|
|
import threading
|
|
from collections import OrderedDict
|
|
|
|
class TTLCache:
|
|
"""LRU cache with per-entry TTL."""
|
|
|
|
def __init__(self, max_size=128, ttl=300):
|
|
self.max_size = max_size
|
|
self.ttl = ttl
|
|
self._store = OrderedDict()
|
|
self._lock = threading.Lock()
|
|
|
|
def get(self, key, now):
|
|
entry = self._store.get(key)
|
|
if entry is None:
|
|
return None
|
|
value, expires_at = entry
|
|
if now > expires_at:
|
|
del self._store[key]
|
|
return None
|
|
self._store.move_to_end(key)
|
|
return value
|
|
|
|
def put(self, key, value, now):
|
|
with self._lock:
|
|
if key in self._store:
|
|
self._store.move_to_end(key)
|
|
self._store[key] = (value, now + self.ttl)
|
|
if len(self._store) > self.max_size:
|
|
self._store.popitem(last=True)
|
|
|
|
def cleanup(self, now):
|
|
for key, (_, expires_at) in self._store.items():
|
|
if now > expires_at:
|
|
del self._store[key]
|
|
'''
|
|
|
|
|
|
def load_env() -> None:
|
|
env_path = Path(__file__).resolve().parent.parent / ".env"
|
|
if not env_path.exists() or os.environ.get("ANTHROPIC_API_KEY"):
|
|
return
|
|
for line in env_path.read_text().splitlines():
|
|
line = line.strip()
|
|
if line and not line.startswith("#") and "=" in line:
|
|
key, _, value = line.partition("=")
|
|
value = value.strip().strip("'\"")
|
|
os.environ.setdefault(key.strip(), value)
|
|
|
|
|
|
def scenario_a_body() -> dict[str, Any]:
|
|
"""Complex single-turn ask — exercises verbosity steering."""
|
|
return {
|
|
"model": MODEL,
|
|
"max_tokens": 8000,
|
|
"system": "You are a senior Python engineer doing code review.",
|
|
"messages": [
|
|
{
|
|
"role": "user",
|
|
"content": (
|
|
"Review this cache implementation. Identify every bug and "
|
|
"thread-safety issue, then show how to fix each one:\n\n"
|
|
f"```python\n{BUGGY_CODE}```"
|
|
),
|
|
}
|
|
],
|
|
}
|
|
|
|
|
|
def run(client: anthropic.Anthropic, body: dict[str, Any]) -> dict[str, int]:
|
|
# The installed SDK may predate output_config as a typed kwarg; the API
|
|
# accepts it either way, so pass it through extra_body.
|
|
body = dict(body)
|
|
extra_body = None
|
|
if "output_config" in body:
|
|
extra_body = {"output_config": body.pop("output_config")}
|
|
response = client.messages.create(**body, extra_body=extra_body)
|
|
if response.stop_reason == "refusal":
|
|
raise RuntimeError("request was refused by safety classifiers")
|
|
return {
|
|
"input_tokens": response.usage.input_tokens,
|
|
"output_tokens": response.usage.output_tokens,
|
|
}
|
|
|
|
|
|
def main() -> int:
|
|
load_env()
|
|
if not os.environ.get("ANTHROPIC_API_KEY"):
|
|
print("ANTHROPIC_API_KEY not found (env or .env)", file=sys.stderr)
|
|
return 1
|
|
client = anthropic.Anthropic()
|
|
which = sys.argv[1].upper() if len(sys.argv) > 1 else "ALL"
|
|
|
|
conditions: list[tuple[str, str, dict[str, Any]]] = []
|
|
|
|
if which in ("A", "ALL"):
|
|
# Scenario A: baseline vs steered.
|
|
conditions.append(("A:verbosity", "baseline", scenario_a_body()))
|
|
for level in (2, 3):
|
|
body = scenario_a_body()
|
|
shape_request(body, OutputShaperSettings(enabled=True, verbosity_level=level))
|
|
conditions.append(("A:verbosity", f"shaped L{level}", body))
|
|
|
|
print(f"model={MODEL} trials={TRIALS}\n")
|
|
print(f"{'scenario':<18} {'condition':<16} {'trial':<6} {'in_tok':>7} {'out_tok':>8}")
|
|
print("-" * 60)
|
|
|
|
results: dict[tuple[str, str], list[int]] = {}
|
|
for scenario, condition, body in conditions:
|
|
for trial in range(1, TRIALS + 1):
|
|
usage = run(client, copy.deepcopy(body))
|
|
results.setdefault((scenario, condition), []).append(usage["output_tokens"])
|
|
print(
|
|
f"{scenario:<18} {condition:<16} {trial:<6} "
|
|
f"{usage['input_tokens']:>7} {usage['output_tokens']:>8}"
|
|
)
|
|
|
|
print("\n=== Summary (mean output tokens, reduction vs baseline) ===")
|
|
baselines: dict[str, float] = {}
|
|
for (scenario, condition), outs in results.items():
|
|
if condition.startswith("baseline"):
|
|
baselines[scenario] = statistics.mean(outs)
|
|
for (scenario, condition), outs in results.items():
|
|
mean = statistics.mean(outs)
|
|
base = baselines.get(scenario, 0)
|
|
if condition.startswith("baseline") or not base:
|
|
print(f"{scenario:<18} {condition:<16} {mean:>8.0f} (baseline)")
|
|
else:
|
|
pct = (base - mean) / base * 100
|
|
print(f"{scenario:<18} {condition:<16} {mean:>8.0f} ({pct:+.1f}% vs baseline)")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|