mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:07:29 +08:00
## What this does
Verbosity steering already shipped (BETA, L2). This branch measures it
properly, fixes two ways it was misbehaving, and changes what Headroom
is willing to *claim* about it.
Net effect for a user: steering saves more than it did, on more models,
and the dashboard stops showing a savings percentage that was never
measured on their traffic.
## Three changes worth reviewing separately
### 1. The L3 steering text had two defects, both measured
**"Reproduce code, error strings, numbers, units, and API or CLI names
exactly"** read as a licence to emit code. Against the same SWE-bench
turn it left gpt-5.2 unchanged but inflated gpt-5.1 by 58% (1631 → 2573
output tokens), cutting L3's saving on that model from 51% to 23%.
Replaced with a completeness floor that forbids dropping task-critical
content — negations included — without instructing the model to
reproduce anything. The whole OpenAI family went from *negative* to
+31.2%.
**"reference by path and line"** was too soft. Told to stop restating
code, models wrote "change the check in the mixin" and the reader had to
ask which file. Measured with a blind judge playing the engineer who
must apply the fix: of four blocker categories only "under-specified
location" moved between L0 and L3 (0 → 3 across 9 paired tasks). Notably
the *predicted* failure — "omit rationale" → "but why?" — scored zero at
both levels. Making the citation obligatory took that category to 0 and
turn inflation from +23.1% to 0.0%.
Suppressing content is safe. Suppressing the pointer to it is not: the
reader buys it back at the cost of a whole turn, which at agentic
context sizes is far more expensive than the tokens saved.
### 2. The savings table now ships empty
`MODELLED_REDUCTION` previously carried built-in factors. It now ships
`{}` plus a `register_modelled_factors()` seam.
A credible reduction factor is not a constant. It depends on the model
family, the shape of the turn, and the exact steering text — and
producing one means running a paired benchmark across models and paying
for both arms. A constant compiled into the source is a number measured
on somebody else's traffic.
So `estimate_from_model` returns `None` for every level and the
dashboard renders its "not measured" fallback. That is the correct
rendering of *not measured*; a number nobody measured is not. Both real
paths are unchanged and documented at the table: run a holdout (which
outranks this tier entirely), or register factors from a benchmark.
The old built-in figure is a good argument for this: it claimed `(0.336,
0.738)` from five synthetic prompts, measured against a version of the
L3 string that has since changed twice. It over-reported by roughly 2×
at the optimistic end — the direction that matters for a number on a
dashboard.
### 3. Everything else
Thinking tokens split from visible output (`thinking_tokens.py`), the
estimator that was wired but never ran now runs, and the
`OUTCOME_OBSERVED` pipeline stage.
An earlier commit in this range adds a `headroom-output` plugin and a
writable `PRE_SEND_PARAMS` body hook; a later one removes both. Live
measurement showed per-turn effort routing ~15× underwater on mechanical
turns, so the hypothesis it was built on did not survive. The net diff
has neither, and a test now pins the hook as deliberately absent — an
unused writable-body seam is a liability, not an extension point.
## How it was measured
SWE-bench Verified, 12 instances carrying the real source files each
gold patch touches (6k–48k chars), stratified by the dataset's
difficulty label, run at L0 and L3 across six models and paired on
instance. Truncated pairs excluded — a capped response's length is the
ceiling, not the model's choice.
| | pairs | reduction |
|---|---|---|
| Anthropic (opus, sonnet, haiku) | 33 | 18.2% |
| OpenAI (terra, luna, sol) | 28 | 31.2% |
61 usable pairs, median +31.8%, mean +23.0% (SE 5.3, t=4.4), all six
positive in sign, 13% of individual pairs negative.
**On power, since it is easy to over-read the per-model figures:** only
luna (t=6.0), terra (t=3.0) and sonnet (t=2.7) are individually
distinguishable from zero. sol, haiku and opus are not (t≈1.3, 1.3,
0.7). They are single samples per instance against a model whose
*unsteered* output varies by a median 33% run to run — measured
directly, by re-running the L0 arm on identical prompts. The aggregate
effect is solid; a single-model number from this run is not.
## Risk
- Steering is the one lever that mutates the provider prefix-cache key,
so the text is byte-stable and applied once per conversation;
`steering_allowed_for` suppresses it entirely in `mode="cache"`.
- The feature stays opt-in (`available_in=STABLE`, no
`default_enabled_in`). L3 is a visible change to every response and
should be a choice.
- `shaper_enabled_for` keeps byte-faithful forwarding intact: default-on
plus `optimize=False` means don't shape, because creating a `system`
field where the client sent none breaks the contract.
## Testing
612 targeted tests pass; full suite green apart from
`test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`,
which fails identically on `main`.
The final commit fixes a UP037 lint error that arrived with the
savings-rendering work on main. It is unrelated to this branch but fails
CI lint for anything opened on top of it.
137 lines
4.2 KiB
Python
137 lines
4.2 KiB
Python
from __future__ import annotations
|
|
|
|
import importlib.metadata
|
|
from dataclasses import dataclass
|
|
|
|
from headroom.pipeline import (
|
|
CANONICAL_PIPELINE_STAGES,
|
|
ENTRY_POINT_GROUP,
|
|
PipelineEvent,
|
|
PipelineExtensionManager,
|
|
PipelineStage,
|
|
discover_pipeline_extensions,
|
|
summarize_routing_markers,
|
|
)
|
|
|
|
|
|
@dataclass
|
|
class FakeEntryPoint:
|
|
name: str
|
|
value: object
|
|
|
|
def load(self):
|
|
if isinstance(self.value, Exception):
|
|
raise self.value
|
|
return self.value
|
|
|
|
|
|
def test_discover_pipeline_extensions_handles_load_and_init_failures(
|
|
monkeypatch,
|
|
) -> None:
|
|
class WorkingExtension:
|
|
def on_pipeline_event(self, event: PipelineEvent): # noqa: ANN001, ANN201
|
|
return event
|
|
|
|
class NeedsInit:
|
|
def __init__(self) -> None:
|
|
raise RuntimeError("bad init")
|
|
|
|
monkeypatch.setattr(
|
|
importlib.metadata,
|
|
"entry_points",
|
|
lambda group=None: (
|
|
[
|
|
FakeEntryPoint("working-instance", WorkingExtension()),
|
|
FakeEntryPoint("working-class", WorkingExtension),
|
|
FakeEntryPoint("bad-load", RuntimeError("bad load")),
|
|
FakeEntryPoint("bad-init", NeedsInit),
|
|
]
|
|
if group == ENTRY_POINT_GROUP
|
|
else []
|
|
),
|
|
)
|
|
|
|
# Discovery is opt-in since the contract gained ``body`` (it can now
|
|
# rewrite max_tokens/effort on live traffic), so name the extensions
|
|
# explicitly. What this test covers is unchanged: a bad load and a bad
|
|
# __init__ are isolated, and the healthy two still come through.
|
|
discovered = discover_pipeline_extensions(["*"])
|
|
assert len(discovered) == 2
|
|
assert all(callable(getattr(ext, "on_pipeline_event", None)) for ext in discovered)
|
|
|
|
|
|
def test_discover_pipeline_extensions_handles_enumeration_failure(monkeypatch) -> None:
|
|
monkeypatch.setattr(
|
|
importlib.metadata,
|
|
"entry_points",
|
|
lambda group=None: (_ for _ in ()).throw(RuntimeError("boom")),
|
|
)
|
|
assert discover_pipeline_extensions() == []
|
|
|
|
|
|
def test_pipeline_manager_emit_and_summary(monkeypatch) -> None:
|
|
class Hook:
|
|
def __init__(self) -> None:
|
|
self.seen: list[str] = []
|
|
|
|
def on_pipeline_event(self, event: PipelineEvent): # noqa: ANN001, ANN201
|
|
self.seen.append(event.stage.value)
|
|
event.metadata["hook"] = True
|
|
return event
|
|
|
|
class ReplacingExtension:
|
|
def on_pipeline_event(self, event: PipelineEvent): # noqa: ANN001, ANN201
|
|
return PipelineEvent(
|
|
stage=event.stage,
|
|
operation=event.operation,
|
|
request_id=event.request_id,
|
|
provider=event.provider,
|
|
model=event.model,
|
|
messages=event.messages,
|
|
tools=event.tools,
|
|
headers=event.headers,
|
|
response=event.response,
|
|
metadata={**event.metadata, "replaced": True},
|
|
)
|
|
|
|
class BrokenExtension:
|
|
def on_pipeline_event(self, event: PipelineEvent): # noqa: ANN001, ANN201
|
|
raise RuntimeError("boom")
|
|
|
|
hook = Hook()
|
|
monkeypatch.setattr(
|
|
"headroom.pipeline.discover_pipeline_extensions",
|
|
lambda: [BrokenExtension()],
|
|
)
|
|
|
|
manager = PipelineExtensionManager(
|
|
hooks=hook,
|
|
extensions=[object(), ReplacingExtension()],
|
|
discover=True,
|
|
)
|
|
|
|
assert manager.enabled is True
|
|
event = manager.emit(
|
|
PipelineStage.INPUT_RECEIVED,
|
|
operation="compress",
|
|
request_id="req-1",
|
|
provider="openai",
|
|
model="gpt-4o",
|
|
messages=[{"role": "user", "content": "hello"}],
|
|
metadata={"start": True},
|
|
)
|
|
|
|
assert hook.seen == ["input_received"]
|
|
assert event.metadata == {"start": True, "hook": True, "replaced": True}
|
|
assert event.request_id == "req-1"
|
|
|
|
disabled = PipelineExtensionManager(discover=False)
|
|
assert disabled.enabled is False
|
|
|
|
assert summarize_routing_markers(["router:smart", "other", "router:cheap"]) == [
|
|
"router:smart",
|
|
"router:cheap",
|
|
]
|
|
assert PipelineStage.SETUP in CANONICAL_PIPELINE_STAGES
|
|
assert PipelineStage.RESPONSE_RECEIVED in CANONICAL_PIPELINE_STAGES
|