fix(output-savings): seed the holdout key on the whole first user message (#3209)

## Description

`conversation_key_from_body` seeds the output-shaping holdout
assignment. It hashed the first 512 characters of the **first** text
block of the first user message.

Agent clients open every conversation with injected context - project
instructions, IDE state, memory digests - that is byte-identical across
conversations in a project and routinely kilobytes long. The user's own
words, the only part that distinguishes one conversation from the next,
sat past the 512-character cut, so whole populations of conversations
shared a single key.

Assignment is deterministic, so one key means one arm permanently. On a
real 78k-request ledger, a holdout nominally set to 3% had produced:

| family | treatment | control |
|---|---|---|
| fable | 22,222 | 1 |
| sonnet | 15,401 | 0 |
| haiku | 523 | 0 |
| opus | 38,891 | 3,532 |

Those clients were not being sampled at 3%. They were frozen into one
arm, so their output-shaping savings could never be measured against a
control no matter how long the holdout ran - and the aggregate measured
estimate, had it ever passed its coverage gate, would have compared one
frozen population against another rather than a randomized contrast.

No tracking issue; this was found from production ledger data.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `conversation_key_from_body` now seeds on every text block of the
first user message, joined and in full, instead of a 512-character
prefix of the first block.
- Added `_MAX_SEED_CHARS = 65_536` as a cost guard only, documented as
such and set far above any client prologue so it cannot recreate the bug
it replaces.
- Non-text blocks (images) still never seed the key.
- Added two regression tests: two conversations differing only past a
long identical prologue, and an image block leaving the key unchanged.

Stability is preserved, which the design depends on: user turns are
never compressed (`protected_user_message` in `compression_units`), so
the first user message is immutable for the life of a conversation, and
identical openers still map to one key and one arm.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/ -q -k "output_sav or shaper or holdout or telemetry_session"
158 passed, 8 skipped, 11927 deselected, 1 warning in 50.48s

$ uvx ruff check .
All checks passed!

$ uvx ruff format --check headroom/proxy/output_savings_policy.py tests/test_output_savings_policy.py
2 files already formatted

$ uv run --frozen --extra dev mypy headroom/proxy/output_savings_policy.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 15.6, Python 3.11, headroom at 91186b40 (0.36.4),
replayed against 170 real Claude Code conversation openers taken from 7
days of local transcripts.
- Exact command / steps: loaded `output_savings_policy.py` from this
branch alongside an inline copy of the previous implementation, built a
body per opener as `[{"type": "text", "text": prologue}, {"type":
"text", "text": opener}]` with a 2,113-character injected prologue, then
counted distinct keys and `assign_arm(key, 0.03)` outcomes under both
versions.
- Observed result: before, 170 conversations collapsed to 3 distinct
keys (one per model family) and 0 of 170 landed in control; after, the
same 170 produced 120 distinct keys with 4 in control (2.4%), and both
arms were reachable for every model family. The 120-rather-than-170
figure is repeated openers, which correctly share a key. The
before-figures reproduce the production ledger in the description, where
whole families sit at 0 or 1 control samples.
- Not tested: no live traffic was routed through a proxy built from this
branch; the OpenAI Responses path is unchanged and still keys off
`_stable_response_identifier`; single-turn templated side-calls whose
entire first message is a fixed template remain one key by construction,
which this change does not address.

## Runtime Rollout Safety

- Rollout-managed feature(s): output shaper holdout arm assignment,
reached only when `HEADROOM_OUTPUT_SHAPER` is enabled and
`HEADROOM_OUTPUT_HOLDOUT` is above zero.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes. Conversations hash to different
keys than before, so a given conversation may be assigned the opposite
arm from the one it had under the previous version.
- Kill switch / disable path: set `HEADROOM_OUTPUT_HOLDOUT=0`, which
routes every conversation to treatment and bypasses the key entirely;
`HEADROOM_OUTPUT_SHAPER=0` disables the surrounding feature.
- Unsafe override required: No.
- Qualification impact: ledger accumulators recorded before this change
stay in whatever arm the old key assigned, so a stratum can hold pre-
and post-change samples from different populations; expect measured
estimates to move as the newly reachable arms fill in.
- Rollback path: revert this commit. The key reverts with it, no
persisted format or stored data changes, and existing ledger entries
remain readable either way.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable; no user-visible surface changes.

## Additional Notes

Documentation checkbox is unchecked because no docs describe the holdout
key; the rationale lives in the function's docstring, which this PR
expands.

Two related gaps are deliberately out of scope. Single-turn templated
side-calls, where the entire first message is a fixed template, still
collapse to one key and stay unmeasurable. And a verbosity baseline
seeded once at install cannot be relearned from shaped traffic, so the
control arm this fix unblocks is the only route to a fresh comparator.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
gglucass
2026-09-30 11:20:40 -05:00
committed by GitHub
co-authored by Claude Fable 5 JD Davis
parent bd0296b5e7
commit 117ff72ec9
2 changed files with 259 additions and 21 deletions
+84 -21
View File
@@ -3,6 +3,7 @@
from __future__ import annotations
import hashlib
from collections.abc import Iterable
from typing import Any, cast
# Coarse input-token buckets. Coarse on purpose: too many strata make
@@ -68,6 +69,33 @@ def stratum_key(
)
def _absorb(digest: Any, text: str) -> None:
"""Fold one seed field into ``digest``.
Incremental so no length cap is needed: the seed is consumed a field at a
time instead of being concatenated into one string first. The NUL is the
field separator, and feeding it separately is byte-identical to hashing
``"\x00" + text``.
"""
digest.update(b"\x00")
digest.update(text.encode("utf-8", "ignore"))
def _absorb_text_blocks(digest: Any, blocks: Iterable[str]) -> None:
"""Fold every text block of one message into ``digest`` as one field.
Byte-identical to absorbing ``"\x00".join(blocks)``, including the empty
case: a message with no text blocks still contributes its separator.
"""
absorbed = False
for text in blocks:
digest.update(b"\x00")
digest.update(text.encode("utf-8", "ignore"))
absorbed = True
if not absorbed:
digest.update(b"\x00")
def _unwrap_response_create_body(body: dict[str, Any]) -> dict[str, Any]:
response = body.get("response")
if body.get("type") == "response.create" and isinstance(response, dict):
@@ -113,52 +141,87 @@ def _stable_response_identifier(body: dict[str, Any]) -> str:
def conversation_key_from_body(body: dict[str, Any]) -> str:
"""Derive a conversation-stable key for holdout assignment."""
"""Derive a conversation-stable key for holdout assignment.
Every text block of the first user message feeds the seed, in full: the
digest is built incrementally, field by field, so there is no length cap to
collapse behind.
Seeding on the first 512 characters of the *first* block collapsed whole
populations of conversations onto one key, because agent clients open a
conversation with injected context -- CLAUDE.md, IDE selection, memory
digests -- that is byte-identical for every conversation in a project and
far longer than 512 characters. The user's own words, the part that makes
one conversation different from the next, sat past the cut.
One key means one arm, permanently, since the assignment is deterministic:
on a real 78k-request ledger that left fable at 22,222 treatment requests
against 1 control and sonnet at 15,401 against 0, while opus accumulated
3,532 control samples -- a holdout nominally set to 3% that had in fact
frozen each client into a single arm. Reading the whole message restores
the intended unit of randomization without weakening stability: user turns
are never compressed, so the first message is immutable for the life of
the conversation.
"""
body = _unwrap_response_create_body(body)
model = str(body.get("model", ""))
seed = model
digest = hashlib.sha256()
digest.update(str(body.get("model", "")).encode("utf-8", "ignore"))
for msg in body.get("messages", []):
if isinstance(msg, dict) and msg.get("role") == "user":
content = msg.get("content")
if isinstance(content, str):
seed += "\x00" + content[:512]
_absorb(digest, content)
elif isinstance(content, list):
for block in content:
if isinstance(block, dict) and block.get("type") == "text":
seed += "\x00" + str(block.get("text", ""))[:512]
break
_absorb_text_blocks(
digest,
(
str(block.get("text", ""))
for block in content
if isinstance(block, dict) and block.get("type") == "text"
),
)
break
if "input" in body:
stable_response_key = _stable_response_identifier(body)
if stable_response_key:
seed += "\x00" + stable_response_key
_absorb(digest, stable_response_key)
elif not body.get("messages"):
seed += "\x00responses"
return hashlib.sha256(seed.encode("utf-8", "ignore")).hexdigest()
_absorb(digest, "responses")
return digest.hexdigest()
def conversation_key_from_responses_body(body: dict[str, Any]) -> str:
"""Conversation-stable key for an OpenAI Responses payload."""
"""Conversation-stable key for an OpenAI Responses payload.
Same seeding rule as :func:`conversation_key_from_body`, and for the same
reason: this path carried the identical 512-character cut, on the first
text part alone, so a Codex-style client whose opener begins with injected
context keyed every conversation the same way and never left one arm.
"""
body = _unwrap_response_create_body(body)
model = str(body.get("model", ""))
seed = model
digest = hashlib.sha256()
digest.update(str(body.get("model", "")).encode("utf-8", "ignore"))
input_data = body.get("input")
if isinstance(input_data, str):
seed += "\x00" + input_data[:512]
_absorb(digest, input_data)
elif isinstance(input_data, list):
for item in input_data:
if not isinstance(item, dict) or item.get("role") != "user":
continue
content = item.get("content")
if isinstance(content, str):
seed += "\x00" + content[:512]
_absorb(digest, content)
elif isinstance(content, list):
for part in content:
if isinstance(part, dict) and isinstance(part.get("text"), str):
seed += "\x00" + part["text"][:512]
break
_absorb_text_blocks(
digest,
(
part["text"]
for part in content
if isinstance(part, dict) and isinstance(part.get("text"), str)
),
)
break
return hashlib.sha256(seed.encode("utf-8", "ignore")).hexdigest()
return digest.hexdigest()
def assign_arm(conversation_key: str, holdout_fraction: float) -> str:
+175
View File
@@ -5,6 +5,7 @@ from __future__ import annotations
from headroom.proxy.output_savings_policy import (
assign_arm,
conversation_key_from_body,
conversation_key_from_responses_body,
input_bucket,
model_family,
parse_stratum_label,
@@ -52,9 +53,183 @@ def test_conversation_key_uses_response_create_payload() -> None:
assert conversation_key_from_body(http_body) == conversation_key_from_body(ws_body)
def test_conversation_key_survives_a_long_identical_client_prologue() -> None:
"""Two conversations differing only past a long injected prologue.
Agent clients open every conversation with the same context blocks
(project instructions, IDE state, memory), commonly kilobytes long. Keying
on a 512-character prefix gave all of them one key, so a whole client sat
in one arm permanently and never produced holdout data.
"""
prologue = "<system-reminder>project instructions. </system-reminder>" * 40
assert len(prologue) > 512
def body(question: str) -> dict[str, object]:
return {
"model": "claude-opus-4-5",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": prologue},
{"type": "text", "text": question},
],
}
],
}
assert conversation_key_from_body(body("add a cache")) != conversation_key_from_body(
body("delete the cache")
)
# Same words, same conversation: still one key, so the arm stays stable.
assert conversation_key_from_body(body("add a cache")) == conversation_key_from_body(
body("add a cache")
)
def test_conversation_key_separates_conversations_behind_a_64kb_prologue() -> None:
"""No cutoff can hide the user's words, however long the prologue.
A character cap is a cutoff wherever it sits: put a prologue past it and
every conversation behind that prologue keys the same way again, which is
the collapse this key exists to avoid. The seed is hashed incrementally, so
there is no cap -- this prologue is deliberately larger than the 64KB guard
an earlier revision of this fix used.
"""
prologue = "x" * 96_000
def body(question: str) -> dict[str, object]:
return {
"model": "claude-opus-4-5",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": prologue},
{"type": "text", "text": question},
],
}
],
}
assert conversation_key_from_body(body("add a cache")) != conversation_key_from_body(
body("delete the cache")
)
# And the same opener is still one key, so the arm holds for the conversation.
assert conversation_key_from_body(body("add a cache")) == conversation_key_from_body(
body("add a cache")
)
def test_responses_conversation_key_separates_past_a_long_prologue() -> None:
"""The Responses path carried the same cut and needs the same seeding."""
prologue = "x" * 96_000
def body(question: str) -> dict[str, object]:
return {
"model": "gpt-5",
"input": [
{
"role": "user",
"content": [
{"type": "input_text", "text": prologue},
{"type": "input_text", "text": question},
],
}
],
}
assert conversation_key_from_responses_body(
body("add a cache")
) != conversation_key_from_responses_body(body("delete the cache"))
assert conversation_key_from_responses_body(
body("add a cache")
) == conversation_key_from_responses_body(body("add a cache"))
def test_conversation_key_ignores_non_text_blocks() -> None:
"""An image block carries no conversation identity and must not seed."""
text_only = {
"model": "claude-opus-4-5",
"messages": [{"role": "user", "content": [{"type": "text", "text": "ship it"}]}],
}
with_image = {
"model": "claude-opus-4-5",
"messages": [
{
"role": "user",
"content": [
{"type": "image", "source": {"data": "iVBORw0KGgo="}},
{"type": "text", "text": "ship it"},
],
}
],
}
assert conversation_key_from_body(text_only) == conversation_key_from_body(with_image)
def test_responses_conversation_key_reads_past_non_user_items() -> None:
"""Seeding starts at the first user item, whatever precedes it."""
def body(question: str) -> dict[str, object]:
return {
"model": "gpt-5",
"input": [
{"role": "system", "content": "you are a helpful assistant"},
{"role": "user", "content": question},
],
}
assert conversation_key_from_responses_body(
body("add a cache")
) != conversation_key_from_responses_body(body("delete the cache"))
def test_conversation_key_handles_a_message_with_no_text() -> None:
"""An image-only opener still keys, and keys apart from an empty one."""
image_only = {
"model": "claude-opus-4-5",
"messages": [{"role": "user", "content": [{"type": "image", "source": {"data": "iVBOR"}}]}],
}
empty = {"model": "claude-opus-4-5", "messages": [{"role": "user", "content": []}]}
no_message = {"model": "claude-opus-4-5", "messages": []}
# No text either way, so these two agree; a message-less body does not.
assert conversation_key_from_body(image_only) == conversation_key_from_body(empty)
assert conversation_key_from_body(image_only) != conversation_key_from_body(no_message)
def test_stratum_label_round_trips_arm_and_key() -> None:
key = "opus|code|m|tools"
assert parse_stratum_label(stratum_label("treatment", key)) == ("treatment", key)
assert parse_stratum_label(stratum_label("control", key)) == ("control", key)
assert parse_stratum_label("unrelated") is None
def test_responses_conversation_key_seeds_on_a_plain_string_input() -> None:
"""/v1/responses accepts `input` as a bare string, not only an item list.
That shape skips the item loop entirely, so it needs its own seeding path
or every string-input conversation hashes to the same key and lands in one
arm together.
"""
def body(question: str) -> dict:
return {"model": "gpt-5.2-codex", "input": question}
assert conversation_key_from_responses_body(
body("add a cache")
) != conversation_key_from_responses_body(body("delete the cache"))
assert conversation_key_from_responses_body(
body("add a cache")
) == conversation_key_from_responses_body(body("add a cache"))
# And it agrees with the equivalent item-list form. The seed is the opening
# user text, not the wire shape carrying it, so a client that switches
# between the two mid-session stays in the arm it was assigned.
assert conversation_key_from_responses_body(body("add a cache")) == (
conversation_key_from_responses_body(
{"model": "gpt-5.2-codex", "input": [{"role": "user", "content": "add a cache"}]}
)
)