refactor(proxy): parse HEADROOM_OUTPUT_HOLDOUT in one place (#4082)

## Description

The A/B holdout fraction (`HEADROOM_OUTPUT_HOLDOUT`) was parsed with the
same five lines in five places:

```python
try:
    holdout = float(runtime_env.getenv("HEADROOM_OUTPUT_HOLDOUT", "0") or "0")
except ValueError:
    holdout = 0.0
```

The five places were `gateway_turn.shape_gateway_body`, the Anthropic
chat handler, the OpenAI chat handler, the OpenAI Responses shaper, and
`openai._output_shaping_holdout_fraction`. Every new shaper path copied
the parse again, and any fix to it would have had to land five times.

This PR moves the parse into `output_shaper.output_holdout_fraction()`.
That module already reads the other live shaper knobs
(`HEADROOM_OUTPUT_SHAPER`, `HEADROOM_VERBOSITY_LEVEL`) through
`runtime_env`, and every call site already imports from it. The parse is
byte-for-byte the same, so behaviour is unchanged, including hot-reload
through `runtime_env` overrides.

Evidence: before this PR, `grep -rn HEADROOM_OUTPUT_HOLDOUT headroom |
grep getenv` found the five sites. After it, it finds only
`output_shaper.py`. Nothing in tests or the consumer repos referenced
`_output_shaping_holdout_fraction`.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/output_shaper.py`: add `output_holdout_fraction()`.
- `headroom/proxy/gateway_turn.py`,
`headroom/proxy/handlers/anthropic.py`,
`headroom/proxy/handlers/openai.py` (three sites): call it instead of
the inline parse, and drop the `runtime_env` imports that only served
that parse.
- `headroom/proxy/handlers/openai.py`: delete
`_output_shaping_holdout_fraction`, since its one caller now uses the
shared function.
- Only the holdout lines in the handler files are touched.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/test_proxy_byte_faithful_forwarding.py tests/test_output_shaper_conversation_label.py tests/test_openai_codex_ws_lifecycle.py tests/test_openai_responses_output_shaper.py tests/test_beacon_schema_v2.py tests/test_rollout.py tests/test_output_savings_cli.py tests/test_output_savings_policy.py tests/test_output_shaper.py tests/test_runtime_env.py tests/test_output_shaper_responses.py tests/test_output_savings.py tests/test_stats_output_reduction_gate.py tests/gateway/test_output_steering.py
============================= 440 passed in 35.98s =============================
$ pytest -q -p no:cacheprovider tests/test_cli_learn.py tests/test_proxy_dashboard_stats_cache.py
======================== 56 passed, 1 skipped in 4.81s =========================
$ uvx ruff@0.16.8 check headroom/proxy/gateway_turn.py headroom/proxy/handlers/openai.py headroom/proxy/handlers/anthropic.py headroom/proxy/output_shaper.py
All checks passed!
$ mypy headroom/proxy/output_shaper.py headroom/proxy/gateway_turn.py --ignore-missing-imports
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, branch checkout at a1098e6c, using
the repo .venv. A temp HOME, workspace and CODEX_HOME were used, with
offline mode on.
- Exact command / steps:
1. Ran the shared smoke harness with `HEADROOM_OUTPUT_SHAPER=1
HEADROOM_OUTPUT_HOLDOUT=1`. It starts a real proxy from this checkout
against a local fake upstream and sends one Anthropic and one OpenAI
request carrying a compressible tool result.
2. Called the new function directly, through the env and through a
`runtime_env` hot-reload override.
- Observed result:
```text
branch: cleanup/proxy-core-a-output-holdout @ a1098e6c
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines (<tmp>):
(smoke dir: <tmp>)
```
```text
$ HEADROOM_OUTPUT_HOLDOUT=0.25 python -c "...; print(output_holdout_fraction()); runtime_env.set_overrides({'HEADROOM_OUTPUT_HOLDOUT': '1'}); ...; runtime_env.set_overrides({'HEADROOM_OUTPUT_HOLDOUT': 'abc'}); ..."
env 0.25        -> 0.25
hot override 1  -> 1.0 control
override abc    -> 0.0 treatment
```
- Both providers return 200 through the edited handlers, with no new
warnings.
- The function reads the env, a hot-reload override wins over it, and an
invalid value still means 0.0 (treatment), as with the old inline parse.
- Two pre-existing offline-mode and Kompress-prefetch warnings are left
out of the smoke output.
- Not tested: In this smoke run the proxy log shows no shaper arm label,
so the rollout-gated shaping branch did not run live. The shaping branch
with a holdout set is covered by `tests/gateway/test_output_steering.py`
and the Responses and WS shaper tests listed above.

## Runtime Rollout Safety

- Rollout-managed feature(s): Output shaping (`proxy_output_shaper`
rollout / `HEADROOM_OUTPUT_SHAPER`). The gates are not touched.
- Minimum rollout channel: N/A; no behaviour change.
- Stable/default behavior changed: No. Unset or invalid still means 0.0,
and the value is still read live through `runtime_env`.
- Kill switch / disable path: Unchanged (`HEADROOM_OUTPUT_HOLDOUT=0`, or
the shaper switch).
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of a maintainer-requested codebase cleanup (dead code, duplication,
debuggability), split into small PRs for review. It is part of
maintainer-requested cleanup #4136.

This touches `handlers/anthropic.py` and `handlers/openai.py`, which
other cleanup branches also edit. This branch changes only the holdout
lines, so any conflict should be trivial. There are no new tests,
because existing holdout tests cover the moved parse. No docs change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
This commit is contained in:
Tejas Chopra
2026-10-10 09:04:35 -07:00
committed by GitHub
parent 7387b8cc11
commit 1a4e4194a0
4 changed files with 22 additions and 36 deletions
+2 -6
View File
@@ -1178,7 +1178,6 @@ def shape_gateway_body(
and the block is re-appended deterministically. Never raises.
"""
try:
from headroom.proxy import runtime_env
from headroom.proxy.output_savings import (
assign_arm,
conversation_key_from_body,
@@ -1188,6 +1187,7 @@ def shape_gateway_body(
from headroom.proxy.output_shaper import (
OutputShaperSettings,
classify_turn,
output_holdout_fraction,
resolve_verbosity_level,
shape_openai_chat_request,
shape_request,
@@ -1201,11 +1201,7 @@ def shape_gateway_body(
)
if not settings.enabled:
return False
try:
holdout = float(runtime_env.getenv("HEADROOM_OUTPUT_HOLDOUT", "0") or "0")
except ValueError:
holdout = 0.0
arm = assign_arm(conversation_key_from_body(provider_body), holdout)
arm = assign_arm(conversation_key_from_body(provider_body), output_holdout_fraction())
messages = provider_body.get("messages")
turn_kind = classify_turn(messages if isinstance(messages, list) else []).value
stratum = stratum_key(
+2 -8
View File
@@ -3558,6 +3558,7 @@ class AnthropicHandlerMixin:
from headroom.proxy.output_shaper import (
OutputShaperSettings,
classify_turn,
output_holdout_fraction,
resolve_verbosity_level,
shape_request,
)
@@ -3571,15 +3572,8 @@ class AnthropicHandlerMixin:
# conversation is treatment or control. This keeps the A/B
# comparison clean AND keeps the prefix cache stable (we
# never flip a conversation's system-prompt tail mid-stream).
from headroom.proxy import runtime_env
_holdout = 0.0
try:
_holdout = float(runtime_env.getenv("HEADROOM_OUTPUT_HOLDOUT", "0") or "0")
except ValueError:
_holdout = 0.0
_conversation = conversation_key_from_body(body)
_arm = assign_arm(_conversation, _holdout)
_arm = assign_arm(_conversation, output_holdout_fraction())
# Stratum from request features observable now (mirrors the
# offline baseline so live and learned strata line up).
+6 -22
View File
@@ -819,7 +819,6 @@ def _shape_openai_responses_payload(
must not be able to break request forwarding.
"""
try:
from headroom.proxy import runtime_env
from headroom.proxy.output_savings import (
assign_arm,
conversation_key_from_responses_body,
@@ -830,6 +829,7 @@ def _shape_openai_responses_payload(
from headroom.proxy.output_shaper import (
OutputShaperSettings,
classify_responses_turn,
output_holdout_fraction,
resolve_verbosity_level,
shape_responses_request,
)
@@ -838,10 +838,7 @@ def _shape_openai_responses_payload(
if not settings.enabled:
return [], False
try:
holdout = float(runtime_env.getenv("HEADROOM_OUTPUT_HOLDOUT", "0") or "0")
except ValueError:
holdout = 0.0
holdout = output_holdout_fraction()
conversation = conversation_key_from_responses_body(payload)
arm = assign_arm(conversation, holdout)
@@ -1487,15 +1484,6 @@ def _openai_responses_from_sse(sse_text: str) -> dict[str, Any] | None:
return completed
def _output_shaping_holdout_fraction() -> float:
from headroom.proxy import runtime_env
try:
return float(runtime_env.getenv("HEADROOM_OUTPUT_HOLDOUT", "0") or "0")
except ValueError:
return 0.0
def _shape_openai_responses_for_output(
payload: dict[str, Any],
*,
@@ -1515,6 +1503,7 @@ def _shape_openai_responses_for_output(
OutputShaperSettings,
ShapeResult,
classify_openai_responses_input,
output_holdout_fraction,
resolve_verbosity_level,
shape_openai_responses_request,
)
@@ -1526,7 +1515,7 @@ def _shape_openai_responses_for_output(
assert result.labels is not None
key = conversation_key or conversation_key_from_body(payload)
arm = assign_arm(key, _output_shaping_holdout_fraction())
arm = assign_arm(key, output_holdout_fraction())
turn_kind = classify_openai_responses_input(payload.get("input")).value
stratum = stratum_key(
turn_kind=turn_kind,
@@ -4486,7 +4475,6 @@ class OpenAIHandlerMixin:
# Mutating `body` in place is sufficient here — the outbound request
# serializes `body` fresh, so no body-mutation tracker is needed.
if not _bypass:
from headroom.proxy import runtime_env
from headroom.proxy.output_savings import (
assign_arm,
conversation_key_from_body,
@@ -4497,6 +4485,7 @@ class OpenAIHandlerMixin:
from headroom.proxy.output_shaper import (
OutputShaperSettings,
classify_turn,
output_holdout_fraction,
resolve_verbosity_level,
shape_openai_chat_request,
)
@@ -4510,13 +4499,8 @@ class OpenAIHandlerMixin:
# or control, which keeps the A/B comparison clean and the
# provider prefix cache stable (the steering block never flips
# mid-conversation).
_holdout = 0.0
try:
_holdout = float(runtime_env.getenv("HEADROOM_OUTPUT_HOLDOUT", "0") or "0")
except ValueError:
_holdout = 0.0
_conversation = conversation_key_from_body(body)
_arm = assign_arm(_conversation, _holdout)
_arm = assign_arm(_conversation, output_holdout_fraction())
_turn_kind = classify_turn(body.get("messages", [])).value
_stratum = stratum_key(
turn_kind=_turn_kind,
+12
View File
@@ -135,6 +135,18 @@ class OutputShaperSettings:
)
def output_holdout_fraction() -> float:
"""Fraction of conversations held out of shaping (``HEADROOM_OUTPUT_HOLDOUT``).
Read live, like the other shaper knobs. Unset or unparseable means 0.0 (no
holdout); :func:`~headroom.proxy.output_savings.assign_arm` clamps the range.
"""
try:
return float(runtime_env.getenv("HEADROOM_OUTPUT_HOLDOUT", "0") or "0")
except ValueError:
return 0.0
def shaper_enabled_for(config: Any) -> bool | None:
"""Resolve the output-shaper gate for a proxy config.