mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(transforms): judge a Codex exec envelope read by its output (#3878)
## Description
Codex code mode runs `exec` as JavaScript. When the script prints the
whole `exec_command` result, `text(r)` instead of `text(r.output)`, a
file read reaches the Responses read-protection gate as the JSON
envelope:
```json
{"chunk_id":"0819ff","wall_time_seconds":3e-06,"exit_code":0,"original_token_count":1540,"output":" 1\tuse std::...\n 2\t..."}
```
The read command is detected (#3621 / #3737), but
`_read_output_should_be_protected` then runs the content detector on the
envelope, i.e. on one line of JSON-escaped code. That verdict flips per
file between `SOURCE_CODE` (protected) and a releasable JSON type. When
it comes back JSON the read is released, the whole file is replaced by a
single `<<ccr:...>>` marker, and the model re-runs the command (in the
session this was found in, a 400-line `nl -ba models.rs` was re-read
twice with line filters: 4 requests instead of 2).
Fix: in `_read_output_should_be_protected`, unwrap a Codex exec envelope
(bare, a JSON list of them, or behind Codex's `Script completed / Wall
time / Output:` preamble, which `_responses_part_text` joins in) and
return the gate's own verdict on the `output` string(s), which is
exactly the verdict `text(r.output)` already gets. Only a body that is
nothing but envelope(s) is unwrapped; anything else is judged as before.
JSON / log / diff reads inside an envelope stay releasable, and non-read
commands never reach this gate. Both callers (the Responses handler and
`ContentRouter.apply`) go through the same function.
**Compression behaviour change, stated plainly:** this lowers
single-request `tokens_saved` on Codex `text(r)` reads of
code/markdown/config, because those reads are now kept verbatim (the
`protect_reads` policy the coding profile already applies to
`text(r.output)` and to plain `exec_command` reads). The trade is fewer
re-read turns, which a single-request savings metric does not see.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/content_router.py`: new
`_codex_exec_envelope_outputs()`; `_read_output_should_be_protected`
judges the envelope's output string(s) when the text is one.
- `tests/test_openai_responses_read_protection.py`:
- `test_read_gate_judges_codex_exec_envelope_by_its_output`: bare,
listed and preamble-prefixed envelopes get the verdict of their output,
for a code read (protected) and a JSON read (releasable).
- `test_responses_codex_exec_whole_result_read_stays_verbatim`: end to
end through `_compress_openai_responses_live_text_units_with_router`
with a lossy router, a `text(r)` code read stays byte-identical.
Both new tests fail on `main` ffc6edb4 and pass with this change.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_openai_responses_read_protection.py -x -q
27 passed in 1.21s
$ uv run --frozen --extra dev pytest tests/test_observed_wire_shapes.py tests/test_transforms_tabular.py tests/test_openai_responses_compression_units.py -x -q
128 passed in 1.93s
# the two new tests with the headroom/ change stashed (main ffc6edb4):
FAILED tests/test_openai_responses_read_protection.py::test_read_gate_judges_codex_exec_envelope_by_its_output
FAILED tests/test_openai_responses_read_protection.py::test_responses_codex_exec_whole_result_read_stays_verbatim
2 failed, 25 passed
$ uvx ruff@0.16.8 check headroom/transforms/content_router.py tests/test_openai_responses_read_protection.py
All checks passed!
$ uvx ruff@0.16.8 format --check headroom/transforms/content_router.py tests/test_openai_responses_read_protection.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/transforms/content_router.py --ignore-missing-imports
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS 15, Python 3.11 worktree venv (`--extra dev --extra
proxy --extra code`, native content detection) on upstream main
ffc6edb4, with and without this commit. A real `HeadroomProxy` built
from `_proxy_config_from_env()` after the coding profile's
`apply_proxy_env_defaults`, so `HEADROOM_PROTECT_READS` is on as in
`headroom proxy`. Kompress model not loaded in the process (cold); the
change is in the protect/release verdict, which does not depend on
Kompress.
- Exact command / steps: 26 Codex `exec` items built from commands run
on a pristine `git archive ffc6edb4` tree (13 commands x {`text(r)`
envelope, `text(r.output)` plain}: `nl -ba` / `sed -n` / `head` reads of
Rust, Python, Markdown, YAML, TOML and JSON files in this repo, plus
`rg`, `git log --stat`, `git diff` controls), each sent as `[user
message, custom_tool_call, custom_tool_call_output]` through
`_compress_openai_responses_live_text_units_with_router`. Every
compressor warmed first on salted copies; 2 rounds per arm; `tokens`
below is o200k of the tool output.
- Observed result: fully deterministic across rounds. Exactly 7 of 26
items change, all `text(r)` reads of code/markdown/config that main
released and replaced with a CCR marker:
| item (`text(r)`) | tokens | main saved | this PR saved |
|---|---:|---:|---:|
| `nl -ba crates/headroom-proxy/src/proxy.rs \| sed -n 1,400p` | 6,540 |
6,475 (marker) | 0 (verbatim) |
| `nl -ba .../responses_items.rs \| sed -n 1,300p` | 4,563 | 4,498
(marker) | 0 |
| `nl -ba headroom/mcp_registry/codex.py` | 4,366 | 4,301 (marker) | 0 |
| `sed -n 1,300p headroom/proxy/server.py` | 2,568 | 2,506 (marker) | 0
|
| `sed -n 1,300p README.md` | 5,054 | 4,991 (marker) | 0 |
| `head -n 250 .github/workflows/ci.yml` | 2,625 | 2,564 (marker) | 0 |
| `head -n 200 pyproject.toml` | 3,071 | 3,007 (marker) | 0 |
Unchanged: every `text(r.output)` item, two `text(r)` code reads main
already happened to protect (`config.rs`, `content_router.py`, the
flip-per-file behaviour), the JSON read and the `git diff` envelope
(still releasable), and the non-read `rg` / `git log` controls. Corpus
totals: 88,869 tokens, saved 40,154 on main vs 11,812 with this change
(-28,342, all from the 7 reads above). An earlier replay of the
originating session's 31-item corpus on the 0.39.0 wheel showed the same
shape: only the `text(r)` code reads the detector called JSON changed.
- Not tested: a live Codex session end to end against the OpenAI API
(the re-read count is from the originating session's rollout, not
re-measured); Codex's classic `exec_command` function-call shape, which
carries plain text rather than a JSON envelope and is unaffected.
## Runtime Rollout Safety
- Rollout-managed feature(s): None. Applies wherever read protection is
already on (`HEADROOM_PROTECT_READS`, on in the coding profile).
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes. A detected read whose output is
a Codex exec JSON envelope is now protected or released by its `output`
text instead of by the envelope; for code/markdown/config reads that
means kept verbatim where some were released. Everything that is not a
whole-envelope body is judged byte-identically to main.
- Kill switch / disable path: `HEADROOM_PROTECT_READS=0` turns read
protection off entirely, as today. No separate switch.
- Unsafe override required: None.
- Qualification impact: Lower single-request `tokens_saved` on Codex
`text(r)` reads by design (-28,342 of 40,154 on the corpus above),
traded for fewer re-read turns. Worth watching the Codex savings rate
and re-read rate across the release that carries it.
- Rollback path: Revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- Documentation: N/A, internal gate.
- Deliberately left out: a script that prints other text as well as the
envelope (`text("x"); text(r)`) keeps today's verdict on the whole body.
Unwrapping only when the body is nothing but envelope(s) keeps the
change narrow.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
ccd9fffce9
commit
922924ef44
@@ -1157,6 +1157,41 @@ _RELEASABLE_READ_TYPES = frozenset(
|
||||
)
|
||||
|
||||
|
||||
def _codex_exec_envelope_outputs(text: str) -> list[str] | None:
|
||||
"""The file text inside a Codex ``exec`` result printed whole, else ``None``.
|
||||
|
||||
Codex code mode runs ``exec`` as JavaScript. A script that prints the whole
|
||||
``exec_command`` result, ``text(r)`` rather than ``text(r.output)``, sends
|
||||
the JSON envelope ``{"chunk_id", "wall_time_seconds", "exit_code", ...,
|
||||
"output": "<file>"}`` (or a list of them), possibly behind Codex's
|
||||
``Script completed / Wall time / Output:`` preamble. Judging the envelope
|
||||
would judge a line of JSON-escaped code, whose verdict flips per file
|
||||
between source and releasable JSON; the read is its ``output`` string(s).
|
||||
Only a body that is nothing but envelope(s) is unwrapped.
|
||||
"""
|
||||
if '"wall_time_seconds"' not in text:
|
||||
return None
|
||||
body = text.strip()
|
||||
if not body.startswith(("{", "[")):
|
||||
_, found, body = text.partition("\nOutput:\n")
|
||||
body = body.strip()
|
||||
if not found or not body.startswith(("{", "[")):
|
||||
return None
|
||||
try:
|
||||
parsed = json.loads(body)
|
||||
except (ValueError, RecursionError):
|
||||
return None
|
||||
envelopes = parsed if isinstance(parsed, list) else [parsed]
|
||||
outputs = [
|
||||
env["output"]
|
||||
for env in envelopes
|
||||
if isinstance(env, dict)
|
||||
and "wall_time_seconds" in env
|
||||
and isinstance(env.get("output"), str)
|
||||
]
|
||||
return outputs if outputs and len(outputs) == len(envelopes) else None
|
||||
|
||||
|
||||
def _read_output_should_be_protected(text: Any) -> bool:
|
||||
"""Finalize read-protection by CONTENT — protect by default, release only DATA.
|
||||
|
||||
@@ -1172,6 +1207,9 @@ def _read_output_should_be_protected(text: Any) -> bool:
|
||||
"""
|
||||
if not isinstance(text, str) or not text:
|
||||
return False
|
||||
outputs = _codex_exec_envelope_outputs(text)
|
||||
if outputs is not None:
|
||||
return any(_read_output_should_be_protected(output) for output in outputs)
|
||||
try:
|
||||
detection = _detect_content(text)
|
||||
except Exception:
|
||||
|
||||
@@ -17,6 +17,7 @@ exactly the bytes the model needs for line-precise edits, forcing re-reads
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from types import MethodType, SimpleNamespace
|
||||
|
||||
from headroom.proxy.handlers.openai import OpenAIHandlerMixin
|
||||
@@ -718,3 +719,61 @@ def test_responses_codex_exec_test_output_still_compresses(monkeypatch):
|
||||
|
||||
assert modified is True
|
||||
assert new_payload["input"][1] != _codex_exec_output("call_pytest", _NL_OUTPUT)
|
||||
|
||||
|
||||
# 40 line-numbered lines of Rust: the detector calls it code on its own, but a
|
||||
# JSON envelope around it reads as a releasable JSON document.
|
||||
_RUST_NL = "".join(
|
||||
f"{i:6d}\tpub fn f{i}(x: Option<Vec<u8>>) -> usize {{ x.map_or(0, |v| v.len()) }}\n"
|
||||
for i in range(1, 41)
|
||||
)
|
||||
|
||||
|
||||
def _exec_envelope(output: str) -> str:
|
||||
"""What Codex prints for `text(r)`: the whole exec_command result."""
|
||||
return json.dumps(
|
||||
{
|
||||
"chunk_id": "6c7260",
|
||||
"wall_time_seconds": 0.0,
|
||||
"exit_code": 0,
|
||||
"original_token_count": 900,
|
||||
"output": output,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def test_read_gate_judges_codex_exec_envelope_by_its_output():
|
||||
"""Bare, listed or behind the `Output:` preamble: the verdict is the output's."""
|
||||
from headroom.transforms.content_router import _read_output_should_be_protected as protect
|
||||
|
||||
data = json.dumps(
|
||||
{
|
||||
"name": "app",
|
||||
"dependencies": [{"name": f"pkg{i}", "version": "1.0.0"} for i in range(30)],
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
assert protect(_RUST_NL) is True
|
||||
assert protect(data) is False
|
||||
for output in (_RUST_NL, data):
|
||||
envelope = _exec_envelope(output)
|
||||
expected = protect(output)
|
||||
assert protect(envelope) is expected
|
||||
assert protect(json.dumps([json.loads(envelope)] * 2)) is expected
|
||||
assert (
|
||||
protect("Script completed\nWall time 0.1 seconds\nOutput:\n\n" + envelope) is expected
|
||||
)
|
||||
|
||||
|
||||
def test_responses_codex_exec_whole_result_read_stays_verbatim(monkeypatch):
|
||||
"""`text(r)` sends the exec envelope; the code read inside must stay verbatim."""
|
||||
monkeypatch.setenv("HEADROOM_PROTECT_READS", "1")
|
||||
handler = _handler_with_router(_lossy_router())
|
||||
call = _codex_exec_call("call_exec", "nl -ba src/models.rs | sed -n '1,400p'")
|
||||
call["input"] = call["input"].replace("text(r0.output)", "text(r0)")
|
||||
output = _codex_exec_output("call_exec", _exec_envelope(_RUST_NL))
|
||||
payload = {"model": "gpt-5", "input": [call, output]}
|
||||
|
||||
new_payload, _modified, _s, _t, _u, _c, _a = _run(handler, payload)
|
||||
|
||||
assert new_payload["input"][1] == output
|
||||
|
||||
Reference in New Issue
Block a user