mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): protect Codex exec file reads on the Responses path (#3621)
## Description
Codex CLI 0.153 sends every shell command as a Responses
`custom_tool_call` named `exec`. Its `input` is JavaScript that calls
`tools.exec_command({"cmd": ...})`, not JSON arguments. Read protection
(`HEADROOM_PROTECT_READS`, on in the default `coding` profile) only
parsed `function_call` and `local_shell_call`, so Codex's `sed -n`,
`nl`, and `cat` reads were lossy-compressed. Codex then re-ran searches
and re-read the same lines, which cancelled the per-request saving.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `content_router._custom_tool_call_commands` pulls each `cmd` out of an
`exec` script by JSON-decoding the object passed to `exec_command(`.
Input that is not that shape, or not strict JSON, yields nothing, so the
output stays compressible as before.
- The Responses read scan in `handlers/openai.py` handles
`custom_tool_call`. A script with several commands is treated as a read
when any command is a read, because it produces one output.
- Tests: helper parsing (including braces and quotes inside the
command), a Codex `exec` read kept verbatim with a content-part output,
a multi-command script with a read, and a control where test output
still compresses.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) on the changed modules
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_openai_responses_read_protection.py
without the fix: 4 failed, 17 passed
with the fix: 21 passed
$ pytest tests/test_openai_responses_read_protection.py tests/test_openai_responses_compression_units.py \
tests/test_responses_pyo3_compression.py tests/test_compression_units.py tests/test_compression_batches.py \
tests/test_output_turn_policy.py tests/test_transforms_content_router.py tests/test_proxy
451 passed
$ uvx ruff@0.16.3 check <changed files>; mypy <changed modules>
All checks passed!
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: Codex CLI 0.153.2, gpt-5.6-terra, Headroom proxy with no
license and default settings, a recorder at the OpenAI edge logging
every request's usage.
- Exact command / steps: five SWE-bench-style bug-fix tasks, three runs
each, `codex exec` through the proxy. Compared against Codex direct to
OpenAI.
- Observed result:
| 15 runs each | Direct | Headroom before | Headroom with this fix |
|---|---|---|---|
| Solved | 15/15 | 15/15 | 15/15 |
| Requests per run | 6.1 | 7.3 | 6.1 |
| Cost per run | $0.0766 | $0.0803 | $0.0741 |
| Cost per request | $0.01245 | $0.01103 | $0.01226 |
| Prefix-cache breaks | 0 | 0 | 0 |
Before the fix, Codex re-read files it had just read (4.1 repeat file
reads per run against 2.1 direct), so the 11% per-request saving turned
into a 5% higher cost per run. With the fix, request count matches
direct and cost per run is 3% below direct (within noise at this sample
size).
- Not tested: interactive Codex, and Codex versions that still use
`function_call` shell tools (already covered by existing tests).
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: yes, Codex `exec` file reads are no
longer lossy-compressed when read protection is on.
- Kill switch / disable path: `HEADROOM_PROTECT_READS=0`.
- Unsafe override required: no.
- Qualification impact: less compression on Codex file reads, fewer
re-reads.
- Rollback path: revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
No docs change: read protection is already documented, and this extends
it to one more wire shape. Codex also reads files with `awk 'NR>=a &&
NR<=b'`, which `_is_read_command` does not treat as a read. That is a
separate follow-up.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
35b11564c6
commit
2c56a1b3a7
@@ -2157,10 +2157,13 @@ class OpenAIHandlerMixin:
|
||||
# _tool_call_command_text helper:
|
||||
# - function_call.arguments (Copilot bash, Codex exec_command, …)
|
||||
# - local_shell_call.action (native Responses shell; argv or string)
|
||||
# - custom_tool_call.input (Codex code-mode `exec`: JavaScript calling
|
||||
# tools.exec_command({"cmd": …}))
|
||||
# Content is gated per-output by _read_output_should_be_protected so
|
||||
# confidently non-code DATA reads (lockfiles, JSON, logs, search) stay
|
||||
# compressible, exactly like the chat path.
|
||||
from headroom.transforms.content_router import (
|
||||
_custom_tool_call_commands,
|
||||
_is_read_command,
|
||||
_read_output_should_be_protected,
|
||||
_tool_call_command_text,
|
||||
@@ -2177,6 +2180,17 @@ class OpenAIHandlerMixin:
|
||||
command = _tool_call_command_text(item.get("arguments"))
|
||||
elif item_type == "local_shell_call":
|
||||
command = _tool_call_command_text(item.get("action"))
|
||||
elif item_type == "custom_tool_call":
|
||||
# One script can run several commands; its single output is a
|
||||
# read when any of them is (over-protecting only costs savings).
|
||||
command = next(
|
||||
(
|
||||
c
|
||||
for c in _custom_tool_call_commands(item.get("input"))
|
||||
if _is_read_command(c)
|
||||
),
|
||||
"",
|
||||
)
|
||||
else:
|
||||
continue
|
||||
call_id = item.get("call_id")
|
||||
|
||||
@@ -567,6 +567,40 @@ def _tool_call_command_text(raw: Any) -> str:
|
||||
return cmd if isinstance(cmd, str) else ""
|
||||
|
||||
|
||||
_EXEC_COMMAND_CALL_RE = re.compile(r"\bexec_command\s*\(")
|
||||
|
||||
|
||||
def _custom_tool_call_commands(raw: Any) -> list[str]:
|
||||
"""Extract shell commands from a Codex code-mode ``exec`` custom tool call.
|
||||
|
||||
Codex sends shell commands as a Responses ``custom_tool_call`` named ``exec``
|
||||
whose ``input`` is a JavaScript snippet rather than JSON arguments::
|
||||
|
||||
const r = await tools.exec_command({"cmd": "sed -n '1,80p' f.py", "workdir": "…"});
|
||||
text(r.output);
|
||||
|
||||
Returns every ``cmd`` passed to ``exec_command``, in order. Returns ``[]`` when
|
||||
the input is not that shape; an argument object that is not strict JSON is
|
||||
skipped, which leaves the output compressible exactly as before.
|
||||
"""
|
||||
if not isinstance(raw, str) or "exec_command" not in raw:
|
||||
return []
|
||||
decoder = json.JSONDecoder()
|
||||
commands: list[str] = []
|
||||
for match in _EXEC_COMMAND_CALL_RE.finditer(raw):
|
||||
start = raw.find("{", match.end())
|
||||
if start < 0 or raw[match.end() : start].strip():
|
||||
continue
|
||||
try:
|
||||
args, _end = decoder.raw_decode(raw, start)
|
||||
except ValueError:
|
||||
continue
|
||||
command = _tool_call_command_text(args)
|
||||
if command:
|
||||
commands.append(command)
|
||||
return commands
|
||||
|
||||
|
||||
def _fenced_shell_command(content: Any) -> str:
|
||||
"""Extract the shell command from a TEXT-BASED agent's fenced code block.
|
||||
|
||||
|
||||
@@ -529,3 +529,109 @@ def test_responses_read_scan_tolerates_non_dict_and_missing_call_id(monkeypatch)
|
||||
assert modified is True
|
||||
assert new_payload["input"][0] == "a bare string item"
|
||||
assert new_payload["input"][3]["output"] == "kept words"
|
||||
|
||||
|
||||
def _codex_exec_call(call_id: str, *commands: str) -> dict:
|
||||
"""Codex code-mode `exec` custom tool call, as Codex 0.15x sends it."""
|
||||
import json as _json
|
||||
|
||||
script = "".join(
|
||||
"const r{i} = await tools.exec_command({args});\ntext(r{i}.output);\n".format(
|
||||
i=i,
|
||||
args=_json.dumps({"cmd": cmd, "workdir": "/repo", "yield_time_ms": 10000}),
|
||||
)
|
||||
for i, cmd in enumerate(commands)
|
||||
)
|
||||
return {"type": "custom_tool_call", "call_id": call_id, "name": "exec", "input": script}
|
||||
|
||||
|
||||
def _codex_exec_output(call_id: str, text: str) -> dict:
|
||||
return {
|
||||
"type": "custom_tool_call_output",
|
||||
"call_id": call_id,
|
||||
"output": [
|
||||
{"type": "input_text", "text": "Script completed\nWall time 0.1 seconds\nOutput:\n"},
|
||||
{"type": "input_text", "text": text},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def test_custom_tool_call_commands_parses_codex_exec_input():
|
||||
from headroom.transforms.content_router import _custom_tool_call_commands
|
||||
|
||||
call = _codex_exec_call("c", "sed -n '1,80p' tenacity/wait.py", "rg -n 'def f' src")
|
||||
assert _custom_tool_call_commands(call["input"]) == [
|
||||
"sed -n '1,80p' tenacity/wait.py",
|
||||
"rg -n 'def f' src",
|
||||
]
|
||||
# Braces and quotes inside the command must not break the JSON scan.
|
||||
awk = _codex_exec_call("c", "awk 'NR >= 3 {printf \"%4d: %s\\n\", NR, $0}' f.py")
|
||||
assert _custom_tool_call_commands(awk["input"]) == [
|
||||
"awk 'NR >= 3 {printf \"%4d: %s\\n\", NR, $0}' f.py"
|
||||
]
|
||||
|
||||
|
||||
def test_custom_tool_call_commands_ignores_other_shapes():
|
||||
from headroom.transforms.content_router import _custom_tool_call_commands
|
||||
|
||||
assert _custom_tool_call_commands(None) == []
|
||||
assert _custom_tool_call_commands({"cmd": "cat f"}) == []
|
||||
assert _custom_tool_call_commands("*** Begin Patch\n*** Update File: f.py\n") == []
|
||||
assert _custom_tool_call_commands("tools.exec_command(notJson)") == []
|
||||
assert _custom_tool_call_commands("tools.exec_command({cmd: 'cat f'})") == []
|
||||
|
||||
|
||||
def test_responses_codex_exec_read_stays_verbatim(monkeypatch):
|
||||
"""Codex's `exec` custom tool: a sed/nl file read must reach the model verbatim."""
|
||||
monkeypatch.setenv("HEADROOM_PROTECT_READS", "1")
|
||||
handler = _handler_with_router(_lossy_router())
|
||||
output = _codex_exec_output("call_exec", _NL_OUTPUT)
|
||||
payload = {
|
||||
"model": "gpt-5",
|
||||
"input": [
|
||||
_codex_exec_call("call_exec", "nl -ba tenacity/wait.py | sed -n '20,115p'"),
|
||||
output,
|
||||
],
|
||||
}
|
||||
|
||||
new_payload, _modified, _s, _t, _u, _c, _a = _run(handler, payload)
|
||||
|
||||
assert new_payload["input"][1] == output
|
||||
|
||||
|
||||
def test_responses_codex_exec_script_with_a_read_among_commands_stays_verbatim(monkeypatch):
|
||||
"""One script, one output: protect it when any command in the script is a read."""
|
||||
monkeypatch.setenv("HEADROOM_PROTECT_READS", "1")
|
||||
handler = _handler_with_router(_lossy_router())
|
||||
output = _codex_exec_output("call_multi", _NL_OUTPUT)
|
||||
payload = {
|
||||
"model": "gpt-5",
|
||||
"input": [
|
||||
_codex_exec_call(
|
||||
"call_multi", "git status --short", "sed -n '1,130p' tenacity/wait.py"
|
||||
),
|
||||
output,
|
||||
],
|
||||
}
|
||||
|
||||
new_payload, _modified, _s, _t, _u, _c, _a = _run(handler, payload)
|
||||
|
||||
assert new_payload["input"][1] == output
|
||||
|
||||
|
||||
def test_responses_codex_exec_test_output_still_compresses(monkeypatch):
|
||||
"""Control: a Codex `exec` running tests is not a read and stays compressible."""
|
||||
monkeypatch.setenv("HEADROOM_PROTECT_READS", "1")
|
||||
handler = _handler_with_router(_lossy_router())
|
||||
payload = {
|
||||
"model": "gpt-5",
|
||||
"input": [
|
||||
_codex_exec_call("call_pytest", "python3 -m pytest -q tests/test_wait.py"),
|
||||
_codex_exec_output("call_pytest", _NL_OUTPUT),
|
||||
],
|
||||
}
|
||||
|
||||
new_payload, modified, _s, _t, _u, _c, _a = _run(handler, payload)
|
||||
|
||||
assert modified is True
|
||||
assert new_payload["input"][1] != _codex_exec_output("call_pytest", _NL_OUTPUT)
|
||||
|
||||
Reference in New Issue
Block a user