mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): protect Codex exec reads whose argument is a JavaScript object literal (#3737)
## Description #3621 added read protection for Codex code-mode `exec` calls: `_custom_tool_call_commands` pulls each `cmd` out of `tools.exec_command(...)` so a `cat` / `sed -n` / `nl` read reaches the model verbatim. It JSON-decodes the argument, and skips it when that fails. Codex usually writes that argument as a JavaScript object literal with a bare key, not as JSON: ```js const r = await tools.exec_command({cmd:"cat /private/tmp/hr-bundle.js","workdir":"/private/tmp"}); const r = await tools.exec_command({ cmd: "sed -n '1,80p' f.py", workdir: "/repo" }); ``` Neither is valid JSON, so these reads were never protected and were lossy-compressed as before #3621. Across 1,125 `exec` calls in real Codex 0.15x session logs (`~/.codex/sessions`, September 2026), the current parser returns commands for 129. With this change it returns commands for 696, and the read output that read protection keeps verbatim grows from about 50k to 425k tokens. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `content_router._custom_tool_call_commands`: when strict JSON decoding of the `exec_command` argument fails, read the `cmd` property from the object literal instead. The key may be bare or quoted; the value must be a plain `"..."`, `'...'` or backtick string. Simple escapes are unescaped. - A template literal with `${...}` substitution, or a `cmd` that is not a string literal (a variable, an expression), still yields nothing, so that output stays compressible exactly as before. - Strict JSON arguments take the existing path unchanged. - Tests: the parser on five literal shapes (bare key, quoted key, single quotes, backticks, key not first), the dynamic shapes that must still yield nothing, and a Codex `exec` read with a bare-key argument kept verbatim through `_compress_openai_responses_live_text_units_with_router`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_openai_responses_read_protection.py -q 23 passed in 1.50s # With only the content_router.py change reverted: FAILED tests/test_openai_responses_read_protection.py::test_custom_tool_call_commands_parses_javascript_object_literals FAILED tests/test_openai_responses_read_protection.py::test_responses_codex_exec_javascript_literal_read_stays_verbatim 2 failed, 21 passed in 1.01s $ uv run --frozen --extra dev pytest tests/ -q -k "read_protect or content_router or responses or codex" 1075 passed, 26 skipped, 12245 deselected $ uvx ruff check headroom/transforms/content_router.py tests/test_openai_responses_read_protection.py All checks passed! $ uv run --frozen --extra dev mypy headroom/transforms/content_router.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS 15, Python 3.12, headroom-ai 0.38.0 as installed by Headroom Desktop, Codex CLI / Codex Desktop 0.15x session logs. - Exact command / steps: Parsed every `custom_tool_call` in all 102 real `~/.codex/sessions/2026/09/**/*.jsonl` rollouts (86 contain `exec` calls) with the current and the patched `_custom_tool_call_commands`, then summed the output of calls whose command `_is_read_command` accepts and whose output `_read_output_should_be_protected` keeps. Separately ran a Codex `exec` read with a bare-key argument through the 0.38.0 wheel's Responses units path with `HEADROOM_PROTECT_READS=1`, with and without the patched parser. - Observed result: 129 of 1,125 `exec` calls parsed before, 696 after; protected read output about 50k tokens before, 425k after. The bare-key read was compressed without the patch and reached the model byte-identical with it. A non-read `exec` output in the same payload still compressed. - Not tested: Codex versions before 0.15x, and whether fewer lossy reads reduce Codex re-reads over a full session (the motivation #3621 gives). ## Runtime Rollout Safety - Rollout-managed feature(s): None. This widens the existing `HEADROOM_PROTECT_READS` read protection to an argument shape it was meant to cover. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes, when `HEADROOM_PROTECT_READS` is on: more Codex file-read outputs are kept verbatim instead of compressed. With it off, nothing changes. - Kill switch / disable path: `HEADROOM_PROTECT_READS=0` disables read protection, including this path. - Unsafe override required: No. - Qualification impact: Codex requests with file reads compress somewhat less. Replaying three real Codex coding sessions with read protection on, the per-session savings rate moved +1.1, -0.4 and 0.0 percentage points. - Rollback path: Revert this commit; the parser returns to JSON-only. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes No documentation change: this is internal to read protection. Follow-up to #3621. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
d971f7c398
commit
ecb5e5afea
@@ -2195,19 +2195,22 @@ class OpenAIHandlerMixin:
|
||||
command = _tool_call_command_text(item.get("action"))
|
||||
elif item_type == "custom_tool_call":
|
||||
# One script can run several commands; its single output is a
|
||||
# read when any of them is (over-protecting only costs savings).
|
||||
# read when any of them is, or when one's cmd is not a string
|
||||
# literal and so might be (over-protecting only costs savings).
|
||||
commands = _custom_tool_call_commands(item.get("input"))
|
||||
command = next(
|
||||
(
|
||||
c
|
||||
for c in _custom_tool_call_commands(item.get("input"))
|
||||
if _is_read_command(c)
|
||||
),
|
||||
"",
|
||||
(c for c in commands if c is not None and _is_read_command(c)),
|
||||
"exec_command(<cmd not a string literal>)" if None in commands else "",
|
||||
)
|
||||
else:
|
||||
continue
|
||||
call_id = item.get("call_id")
|
||||
if command and isinstance(call_id, str) and call_id and _is_read_command(command):
|
||||
if (
|
||||
command
|
||||
and isinstance(call_id, str)
|
||||
and call_id
|
||||
and (item_type == "custom_tool_call" or _is_read_command(command))
|
||||
):
|
||||
read_command_by_call_id[call_id] = command
|
||||
# Outputs protected by read-command detection. Also unioned into the
|
||||
# cross-turn dedup protection set below: a [↑…] fold of a read would
|
||||
|
||||
@@ -572,8 +572,36 @@ def _tool_call_command_text(raw: Any) -> str:
|
||||
|
||||
_EXEC_COMMAND_CALL_RE = re.compile(r"\bexec_command\s*\(")
|
||||
|
||||
# The `cmd` property of a JavaScript object literal, as Codex code-mode usually
|
||||
# writes it: `{cmd: "sed -n '1,80p' f.py", workdir: "/repo"}` or
|
||||
# `{cmd:"cat f","workdir":"/repo"}`. Matched from the opening brace; the key may
|
||||
# be bare or quoted and the value a "...", '...' or `...` string that is the
|
||||
# WHOLE property value (the next token ends it: `,` or `}`). Anything else -
|
||||
# `"c" + "at f"`, `"cat f".trim()`, a template literal with `${...}` - is an
|
||||
# expression whose value is not the literal and does not match.
|
||||
_JS_CMD_PROPERTY_RE = re.compile(
|
||||
r"""\{[^{}]*?(?<![\w$])(?:cmd|"cmd"|'cmd')\s*:\s*"""
|
||||
r"""(?:"((?:[^"\\\n]|\\.)*)"|'((?:[^'\\\n]|\\.)*)'|`((?:[^`\\$]|\\.|\$(?!\{))*)`)"""
|
||||
r"""(?=\s*[,}])"""
|
||||
)
|
||||
_JS_STRING_ESCAPES = {"n": "\n", "t": "\t", "r": "\r", "b": "\b", "f": "\f", "v": "\v"}
|
||||
|
||||
def _custom_tool_call_commands(raw: Any) -> list[str]:
|
||||
|
||||
def _js_string_value(body: str) -> str | None:
|
||||
"""Unescape the body of a JavaScript string literal.
|
||||
|
||||
Returns None for an escape it does not decode (``\\x61``, ``\\u0061``, octal),
|
||||
since guessing would misread ``c\\x61t f`` as something other than ``cat f``.
|
||||
"""
|
||||
escaped = re.findall(r"\\(.)", body, flags=re.S)
|
||||
if any(c.isalnum() and c not in _JS_STRING_ESCAPES for c in escaped):
|
||||
return None
|
||||
return re.sub(
|
||||
r"\\(.)", lambda m: _JS_STRING_ESCAPES.get(m.group(1), m.group(1)), body, flags=re.S
|
||||
)
|
||||
|
||||
|
||||
def _custom_tool_call_commands(raw: Any) -> list[str | None]:
|
||||
"""Extract shell commands from a Codex code-mode ``exec`` custom tool call.
|
||||
|
||||
Codex sends shell commands as a Responses ``custom_tool_call`` named ``exec``
|
||||
@@ -582,21 +610,34 @@ def _custom_tool_call_commands(raw: Any) -> list[str]:
|
||||
const r = await tools.exec_command({"cmd": "sed -n '1,80p' f.py", "workdir": "…"});
|
||||
text(r.output);
|
||||
|
||||
The argument is usually a JavaScript object literal rather than JSON
|
||||
(``{cmd: "cat f.py"}``, unquoted key); when strict JSON decoding fails, the
|
||||
``cmd`` property is read from the literal instead.
|
||||
|
||||
Returns every ``cmd`` passed to ``exec_command``, in order. Returns ``[]`` when
|
||||
the input is not that shape; an argument object that is not strict JSON is
|
||||
skipped, which leaves the output compressible exactly as before.
|
||||
the input is not that shape. A call whose ``cmd`` is not statically known (a
|
||||
variable, a concatenation, a template with substitution, an escape this does
|
||||
not decode) yields ``None``: the command may be a read, and callers must treat
|
||||
it as one rather than let an incomplete parse release the output.
|
||||
"""
|
||||
if not isinstance(raw, str) or "exec_command" not in raw:
|
||||
return []
|
||||
decoder = json.JSONDecoder()
|
||||
commands: list[str] = []
|
||||
commands: list[str | None] = []
|
||||
for match in _EXEC_COMMAND_CALL_RE.finditer(raw):
|
||||
start = raw.find("{", match.end())
|
||||
if start < 0 or raw[match.end() : start].strip():
|
||||
commands.append(None) # argument is not an object literal: dynamic
|
||||
continue
|
||||
try:
|
||||
args, _end = decoder.raw_decode(raw, start)
|
||||
except ValueError:
|
||||
literal = _JS_CMD_PROPERTY_RE.match(raw, start)
|
||||
if literal is None:
|
||||
commands.append(None)
|
||||
continue
|
||||
body = next(group for group in literal.groups() if group is not None)
|
||||
commands.append(_js_string_value(body) or None)
|
||||
continue
|
||||
command = _tool_call_command_text(args)
|
||||
if command:
|
||||
|
||||
@@ -577,8 +577,91 @@ def test_custom_tool_call_commands_ignores_other_shapes():
|
||||
assert _custom_tool_call_commands(None) == []
|
||||
assert _custom_tool_call_commands({"cmd": "cat f"}) == []
|
||||
assert _custom_tool_call_commands("*** Begin Patch\n*** Update File: f.py\n") == []
|
||||
assert _custom_tool_call_commands("tools.exec_command(notJson)") == []
|
||||
assert _custom_tool_call_commands("tools.exec_command({cmd: 'cat f'})") == []
|
||||
|
||||
|
||||
def test_custom_tool_call_commands_marks_non_literal_cmd_unknown():
|
||||
"""A cmd that is not a whole string literal may still be a read: None, not skipped."""
|
||||
from headroom.transforms.content_router import _custom_tool_call_commands
|
||||
|
||||
for script in (
|
||||
"tools.exec_command(notJson)",
|
||||
"tools.exec_command({cmd: someVariable})",
|
||||
"tools.exec_command({cmd: `cat ${f}`})",
|
||||
'tools.exec_command({cmd:"c"+"at f"})',
|
||||
"tools.exec_command({cmd: 'c' + 'at f', workdir: '/repo'})",
|
||||
'tools.exec_command({cmd: "cat f".trim()})',
|
||||
'tools.exec_command({cmd: "c\\x61t f"})',
|
||||
'tools.exec_command({cmd: "c\\u0061t f"})',
|
||||
):
|
||||
assert _custom_tool_call_commands(script) == [None], script
|
||||
# The known commands around an unknown one are still returned, in order.
|
||||
assert _custom_tool_call_commands(
|
||||
'tools.exec_command({cmd: "ls"}); tools.exec_command({cmd: "c" + "at f"});'
|
||||
) == ["ls", None]
|
||||
|
||||
|
||||
def test_custom_tool_call_commands_parses_javascript_object_literals():
|
||||
"""Codex usually writes the argument as a JS literal, not JSON (bare `cmd` key)."""
|
||||
from headroom.transforms.content_router import _custom_tool_call_commands
|
||||
|
||||
script = (
|
||||
'const a = await tools.exec_command({cmd:"cat /tmp/app.js","workdir":"/tmp"});\n'
|
||||
"const b = await tools.exec_command({cmd: 'sed -n \\'1,80p\\' f.py', workdir: '/repo'});\n"
|
||||
"const c = await tools.exec_command({ workdir: '/repo', cmd: `nl -ba f.py` });\n"
|
||||
'const d = await tools.exec_command({cmd: "rg -n \\"def f\\" src", yield_time_ms: 1000});\n'
|
||||
"const e = await tools.exec_command({'cmd': 'head -n 5 f.py'});\n"
|
||||
)
|
||||
assert _custom_tool_call_commands(script) == [
|
||||
"cat /tmp/app.js",
|
||||
"sed -n '1,80p' f.py",
|
||||
"nl -ba f.py",
|
||||
'rg -n "def f" src',
|
||||
"head -n 5 f.py",
|
||||
]
|
||||
|
||||
|
||||
def test_responses_codex_exec_javascript_literal_read_stays_verbatim(monkeypatch):
|
||||
"""The same read with Codex's usual bare-key argument must also stay verbatim."""
|
||||
monkeypatch.setenv("HEADROOM_PROTECT_READS", "1")
|
||||
handler = _handler_with_router(_lossy_router())
|
||||
output = _codex_exec_output("call_exec", _NL_OUTPUT)
|
||||
call = {
|
||||
"type": "custom_tool_call",
|
||||
"call_id": "call_exec",
|
||||
"name": "exec",
|
||||
"input": (
|
||||
'const r = await tools.exec_command({cmd: "nl -ba tenacity/wait.py | '
|
||||
'sed -n \'20,115p\'", workdir: "/repo", yield_time_ms: 10000});\n'
|
||||
"text(r.output);\n"
|
||||
),
|
||||
}
|
||||
payload = {"model": "gpt-5", "input": [call, output]}
|
||||
|
||||
new_payload, _modified, _s, _t, _u, _c, _a = _run(handler, payload)
|
||||
|
||||
assert new_payload["input"][1] == output
|
||||
|
||||
|
||||
def test_responses_codex_exec_concatenated_cmd_stays_verbatim(monkeypatch):
|
||||
"""`"c" + "at f"` runs `cat f`: an unparsed cmd must not release the read."""
|
||||
monkeypatch.setenv("HEADROOM_PROTECT_READS", "1")
|
||||
handler = _handler_with_router(_lossy_router())
|
||||
output = _codex_exec_output("call_exec", _NL_OUTPUT)
|
||||
call = {
|
||||
"type": "custom_tool_call",
|
||||
"call_id": "call_exec",
|
||||
"name": "exec",
|
||||
"input": (
|
||||
'const r = await tools.exec_command({cmd: "c" + "at tenacity/wait.py", '
|
||||
'workdir: "/repo"});\n'
|
||||
"text(r.output);\n"
|
||||
),
|
||||
}
|
||||
payload = {"model": "gpt-5", "input": [call, output]}
|
||||
|
||||
new_payload, _modified, _s, _t, _u, _c, _a = _run(handler, payload)
|
||||
|
||||
assert new_payload["input"][1] == output
|
||||
|
||||
|
||||
def test_responses_codex_exec_read_stays_verbatim(monkeypatch):
|
||||
|
||||
Reference in New Issue
Block a user