mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(learn): drop the echoed prompt from a failed cli's error (#3928)
## Description
`codex exec` echoes the whole stdin prompt to stderr ahead of its own
error, and a failed CLI's error message keeps the first 2000 chars of
stderr. The analyzer's system prompt alone is 4116 chars, so every
codex-cli failure reported Codex's banner and the start of our system
prompt and never the reason (a 401, an unknown model, a rate limit). The
echo also carries the session digest, which does not belong in an error
message.
`_failure_detail` now takes the prompt that was sent on stdin. When
stderr contains it, the echo is replaced with `[prompt omitted]` and the
end of what the CLI printed after it is kept, cut at a line boundary,
inside the same 2000-char cap. Stderr from a CLI that does not echo the
prompt is handled exactly as before.
Closes #3927
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/learn/analyzer.py`: `_failure_detail` gets a keyword-only
`prompt` argument; when the prompt (newlines normalized the way a
text-mode pipe reads them back) appears in stderr, it is replaced with a
marker and the text after it is tailed to fit the cap, whole lines where
possible.
- Both callers pass the prompt they wrote to stdin: the `run(...)` path
used by codex-cli and gemini-cli, and the claude-cli streaming path
(claude does not echo today, so nothing changes there).
- `tests/test_learn/test_analyzer.py`:
`test_codex_failure_drops_prompt_echo_and_keeps_reason` feeds stderr in
the shape codex-cli 0.156.1 prints (banner, `user`, the prompt, 100
reconnect lines, the final error) and checks the message ends with the
real reason, carries the marker, has no digest text, and stays inside
the cap.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Ran on the changed files only (`ruff check`/`ruff format --check` on the
two changed files, `mypy headroom/learn/analyzer.py`), not the whole
tree.
### Test Output
Before the fix (new test against upstream/main's analyzer):
```text
$ pytest tests/test_learn/test_analyzer.py -q -k prompt_echo
E AssertionError: assert False
E + where False = <built-in method endswith of str object at 0x77522a6400>('ERROR: unexpected status 401 Unauthorized: Missing bearer')
E + where <built-in method endswith of str object at 0x77522a6400> = '`codex exec --skip-git-repo-check` failed (exit 1):\nReading prompt from stdin...\nOpenAI Codex v0.156.1\n--------\nm...endation (how many tokens would be saved per session if this rule existed)\n- Separate stable project facts (CONTEXT_F'.endswith
```
After:
```text
$ pytest tests/test_learn/test_analyzer.py tests/test_learn/test_error_classification.py -q
============================= 102 passed in 4.73s ==============================
$ ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
2 files already formatted
$ mypy headroom/learn/analyzer.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS 27, Python 3.12, codex-cli 0.156.1 (Homebrew),
upstream/main d5318ac2 vs this branch.
- Exact command / steps: `CODEX_HOME=<empty dir> python run_proof.py`,
where the script calls the real `analyzer._call_cli_llm(digest,
"codex-cli")` with a 300-line fake digest (28,586 prompt chars on
stdin). The empty `CODEX_HOME` gives codex no credentials, so the real
`codex exec --skip-git-repo-check` fails with a 401 and spends no
tokens. The script prints the RuntimeError and whether the system prompt
or digest text appears in it.
- Observed result: before the fix the error is 2052 chars of Codex
banner plus our system prompt with no 401 in it; with it the echo is
replaced by `[prompt omitted]` and the message ends with codex's own 401
line (output below).
Before (upstream/main), last lines of the message:
```text
error chars: 2052
system prompt in error: True
digest in error: False
...
its guardrail even if it appears only once as a loop, and set its
estimated_tokens_saved to at least the measured wasted tokens reported
there.
- Only include patterns with CLEAR evidence from the data (2+
```
After (this branch), the whole message:
```text
error chars: 2037
system prompt in error: False
digest in error: False
`codex exec --skip-git-repo-check` failed (exit 1):
Reading prompt from stdin...
OpenAI Codex v0.156.1
--------
workdir: <scratch dir>/echo-proof
model: gpt-6-astra
provider: openai
approval: never
sandbox: read-only
reasoning effort: none
reasoning summaries: none
session id: 01a0fcc7-cd87-7333-be41-6981201cf854
--------
user
[prompt omitted]
ERROR: Reconnecting... 2/5
2026-10-02T13:22:30.743051Z ERROR
codex_api::endpoint::responses_websocket: failed to connect to
websocket: HTTP error: 401 Unauthorized, url:
wss://api.openai.com/v1/responses
ERROR: Reconnecting... 3/5
...
warning: Falling back from WebSockets to HTTPS transport. unexpected
status 401 Unauthorized: Missing bearer or basic authentication in
header, url: wss://api.openai.com/v1/responses, cf-ray:
a444045a1da3d919-AMS
ERROR: Reconnecting... 1/5
...
ERROR: Reconnecting... 5/5
ERROR: unexpected status 401 Unauthorized: Missing bearer or basic
authentication in header, url: https://api.openai.com/v1/responses,
cf-ray: a444048d19332162-AMS, request id:
req_c4fd2b8fee604e3f932b68bc90488ac1
ERROR: unexpected status 401 Unauthorized: Missing bearer or basic
authentication in header, url: https://api.openai.com/v1/responses,
cf-ray: a444048d19332162-AMS, request id:
req_c4fd2b8fee604e3f932b68bc90488ac1
```
(`...` marks lines elided here for length; the real message keeps them.)
- Not tested: other codex failure reasons (bad model name, rate limit,
network) were not triggered for real; they print after the echo the same
way, which is what the fix keys on. gemini-cli was not run (not
installed here); if it does not echo the prompt its errors are
unchanged. Windows was not tested: the newline normalization covers a
CRLF echo read back through a text-mode pipe, but that path has only
been reasoned about, not run. The success-path `logger.debug` of stderr
is untouched.
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a, not rollout-managed.
- Stable/default behavior changed: only the text of the RuntimeError
raised when a learn CLI backend exits non-zero and echoed the prompt to
stderr. No change to what is sent to any CLI or model, or to anything
outside `headroom learn`.
- Kill switch / disable path: none; diagnostics text only.
- Unsafe override required: no.
- Qualification impact: none; learn error reporting only, no proxy,
compression or cache path.
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- Documentation: no user-facing docs describe the error text, so nothing
to update.
- Builds on #3080 (stdout in CLI failure messages) and #3517 (both ends
of unparseable output); this is the stderr half for a CLI that echoes
its input.
- Headroom Desktop has shipped this as a downstream patch to the pinned
wheel; this upstreams it.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3928"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
2a4d34c476
commit
f00d425614
@@ -642,7 +642,11 @@ def _output_snippet(text: str, limit: int = _MAX_SNIPPET_LEN) -> str:
|
||||
|
||||
|
||||
def _failure_detail(
|
||||
stderr: str | None, stdout: str | None, *, result_text: str | None = None
|
||||
stderr: str | None,
|
||||
stdout: str | None,
|
||||
*,
|
||||
result_text: str | None = None,
|
||||
prompt: str | None = None,
|
||||
) -> str:
|
||||
"""Build the operator-facing reason for a non-zero CLI exit.
|
||||
|
||||
@@ -656,17 +660,39 @@ def _failure_detail(
|
||||
than headed because CLI backends emit the error last (a streaming backend's
|
||||
whole event log precedes it).
|
||||
|
||||
A CLI that echoes its stdin prompt to stderr (``codex exec`` prints the whole
|
||||
prompt ahead of its own error) would otherwise fill the head-of-stderr cut
|
||||
with our system prompt and never reach the reason. The echo is replaced with
|
||||
a marker and only the end of what follows it is kept, which also keeps the
|
||||
session digest out of the message.
|
||||
|
||||
Args:
|
||||
stderr: Captured stderr, if any.
|
||||
stdout: Captured stdout, if any.
|
||||
result_text: Pre-extracted reason (claude-cli's final ``result`` field),
|
||||
used in place of the raw stdout tail when available.
|
||||
prompt: The prompt sent on stdin, dropped from stderr when echoed there.
|
||||
|
||||
Returns:
|
||||
A non-empty snippet, or ``"(no output captured)"`` when both streams were
|
||||
empty, so the message is never a dangling colon.
|
||||
"""
|
||||
parts: list[str] = []
|
||||
# Text-mode pipes read the echo back with universal newlines.
|
||||
echo = prompt.replace("\r\n", "\n").replace("\r", "\n").strip() if prompt else ""
|
||||
if stderr and echo and echo in stderr:
|
||||
head, _, after = stderr.strip().partition(echo)
|
||||
marker = "[prompt omitted]\n"
|
||||
after = after.strip()
|
||||
# At least half the cap goes to what follows the echo, so a long
|
||||
# banner ahead of it cannot push the CLI's verdict out.
|
||||
room = max(_MAX_SNIPPET_LEN - len(head) - len(marker), _MAX_SNIPPET_LEN // 2)
|
||||
head = head[: _MAX_SNIPPET_LEN - room - len(marker)] + marker
|
||||
if len(after) > room:
|
||||
# Keep whole lines where possible: the CLI's verdict is the last one.
|
||||
cut = after[len(after) - room :]
|
||||
after = cut.partition("\n")[2] or cut
|
||||
stderr = head + after
|
||||
if stderr and stderr.strip():
|
||||
parts.append(stderr.strip()[:_MAX_SNIPPET_LEN])
|
||||
tail = result_text if result_text and result_text.strip() else stdout
|
||||
@@ -754,7 +780,7 @@ def _call_cli_llm(
|
||||
) from None
|
||||
|
||||
if result.returncode != 0:
|
||||
detail = _failure_detail(result.stderr, result.stdout)
|
||||
detail = _failure_detail(result.stderr, result.stdout, prompt=prompt)
|
||||
raise RuntimeError(f"`{' '.join(cmd)}` failed (exit {result.returncode}):\n{detail}")
|
||||
|
||||
# Log stderr warnings even on success (auth refreshes, deprecation notices).
|
||||
@@ -924,7 +950,7 @@ def _call_claude_cli_streaming(
|
||||
# the human-readable reason ("API Error: ...", "Not logged in", usage
|
||||
# limits).
|
||||
detail = _failure_detail(
|
||||
"".join(stderr_lines), "".join(stdout_lines), result_text=final_result
|
||||
"".join(stderr_lines), "".join(stdout_lines), result_text=final_result, prompt=prompt
|
||||
)
|
||||
raise RuntimeError(f"`{' '.join(cmd)}` failed (exit {proc.returncode}):\n{detail}")
|
||||
|
||||
|
||||
@@ -1064,6 +1064,50 @@ class TestCallCliLlm:
|
||||
_call_cli_llm("test digest", "codex-cli")
|
||||
assert long_stderr not in str(exc_info.value)
|
||||
|
||||
@patch("headroom.learn.analyzer.subprocess.run")
|
||||
def test_codex_failure_drops_prompt_echo_and_keeps_reason(self, mock_run: MagicMock):
|
||||
# `codex exec` echoes the whole stdin prompt to stderr ahead of its own
|
||||
# error, so a head-of-stderr excerpt is the banner plus our system prompt
|
||||
# and never the reason. Shape captured from codex-cli 0.156.1.
|
||||
def _codex(cmd, *, input, **kwargs):
|
||||
stderr = (
|
||||
"Reading prompt from stdin...\nOpenAI Codex v0.156.1\n--------\n"
|
||||
"model: gpt-5\n--------\nuser\n" + input + "\n\n"
|
||||
)
|
||||
stderr += "ERROR: Reconnecting... 1/5\n" * 100
|
||||
stderr += "ERROR: unexpected status 401 Unauthorized: Missing bearer\n"
|
||||
return MagicMock(returncode=1, stdout="", stderr=stderr)
|
||||
|
||||
mock_run.side_effect = _codex
|
||||
with pytest.raises(RuntimeError) as exc_info:
|
||||
_call_cli_llm("SECRET-SESSION-DIGEST", "codex-cli")
|
||||
message = str(exc_info.value)
|
||||
assert message.endswith("ERROR: unexpected status 401 Unauthorized: Missing bearer")
|
||||
assert "[prompt omitted]" in message
|
||||
assert "SECRET-SESSION-DIGEST" not in message
|
||||
assert "OpenAI Codex v0.156.1" in message
|
||||
# Whole lines only, still inside the snippet cap.
|
||||
assert "\nERROR: Reconnecting... 1/5\n" in message
|
||||
assert len(message) < 2200
|
||||
|
||||
@patch("headroom.learn.analyzer.subprocess.run")
|
||||
def test_codex_failure_keeps_reason_behind_a_long_banner(self, mock_run: MagicMock):
|
||||
# Diagnostics ahead of the echo longer than the whole snippet cap must
|
||||
# not push the CLI's final error out of the message.
|
||||
def _codex(cmd, *, input, **kwargs):
|
||||
stderr = "WARN startup diagnostic line\n" * 100 + input + "\n"
|
||||
stderr += "ERROR: unknown model gpt-x\n"
|
||||
return MagicMock(returncode=1, stdout="", stderr=stderr)
|
||||
|
||||
mock_run.side_effect = _codex
|
||||
with pytest.raises(RuntimeError) as exc_info:
|
||||
_call_cli_llm("SECRET-SESSION-DIGEST", "codex-cli")
|
||||
message = str(exc_info.value)
|
||||
assert message.endswith("[prompt omitted]\nERROR: unknown model gpt-x")
|
||||
assert message.startswith("`codex exec")
|
||||
assert "SECRET-SESSION-DIGEST" not in message
|
||||
assert len(message) < 2200
|
||||
|
||||
def test_unknown_cli_model_raises(self):
|
||||
with pytest.raises(ValueError, match="Unknown CLI model"):
|
||||
_call_cli_llm("test digest", "unknown-cli")
|
||||
|
||||
Reference in New Issue
Block a user