fix(learn): keep the digest's loop section inside its budget (#4269)

## Description

`headroom learn` can build a prompt larger than the CLI it runs will
accept, and the prompt-too-long retry cannot shrink it.

`_build_digest` budgets the per-session event stream (`max_tokens * 4`
chars), but it renders the "Detected Loops" section ahead of that stream
without any bound: every detected loop, each with every message index it
repeated at. Codex learn scans all of a machine's Codex sessions as one
project, so loops accumulate across every session the user ever ran. On
one Windows machine this produced a 1,786,526-char prompt, and `codex
exec` refused it on every scan:

```text
Error: turn/start: turn/start failed: Input exceeds the maximum length of 1048576 characters. (code -32602), data: {"input_error_code":"input_too_large","max_chars":1048576,"actual_chars":1786526}
```

Halving `max_tokens` on a prompt-too-long error does not help here: it
only trims the event stream, never the loop section.

This PR bounds the rendered loop section by the same budget, and lets a
Codex overflow take the smaller-digest retry.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `learn/loops.py`: `format_loops_for_digest` takes `max_chars` and
counts the rendered text, header and newlines included. It adds loops in
the given order (`detect_loops` sorts by measured waste) and stops at
the first one whose line would pass `max_chars`, so the costliest loops
are kept. Each line shows its sample input cut to 120 chars (the same
cut `detect_loops` makes) and its first 20 message indices beside the
full repeat count. It returns "" when no loop fits.
- `learn/analyzer.py`: `_build_digest` gives the loop section a quarter
of the `max_tokens * 4` char budget, less whatever the header and the
prior learned patterns leave short, so header + loops + prior patterns
stay inside the budget and the event stream fills the rest. The prior
patterns are never cut: a re-emitted section replaces its prior one
wholesale, so a partial baseline would drop bullets from the user's
file.
- `learn/analyzer.py`: the event stream counts every line with the
newline that joins it, and a session header, event or tool-call line
goes in only if it leaves 100 chars for the truncation markers that can
follow it. Once a session is cut, no later session is started. So the
whole digest stays inside `max_tokens * 4`; the only exception is a
prior-patterns section that leaves no room on its own, since it is never
cut.
- `learn/analyzer.py`: `_is_prompt_too_long` also matches Codex's "Input
exceeds the maximum length" refusal, so a Codex overflow is retried at
half the budget like the other backends.
- The caller's `LoopPattern`s are not touched (slices only), and
`apply_loop_weighting` still receives every loop.
- Tests: `test_loop_section_stays_inside_the_digest_budget` is
parametrized over a 120-char Bash input, a 2,000-char Grep pattern and a
2,000-char Glob path, at the 80k and 10k (retry floor) budgets. New
`test_loops_fit_beside_a_large_prior_patterns_section` (80k, 10k). New
`test_whole_digest_stays_inside_the_budget`: 5 sessions of 3,000 failing
calls each, at 80k and 10k, as tool calls and as events, with and
without loops, asserts `len(digest) <= max_tokens * 4`. New
`test_many_tiny_sessions_stay_inside_the_budget`: 20,000 one-call
sessions at 80k and 10k, as tool calls and as events (614 to 4,862
sessions kept), so the blank line closing each session is counted too.
`test_retries_smaller_digest_on_prompt_too_long` adds the real Codex
error text.
- `pyproject.toml` / `uv.lock` (review: Dependency audit): the `aiohttp`
constraint floor is now `>=3.14.4` (CVE-2026-111063, CVE-2026-111064 and
GHSA-2g87-pp6c-29x5, all fixed in 3.14.4), relocked with `uv lock
--upgrade-package aiohttp==3.14.4`. Only aiohttp (3.14.3 -> 3.14.4) and
yarl (1.22.0 -> 1.25.1, which 3.14.4 requires) move. Main still resolves
3.14.3, so the same commit is on #4299, #4286, #4285 and #4269.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

`mypy` passes on the changed module.

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_learn/ -q
384 passed, 4 skipped in 34.16s

$ uvx ruff@0.17.0 check headroom/learn/analyzer.py headroom/learn/loops.py tests/test_learn/test_loop_weighting.py tests/test_learn/test_analyzer.py
All checks passed!
$ uvx ruff@0.17.0 format --check headroom/learn/analyzer.py headroom/learn/loops.py tests/test_learn/test_loop_weighting.py tests/test_learn/test_analyzer.py
4 files already formatted

$ uv run --frozen --extra dev mypy headroom/learn/analyzer.py headroom/learn/loops.py
Success: no issues found in 2 source files
```

With `headroom/learn/` reverted to f52a4356, 7 of the new cases fail:
the 4 long Grep/Glob cases, both prior-patterns cases and the Codex
retry case. With it reverted to d9391dab, all 8
`test_whole_digest_stays_inside_the_budget` cases fail (321,135 to
321,420 chars at 80k, 40,300 to 40,475 at 10k).
`test_many_tiny_sessions_stay_inside_the_budget` passes on df75e3fa and
fails if the per-session blank line is left out of the count (324,858
and 40,590 chars).

## Real Behavior Proof

- Environment: macOS, Python 3.11, a real `~/.codex/sessions` (283
sessions, 407 MB), read only.
- Exact command / steps: `CodexPlugin().scan_project(...)`,
`detect_loops(sessions)`, then `_build_digest(project, sessions,
loops=loops, max_tokens=B)` for B = 80,000 and 10,000 (the retry floor)
on `main`, on f52a4356, on d9391dab and on df75e3fa (c95f5fe8 only adds
a test); plus the review repro (Grep loops with 2,000-char patterns, no
sessions).
- Observed result: on df75e3fa the whole digest stays inside `max_tokens
* 4` at both budgets (319,721 of 320,000 and 39,933 of 40,000 chars on
real sessions), and the loop section inside a quarter of it (tables
below).

| real sessions (291 loops) | 80k tokens (320,000 chars) | 10k tokens
(40,000 chars) |
  |---|---|---|
| `main` | 291 loop lines, 47,546 chars; digest 322,613 | 291 loop
lines, 47,546 chars; digest 58,487 |
| f52a4356 | 200 loop lines, 32,208 chars; digest 322,765 | 25 loop
lines, 4,678 chars; digest 40,376 |
| d9391dab | 291 loop lines, 46,177 chars; digest 322,604 | 60 loop
lines, 9,985 chars; digest 40,283 |
| df75e3fa | 291 loop lines, 46,177 chars; digest 319,721 | 60 loop
lines, 9,985 chars; digest 39,933 |

| review repro, digest chars | 25 loops, 80k | 25 loops, 10k | 200
loops, 80k | 200 loops, 10k |
  |---|---|---|---|---|
  | `main` | 87,080 | 87,080 | 693,630 | 693,630 |
  | f52a4356 | 54,080 | 54,080 | 429,630 | 54,080 |
  | d9391dab, df75e3fa | 7,080 | 7,080 | 53,630 | 10,006 (36 loops) |

In every repro run the caller's `LoopPattern`s still held the 2,000-char
sample and all 300 indices afterwards. Real loops never had long
samples: `detect_loops` already cuts `sample_input` to 120 chars
(longest here: 103), so the long-pattern case only reaches
`_build_digest` through caller-supplied loops.
- Not tested: the Windows machine that produced the 1.79M-char prompt (I
have its error, not its sessions). I did not run `codex exec` on an
oversized prompt (it would write a session to `~/.codex`); the needle
comes from that machine's error text above, and the same string is in
the codex-cli 0.156.1 binary.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes. A full digest now stops at the
last line that fits (it used to let in the line that crossed the budget
and skip newlines in the count, ~0.8% over), so it carries slightly
fewer events. Also, when the rendered loop section would pass a quarter
of the digest's char budget (less with large prior learned patterns), or
a loop repeated more than 20 times: the digest lists only the costliest
loops that fit, with at most 20 message indices each. A Codex "Input
exceeds the maximum length" failure is now retried at half the budget.
Other projects get an identical digest.
- Kill switch / disable path: Not applicable.
- Unsafe override required: No.
- Qualification impact: None. Learn only; the proxy request path and
compression are untouched.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

- No documentation change: the learn docs don't describe the digest's
layout.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
gglucass
2026-10-20 21:48:07 -05:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 290f72c550
commit 506ba16c4a
4 changed files with 209 additions and 44 deletions
+55 -32
View File
@@ -336,8 +336,9 @@ def _is_prompt_too_long(exc: Exception) -> bool:
Matches the known phrasings across backends: claude CLI ("Prompt is too
long"), Anthropic via LiteLLM ("prompt is too long: N tokens > M maximum"),
OpenAI ("context_length_exceeded" / "maximum context length"), and Gemini
("input token count exceeds the maximum").
OpenAI ("context_length_exceeded" / "maximum context length"), Gemini
("input token count exceeds the maximum"), and the codex CLI ("Input exceeds
the maximum length of 1048576 characters").
"""
msg = str(exc).lower()
return any(
@@ -348,6 +349,7 @@ def _is_prompt_too_long(exc: Exception) -> bool:
"maximum context length",
"context window",
"token count exceeds",
"input exceeds the maximum length",
)
)
@@ -392,30 +394,42 @@ def _build_digest(
lines.append(f"Tokens used: {total_tokens_in:,} in / {total_tokens_out:,} out")
lines.append("")
# Detected loops first — the most expensive waste pattern, so the LLM sees
# it before the (budget-truncatable) per-session event stream.
loop_section = format_loops_for_digest(loops)
if loop_section:
lines.append(loop_section)
# Prior learned patterns (if any) — gives the LLM the current baseline so
# it can produce complete updated sections instead of condensed deltas.
prior_section = _build_prior_patterns_section(project)
if prior_section:
lines.append(prior_section)
# Budget tracking — stop adding events when we approach the limit
# Rough estimate: 4 chars per token
char_budget = max_tokens * 4
chars_used = sum(len(ln) for ln in lines)
# Every line costs its length plus the newline joining it, and goes in only
# if it leaves this room for the truncation markers that can follow it (an
# events line, a blank, a sessions line), so the digest stays inside
# char_budget. Only a prior-patterns section that leaves no such room on
# its own (never cut, see below) can push it past.
room = char_budget - 100
# Prior learned patterns (if any) — gives the LLM the current baseline so
# it can produce complete updated sections instead of condensed deltas.
# Never cut: a re-emitted section replaces its prior one wholesale, so a
# partial baseline would drop the missing bullets from the written file.
prior_section = _build_prior_patterns_section(project)
# Detected loops first — the most expensive waste pattern, so the LLM sees
# it before the (budget-truncatable) per-session event stream. The event
# stream cannot trim a section placed ahead of it, so the loops get their
# own cap: a quarter of the budget, less whatever the header and prior
# patterns leave short. Rendered whole, every loop with every index, the
# section alone outgrew the digest on a machine with years of Codex
# sessions, which learn scans as one project.
pre_event_chars = sum(len(ln) + 1 for ln in lines) + len(prior_section) + 2
loop_section = format_loops_for_digest(
loops, max_chars=min(char_budget // 4, room - pre_event_chars)
)
if loop_section:
lines.append(loop_section)
if prior_section:
lines.append(prior_section)
chars_used = sum(len(ln) + 1 for ln in lines)
full = False
for session in sessions:
if chars_used > char_budget:
lines.append(
f"... (remaining {len(sessions) - sessions.index(session)} sessions truncated)"
)
break
session_header = (
f"=== Session {session.session_id[:12]} "
f"({len(session.tool_calls)} calls, {session.failure_count} failures"
@@ -423,29 +437,38 @@ def _build_digest(
if session.total_input_tokens:
session_header += f", {session.total_input_tokens:,} input tokens"
session_header += ") ==="
if full or chars_used + len(session_header) + 1 > room:
lines.append(
f"... (remaining {len(sessions) - sessions.index(session)} sessions truncated)"
)
break
lines.append(session_header)
chars_used += len(session_header)
chars_used += len(session_header) + 1
# Use events if available (richer context), fall back to tool_calls
if session.events:
for event in session.events:
if chars_used > char_budget:
lines.append(" ... (remaining events truncated)")
break
event_line = _format_event(event)
if event_line:
lines.append(event_line)
chars_used += len(event_line)
if not event_line:
continue
if chars_used + len(event_line) + 1 > room:
lines.append(" ... (remaining events truncated)")
full = True
break
lines.append(event_line)
chars_used += len(event_line) + 1
else:
for tc in session.tool_calls:
if chars_used > char_budget:
lines.append(" ... (remaining calls truncated)")
break
tc_line = _format_tool_call(tc)
if chars_used + len(tc_line) + 1 > room:
lines.append(" ... (remaining calls truncated)")
full = True
break
lines.append(tc_line)
chars_used += len(tc_line)
chars_used += len(tc_line) + 1
lines.append("")
chars_used += 1
return "\n".join(lines)
+26 -10
View File
@@ -69,6 +69,11 @@ _PAGINATION_RE = re.compile("|".join(_PAGINATION_PATTERNS), re.IGNORECASE)
# otherwise identical commands do not split a loop into singletons.
_INT_RE = re.compile(r"\b\d+\b")
# A loop's digest line shows its sample input cut to this many chars and its
# first _DIGEST_MAX_INDICES message indices beside its full repeat count.
_SAMPLE_INPUT_CHARS = 120
_DIGEST_MAX_INDICES = 20
@dataclass
class LoopPattern:
@@ -290,7 +295,7 @@ def detect_loops(
LoopPattern(
tool=calls[0].name,
signature=_fuzzy_signature(signatures[key]),
sample_input=calls[0].input_summary[:120],
sample_input=calls[0].input_summary[:_SAMPLE_INPUT_CHARS],
count=count,
is_error_loop=is_error_loop,
wasted_tokens=wasted,
@@ -302,15 +307,19 @@ def detect_loops(
return loops
def format_loops_for_digest(loops: list[LoopPattern]) -> str:
def format_loops_for_digest(loops: list[LoopPattern], max_chars: int | None = None) -> str:
"""Render detected loops as a high-priority digest section for the LLM.
Returns "" when there are no loops so the digest is unchanged in the
common case.
Loops are rendered in the given order (detect_loops sorts them costliest
first). With *max_chars*, rendering stops at the first loop that would
take the section past that many chars, so the costliest ones are kept.
Returns "" when there are no loops, or none fits, so the digest is
unchanged in the common case.
"""
if not loops:
return ""
lines = [
header = [
"=== Detected Loops (HIGHEST PRIORITY) ===",
(
"These tool-call patterns REPEATED within a session — the most "
@@ -321,14 +330,21 @@ def format_loops_for_digest(loops: list[LoopPattern]) -> str:
),
"",
]
used = sum(len(ln) + 1 for ln in header) # each line plus its newline
entries: list[str] = []
for lp in loops:
lines.append(
f'- [{lp.kind}] {lp.tool}: "{lp.sample_input}" '
entry = (
f'- [{lp.kind}] {lp.tool}: "{lp.sample_input[:_SAMPLE_INPUT_CHARS]}" '
f"repeated {lp.count}x, ~{lp.wasted_tokens:,} tokens wasted "
f"(messages {lp.msg_indices})"
f"(messages {lp.msg_indices[:_DIGEST_MAX_INDICES]})"
)
lines.append("")
return "\n".join(lines)
used += len(entry) + 1
if max_chars is not None and used > max_chars:
break
entries.append(entry)
if not entries:
return ""
return "\n".join([*header, *entries, ""])
def _signature_tokens(signature: str) -> set[str]:
+13 -2
View File
@@ -475,11 +475,22 @@ class TestSessionAnalyzer:
assert result.recommendations == []
assert result.analysis_error == "API key not set"
@pytest.mark.parametrize(
"error",
[
"`claude -p` failed (exit 1):\nPrompt is too long",
# What codex exec printed on a real learn run: it refuses a prompt over 2**20 chars.
"`codex exec` failed (exit 1):\nError: turn/start: turn/start failed: Input exceeds "
"the maximum length of 1048576 characters. (code -32602), data: "
'{"input_error_code":"input_too_large","max_chars":1048576,"actual_chars":1786526}',
],
ids=["claude", "codex"],
)
@patch("headroom.learn.analyzer._call_llm")
def test_retries_smaller_digest_on_prompt_too_long(self, mock_call_llm: MagicMock):
def test_retries_smaller_digest_on_prompt_too_long(self, mock_call_llm: MagicMock, error):
"""A context-overflow error rebuilds the digest at half the budget and retries."""
mock_call_llm.side_effect = [
RuntimeError("`claude -p` failed (exit 1):\nPrompt is too long"),
RuntimeError(error),
{"context_file_rules": [], "memory_file_rules": []},
]
+115
View File
@@ -26,6 +26,7 @@ from headroom.learn.fixtures import (
)
from headroom.learn.loops import (
_IDENTITY_FIELDS,
LoopPattern,
_canonical_signature,
_identity_input,
_signature_tokens,
@@ -37,6 +38,7 @@ from headroom.learn.models import (
Recommendation,
RecommendationTarget,
SessionData,
SessionEvent,
ToolCall,
)
@@ -150,6 +152,119 @@ class TestDigestSurfacesLoops:
digest = _build_digest(_project(), [one_off_error_session()])
assert "Detected Loops" not in digest
@pytest.mark.parametrize("max_tokens", [80_000, 10_000])
@pytest.mark.parametrize(
("tool", "sample"),
[
("Bash", "x" * 120),
("Grep", "x" * 2000), # a long search pattern
("Glob", "src/" * 500 + "*.py"), # a long path
],
ids=["bash", "long-grep-pattern", "long-glob-path"],
)
def test_loop_section_stays_inside_the_digest_budget(self, tool, sample, max_tokens):
# Rendered whole, thousands of loops with hundreds of indices each made
# a 1.79M-char prompt, over Codex's 1,048,576-char input cap.
loops = [
LoopPattern(tool, "s", sample, 300, False, 1_000_000 - i, list(range(300)))
for i in range(2000)
]
digest = _build_digest(_project(), [], loops=loops, max_tokens=max_tokens)
section = digest[digest.index("=== Detected Loops") :]
assert len(section) <= max_tokens # a quarter of the budget's chars
# The costliest loops are the ones kept, and the caller's list is intact.
assert "~1,000,000 tokens wasted" in section
assert "~999,000 tokens wasted" not in section
assert loops[0].sample_input == sample
assert len(loops[0].msg_indices) == 300
@pytest.mark.parametrize("max_tokens", [80_000, 10_000])
def test_loops_fit_beside_a_large_prior_patterns_section(self, tmp_path, max_tokens):
# The event stream cannot trim sections placed ahead of it, so the
# loops give way to the prior patterns, which are never cut.
bullets = "".join(f"- rule {i:06d} {'y' * 80}\n" for i in range(max_tokens * 4 // 100))
claude_md = tmp_path / "CLAUDE.md"
claude_md.write_text(
f"<!-- headroom:learn:start -->\n{bullets}<!-- headroom:learn:end -->\n"
)
project = ProjectInfo(
name="proj",
project_path=tmp_path,
data_path=tmp_path / "data",
context_file=claude_md,
)
loops = [
LoopPattern("Grep", "s", "x" * 2000, 300, False, 1_000_000 - i, list(range(300)))
for i in range(25)
]
digest = _build_digest(project, [], loops=loops, max_tokens=max_tokens)
assert bullets.splitlines()[-1] in digest
assert "~1,000,000 tokens wasted" in digest
assert len(digest) <= max_tokens * 4
@pytest.mark.parametrize("with_loops", [False, True], ids=["no-loops", "loops"])
@pytest.mark.parametrize("with_events", [False, True], ids=["tool-calls", "events"])
@pytest.mark.parametrize("max_tokens", [80_000, 10_000])
def test_whole_digest_stays_inside_the_budget(self, max_tokens, with_events, with_loops):
# Far more session text than the budget: every line and its newline
# counts, and none goes in that would leave no room for the markers.
sessions = []
for s in range(5):
calls = [
ToolCall(
name="Bash",
tool_call_id=f"{s}-{i}",
input_data={"command": f"run {i}"},
output="e" * 300,
is_error=True,
msg_index=i,
)
for i in range(3000)
]
events = (
[SessionEvent(type="tool_call", msg_index=c.msg_index, tool_call=c) for c in calls]
if with_events
else []
)
sessions.append(SessionData(session_id=f"s{s}", tool_calls=calls, events=events))
loops = (
[
LoopPattern("Grep", "s", "x" * 2000, 300, False, 1_000_000 - i, list(range(300)))
for i in range(2000)
]
if with_loops
else []
)
digest = _build_digest(_project(), sessions, loops=loops, max_tokens=max_tokens)
assert len(digest) <= max_tokens * 4
assert "remaining" in digest and "sessions truncated" in digest
@pytest.mark.parametrize("with_events", [False, True], ids=["tool-calls", "events"])
@pytest.mark.parametrize("max_tokens", [80_000, 10_000])
def test_many_tiny_sessions_stay_inside_the_budget(self, max_tokens, with_events):
# Thousands of kept sessions, each closed by a blank line whose newline
# counts too; the marker reserve alone would absorb only ~100 of them.
sessions = []
for s in range(20_000):
call = ToolCall(
name="Bash",
tool_call_id=str(s),
input_data={"command": "ls"},
output="ok",
is_error=False,
msg_index=0,
)
events = [SessionEvent(type="tool_call", msg_index=0, tool_call=call)]
sessions.append(
SessionData(
session_id=f"s{s}", tool_calls=[call], events=events if with_events else []
)
)
digest = _build_digest(_project(), sessions, loops=[], max_tokens=max_tokens)
assert len(digest) <= max_tokens * 4
assert digest.count("=== Session ") > 500
assert "sessions truncated" in digest
# =============================================================================
# apply_loop_weighting