mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(learn): keep the digest's loop section inside its budget (#4269)
## Description
`headroom learn` can build a prompt larger than the CLI it runs will
accept, and the prompt-too-long retry cannot shrink it.
`_build_digest` budgets the per-session event stream (`max_tokens * 4`
chars), but it renders the "Detected Loops" section ahead of that stream
without any bound: every detected loop, each with every message index it
repeated at. Codex learn scans all of a machine's Codex sessions as one
project, so loops accumulate across every session the user ever ran. On
one Windows machine this produced a 1,786,526-char prompt, and `codex
exec` refused it on every scan:
```text
Error: turn/start: turn/start failed: Input exceeds the maximum length of 1048576 characters. (code -32602), data: {"input_error_code":"input_too_large","max_chars":1048576,"actual_chars":1786526}
```
Halving `max_tokens` on a prompt-too-long error does not help here: it
only trims the event stream, never the loop section.
This PR bounds the rendered loop section by the same budget, and lets a
Codex overflow take the smaller-digest retry.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `learn/loops.py`: `format_loops_for_digest` takes `max_chars` and
counts the rendered text, header and newlines included. It adds loops in
the given order (`detect_loops` sorts by measured waste) and stops at
the first one whose line would pass `max_chars`, so the costliest loops
are kept. Each line shows its sample input cut to 120 chars (the same
cut `detect_loops` makes) and its first 20 message indices beside the
full repeat count. It returns "" when no loop fits.
- `learn/analyzer.py`: `_build_digest` gives the loop section a quarter
of the `max_tokens * 4` char budget, less whatever the header and the
prior learned patterns leave short, so header + loops + prior patterns
stay inside the budget and the event stream fills the rest. The prior
patterns are never cut: a re-emitted section replaces its prior one
wholesale, so a partial baseline would drop bullets from the user's
file.
- `learn/analyzer.py`: the event stream counts every line with the
newline that joins it, and a session header, event or tool-call line
goes in only if it leaves 100 chars for the truncation markers that can
follow it. Once a session is cut, no later session is started. So the
whole digest stays inside `max_tokens * 4`; the only exception is a
prior-patterns section that leaves no room on its own, since it is never
cut.
- `learn/analyzer.py`: `_is_prompt_too_long` also matches Codex's "Input
exceeds the maximum length" refusal, so a Codex overflow is retried at
half the budget like the other backends.
- The caller's `LoopPattern`s are not touched (slices only), and
`apply_loop_weighting` still receives every loop.
- Tests: `test_loop_section_stays_inside_the_digest_budget` is
parametrized over a 120-char Bash input, a 2,000-char Grep pattern and a
2,000-char Glob path, at the 80k and 10k (retry floor) budgets. New
`test_loops_fit_beside_a_large_prior_patterns_section` (80k, 10k). New
`test_whole_digest_stays_inside_the_budget`: 5 sessions of 3,000 failing
calls each, at 80k and 10k, as tool calls and as events, with and
without loops, asserts `len(digest) <= max_tokens * 4`. New
`test_many_tiny_sessions_stay_inside_the_budget`: 20,000 one-call
sessions at 80k and 10k, as tool calls and as events (614 to 4,862
sessions kept), so the blank line closing each session is counted too.
`test_retries_smaller_digest_on_prompt_too_long` adds the real Codex
error text.
- `pyproject.toml` / `uv.lock` (review: Dependency audit): the `aiohttp`
constraint floor is now `>=3.14.4` (CVE-2026-111063, CVE-2026-111064 and
GHSA-2g87-pp6c-29x5, all fixed in 3.14.4), relocked with `uv lock
--upgrade-package aiohttp==3.14.4`. Only aiohttp (3.14.3 -> 3.14.4) and
yarl (1.22.0 -> 1.25.1, which 3.14.4 requires) move. Main still resolves
3.14.3, so the same commit is on #4299, #4286, #4285 and #4269.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
`mypy` passes on the changed module.
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_learn/ -q
384 passed, 4 skipped in 34.16s
$ uvx ruff@0.17.0 check headroom/learn/analyzer.py headroom/learn/loops.py tests/test_learn/test_loop_weighting.py tests/test_learn/test_analyzer.py
All checks passed!
$ uvx ruff@0.17.0 format --check headroom/learn/analyzer.py headroom/learn/loops.py tests/test_learn/test_loop_weighting.py tests/test_learn/test_analyzer.py
4 files already formatted
$ uv run --frozen --extra dev mypy headroom/learn/analyzer.py headroom/learn/loops.py
Success: no issues found in 2 source files
```
With `headroom/learn/` reverted to f52a4356, 7 of the new cases fail:
the 4 long Grep/Glob cases, both prior-patterns cases and the Codex
retry case. With it reverted to d9391dab, all 8
`test_whole_digest_stays_inside_the_budget` cases fail (321,135 to
321,420 chars at 80k, 40,300 to 40,475 at 10k).
`test_many_tiny_sessions_stay_inside_the_budget` passes on df75e3fa and
fails if the per-session blank line is left out of the count (324,858
and 40,590 chars).
## Real Behavior Proof
- Environment: macOS, Python 3.11, a real `~/.codex/sessions` (283
sessions, 407 MB), read only.
- Exact command / steps: `CodexPlugin().scan_project(...)`,
`detect_loops(sessions)`, then `_build_digest(project, sessions,
loops=loops, max_tokens=B)` for B = 80,000 and 10,000 (the retry floor)
on `main`, on f52a4356, on d9391dab and on df75e3fa (c95f5fe8 only adds
a test); plus the review repro (Grep loops with 2,000-char patterns, no
sessions).
- Observed result: on df75e3fa the whole digest stays inside `max_tokens
* 4` at both budgets (319,721 of 320,000 and 39,933 of 40,000 chars on
real sessions), and the loop section inside a quarter of it (tables
below).
| real sessions (291 loops) | 80k tokens (320,000 chars) | 10k tokens
(40,000 chars) |
|---|---|---|
| `main` | 291 loop lines, 47,546 chars; digest 322,613 | 291 loop
lines, 47,546 chars; digest 58,487 |
| f52a4356 | 200 loop lines, 32,208 chars; digest 322,765 | 25 loop
lines, 4,678 chars; digest 40,376 |
| d9391dab | 291 loop lines, 46,177 chars; digest 322,604 | 60 loop
lines, 9,985 chars; digest 40,283 |
| df75e3fa | 291 loop lines, 46,177 chars; digest 319,721 | 60 loop
lines, 9,985 chars; digest 39,933 |
| review repro, digest chars | 25 loops, 80k | 25 loops, 10k | 200
loops, 80k | 200 loops, 10k |
|---|---|---|---|---|
| `main` | 87,080 | 87,080 | 693,630 | 693,630 |
| f52a4356 | 54,080 | 54,080 | 429,630 | 54,080 |
| d9391dab, df75e3fa | 7,080 | 7,080 | 53,630 | 10,006 (36 loops) |
In every repro run the caller's `LoopPattern`s still held the 2,000-char
sample and all 300 indices afterwards. Real loops never had long
samples: `detect_loops` already cuts `sample_input` to 120 chars
(longest here: 103), so the long-pattern case only reaches
`_build_digest` through caller-supplied loops.
- Not tested: the Windows machine that produced the 1.79M-char prompt (I
have its error, not its sessions). I did not run `codex exec` on an
oversized prompt (it would write a session to `~/.codex`); the needle
comes from that machine's error text above, and the same string is in
the codex-cli 0.156.1 binary.
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes. A full digest now stops at the
last line that fits (it used to let in the line that crossed the budget
and skip newlines in the count, ~0.8% over), so it carries slightly
fewer events. Also, when the rendered loop section would pass a quarter
of the digest's char budget (less with large prior learned patterns), or
a loop repeated more than 20 times: the digest lists only the costliest
loops that fit, with at most 20 message indices each. A Codex "Input
exceeds the maximum length" failure is now retried at half the budget.
Other projects get an identical digest.
- Kill switch / disable path: Not applicable.
- Unsafe override required: No.
- Qualification impact: None. Learn only; the proxy request path and
compression are untouched.
- Rollback path: Revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
Not applicable.
## Additional Notes
- No documentation change: the learn docs don't describe the digest's
layout.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
290f72c550
commit
506ba16c4a
+55
-32
@@ -336,8 +336,9 @@ def _is_prompt_too_long(exc: Exception) -> bool:
|
||||
|
||||
Matches the known phrasings across backends: claude CLI ("Prompt is too
|
||||
long"), Anthropic via LiteLLM ("prompt is too long: N tokens > M maximum"),
|
||||
OpenAI ("context_length_exceeded" / "maximum context length"), and Gemini
|
||||
("input token count exceeds the maximum").
|
||||
OpenAI ("context_length_exceeded" / "maximum context length"), Gemini
|
||||
("input token count exceeds the maximum"), and the codex CLI ("Input exceeds
|
||||
the maximum length of 1048576 characters").
|
||||
"""
|
||||
msg = str(exc).lower()
|
||||
return any(
|
||||
@@ -348,6 +349,7 @@ def _is_prompt_too_long(exc: Exception) -> bool:
|
||||
"maximum context length",
|
||||
"context window",
|
||||
"token count exceeds",
|
||||
"input exceeds the maximum length",
|
||||
)
|
||||
)
|
||||
|
||||
@@ -392,30 +394,42 @@ def _build_digest(
|
||||
lines.append(f"Tokens used: {total_tokens_in:,} in / {total_tokens_out:,} out")
|
||||
lines.append("")
|
||||
|
||||
# Detected loops first — the most expensive waste pattern, so the LLM sees
|
||||
# it before the (budget-truncatable) per-session event stream.
|
||||
loop_section = format_loops_for_digest(loops)
|
||||
if loop_section:
|
||||
lines.append(loop_section)
|
||||
|
||||
# Prior learned patterns (if any) — gives the LLM the current baseline so
|
||||
# it can produce complete updated sections instead of condensed deltas.
|
||||
prior_section = _build_prior_patterns_section(project)
|
||||
if prior_section:
|
||||
lines.append(prior_section)
|
||||
|
||||
# Budget tracking — stop adding events when we approach the limit
|
||||
# Rough estimate: 4 chars per token
|
||||
char_budget = max_tokens * 4
|
||||
chars_used = sum(len(ln) for ln in lines)
|
||||
# Every line costs its length plus the newline joining it, and goes in only
|
||||
# if it leaves this room for the truncation markers that can follow it (an
|
||||
# events line, a blank, a sessions line), so the digest stays inside
|
||||
# char_budget. Only a prior-patterns section that leaves no such room on
|
||||
# its own (never cut, see below) can push it past.
|
||||
room = char_budget - 100
|
||||
|
||||
# Prior learned patterns (if any) — gives the LLM the current baseline so
|
||||
# it can produce complete updated sections instead of condensed deltas.
|
||||
# Never cut: a re-emitted section replaces its prior one wholesale, so a
|
||||
# partial baseline would drop the missing bullets from the written file.
|
||||
prior_section = _build_prior_patterns_section(project)
|
||||
|
||||
# Detected loops first — the most expensive waste pattern, so the LLM sees
|
||||
# it before the (budget-truncatable) per-session event stream. The event
|
||||
# stream cannot trim a section placed ahead of it, so the loops get their
|
||||
# own cap: a quarter of the budget, less whatever the header and prior
|
||||
# patterns leave short. Rendered whole, every loop with every index, the
|
||||
# section alone outgrew the digest on a machine with years of Codex
|
||||
# sessions, which learn scans as one project.
|
||||
pre_event_chars = sum(len(ln) + 1 for ln in lines) + len(prior_section) + 2
|
||||
loop_section = format_loops_for_digest(
|
||||
loops, max_chars=min(char_budget // 4, room - pre_event_chars)
|
||||
)
|
||||
if loop_section:
|
||||
lines.append(loop_section)
|
||||
if prior_section:
|
||||
lines.append(prior_section)
|
||||
|
||||
chars_used = sum(len(ln) + 1 for ln in lines)
|
||||
full = False
|
||||
|
||||
for session in sessions:
|
||||
if chars_used > char_budget:
|
||||
lines.append(
|
||||
f"... (remaining {len(sessions) - sessions.index(session)} sessions truncated)"
|
||||
)
|
||||
break
|
||||
|
||||
session_header = (
|
||||
f"=== Session {session.session_id[:12]} "
|
||||
f"({len(session.tool_calls)} calls, {session.failure_count} failures"
|
||||
@@ -423,29 +437,38 @@ def _build_digest(
|
||||
if session.total_input_tokens:
|
||||
session_header += f", {session.total_input_tokens:,} input tokens"
|
||||
session_header += ") ==="
|
||||
if full or chars_used + len(session_header) + 1 > room:
|
||||
lines.append(
|
||||
f"... (remaining {len(sessions) - sessions.index(session)} sessions truncated)"
|
||||
)
|
||||
break
|
||||
lines.append(session_header)
|
||||
chars_used += len(session_header)
|
||||
chars_used += len(session_header) + 1
|
||||
|
||||
# Use events if available (richer context), fall back to tool_calls
|
||||
if session.events:
|
||||
for event in session.events:
|
||||
if chars_used > char_budget:
|
||||
lines.append(" ... (remaining events truncated)")
|
||||
break
|
||||
event_line = _format_event(event)
|
||||
if event_line:
|
||||
lines.append(event_line)
|
||||
chars_used += len(event_line)
|
||||
if not event_line:
|
||||
continue
|
||||
if chars_used + len(event_line) + 1 > room:
|
||||
lines.append(" ... (remaining events truncated)")
|
||||
full = True
|
||||
break
|
||||
lines.append(event_line)
|
||||
chars_used += len(event_line) + 1
|
||||
else:
|
||||
for tc in session.tool_calls:
|
||||
if chars_used > char_budget:
|
||||
lines.append(" ... (remaining calls truncated)")
|
||||
break
|
||||
tc_line = _format_tool_call(tc)
|
||||
if chars_used + len(tc_line) + 1 > room:
|
||||
lines.append(" ... (remaining calls truncated)")
|
||||
full = True
|
||||
break
|
||||
lines.append(tc_line)
|
||||
chars_used += len(tc_line)
|
||||
chars_used += len(tc_line) + 1
|
||||
|
||||
lines.append("")
|
||||
chars_used += 1
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
+26
-10
@@ -69,6 +69,11 @@ _PAGINATION_RE = re.compile("|".join(_PAGINATION_PATTERNS), re.IGNORECASE)
|
||||
# otherwise identical commands do not split a loop into singletons.
|
||||
_INT_RE = re.compile(r"\b\d+\b")
|
||||
|
||||
# A loop's digest line shows its sample input cut to this many chars and its
|
||||
# first _DIGEST_MAX_INDICES message indices beside its full repeat count.
|
||||
_SAMPLE_INPUT_CHARS = 120
|
||||
_DIGEST_MAX_INDICES = 20
|
||||
|
||||
|
||||
@dataclass
|
||||
class LoopPattern:
|
||||
@@ -290,7 +295,7 @@ def detect_loops(
|
||||
LoopPattern(
|
||||
tool=calls[0].name,
|
||||
signature=_fuzzy_signature(signatures[key]),
|
||||
sample_input=calls[0].input_summary[:120],
|
||||
sample_input=calls[0].input_summary[:_SAMPLE_INPUT_CHARS],
|
||||
count=count,
|
||||
is_error_loop=is_error_loop,
|
||||
wasted_tokens=wasted,
|
||||
@@ -302,15 +307,19 @@ def detect_loops(
|
||||
return loops
|
||||
|
||||
|
||||
def format_loops_for_digest(loops: list[LoopPattern]) -> str:
|
||||
def format_loops_for_digest(loops: list[LoopPattern], max_chars: int | None = None) -> str:
|
||||
"""Render detected loops as a high-priority digest section for the LLM.
|
||||
|
||||
Returns "" when there are no loops so the digest is unchanged in the
|
||||
common case.
|
||||
Loops are rendered in the given order (detect_loops sorts them costliest
|
||||
first). With *max_chars*, rendering stops at the first loop that would
|
||||
take the section past that many chars, so the costliest ones are kept.
|
||||
|
||||
Returns "" when there are no loops, or none fits, so the digest is
|
||||
unchanged in the common case.
|
||||
"""
|
||||
if not loops:
|
||||
return ""
|
||||
lines = [
|
||||
header = [
|
||||
"=== Detected Loops (HIGHEST PRIORITY) ===",
|
||||
(
|
||||
"These tool-call patterns REPEATED within a session — the most "
|
||||
@@ -321,14 +330,21 @@ def format_loops_for_digest(loops: list[LoopPattern]) -> str:
|
||||
),
|
||||
"",
|
||||
]
|
||||
used = sum(len(ln) + 1 for ln in header) # each line plus its newline
|
||||
entries: list[str] = []
|
||||
for lp in loops:
|
||||
lines.append(
|
||||
f'- [{lp.kind}] {lp.tool}: "{lp.sample_input}" '
|
||||
entry = (
|
||||
f'- [{lp.kind}] {lp.tool}: "{lp.sample_input[:_SAMPLE_INPUT_CHARS]}" '
|
||||
f"repeated {lp.count}x, ~{lp.wasted_tokens:,} tokens wasted "
|
||||
f"(messages {lp.msg_indices})"
|
||||
f"(messages {lp.msg_indices[:_DIGEST_MAX_INDICES]})"
|
||||
)
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
used += len(entry) + 1
|
||||
if max_chars is not None and used > max_chars:
|
||||
break
|
||||
entries.append(entry)
|
||||
if not entries:
|
||||
return ""
|
||||
return "\n".join([*header, *entries, ""])
|
||||
|
||||
|
||||
def _signature_tokens(signature: str) -> set[str]:
|
||||
|
||||
@@ -475,11 +475,22 @@ class TestSessionAnalyzer:
|
||||
assert result.recommendations == []
|
||||
assert result.analysis_error == "API key not set"
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"error",
|
||||
[
|
||||
"`claude -p` failed (exit 1):\nPrompt is too long",
|
||||
# What codex exec printed on a real learn run: it refuses a prompt over 2**20 chars.
|
||||
"`codex exec` failed (exit 1):\nError: turn/start: turn/start failed: Input exceeds "
|
||||
"the maximum length of 1048576 characters. (code -32602), data: "
|
||||
'{"input_error_code":"input_too_large","max_chars":1048576,"actual_chars":1786526}',
|
||||
],
|
||||
ids=["claude", "codex"],
|
||||
)
|
||||
@patch("headroom.learn.analyzer._call_llm")
|
||||
def test_retries_smaller_digest_on_prompt_too_long(self, mock_call_llm: MagicMock):
|
||||
def test_retries_smaller_digest_on_prompt_too_long(self, mock_call_llm: MagicMock, error):
|
||||
"""A context-overflow error rebuilds the digest at half the budget and retries."""
|
||||
mock_call_llm.side_effect = [
|
||||
RuntimeError("`claude -p` failed (exit 1):\nPrompt is too long"),
|
||||
RuntimeError(error),
|
||||
{"context_file_rules": [], "memory_file_rules": []},
|
||||
]
|
||||
|
||||
|
||||
@@ -26,6 +26,7 @@ from headroom.learn.fixtures import (
|
||||
)
|
||||
from headroom.learn.loops import (
|
||||
_IDENTITY_FIELDS,
|
||||
LoopPattern,
|
||||
_canonical_signature,
|
||||
_identity_input,
|
||||
_signature_tokens,
|
||||
@@ -37,6 +38,7 @@ from headroom.learn.models import (
|
||||
Recommendation,
|
||||
RecommendationTarget,
|
||||
SessionData,
|
||||
SessionEvent,
|
||||
ToolCall,
|
||||
)
|
||||
|
||||
@@ -150,6 +152,119 @@ class TestDigestSurfacesLoops:
|
||||
digest = _build_digest(_project(), [one_off_error_session()])
|
||||
assert "Detected Loops" not in digest
|
||||
|
||||
@pytest.mark.parametrize("max_tokens", [80_000, 10_000])
|
||||
@pytest.mark.parametrize(
|
||||
("tool", "sample"),
|
||||
[
|
||||
("Bash", "x" * 120),
|
||||
("Grep", "x" * 2000), # a long search pattern
|
||||
("Glob", "src/" * 500 + "*.py"), # a long path
|
||||
],
|
||||
ids=["bash", "long-grep-pattern", "long-glob-path"],
|
||||
)
|
||||
def test_loop_section_stays_inside_the_digest_budget(self, tool, sample, max_tokens):
|
||||
# Rendered whole, thousands of loops with hundreds of indices each made
|
||||
# a 1.79M-char prompt, over Codex's 1,048,576-char input cap.
|
||||
loops = [
|
||||
LoopPattern(tool, "s", sample, 300, False, 1_000_000 - i, list(range(300)))
|
||||
for i in range(2000)
|
||||
]
|
||||
digest = _build_digest(_project(), [], loops=loops, max_tokens=max_tokens)
|
||||
section = digest[digest.index("=== Detected Loops") :]
|
||||
assert len(section) <= max_tokens # a quarter of the budget's chars
|
||||
# The costliest loops are the ones kept, and the caller's list is intact.
|
||||
assert "~1,000,000 tokens wasted" in section
|
||||
assert "~999,000 tokens wasted" not in section
|
||||
assert loops[0].sample_input == sample
|
||||
assert len(loops[0].msg_indices) == 300
|
||||
|
||||
@pytest.mark.parametrize("max_tokens", [80_000, 10_000])
|
||||
def test_loops_fit_beside_a_large_prior_patterns_section(self, tmp_path, max_tokens):
|
||||
# The event stream cannot trim sections placed ahead of it, so the
|
||||
# loops give way to the prior patterns, which are never cut.
|
||||
bullets = "".join(f"- rule {i:06d} {'y' * 80}\n" for i in range(max_tokens * 4 // 100))
|
||||
claude_md = tmp_path / "CLAUDE.md"
|
||||
claude_md.write_text(
|
||||
f"<!-- headroom:learn:start -->\n{bullets}<!-- headroom:learn:end -->\n"
|
||||
)
|
||||
project = ProjectInfo(
|
||||
name="proj",
|
||||
project_path=tmp_path,
|
||||
data_path=tmp_path / "data",
|
||||
context_file=claude_md,
|
||||
)
|
||||
loops = [
|
||||
LoopPattern("Grep", "s", "x" * 2000, 300, False, 1_000_000 - i, list(range(300)))
|
||||
for i in range(25)
|
||||
]
|
||||
digest = _build_digest(project, [], loops=loops, max_tokens=max_tokens)
|
||||
assert bullets.splitlines()[-1] in digest
|
||||
assert "~1,000,000 tokens wasted" in digest
|
||||
assert len(digest) <= max_tokens * 4
|
||||
|
||||
@pytest.mark.parametrize("with_loops", [False, True], ids=["no-loops", "loops"])
|
||||
@pytest.mark.parametrize("with_events", [False, True], ids=["tool-calls", "events"])
|
||||
@pytest.mark.parametrize("max_tokens", [80_000, 10_000])
|
||||
def test_whole_digest_stays_inside_the_budget(self, max_tokens, with_events, with_loops):
|
||||
# Far more session text than the budget: every line and its newline
|
||||
# counts, and none goes in that would leave no room for the markers.
|
||||
sessions = []
|
||||
for s in range(5):
|
||||
calls = [
|
||||
ToolCall(
|
||||
name="Bash",
|
||||
tool_call_id=f"{s}-{i}",
|
||||
input_data={"command": f"run {i}"},
|
||||
output="e" * 300,
|
||||
is_error=True,
|
||||
msg_index=i,
|
||||
)
|
||||
for i in range(3000)
|
||||
]
|
||||
events = (
|
||||
[SessionEvent(type="tool_call", msg_index=c.msg_index, tool_call=c) for c in calls]
|
||||
if with_events
|
||||
else []
|
||||
)
|
||||
sessions.append(SessionData(session_id=f"s{s}", tool_calls=calls, events=events))
|
||||
loops = (
|
||||
[
|
||||
LoopPattern("Grep", "s", "x" * 2000, 300, False, 1_000_000 - i, list(range(300)))
|
||||
for i in range(2000)
|
||||
]
|
||||
if with_loops
|
||||
else []
|
||||
)
|
||||
digest = _build_digest(_project(), sessions, loops=loops, max_tokens=max_tokens)
|
||||
assert len(digest) <= max_tokens * 4
|
||||
assert "remaining" in digest and "sessions truncated" in digest
|
||||
|
||||
@pytest.mark.parametrize("with_events", [False, True], ids=["tool-calls", "events"])
|
||||
@pytest.mark.parametrize("max_tokens", [80_000, 10_000])
|
||||
def test_many_tiny_sessions_stay_inside_the_budget(self, max_tokens, with_events):
|
||||
# Thousands of kept sessions, each closed by a blank line whose newline
|
||||
# counts too; the marker reserve alone would absorb only ~100 of them.
|
||||
sessions = []
|
||||
for s in range(20_000):
|
||||
call = ToolCall(
|
||||
name="Bash",
|
||||
tool_call_id=str(s),
|
||||
input_data={"command": "ls"},
|
||||
output="ok",
|
||||
is_error=False,
|
||||
msg_index=0,
|
||||
)
|
||||
events = [SessionEvent(type="tool_call", msg_index=0, tool_call=call)]
|
||||
sessions.append(
|
||||
SessionData(
|
||||
session_id=f"s{s}", tool_calls=[call], events=events if with_events else []
|
||||
)
|
||||
)
|
||||
digest = _build_digest(_project(), sessions, loops=[], max_tokens=max_tokens)
|
||||
assert len(digest) <= max_tokens * 4
|
||||
assert digest.count("=== Session ") > 500
|
||||
assert "sessions truncated" in digest
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# apply_loop_weighting
|
||||
|
||||
Reference in New Issue
Block a user