mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description
Two spots in the `learn` analyzer crash on JSON a model can
realistically emit. Both parse fully model-controlled output.
**1. `_strip_fenced_json` returns a non-object instead of raising.** It
is documented to return a dict and to raise `json.JSONDecodeError` when
no candidate is an object, but its final fallback was `result: dict =
json.loads(text); return result`. `json.loads` raises for invalid JSON,
but for valid JSON that is not an object (a bare array, string, or
number) it returns a non-dict. The `-> dict`-annotated value is then a
list/str, and all three callers do `.get(...)` on it, raising
`AttributeError` instead of the clean `JSONDecodeError` they already
handle.
**2. `_parse_llm_response` crashes on null rule lists/fields.** It read
`raw.get("context_file_rules", [])` / `raw.get("memory_file_rules", [])`
and `rule.get("section", "").strip()` / `rule.get("content",
"").strip()`. `.get(key, default)` only falls back when the key is
absent, so a present-but-null `"context_file_rules": null` made `for
rule in None` raise `TypeError`, and a null `section`/`content` made
`None.strip()` raise `AttributeError`.
## Fix
- `_strip_fenced_json`: reject a non-dict fallback result explicitly
(`raise json.JSONDecodeError(...)`), restoring the documented contract.
The lenient first-brace..last-brace slice still recovers a single object
a model wrapped in an array; only genuinely object-less JSON raises.
- `_parse_llm_response`: coerce the rule lists and the
`section`/`content` fields with `or` so a null is treated like an absent
value. Valid rules parse unchanged.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/learn/analyzer.py`: raise `JSONDecodeError` when
`_strip_fenced_json`'s fallback parse yields a non-dict; `or`-coerce
null rule lists and null `section`/`content` in `_parse_llm_response`.
- `tests/test_learn/test_analyzer.py`: regressions for non-object JSON
raising (and array-wrapped-object still extracted), and for null rule
lists / null fields being tolerated with valid rules still parsed.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_learn/test_analyzer.py::TestStripFencedJson tests/test_learn/test_analyzer.py::TestLLMResponseParser -q
# all pass; each new test fails with the corresponding fix reverted
# (JSONDecodeError contract / 'NoneType' object is not iterable / has no attribute 'strip')
$ uvx ruff@0.15.17 check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/learn/analyzer.py
Success: no issues found in 1 source file
```
Note: one unrelated digest-builder test
(`TestDigestBuilder::test_includes_project_info`) fails in my local
Windows venv because it hardcodes a `/tmp/...` path; it fails
identically on a clean `main` and is not touched by this change.
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: called the real `_strip_fenced_json` with
object-less JSON (`[1,2,3]`, `"str"`, `42`, ...) and valid/extractable
objects, and the real `_parse_llm_response` with `{"context_file_rules":
null}`, a rule with `section: null`, and a valid rule; then reverted
each fix and re-ran.
- Observed result: with the fixes the object-less inputs raise
`JSONDecodeError`, the null shapes yield an empty recommendation list,
and valid inputs parse as before; with each fix reverted the respective
input raises (`AttributeError: 'list' object has no attribute 'get'`,
`TypeError: 'NoneType' object is not iterable`, `AttributeError:
'NoneType' object has no attribute 'strip'`). Ran against the actual
module.
- Not tested: a live `headroom learn` run where the backing model
returns these shapes.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2471"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
### Maintainer current-main requalification
Exact head `27fa31eac507413415a00b54e29679bef08eabe9`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.
Clean current upstream/main0ddaa56c integration; production
_strip_fenced_json/_parse_llm_response/_as_str ASTs are identical to
contributor published head. Main-relative scope remains two files. Small
maintainer test-only repair uses str(project.project_path) instead of a
POSIX-only literal in existing digest assertion; source and test fixture
data unchanged. Initial full analyzer run109pass/1fail and separate
clean-main failure independently establish inherited Windows path
expectation before repair. After repair399tests pass,11skip,1warning
(all110analyzer cases plus289incoming-main regressions). Two changed
files pass Rufflint/format,candidate offline lock and whitespace pass.
Complete native inputset/allrecordedhashes/artifactchecksum verified
before existing3982DLL reuse; no fresh build.
Independent paired parser probes: branch passes both truthy non-string
section and objectless JSON contract cases; clean main fails with
AttributeError and missing JSONDecodeError respectively. These are
controlled parser executions, not live LLM quality/provider behavior.
Outstanding Devin truthy non-list rule-container report also
independently reproduced on BOTH branch and clean main:
context_file_rules:42 and memory_file_rules:42 raise TypeError in both.
This is an inherited limitation, not a PR-introduced regression or
grounds to withhold this narrower fix. Do not claim arbitrary-container
safety. The later author suggestion to join list-valued rule content
changes intended normalization semantics; not selected or implemented by
maintainer. Existing _as_str drops non-string fields, preserving
contributor intent.
Read author replies/current discussions and unresolved thread. No new
unreviewed external findings at prepublication audit. Source/test diff
contains no workflow/setup/dependency/native/CLI paths; central
publication gate passed. Exact published-head Actions/rollup, current
main ancestry/mergeability and fresh discussion remain before final
approval. Maintainers handle CI; no PR merged.
Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
1505 lines
62 KiB
Python
1505 lines
62 KiB
Python
"""Tests for session analyzer — digest builder and LLM-based analysis."""
|
||
|
||
import io
|
||
import json
|
||
import subprocess
|
||
import time
|
||
from pathlib import Path
|
||
from unittest.mock import MagicMock, patch
|
||
|
||
import pytest
|
||
|
||
from headroom.learn.analyzer import (
|
||
SessionAnalyzer,
|
||
_build_digest,
|
||
_call_cli_llm,
|
||
_call_llm,
|
||
_detect_default_model,
|
||
_output_snippet,
|
||
_parse_llm_response,
|
||
_resolve_timeout_secs,
|
||
_strip_fenced_json,
|
||
)
|
||
from headroom.learn.models import (
|
||
ErrorCategory,
|
||
ProjectInfo,
|
||
RecommendationTarget,
|
||
SessionData,
|
||
SessionEvent,
|
||
ToolCall,
|
||
)
|
||
|
||
|
||
def _project() -> ProjectInfo:
|
||
return ProjectInfo(
|
||
name="test-project",
|
||
project_path=Path("/tmp/test-project"),
|
||
data_path=Path("/tmp/test-data"),
|
||
)
|
||
|
||
|
||
def _tc(
|
||
name: str = "Bash",
|
||
input_data: dict | None = None,
|
||
output: str = "ok",
|
||
is_error: bool = False,
|
||
error_category: ErrorCategory = ErrorCategory.UNKNOWN,
|
||
msg_index: int = 0,
|
||
output_bytes: int = 0,
|
||
) -> ToolCall:
|
||
return ToolCall(
|
||
name=name,
|
||
tool_call_id=f"tc_{msg_index}",
|
||
input_data=input_data or {},
|
||
output=output,
|
||
is_error=is_error,
|
||
error_category=error_category,
|
||
msg_index=msg_index,
|
||
output_bytes=output_bytes or len(output),
|
||
)
|
||
|
||
|
||
# =============================================================================
|
||
# Digest Builder Tests
|
||
# =============================================================================
|
||
|
||
|
||
class TestDigestBuilder:
|
||
def test_includes_project_info(self):
|
||
project = _project()
|
||
sessions = [SessionData(session_id="s1", tool_calls=[_tc()])]
|
||
digest = _build_digest(project, sessions)
|
||
assert "test-project" in digest
|
||
assert str(project.project_path) in digest
|
||
|
||
def test_includes_session_stats(self):
|
||
sessions = [
|
||
SessionData(
|
||
session_id="abc123",
|
||
tool_calls=[_tc(msg_index=0), _tc(msg_index=1, is_error=True, output="Error!")],
|
||
)
|
||
]
|
||
digest = _build_digest(_project(), sessions)
|
||
assert "abc123" in digest
|
||
assert "2 calls" in digest
|
||
assert "1 failure" in digest
|
||
|
||
def test_includes_tool_call_details(self):
|
||
sessions = [
|
||
SessionData(
|
||
session_id="s1",
|
||
tool_calls=[
|
||
_tc(
|
||
name="Read",
|
||
input_data={"file_path": "/src/foo.py"},
|
||
output="contents",
|
||
msg_index=0,
|
||
),
|
||
_tc(
|
||
name="Bash",
|
||
input_data={"command": "python3 run.py"},
|
||
output="ModuleNotFoundError",
|
||
is_error=True,
|
||
error_category=ErrorCategory.MODULE_NOT_FOUND,
|
||
msg_index=1,
|
||
),
|
||
],
|
||
)
|
||
]
|
||
digest = _build_digest(_project(), sessions)
|
||
assert "/src/foo.py" in digest
|
||
assert "python3 run.py" in digest
|
||
assert "ERROR" in digest
|
||
assert "ModuleNotFoundError" in digest
|
||
|
||
def test_includes_user_messages(self):
|
||
tc = _tc(msg_index=0)
|
||
events = [
|
||
SessionEvent(type="tool_call", msg_index=0, tool_call=tc),
|
||
SessionEvent(type="user_message", msg_index=1, text="Use uv run instead"),
|
||
]
|
||
sessions = [SessionData(session_id="s1", tool_calls=[tc], events=events)]
|
||
digest = _build_digest(_project(), sessions)
|
||
assert "USER:" in digest
|
||
assert "Use uv run instead" in digest
|
||
|
||
def test_includes_subagent_summaries(self):
|
||
events = [
|
||
SessionEvent(
|
||
type="agent_summary",
|
||
msg_index=0,
|
||
agent_tool_count=150,
|
||
agent_tokens=60000,
|
||
agent_prompt="Explore all test files",
|
||
),
|
||
]
|
||
sessions = [SessionData(session_id="s1", events=events)]
|
||
digest = _build_digest(_project(), sessions)
|
||
assert "SUBAGENT" in digest
|
||
assert "150 tool calls" in digest
|
||
assert "Explore all test files" in digest
|
||
|
||
def test_includes_interruptions(self):
|
||
events = [
|
||
SessionEvent(
|
||
type="interruption",
|
||
msg_index=0,
|
||
text="[Request interrupted by user]",
|
||
),
|
||
]
|
||
sessions = [SessionData(session_id="s1", events=events)]
|
||
digest = _build_digest(_project(), sessions)
|
||
assert "INTERRUPTED" in digest
|
||
|
||
def test_empty_sessions(self):
|
||
digest = _build_digest(_project(), [])
|
||
assert "0 sessions" in digest or "test-project" in digest
|
||
|
||
def test_long_error_output_preserves_tail_root_cause(self):
|
||
# A traceback's diagnosis lives at the tail; a head-only slice would drop it (#2590).
|
||
traceback = (
|
||
"Traceback (most recent call last):\n"
|
||
+ "\n".join(
|
||
f' File "mod{i}.py", line {i}, in fn{i}\n call_{i}()' for i in range(40)
|
||
)
|
||
+ "\nKeyError: 'the-actual-root-cause'"
|
||
)
|
||
sessions = [
|
||
SessionData(
|
||
session_id="s1",
|
||
tool_calls=[
|
||
_tc(
|
||
name="Bash",
|
||
output=traceback,
|
||
is_error=True,
|
||
error_category=ErrorCategory.UNKNOWN,
|
||
msg_index=0,
|
||
)
|
||
],
|
||
)
|
||
]
|
||
digest = _build_digest(_project(), sessions)
|
||
assert "Traceback" in digest
|
||
assert "KeyError: 'the-actual-root-cause'" in digest
|
||
assert "…" in digest
|
||
|
||
def test_short_error_output_not_truncated(self):
|
||
short = "ModuleNotFoundError: No module named 'foo'"
|
||
sessions = [
|
||
SessionData(
|
||
session_id="s1",
|
||
tool_calls=[
|
||
_tc(
|
||
name="Bash",
|
||
output=short,
|
||
is_error=True,
|
||
error_category=ErrorCategory.MODULE_NOT_FOUND,
|
||
msg_index=0,
|
||
)
|
||
],
|
||
)
|
||
]
|
||
digest = _build_digest(_project(), sessions)
|
||
assert short in digest
|
||
assert "…" not in digest
|
||
|
||
|
||
# =============================================================================
|
||
# Prior Patterns Injection Tests
|
||
# =============================================================================
|
||
|
||
|
||
_MARKER_BLOCK = (
|
||
"<!-- headroom:learn:start -->\n"
|
||
"## Headroom Learned Patterns\n"
|
||
"*Auto-generated by `headroom learn` on 2026-04-01 — do not edit manually*\n"
|
||
"\n"
|
||
"### Large Files\n"
|
||
"- `src/App.tsx` is very large (~40k tokens) — use offset/limit reads\n"
|
||
"- `src/lib.rs` frequently exceeds 10k tokens\n"
|
||
"\n"
|
||
"<!-- headroom:learn:end -->"
|
||
)
|
||
|
||
|
||
def _project_with_files(
|
||
tmp_path: Path, claude_md_text: str | None, memory_md_text: str | None
|
||
) -> ProjectInfo:
|
||
"""Build a ProjectInfo pointing at temp CLAUDE.md / MEMORY.md files."""
|
||
proj_dir = tmp_path / "proj"
|
||
proj_dir.mkdir()
|
||
data_dir = tmp_path / "data"
|
||
(data_dir / "memory").mkdir(parents=True)
|
||
|
||
context_file: Path | None = None
|
||
if claude_md_text is not None:
|
||
context_file = proj_dir / "CLAUDE.md"
|
||
context_file.write_text(claude_md_text)
|
||
|
||
memory_file: Path | None = None
|
||
if memory_md_text is not None:
|
||
memory_file = data_dir / "memory" / "MEMORY.md"
|
||
memory_file.write_text(memory_md_text)
|
||
|
||
return ProjectInfo(
|
||
name="proj",
|
||
project_path=proj_dir,
|
||
data_path=data_dir,
|
||
context_file=context_file,
|
||
memory_file=memory_file,
|
||
)
|
||
|
||
|
||
class TestPriorPatternsInjection:
|
||
"""The digest should include the prior marker block so the LLM can emit
|
||
COMPLETE updated sections instead of condensed deltas that reference
|
||
now-dropped siblings (the "X is also large — same rule as Y, Z" bug)."""
|
||
|
||
def test_digest_includes_prior_block_from_claude_md(self, tmp_path):
|
||
project = _project_with_files(
|
||
tmp_path, claude_md_text=f"# Project\n\n{_MARKER_BLOCK}\n", memory_md_text=None
|
||
)
|
||
digest = _build_digest(project, [])
|
||
assert "Prior Learned Patterns" in digest
|
||
assert "### Large Files" in digest
|
||
assert "App.tsx" in digest
|
||
|
||
def test_digest_includes_prior_block_from_memory_md(self, tmp_path):
|
||
project = _project_with_files(
|
||
tmp_path, claude_md_text=None, memory_md_text=f"{_MARKER_BLOCK}\n"
|
||
)
|
||
digest = _build_digest(project, [])
|
||
assert "Prior Learned Patterns" in digest
|
||
assert "MEMORY.md" in digest
|
||
assert "### Large Files" in digest
|
||
|
||
def test_digest_omits_section_when_no_files_exist(self, tmp_path):
|
||
project = _project_with_files(tmp_path, claude_md_text=None, memory_md_text=None)
|
||
digest = _build_digest(project, [])
|
||
assert "Prior Learned Patterns" not in digest
|
||
assert "<!-- headroom:learn" not in digest
|
||
|
||
def test_digest_omits_section_when_file_has_no_marker_block(self, tmp_path):
|
||
"""CLAUDE.md exists but has no headroom block → no prior section emitted."""
|
||
project = _project_with_files(
|
||
tmp_path,
|
||
claude_md_text="# Project\n\nJust a regular readme, no headroom block.\n",
|
||
memory_md_text=None,
|
||
)
|
||
digest = _build_digest(project, [])
|
||
assert "Prior Learned Patterns" not in digest
|
||
|
||
def test_digest_surfaces_both_files_when_both_present(self, tmp_path):
|
||
project = _project_with_files(
|
||
tmp_path,
|
||
claude_md_text=f"# Project\n\n{_MARKER_BLOCK}\n",
|
||
memory_md_text=f"{_MARKER_BLOCK}\n",
|
||
)
|
||
digest = _build_digest(project, [])
|
||
assert digest.count("### Large Files") >= 2 # once per file
|
||
assert "CLAUDE.md" in digest
|
||
assert "MEMORY.md" in digest
|
||
|
||
@patch("headroom.learn.analyzer._call_llm")
|
||
def test_analyze_passes_prior_block_through_to_llm(self, mock_call_llm: MagicMock, tmp_path):
|
||
"""End-to-end: SessionAnalyzer.analyze() → _call_llm receives digest
|
||
containing the prior marker block content."""
|
||
mock_call_llm.return_value = {"context_file_rules": [], "memory_file_rules": []}
|
||
project = _project_with_files(
|
||
tmp_path, claude_md_text=f"# Project\n\n{_MARKER_BLOCK}\n", memory_md_text=None
|
||
)
|
||
sessions = [
|
||
SessionData(
|
||
session_id="s1",
|
||
tool_calls=[_tc(msg_index=0, is_error=True, output="error")],
|
||
)
|
||
]
|
||
|
||
SessionAnalyzer(model="test-model").analyze(project, sessions)
|
||
|
||
mock_call_llm.assert_called_once()
|
||
digest_arg = mock_call_llm.call_args[0][0]
|
||
assert "Prior Learned Patterns" in digest_arg
|
||
assert "App.tsx" in digest_arg
|
||
|
||
|
||
# =============================================================================
|
||
# LLM Response Parser Tests
|
||
# =============================================================================
|
||
|
||
|
||
class TestLLMResponseParser:
|
||
def test_parses_context_file_rules(self):
|
||
raw = {
|
||
"context_file_rules": [
|
||
{
|
||
"section": "Environment",
|
||
"content": "- Use `uv run python` instead of `python3`",
|
||
"estimated_tokens_saved": 800,
|
||
"evidence_count": 5,
|
||
}
|
||
],
|
||
"memory_file_rules": [],
|
||
}
|
||
recs = _parse_llm_response(raw)
|
||
assert len(recs) == 1
|
||
assert recs[0].target == RecommendationTarget.CONTEXT_FILE
|
||
assert recs[0].section == "Environment"
|
||
assert "uv run python" in recs[0].content
|
||
assert recs[0].estimated_tokens_saved == 800
|
||
assert recs[0].evidence_count == 5
|
||
|
||
def test_parses_memory_file_rules(self):
|
||
raw = {
|
||
"context_file_rules": [],
|
||
"memory_file_rules": [
|
||
{
|
||
"section": "User Preferences",
|
||
"content": "- Do not auto-execute curl commands",
|
||
"estimated_tokens_saved": 500,
|
||
"evidence_count": 3,
|
||
}
|
||
],
|
||
}
|
||
recs = _parse_llm_response(raw)
|
||
assert len(recs) == 1
|
||
assert recs[0].target == RecommendationTarget.MEMORY_FILE
|
||
assert "curl" in recs[0].content
|
||
|
||
def test_sorts_by_token_savings(self):
|
||
raw = {
|
||
"context_file_rules": [
|
||
{
|
||
"section": "Paths",
|
||
"content": "- Use correct paths",
|
||
"estimated_tokens_saved": 200,
|
||
"evidence_count": 2,
|
||
},
|
||
{
|
||
"section": "Environment",
|
||
"content": "- Use uv",
|
||
"estimated_tokens_saved": 1000,
|
||
"evidence_count": 5,
|
||
},
|
||
],
|
||
"memory_file_rules": [],
|
||
}
|
||
recs = _parse_llm_response(raw)
|
||
assert recs[0].estimated_tokens_saved == 1000
|
||
assert recs[1].estimated_tokens_saved == 200
|
||
|
||
def test_handles_missing_fields(self):
|
||
raw = {
|
||
"context_file_rules": [
|
||
{"section": "Env", "content": "- stuff"},
|
||
{"section": "", "content": ""}, # should be skipped
|
||
{"not_a_real_field": True}, # should be skipped
|
||
],
|
||
"memory_file_rules": [],
|
||
}
|
||
recs = _parse_llm_response(raw)
|
||
assert len(recs) == 1
|
||
|
||
def test_handles_empty_response(self):
|
||
recs = _parse_llm_response({})
|
||
assert recs == []
|
||
|
||
def test_handles_non_dict_entries(self):
|
||
raw = {"context_file_rules": ["not a dict", 42], "memory_file_rules": []}
|
||
recs = _parse_llm_response(raw)
|
||
assert recs == []
|
||
|
||
def test_handles_null_rule_lists_and_fields(self):
|
||
# The model controls this JSON. A present-but-null rules list or a null
|
||
# section/content must be treated like an absent value, not crash the
|
||
# iteration / `.strip()`.
|
||
assert _parse_llm_response({"context_file_rules": None, "memory_file_rules": None}) == []
|
||
assert (
|
||
_parse_llm_response({"context_file_rules": [{"section": None, "content": "x"}]}) == []
|
||
)
|
||
assert _parse_llm_response({"memory_file_rules": [{"section": "s", "content": None}]}) == []
|
||
|
||
# A truthy non-string section/content (number, list, object) is also
|
||
# model-controlled and must not crash `.strip()`; it is dropped like null.
|
||
assert _parse_llm_response({"context_file_rules": [{"section": 123, "content": "x"}]}) == []
|
||
assert _parse_llm_response({"context_file_rules": [{"section": "s", "content": 123}]}) == []
|
||
assert (
|
||
_parse_llm_response({"memory_file_rules": [{"section": ["s"], "content": "x"}]}) == []
|
||
)
|
||
assert (
|
||
_parse_llm_response({"memory_file_rules": [{"section": "s", "content": {"a": 1}}]})
|
||
== []
|
||
)
|
||
|
||
# A valid rule alongside the null shapes is still parsed.
|
||
recs = _parse_llm_response(
|
||
{
|
||
"context_file_rules": [
|
||
None,
|
||
{"section": "Large Files", "content": "use offset", "evidence_count": 3},
|
||
]
|
||
}
|
||
)
|
||
assert [r.section for r in recs] == ["Large Files"]
|
||
assert recs[0].evidence_count == 3
|
||
|
||
|
||
# =============================================================================
|
||
# Full Analyzer Integration Tests (mocked LLM)
|
||
# =============================================================================
|
||
|
||
|
||
class TestSessionAnalyzer:
|
||
def test_empty_sessions_no_llm_call(self):
|
||
"""No failures + no events → no LLM call, empty result."""
|
||
analyzer = SessionAnalyzer()
|
||
result = analyzer.analyze(_project(), [])
|
||
assert result.total_calls == 0
|
||
assert result.total_failures == 0
|
||
assert result.recommendations == []
|
||
|
||
@patch("headroom.learn.analyzer._call_llm")
|
||
def test_calls_llm_with_digest(self, mock_call_llm: MagicMock):
|
||
mock_call_llm.return_value = {
|
||
"context_file_rules": [
|
||
{
|
||
"section": "Environment",
|
||
"content": "- Use uv run python",
|
||
"estimated_tokens_saved": 800,
|
||
"evidence_count": 3,
|
||
}
|
||
],
|
||
"memory_file_rules": [],
|
||
}
|
||
|
||
analyzer = SessionAnalyzer(model="test-model")
|
||
sessions = [
|
||
SessionData(
|
||
session_id="s1",
|
||
tool_calls=[
|
||
_tc(msg_index=0, is_error=True, output="ModuleNotFoundError"),
|
||
_tc(msg_index=1),
|
||
],
|
||
)
|
||
]
|
||
result = analyzer.analyze(_project(), sessions)
|
||
|
||
mock_call_llm.assert_called_once()
|
||
assert result.total_calls == 2
|
||
assert result.total_failures == 1
|
||
assert len(result.recommendations) == 1
|
||
assert "uv run python" in result.recommendations[0].content
|
||
|
||
@patch("headroom.learn.analyzer._call_llm")
|
||
def test_handles_llm_failure_gracefully(self, mock_call_llm: MagicMock):
|
||
mock_call_llm.side_effect = RuntimeError("API key not set")
|
||
|
||
analyzer = SessionAnalyzer(model="test-model")
|
||
sessions = [
|
||
SessionData(
|
||
session_id="s1",
|
||
tool_calls=[_tc(msg_index=0, is_error=True, output="error")],
|
||
)
|
||
]
|
||
result = analyzer.analyze(_project(), sessions)
|
||
|
||
# Stats should still work, just no recommendations
|
||
assert result.total_calls == 1
|
||
assert result.total_failures == 1
|
||
assert result.recommendations == []
|
||
assert result.analysis_error == "API key not set"
|
||
|
||
@pytest.mark.parametrize(
|
||
"error",
|
||
[
|
||
"`claude -p` failed (exit 1):\nPrompt is too long",
|
||
# What codex exec printed on a real learn run: it refuses a prompt over 2**20 chars.
|
||
"`codex exec` failed (exit 1):\nError: turn/start: turn/start failed: Input exceeds "
|
||
"the maximum length of 1048576 characters. (code -32602), data: "
|
||
'{"input_error_code":"input_too_large","max_chars":1048576,"actual_chars":1786526}',
|
||
],
|
||
ids=["claude", "codex"],
|
||
)
|
||
@patch("headroom.learn.analyzer._call_llm")
|
||
def test_retries_smaller_digest_on_prompt_too_long(self, mock_call_llm: MagicMock, error):
|
||
"""A context-overflow error rebuilds the digest at half the budget and retries."""
|
||
mock_call_llm.side_effect = [
|
||
RuntimeError(error),
|
||
{"context_file_rules": [], "memory_file_rules": []},
|
||
]
|
||
|
||
analyzer = SessionAnalyzer(model="test-model")
|
||
sessions = [
|
||
SessionData(
|
||
session_id="s1",
|
||
tool_calls=[
|
||
# Enough volume to overflow even the halved (40k-token) budget,
|
||
# so the retry digest is visibly shorter than the first.
|
||
_tc(msg_index=i, is_error=True, output="x" * 200)
|
||
for i in range(2000)
|
||
],
|
||
)
|
||
]
|
||
result = analyzer.analyze(_project(), sessions)
|
||
|
||
assert mock_call_llm.call_count == 2
|
||
first_digest = mock_call_llm.call_args_list[0][0][0]
|
||
second_digest = mock_call_llm.call_args_list[1][0][0]
|
||
assert len(second_digest) < len(first_digest)
|
||
assert result.recommendations == []
|
||
|
||
@patch("headroom.learn.analyzer._call_llm")
|
||
def test_gives_up_when_min_digest_budget_still_too_long(self, mock_call_llm: MagicMock):
|
||
"""Persistent overflow stops at the minimum budget instead of looping forever."""
|
||
mock_call_llm.side_effect = RuntimeError("Prompt is too long")
|
||
|
||
analyzer = SessionAnalyzer(model="test-model")
|
||
sessions = [
|
||
SessionData(
|
||
session_id="s1",
|
||
tool_calls=[_tc(msg_index=0, is_error=True, output="error")],
|
||
)
|
||
]
|
||
result = analyzer.analyze(_project(), sessions)
|
||
|
||
# Budgets: 80k, 40k, 20k, 10k — then give up.
|
||
assert mock_call_llm.call_count == 4
|
||
assert result.recommendations == []
|
||
|
||
@patch("headroom.learn.analyzer._call_llm")
|
||
def test_non_overflow_failure_does_not_retry(self, mock_call_llm: MagicMock):
|
||
mock_call_llm.side_effect = RuntimeError("API key not set")
|
||
|
||
analyzer = SessionAnalyzer(model="test-model")
|
||
sessions = [
|
||
SessionData(
|
||
session_id="s1",
|
||
tool_calls=[_tc(msg_index=0, is_error=True, output="error")],
|
||
)
|
||
]
|
||
analyzer.analyze(_project(), sessions)
|
||
|
||
mock_call_llm.assert_called_once()
|
||
|
||
@patch("headroom.learn.analyzer._call_llm")
|
||
def test_passes_events_to_digest(self, mock_call_llm: MagicMock):
|
||
"""User messages and subagent events should appear in the digest."""
|
||
mock_call_llm.return_value = {"context_file_rules": [], "memory_file_rules": []}
|
||
|
||
tc = _tc(msg_index=0, is_error=True, output="error")
|
||
events = [
|
||
SessionEvent(type="tool_call", msg_index=0, tool_call=tc),
|
||
SessionEvent(type="user_message", msg_index=1, text="use venv python"),
|
||
]
|
||
sessions = [SessionData(session_id="s1", tool_calls=[tc], events=events)]
|
||
|
||
analyzer = SessionAnalyzer(model="test-model")
|
||
analyzer.analyze(_project(), sessions)
|
||
|
||
# Check that the digest passed to the LLM includes user message
|
||
call_args = mock_call_llm.call_args
|
||
digest = call_args[0][0] # first positional arg
|
||
assert "use venv python" in digest
|
||
|
||
|
||
# =============================================================================
|
||
# Model Auto-Detection
|
||
# =============================================================================
|
||
|
||
|
||
class TestDetectDefaultModel:
|
||
def test_anthropic_key(self, monkeypatch):
|
||
monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-test")
|
||
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
|
||
monkeypatch.delenv("GEMINI_API_KEY", raising=False)
|
||
assert _detect_default_model() == "claude-sonnet-4-6"
|
||
|
||
def test_openai_key(self, monkeypatch):
|
||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||
monkeypatch.setenv("OPENAI_API_KEY", "sk-test")
|
||
monkeypatch.delenv("GEMINI_API_KEY", raising=False)
|
||
assert _detect_default_model() == "gpt-4o"
|
||
|
||
def test_gemini_key(self, monkeypatch):
|
||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
|
||
monkeypatch.setenv("GEMINI_API_KEY", "test")
|
||
assert _detect_default_model() == "gemini/gemini-flash-latest"
|
||
|
||
def test_anthropic_preferred_over_openai(self, monkeypatch):
|
||
monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-test")
|
||
monkeypatch.setenv("OPENAI_API_KEY", "sk-test")
|
||
assert _detect_default_model() == "claude-sonnet-4-6"
|
||
|
||
def test_no_keys_no_cli_raises(self, monkeypatch):
|
||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
|
||
monkeypatch.delenv("GEMINI_API_KEY", raising=False)
|
||
monkeypatch.setattr("headroom.learn.analyzer.shutil.which", lambda _name: None)
|
||
|
||
with pytest.raises(RuntimeError, match="No LLM API key found"):
|
||
_detect_default_model()
|
||
|
||
def test_cli_fallback_claude(self, monkeypatch):
|
||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
|
||
monkeypatch.delenv("GEMINI_API_KEY", raising=False)
|
||
monkeypatch.setattr(
|
||
"headroom.learn.analyzer.shutil.which",
|
||
lambda name: f"/usr/bin/{name}" if name == "claude" else None,
|
||
)
|
||
assert _detect_default_model() == "claude-cli"
|
||
|
||
def test_cli_fallback_gemini(self, monkeypatch):
|
||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
|
||
monkeypatch.delenv("GEMINI_API_KEY", raising=False)
|
||
monkeypatch.setattr(
|
||
"headroom.learn.analyzer.shutil.which",
|
||
lambda name: f"/usr/bin/{name}" if name == "gemini" else None,
|
||
)
|
||
assert _detect_default_model() == "gemini-cli"
|
||
|
||
def test_cli_fallback_codex(self, monkeypatch):
|
||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
|
||
monkeypatch.delenv("GEMINI_API_KEY", raising=False)
|
||
monkeypatch.setattr(
|
||
"headroom.learn.analyzer.shutil.which",
|
||
lambda name: f"/usr/bin/{name}" if name == "codex" else None,
|
||
)
|
||
assert _detect_default_model() == "codex-cli"
|
||
|
||
def test_cli_fallback_never_auto_detects_agy(self, monkeypatch):
|
||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
|
||
monkeypatch.delenv("GEMINI_API_KEY", raising=False)
|
||
monkeypatch.delenv("HEADROOM_LEARN_CLI", raising=False)
|
||
monkeypatch.setenv("HEADROOM_LEARN_ALLOW_UNSAFE_AGY", "1")
|
||
monkeypatch.setattr(
|
||
"headroom.learn.analyzer.shutil.which",
|
||
lambda name: f"/usr/bin/{name}" if name == "agy" else None,
|
||
)
|
||
with pytest.raises(RuntimeError, match="No LLM API key found"):
|
||
_detect_default_model()
|
||
|
||
def test_cli_fallback_prefers_existing_clis_over_agy(self, monkeypatch):
|
||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
|
||
monkeypatch.delenv("GEMINI_API_KEY", raising=False)
|
||
monkeypatch.setattr(
|
||
"headroom.learn.analyzer.shutil.which",
|
||
lambda name: f"/usr/bin/{name}" if name in ("codex", "agy") else None,
|
||
)
|
||
assert _detect_default_model() == "codex-cli"
|
||
|
||
def test_api_key_preferred_over_cli(self, monkeypatch):
|
||
monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-test")
|
||
monkeypatch.setattr(
|
||
"headroom.learn.analyzer.shutil.which",
|
||
lambda name: f"/usr/bin/{name}" if name == "claude" else None,
|
||
)
|
||
assert _detect_default_model() == "claude-sonnet-4-6"
|
||
|
||
def test_env_var_selects_gemini(self, monkeypatch):
|
||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
|
||
monkeypatch.delenv("GEMINI_API_KEY", raising=False)
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI", "gemini")
|
||
assert _detect_default_model() == "gemini-cli"
|
||
|
||
def test_env_var_selects_codex(self, monkeypatch):
|
||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
|
||
monkeypatch.delenv("GEMINI_API_KEY", raising=False)
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI", "codex")
|
||
assert _detect_default_model() == "codex-cli"
|
||
|
||
def test_env_var_selects_agy_with_unsafe_opt_in(self, monkeypatch):
|
||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
|
||
monkeypatch.delenv("GEMINI_API_KEY", raising=False)
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI", "agy")
|
||
monkeypatch.setenv("HEADROOM_LEARN_ALLOW_UNSAFE_AGY", "1")
|
||
assert _detect_default_model() == "agy-cli"
|
||
|
||
def test_env_var_agy_without_unsafe_opt_in_raises(self, monkeypatch):
|
||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
|
||
monkeypatch.delenv("GEMINI_API_KEY", raising=False)
|
||
monkeypatch.delenv("HEADROOM_LEARN_ALLOW_UNSAFE_AGY", raising=False)
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI", "agy")
|
||
with pytest.raises(ValueError, match="HEADROOM_LEARN_ALLOW_UNSAFE_AGY=1"):
|
||
_detect_default_model()
|
||
|
||
def test_env_var_invalid_raises(self, monkeypatch):
|
||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
|
||
monkeypatch.delenv("GEMINI_API_KEY", raising=False)
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI", "unknown-tool")
|
||
with pytest.raises(ValueError, match="not a supported CLI"):
|
||
_detect_default_model()
|
||
|
||
def test_api_key_preferred_over_env_var(self, monkeypatch):
|
||
monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-test")
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI", "gemini")
|
||
assert _detect_default_model() == "claude-sonnet-4-6"
|
||
|
||
def test_env_var_preferred_over_auto_detect(self, monkeypatch):
|
||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
|
||
monkeypatch.delenv("GEMINI_API_KEY", raising=False)
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI", "codex")
|
||
monkeypatch.setattr(
|
||
"headroom.learn.analyzer.shutil.which",
|
||
lambda name: f"/usr/bin/{name}" if name == "claude" else None,
|
||
)
|
||
# codex selected via env var, even though claude is in PATH
|
||
assert _detect_default_model() == "codex-cli"
|
||
|
||
|
||
# =============================================================================
|
||
# CLI LLM Backend
|
||
# =============================================================================
|
||
|
||
|
||
class TestStripFencedJson:
|
||
def test_raw_json(self):
|
||
result = _strip_fenced_json('{"key": "value"}')
|
||
assert result == {"key": "value"}
|
||
|
||
def test_fenced_json(self):
|
||
raw = '```json\n{"key": "value"}\n```'
|
||
result = _strip_fenced_json(raw)
|
||
assert result == {"key": "value"}
|
||
|
||
def test_fenced_no_language_tag(self):
|
||
raw = '```\n{"key": "value"}\n```'
|
||
result = _strip_fenced_json(raw)
|
||
assert result == {"key": "value"}
|
||
|
||
def test_whitespace_padding(self):
|
||
raw = ' \n```json\n{"key": "value"}\n```\n '
|
||
result = _strip_fenced_json(raw)
|
||
assert result == {"key": "value"}
|
||
|
||
def test_invalid_json_raises(self):
|
||
with pytest.raises(json.JSONDecodeError):
|
||
_strip_fenced_json("not json at all")
|
||
|
||
def test_non_object_json_raises(self):
|
||
# A model may return valid JSON that is not an object (a bare array,
|
||
# string, number, ...). The contract is to raise JSONDecodeError so
|
||
# callers do not receive a non-dict and crash on `.get`.
|
||
for raw in ("[1, 2, 3]", '["a", "b"]', '[{"a": 1}, {"b": 2}]', '"a string"', "42", "true"):
|
||
with pytest.raises(json.JSONDecodeError):
|
||
_strip_fenced_json(raw)
|
||
|
||
def test_array_wrapping_a_single_object_is_extracted(self):
|
||
# The lenient first-{ .. last-} slice still recovers a single object a
|
||
# model wrapped in an array; only genuinely object-less JSON raises.
|
||
assert _strip_fenced_json('[{"key": "value"}]') == {"key": "value"}
|
||
|
||
def test_prose_preamble_before_fence(self):
|
||
# Models sometimes add a preamble before the fence despite being told
|
||
# to return JSON only (e.g. "Here it is:\n\n```json ...").
|
||
raw = 'The JSON is my deliverable. Here it is:\n\n```json\n{"key": "value"}\n```'
|
||
result = _strip_fenced_json(raw)
|
||
assert result == {"key": "value"}
|
||
|
||
def test_prose_around_bare_object(self):
|
||
raw = 'Sure, here you go: {"key": "value"} hope that helps!'
|
||
result = _strip_fenced_json(raw)
|
||
assert result == {"key": "value"}
|
||
|
||
def test_triple_backtick_inside_payload(self):
|
||
raw = '```json\n{"note": "run ```code``` here", "n": 1}\n```'
|
||
result = _strip_fenced_json(raw)
|
||
assert result == {"note": "run ```code``` here", "n": 1}
|
||
|
||
|
||
def _fake_claude_popen(
|
||
*,
|
||
stdout_lines: list[str],
|
||
stderr_lines: list[str] | None = None,
|
||
returncode: int = 0,
|
||
stdout_delay: float = 0.0,
|
||
) -> MagicMock:
|
||
"""Build a Popen mock factory for the streaming claude-cli path.
|
||
|
||
Returns a MagicMock that, when called as ``Popen(cmd, ...)``, yields a
|
||
fake process whose stdout/stderr behave like line-iterable text streams.
|
||
Each stdout line is sleep(*stdout_delay*)-gated to let tests simulate slow
|
||
or hung processes.
|
||
"""
|
||
if stderr_lines is None:
|
||
stderr_lines = []
|
||
|
||
def _make_iter(lines: list[str], delay: float):
|
||
def _gen():
|
||
for line in lines:
|
||
if delay:
|
||
time.sleep(delay)
|
||
yield line
|
||
|
||
return _gen()
|
||
|
||
factory = MagicMock()
|
||
|
||
def _construct(*args, **kwargs):
|
||
proc = MagicMock()
|
||
proc.stdin = io.StringIO()
|
||
proc.stdout = _make_iter(stdout_lines, stdout_delay)
|
||
proc.stderr = _make_iter(stderr_lines, 0.0)
|
||
proc.returncode = returncode
|
||
proc.wait = MagicMock(return_value=returncode)
|
||
proc.kill = MagicMock()
|
||
proc.poll = MagicMock(return_value=returncode)
|
||
return proc
|
||
|
||
factory.side_effect = _construct
|
||
return factory
|
||
|
||
|
||
def _stream_event(event_type: str, **fields) -> str:
|
||
return json.dumps({"type": event_type, **fields}) + "\n"
|
||
|
||
|
||
def _result_event(text: str) -> str:
|
||
return _stream_event("result", subtype="success", is_error=False, result=text)
|
||
|
||
|
||
class TestCallCliLlm:
|
||
def test_claude_cli_streams_and_parses_result_event(self):
|
||
stdout = [
|
||
_stream_event("system", subtype="init"),
|
||
_stream_event("assistant", message={"content": "thinking..."}),
|
||
_result_event('{"context_file_rules": [], "memory_file_rules": []}'),
|
||
]
|
||
with patch(
|
||
"headroom.learn.analyzer.subprocess.Popen", _fake_claude_popen(stdout_lines=stdout)
|
||
) as popen:
|
||
result = _call_cli_llm("test digest", "claude-cli")
|
||
assert result == {"context_file_rules": [], "memory_file_rules": []}
|
||
cmd = popen.call_args[0][0]
|
||
assert cmd == [
|
||
"claude",
|
||
"-p",
|
||
"--output-format",
|
||
"stream-json",
|
||
"--verbose",
|
||
"--include-partial-messages",
|
||
"--tools",
|
||
"",
|
||
"--strict-mcp-config",
|
||
"--settings",
|
||
'{"disableAllHooks":true}',
|
||
]
|
||
|
||
def test_claude_cli_progress_callback_is_throttled(self):
|
||
stdout = [
|
||
_stream_event("system", subtype="init"),
|
||
*[_stream_event("assistant", message={"content": "..."}) for _ in range(5)],
|
||
_result_event('{"context_file_rules": [], "memory_file_rules": []}'),
|
||
]
|
||
progress: list[str] = []
|
||
with patch(
|
||
"headroom.learn.analyzer.subprocess.Popen", _fake_claude_popen(stdout_lines=stdout)
|
||
):
|
||
result = _call_cli_llm("test digest", "claude-cli", on_progress=progress.append)
|
||
assert result == {"context_file_rules": [], "memory_file_rules": []}
|
||
# All 6 progress-worthy events arrive well within one 3s throttle
|
||
# window, so only the first ("session started") should be echoed.
|
||
assert progress == ["session started"]
|
||
|
||
def test_claude_cli_progress_callback_reports_spaced_events(self, monkeypatch):
|
||
monkeypatch.setattr("headroom.learn.analyzer._PROGRESS_THROTTLE_SECS", 0.01)
|
||
stdout = [
|
||
_stream_event("system", subtype="init"),
|
||
_stream_event("assistant", message={"content": "thinking..."}),
|
||
_stream_event("stream_event", event={"type": "content_block_delta"}),
|
||
_stream_event("user", message={"content": []}),
|
||
_result_event('{"context_file_rules": [], "memory_file_rules": []}'),
|
||
]
|
||
progress: list[str] = []
|
||
with patch(
|
||
"headroom.learn.analyzer.subprocess.Popen",
|
||
_fake_claude_popen(stdout_lines=stdout, stdout_delay=0.03),
|
||
):
|
||
result = _call_cli_llm("test digest", "claude-cli", on_progress=progress.append)
|
||
assert result == {"context_file_rules": [], "memory_file_rules": []}
|
||
assert progress[0] == "session started"
|
||
assert progress[1].startswith("assistant responding, ")
|
||
assert progress[2].startswith("assistant responding, ") # stream_event maps like assistant
|
||
assert progress[3].startswith("tool running, ")
|
||
assert len(progress) == 4 # the terminal "result" event is never echoed as progress
|
||
|
||
def test_claude_cli_progress_callback_exception_does_not_abort_analysis(self):
|
||
stdout = [
|
||
_stream_event("system", subtype="init"),
|
||
_result_event('{"context_file_rules": [], "memory_file_rules": []}'),
|
||
]
|
||
|
||
def _boom(_detail: str) -> None:
|
||
raise RuntimeError("wrapper UI pipe closed")
|
||
|
||
with patch(
|
||
"headroom.learn.analyzer.subprocess.Popen", _fake_claude_popen(stdout_lines=stdout)
|
||
):
|
||
result = _call_cli_llm("test digest", "claude-cli", on_progress=_boom)
|
||
assert result == {"context_file_rules": [], "memory_file_rules": []}
|
||
|
||
def test_claude_cli_parses_fenced_result(self):
|
||
stdout = [
|
||
_result_event('```json\n{"context_file_rules": [], "memory_file_rules": []}\n```'),
|
||
]
|
||
with patch(
|
||
"headroom.learn.analyzer.subprocess.Popen", _fake_claude_popen(stdout_lines=stdout)
|
||
):
|
||
result = _call_cli_llm("test digest", "claude-cli")
|
||
assert result == {"context_file_rules": [], "memory_file_rules": []}
|
||
|
||
def test_claude_cli_idle_timeout_kills_hang(self, monkeypatch):
|
||
import threading as _threading
|
||
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS", "1")
|
||
|
||
# An iterator that never yields and never EOFs — simulates a hung CLI.
|
||
# The pump thread blocks in __next__, so no events reach the watchdog.
|
||
blocked = _threading.Event() # never set
|
||
|
||
class _HangingStream:
|
||
def __iter__(self):
|
||
return self
|
||
|
||
def __next__(self):
|
||
blocked.wait(timeout=10)
|
||
raise StopIteration
|
||
|
||
def _construct(*args, **kwargs):
|
||
proc = MagicMock()
|
||
proc.stdin = io.StringIO()
|
||
proc.stdout = _HangingStream()
|
||
proc.stderr = _HangingStream()
|
||
proc.returncode = 0
|
||
proc.wait = MagicMock(return_value=0)
|
||
proc.kill = MagicMock(side_effect=lambda: blocked.set())
|
||
proc.poll = MagicMock(return_value=None)
|
||
return proc
|
||
|
||
popen = MagicMock(side_effect=_construct)
|
||
with patch("headroom.learn.analyzer.subprocess.Popen", popen):
|
||
with pytest.raises(RuntimeError, match="produced no output"):
|
||
_call_cli_llm("test digest", "claude-cli")
|
||
|
||
def test_claude_cli_hard_cap_kills_continuous_chatter(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI_TIMEOUT_SECS", "1")
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS", "10")
|
||
|
||
# Continuous output every 50ms so idle never fires; hard cap should.
|
||
chatter = [_stream_event("assistant", message={"i": i}) for i in range(1000)]
|
||
popen = _fake_claude_popen(stdout_lines=chatter, stdout_delay=0.05)
|
||
with patch("headroom.learn.analyzer.subprocess.Popen", popen):
|
||
with pytest.raises(RuntimeError, match="exceeded the 1s hard cap"):
|
||
_call_cli_llm("test digest", "claude-cli")
|
||
|
||
def test_claude_cli_missing_result_event_raises(self):
|
||
stdout = [_stream_event("assistant", message={"content": "no result"})]
|
||
with patch(
|
||
"headroom.learn.analyzer.subprocess.Popen", _fake_claude_popen(stdout_lines=stdout)
|
||
):
|
||
with pytest.raises(RuntimeError, match="did not emit a final `result` event"):
|
||
_call_cli_llm("test digest", "claude-cli")
|
||
|
||
def test_claude_cli_nonzero_exit_raises(self):
|
||
popen = _fake_claude_popen(
|
||
stdout_lines=[],
|
||
stderr_lines=["Error: auth required\n"],
|
||
returncode=1,
|
||
)
|
||
with patch("headroom.learn.analyzer.subprocess.Popen", popen):
|
||
with pytest.raises(RuntimeError, match="failed.*exit 1"):
|
||
_call_cli_llm("test digest", "claude-cli")
|
||
|
||
def test_claude_cli_nonzero_exit_includes_api_error_from_stdout(self):
|
||
# Regression: `claude -p --output-format stream-json` writes NOTHING to
|
||
# stderr on an API failure. It still emits a final `result` event whose
|
||
# `result` field carries the reason, so a stderr-only message rendered as
|
||
# a bare "failed (exit 1):" with no cause for the user or the logs.
|
||
api_error = "API Error: Connection refused — a firewall or proxy may be blocking it"
|
||
stdout = [
|
||
_stream_event("system", subtype="init"),
|
||
_stream_event(
|
||
"result",
|
||
subtype="success",
|
||
is_error=False,
|
||
terminal_reason="api_error",
|
||
result=api_error,
|
||
),
|
||
]
|
||
popen = _fake_claude_popen(stdout_lines=stdout, stderr_lines=[], returncode=1)
|
||
with patch("headroom.learn.analyzer.subprocess.Popen", popen):
|
||
with pytest.raises(RuntimeError) as exc_info:
|
||
_call_cli_llm("test digest", "claude-cli")
|
||
message = str(exc_info.value)
|
||
assert "failed (exit 1)" in message
|
||
assert api_error in message
|
||
|
||
def test_claude_cli_nonzero_exit_with_no_output_says_so(self):
|
||
popen = _fake_claude_popen(stdout_lines=[], stderr_lines=[], returncode=1)
|
||
with patch("headroom.learn.analyzer.subprocess.Popen", popen):
|
||
with pytest.raises(RuntimeError) as exc_info:
|
||
_call_cli_llm("test digest", "claude-cli")
|
||
# Never a dangling colon: the message states that nothing was captured.
|
||
assert "(no output captured)" in str(exc_info.value)
|
||
|
||
def test_claude_cli_nonzero_exit_keeps_stderr_when_present(self):
|
||
popen = _fake_claude_popen(
|
||
stdout_lines=[_result_event("partial")],
|
||
stderr_lines=["Error: auth required\n"],
|
||
returncode=1,
|
||
)
|
||
with patch("headroom.learn.analyzer.subprocess.Popen", popen):
|
||
with pytest.raises(RuntimeError) as exc_info:
|
||
_call_cli_llm("test digest", "claude-cli")
|
||
message = str(exc_info.value)
|
||
assert "auth required" in message
|
||
assert "partial" in message
|
||
|
||
def test_claude_cli_unparseable_result_raises_with_context(self):
|
||
stdout = [_result_event("This is not JSON at all")]
|
||
with patch(
|
||
"headroom.learn.analyzer.subprocess.Popen", _fake_claude_popen(stdout_lines=stdout)
|
||
):
|
||
with pytest.raises(RuntimeError, match="unparseable output"):
|
||
_call_cli_llm("test digest", "claude-cli")
|
||
|
||
def test_claude_cli_unparseable_result_shows_both_ends(self):
|
||
# A head-only excerpt of a long answer is valid-looking JSON in every
|
||
# case; the reason it did not parse -- truncated mid-value, or prose
|
||
# after the closing fence -- is only visible at the end.
|
||
payload = '```json\n{"context_file_rules": [' + '{"section": "x"}, ' * 400 + "]}\n```"
|
||
payload += "\n\nLet me know if you want this in another shape!"
|
||
with patch(
|
||
"headroom.learn.analyzer.subprocess.Popen",
|
||
_fake_claude_popen(stdout_lines=[_result_event(payload)]),
|
||
):
|
||
with pytest.raises(RuntimeError) as exc_info:
|
||
_call_cli_llm("test digest", "claude-cli")
|
||
message = str(exc_info.value)
|
||
assert "unparseable output" in message
|
||
assert "```json" in message, "head is missing"
|
||
assert "another shape" in message, "tail is missing"
|
||
assert "chars omitted" in message
|
||
# Still bounded: the excerpt cannot grow with the payload.
|
||
assert len(message) < len(payload)
|
||
|
||
def test_output_snippet_leaves_short_output_alone(self):
|
||
assert _output_snippet("short") == "short"
|
||
edge = "x" * 2000
|
||
assert _output_snippet(edge) == edge
|
||
|
||
def test_claude_cli_not_installed_raises(self):
|
||
popen = MagicMock(side_effect=FileNotFoundError("No such file or directory: 'claude'"))
|
||
with patch("headroom.learn.analyzer.subprocess.Popen", popen):
|
||
with pytest.raises(RuntimeError, match="not found in PATH"):
|
||
_call_cli_llm("test digest", "claude-cli")
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_codex_cli_uses_exec(self, mock_run: MagicMock):
|
||
mock_run.return_value = MagicMock(
|
||
returncode=0,
|
||
stdout='{"context_file_rules": [], "memory_file_rules": []}',
|
||
stderr="",
|
||
)
|
||
result = _call_cli_llm("test digest", "codex-cli")
|
||
assert result == {"context_file_rules": [], "memory_file_rules": []}
|
||
cmd = mock_run.call_args[0][0]
|
||
assert cmd == ["codex", "exec", "--skip-git-repo-check"]
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_gemini_cli_uses_p_flag(self, mock_run: MagicMock):
|
||
mock_run.return_value = MagicMock(
|
||
returncode=0,
|
||
stdout='{"context_file_rules": [], "memory_file_rules": []}',
|
||
stderr="",
|
||
)
|
||
_call_cli_llm("test digest", "gemini-cli")
|
||
cmd = mock_run.call_args[0][0]
|
||
assert cmd == ["gemini", "-p"]
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_agy_cli_sends_stream_json_and_parses_result_event(
|
||
self, mock_run: MagicMock, monkeypatch
|
||
):
|
||
monkeypatch.setenv("HEADROOM_LEARN_ALLOW_UNSAFE_AGY", "1")
|
||
response = '```json\n{"context_file_rules": [], "memory_file_rules": []}\n```\n'
|
||
mock_run.return_value = MagicMock(
|
||
returncode=0,
|
||
stdout="\n".join(
|
||
[
|
||
'{"event": "init", "init": {"permission_mode": "default"}}',
|
||
'{"event": "step_update", "step_update": {"text_delta": "{"}}',
|
||
json.dumps({"event": "result", "result": {"response": response}}),
|
||
]
|
||
),
|
||
stderr="",
|
||
)
|
||
result = _call_cli_llm("test digest", "agy-cli")
|
||
assert result == {"context_file_rules": [], "memory_file_rules": []}
|
||
cmd = mock_run.call_args[0][0]
|
||
assert cmd == ["agy", "--input-format", "stream-json", "--output-format", "stream-json"]
|
||
sent = json.loads(mock_run.call_args.kwargs["input"])
|
||
assert sent["event"] == "user"
|
||
assert sent["message"]["content"].endswith("test digest")
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_agy_cli_result_with_unicode_line_separator(self, mock_run: MagicMock, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_LEARN_ALLOW_UNSAFE_AGY", "1")
|
||
response = '{"context_file_rules": [], "memory_file_rules": [], "note": "a
b"}'
|
||
mock_run.return_value = MagicMock(
|
||
returncode=0,
|
||
stdout=json.dumps(
|
||
{"event": "result", "result": {"response": response}}, ensure_ascii=False
|
||
)
|
||
+ "\n",
|
||
stderr="",
|
||
)
|
||
result = _call_cli_llm("test digest", "agy-cli")
|
||
assert result["note"] == "a
b"
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_agy_cli_without_unsafe_opt_in_does_not_run(self, mock_run: MagicMock, monkeypatch):
|
||
monkeypatch.delenv("HEADROOM_LEARN_ALLOW_UNSAFE_AGY", raising=False)
|
||
with pytest.raises(ValueError, match="HEADROOM_LEARN_ALLOW_UNSAFE_AGY=1"):
|
||
_call_cli_llm("test digest", "agy-cli")
|
||
mock_run.assert_not_called()
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_agy_cli_missing_result_event_raises(self, mock_run: MagicMock, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_LEARN_ALLOW_UNSAFE_AGY", "1")
|
||
mock_run.return_value = MagicMock(
|
||
returncode=0,
|
||
stdout='{"event": "init", "init": {}}\n',
|
||
stderr="",
|
||
)
|
||
with pytest.raises(RuntimeError, match="unparseable output"):
|
||
_call_cli_llm("test digest", "agy-cli")
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_codex_nonzero_exit_raises(self, mock_run: MagicMock):
|
||
mock_run.return_value = MagicMock(
|
||
returncode=1,
|
||
stdout="",
|
||
stderr="Error: auth required",
|
||
)
|
||
with pytest.raises(RuntimeError, match="failed.*exit 1"):
|
||
_call_cli_llm("test digest", "codex-cli")
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_codex_nonzero_exit_includes_stdout_when_stderr_empty(self, mock_run: MagicMock):
|
||
mock_run.return_value = MagicMock(
|
||
returncode=1,
|
||
stdout="stream error: rate limit exceeded",
|
||
stderr="",
|
||
)
|
||
with pytest.raises(RuntimeError) as exc_info:
|
||
_call_cli_llm("test digest", "codex-cli")
|
||
assert "rate limit exceeded" in str(exc_info.value)
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_codex_stderr_truncated_in_error(self, mock_run: MagicMock):
|
||
long_stderr = "x" * 5000
|
||
mock_run.return_value = MagicMock(
|
||
returncode=1,
|
||
stdout="",
|
||
stderr=long_stderr,
|
||
)
|
||
with pytest.raises(RuntimeError) as exc_info:
|
||
_call_cli_llm("test digest", "codex-cli")
|
||
assert long_stderr not in str(exc_info.value)
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_codex_failure_drops_prompt_echo_and_keeps_reason(self, mock_run: MagicMock):
|
||
# `codex exec` echoes the whole stdin prompt to stderr ahead of its own
|
||
# error, so a head-of-stderr excerpt is the banner plus our system prompt
|
||
# and never the reason. Shape captured from codex-cli 0.156.1.
|
||
def _codex(cmd, *, input, **kwargs):
|
||
stderr = (
|
||
"Reading prompt from stdin...\nOpenAI Codex v0.156.1\n--------\n"
|
||
"model: gpt-5\n--------\nuser\n" + input + "\n\n"
|
||
)
|
||
stderr += "ERROR: Reconnecting... 1/5\n" * 100
|
||
stderr += "ERROR: unexpected status 401 Unauthorized: Missing bearer\n"
|
||
return MagicMock(returncode=1, stdout="", stderr=stderr)
|
||
|
||
mock_run.side_effect = _codex
|
||
with pytest.raises(RuntimeError) as exc_info:
|
||
_call_cli_llm("SECRET-SESSION-DIGEST", "codex-cli")
|
||
message = str(exc_info.value)
|
||
assert message.endswith("ERROR: unexpected status 401 Unauthorized: Missing bearer")
|
||
assert "[prompt omitted]" in message
|
||
assert "SECRET-SESSION-DIGEST" not in message
|
||
assert "OpenAI Codex v0.156.1" in message
|
||
# Whole lines only, still inside the snippet cap.
|
||
assert "\nERROR: Reconnecting... 1/5\n" in message
|
||
assert len(message) < 2200
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_codex_failure_keeps_reason_behind_a_long_banner(self, mock_run: MagicMock):
|
||
# Diagnostics ahead of the echo longer than the whole snippet cap must
|
||
# not push the CLI's final error out of the message.
|
||
def _codex(cmd, *, input, **kwargs):
|
||
stderr = "WARN startup diagnostic line\n" * 100 + input + "\n"
|
||
stderr += "ERROR: unknown model gpt-x\n"
|
||
return MagicMock(returncode=1, stdout="", stderr=stderr)
|
||
|
||
mock_run.side_effect = _codex
|
||
with pytest.raises(RuntimeError) as exc_info:
|
||
_call_cli_llm("SECRET-SESSION-DIGEST", "codex-cli")
|
||
message = str(exc_info.value)
|
||
assert message.endswith("[prompt omitted]\nERROR: unknown model gpt-x")
|
||
assert message.startswith("`codex exec")
|
||
assert "SECRET-SESSION-DIGEST" not in message
|
||
assert len(message) < 2200
|
||
|
||
def test_unknown_cli_model_raises(self):
|
||
with pytest.raises(ValueError, match="Unknown CLI model"):
|
||
_call_cli_llm("test digest", "unknown-cli")
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_codex_not_installed_raises(self, mock_run: MagicMock):
|
||
mock_run.side_effect = FileNotFoundError("No such file or directory: 'codex'")
|
||
with pytest.raises(RuntimeError, match="not found in PATH"):
|
||
_call_cli_llm("test digest", "codex-cli")
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_codex_timeout_raises_runtime_error(self, mock_run: MagicMock):
|
||
mock_run.side_effect = subprocess.TimeoutExpired(cmd=["codex", "exec"], timeout=300)
|
||
with pytest.raises(RuntimeError, match="did not respond within"):
|
||
_call_cli_llm("test digest", "codex-cli")
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_codex_timeout_honors_env_override(self, mock_run: MagicMock, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI_TIMEOUT_SECS", "42")
|
||
mock_run.return_value = MagicMock(returncode=0, stdout="{}", stderr="")
|
||
_call_cli_llm("test digest", "codex-cli")
|
||
assert mock_run.call_args.kwargs["timeout"] == 42
|
||
|
||
@patch("headroom.learn.analyzer.subprocess.run")
|
||
def test_codex_unparseable_output_raises_with_context(self, mock_run: MagicMock):
|
||
mock_run.return_value = MagicMock(
|
||
returncode=0,
|
||
stdout="This is not JSON at all",
|
||
stderr="",
|
||
)
|
||
with pytest.raises(RuntimeError, match="unparseable output"):
|
||
_call_cli_llm("test digest", "codex-cli")
|
||
|
||
|
||
class TestWindowsCliShimFallback:
|
||
"""npm-installed CLIs on Windows are ``.cmd``/``.bat`` shims, not directly
|
||
executable — ``subprocess`` uses ``CreateProcess``, which (unlike a shell)
|
||
does not apply the ``PATHEXT`` extension search, so ``Popen`` raises
|
||
``FileNotFoundError`` even though the shell (and ``shutil.which``) finds
|
||
the CLI fine (issue #1624). A FileNotFoundError on Windows should trigger
|
||
one ``shutil.which``-based retry with the resolved executable path.
|
||
"""
|
||
|
||
def test_streaming_cli_retries_with_resolved_shim(self, monkeypatch):
|
||
monkeypatch.setattr("headroom.learn.analyzer.os.name", "nt")
|
||
monkeypatch.setattr(
|
||
"headroom.learn.analyzer.shutil.which",
|
||
lambda name: r"C:\npm\claude.cmd" if name == "claude" else None,
|
||
)
|
||
stdout = [_result_event('{"context_file_rules": [], "memory_file_rules": []}')]
|
||
fake_popen = _fake_claude_popen(stdout_lines=stdout)
|
||
calls: list[list[str]] = []
|
||
|
||
def _construct(cmd, *args, **kwargs):
|
||
calls.append(cmd)
|
||
if cmd[0] == "claude":
|
||
raise FileNotFoundError("No such file or directory: 'claude'")
|
||
return fake_popen.side_effect(cmd, *args, **kwargs)
|
||
|
||
popen = MagicMock(side_effect=_construct)
|
||
with patch("headroom.learn.analyzer.subprocess.Popen", popen):
|
||
result = _call_cli_llm("test digest", "claude-cli")
|
||
|
||
assert result == {"context_file_rules": [], "memory_file_rules": []}
|
||
assert calls[0][0] == "claude"
|
||
assert calls[1][0] == r"C:\npm\claude.cmd"
|
||
assert calls[1][1:] == calls[0][1:] # remaining args preserved
|
||
|
||
def test_non_streaming_cli_retries_with_resolved_shim(self, monkeypatch):
|
||
monkeypatch.setattr("headroom.learn.analyzer.os.name", "nt")
|
||
monkeypatch.setattr(
|
||
"headroom.learn.analyzer.shutil.which",
|
||
lambda name: r"C:\npm\codex.cmd" if name == "codex" else None,
|
||
)
|
||
calls: list[list[str]] = []
|
||
|
||
def _run(cmd, *args, **kwargs):
|
||
calls.append(cmd)
|
||
if cmd[0] == "codex":
|
||
raise FileNotFoundError("No such file or directory: 'codex'")
|
||
return MagicMock(
|
||
returncode=0,
|
||
stdout='{"context_file_rules": [], "memory_file_rules": []}',
|
||
stderr="",
|
||
)
|
||
|
||
with patch("headroom.learn.analyzer.subprocess.run", side_effect=_run):
|
||
result = _call_cli_llm("test digest", "codex-cli")
|
||
|
||
assert result == {"context_file_rules": [], "memory_file_rules": []}
|
||
assert calls[0][0] == "codex"
|
||
assert calls[1][0] == r"C:\npm\codex.cmd"
|
||
|
||
def test_shim_unresolvable_still_raises_not_found_in_path(self, monkeypatch):
|
||
monkeypatch.setattr("headroom.learn.analyzer.os.name", "nt")
|
||
monkeypatch.setattr("headroom.learn.analyzer.shutil.which", lambda name: None)
|
||
mock_run = MagicMock(side_effect=FileNotFoundError("No such file or directory: 'codex'"))
|
||
with patch("headroom.learn.analyzer.subprocess.run", mock_run):
|
||
with pytest.raises(RuntimeError, match="not found in PATH"):
|
||
_call_cli_llm("test digest", "codex-cli")
|
||
# shutil.which couldn't resolve anything — no retry attempted.
|
||
assert mock_run.call_count == 1
|
||
|
||
def test_non_windows_skips_shim_resolution(self, monkeypatch):
|
||
monkeypatch.setattr("headroom.learn.analyzer.os.name", "posix")
|
||
which = MagicMock(return_value=r"/usr/local/bin/codex")
|
||
monkeypatch.setattr("headroom.learn.analyzer.shutil.which", which)
|
||
mock_run = MagicMock(side_effect=FileNotFoundError("No such file or directory: 'codex'"))
|
||
with patch("headroom.learn.analyzer.subprocess.run", mock_run):
|
||
with pytest.raises(RuntimeError, match="not found in PATH"):
|
||
_call_cli_llm("test digest", "codex-cli")
|
||
which.assert_not_called()
|
||
assert mock_run.call_count == 1
|
||
|
||
|
||
class TestParseStreamEvent:
|
||
def test_returns_none_for_empty_line(self):
|
||
from headroom.learn.analyzer import _parse_stream_event
|
||
|
||
assert _parse_stream_event("") is None
|
||
assert _parse_stream_event(" \n") is None
|
||
|
||
def test_returns_none_for_invalid_json(self):
|
||
from headroom.learn.analyzer import _parse_stream_event
|
||
|
||
assert _parse_stream_event("not json at all") is None
|
||
assert _parse_stream_event("{unclosed") is None
|
||
|
||
def test_returns_none_for_non_dict_json(self):
|
||
from headroom.learn.analyzer import _parse_stream_event
|
||
|
||
assert _parse_stream_event('"a string"') is None
|
||
assert _parse_stream_event("[1, 2, 3]") is None
|
||
|
||
def test_parses_valid_event(self):
|
||
from headroom.learn.analyzer import _parse_stream_event
|
||
|
||
assert _parse_stream_event('{"type": "result", "result": "x"}') == {
|
||
"type": "result",
|
||
"result": "x",
|
||
}
|
||
|
||
|
||
class TestClaudeCliEdgeCases:
|
||
"""Coverage for less-traveled branches in the streaming claude-cli path."""
|
||
|
||
def test_non_string_result_field_falls_through_to_missing(self):
|
||
# `result` event present but the `result` field is a dict, not a string.
|
||
# The watchdog should not store it as final_result, so the path raises
|
||
# the "did not emit a final result event" error.
|
||
stdout = [_stream_event("result", subtype="success", result={"unexpected": "shape"})]
|
||
with patch(
|
||
"headroom.learn.analyzer.subprocess.Popen", _fake_claude_popen(stdout_lines=stdout)
|
||
):
|
||
with pytest.raises(RuntimeError, match="did not emit a final `result` event"):
|
||
_call_cli_llm("test digest", "claude-cli")
|
||
|
||
def test_stderr_on_success_is_logged_not_raised(self, caplog):
|
||
import logging
|
||
|
||
stdout = [_result_event('{"context_file_rules": [], "memory_file_rules": []}')]
|
||
stderr_warning = "deprecation: --foo will be removed in v2\n"
|
||
popen = _fake_claude_popen(stdout_lines=stdout, stderr_lines=[stderr_warning])
|
||
with caplog.at_level(logging.DEBUG, logger="headroom.learn.analyzer"):
|
||
with patch("headroom.learn.analyzer.subprocess.Popen", popen):
|
||
result = _call_cli_llm("test digest", "claude-cli")
|
||
assert result == {"context_file_rules": [], "memory_file_rules": []}
|
||
assert any("CLI stderr (exit 0)" in rec.message for rec in caplog.records)
|
||
|
||
def test_non_result_stdout_lines_are_buffered_into_snippet_on_failure(self):
|
||
# If only assistant/system events arrive (no result), the missing-result
|
||
# error should include a snippet from stdout.
|
||
stdout = [
|
||
_stream_event("system", subtype="init"),
|
||
_stream_event("assistant", message={"content": "thinking..."}),
|
||
]
|
||
with patch(
|
||
"headroom.learn.analyzer.subprocess.Popen", _fake_claude_popen(stdout_lines=stdout)
|
||
):
|
||
with pytest.raises(RuntimeError) as exc_info:
|
||
_call_cli_llm("test digest", "claude-cli")
|
||
message = str(exc_info.value)
|
||
assert "did not emit a final `result` event" in message
|
||
assert "thinking" in message # stdout snippet was included
|
||
|
||
def test_resolve_timeout_logs_warning_for_invalid(self, caplog, monkeypatch):
|
||
import logging
|
||
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI_TIMEOUT_SECS", "abc")
|
||
with caplog.at_level(logging.WARNING, logger="headroom.learn.analyzer"):
|
||
assert _resolve_timeout_secs("HEADROOM_LEARN_CLI_TIMEOUT_SECS", 300) == 300
|
||
assert any(
|
||
"Invalid HEADROOM_LEARN_CLI_TIMEOUT_SECS" in rec.message for rec in caplog.records
|
||
)
|
||
|
||
def test_resolve_timeout_logs_warning_for_non_positive(self, caplog, monkeypatch):
|
||
import logging
|
||
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI_TIMEOUT_SECS", "-5")
|
||
with caplog.at_level(logging.WARNING, logger="headroom.learn.analyzer"):
|
||
assert _resolve_timeout_secs("HEADROOM_LEARN_CLI_TIMEOUT_SECS", 300) == 300
|
||
assert any("must be positive" in rec.message for rec in caplog.records)
|
||
|
||
|
||
class TestResolveTimeoutSecs:
|
||
def test_uses_default_when_unset(self, monkeypatch):
|
||
monkeypatch.delenv("HEADROOM_LEARN_CLI_TIMEOUT_SECS", raising=False)
|
||
assert _resolve_timeout_secs("HEADROOM_LEARN_CLI_TIMEOUT_SECS", 300) == 300
|
||
|
||
def test_uses_default_when_empty(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI_TIMEOUT_SECS", "")
|
||
assert _resolve_timeout_secs("HEADROOM_LEARN_CLI_TIMEOUT_SECS", 300) == 300
|
||
|
||
def test_uses_default_for_non_integer(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI_TIMEOUT_SECS", "not-a-number")
|
||
assert _resolve_timeout_secs("HEADROOM_LEARN_CLI_TIMEOUT_SECS", 300) == 300
|
||
|
||
def test_uses_default_for_non_positive(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI_TIMEOUT_SECS", "0")
|
||
assert _resolve_timeout_secs("HEADROOM_LEARN_CLI_TIMEOUT_SECS", 300) == 300
|
||
|
||
def test_returns_overridden_value(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_LEARN_CLI_TIMEOUT_SECS", "777")
|
||
assert _resolve_timeout_secs("HEADROOM_LEARN_CLI_TIMEOUT_SECS", 300) == 777
|
||
|
||
|
||
class TestCallLlmRouting:
|
||
@patch("headroom.learn.analyzer._call_cli_llm")
|
||
def test_routes_cli_model_to_cli_backend(self, mock_cli: MagicMock):
|
||
mock_cli.return_value = {"context_file_rules": [], "memory_file_rules": []}
|
||
result = _call_llm("test digest", "claude-cli")
|
||
mock_cli.assert_called_once_with("test digest", "claude-cli", on_progress=None)
|
||
assert result == {"context_file_rules": [], "memory_file_rules": []}
|
||
|
||
@patch("headroom.learn.analyzer._call_cli_llm")
|
||
def test_routes_codex_cli(self, mock_cli: MagicMock):
|
||
mock_cli.return_value = {}
|
||
_call_llm("digest", "codex-cli")
|
||
mock_cli.assert_called_once_with("digest", "codex-cli", on_progress=None)
|