Commit Graph
24 Commits
Author SHA1 Message Date
Abhay SinghandJerrettDavis 12864fc8f0 fix(learn): harden analyzer JSON parsing against malformed LLM output (#2471)
## Description

Two spots in the `learn` analyzer crash on JSON a model can
realistically emit. Both parse fully model-controlled output.

**1. `_strip_fenced_json` returns a non-object instead of raising.** It
is documented to return a dict and to raise `json.JSONDecodeError` when
no candidate is an object, but its final fallback was `result: dict =
json.loads(text); return result`. `json.loads` raises for invalid JSON,
but for valid JSON that is not an object (a bare array, string, or
number) it returns a non-dict. The `-> dict`-annotated value is then a
list/str, and all three callers do `.get(...)` on it, raising
`AttributeError` instead of the clean `JSONDecodeError` they already
handle.

**2. `_parse_llm_response` crashes on null rule lists/fields.** It read
`raw.get("context_file_rules", [])` / `raw.get("memory_file_rules", [])`
and `rule.get("section", "").strip()` / `rule.get("content",
"").strip()`. `.get(key, default)` only falls back when the key is
absent, so a present-but-null `"context_file_rules": null` made `for
rule in None` raise `TypeError`, and a null `section`/`content` made
`None.strip()` raise `AttributeError`.

## Fix

- `_strip_fenced_json`: reject a non-dict fallback result explicitly
(`raise json.JSONDecodeError(...)`), restoring the documented contract.
The lenient first-brace..last-brace slice still recovers a single object
a model wrapped in an array; only genuinely object-less JSON raises.
- `_parse_llm_response`: coerce the rule lists and the
`section`/`content` fields with `or` so a null is treated like an absent
value. Valid rules parse unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/learn/analyzer.py`: raise `JSONDecodeError` when
`_strip_fenced_json`'s fallback parse yields a non-dict; `or`-coerce
null rule lists and null `section`/`content` in `_parse_llm_response`.
- `tests/test_learn/test_analyzer.py`: regressions for non-object JSON
raising (and array-wrapped-object still extracted), and for null rule
lists / null fields being tolerated with valid rules still parsed.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_learn/test_analyzer.py::TestStripFencedJson tests/test_learn/test_analyzer.py::TestLLMResponseParser -q
# all pass; each new test fails with the corresponding fix reverted
# (JSONDecodeError contract / 'NoneType' object is not iterable / has no attribute 'strip')

$ uvx ruff@0.15.17 check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/learn/analyzer.py
Success: no issues found in 1 source file
```

Note: one unrelated digest-builder test
(`TestDigestBuilder::test_includes_project_info`) fails in my local
Windows venv because it hardcodes a `/tmp/...` path; it fails
identically on a clean `main` and is not touched by this change.

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: called the real `_strip_fenced_json` with
object-less JSON (`[1,2,3]`, `"str"`, `42`, ...) and valid/extractable
objects, and the real `_parse_llm_response` with `{"context_file_rules":
null}`, a rule with `section: null`, and a valid rule; then reverted
each fix and re-ran.
- Observed result: with the fixes the object-less inputs raise
`JSONDecodeError`, the null shapes yield an empty recommendation list,
and valid inputs parse as before; with each fix reverted the respective
input raises (`AttributeError: 'list' object has no attribute 'get'`,
`TypeError: 'NoneType' object is not iterable`, `AttributeError:
'NoneType' object has no attribute 'strip'`). Ran against the actual
module.
- Not tested: a live `headroom learn` run where the backing model
returns these shapes.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2471"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `27fa31eac507413415a00b54e29679bef08eabe9`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean current upstream/main0ddaa56c integration; production
_strip_fenced_json/_parse_llm_response/_as_str ASTs are identical to
contributor published head. Main-relative scope remains two files. Small
maintainer test-only repair uses str(project.project_path) instead of a
POSIX-only literal in existing digest assertion; source and test fixture
data unchanged. Initial full analyzer run109pass/1fail and separate
clean-main failure independently establish inherited Windows path
expectation before repair. After repair399tests pass,11skip,1warning
(all110analyzer cases plus289incoming-main regressions). Two changed
files pass Rufflint/format,candidate offline lock and whitespace pass.
Complete native inputset/allrecordedhashes/artifactchecksum verified
before existing3982DLL reuse; no fresh build.

Independent paired parser probes: branch passes both truthy non-string
section and objectless JSON contract cases; clean main fails with
AttributeError and missing JSONDecodeError respectively. These are
controlled parser executions, not live LLM quality/provider behavior.
Outstanding Devin truthy non-list rule-container report also
independently reproduced on BOTH branch and clean main:
context_file_rules:42 and memory_file_rules:42 raise TypeError in both.
This is an inherited limitation, not a PR-introduced regression or
grounds to withhold this narrower fix. Do not claim arbitrary-container
safety. The later author suggestion to join list-valued rule content
changes intended normalization semantics; not selected or implemented by
maintainer. Existing _as_str drops non-string fields, preserving
contributor intent.

Read author replies/current discussions and unresolved thread. No new
unreviewed external findings at prepublication audit. Source/test diff
contains no workflow/setup/dependency/native/CLI paths; central
publication gate passed. Exact published-head Actions/rollup, current
main ancestry/mergeability and fresh discussion remain before final
approval. Maintainers handle CI; no PR merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-06 21:32:04 -06:00
gglucassandClaude Opus 5.5 506ba16c4a fix(learn): keep the digest's loop section inside its budget (#4269)
## Description

`headroom learn` can build a prompt larger than the CLI it runs will
accept, and the prompt-too-long retry cannot shrink it.

`_build_digest` budgets the per-session event stream (`max_tokens * 4`
chars), but it renders the "Detected Loops" section ahead of that stream
without any bound: every detected loop, each with every message index it
repeated at. Codex learn scans all of a machine's Codex sessions as one
project, so loops accumulate across every session the user ever ran. On
one Windows machine this produced a 1,786,526-char prompt, and `codex
exec` refused it on every scan:

```text
Error: turn/start: turn/start failed: Input exceeds the maximum length of 1048576 characters. (code -32602), data: {"input_error_code":"input_too_large","max_chars":1048576,"actual_chars":1786526}
```

Halving `max_tokens` on a prompt-too-long error does not help here: it
only trims the event stream, never the loop section.

This PR bounds the rendered loop section by the same budget, and lets a
Codex overflow take the smaller-digest retry.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `learn/loops.py`: `format_loops_for_digest` takes `max_chars` and
counts the rendered text, header and newlines included. It adds loops in
the given order (`detect_loops` sorts by measured waste) and stops at
the first one whose line would pass `max_chars`, so the costliest loops
are kept. Each line shows its sample input cut to 120 chars (the same
cut `detect_loops` makes) and its first 20 message indices beside the
full repeat count. It returns "" when no loop fits.
- `learn/analyzer.py`: `_build_digest` gives the loop section a quarter
of the `max_tokens * 4` char budget, less whatever the header and the
prior learned patterns leave short, so header + loops + prior patterns
stay inside the budget and the event stream fills the rest. The prior
patterns are never cut: a re-emitted section replaces its prior one
wholesale, so a partial baseline would drop bullets from the user's
file.
- `learn/analyzer.py`: the event stream counts every line with the
newline that joins it, and a session header, event or tool-call line
goes in only if it leaves 100 chars for the truncation markers that can
follow it. Once a session is cut, no later session is started. So the
whole digest stays inside `max_tokens * 4`; the only exception is a
prior-patterns section that leaves no room on its own, since it is never
cut.
- `learn/analyzer.py`: `_is_prompt_too_long` also matches Codex's "Input
exceeds the maximum length" refusal, so a Codex overflow is retried at
half the budget like the other backends.
- The caller's `LoopPattern`s are not touched (slices only), and
`apply_loop_weighting` still receives every loop.
- Tests: `test_loop_section_stays_inside_the_digest_budget` is
parametrized over a 120-char Bash input, a 2,000-char Grep pattern and a
2,000-char Glob path, at the 80k and 10k (retry floor) budgets. New
`test_loops_fit_beside_a_large_prior_patterns_section` (80k, 10k). New
`test_whole_digest_stays_inside_the_budget`: 5 sessions of 3,000 failing
calls each, at 80k and 10k, as tool calls and as events, with and
without loops, asserts `len(digest) <= max_tokens * 4`. New
`test_many_tiny_sessions_stay_inside_the_budget`: 20,000 one-call
sessions at 80k and 10k, as tool calls and as events (614 to 4,862
sessions kept), so the blank line closing each session is counted too.
`test_retries_smaller_digest_on_prompt_too_long` adds the real Codex
error text.
- `pyproject.toml` / `uv.lock` (review: Dependency audit): the `aiohttp`
constraint floor is now `>=3.14.4` (CVE-2026-111063, CVE-2026-111064 and
GHSA-2g87-pp6c-29x5, all fixed in 3.14.4), relocked with `uv lock
--upgrade-package aiohttp==3.14.4`. Only aiohttp (3.14.3 -> 3.14.4) and
yarl (1.22.0 -> 1.25.1, which 3.14.4 requires) move. Main still resolves
3.14.3, so the same commit is on #4299, #4286, #4285 and #4269.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

`mypy` passes on the changed module.

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_learn/ -q
384 passed, 4 skipped in 34.16s

$ uvx ruff@0.17.0 check headroom/learn/analyzer.py headroom/learn/loops.py tests/test_learn/test_loop_weighting.py tests/test_learn/test_analyzer.py
All checks passed!
$ uvx ruff@0.17.0 format --check headroom/learn/analyzer.py headroom/learn/loops.py tests/test_learn/test_loop_weighting.py tests/test_learn/test_analyzer.py
4 files already formatted

$ uv run --frozen --extra dev mypy headroom/learn/analyzer.py headroom/learn/loops.py
Success: no issues found in 2 source files
```

With `headroom/learn/` reverted to f52a4356, 7 of the new cases fail:
the 4 long Grep/Glob cases, both prior-patterns cases and the Codex
retry case. With it reverted to d9391dab, all 8
`test_whole_digest_stays_inside_the_budget` cases fail (321,135 to
321,420 chars at 80k, 40,300 to 40,475 at 10k).
`test_many_tiny_sessions_stay_inside_the_budget` passes on df75e3fa and
fails if the per-session blank line is left out of the count (324,858
and 40,590 chars).

## Real Behavior Proof

- Environment: macOS, Python 3.11, a real `~/.codex/sessions` (283
sessions, 407 MB), read only.
- Exact command / steps: `CodexPlugin().scan_project(...)`,
`detect_loops(sessions)`, then `_build_digest(project, sessions,
loops=loops, max_tokens=B)` for B = 80,000 and 10,000 (the retry floor)
on `main`, on f52a4356, on d9391dab and on df75e3fa (c95f5fe8 only adds
a test); plus the review repro (Grep loops with 2,000-char patterns, no
sessions).
- Observed result: on df75e3fa the whole digest stays inside `max_tokens
* 4` at both budgets (319,721 of 320,000 and 39,933 of 40,000 chars on
real sessions), and the loop section inside a quarter of it (tables
below).

| real sessions (291 loops) | 80k tokens (320,000 chars) | 10k tokens
(40,000 chars) |
  |---|---|---|
| `main` | 291 loop lines, 47,546 chars; digest 322,613 | 291 loop
lines, 47,546 chars; digest 58,487 |
| f52a4356 | 200 loop lines, 32,208 chars; digest 322,765 | 25 loop
lines, 4,678 chars; digest 40,376 |
| d9391dab | 291 loop lines, 46,177 chars; digest 322,604 | 60 loop
lines, 9,985 chars; digest 40,283 |
| df75e3fa | 291 loop lines, 46,177 chars; digest 319,721 | 60 loop
lines, 9,985 chars; digest 39,933 |

| review repro, digest chars | 25 loops, 80k | 25 loops, 10k | 200
loops, 80k | 200 loops, 10k |
  |---|---|---|---|---|
  | `main` | 87,080 | 87,080 | 693,630 | 693,630 |
  | f52a4356 | 54,080 | 54,080 | 429,630 | 54,080 |
  | d9391dab, df75e3fa | 7,080 | 7,080 | 53,630 | 10,006 (36 loops) |

In every repro run the caller's `LoopPattern`s still held the 2,000-char
sample and all 300 indices afterwards. Real loops never had long
samples: `detect_loops` already cuts `sample_input` to 120 chars
(longest here: 103), so the long-pattern case only reaches
`_build_digest` through caller-supplied loops.
- Not tested: the Windows machine that produced the 1.79M-char prompt (I
have its error, not its sessions). I did not run `codex exec` on an
oversized prompt (it would write a session to `~/.codex`); the needle
comes from that machine's error text above, and the same string is in
the codex-cli 0.156.1 binary.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes. A full digest now stops at the
last line that fits (it used to let in the line that crossed the budget
and skip newlines in the count, ~0.8% over), so it carries slightly
fewer events. Also, when the rendered loop section would pass a quarter
of the digest's char budget (less with large prior learned patterns), or
a loop repeated more than 20 times: the digest lists only the costliest
loops that fit, with at most 20 message indices each. A Codex "Input
exceeds the maximum length" failure is now retried at half the budget.
Other projects get an identical digest.
- Kill switch / disable path: Not applicable.
- Unsafe override required: No.
- Qualification impact: None. Learn only; the proxy request path and
compression are untouched.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

- No documentation change: the learn docs don't describe the digest's
layout.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 21:48:07 -05:00
Tejas ChopraandClaude Opus 5.5 abbf36f052 refactor(memory,learn,cache): remove unreferenced classes, methods and constants (#4112)
## Description

This removes nine symbols in `headroom/memory`, `headroom/learn` and
`headroom/cache` that nothing references outside their own definition.
One is a legacy alias whose only user was its own test.

Evidence: for each name I ran `grep -rnw <name>` across `headroom crates
plugins sdk docs wiki scripts benchmarks examples e2e .github
pyproject.toml tests` and the org's consumer repos. The only hit was the
definition, plus the `FailureAnalyzer` alias test removed here. None of
the names is public API. None is in an `__all__`, a re-export or lazy
export map, an entry-point group or the docs, and no consumer imports
it.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `memory/inline_extractor.py`: remove `InlineMemoryWrapper`, with
`chat` and `chat_with_response`. `with_memory` (`memory/wrapper.py`)
uses only the module's functions, `inject_memory_instruction` and
`parse_response_with_memory`, and those stay.
- `memory/traffic_learner.py`: remove these:
  - `AgentType`;
- `_COMMAND_RE` and `_FILE_PATH_RE`. `_MODULE_RE` and `_COMMAND_NF_RE`
next to them are used and stay;
- `TrafficLearner.get_learned_patterns`. Its docstring said "retained
for backwards compatibility", but `TrafficLearner` is internal to the
proxy.
- `memory/core.py`: remove the private
`HierarchicalMemory._scope_level_value`.
- `cache/compression_feedback.py`: remove the private
`CompressionFeedback._truncate_strategy_dicts`.
- `learn/analyzer.py`: remove `FailureAnalyzer`. Its docstring said
"used by existing CLI code", but `cli/learn.py` uses `SessionAnalyzer`
directly. Also remove `TestFailureAnalyzerCompat` from
`tests/test_learn/test_analyzer.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/test_memory_wrapper.py tests/test_inline_extractor_inject.py tests/test_memory/test_traffic_learner.py tests/test_openai_responses_traffic_learner.py tests/test_memory/test_core_operations.py tests/test_learn/test_analyzer.py tests/test_ccr_feedback.py tests/test_memory/test_hierarchical.py tests/test_toin_fixes.py
================ 417 passed, 8 skipped, 772 warnings in 49.26s =================
$ mypy <5 changed product files> --ignore-missing-imports
Success: no issues found in 5 source files
$ uvx ruff@0.16.8 check <changed files>
All checks passed!
```

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, repo .venv, branch checkout at
93c23cb2 (base a9815bbb), temp HOME / HEADROOM_WORKSPACE_DIR /
CODEX_HOME, offline.
- Exact command / steps:
- Ran the shared smoke harness, which starts a real proxy from this
checkout against a local fake upstream and sends one Anthropic and one
OpenAI request with a compressible tool result.
- Ran a direct script. It imports the touched modules, checks that the
removed names are gone, and drives a real `TrafficLearner` through two
`on_tool_result` calls.
- Ran the tests for every touched module: the memory wrapper, inline
extractor, traffic learner, core, CCR feedback and learn analyzer.
- Observed result:
  - Both proxy requests return HTTP 200 with savings.
- The modules import without the removed symbols, and the learner
processes both results.
  - 417 passed. The 8 skips are optional-dependency skips.
- Not tested: No runtime path changed, because nothing called the
removed symbols.

```text
$ bash smoke.sh <worktree>
branch: cleanup/memory-cache-dead-symbols @ 93c23cb2
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines:
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you 
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.

$ HOME=$TMP/home HEADROOM_WORKSPACE_DIR=$TMP/ws CODEX_HOME=$TMP/codex python -u -c "<import touched modules; hasattr(removed name); TrafficLearner().on_tool_result(...) x2; get_stats()>"
headroom.memory.inline_extractor: imports, InlineMemoryWrapper present=False
headroom.memory.traffic_learner: imports, AgentType present=False
headroom.learn.analyzer: imports, FailureAnalyzer present=False
traffic learner stats: {'requests_processed': 2, 'patterns_extracted': 0, 'patterns_saved': 0, 'pending_patterns': 0, 'min_evidence': 1, 'history_size': 2}
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; dead-code removal.
- Stable/default behavior changed: No.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

- Documentation: N/A, because none of these names is documented.
- `SQLiteMemoryStore.count_sync` and `HNSWVectorIndex.set_ef_search` are
also unused. Their classes are re-exported from
`headroom.memory.adapters`, so they are deprecated in a separate PR
instead of being deleted here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 09:00:02 -07:00
TrîggeredandJD Davis 4ba231ab3d feat(learn): add agy (Antigravity CLI) as an analysis backend (#3939)
## Description

Adds the Antigravity CLI (`agy`) as a keyless LLM backend for `headroom
learn`, next to `claude`, `gemini` and `codex`. Antigravity subscription
users with no raw API key can now run the analysis through `agy`.

`agy` cannot use the existing stdin path as is. `agy -p` takes the
prompt as a flag argument and never reads stdin, so a large digest would
hit OS argument-length limits. The only stdin path is `--input-format
stream-json`, which requires `--output-format stream-json`. The backend
therefore sends the prompt as a single NDJSON user event on stdin and
reads the answer from `result.response` in the final `result` event.
Everything else (wall-clock timeout, Windows shim fallback, failure
detail, fenced-JSON parsing) reuses the existing `run()` path that
`gemini` and `codex` use.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/learn/analyzer.py`: add an `agy` / `agy-cli` entry to
`_CLI_BACKENDS` (`agy --input-format stream-json --output-format
stream-json`). `agy` is never auto-detected, and it only runs when
`HEADROOM_LEARN_ALLOW_UNSAFE_AGY=1` is set: `agy` has no zero-tool mode,
so transcript text could drive its tools. Without the opt-in,
`HEADROOM_LEARN_CLI=agy` and `--model agy-cli` fail before any
subprocess starts; with it, a warning is logged on every run.
- `headroom/learn/analyzer.py`: in `_call_cli_llm`, wrap the prompt as
`{"event": "user", "message": {"content": ...}}` for `agy-cli`, and
extract `result.response` from the stream with a new
`_agy_result_text()` helper before the existing `_strip_fenced_json()`
parsing. A missing `result` event surfaces as the existing "unparseable
output" error, with both ends of stdout.
- `tests/test_learn/test_analyzer.py`: tests that auto-detection never
selects `agy` (even when it is the only CLI on PATH and the opt-in is
set), that `HEADROOM_LEARN_CLI=agy` and `_call_cli_llm(..., "agy-cli")`
fail without the opt-in (no subprocess call), `HEADROOM_LEARN_CLI=agy`
with the opt-in, existing CLIs still taking precedence, the exact
command and stdin payload, result-event parsing (including a fenced
response and a literal U+2028), and a missing `result` event.
- `headroom/cli/learn.py`: check the agy opt-in during model resolution,
so `HEADROOM_LEARN_CLI=agy` and `--model agy-cli` without it exit 1 with
an `Error:` line before any sessions are scanned.
- `tests/test_cli_learn.py`: a CLI test for both paths without the
opt-in.
- `wiki/learn.md`: list `agy` among valid `HEADROOM_LEARN_CLI` values
and document the `HEADROOM_LEARN_ALLOW_UNSAFE_AGY=1` opt-in and its
risk.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_learn/ tests/test_cli_learn.py -q
5 failed, 345 passed, 4 skipped in 10.78s
# The same 5 fail on unmodified upstream/main on this Windows machine (5 failed, 340 passed),
# all path/git related and unrelated to this change:
#   test_analyzer.py::TestDigestBuilder::test_includes_project_info
#   test_scanner.py::TestDecodeProjectPath::test_home_dir_username_stays_single_component
#   test_writer.py::TestClaudeLocalMdStaysOutOfGit::{test_apply_adds_exclude_entry,
#     test_second_run_does_not_duplicate_the_entry, test_tracked_file_warns_instead_of_excluding}

$ uv run --frozen --extra dev pytest tests/test_learn/test_analyzer.py -q -k agy   # analyzer.py reverted
FAILED TestDetectDefaultModel::test_env_var_selects_agy_with_unsafe_opt_in
FAILED TestDetectDefaultModel::test_env_var_agy_without_unsafe_opt_in_raises
FAILED TestCallCliLlm::test_agy_cli_sends_stream_json_and_parses_result_event
FAILED TestCallCliLlm::test_agy_cli_result_with_unicode_line_separator
FAILED TestCallCliLlm::test_agy_cli_without_unsafe_opt_in_does_not_run
FAILED TestCallCliLlm::test_agy_cli_missing_result_event_raises
6 failed, 2 passed, 97 deselected
# All 8 pass with the change.

$ uvx ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ uvx ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/learn/analyzer.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, `agy` 1.2.14 signed in via subscription, no
API keys set.
- Exact command / steps: `HEADROOM_LEARN_CLI=agy
HEADROOM_LEARN_ALLOW_UNSAFE_AGY=1 ANTHROPIC_API_KEY= OPENAI_API_KEY=
GEMINI_API_KEY= uv run --frozen --extra dev headroom learn` in this
repository (dry run).
- Observed result: `Analyzing with agy-cli...`, then `Sessions: 3 |
Calls: 109 | Failures: 19 (17.4%)`, `Recommendations: 8`, with `[WOULD
WRITE]` blocks for `CLAUDE.local.md` and `MEMORY.md`; exit 0. Before
wiring the backend I confirmed the protocol by hand: `agy -p -` answers
the literal prompt `-` and ignores stdin, while the stream-json
invocation returns the answer in `{"event": "result", "result":
{"status": "SUCCESS", "response": "..."}}`.
- Not tested: macOS/Linux `agy` installs; `agy` under a permission mode
that prompts for tool approval (my config is `always-proceed`), where a
tool request in print mode would be bounded by the existing
`HEADROOM_LEARN_CLI_TIMEOUT_SECS` wall-clock cap.

## Runtime Rollout Safety

- Rollout-managed feature(s): None (offline `headroom learn` CLI)
- Minimum rollout channel: dev
- Stable/default behavior changed: No. `agy` is never auto-detected; it
is only used when selected with `HEADROOM_LEARN_CLI=agy` / `--model
agy-cli` together with `HEADROOM_LEARN_ALLOW_UNSAFE_AGY=1`.
- Kill switch / disable path: Set an API key, `HEADROOM_LEARN_CLI`, or
`--model` to pick another backend.
- Unsafe override required: None
- Qualification impact: None
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A (CLI change).

## Additional Notes

- No tracking issue exists for an `agy` analysis backend, so `Closes #`
is left empty. Related Antigravity work (#3428 `headroom init -g agy`,
#3610 Antigravity transcripts in `headroom learn`) is independent of
this change.
- Unlike `claude-cli` (#3892), `agy` has no flag to start a session with
zero tools. I deliberately did not add `--dangerously-skip-permissions`:
the digest contains untrusted transcript content, and the backend should
run under the user's own permission settings.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3939"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-04 13:56:26 -07:00
f00d425614 fix(learn): drop the echoed prompt from a failed cli's error (#3928)
## Description

`codex exec` echoes the whole stdin prompt to stderr ahead of its own
error, and a failed CLI's error message keeps the first 2000 chars of
stderr. The analyzer's system prompt alone is 4116 chars, so every
codex-cli failure reported Codex's banner and the start of our system
prompt and never the reason (a 401, an unknown model, a rate limit). The
echo also carries the session digest, which does not belong in an error
message.

`_failure_detail` now takes the prompt that was sent on stdin. When
stderr contains it, the echo is replaced with `[prompt omitted]` and the
end of what the CLI printed after it is kept, cut at a line boundary,
inside the same 2000-char cap. Stderr from a CLI that does not echo the
prompt is handled exactly as before.

Closes #3927

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/learn/analyzer.py`: `_failure_detail` gets a keyword-only
`prompt` argument; when the prompt (newlines normalized the way a
text-mode pipe reads them back) appears in stderr, it is replaced with a
marker and the text after it is tailed to fit the cap, whole lines where
possible.
- Both callers pass the prompt they wrote to stdin: the `run(...)` path
used by codex-cli and gemini-cli, and the claude-cli streaming path
(claude does not echo today, so nothing changes there).
- `tests/test_learn/test_analyzer.py`:
`test_codex_failure_drops_prompt_echo_and_keeps_reason` feeds stderr in
the shape codex-cli 0.156.1 prints (banner, `user`, the prompt, 100
reconnect lines, the final error) and checks the message ends with the
real reason, carries the marker, has no digest text, and stays inside
the cap.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Ran on the changed files only (`ruff check`/`ruff format --check` on the
two changed files, `mypy headroom/learn/analyzer.py`), not the whole
tree.

### Test Output

Before the fix (new test against upstream/main's analyzer):

```text
$ pytest tests/test_learn/test_analyzer.py -q -k prompt_echo
E   AssertionError: assert False
E    +  where False = <built-in method endswith of str object at 0x77522a6400>('ERROR: unexpected status 401 Unauthorized: Missing bearer')
E    +    where <built-in method endswith of str object at 0x77522a6400> = '`codex exec --skip-git-repo-check` failed (exit 1):\nReading prompt from stdin...\nOpenAI Codex v0.156.1\n--------\nm...endation (how many tokens would be saved per session if this rule existed)\n- Separate stable project facts (CONTEXT_F'.endswith
```

After:

```text
$ pytest tests/test_learn/test_analyzer.py tests/test_learn/test_error_classification.py -q
============================= 102 passed in 4.73s ==============================
$ ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
2 files already formatted
$ mypy headroom/learn/analyzer.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 27, Python 3.12, codex-cli 0.156.1 (Homebrew),
upstream/main d5318ac2 vs this branch.
- Exact command / steps: `CODEX_HOME=<empty dir> python run_proof.py`,
where the script calls the real `analyzer._call_cli_llm(digest,
"codex-cli")` with a 300-line fake digest (28,586 prompt chars on
stdin). The empty `CODEX_HOME` gives codex no credentials, so the real
`codex exec --skip-git-repo-check` fails with a 401 and spends no
tokens. The script prints the RuntimeError and whether the system prompt
or digest text appears in it.
- Observed result: before the fix the error is 2052 chars of Codex
banner plus our system prompt with no 401 in it; with it the echo is
replaced by `[prompt omitted]` and the message ends with codex's own 401
line (output below).

  Before (upstream/main), last lines of the message:
  ```text
  error chars: 2052
  system prompt in error: True
  digest in error: False
  ...
    its guardrail even if it appears only once as a loop, and set its
estimated_tokens_saved to at least the measured wasted tokens reported
there.
  - Only include patterns with CLEAR evidence from the data (2+
  ```
  After (this branch), the whole message:
  ```text
  error chars: 2037
  system prompt in error: False
  digest in error: False
  `codex exec --skip-git-repo-check` failed (exit 1):
  Reading prompt from stdin...
  OpenAI Codex v0.156.1
  --------
  workdir: <scratch dir>/echo-proof
  model: gpt-6-astra
  provider: openai
  approval: never
  sandbox: read-only
  reasoning effort: none
  reasoning summaries: none
  session id: 01a0fcc7-cd87-7333-be41-6981201cf854
  --------
  user
  [prompt omitted]
  ERROR: Reconnecting... 2/5
2026-10-02T13:22:30.743051Z ERROR
codex_api::endpoint::responses_websocket: failed to connect to
websocket: HTTP error: 401 Unauthorized, url:
wss://api.openai.com/v1/responses
  ERROR: Reconnecting... 3/5
  ...
warning: Falling back from WebSockets to HTTPS transport. unexpected
status 401 Unauthorized: Missing bearer or basic authentication in
header, url: wss://api.openai.com/v1/responses, cf-ray:
a444045a1da3d919-AMS
  ERROR: Reconnecting... 1/5
  ...
  ERROR: Reconnecting... 5/5
ERROR: unexpected status 401 Unauthorized: Missing bearer or basic
authentication in header, url: https://api.openai.com/v1/responses,
cf-ray: a444048d19332162-AMS, request id:
req_c4fd2b8fee604e3f932b68bc90488ac1
ERROR: unexpected status 401 Unauthorized: Missing bearer or basic
authentication in header, url: https://api.openai.com/v1/responses,
cf-ray: a444048d19332162-AMS, request id:
req_c4fd2b8fee604e3f932b68bc90488ac1
  ```
(`...` marks lines elided here for length; the real message keeps them.)
- Not tested: other codex failure reasons (bad model name, rate limit,
network) were not triggered for real; they print after the echo the same
way, which is what the fix keys on. gemini-cli was not run (not
installed here); if it does not echo the prompt its errors are
unchanged. Windows was not tested: the newline normalization covers a
CRLF echo read back through a text-mode pipe, but that path has only
been reasoned about, not run. The success-path `logger.debug` of stderr
is untouched.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a, not rollout-managed.
- Stable/default behavior changed: only the text of the RuntimeError
raised when a learn CLI backend exits non-zero and echoed the prompt to
stderr. No change to what is sent to any CLI or model, or to anything
outside `headroom learn`.
- Kill switch / disable path: none; diagnostics text only.
- Unsafe override required: no.
- Qualification impact: none; learn error reporting only, no proxy,
compression or cache path.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: no user-facing docs describe the error text, so nothing
to update.
- Builds on #3080 (stdout in CLI failure messages) and #3517 (both ends
of unparseable output); this is the stderr half for a CLI that echoes
its input.
- Headroom Desktop has shipped this as a downstream patch to the pinned
wheel; this upstreams it.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3928"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-04 13:56:12 -07:00
2a4d34c476 fix(learn): run the claude-cli analysis with hooks disabled (#3926)
## Description

`headroom learn`'s claude-cli backend runs `claude -p` as the user, so
the user's hooks (from settings files and installed plugins) run inside
the analysis session. #3892 took the session's tools away; this takes
its hooks away. A blocking Stop hook makes Claude take a second turn
whose reply becomes the final `result`, replacing the analysis, so learn
finds no rules and prints "No actionable patterns found." with exit 0
and no error.

The fix passes `--settings '{"disableAllHooks":true}'`. `--settings` is
merged on top of the user's own settings, which still load, so env
routing (`ANTHROPIC_BASE_URL` in settings) and OAuth keep working.
`--bare` would also skip hooks but never reads OAuth or the keychain.

Closes #3925

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/learn/analyzer.py`: add `--settings
'{"disableAllHooks":true}'` to the claude-cli entry in `_CLI_BACKENDS`,
with a comment on why hooks must be off and why `--bare` is not used.
- `tests/test_learn/test_analyzer.py`:
`test_claude_cli_streams_and_parses_result_event` pins the new argv.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

Ran on the changed files only (`ruff check`/`ruff format --check` on the
two changed files, `mypy headroom/learn/analyzer.py`), not the whole
tree. The existing argv test was updated rather than a new one added.

### Test Output

Before the fix (updated test against upstream/main's analyzer):

```text
$ pytest tests/test_learn/test_analyzer.py -q -x -k test_claude_cli_streams_and_parses_result_event
E   AssertionError: assert ['claude', '-...essages', ...] == ['claude', '-...essages', ...]
E     Right contains 2 more items, first extra item: '--settings'
FAILED tests/test_learn/test_analyzer.py::TestCallCliLlm::test_claude_cli_streams_and_parses_result_event
======================= 1 failed, 96 deselected in 3.35s =======================
```

After:

```text
$ pytest tests/test_learn/test_analyzer.py tests/test_learn/test_error_classification.py -q
============================= 101 passed in 4.19s ==============================
$ ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
2 files already formatted
$ mypy headroom/learn/analyzer.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 27, Python 3.12, Claude Code 2.1.287 logged in with
a subscription (OAuth), upstream/main d5318ac2 vs this branch.
- Exact command / steps: a temp project whose `.claude/settings.json`
has a Stop hook that logs each firing and blocks the first stop (script
in the linked issue). From that directory, a script takes the claude-cli
argv from `_CLI_BACKENDS` and calls the real
`_call_claude_cli_streaming(cmd, prompt, ...)` against the real `claude`
binary, with the prompt `Reply with exactly this JSON and nothing else:
{"context_file_rules": [{"section": "Proof", "content": "analysis
survived"}], "memory_file_rules": []}`.
- Observed result: before the fix the Stop hook ran twice and its forced
second turn replaced the analysis; with it the hook never ran and the
analysis came back intact (output below).

Before (upstream/main): the hook ran twice and its forced second turn
replaced the answer.
  ```text
argv: ['claude', '-p', '--output-format', 'stream-json', '--verbose',
'--include-partial-messages', '--tools', '', '--strict-mcp-config']
  parsed result: {'context_file_rules': [], 'memory_file_rules': []}
  stop-hook-fired.log: fired (blocked) 15:11:48
  fired (allowed) 15:11:50
  ```
After (this branch): the hook never ran and the analysis came back
intact.
  ```text
argv: ['claude', '-p', '--output-format', 'stream-json', '--verbose',
'--include-partial-messages', '--tools', '', '--strict-mcp-config',
'--settings', '{"disableAllHooks":true}']
parsed result: {'context_file_rules': [{'section': 'Proof', 'content':
'analysis survived'}], 'memory_file_rules': []}
  stop-hook-fired.log: (absent: hook never ran)
  ```
Settings still load: with `"model": "haiku"` added to the same project
`settings.json`, the same argv's `system/init` event reports the model
from that file while the file's Stop hook stays silent, and auth is the
subscription login (`apiKeySource: none`):
  ```text
init: {'model': 'claude-haiku-4-5-20251001', 'apiKeySource': 'none',
'tools': [], 'mcp_servers': []}
  result: {"ok": true}
  stop-hook-fired.log: (absent: hook never ran)
  ```
- Not tested: a hook defined by an installed plugin (as opposed to a
settings file) in this exact setup; `disableAllHooks` is documented to
cover both and the field report was a plugin hook. Managed (enterprise
policy) hooks: a user-level `disableAllHooks` is not expected to
override managed hooks, so those may still run. Did not run a full
`headroom learn` end to end against real session history; the proof
drives the analyzer's claude-cli runner directly with a tiny prompt.
Windows and Linux not tested.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a, not rollout-managed.
- Stable/default behavior changed: yes, for the claude-cli learn backend
only: the analysis `claude -p` session no longer runs the user's hooks.
Nothing outside `headroom learn` is affected.
- Kill switch / disable path: none added. The analysis session has no
tools and returns one JSON object, so no hook has a legitimate job in
it; auth and routing come from settings, which still load.
- Unsafe override required: no.
- Qualification impact: none; learn analysis only, no proxy, compression
or cache path.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: no doc lists the claude-cli argv, so there is nothing
to update.
- Follow-up to #3892, which added `--tools ""` and `--strict-mcp-config`
to the same command.
- Headroom Desktop has shipped this flag as a downstream patch to the
pinned wheel; this upstreams it.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3926"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-04 13:55:57 -07:00
db90b93da9 fix(learn): run the claude-cli analysis with no tools (#3892)
## Description

`headroom learn`'s claude-cli backend runs the analysis as `claude -p`,
and a headless `claude -p` is a full Claude Code agent. With Claude Code
2.1.285, its init event lists 127 tools from 7 MCP servers: Bash, Edit,
Task, cron, every claude.ai connector. They run under the user's own
permissions, hooks and skills. The analysis needs none of them, because
the digest is the whole input and the answer is one JSON object.

When the model decides to go read or verify things, the run turns
multi-turn. Each tool call emits stream events, so the idle cap never
fires, and the run is killed at the hard cap with nothing written. We
saw this in the field as `claude -p --output-format stream-json
--verbose exceeded the 900s hard cap` (Headroom Desktop raises the hard
cap to 900s). A normal analysis finishes in seconds to under a minute in
a single turn.

This adds `--tools ""` and `--strict-mcp-config` to the claude-cli
command, so the session has zero tools and answers in one turn. You need
both flags: `--tools ""` alone removes the built-in tools but still
leaves every MCP server's tools.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/learn/analyzer.py`: the `claude-cli` entry of
`_CLI_BACKENDS` now also passes `--tools "" --strict-mcp-config`, with a
comment explaining why.
- `tests/test_learn/test_analyzer.py`:
`test_claude_cli_streams_and_parses_result_event` pins the new command.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_learn/
======================= 320 passed, 4 skipped in 22.78s ========================
$ uvx ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ uvx ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/learn/analyzer.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 15 (Darwin 24.6), Claude Code 2.1.285 logged in
with OAuth, user settings with plugins, hooks and claude.ai MCP
connectors, this branch run through `uv run`.
- Exact command / steps: (1) Piped `hi` into the old and the new
claude-cli command and read the `system`/`init` stream event. (2) Called
`_call_cli_llm(<small two-call digest>, "claude-cli")` against the real
CLI.
- Observed result: (1) The old command's init event listed 127 tools and
7 MCP servers; the new command's listed 0 tools and 0 MCP servers. (2)
The analysis returned parsed JSON with `context_file_rules` and
`memory_file_rules` in 8.5s.
- Not tested: Windows (the command is still a list passed to Popen, and
the empty `--tools` value is a separate argv entry); Claude Code
versions without `--tools`, which will fail with `unknown option
'--tools'` and need a `claude update`.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; this only changes the learn
claude-cli command.
- Minimum rollout channel: N/A, not rollout-managed.
- Stable/default behavior changed: yes, the learn analysis session runs
with no tools. The recommendations come from the digest alone, as the
prompt already asks.
- Kill switch / disable path: `--model <litellm-model-name>`, or the
gemini-cli or codex-cli backends, skip this command entirely.
- Unsafe override required: no.
- Qualification impact: none; learn is not in the proxy request path.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: no docs describe the claude-cli argv, so there was
nothing to update.
- The existing command test was updated rather than a new one added; it
asserts the exact argv.
- Stop hooks that block can still make a `claude -p` run multi-turn.
Only Claude Code's `--safe-mode` disables hooks under OAuth (`--bare`
needs an API key), and that flag is too new to pass unconditionally.
- Headroom Desktop carries this as a sitecustomize patch until a wheel
with it ships. The patch does nothing when the wheel already passes
either flag.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 15:53:42 -05:00
gglucassandClaude Opus 5 cb3ab61d4d fix(learn): show both ends of CLI output on a parse failure (#3517)
## Description

`_call_cli_llm` and `_call_claude_cli_streaming` attach a HEAD-only
excerpt of
the CLI's output to their parse failures ("returned unparseable output.
First
2000 chars:", "did not emit a final `result` event"). For those two
failures
specifically, the head is the one part that is never diagnostic: a
model's JSON
answer that does not parse looks like valid JSON for its first 2000
chars in
every case. What separates a payload truncated mid-value from one with
prose
after the closing fence is the END of it, and that is exactly what gets
cut.

This is not hypothetical. Downstream (the Headroom desktop app) these
RuntimeErrors are reported with the CLI's stderr attached, and a real
report
arrived with 2000 characters of well-formed ```json ... and no way to
tell which
of the two shapes it was, so the defect could not be acted on at all.

`_failure_detail` already makes this argument for the non-zero-exit path
("stdout
is tailed rather than headed because CLI backends emit the error last").
This
extends the same reasoning to the parse failures, which need both ends:
the head
is where a wrapper (fence, preamble) shows up, the tail is where the
break is.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Added `_output_snippet(text, limit=_MAX_SNIPPET_LEN)`: returns short
output
unchanged, otherwise `head + "...[N chars omitted]..." + tail`, still
bounded
  by the same `_MAX_SNIPPET_LEN` budget.
- Used it at the three sites that excerpt CLI OUTPUT for a parse
failure: the
non-streaming unparseable stdout, the missing final `result` event, and
the
  unparseable `result` text.
- Reworded those messages from "First 2000 chars:" to "Head and tail of
the
output:". Deliberately no counts in the marker line: downstream groups
these
errors by that line, so a length in it would open a new group per
failure.
- Left the stderr excerpts and `_failure_detail` alone.

## Testing

- [x] Unit tests pass (`pytest`)

### Test Output

```text
uv run --frozen --extra dev pytest tests/test_learn/ -q
260 passed, 4 skipped in 11.02s
```

## Real Behavior Proof

- Environment: macOS 15, Python 3.12, uv --frozen, headroom main @
4e1f7769
- Exact command / steps: Added
`test_claude_cli_unparseable_result_shows_both_ends`, which feeds
`_call_cli_llm` a claude-cli `result` event carrying a ~7 KB fenced JSON
payload followed by trailing prose (the shape that cannot be diagnosed
today), then ran `uv run --frozen --extra dev pytest
tests/test_learn/test_analyzer.py -q`.
- Observed result: 97 passed. The RuntimeError now contains both the
opening ```json fence and the trailing "another shape" prose, plus a
"chars omitted" marker, and stays shorter than the payload. Before the
change the same assertion on the tail fails, since the message stopped
at 2000 chars.
- Not tested: A live `claude -p` run against a real API failure; the CLI
is mocked, as in the surrounding tests.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: No. Only the text of three error
messages; short output is byte-identical to before.
- Kill switch / disable path: Not applicable; no flag, no new
dependency, no I/O.
- Unsafe override required: No.
- Qualification impact: None. `headroom learn` success paths are
untouched.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 16:05:22 -05:00
4c6bd3e8fc fix(learn): echo live progress during claude-cli analysis (#3158)
## Description

`headroom learn` already streams claude-cli's `stream-json` output
internally, using it to drive an idle-timeout watchdog, but none of that
liveness reaches stdout. Wrapper UIs that tail the CLI output (the
Headroom desktop app included) see nothing print between "Analyzing with
claude-cli..." and the final result, sometimes for minutes, which makes
the scan look hung. This closes #3105 by surfacing that liveness as
short, throttled progress lines.

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)

## Changes Made

- Added an optional `on_progress: typing.Callable[[str], None] | None =
None` callback, threaded through `SessionAnalyzer.analyze` ->
`_call_llm` -> `_call_cli_llm` -> `_call_claude_cli_streaming`. It
defaults to `None` at every layer, so no existing caller changes
behavior.
- Inside the streaming event loop, added `_progress_detail()`, which
maps each non-terminal `stream-json` event
(`system`/`assistant`/`stream_event`/`user`) to a short phrase like
`"assistant responding, 12s"`. The terminal `result` event is handled
separately and is never echoed as progress.
- Throttled the callback to at most once per 3 seconds
(`_PROGRESS_THROTTLE_SECS`) via the same `time.monotonic()` idiom the
function already used for its idle-timeout watchdog, so a burst of
partial-message events doesn't spam the console.
- The callback is invoked defensively: any exception it raises is caught
and logged via `logger.debug`, never propagated, so a broken UI pipe
can't abort an otherwise-successful analysis.
- `headroom/cli/learn.py` supplies the one concrete callback, reusing
the exact existing `" Analyzing with {model}..."` prefix with `(detail)`
appended, so any wrapper UI that whitelists that prefix keeps working
unchanged.
- Added `--include-partial-messages` to the claude-cli invocation, as
suggested in the issue. Without it, claude only emits about three events
for an entire run (session init, one assistant message, result), which
is too sparse for the throttle to produce useful incremental updates.

## Testing

- [x] Added new tests
- [x] All tests pass locally

```
$ .venv/Scripts/python.exe -m pytest tests/test_learn/test_analyzer.py tests/test_cli_learn.py -q
tests\test_learn\test_analyzer.py .......................................
......................................................                  [ 84%]
tests\test_cli_learn.py .................                                [100%]
109 items collected, 108 passed, 1 failed

FAILED tests/test_learn/test_analyzer.py::TestDigestBuilder::test_includes_project_info
  (pre-existing, unrelated: a Windows-path-separator assertion that also
  fails identically on a clean upstream/main checkout, before this change)

$ .venv/Scripts/ruff.exe check headroom/learn/analyzer.py headroom/cli/learn.py tests/test_learn/test_analyzer.py tests/test_cli_learn.py
All checks passed!

$ .venv/Scripts/ruff.exe format --check headroom/learn/analyzer.py headroom/cli/learn.py tests/test_learn/test_analyzer.py tests/test_cli_learn.py
4 files already formatted

$ .venv/Scripts/mypy.exe headroom/learn/analyzer.py headroom/cli/learn.py
Success: no issues found in 2 source files
```

Three new tests were added covering: a burst of events within one
throttle window only echoing the first one, spaced-out events each
mapping to the correct phrase in order (with the terminal `result` event
never echoed), and a raising `on_progress` callback not aborting a
successful analysis. A fourth new test at the CLI layer asserts the `"
Analyzing with ... (detail)"` lines appear in `headroom learn` output.

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.11, local `.venv`, no live
`claude` CLI invoked (tests mock `subprocess.Popen`, matching the
existing test style in this file)
- Exact command / steps: `.venv/Scripts/python.exe -m pytest
tests/test_learn/test_analyzer.py tests/test_cli_learn.py -v -k
progress`
- Observed result: all 4 new progress-related tests pass, confirming the
throttling, event-to-phrase mapping, defensive-callback behavior, and
the CLI's rendered output line all work as designed
- Not tested: an actual live `claude` CLI process (no `claude` binary
available in this sandboxed environment); the fix only changes what data
is threaded through an existing, already-live streaming loop, so this is
low risk, but a manual check against the real CLI before/after would be
good follow-up validation

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no — `on_progress` defaults to `None`
everywhere, and when `None` the code path is unchanged from before this
PR
- Kill switch / disable path: none needed; the only new default-on
behavior is the extra progress lines in `headroom learn` CLI output,
which is additive to existing output (same prefix, one line appended per
event) and does not change any parsed/machine-readable output
- Unsafe override required: no
- Qualification impact: none — no changes to routing, caching, or
backend selection logic
- Rollback path: revert this commit; nothing persists state or changes
on-disk formats

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Jerrett Davis <mxjerrett@gmail.com>
2026-09-09 22:00:36 -05:00
gglucassandClaude Fable 5 d329a04b4b fix(learn): retry LLM analysis with a smaller digest when the prompt overflows the context window (#3404)
## Description

`headroom learn` dies with zero recommendations when the analysis prompt
overflows the model's context window. Field report (desktop telemetry):
a 4-session / 443-call scan failed with ``LLM analysis failed: `claude
-p --output-format stream-json --verbose` failed (exit 1): Prompt is too
long``, and the whole project's analysis was lost.

The root cause is that `_build_digest` budgets `_MAX_DIGEST_TOKENS`
(80k) for the digest as if the prompt had the window to itself. CLI
backends do not work that way: `claude -p` loads the user's own context
on top - CLAUDE.md files, MCP tool definitions, skills. The digest
budget cannot know how big that share is, and the 4-chars-per-token
estimate also runs hot on dense tool-call lines. On installs with heavy
context the combined prompt overflows a 200k window and the analysis
fails outright, every scan, until the user's sessions age out.

The fix makes the budget adaptive instead of guessing a universally safe
constant: when the backend reports a context overflow, rebuild the
digest at half the budget and retry, down to a 10k-token floor (80k ->
40k -> 20k -> 10k, then give up). A truncated digest that produces
recommendations beats a full digest that produces an error. Non-overflow
failures are re-raised on the first attempt exactly as before.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `SessionAnalyzer.analyze` wraps the LLM call in a shrinking-budget
retry loop; overflow detection via new `_is_prompt_too_long(exc)`, which
matches the known phrasings across backends: claude CLI ("Prompt is too
long"), Anthropic via LiteLLM ("prompt is too long: N tokens > M
maximum"), OpenAI ("context_length_exceeded" / "maximum context
length"), Gemini ("input token count exceeds the maximum").
- `_build_digest` gains a `max_tokens` parameter (default
`_MAX_DIGEST_TOKENS`, so all other callers are unchanged); the char
budget derives from it.
- New `_MIN_DIGEST_TOKENS = 10_000` floor so a persistent overflow
terminates after 4 attempts instead of looping.
- Tests: retry produces a strictly shorter second digest and succeeds;
persistent overflow stops after the 10k floor (4 calls) and returns
stats-only; non-overflow failures still fail on the first call with no
retry.

## Testing

- [x] Unit tests pass (`pytest`)

### Test Output

```text
uv run --frozen --extra dev pytest tests/test_learn/ -q
216 passed, 3 skipped in 8.18s
```

## Real Behavior Proof

- Environment: macOS (arm64), Python 3.12 via uv, this branch's
worktree, real `claude` CLI installed.
- Exact command / steps: Drove the real CLI through `_call_cli_llm` with
oversized digests to verify the trigger end-to-end.
- Observed result: Could not force the overflow on this machine - the
account resolves to a 1M-context model, so even a ~950k-token prompt
succeeds (confirmed via the stream-json `result` event's `contextWindow:
1000000`). The failure is real on 200k-window installs: the field
telemetry's stderr carries `Prompt is too long` verbatim, which is the
exact string `_is_prompt_too_long` matches and the exact `RuntimeError`
message the retry test injects through the real `analyze` -> `_call_llm`
path.
- Not tested: a live end-to-end overflow against a 200k-window account
(none at hand); the gemini/codex CLI phrasings are covered by the
pattern list, not live runs.

## Runtime Rollout Safety

- Rollout-managed feature(s): None; `learn` is not rollout-gated.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Only on the previously-failing path:
an overflow now retries with a smaller digest instead of aborting the
analysis. Successful analyses are byte-identical (same budget, same
digest).
- Kill switch / disable path: Not needed; a persistent overflow degrades
to today's behavior (stats-only result, warning logged) after 4
attempts.
- Unsafe override required: No.
- Qualification impact: Heavy-context installs (large CLAUDE.md, many
MCP servers) go from 100% learn failure to a truncated-digest analysis.
- Rollback path: Revert the commit; no persisted formats touched.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-09 11:30:19 -05:00
JD Davis 632cb81dbe fix(learn): surface Codex analysis failures (#3016)
## Description

`headroom learn` could invoke Codex CLI from a non-Git working directory
without Codex’s required bypass flag. The resulting backend error was
then swallowed by the analyzer and rendered as “No actionable patterns
found” with exit code 0. This fixes both coupled defects so Codex can
run from discovered project locations and genuine analysis failures
remain visible and machine-detectable.

Closes #3008

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `--skip-git-repo-check` to the Codex CLI analysis backend
command.
- Added an explicit `analysis_error` result field instead of conflating
backend failure with an empty recommendation set.
- Kept multi-project analysis best-effort, while returning exit code 1
after any project analysis fails.
- Prevented failed analysis from printing a misleading no-pattern
success message.
- Added analyzer and CLI regression coverage for the command and
failure-propagation contracts.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run pytest -q tests/test_learn/test_analyzer.py tests/test_cli_learn.py
102 passed in 2.34s

uv run pytest -q tests/test_learn tests/test_cli_learn.py
257 passed, 7 skipped in 3.11s

uv run mypy headroom
Success: no issues found in 520 source files

uv run ruff check <changed files>
All checks passed!
uv run ruff format --check <changed files>
5 files already formatted

uv run pytest tests scripts/tests --splits 4 --group N --tb=short -q
shard 1: 2766 passed, 140 skipped in 174.08s
shard 2: 2699 passed, 207 skipped in 60.00s
shard 3: 2822 passed, 84 skipped in 76.10s
shard 4: 2734 passed, 172 skipped in 80.29s
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.13, Codex CLI 0.147.0-compatible
command surface, current `main` including #2996.
- Exact command / steps: verified `codex exec --help`; exercised
`_call_cli_llm` with a captured subprocess command; invoked the Click
command with a simulated Codex nonzero backend result.
- Observed result: the subprocess command is `codex exec
--skip-git-repo-check`; backend failure text is printed as `Analysis
failed`, the misleading no-pattern message is absent, and the CLI exits
1.
- Not tested: live paid Codex analysis against production account
credentials; subprocess and CLI behavior are covered deterministically.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; this is CLI-only failure handling.
- Minimum rollout channel: normal patch release after exact-head CI is
entirely green.
- Stable/default behavior changed: failed LLM analysis now exits nonzero
instead of reporting success; successful and genuinely empty analyses
are unchanged.
- Kill switch / disable path: select another backend with
`HEADROOM_LEARN_CLI` or `--model` if Codex CLI is unavailable.
- Unsafe override required: none.
- Qualification impact: all four Python CI shards, static checks,
security checks, and command-level regression tests must pass.
- Rollback path: fix forward through a human-reviewed corrective PR; no
persisted data or migration is involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project’s style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation — inline
result-contract documentation; no separate user guide change is required
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable; command-line backend and exit semantics only.

## Additional Notes

Human review only. No merge or auto-merge is configured. This corrects
the root failure and exit semantics without extending any timeout.
2026-08-25 21:40:12 -05:00
gglucassandClaude Opus 5 c5563d3a7d fix(learn): include stdout in CLI failure messages, not just stderr (#3080)
## Description

`headroom learn` reports CLI backend failures using **stderr only**.
`claude -p --output-format stream-json --verbose` writes *nothing* to
stderr when the run fails at the API layer, so the failure a user
actually sees is a message that stops at the colon:

```text
LLM analysis failed: `claude -p --output-format stream-json --verbose` failed (exit 1):
```

The reason is not missing, it is discarded. Claude Code still emits a
final `result` event on stdout whose `result` field is the
human-readable cause, and the streaming path has already parsed it into
`final_result` one line above the `raise`. This makes a whole class of
failures undiagnosable for users and maintainers alike: a usage limit,
an unreachable local proxy, and an expired login all render identically
as an empty message.

Reported by a desktop user who could only tell us "sometimes i have this
LLM analysis failed" with nothing after the colon.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Add `_failure_detail(stderr, stdout, *, result_text=None)` in
`headroom/learn/analyzer.py`. Prefers the already-parsed `result` text,
falls back to the **tail** of stdout (CLI backends emit the error last,
after their whole event log), keeps stderr when present, and returns
`"(no output captured)"` so the message is never a dangling colon.
- Use it in `_call_claude_cli_streaming` (streaming claude-cli path) and
in `_call_cli_llm` (the `subprocess.run` backends, gemini-cli /
codex-cli), so the same blind spot is closed for every CLI backend
rather than only the one that was reported.
- Existing truncation behaviour is unchanged: each stream is still
capped at `_MAX_SNIPPET_LEN`.

Complements #3016, which makes an analysis failure propagate instead of
being swallowed as success; that PR fixes *whether* the user learns a
failure happened, this one fixes *what* the failure says. No overlapping
lines.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_learn/ -q
247 passed, 4 skipped in 27.08s

$ uvx ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!

$ uvx ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
2 files already formatted

$ uv run --frozen --extra dev mypy headroom/learn/analyzer.py
Success: no issues found in 1 source file
```

New tests:
`test_claude_cli_nonzero_exit_includes_api_error_from_stdout`,
`test_claude_cli_nonzero_exit_with_no_output_says_so`,
`test_claude_cli_nonzero_exit_keeps_stderr_when_present`,
`test_codex_nonzero_exit_includes_stdout_when_stderr_empty`.

## Real Behavior Proof

- Environment: macOS 15.6 (Darwin 24.6.0), Claude Code 2.1.228, Python
3.10.18, headroom on this branch.
- Exact command / steps: forced an API-layer failure in the exact
command the analyzer runs, capturing the streams separately: `echo "say
hi" | claude -p --output-format stream-json --verbose --settings
'{"env":{"ANTHROPIC_BASE_URL":"http://127.0.0.1:9"}}' > out.txt 2>
err.txt; echo "EXIT=$?"; wc -c err.txt; tail -c 400 out.txt`
- Observed result: `EXIT=1`, `err.txt` is **0 bytes**, and the reason
appears only in the last stdout line: `"terminal_reason":"api_error",
..., "result":"API Error: Connection refused — a firewall or proxy may
be blocking it (ConnectionRefused)"`. A second run with `--bare`
produced the same shape with `"result":"Not logged in · Please run
/login"`. Before this change both surface as `failed (exit 1):` with
nothing after the colon; after it, the `result` text is in the message.
The unit tests encode this exact stream shape (stdout `result` event,
empty stderr, exit 1).
- Not tested: real usage-limit and 429 responses, which I cannot provoke
on demand. They travel the same code path as the reproduced `api_error`
case (final `result` event on stdout, empty stderr), so they are covered
by construction rather than by observation. Windows and the gemini-cli
backend were not exercised manually; the shared helper is covered by
unit tests for both the streaming and `subprocess.run` paths.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. This touches only the error text
raised by `headroom learn`'s CLI backends; no rollout-gated feature,
flag, or runtime component is involved.
- Minimum rollout channel: N/A, not rollout-gated. Ships with the
package like any other library fix.
- Stable/default behavior changed: Yes, narrowly. The message text of an
existing `RuntimeError` on a non-zero CLI exit now includes the
stdout/`result` reason alongside stderr. No control flow, exit code,
public API, or return value changes: the same exception is raised in the
same cases.
- Kill switch / disable path: None needed. Nothing is enabled or newly
executed, so there is nothing to switch off; the only behavioral surface
is the string inside an exception that was already being raised.
- Unsafe override required: No.
- Qualification impact: None. No qualification-gated path, model, or
provider behavior is touched. Callers that pattern-match this message on
`"failed (exit N)"` still match, since that prefix is unchanged.
- Rollback path: Revert this commit. The previous stderr-only message
returns with no migration, state, or config to undo.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 20:20:45 -07:00
TenderDeve 85e8699451 fix(learn): keep traceback tail in tool-error digest preview (#2596)
## Description

`_format_tool_call` in `headroom/learn/analyzer.py` built the error
preview with a head-only slice — `tc.output[:200]`. For tracebacks the
root cause (`ExceptionType: message`) is at the **tail**, so the digest
showed only `Traceback (most recent call last):` plus the first frame
and dropped the actual diagnosis. The issue reports 46% of 715 measured
errors were truncated past the 200-char head.

Closes #2590

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Added `_truncate_head_tail()` helper that collapses newlines and, when
over budget, keeps both the head and the tail joined by `…`.
- `_format_tool_call` now uses it for error output so the exception line
survives truncation. Short errors are returned unchanged (no marker).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
$ uv run pytest tests/test_learn/test_analyzer.py::TestDigestBuilder -q
9 passed in 1.47s

$ uv run ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
```

## Real Behavior Proof

- Environment: headroom @ main, Python 3.14, uv
- Exact command / steps: added a long synthetic traceback (`KeyError:
'the-actual-root-cause'` at the tail) as a failing tool call and built
the digest.
- Observed result: digest now contains both `Traceback` and `KeyError:
'the-actual-root-cause'`, separated by `…`; short errors have no `…`.
- Not tested: mypy not run locally.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Truncation budget stays at 200 chars (now split head/tail). mypy not run
locally; happy to adjust if CI flags anything.
2026-07-27 06:44:51 -07:00
Fabien Culpo d2170b1922 fix(learn): parse fenced JSON even with a prose preamble (#1988)
## Description

`_strip_fenced_json` only stripped a markdown fence when the string
*started with* ```` ``` ````. When the model prefixed prose before the
fence (e.g. `Here is the JSON:\n\n```json ...`) despite being told to
return JSON only, the guard was skipped and `json.loads` ran on the
prose, raising `JSONDecodeError`. The claude-cli streaming path surfaced
this as `returned unparseable output`, and `headroom learn` silently
discarded the LLM analysis, degrading to "No actionable patterns found".
This is the parsing-side cousin of the silent-degradation issue fixed in
#373.

Closes #1989. Related: #373.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/learn/analyzer.py`: rewrote `_strip_fenced_json` to locate
the fenced block wherever it appears, then fall back to the whole text,
then to a first-`{` / last-`}` slice, only re-raising `JSONDecodeError`
if nothing parses as a JSON object. Preserves the prior "first opening /
last closing fence" behaviour and triple-backtick content inside the
payload. Fixes all three call sites (non-streaming CLI, claude-cli
streaming, litellm).
- `tests/test_learn/test_analyzer.py`: added regression cases to
`TestStripFencedJson` for preamble-before-fence, prose around a bare
object, and triple-backticks inside the payload.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) — scoped to the changed
module (see Additional Notes)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_learn/test_analyzer.py -q
........................................................................ [ 86%]
...........                                                              [100%]
83 passed, 1 warning in 2.18s

$ ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
2 files already formatted

$ mypy --ignore-missing-imports --follow-imports=silent headroom/learn/analyzer.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.13.12, headroom-ai at this branch
(runtime deps from an installed 0.30.0 env).
- Exact command / steps: ran the old vs new `_strip_fenced_json` on the
exact failing model output (a prose preamble followed by a ```json
fence), then applied the fix over an installed 0.30.0 and re-ran the
previously failing `headroom learn --apply`. Input sample: `'The JSON is
my deliverable for this analysis task. Here it
is:\n\n```json\n{"context_file_rules": [], "memory_file_rules":
[]}\n```'`
- Observed result: OLD raised `JSONDecodeError: Expecting value: line 1
column 1 (char 0)`; NEW returned `{'context_file_rules': [],
'memory_file_rules': []}`. The real `headroom learn --apply` run that
had been failing with `returned unparseable output` then completed and
consumed the LLM analysis instead of dropping it. Full transcript:
  ```text
  OLD: JSONDecodeError -> Expecting value: line 1 column 1 (char 0)
  NEW: {'context_file_rules': [], 'memory_file_rules': []}
  ```
- Not tested: full end-to-end `headroom learn --apply` was not re-run
inside CI here (it shells out to a live `claude` CLI); the parser is
exercised deterministically by the added unit tests and the before/after
repro above.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A —
updated the function docstring only; no external docs affected)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable (N/A — no CHANGELOG
entry convention observed for this fix; happy to add if maintainers
prefer)

## Additional Notes

- `mypy` was run against the changed module in isolation
(`--ignore-missing-imports --follow-imports=silent`) rather than the
full project, because I validated in an ad-hoc environment; the change
keeps the existing `-> dict` signature and annotations, so it is
type-neutral.
- Not addressed here (possible follow-up): the failure is swallowed as a
warning in `analyze()`, so users only see "No actionable patterns found"
with no signal the LLM pass produced nothing — the same
silent-degradation class as #373, on the parsing side.
2026-07-10 11:15:36 -07:00
Rod BoevandJD Davis e3b45e402b fix(learn): handle Windows UTF-8, drive-letter paths, and CLI shim fallback (#1895)
## Description

`headroom learn --verbosity` is broken on Windows in three related ways:

- Transcript/profile reads can use the platform default codec, so
non-ASCII content can raise `UnicodeDecodeError` and collapse learning
signals to empty output.
- `--project <path>` can miss real Claude project directories because
Windows profile junctions can raise `PermissionError` during directory
walks, and escaped Claude project folder names cannot always distinguish
`vibe-remote` from `vibe\remote`.
- `headroom learn --agent codex` can fail with `` `claude` not found in
PATH `` even when the npm-installed CLI exists, because Windows `.cmd`
shims require `PATHEXT` resolution.

Refs https://github.com/headroomlabs-ai/headroom/issues/1624 for the
Windows learn failures. The dashboard-hint UX and
third-party-provider-auth items in that issue are unrelated and out of
scope for this PR.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/learn/verbosity.py`: read and write verbosity
transcripts/profiles with `encoding="utf-8"` so non-ASCII content works
regardless of the Windows locale codec.
- `headroom/learn/plugins/claude.py`: skip inaccessible siblings one
entry at a time during greedy project path decoding, so one Windows
junction no longer hides valid project directories.
- `headroom/learn/plugins/claude.py`: prefer a valid `cwd` found in
Claude session JSONL when discovering project paths, which resolves
ambiguous escaped folder names such as `vibe-remote` versus
`vibe\remote`.
- `headroom/learn/analyzer.py`: resolve Windows CLI shim paths through
`shutil.which()` after `FileNotFoundError`, then retry once for
streaming and non-streaming CLI calls.
- `CHANGELOG.md`: document the Windows learn fixes under `Unreleased`.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_verbosity_learn.py::TestWindowsEncoding
tests/test_learn/test_scanner.py::TestGreedyPathDecode::test_permission_denied_sibling_does_not_abort_the_walk
tests/test_learn/test_analyzer.py::TestWindowsCliShimFallback
tests/test_learn/test_scanner.py::TestDecodeProjectPath::test_discover_project_prefers_session_cwd_over_ambiguous_folder_name
-q`)
- [x] Linting passes (`uv run ruff check
headroom/learn/plugins/claude.py tests/test_learn/test_scanner.py`)
- [x] Formatting passes (`uv run ruff format --check
headroom/learn/plugins/claude.py tests/test_learn/test_scanner.py`)
- [ ] Type checking passes (`uv run mypy headroom`) not run; no new
public type surface
- [x] New tests added for the Windows `cwd` disambiguation regression
- [x] Manual testing performed

### Test Output

```text
uv run ruff format headroom/learn/plugins/claude.py
1 file reformatted

uv run ruff check headroom/learn/plugins/claude.py tests/test_learn/test_scanner.py
All checks passed!

uv run ruff format --check headroom/learn/plugins/claude.py tests/test_learn/test_scanner.py
2 files already formatted

uv run pytest tests/test_verbosity_learn.py::TestWindowsEncoding tests/test_learn/test_scanner.py::TestGreedyPathDecode::test_permission_denied_sibling_does_not_abort_the_walk tests/test_learn/test_analyzer.py::TestWindowsCliShimFallback tests/test_learn/test_scanner.py::TestDecodeProjectPath::test_discover_project_prefers_session_cwd_over_ambiguous_folder_name -q
9 passed in 0.25s
```

CI on current head `c6dbac40` is green. A prior `test (1)` run hit an
unrelated timing-sensitive scheduler assertion; GitHub did not permit
direct rerun without admin rights, so the empty commit `c6dbac40`
retriggered CI and the shard passed.

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, real filesystem for the
path-decoding reproduction.
- Exact command / steps: Ran `uv run pytest
tests/test_verbosity_learn.py::TestWindowsEncoding
tests/test_learn/test_scanner.py::TestGreedyPathDecode::test_permission_denied_sibling_does_not_abort_the_walk
tests/test_learn/test_analyzer.py::TestWindowsCliShimFallback
tests/test_learn/test_scanner.py::TestDecodeProjectPath::test_discover_project_prefers_session_cwd_over_ambiguous_folder_name
-q`, `uv run ruff check headroom/learn/plugins/claude.py
tests/test_learn/test_scanner.py`, and `uv run ruff format --check
headroom/learn/plugins/claude.py tests/test_learn/test_scanner.py`; the
path tests create real Windows-style project directories, inaccessible
siblings, ambiguous `vibe\remote` versus `vibe-remote` folders, and
Claude session JSONL with `cwd` pointing at the intended project.
- Observed result: The focused test command returned `9 passed in
0.25s`, Ruff check passed, and Ruff format check passed. The decoder
skips the inaccessible sibling and reaches `vibe-remote`; the
session-`cwd` test returns `vibe-remote` instead of trusting the
ambiguous escaped folder name; the UTF-8 tests round-trip non-ASCII
transcript/profile content under a non-UTF-8 Windows-style codec; the
CLI shim tests retry once through `shutil.which()` after
`FileNotFoundError`.
- Not tested: real npm-installed `claude`/`codex` CLI shims,
dashboard-hint UX, and third-party-provider auth.

## Review Readiness

- [x] I have performed a self-review
- [x] Retrospective review completed after opening; it found one missing
`cwd` disambiguation case, now fixed in this PR
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- The post-open retrospective review concluded this needed targeted
rework rather than only a retrospective sign-off.
- The `cwd` recovery commit is `ee720bb1`; `0905be7b` contains the
required formatter cleanup; current head `c6dbac40` is an empty CI-rerun
commit after GitHub denied direct rerun without admin rights.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-07-09 12:49:38 -05:00
gglucassandClaude Opus 4.7 9bff5752bb fix(learn): claude-cli streams output with idle timeout (#373)
## Description

`headroom learn` with the claude-cli backend used `subprocess.run` with
a hard 120s wall-clock cap and no liveness signal. A successful long
analysis and a hung connection looked identical — exit 0 with "0
recommendations" was the only user-visible signal when the LLM call
timed out, which silently hides genuine learnings.

This PR makes the CLI backend timeout-aware, with progress detection for
claude-cli and configurable wall-clock caps for every backend.

Fixes #(issue number)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Streaming claude-cli with idle timeout**: invoke `claude -p
--output-format stream-json --verbose` and run a watchdog loop that
drains stdout/stderr via reader threads. Each stream-json event resets
an idle deadline. Kill the process if no output for
`HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS` (default 60s) or if total elapsed
exceeds `HEADROOM_LEARN_CLI_TIMEOUT_SECS` (default 300s, was 120s). The
final `type:"result"` event carries the assistant response, which is
then parsed as JSON. Reader threads (rather than `select`) are used so
the watchdog works on Windows where `select` does not support pipe
handles.
- **Bumped default `_CLI_TIMEOUT` from 120s to 300s** as the hard cap
for all CLI backends. The previous 120s was too tight for large digests
on slower networks.
- **Env-var overrides** via new helper `_resolve_timeout_secs(env_var,
default)`:
- `HEADROOM_LEARN_CLI_TIMEOUT_SECS` — hard wall-clock cap (all CLI
backends)
- `HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS` — idle cap (streaming
claude-cli only)
- Non-positive or non-integer values log a warning and fall back to
defaults, so a typo can't disable the timeout.
- **gemini-cli and codex-cli** keep `subprocess.run(timeout=hard_cap)`
since they do not emit progress events. They benefit from the bumped
default and the env-var override.
- **CHANGELOG.md** updated under `[Unreleased]` → `### Fixed`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed (existing repro: 16k-call digest that
previously timed out at 120s)

New test coverage in `tests/test_learn/test_analyzer.py`:

- `test_claude_cli_streams_and_parses_result_event` — happy path, fake
Popen yields system/assistant/result events
- `test_claude_cli_parses_fenced_result` — markdown fences in the result
event still parse
- `test_claude_cli_idle_timeout_kills_hang` —
`HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS=1` + a hanging stdout iterator
triggers the idle watchdog
- `test_claude_cli_hard_cap_kills_continuous_chatter` — continuous
events with a low hard cap fire the wall-clock kill (proves idle reset
alone can't keep a runaway alive)
- `test_claude_cli_missing_result_event_raises` — graceful failure when
no `result` event is emitted
- `test_claude_cli_nonzero_exit_raises` /
`test_claude_cli_unparseable_result_raises_with_context` /
`test_claude_cli_not_installed_raises` — error paths
- Parallel codex-cli error coverage (timeout-honors-env-override
included) so the wall-clock path is exercised
- `TestResolveTimeoutSecs` — unset / empty / non-integer / non-positive
/ valid override

## Test Output

```
$ uv run pytest tests/test_learn/test_analyzer.py
============================== 67 passed in 2.14s ==============================

$ uv run ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!

$ uv run ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
2 files already formatted

$ uv run mypy headroom/learn/analyzer.py
Success: no issues found in 1 source file
```

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- The contract assumed for claude-cli stream-json output is: each line
is a JSON object with a `type` field; the final event has
`type:"result"` with a string `result` field carrying the assistant
text. This matches the documented Anthropic CLI behavior. If the
contract changes upstream, `_call_claude_cli_streaming` raises a clear
"did not emit a final \`result\` event" error rather than silently
succeeding.
- Backwards-compatible for users without env-var configuration: behavior
just becomes "longer hard cap, plus idle watchdog for claude-cli",
neither of which can falsely succeed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-11 11:55:19 -05:00
Evan Alferez d7973665f4 fix(learn): finish gemini-flash-latest default model sweep (#532)
Google deprecated gemini/gemini-2.0-flash; headroom learn silently fails
when GEMINI_API_KEY is set. PR #532 updated the default in analyzer.py
but left stale references in the CLI help text and unit test assertion.
2026-06-03 08:32:43 +09:00
GarmandClaude Opus 4.7 35073ccb23 style(learn): ruff format test_analyzer.py
Line-length wrapping only. No behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 13:24:46 +02:00
GarmandClaude Opus 4.7 6d2aba8741 fix(learn): show prior patterns block to LLM to prevent dangling refs
When `headroom learn` re-surfaced a section heading that already existed
in CLAUDE.md / MEMORY.md, the writer replaced that section wholesale —
but the LLM never saw the prior block, so it emitted condensed bullets
like "X is *also* large — same rule as Y, Z" assuming Y and Z would
remain siblings. After replacement, Y and Z were gone and the "also"
dangled.

This threads the project's current `<!-- headroom:learn -->` block (from
both CLAUDE.md and MEMORY.md) into the digest as a "Prior Learned
Patterns" section, and extends the system prompt to make the re-emission
contract explicit: re-stating a section replaces it wholesale, so the
LLM must copy forward prior bullets it still agrees with. Prior sections
the LLM omits entirely are still carried forward by the writer (#231
behavior preserved as a safety net).

Changes:
- New `extract_marker_block(file_content)` helper in `learn.writer` that
  returns the raw marker block (delimiters included) or None.
- New `_build_prior_patterns_section(project)` in `learn.analyzer` reads
  `project.context_file` and `project.memory_file` via the new helper
  and formats a labeled section ahead of the per-session event stream.
- `_build_digest` emits the prior-patterns section when present; char
  budget accounting unchanged (prior blocks are small).
- `_SYSTEM_PROMPT` gains a "Prior Learned Patterns" rule block telling
  the LLM how to integrate prior bullets (preserve / revise / drop-only-
  if-contradicted) and warning against unresolved cross-references.
- Tests: 6 new `TestPriorPatternsInjection` cases (present/absent files,
  no-marker-block, both-files, end-to-end via mocked `_call_llm`); 4 new
  `TestExtractMarkerBlock` cases.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 12:59:02 +02:00
Gyeonghun ParkandClaude Sonnet 4.6 c3cf022886 fix(learn): handle FileNotFoundError when CLI tool is not installed
When --model codex-cli is used but codex is not in PATH,
subprocess.run raises FileNotFoundError. Catch it and raise
a clear RuntimeError with guidance.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 22:11:19 +09:00
Gyeonghun ParkandClaude Sonnet 4.6 e98c975153 feat(learn): add CLI-based LLM backends for keyless headroom learn
Allow `headroom learn` to use locally installed coding agent CLIs
(claude, gemini, codex) as LLM backends, so subscription users
without raw API keys can run failure analysis.

Priority: --model flag > API key > HEADROOM_LEARN_CLI env var > auto-detect

- Pass prompts via stdin to avoid ARG_MAX limits
- Handle TimeoutExpired, truncate stderr, enrich JSONDecodeError
- Add 31 new tests (48 total), all passing
- Update docs/learn.md with CLI backend documentation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 22:02:06 +09:00
Tejas Chopra da481a359b fix(learn): pass explicit model in tests to avoid API key requirement
SessionAnalyzer() without a model calls _detect_default_model() which
raises when no API keys are set (e.g., in CI). Pass model="test-model"
in the three tests that mock _call_llm.
2026-03-07 15:10:25 -08:00
Tejas Chopra 4d14012c2f feat: add headroom perf CLI and rewrite headroom learn to use LLM analysis
Proxy performance logging (`headroom perf`):
- Add always-on RotatingFileHandler to ~/.headroom/logs/proxy.log (10MB x 5 backups)
- Replace scattered log lines with structured PERF lines containing model, msgs,
  tok_before/after/saved, cache_read/write/hit_pct, opt_ms, and transforms
- Emit PERF lines from all three response paths (streaming Anthropic, non-streaming
  Anthropic, Bedrock streaming)
- Add `headroom perf` CLI that parses proxy logs and reports token savings, cache
  hit rates, prefix stability, transform effectiveness, routing breakdown, TOIN
  status, and actionable recommendations
- Support --hours and --raw flags for time filtering and raw record output

Learn module rewrite (LLM-based analysis):
- Replace all regex/heuristic analyzers with a single LLM call via LiteLLM
- New SessionAnalyzer builds compact digests and sends to any of 100+ models
- Auto-detect best model from API keys (Anthropic → OpenAI → Gemini)
- Add --model flag for explicit model selection
- Enrich scanner with SessionEvent (user messages, interruptions, subagent summaries),
  token usage tracking, and timestamps
- Simplify models: remove EnvironmentFact, StructureNote, Correction, CommandPattern,
  RetryPattern, AnalysisReport; add SessionEvent, AnalysisResult
- Simplify writer: remove Recommender class (LLM now produces recommendations directly)
- Update tests for new analyzer and models
2026-03-07 14:15:49 -08:00
chopratejas 17442c2dcc Add headroom learn: offline failure learning for coding agents
Analyzes past conversation history to find tool call failure patterns,
correlates each failure with what eventually succeeded, and writes
specific project-level learnings to CLAUDE.md and MEMORY.md.

Key design:
- Success correlation: extracts the diff between failed and successful
  inputs as the learning (not generic advice)
- Generic architecture: tool-agnostic ToolCall model with pluggable
  Scanner/Writer adapters (Claude Code first, extensible to Cursor/Codex)
- 5 analyzers: Environment, Structure, Commands, Retries, Cross-Session
- Dry-run by default, --apply to write, --all for all projects

Also fixes mypy errors in litellm_callback, asgi, langchain chat_model,
and anthropic provider (AsyncClient typing, ToolCall arg-type, int cast).
2026-02-27 21:19:03 -08:00