mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
feat/https-compose-caddy
24
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
12864fc8f0 |
fix(learn): harden analyzer JSON parsing against malformed LLM output (#2471)
## Description
Two spots in the `learn` analyzer crash on JSON a model can
realistically emit. Both parse fully model-controlled output.
**1. `_strip_fenced_json` returns a non-object instead of raising.** It
is documented to return a dict and to raise `json.JSONDecodeError` when
no candidate is an object, but its final fallback was `result: dict =
json.loads(text); return result`. `json.loads` raises for invalid JSON,
but for valid JSON that is not an object (a bare array, string, or
number) it returns a non-dict. The `-> dict`-annotated value is then a
list/str, and all three callers do `.get(...)` on it, raising
`AttributeError` instead of the clean `JSONDecodeError` they already
handle.
**2. `_parse_llm_response` crashes on null rule lists/fields.** It read
`raw.get("context_file_rules", [])` / `raw.get("memory_file_rules", [])`
and `rule.get("section", "").strip()` / `rule.get("content",
"").strip()`. `.get(key, default)` only falls back when the key is
absent, so a present-but-null `"context_file_rules": null` made `for
rule in None` raise `TypeError`, and a null `section`/`content` made
`None.strip()` raise `AttributeError`.
## Fix
- `_strip_fenced_json`: reject a non-dict fallback result explicitly
(`raise json.JSONDecodeError(...)`), restoring the documented contract.
The lenient first-brace..last-brace slice still recovers a single object
a model wrapped in an array; only genuinely object-less JSON raises.
- `_parse_llm_response`: coerce the rule lists and the
`section`/`content` fields with `or` so a null is treated like an absent
value. Valid rules parse unchanged.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/learn/analyzer.py`: raise `JSONDecodeError` when
`_strip_fenced_json`'s fallback parse yields a non-dict; `or`-coerce
null rule lists and null `section`/`content` in `_parse_llm_response`.
- `tests/test_learn/test_analyzer.py`: regressions for non-object JSON
raising (and array-wrapped-object still extracted), and for null rule
lists / null fields being tolerated with valid rules still parsed.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_learn/test_analyzer.py::TestStripFencedJson tests/test_learn/test_analyzer.py::TestLLMResponseParser -q
# all pass; each new test fails with the corresponding fix reverted
# (JSONDecodeError contract / 'NoneType' object is not iterable / has no attribute 'strip')
$ uvx ruff@0.15.17 check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/learn/analyzer.py
Success: no issues found in 1 source file
```
Note: one unrelated digest-builder test
(`TestDigestBuilder::test_includes_project_info`) fails in my local
Windows venv because it hardcodes a `/tmp/...` path; it fails
identically on a clean `main` and is not touched by this change.
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: called the real `_strip_fenced_json` with
object-less JSON (`[1,2,3]`, `"str"`, `42`, ...) and valid/extractable
objects, and the real `_parse_llm_response` with `{"context_file_rules":
null}`, a rule with `section: null`, and a valid rule; then reverted
each fix and re-ran.
- Observed result: with the fixes the object-less inputs raise
`JSONDecodeError`, the null shapes yield an empty recommendation list,
and valid inputs parse as before; with each fix reverted the respective
input raises (`AttributeError: 'list' object has no attribute 'get'`,
`TypeError: 'NoneType' object is not iterable`, `AttributeError:
'NoneType' object has no attribute 'strip'`). Ran against the actual
module.
- Not tested: a live `headroom learn` run where the backing model
returns these shapes.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2471"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
### Maintainer current-main requalification
Exact head `27fa31eac507413415a00b54e29679bef08eabe9`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.
Clean current upstream/main0ddaa56c integration; production
_strip_fenced_json/_parse_llm_response/_as_str ASTs are identical to
contributor published head. Main-relative scope remains two files. Small
maintainer test-only repair uses str(project.project_path) instead of a
POSIX-only literal in existing digest assertion; source and test fixture
data unchanged. Initial full analyzer run109pass/1fail and separate
clean-main failure independently establish inherited Windows path
expectation before repair. After repair399tests pass,11skip,1warning
(all110analyzer cases plus289incoming-main regressions). Two changed
files pass Rufflint/format,candidate offline lock and whitespace pass.
Complete native inputset/allrecordedhashes/artifactchecksum verified
before existing3982DLL reuse; no fresh build.
Independent paired parser probes: branch passes both truthy non-string
section and objectless JSON contract cases; clean main fails with
AttributeError and missing JSONDecodeError respectively. These are
controlled parser executions, not live LLM quality/provider behavior.
Outstanding Devin truthy non-list rule-container report also
independently reproduced on BOTH branch and clean main:
context_file_rules:42 and memory_file_rules:42 raise TypeError in both.
This is an inherited limitation, not a PR-introduced regression or
grounds to withhold this narrower fix. Do not claim arbitrary-container
safety. The later author suggestion to join list-valued rule content
changes intended normalization semantics; not selected or implemented by
maintainer. Existing _as_str drops non-string fields, preserving
contributor intent.
Read author replies/current discussions and unresolved thread. No new
unreviewed external findings at prepublication audit. Source/test diff
contains no workflow/setup/dependency/native/CLI paths; central
publication gate passed. Exact published-head Actions/rollup, current
main ancestry/mergeability and fresh discussion remain before final
approval. Maintainers handle CI; no PR merged.
Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
506ba16c4a |
fix(learn): keep the digest's loop section inside its budget (#4269)
## Description
`headroom learn` can build a prompt larger than the CLI it runs will
accept, and the prompt-too-long retry cannot shrink it.
`_build_digest` budgets the per-session event stream (`max_tokens * 4`
chars), but it renders the "Detected Loops" section ahead of that stream
without any bound: every detected loop, each with every message index it
repeated at. Codex learn scans all of a machine's Codex sessions as one
project, so loops accumulate across every session the user ever ran. On
one Windows machine this produced a 1,786,526-char prompt, and `codex
exec` refused it on every scan:
```text
Error: turn/start: turn/start failed: Input exceeds the maximum length of 1048576 characters. (code -32602), data: {"input_error_code":"input_too_large","max_chars":1048576,"actual_chars":1786526}
```
Halving `max_tokens` on a prompt-too-long error does not help here: it
only trims the event stream, never the loop section.
This PR bounds the rendered loop section by the same budget, and lets a
Codex overflow take the smaller-digest retry.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `learn/loops.py`: `format_loops_for_digest` takes `max_chars` and
counts the rendered text, header and newlines included. It adds loops in
the given order (`detect_loops` sorts by measured waste) and stops at
the first one whose line would pass `max_chars`, so the costliest loops
are kept. Each line shows its sample input cut to 120 chars (the same
cut `detect_loops` makes) and its first 20 message indices beside the
full repeat count. It returns "" when no loop fits.
- `learn/analyzer.py`: `_build_digest` gives the loop section a quarter
of the `max_tokens * 4` char budget, less whatever the header and the
prior learned patterns leave short, so header + loops + prior patterns
stay inside the budget and the event stream fills the rest. The prior
patterns are never cut: a re-emitted section replaces its prior one
wholesale, so a partial baseline would drop bullets from the user's
file.
- `learn/analyzer.py`: the event stream counts every line with the
newline that joins it, and a session header, event or tool-call line
goes in only if it leaves 100 chars for the truncation markers that can
follow it. Once a session is cut, no later session is started. So the
whole digest stays inside `max_tokens * 4`; the only exception is a
prior-patterns section that leaves no room on its own, since it is never
cut.
- `learn/analyzer.py`: `_is_prompt_too_long` also matches Codex's "Input
exceeds the maximum length" refusal, so a Codex overflow is retried at
half the budget like the other backends.
- The caller's `LoopPattern`s are not touched (slices only), and
`apply_loop_weighting` still receives every loop.
- Tests: `test_loop_section_stays_inside_the_digest_budget` is
parametrized over a 120-char Bash input, a 2,000-char Grep pattern and a
2,000-char Glob path, at the 80k and 10k (retry floor) budgets. New
`test_loops_fit_beside_a_large_prior_patterns_section` (80k, 10k). New
`test_whole_digest_stays_inside_the_budget`: 5 sessions of 3,000 failing
calls each, at 80k and 10k, as tool calls and as events, with and
without loops, asserts `len(digest) <= max_tokens * 4`. New
`test_many_tiny_sessions_stay_inside_the_budget`: 20,000 one-call
sessions at 80k and 10k, as tool calls and as events (614 to 4,862
sessions kept), so the blank line closing each session is counted too.
`test_retries_smaller_digest_on_prompt_too_long` adds the real Codex
error text.
- `pyproject.toml` / `uv.lock` (review: Dependency audit): the `aiohttp`
constraint floor is now `>=3.14.4` (CVE-2026-111063, CVE-2026-111064 and
GHSA-2g87-pp6c-29x5, all fixed in 3.14.4), relocked with `uv lock
--upgrade-package aiohttp==3.14.4`. Only aiohttp (3.14.3 -> 3.14.4) and
yarl (1.22.0 -> 1.25.1, which 3.14.4 requires) move. Main still resolves
3.14.3, so the same commit is on #4299, #4286, #4285 and #4269.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
`mypy` passes on the changed module.
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_learn/ -q
384 passed, 4 skipped in 34.16s
$ uvx ruff@0.17.0 check headroom/learn/analyzer.py headroom/learn/loops.py tests/test_learn/test_loop_weighting.py tests/test_learn/test_analyzer.py
All checks passed!
$ uvx ruff@0.17.0 format --check headroom/learn/analyzer.py headroom/learn/loops.py tests/test_learn/test_loop_weighting.py tests/test_learn/test_analyzer.py
4 files already formatted
$ uv run --frozen --extra dev mypy headroom/learn/analyzer.py headroom/learn/loops.py
Success: no issues found in 2 source files
```
With `headroom/learn/` reverted to
|
||
|
|
abbf36f052 |
refactor(memory,learn,cache): remove unreferenced classes, methods and constants (#4112)
## Description This removes nine symbols in `headroom/memory`, `headroom/learn` and `headroom/cache` that nothing references outside their own definition. One is a legacy alias whose only user was its own test. Evidence: for each name I ran `grep -rnw <name>` across `headroom crates plugins sdk docs wiki scripts benchmarks examples e2e .github pyproject.toml tests` and the org's consumer repos. The only hit was the definition, plus the `FailureAnalyzer` alias test removed here. None of the names is public API. None is in an `__all__`, a re-export or lazy export map, an entry-point group or the docs, and no consumer imports it. Refs #4136 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - `memory/inline_extractor.py`: remove `InlineMemoryWrapper`, with `chat` and `chat_with_response`. `with_memory` (`memory/wrapper.py`) uses only the module's functions, `inject_memory_instruction` and `parse_response_with_memory`, and those stay. - `memory/traffic_learner.py`: remove these: - `AgentType`; - `_COMMAND_RE` and `_FILE_PATH_RE`. `_MODULE_RE` and `_COMMAND_NF_RE` next to them are used and stay; - `TrafficLearner.get_learned_patterns`. Its docstring said "retained for backwards compatibility", but `TrafficLearner` is internal to the proxy. - `memory/core.py`: remove the private `HierarchicalMemory._scope_level_value`. - `cache/compression_feedback.py`: remove the private `CompressionFeedback._truncate_strategy_dicts`. - `learn/analyzer.py`: remove `FailureAnalyzer`. Its docstring said "used by existing CLI code", but `cli/learn.py` uses `SessionAnalyzer` directly. Also remove `TestFailureAnalyzerCompat` from `tests/test_learn/test_analyzer.py`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q -p no:cacheprovider tests/test_memory_wrapper.py tests/test_inline_extractor_inject.py tests/test_memory/test_traffic_learner.py tests/test_openai_responses_traffic_learner.py tests/test_memory/test_core_operations.py tests/test_learn/test_analyzer.py tests/test_ccr_feedback.py tests/test_memory/test_hierarchical.py tests/test_toin_fixes.py ================ 417 passed, 8 skipped, 772 warnings in 49.26s ================= $ mypy <5 changed product files> --ignore-missing-imports Success: no issues found in 5 source files $ uvx ruff@0.16.8 check <changed files> All checks passed! ``` ## Real Behavior Proof - Environment: macOS, Python 3.12.13, repo .venv, branch checkout at |
||
|
|
4ba231ab3d |
feat(learn): add agy (Antigravity CLI) as an analysis backend (#3939)
## Description
Adds the Antigravity CLI (`agy`) as a keyless LLM backend for `headroom
learn`, next to `claude`, `gemini` and `codex`. Antigravity subscription
users with no raw API key can now run the analysis through `agy`.
`agy` cannot use the existing stdin path as is. `agy -p` takes the
prompt as a flag argument and never reads stdin, so a large digest would
hit OS argument-length limits. The only stdin path is `--input-format
stream-json`, which requires `--output-format stream-json`. The backend
therefore sends the prompt as a single NDJSON user event on stdin and
reads the answer from `result.response` in the final `result` event.
Everything else (wall-clock timeout, Windows shim fallback, failure
detail, fenced-JSON parsing) reuses the existing `run()` path that
`gemini` and `codex` use.
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/learn/analyzer.py`: add an `agy` / `agy-cli` entry to
`_CLI_BACKENDS` (`agy --input-format stream-json --output-format
stream-json`). `agy` is never auto-detected, and it only runs when
`HEADROOM_LEARN_ALLOW_UNSAFE_AGY=1` is set: `agy` has no zero-tool mode,
so transcript text could drive its tools. Without the opt-in,
`HEADROOM_LEARN_CLI=agy` and `--model agy-cli` fail before any
subprocess starts; with it, a warning is logged on every run.
- `headroom/learn/analyzer.py`: in `_call_cli_llm`, wrap the prompt as
`{"event": "user", "message": {"content": ...}}` for `agy-cli`, and
extract `result.response` from the stream with a new
`_agy_result_text()` helper before the existing `_strip_fenced_json()`
parsing. A missing `result` event surfaces as the existing "unparseable
output" error, with both ends of stdout.
- `tests/test_learn/test_analyzer.py`: tests that auto-detection never
selects `agy` (even when it is the only CLI on PATH and the opt-in is
set), that `HEADROOM_LEARN_CLI=agy` and `_call_cli_llm(..., "agy-cli")`
fail without the opt-in (no subprocess call), `HEADROOM_LEARN_CLI=agy`
with the opt-in, existing CLIs still taking precedence, the exact
command and stdin payload, result-event parsing (including a fenced
response and a literal U+2028), and a missing `result` event.
- `headroom/cli/learn.py`: check the agy opt-in during model resolution,
so `HEADROOM_LEARN_CLI=agy` and `--model agy-cli` without it exit 1 with
an `Error:` line before any sessions are scanned.
- `tests/test_cli_learn.py`: a CLI test for both paths without the
opt-in.
- `wiki/learn.md`: list `agy` among valid `HEADROOM_LEARN_CLI` values
and document the `HEADROOM_LEARN_ALLOW_UNSAFE_AGY=1` opt-in and its
risk.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_learn/ tests/test_cli_learn.py -q
5 failed, 345 passed, 4 skipped in 10.78s
# The same 5 fail on unmodified upstream/main on this Windows machine (5 failed, 340 passed),
# all path/git related and unrelated to this change:
# test_analyzer.py::TestDigestBuilder::test_includes_project_info
# test_scanner.py::TestDecodeProjectPath::test_home_dir_username_stays_single_component
# test_writer.py::TestClaudeLocalMdStaysOutOfGit::{test_apply_adds_exclude_entry,
# test_second_run_does_not_duplicate_the_entry, test_tracked_file_warns_instead_of_excluding}
$ uv run --frozen --extra dev pytest tests/test_learn/test_analyzer.py -q -k agy # analyzer.py reverted
FAILED TestDetectDefaultModel::test_env_var_selects_agy_with_unsafe_opt_in
FAILED TestDetectDefaultModel::test_env_var_agy_without_unsafe_opt_in_raises
FAILED TestCallCliLlm::test_agy_cli_sends_stream_json_and_parses_result_event
FAILED TestCallCliLlm::test_agy_cli_result_with_unicode_line_separator
FAILED TestCallCliLlm::test_agy_cli_without_unsafe_opt_in_does_not_run
FAILED TestCallCliLlm::test_agy_cli_missing_result_event_raises
6 failed, 2 passed, 97 deselected
# All 8 pass with the change.
$ uvx ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ uvx ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/learn/analyzer.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, `agy` 1.2.14 signed in via subscription, no
API keys set.
- Exact command / steps: `HEADROOM_LEARN_CLI=agy
HEADROOM_LEARN_ALLOW_UNSAFE_AGY=1 ANTHROPIC_API_KEY= OPENAI_API_KEY=
GEMINI_API_KEY= uv run --frozen --extra dev headroom learn` in this
repository (dry run).
- Observed result: `Analyzing with agy-cli...`, then `Sessions: 3 |
Calls: 109 | Failures: 19 (17.4%)`, `Recommendations: 8`, with `[WOULD
WRITE]` blocks for `CLAUDE.local.md` and `MEMORY.md`; exit 0. Before
wiring the backend I confirmed the protocol by hand: `agy -p -` answers
the literal prompt `-` and ignores stdin, while the stream-json
invocation returns the answer in `{"event": "result", "result":
{"status": "SUCCESS", "response": "..."}}`.
- Not tested: macOS/Linux `agy` installs; `agy` under a permission mode
that prompts for tool approval (my config is `always-proceed`), where a
tool request in print mode would be bounded by the existing
`HEADROOM_LEARN_CLI_TIMEOUT_SECS` wall-clock cap.
## Runtime Rollout Safety
- Rollout-managed feature(s): None (offline `headroom learn` CLI)
- Minimum rollout channel: dev
- Stable/default behavior changed: No. `agy` is never auto-detected; it
is only used when selected with `HEADROOM_LEARN_CLI=agy` / `--model
agy-cli` together with `HEADROOM_LEARN_ALLOW_UNSAFE_AGY=1`.
- Kill switch / disable path: Set an API key, `HEADROOM_LEARN_CLI`, or
`--model` to pick another backend.
- Unsafe override required: None
- Qualification impact: None
- Rollback path: Revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A (CLI change).
## Additional Notes
- No tracking issue exists for an `agy` analysis backend, so `Closes #`
is left empty. Related Antigravity work (#3428 `headroom init -g agy`,
#3610 Antigravity transcripts in `headroom learn`) is independent of
this change.
- Unlike `claude-cli` (#3892), `agy` has no flag to start a session with
zero tools. I deliberately did not add `--dangerously-skip-permissions`:
the digest contains untrusted transcript content, and the backend should
run under the user's own permission settings.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3939"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
f00d425614 |
fix(learn): drop the echoed prompt from a failed cli's error (#3928)
## Description
`codex exec` echoes the whole stdin prompt to stderr ahead of its own
error, and a failed CLI's error message keeps the first 2000 chars of
stderr. The analyzer's system prompt alone is 4116 chars, so every
codex-cli failure reported Codex's banner and the start of our system
prompt and never the reason (a 401, an unknown model, a rate limit). The
echo also carries the session digest, which does not belong in an error
message.
`_failure_detail` now takes the prompt that was sent on stdin. When
stderr contains it, the echo is replaced with `[prompt omitted]` and the
end of what the CLI printed after it is kept, cut at a line boundary,
inside the same 2000-char cap. Stderr from a CLI that does not echo the
prompt is handled exactly as before.
Closes #3927
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/learn/analyzer.py`: `_failure_detail` gets a keyword-only
`prompt` argument; when the prompt (newlines normalized the way a
text-mode pipe reads them back) appears in stderr, it is replaced with a
marker and the text after it is tailed to fit the cap, whole lines where
possible.
- Both callers pass the prompt they wrote to stdin: the `run(...)` path
used by codex-cli and gemini-cli, and the claude-cli streaming path
(claude does not echo today, so nothing changes there).
- `tests/test_learn/test_analyzer.py`:
`test_codex_failure_drops_prompt_echo_and_keeps_reason` feeds stderr in
the shape codex-cli 0.156.1 prints (banner, `user`, the prompt, 100
reconnect lines, the final error) and checks the message ends with the
real reason, carries the marker, has no digest text, and stays inside
the cap.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Ran on the changed files only (`ruff check`/`ruff format --check` on the
two changed files, `mypy headroom/learn/analyzer.py`), not the whole
tree.
### Test Output
Before the fix (new test against upstream/main's analyzer):
```text
$ pytest tests/test_learn/test_analyzer.py -q -k prompt_echo
E AssertionError: assert False
E + where False = <built-in method endswith of str object at 0x77522a6400>('ERROR: unexpected status 401 Unauthorized: Missing bearer')
E + where <built-in method endswith of str object at 0x77522a6400> = '`codex exec --skip-git-repo-check` failed (exit 1):\nReading prompt from stdin...\nOpenAI Codex v0.156.1\n--------\nm...endation (how many tokens would be saved per session if this rule existed)\n- Separate stable project facts (CONTEXT_F'.endswith
```
After:
```text
$ pytest tests/test_learn/test_analyzer.py tests/test_learn/test_error_classification.py -q
============================= 102 passed in 4.73s ==============================
$ ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
2 files already formatted
$ mypy headroom/learn/analyzer.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS 27, Python 3.12, codex-cli 0.156.1 (Homebrew),
upstream/main
|
||
|
|
2a4d34c476 |
fix(learn): run the claude-cli analysis with hooks disabled (#3926)
## Description
`headroom learn`'s claude-cli backend runs `claude -p` as the user, so
the user's hooks (from settings files and installed plugins) run inside
the analysis session. #3892 took the session's tools away; this takes
its hooks away. A blocking Stop hook makes Claude take a second turn
whose reply becomes the final `result`, replacing the analysis, so learn
finds no rules and prints "No actionable patterns found." with exit 0
and no error.
The fix passes `--settings '{"disableAllHooks":true}'`. `--settings` is
merged on top of the user's own settings, which still load, so env
routing (`ANTHROPIC_BASE_URL` in settings) and OAuth keep working.
`--bare` would also skip hooks but never reads OAuth or the keychain.
Closes #3925
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/learn/analyzer.py`: add `--settings
'{"disableAllHooks":true}'` to the claude-cli entry in `_CLI_BACKENDS`,
with a comment on why hooks must be off and why `--bare` is not used.
- `tests/test_learn/test_analyzer.py`:
`test_claude_cli_streams_and_parses_result_event` pins the new argv.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
Ran on the changed files only (`ruff check`/`ruff format --check` on the
two changed files, `mypy headroom/learn/analyzer.py`), not the whole
tree. The existing argv test was updated rather than a new one added.
### Test Output
Before the fix (updated test against upstream/main's analyzer):
```text
$ pytest tests/test_learn/test_analyzer.py -q -x -k test_claude_cli_streams_and_parses_result_event
E AssertionError: assert ['claude', '-...essages', ...] == ['claude', '-...essages', ...]
E Right contains 2 more items, first extra item: '--settings'
FAILED tests/test_learn/test_analyzer.py::TestCallCliLlm::test_claude_cli_streams_and_parses_result_event
======================= 1 failed, 96 deselected in 3.35s =======================
```
After:
```text
$ pytest tests/test_learn/test_analyzer.py tests/test_learn/test_error_classification.py -q
============================= 101 passed in 4.19s ==============================
$ ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
2 files already formatted
$ mypy headroom/learn/analyzer.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS 27, Python 3.12, Claude Code 2.1.287 logged in with
a subscription (OAuth), upstream/main
|
||
|
|
db90b93da9 |
fix(learn): run the claude-cli analysis with no tools (#3892)
## Description `headroom learn`'s claude-cli backend runs the analysis as `claude -p`, and a headless `claude -p` is a full Claude Code agent. With Claude Code 2.1.285, its init event lists 127 tools from 7 MCP servers: Bash, Edit, Task, cron, every claude.ai connector. They run under the user's own permissions, hooks and skills. The analysis needs none of them, because the digest is the whole input and the answer is one JSON object. When the model decides to go read or verify things, the run turns multi-turn. Each tool call emits stream events, so the idle cap never fires, and the run is killed at the hard cap with nothing written. We saw this in the field as `claude -p --output-format stream-json --verbose exceeded the 900s hard cap` (Headroom Desktop raises the hard cap to 900s). A normal analysis finishes in seconds to under a minute in a single turn. This adds `--tools ""` and `--strict-mcp-config` to the claude-cli command, so the session has zero tools and answers in one turn. You need both flags: `--tools ""` alone removes the built-in tools but still leaves every MCP server's tools. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/learn/analyzer.py`: the `claude-cli` entry of `_CLI_BACKENDS` now also passes `--tools "" --strict-mcp-config`, with a comment explaining why. - `tests/test_learn/test_analyzer.py`: `test_claude_cli_streams_and_parses_result_event` pins the new command. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_learn/ ======================= 320 passed, 4 skipped in 22.78s ======================== $ uvx ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py All checks passed! $ uvx ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py 2 files already formatted $ uv run --frozen --extra dev mypy headroom/learn/analyzer.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS 15 (Darwin 24.6), Claude Code 2.1.285 logged in with OAuth, user settings with plugins, hooks and claude.ai MCP connectors, this branch run through `uv run`. - Exact command / steps: (1) Piped `hi` into the old and the new claude-cli command and read the `system`/`init` stream event. (2) Called `_call_cli_llm(<small two-call digest>, "claude-cli")` against the real CLI. - Observed result: (1) The old command's init event listed 127 tools and 7 MCP servers; the new command's listed 0 tools and 0 MCP servers. (2) The analysis returned parsed JSON with `context_file_rules` and `memory_file_rules` in 8.5s. - Not tested: Windows (the command is still a list passed to Popen, and the empty `--tools` value is a separate argv entry); Claude Code versions without `--tools`, which will fail with `unknown option '--tools'` and need a `claude update`. ## Runtime Rollout Safety - Rollout-managed feature(s): none; this only changes the learn claude-cli command. - Minimum rollout channel: N/A, not rollout-managed. - Stable/default behavior changed: yes, the learn analysis session runs with no tools. The recommendations come from the digest alone, as the prompt already asks. - Kill switch / disable path: `--model <litellm-model-name>`, or the gemini-cli or codex-cli backends, skip this command entirely. - Unsafe override required: no. - Qualification impact: none; learn is not in the proxy request path. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Documentation: no docs describe the claude-cli argv, so there was nothing to update. - The existing command test was updated rather than a new one added; it asserts the exact argv. - Stop hooks that block can still make a `claude -p` run multi-turn. Only Claude Code's `--safe-mode` disables hooks under OAuth (`--bare` needs an API key), and that flag is too new to pass unconditionally. - Headroom Desktop carries this as a sitecustomize patch until a wheel with it ships. The patch does nothing when the wheel already passes either flag. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
cb3ab61d4d |
fix(learn): show both ends of CLI output on a parse failure (#3517)
## Description
`_call_cli_llm` and `_call_claude_cli_streaming` attach a HEAD-only
excerpt of
the CLI's output to their parse failures ("returned unparseable output.
First
2000 chars:", "did not emit a final `result` event"). For those two
failures
specifically, the head is the one part that is never diagnostic: a
model's JSON
answer that does not parse looks like valid JSON for its first 2000
chars in
every case. What separates a payload truncated mid-value from one with
prose
after the closing fence is the END of it, and that is exactly what gets
cut.
This is not hypothetical. Downstream (the Headroom desktop app) these
RuntimeErrors are reported with the CLI's stderr attached, and a real
report
arrived with 2000 characters of well-formed ```json ... and no way to
tell which
of the two shapes it was, so the defect could not be acted on at all.
`_failure_detail` already makes this argument for the non-zero-exit path
("stdout
is tailed rather than headed because CLI backends emit the error last").
This
extends the same reasoning to the parse failures, which need both ends:
the head
is where a wrapper (fence, preamble) shows up, the tail is where the
break is.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Added `_output_snippet(text, limit=_MAX_SNIPPET_LEN)`: returns short
output
unchanged, otherwise `head + "...[N chars omitted]..." + tail`, still
bounded
by the same `_MAX_SNIPPET_LEN` budget.
- Used it at the three sites that excerpt CLI OUTPUT for a parse
failure: the
non-streaming unparseable stdout, the missing final `result` event, and
the
unparseable `result` text.
- Reworded those messages from "First 2000 chars:" to "Head and tail of
the
output:". Deliberately no counts in the marker line: downstream groups
these
errors by that line, so a length in it would open a new group per
failure.
- Left the stderr excerpts and `_failure_detail` alone.
## Testing
- [x] Unit tests pass (`pytest`)
### Test Output
```text
uv run --frozen --extra dev pytest tests/test_learn/ -q
260 passed, 4 skipped in 11.02s
```
## Real Behavior Proof
- Environment: macOS 15, Python 3.12, uv --frozen, headroom main @
|
||
|
|
4c6bd3e8fc |
fix(learn): echo live progress during claude-cli analysis (#3158)
## Description `headroom learn` already streams claude-cli's `stream-json` output internally, using it to drive an idle-timeout watchdog, but none of that liveness reaches stdout. Wrapper UIs that tail the CLI output (the Headroom desktop app included) see nothing print between "Analyzing with claude-cli..." and the final result, sometimes for minutes, which makes the scan look hung. This closes #3105 by surfacing that liveness as short, throttled progress lines. ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) ## Changes Made - Added an optional `on_progress: typing.Callable[[str], None] | None = None` callback, threaded through `SessionAnalyzer.analyze` -> `_call_llm` -> `_call_cli_llm` -> `_call_claude_cli_streaming`. It defaults to `None` at every layer, so no existing caller changes behavior. - Inside the streaming event loop, added `_progress_detail()`, which maps each non-terminal `stream-json` event (`system`/`assistant`/`stream_event`/`user`) to a short phrase like `"assistant responding, 12s"`. The terminal `result` event is handled separately and is never echoed as progress. - Throttled the callback to at most once per 3 seconds (`_PROGRESS_THROTTLE_SECS`) via the same `time.monotonic()` idiom the function already used for its idle-timeout watchdog, so a burst of partial-message events doesn't spam the console. - The callback is invoked defensively: any exception it raises is caught and logged via `logger.debug`, never propagated, so a broken UI pipe can't abort an otherwise-successful analysis. - `headroom/cli/learn.py` supplies the one concrete callback, reusing the exact existing `" Analyzing with {model}..."` prefix with `(detail)` appended, so any wrapper UI that whitelists that prefix keeps working unchanged. - Added `--include-partial-messages` to the claude-cli invocation, as suggested in the issue. Without it, claude only emits about three events for an entire run (session init, one assistant message, result), which is too sparse for the throttle to produce useful incremental updates. ## Testing - [x] Added new tests - [x] All tests pass locally ``` $ .venv/Scripts/python.exe -m pytest tests/test_learn/test_analyzer.py tests/test_cli_learn.py -q tests\test_learn\test_analyzer.py ....................................... ...................................................... [ 84%] tests\test_cli_learn.py ................. [100%] 109 items collected, 108 passed, 1 failed FAILED tests/test_learn/test_analyzer.py::TestDigestBuilder::test_includes_project_info (pre-existing, unrelated: a Windows-path-separator assertion that also fails identically on a clean upstream/main checkout, before this change) $ .venv/Scripts/ruff.exe check headroom/learn/analyzer.py headroom/cli/learn.py tests/test_learn/test_analyzer.py tests/test_cli_learn.py All checks passed! $ .venv/Scripts/ruff.exe format --check headroom/learn/analyzer.py headroom/cli/learn.py tests/test_learn/test_analyzer.py tests/test_cli_learn.py 4 files already formatted $ .venv/Scripts/mypy.exe headroom/learn/analyzer.py headroom/cli/learn.py Success: no issues found in 2 source files ``` Three new tests were added covering: a burst of events within one throttle window only echoing the first one, spaced-out events each mapping to the correct phrase in order (with the terminal `result` event never echoed), and a raising `on_progress` callback not aborting a successful analysis. A fourth new test at the CLI layer asserts the `" Analyzing with ... (detail)"` lines appear in `headroom learn` output. ## Real Behavior Proof - Environment: Windows 11, Python 3.13.11, local `.venv`, no live `claude` CLI invoked (tests mock `subprocess.Popen`, matching the existing test style in this file) - Exact command / steps: `.venv/Scripts/python.exe -m pytest tests/test_learn/test_analyzer.py tests/test_cli_learn.py -v -k progress` - Observed result: all 4 new progress-related tests pass, confirming the throttling, event-to-phrase mapping, defensive-callback behavior, and the CLI's rendered output line all work as designed - Not tested: an actual live `claude` CLI process (no `claude` binary available in this sandboxed environment); the fix only changes what data is threaded through an existing, already-live streaming loop, so this is low risk, but a manual check against the real CLI before/after would be good follow-up validation ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: no — `on_progress` defaults to `None` everywhere, and when `None` the code path is unchanged from before this PR - Kill switch / disable path: none needed; the only new default-on behavior is the extra progress lines in `headroom learn` CLI output, which is additive to existing output (same prefix, one line appended per event) and does not change any parsed/machine-readable output - Unsafe override required: no - Qualification impact: none — no changes to routing, caching, or backend selection logic - Rollback path: revert this commit; nothing persists state or changes on-disk formats ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Jerrett Davis <mxjerrett@gmail.com> |
||
|
|
d329a04b4b |
fix(learn): retry LLM analysis with a smaller digest when the prompt overflows the context window (#3404)
## Description
`headroom learn` dies with zero recommendations when the analysis prompt
overflows the model's context window. Field report (desktop telemetry):
a 4-session / 443-call scan failed with ``LLM analysis failed: `claude
-p --output-format stream-json --verbose` failed (exit 1): Prompt is too
long``, and the whole project's analysis was lost.
The root cause is that `_build_digest` budgets `_MAX_DIGEST_TOKENS`
(80k) for the digest as if the prompt had the window to itself. CLI
backends do not work that way: `claude -p` loads the user's own context
on top - CLAUDE.md files, MCP tool definitions, skills. The digest
budget cannot know how big that share is, and the 4-chars-per-token
estimate also runs hot on dense tool-call lines. On installs with heavy
context the combined prompt overflows a 200k window and the analysis
fails outright, every scan, until the user's sessions age out.
The fix makes the budget adaptive instead of guessing a universally safe
constant: when the backend reports a context overflow, rebuild the
digest at half the budget and retry, down to a 10k-token floor (80k ->
40k -> 20k -> 10k, then give up). A truncated digest that produces
recommendations beats a full digest that produces an error. Non-overflow
failures are re-raised on the first attempt exactly as before.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `SessionAnalyzer.analyze` wraps the LLM call in a shrinking-budget
retry loop; overflow detection via new `_is_prompt_too_long(exc)`, which
matches the known phrasings across backends: claude CLI ("Prompt is too
long"), Anthropic via LiteLLM ("prompt is too long: N tokens > M
maximum"), OpenAI ("context_length_exceeded" / "maximum context
length"), Gemini ("input token count exceeds the maximum").
- `_build_digest` gains a `max_tokens` parameter (default
`_MAX_DIGEST_TOKENS`, so all other callers are unchanged); the char
budget derives from it.
- New `_MIN_DIGEST_TOKENS = 10_000` floor so a persistent overflow
terminates after 4 attempts instead of looping.
- Tests: retry produces a strictly shorter second digest and succeeds;
persistent overflow stops after the 10k floor (4 calls) and returns
stats-only; non-overflow failures still fail on the first call with no
retry.
## Testing
- [x] Unit tests pass (`pytest`)
### Test Output
```text
uv run --frozen --extra dev pytest tests/test_learn/ -q
216 passed, 3 skipped in 8.18s
```
## Real Behavior Proof
- Environment: macOS (arm64), Python 3.12 via uv, this branch's
worktree, real `claude` CLI installed.
- Exact command / steps: Drove the real CLI through `_call_cli_llm` with
oversized digests to verify the trigger end-to-end.
- Observed result: Could not force the overflow on this machine - the
account resolves to a 1M-context model, so even a ~950k-token prompt
succeeds (confirmed via the stream-json `result` event's `contextWindow:
1000000`). The failure is real on 200k-window installs: the field
telemetry's stderr carries `Prompt is too long` verbatim, which is the
exact string `_is_prompt_too_long` matches and the exact `RuntimeError`
message the retry test injects through the real `analyze` -> `_call_llm`
path.
- Not tested: a live end-to-end overflow against a 200k-window account
(none at hand); the gemini/codex CLI phrasings are covered by the
pattern list, not live runs.
## Runtime Rollout Safety
- Rollout-managed feature(s): None; `learn` is not rollout-gated.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Only on the previously-failing path:
an overflow now retries with a smaller digest instead of aborting the
analysis. Successful analyses are byte-identical (same budget, same
digest).
- Kill switch / disable path: Not needed; a persistent overflow degrades
to today's behavior (stats-only result, warning logged) after 4
attempts.
- Unsafe override required: No.
- Qualification impact: Heavy-context installs (large CLAUDE.md, many
MCP servers) go from 100% learn failure to a truncated-digest analysis.
- Rollback path: Revert the commit; no persisted formats touched.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
632cb81dbe |
fix(learn): surface Codex analysis failures (#3016)
## Description `headroom learn` could invoke Codex CLI from a non-Git working directory without Codex’s required bypass flag. The resulting backend error was then swallowed by the analyzer and rendered as “No actionable patterns found” with exit code 0. This fixes both coupled defects so Codex can run from discovered project locations and genuine analysis failures remain visible and machine-detectable. Closes #3008 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `--skip-git-repo-check` to the Codex CLI analysis backend command. - Added an explicit `analysis_error` result field instead of conflating backend failure with an empty recommendation set. - Kept multi-project analysis best-effort, while returning exit code 1 after any project analysis fails. - Prevented failed analysis from printing a misleading no-pattern success message. - Added analyzer and CLI regression coverage for the command and failure-propagation contracts. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run pytest -q tests/test_learn/test_analyzer.py tests/test_cli_learn.py 102 passed in 2.34s uv run pytest -q tests/test_learn tests/test_cli_learn.py 257 passed, 7 skipped in 3.11s uv run mypy headroom Success: no issues found in 520 source files uv run ruff check <changed files> All checks passed! uv run ruff format --check <changed files> 5 files already formatted uv run pytest tests scripts/tests --splits 4 --group N --tb=short -q shard 1: 2766 passed, 140 skipped in 174.08s shard 2: 2699 passed, 207 skipped in 60.00s shard 3: 2822 passed, 84 skipped in 76.10s shard 4: 2734 passed, 172 skipped in 80.29s ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.13, Codex CLI 0.147.0-compatible command surface, current `main` including #2996. - Exact command / steps: verified `codex exec --help`; exercised `_call_cli_llm` with a captured subprocess command; invoked the Click command with a simulated Codex nonzero backend result. - Observed result: the subprocess command is `codex exec --skip-git-repo-check`; backend failure text is printed as `Analysis failed`, the misleading no-pattern message is absent, and the CLI exits 1. - Not tested: live paid Codex analysis against production account credentials; subprocess and CLI behavior are covered deterministically. ## Runtime Rollout Safety - Rollout-managed feature(s): none; this is CLI-only failure handling. - Minimum rollout channel: normal patch release after exact-head CI is entirely green. - Stable/default behavior changed: failed LLM analysis now exits nonzero instead of reporting success; successful and genuinely empty analyses are unchanged. - Kill switch / disable path: select another backend with `HEADROOM_LEARN_CLI` or `--model` if Codex CLI is unavailable. - Unsafe override required: none. - Qualification impact: all four Python CI shards, static checks, security checks, and command-level regression tests must pass. - Rollback path: fix forward through a human-reviewed corrective PR; no persisted data or migration is involved. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project’s style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation — inline result-contract documentation; no separate user guide change is required - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable; command-line backend and exit semantics only. ## Additional Notes Human review only. No merge or auto-merge is configured. This corrects the root failure and exit semantics without extending any timeout. |
||
|
|
c5563d3a7d |
fix(learn): include stdout in CLI failure messages, not just stderr (#3080)
## Description `headroom learn` reports CLI backend failures using **stderr only**. `claude -p --output-format stream-json --verbose` writes *nothing* to stderr when the run fails at the API layer, so the failure a user actually sees is a message that stops at the colon: ```text LLM analysis failed: `claude -p --output-format stream-json --verbose` failed (exit 1): ``` The reason is not missing, it is discarded. Claude Code still emits a final `result` event on stdout whose `result` field is the human-readable cause, and the streaming path has already parsed it into `final_result` one line above the `raise`. This makes a whole class of failures undiagnosable for users and maintainers alike: a usage limit, an unreachable local proxy, and an expired login all render identically as an empty message. Reported by a desktop user who could only tell us "sometimes i have this LLM analysis failed" with nothing after the colon. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Add `_failure_detail(stderr, stdout, *, result_text=None)` in `headroom/learn/analyzer.py`. Prefers the already-parsed `result` text, falls back to the **tail** of stdout (CLI backends emit the error last, after their whole event log), keeps stderr when present, and returns `"(no output captured)"` so the message is never a dangling colon. - Use it in `_call_claude_cli_streaming` (streaming claude-cli path) and in `_call_cli_llm` (the `subprocess.run` backends, gemini-cli / codex-cli), so the same blind spot is closed for every CLI backend rather than only the one that was reported. - Existing truncation behaviour is unchanged: each stream is still capped at `_MAX_SNIPPET_LEN`. Complements #3016, which makes an analysis failure propagate instead of being swallowed as success; that PR fixes *whether* the user learns a failure happened, this one fixes *what* the failure says. No overlapping lines. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_learn/ -q 247 passed, 4 skipped in 27.08s $ uvx ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py All checks passed! $ uvx ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py 2 files already formatted $ uv run --frozen --extra dev mypy headroom/learn/analyzer.py Success: no issues found in 1 source file ``` New tests: `test_claude_cli_nonzero_exit_includes_api_error_from_stdout`, `test_claude_cli_nonzero_exit_with_no_output_says_so`, `test_claude_cli_nonzero_exit_keeps_stderr_when_present`, `test_codex_nonzero_exit_includes_stdout_when_stderr_empty`. ## Real Behavior Proof - Environment: macOS 15.6 (Darwin 24.6.0), Claude Code 2.1.228, Python 3.10.18, headroom on this branch. - Exact command / steps: forced an API-layer failure in the exact command the analyzer runs, capturing the streams separately: `echo "say hi" | claude -p --output-format stream-json --verbose --settings '{"env":{"ANTHROPIC_BASE_URL":"http://127.0.0.1:9"}}' > out.txt 2> err.txt; echo "EXIT=$?"; wc -c err.txt; tail -c 400 out.txt` - Observed result: `EXIT=1`, `err.txt` is **0 bytes**, and the reason appears only in the last stdout line: `"terminal_reason":"api_error", ..., "result":"API Error: Connection refused — a firewall or proxy may be blocking it (ConnectionRefused)"`. A second run with `--bare` produced the same shape with `"result":"Not logged in · Please run /login"`. Before this change both surface as `failed (exit 1):` with nothing after the colon; after it, the `result` text is in the message. The unit tests encode this exact stream shape (stdout `result` event, empty stderr, exit 1). - Not tested: real usage-limit and 429 responses, which I cannot provoke on demand. They travel the same code path as the reproduced `api_error` case (final `result` event on stdout, empty stderr), so they are covered by construction rather than by observation. Windows and the gemini-cli backend were not exercised manually; the shared helper is covered by unit tests for both the streaming and `subprocess.run` paths. ## Runtime Rollout Safety - Rollout-managed feature(s): None. This touches only the error text raised by `headroom learn`'s CLI backends; no rollout-gated feature, flag, or runtime component is involved. - Minimum rollout channel: N/A, not rollout-gated. Ships with the package like any other library fix. - Stable/default behavior changed: Yes, narrowly. The message text of an existing `RuntimeError` on a non-zero CLI exit now includes the stdout/`result` reason alongside stderr. No control flow, exit code, public API, or return value changes: the same exception is raised in the same cases. - Kill switch / disable path: None needed. Nothing is enabled or newly executed, so there is nothing to switch off; the only behavioral surface is the string inside an exception that was already being raised. - Unsafe override required: No. - Qualification impact: None. No qualification-gated path, model, or provider behavior is touched. Callers that pattern-match this message on `"failed (exit N)"` still match, since that prefix is unchanged. - Rollback path: Revert this commit. The previous stderr-only message returns with no migration, state, or config to undo. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
85e8699451 |
fix(learn): keep traceback tail in tool-error digest preview (#2596)
## Description `_format_tool_call` in `headroom/learn/analyzer.py` built the error preview with a head-only slice — `tc.output[:200]`. For tracebacks the root cause (`ExceptionType: message`) is at the **tail**, so the digest showed only `Traceback (most recent call last):` plus the first frame and dropped the actual diagnosis. The issue reports 46% of 715 measured errors were truncated past the 200-char head. Closes #2590 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `_truncate_head_tail()` helper that collapses newlines and, when over budget, keeps both the head and the tail joined by `…`. - `_format_tool_call` now uses it for error output so the exception line survives truncation. Short errors are returned unchanged (no marker). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text $ uv run pytest tests/test_learn/test_analyzer.py::TestDigestBuilder -q 9 passed in 1.47s $ uv run ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py All checks passed! ``` ## Real Behavior Proof - Environment: headroom @ main, Python 3.14, uv - Exact command / steps: added a long synthetic traceback (`KeyError: 'the-actual-root-cause'` at the tail) as a failing tool call and built the digest. - Observed result: digest now contains both `Traceback` and `KeyError: 'the-actual-root-cause'`, separated by `…`; short errors have no `…`. - Not tested: mypy not run locally. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Truncation budget stays at 200 chars (now split head/tail). mypy not run locally; happy to adjust if CI flags anything. |
||
|
|
d2170b1922 |
fix(learn): parse fenced JSON even with a prose preamble (#1988)
## Description `_strip_fenced_json` only stripped a markdown fence when the string *started with* ```` ``` ````. When the model prefixed prose before the fence (e.g. `Here is the JSON:\n\n```json ...`) despite being told to return JSON only, the guard was skipped and `json.loads` ran on the prose, raising `JSONDecodeError`. The claude-cli streaming path surfaced this as `returned unparseable output`, and `headroom learn` silently discarded the LLM analysis, degrading to "No actionable patterns found". This is the parsing-side cousin of the silent-degradation issue fixed in #373. Closes #1989. Related: #373. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/learn/analyzer.py`: rewrote `_strip_fenced_json` to locate the fenced block wherever it appears, then fall back to the whole text, then to a first-`{` / last-`}` slice, only re-raising `JSONDecodeError` if nothing parses as a JSON object. Preserves the prior "first opening / last closing fence" behaviour and triple-backtick content inside the payload. Fixes all three call sites (non-streaming CLI, claude-cli streaming, litellm). - `tests/test_learn/test_analyzer.py`: added regression cases to `TestStripFencedJson` for preamble-before-fence, prose around a bare object, and triple-backticks inside the payload. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) — scoped to the changed module (see Additional Notes) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_learn/test_analyzer.py -q ........................................................................ [ 86%] ........... [100%] 83 passed, 1 warning in 2.18s $ ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py All checks passed! $ ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py 2 files already formatted $ mypy --ignore-missing-imports --follow-imports=silent headroom/learn/analyzer.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS, Python 3.13.12, headroom-ai at this branch (runtime deps from an installed 0.30.0 env). - Exact command / steps: ran the old vs new `_strip_fenced_json` on the exact failing model output (a prose preamble followed by a ```json fence), then applied the fix over an installed 0.30.0 and re-ran the previously failing `headroom learn --apply`. Input sample: `'The JSON is my deliverable for this analysis task. Here it is:\n\n```json\n{"context_file_rules": [], "memory_file_rules": []}\n```'` - Observed result: OLD raised `JSONDecodeError: Expecting value: line 1 column 1 (char 0)`; NEW returned `{'context_file_rules': [], 'memory_file_rules': []}`. The real `headroom learn --apply` run that had been failing with `returned unparseable output` then completed and consumed the LLM analysis instead of dropping it. Full transcript: ```text OLD: JSONDecodeError -> Expecting value: line 1 column 1 (char 0) NEW: {'context_file_rules': [], 'memory_file_rules': []} ``` - Not tested: full end-to-end `headroom learn --apply` was not re-run inside CI here (it shells out to a live `claude` CLI); the parser is exercised deterministically by the added unit tests and the before/after repro above. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A — updated the function docstring only; no external docs affected) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (N/A — no CHANGELOG entry convention observed for this fix; happy to add if maintainers prefer) ## Additional Notes - `mypy` was run against the changed module in isolation (`--ignore-missing-imports --follow-imports=silent`) rather than the full project, because I validated in an ad-hoc environment; the change keeps the existing `-> dict` signature and annotations, so it is type-neutral. - Not addressed here (possible follow-up): the failure is swallowed as a warning in `analyze()`, so users only see "No actionable patterns found" with no signal the LLM pass produced nothing — the same silent-degradation class as #373, on the parsing side. |
||
|
|
e3b45e402b |
fix(learn): handle Windows UTF-8, drive-letter paths, and CLI shim fallback (#1895)
## Description `headroom learn --verbosity` is broken on Windows in three related ways: - Transcript/profile reads can use the platform default codec, so non-ASCII content can raise `UnicodeDecodeError` and collapse learning signals to empty output. - `--project <path>` can miss real Claude project directories because Windows profile junctions can raise `PermissionError` during directory walks, and escaped Claude project folder names cannot always distinguish `vibe-remote` from `vibe\remote`. - `headroom learn --agent codex` can fail with `` `claude` not found in PATH `` even when the npm-installed CLI exists, because Windows `.cmd` shims require `PATHEXT` resolution. Refs https://github.com/headroomlabs-ai/headroom/issues/1624 for the Windows learn failures. The dashboard-hint UX and third-party-provider-auth items in that issue are unrelated and out of scope for this PR. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/learn/verbosity.py`: read and write verbosity transcripts/profiles with `encoding="utf-8"` so non-ASCII content works regardless of the Windows locale codec. - `headroom/learn/plugins/claude.py`: skip inaccessible siblings one entry at a time during greedy project path decoding, so one Windows junction no longer hides valid project directories. - `headroom/learn/plugins/claude.py`: prefer a valid `cwd` found in Claude session JSONL when discovering project paths, which resolves ambiguous escaped folder names such as `vibe-remote` versus `vibe\remote`. - `headroom/learn/analyzer.py`: resolve Windows CLI shim paths through `shutil.which()` after `FileNotFoundError`, then retry once for streaming and non-streaming CLI calls. - `CHANGELOG.md`: document the Windows learn fixes under `Unreleased`. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_verbosity_learn.py::TestWindowsEncoding tests/test_learn/test_scanner.py::TestGreedyPathDecode::test_permission_denied_sibling_does_not_abort_the_walk tests/test_learn/test_analyzer.py::TestWindowsCliShimFallback tests/test_learn/test_scanner.py::TestDecodeProjectPath::test_discover_project_prefers_session_cwd_over_ambiguous_folder_name -q`) - [x] Linting passes (`uv run ruff check headroom/learn/plugins/claude.py tests/test_learn/test_scanner.py`) - [x] Formatting passes (`uv run ruff format --check headroom/learn/plugins/claude.py tests/test_learn/test_scanner.py`) - [ ] Type checking passes (`uv run mypy headroom`) not run; no new public type surface - [x] New tests added for the Windows `cwd` disambiguation regression - [x] Manual testing performed ### Test Output ```text uv run ruff format headroom/learn/plugins/claude.py 1 file reformatted uv run ruff check headroom/learn/plugins/claude.py tests/test_learn/test_scanner.py All checks passed! uv run ruff format --check headroom/learn/plugins/claude.py tests/test_learn/test_scanner.py 2 files already formatted uv run pytest tests/test_verbosity_learn.py::TestWindowsEncoding tests/test_learn/test_scanner.py::TestGreedyPathDecode::test_permission_denied_sibling_does_not_abort_the_walk tests/test_learn/test_analyzer.py::TestWindowsCliShimFallback tests/test_learn/test_scanner.py::TestDecodeProjectPath::test_discover_project_prefers_session_cwd_over_ambiguous_folder_name -q 9 passed in 0.25s ``` CI on current head `c6dbac40` is green. A prior `test (1)` run hit an unrelated timing-sensitive scheduler assertion; GitHub did not permit direct rerun without admin rights, so the empty commit `c6dbac40` retriggered CI and the shard passed. ## Real Behavior Proof - Environment: Windows 11, Python 3.12, real filesystem for the path-decoding reproduction. - Exact command / steps: Ran `uv run pytest tests/test_verbosity_learn.py::TestWindowsEncoding tests/test_learn/test_scanner.py::TestGreedyPathDecode::test_permission_denied_sibling_does_not_abort_the_walk tests/test_learn/test_analyzer.py::TestWindowsCliShimFallback tests/test_learn/test_scanner.py::TestDecodeProjectPath::test_discover_project_prefers_session_cwd_over_ambiguous_folder_name -q`, `uv run ruff check headroom/learn/plugins/claude.py tests/test_learn/test_scanner.py`, and `uv run ruff format --check headroom/learn/plugins/claude.py tests/test_learn/test_scanner.py`; the path tests create real Windows-style project directories, inaccessible siblings, ambiguous `vibe\remote` versus `vibe-remote` folders, and Claude session JSONL with `cwd` pointing at the intended project. - Observed result: The focused test command returned `9 passed in 0.25s`, Ruff check passed, and Ruff format check passed. The decoder skips the inaccessible sibling and reaches `vibe-remote`; the session-`cwd` test returns `vibe-remote` instead of trusting the ambiguous escaped folder name; the UTF-8 tests round-trip non-ASCII transcript/profile content under a non-UTF-8 Windows-style codec; the CLI shim tests retry once through `shutil.which()` after `FileNotFoundError`. - Not tested: real npm-installed `claude`/`codex` CLI shims, dashboard-hint UX, and third-party-provider auth. ## Review Readiness - [x] I have performed a self-review - [x] Retrospective review completed after opening; it found one missing `cwd` disambiguation case, now fixed in this PR - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - The post-open retrospective review concluded this needed targeted rework rather than only a retrospective sign-off. - The `cwd` recovery commit is `ee720bb1`; `0905be7b` contains the required formatter cleanup; current head `c6dbac40` is an empty CI-rerun commit after GitHub denied direct rerun without admin rights. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
9bff5752bb |
fix(learn): claude-cli streams output with idle timeout (#373)
## Description `headroom learn` with the claude-cli backend used `subprocess.run` with a hard 120s wall-clock cap and no liveness signal. A successful long analysis and a hung connection looked identical — exit 0 with "0 recommendations" was the only user-visible signal when the LLM call timed out, which silently hides genuine learnings. This PR makes the CLI backend timeout-aware, with progress detection for claude-cli and configurable wall-clock caps for every backend. Fixes #(issue number) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Streaming claude-cli with idle timeout**: invoke `claude -p --output-format stream-json --verbose` and run a watchdog loop that drains stdout/stderr via reader threads. Each stream-json event resets an idle deadline. Kill the process if no output for `HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS` (default 60s) or if total elapsed exceeds `HEADROOM_LEARN_CLI_TIMEOUT_SECS` (default 300s, was 120s). The final `type:"result"` event carries the assistant response, which is then parsed as JSON. Reader threads (rather than `select`) are used so the watchdog works on Windows where `select` does not support pipe handles. - **Bumped default `_CLI_TIMEOUT` from 120s to 300s** as the hard cap for all CLI backends. The previous 120s was too tight for large digests on slower networks. - **Env-var overrides** via new helper `_resolve_timeout_secs(env_var, default)`: - `HEADROOM_LEARN_CLI_TIMEOUT_SECS` — hard wall-clock cap (all CLI backends) - `HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS` — idle cap (streaming claude-cli only) - Non-positive or non-integer values log a warning and fall back to defaults, so a typo can't disable the timeout. - **gemini-cli and codex-cli** keep `subprocess.run(timeout=hard_cap)` since they do not emit progress events. They benefit from the bumped default and the env-var override. - **CHANGELOG.md** updated under `[Unreleased]` → `### Fixed`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed (existing repro: 16k-call digest that previously timed out at 120s) New test coverage in `tests/test_learn/test_analyzer.py`: - `test_claude_cli_streams_and_parses_result_event` — happy path, fake Popen yields system/assistant/result events - `test_claude_cli_parses_fenced_result` — markdown fences in the result event still parse - `test_claude_cli_idle_timeout_kills_hang` — `HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS=1` + a hanging stdout iterator triggers the idle watchdog - `test_claude_cli_hard_cap_kills_continuous_chatter` — continuous events with a low hard cap fire the wall-clock kill (proves idle reset alone can't keep a runaway alive) - `test_claude_cli_missing_result_event_raises` — graceful failure when no `result` event is emitted - `test_claude_cli_nonzero_exit_raises` / `test_claude_cli_unparseable_result_raises_with_context` / `test_claude_cli_not_installed_raises` — error paths - Parallel codex-cli error coverage (timeout-honors-env-override included) so the wall-clock path is exercised - `TestResolveTimeoutSecs` — unset / empty / non-integer / non-positive / valid override ## Test Output ``` $ uv run pytest tests/test_learn/test_analyzer.py ============================== 67 passed in 2.14s ============================== $ uv run ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py All checks passed! $ uv run ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py 2 files already formatted $ uv run mypy headroom/learn/analyzer.py Success: no issues found in 1 source file ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - The contract assumed for claude-cli stream-json output is: each line is a JSON object with a `type` field; the final event has `type:"result"` with a string `result` field carrying the assistant text. This matches the documented Anthropic CLI behavior. If the contract changes upstream, `_call_claude_cli_streaming` raises a clear "did not emit a final \`result\` event" error rather than silently succeeding. - Backwards-compatible for users without env-var configuration: behavior just becomes "longer hard cap, plus idle watchdog for claude-cli", neither of which can falsely succeed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
d7973665f4 |
fix(learn): finish gemini-flash-latest default model sweep (#532)
Google deprecated gemini/gemini-2.0-flash; headroom learn silently fails when GEMINI_API_KEY is set. PR #532 updated the default in analyzer.py but left stale references in the CLI help text and unit test assertion. |
||
|
|
35073ccb23 |
style(learn): ruff format test_analyzer.py
Line-length wrapping only. No behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
6d2aba8741 |
fix(learn): show prior patterns block to LLM to prevent dangling refs
When `headroom learn` re-surfaced a section heading that already existed in CLAUDE.md / MEMORY.md, the writer replaced that section wholesale — but the LLM never saw the prior block, so it emitted condensed bullets like "X is *also* large — same rule as Y, Z" assuming Y and Z would remain siblings. After replacement, Y and Z were gone and the "also" dangled. This threads the project's current `<!-- headroom:learn -->` block (from both CLAUDE.md and MEMORY.md) into the digest as a "Prior Learned Patterns" section, and extends the system prompt to make the re-emission contract explicit: re-stating a section replaces it wholesale, so the LLM must copy forward prior bullets it still agrees with. Prior sections the LLM omits entirely are still carried forward by the writer (#231 behavior preserved as a safety net). Changes: - New `extract_marker_block(file_content)` helper in `learn.writer` that returns the raw marker block (delimiters included) or None. - New `_build_prior_patterns_section(project)` in `learn.analyzer` reads `project.context_file` and `project.memory_file` via the new helper and formats a labeled section ahead of the per-session event stream. - `_build_digest` emits the prior-patterns section when present; char budget accounting unchanged (prior blocks are small). - `_SYSTEM_PROMPT` gains a "Prior Learned Patterns" rule block telling the LLM how to integrate prior bullets (preserve / revise / drop-only- if-contradicted) and warning against unresolved cross-references. - Tests: 6 new `TestPriorPatternsInjection` cases (present/absent files, no-marker-block, both-files, end-to-end via mocked `_call_llm`); 4 new `TestExtractMarkerBlock` cases. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
c3cf022886 |
fix(learn): handle FileNotFoundError when CLI tool is not installed
When --model codex-cli is used but codex is not in PATH, subprocess.run raises FileNotFoundError. Catch it and raise a clear RuntimeError with guidance. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
|
e98c975153 |
feat(learn): add CLI-based LLM backends for keyless headroom learn
Allow `headroom learn` to use locally installed coding agent CLIs (claude, gemini, codex) as LLM backends, so subscription users without raw API keys can run failure analysis. Priority: --model flag > API key > HEADROOM_LEARN_CLI env var > auto-detect - Pass prompts via stdin to avoid ARG_MAX limits - Handle TimeoutExpired, truncate stderr, enrich JSONDecodeError - Add 31 new tests (48 total), all passing - Update docs/learn.md with CLI backend documentation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
|
da481a359b |
fix(learn): pass explicit model in tests to avoid API key requirement
SessionAnalyzer() without a model calls _detect_default_model() which raises when no API keys are set (e.g., in CI). Pass model="test-model" in the three tests that mock _call_llm. |
||
|
|
4d14012c2f |
feat: add headroom perf CLI and rewrite headroom learn to use LLM analysis
Proxy performance logging (`headroom perf`): - Add always-on RotatingFileHandler to ~/.headroom/logs/proxy.log (10MB x 5 backups) - Replace scattered log lines with structured PERF lines containing model, msgs, tok_before/after/saved, cache_read/write/hit_pct, opt_ms, and transforms - Emit PERF lines from all three response paths (streaming Anthropic, non-streaming Anthropic, Bedrock streaming) - Add `headroom perf` CLI that parses proxy logs and reports token savings, cache hit rates, prefix stability, transform effectiveness, routing breakdown, TOIN status, and actionable recommendations - Support --hours and --raw flags for time filtering and raw record output Learn module rewrite (LLM-based analysis): - Replace all regex/heuristic analyzers with a single LLM call via LiteLLM - New SessionAnalyzer builds compact digests and sends to any of 100+ models - Auto-detect best model from API keys (Anthropic → OpenAI → Gemini) - Add --model flag for explicit model selection - Enrich scanner with SessionEvent (user messages, interruptions, subagent summaries), token usage tracking, and timestamps - Simplify models: remove EnvironmentFact, StructureNote, Correction, CommandPattern, RetryPattern, AnalysisReport; add SessionEvent, AnalysisResult - Simplify writer: remove Recommender class (LLM now produces recommendations directly) - Update tests for new analyzer and models |
||
|
|
17442c2dcc |
Add headroom learn: offline failure learning for coding agents
Analyzes past conversation history to find tool call failure patterns, correlates each failure with what eventually succeeded, and writes specific project-level learnings to CLAUDE.md and MEMORY.md. Key design: - Success correlation: extracts the diff between failed and successful inputs as the learning (not generic advice) - Generic architecture: tool-agnostic ToolCall model with pluggable Scanner/Writer adapters (Claude Code first, extensible to Cursor/Codex) - 5 analyzers: Environment, Structure, Commands, Retries, Cross-Session - Dry-run by default, --apply to write, --all for all projects Also fixes mypy errors in litellm_callback, asgi, langchain chat_model, and anthropic provider (AsyncClient typing, ToolCall arg-type, int cast). |