Commit Graph
9 Commits
Author SHA1 Message Date
5d25abd356 test: repair three suite failures that are red on main (#3196)
## Summary

Three tests fail on a clean `main` full-suite run. None is a product
defect — all three are tests that stopped describing reality, and they
will noise up or block the 0.36.4 release.

| Test | Why it fails | Fix |
|---|---|---|
| `test_release_workflows::test_no_native_tls_in_wheel_build_tree` |
Shells out to `cargo`; raises `FileNotFoundError` wherever the Rust
toolchain is absent | Copied the skip guards its own dual already had |
|
`test_learn/test_integration::TestCodexIntegration::test_full_pipeline`
| Asserts `"Bash" in all_tools` against **real local Codex data**; Codex
renamed its shell tool | Assert what the test is for, across Codex
versions |
|
`test_graceful_shutdown::test_run_server_installs_cancelled_error_filter`
| Counts installs on the **process-global** `uvicorn.error` logger;
order-dependent | Isolate the global state; assert the real contract |

## 1. native-tls / cargo

The `openssl-sys` gate 30 lines above is described in-code as this
test's dual. It already skips when `cargo` is missing, **and** when
cargo fails for a reason other than `"package did not match"` (the Linux
wheel target not being installed locally). The native-tls test never
copied either guard.

Not disabled: CI installs the toolchain via `dtolnay/rust-toolchain`, so
the check still executes there. The skip only applies where cargo is
genuinely absent.

## 2. Codex tool vocabulary

This test runs against whatever Codex sessions the machine actually has
(gated by `HAS_CODEX_DATA`), and asserted:

```python
# Codex has only Bash tool (shell)
assert "Bash" in all_tools
```

Codex has since renamed its shell tool (`Bash` → `shell` → `exec`), and
0.149.0 added agent tools (`spawn_agent`, `send_message`, `wait`) beside
it. The assertion pinned one release's vocabulary, so it fails on any
current install.

It now asserts what the pipeline is actually being tested for — that
tool calls were extracted, including a shell-execution tool under any of
its known names — and names the remedy in the failure message for the
next rename.

**Still discriminating** (verified, not assumed):

| Scenario | Result |
|---|---|
| pipeline parsed nothing | fails ✓ |
| tool names garbled | fails ✓ |
| agent tools only, no shell tool | fails ✓ |
| real current Codex data | passes ✓ |

## 3. Global logger state

```python
if not any(isinstance(item, _SuppressCancelledErrorFilter) for item in uvicorn_error_logger.filters):
    uvicorn_error_logger.addFilter(_SuppressCancelledErrorFilter())
```

`run_server` is deliberately idempotent and `uvicorn.error` is a
process-global logger, so any earlier test in the session that reached
`run_server` leaves the filter attached — and this test then observes
**zero** installs against its `== 1` assertion. It passes alone and
fails in a full run, which is exactly the symptom.

The test now clears and restores that global state around itself, and
additionally asserts the idempotence guard that is the real contract:
calling `run_server` twice must not stack a duplicate filter. The test
got stronger, not just quieter.

## Scope

Tests only — no product code is touched.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 23:18:40 -07:00
Abhay SinghandJerrettDavis a24fe7dcbf fix(learn): stop classifying a successful exit code 0 as an error (#2289)
## Description

`is_error_content` classifies successful shell commands as errors,
inflating the failure stats that `headroom learn` reports.

The heuristic flags a tool result as an error when it contains any of a
list of substrings, one of which is the bare `"exit code"`:

```python
indicators = [
    ..., "timed out", "exit code", "FileNotFoundError",
]
return any(ind in snippet for ind in indicators)
```

But agent harnesses (Codex, Grok, opencode, ...) append `exit code 0` to
the output of every **successful** shell command. `"exit code" in
snippet` is `True` for `exit code 0`, so those successes are counted as
failures.

That is not cosmetic: `is_error_content` sets `ToolCall.is_error`, which
feeds:
- the per-project failure rate the digest shows the LLM
(`_build_digest`: "N failures (X%)"), and
- loop classification (`detect_loops` treats a group as an *error loop*
when ≥ half its calls are errors),

so a project where most shell commands succeed can read as one riddled
with failures, biasing the learned recommendations.

## Fix

Match a **nonzero** exit code instead of the bare substring:

```python
_NONZERO_EXIT_RE = re.compile(r"exit code:?\s*(?!0\b)\d", re.IGNORECASE)
...
if any(ind in snippet for ind in indicators):
    return True
return bool(_NONZERO_EXIT_RE.search(snippet))
```

`exit code 0` no longer matches. A nonzero code still does — and, as a
small bonus, the case-insensitive regex now also catches `Exit code: 1`
(colon + capitalized), which the old case-sensitive lowercase substring
missed.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/learn/_shared.py`: replace the `"exit code"` substring
indicator with a nonzero-exit-code regex (`_NONZERO_EXIT_RE`) checked
after the other indicators.
- `tests/test_learn/test_integration.py`: new tests that `exit code 0`
is not an error and a nonzero code (any casing / with a colon) still is.
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/learn/_shared.py tests/test_learn/test_integration.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/learn/_shared.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOM-kills this box (ML stack import), so I
reproduced the classifier with a dependency-free script and left the
full pytest to CI.
- Exact command / steps: ran a successful output ending `Process
finished with exit code 0`, plus several nonzero-code failures (`exit
code 1`, `Exit code: 127`, `exit code 137`) and control strings, through
the OLD substring form and the NEW regex form.
- Observed result: OLD flags `exit code 0` as an error; NEW returns
`False` for it, still returns `True` for every nonzero code (including
the colon/capitalized form the old lowercase substring missed), and
leaves the other indicators unchanged.
- Not tested: a full `learn` run over a real history; full local
`pytest` deferred to CI (OOM).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" box is unchecked because the full suite
imports the ML stack, which I can't run here. The new tests live
alongside the existing `is_error_content` false-positive/true-positive
tests in `tests/test_learn/test_integration.py`, so they run under the
normal CI pytest job; behaviour is additionally verified by the
standalone proof above.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-08-11 23:45:32 -05:00
Focused Instability ced75e4718 feat(learn): write per-project learnings to CLAUDE.local.md by default (#1115)
## Description

`headroom learn` wrote per-project learnings into the project's
`CLAUDE.md`, which Claude Code treats as team-shared and git-tracked.
That meant machine-specific absolute paths and tool-discovery byproducts
polluted the shared file for every teammate. This switches the default
to the personal, gitignored `CLAUDE.local.md`, adds a `--target`
override, and migrates any stale block out of `CLAUDE.md`.

Closes #1072.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `ClaudeCodeWriter` now writes CONTEXT_FILE recommendations to
`CLAUDE.local.md` by default instead of `CLAUDE.md` (the home-directory
case still uses `~/.claude/CLAUDE.md`, which is personal global memory).
- Added a `--target` flag (Claude Code only) and `set_context_target()`
to override the destination — e.g. `--target CLAUDE.md` to opt back into
the shared file, or any relative/absolute path.
- On first run after upgrade, a stale Headroom block left in `CLAUDE.md`
is moved into `CLAUDE.local.md` and stripped from `CLAUDE.md`, with a
warning surfaced by the CLI. If `CLAUDE.md` held nothing but the block,
the empty file is removed.
- `WriteResult` carries `warnings`; the `learn` CLI prints them.
- Updated docs (`failure-learning.mdx`) and `CHANGELOG.md`.

This implements the maintainer's stated preference order from the issue
(default → `CLAUDE.local.md`, plus a `--target` flag), scoped to the
Claude writer only — `AGENTS.md`/`GEMINI.md` have no `.local` convention
and are untouched.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_learn/ tests/test_cli_learn.py -q
196 passed, 2 skipped in 17.80s

$ ruff check headroom/learn/writer.py headroom/cli/learn.py
All checks passed!

$ mypy headroom/learn/writer.py headroom/cli/learn.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS (Darwin), Python 3.11, headroom on rebased
upstream/main
- Exact command / steps: ran ClaudeCodeWriter against a temp project
whose `CLAUDE.md` held hand-written content plus a legacy Headroom
block, then `writer.write([...], dry_run=False)`
- Observed result: `CLAUDE.md` kept its hand-written content with the
block removed; `CLAUDE.local.md` gained both the migrated `### Old`
section and the new `### Env` section; `result.warnings` contained the
"Moved Headroom learnings out of …" notice. A block-only `CLAUDE.md` was
deleted and a "Removed …" warning emitted.
- Not tested: live end-to-end `headroom learn --apply` against real LLM
analysis (writer + CLI plumbing covered by unit/CLI tests with mocked
analysis)

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated CHANGELOG.md if applicable

## Additional Notes

Scoped to the Claude Code writer per the issue. After migration,
`discover_projects` may briefly re-surface a section the LLM re-derives,
but the write-side merge dedups by section name so the file stays
correct.
2026-06-22 15:05:06 -05:00
Kayzo 0264e03d33 fix(testing): stabilize 3.12 suite and fingerprints 2026-04-28 21:35:32 +00:00
Garm 47d0e4d5a8 Fix tests 2026-03-23 13:24:02 +01:00
Garm af448a568f . 2026-03-23 13:21:06 +01:00
Garm a24daf35ab . 2026-03-20 18:45:28 +01:00
Tejas Chopra 4d14012c2f feat: add headroom perf CLI and rewrite headroom learn to use LLM analysis
Proxy performance logging (`headroom perf`):
- Add always-on RotatingFileHandler to ~/.headroom/logs/proxy.log (10MB x 5 backups)
- Replace scattered log lines with structured PERF lines containing model, msgs,
  tok_before/after/saved, cache_read/write/hit_pct, opt_ms, and transforms
- Emit PERF lines from all three response paths (streaming Anthropic, non-streaming
  Anthropic, Bedrock streaming)
- Add `headroom perf` CLI that parses proxy logs and reports token savings, cache
  hit rates, prefix stability, transform effectiveness, routing breakdown, TOIN
  status, and actionable recommendations
- Support --hours and --raw flags for time filtering and raw record output

Learn module rewrite (LLM-based analysis):
- Replace all regex/heuristic analyzers with a single LLM call via LiteLLM
- New SessionAnalyzer builds compact digests and sends to any of 100+ models
- Auto-detect best model from API keys (Anthropic → OpenAI → Gemini)
- Add --model flag for explicit model selection
- Enrich scanner with SessionEvent (user messages, interruptions, subagent summaries),
  token usage tracking, and timestamps
- Simplify models: remove EnvironmentFact, StructureNote, Correction, CommandPattern,
  RetryPattern, AnalysisReport; add SessionEvent, AnalysisResult
- Simplify writer: remove Recommender class (LLM now produces recommendations directly)
- Update tests for new analyzer and models
2026-03-07 14:15:49 -08:00
chopratejas 7cf086c2e8 Add multi-agent support, quality gates, and integration tests for headroom learn
- Codex adapter: CodexScanner reads ~/.codex/sessions/*.json, CodexWriter
  writes to AGENTS.md + instructions.md. Tested on 328 real sessions.
- Gemini writer: GeminiWriter writes to GEMINI.md (scanner deferred,
  sessions stored in protobuf).
- CLI --agent flag: auto-detect available agents or specify claude/codex/gemini.
- Quality gates: min_evidence, min_confidence, min_total_evidence thresholds
  prevent weak signals from writing noise to project files.
- Integration tests against real Claude Code and Codex session data on disk.
  Tests skip gracefully if data directories don't exist.
- Bash path extraction for Codex (reads files via sed/cat, not Read tool).
- Idempotency, false positive filtering, and skip-write-on-empty tests.
2026-02-28 23:29:02 -08:00