mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-23 02:49:17 +08:00
feat/https-compose-caddy
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
ea947ca421 |
refactor(transforms): share one CCR persist helper for the Rust-backed compressors (#4094)
## Description DiffCompressor, LogCompressor and SearchCompressor each carried an identical private `_persist_to_python_ccr` method. It writes the Rust-emitted CCR entry through to the production CompressionStore with `explicit_hash` (#816). This PR replaces the three copies with one function, `headroom.transforms.base.persist_rust_ccr_entry`. Its warnings name the compressor that failed and keep the store's exception text out of every log level. Refs #4136 Depends on #4138 (`headroom/log_safety.py`); its files show in this diff until it merges. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Add `persist_rust_ccr_entry(original, compressed, cache_key, *, source)` to `transforms/base.py`, using a lazy import of the compression store as before. - The three `compress()` methods call it with `source="diff"`, `"log"` or `"search"`. - Remove the three private copies, and the loggers in `diff_compressor` and `search_compressor` that are now unused. - The write-failure WARNING names the source and exception type. The detail goes to DEBUG as `describe_exception(e)` (types and code locations, no message or traceback), because store backends can echo the payload. The import-failure warning uses it too. `explicit_hash` and fail-open are unchanged. - Tests: - the three #816 bridge tests become one parametrized test over the helper; - they no longer skip without the Rust extension, because the helper is pure Python; - a new test formats every captured DEBUG and WARNING record and asserts the original-content and credential canaries are absent. It fails on the earlier `exc_info=True` version of this PR. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest tests/test_ccr_rust_marker_hash_bridge.py tests/test_transforms_search_compressor.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/test_transforms/test_diff_compressor.py tests/test_transforms_log_compressor.py tests/test_log_compressor.py tests/test_search_compressor.py tests/test_transforms_content_router.py 280 passed in 15.69s $ python -c "import sys; sys.modules['headroom._core'] = None; import pytest; pytest.main([... 'tests/test_ccr_rust_marker_hash_bridge.py'])" 3 passed $ ruff check / format: All checks passed; mypy base.py: Success ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree (head |
||
|
|
47f8710817 |
refactor(transforms): drop the Python mirror of the Rust search selector (#4103)
## Description
`SearchCompressor.compress()` runs entirely in Rust. The class still
carried a Python re-implementation of the scorer, selector and
formatter, plus a parser wrapper, kept only so old tests could call
them. Production never ran them, and the module's own docstring noted
the Python error-boost keywords had drifted from Rust. All removed names
are private (leading underscore).
Refs #4136
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- Remove `_parse_search_results`, `_score_matches`, `_select_matches`,
`_format_output`, `_cjk_bigrams` and `_is_cjk_char`.
- Parser tests now call `headroom._core.parse_search_lines`, the live
Rust parser, through a small test helper.
- These tests are converted to `compress()`-level tests, because Rust
only unit-tests error boosting and bigram extraction: context-word
overlap, error/warning/TODO boost, `context_keywords`, the short-word
filter, `boost_errors=False`, the CJK query bigram boost, and the
combined parse/score/select/format test.
- `test_score_capped_at_one` is dropped: the cap cannot be observed
through `compress()`.
- The `_cjk_bigrams` unit test is dropped: Rust covers it in
`cjk_bigrams_from_runs`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_search_compressor.py tests/test_transforms_search_compressor.py tests/test_search_compressor_cjk.py tests/test_ccr_rust_marker_hash_bridge.py tests/test_transforms_content_router.py tests/test_error_detection.py tests/test_signals_keyword_parity.py
170 passed in 7.31s
$ ruff check / format: All checks passed; mypy search_compressor.py: Success
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) a real `SearchCompressor.compress()` keeping one match
per file, with a CJK query whose bigrams match only one row.
- Observed result: The proxy serves both requests (200) with the same
savings as main. The direct check keeps the CJK row ranked first by the
Rust scorer. Output below.
- Not tested: Nothing beyond the listed files.
```text
branch: cleanup/transforms-search-drop-python-mirror @
|
||
|
|
bfcb07d78e |
fix(ccr): key Rust search/diff/log markers with explicit_hash (#852)
Fixes #816 ## What The three `_persist_to_python_ccr` shims (`search_compressor.py`, `diff_compressor.py`, `log_compressor.py`) called `store.store(original, compressed)` with the default key — `SHA-256(original)[:24]` since PR #395 — while the Rust side embeds `MD5(original)[:24]` in the emitted `Retrieve more: hash=...` marker. Marker key and storage key never matched, so **every retrieval of a Rust search/diff/log marker returned "Entry not found or expired"** (inside any TTL — the symptom class reported in #714). Fix is exactly what #816 proposed: pass the marker's key via `explicit_hash=cache_key` at all three call sites, the same contract SmartCrusher has used since PR #395. No store changes needed — `store()` already validates and honors `explicit_hash`. Also corrected the stale comment in `search_compressor.py` that still claimed "both use MD5(original)[:24]". ## Tests `tests/test_ccr_rust_marker_hash_bridge.py` (companion to `test_ccr_row_drop_store_bridge.py`, which pinned the same bug class for SmartCrusher in #389): for each shim, the store entry must be retrievable under the Rust marker key AND absent under the SHA-256 default key. Verified red→green: all 3 tests fail on main with the exact issue symptom ("store has no entry under the Rust marker key ...; the marker dangles") and pass with the fix. `test_ccr_row_drop_store_bridge.py` still green. `ruff check` + `ruff format --check` clean. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
|
1de32b1437 | style: ruff format tests/test_transforms_search_compressor.py | ||
|
|
4d799d5264 |
feat(rust): port search_compressor to Rust + signals trait consumer (Phase 3e.2)
Ports `headroom.transforms.search_compressor` to Rust as the first consumer of the `signals::LineImportanceDetector` trait shipped in Phase 3e.1. The Python regex registry (`_GREP_PATTERN`/`_RG_CONTEXT_PATTERN`) silently misparsed two real-world inputs. The hand-rolled Rust parser fixes both: * **Windows paths.** `^([^:]+):(\d+):(.*)$` captured only the drive letter from `C:\Users\foo\bar.py:42:line`, then the `\d+` group failed on `\`. Result: every Windows-formatted line was silently dropped from `file_matches`. The Rust parser detects the drive prefix and starts the line-number scan after the drive colon. * **Filenames with `-`.** `_RG_CONTEXT_PATTERN`'s `[^:-]+` excluded dashes from the path, so legitimate names like `pre-commit-config.yaml-42-line` parsed wrong. The Rust parser anchors on the *line-number marker* (`<sep>\d+<sep>`), so paths can contain dashes freely. Two further hardening changes: * CCR storage failures are loud (Python silently swallowed them). * Per-file dedup is `O(n log n)` via `BTreeSet<(line_no, content_hash)>` (Python used linear `match not in file_selected`, worst-case quadratic for big files). The Rust `SearchCompressor` owns a `Box<dyn LineImportanceDetector>` defaulting to `KeywordDetector`. Priority scoring routes through the trait instead of a hardcoded regex list, so a future BGE classifier head (per the trait extension docs) can take over without touching the compressor. Sidecar `SearchCompressorStats` captures lines unparsed, files dropped by `max_files`, matches dropped by per-file vs global caps, and the CCR skip reason -- diagnostics Python never emitted. `headroom.transforms.search_compressor` is now a thin shim that delegates `compress()` to Rust end-to-end (so the parser bug fixes land in production), and keeps the legacy `_parse_search_results` helper routed through the same Rust parser. The other internal helpers (`_score_matches`, `_select_matches`, `_format_output`) stay Python -- they're heavily covered by existing direct-call tests and Rust scoring is byte-equivalent. The 4 public dataclasses are unchanged. Tests that monkeypatched the old internal `_store_in_ccr` helper are updated to exercise the new `_persist_to_python_ccr` boundary instead. * 16 Rust unit tests (parser, scoring, selection, CCR round-trip) + 3 explicit `fixed_in_3e2` markers for the bug-fix lines * 53 Python tests (existing suite intact; 2 updated for new shape) * `make ci-precheck` clean Stacks on PR #317 (signals trait module). |
||
|
|
efd2ac1ca4 |
chore: renormalize line endings to LF
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but 74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings, violating that contract. Every macOS/Linux clone reports these files as "modified" on fresh checkout because git's diff engine sees the stored bytes don't match the attribute contract, even though the working tree and index match byte-for-byte. Running `git add --renormalize .` rewrites each affected blob so the stored form matches the attribute declaration. No semantic changes — every affected file's diff is "N insertions, N deletions" with inserts and deletes being the same lines modulo line endings. Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub blame skip this mechanical commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
38bf3e639c |
test: expand coverage across helper slices
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |