Commit Graph
7 Commits
Author SHA1 Message Date
ea947ca421 refactor(transforms): share one CCR persist helper for the Rust-backed compressors (#4094)
## Description

DiffCompressor, LogCompressor and SearchCompressor each carried an
identical private `_persist_to_python_ccr` method. It writes the
Rust-emitted CCR entry through to the production CompressionStore with
`explicit_hash` (#816). This PR replaces the three copies with one
function, `headroom.transforms.base.persist_rust_ccr_entry`. Its
warnings name the compressor that failed and keep the store's exception
text out of every log level.

Refs #4136

Depends on #4138 (`headroom/log_safety.py`); its files show in this diff
until it merges.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Add `persist_rust_ccr_entry(original, compressed, cache_key, *,
source)` to `transforms/base.py`, using a lazy import of the compression
store as before.
- The three `compress()` methods call it with `source="diff"`, `"log"`
or `"search"`.
- Remove the three private copies, and the loggers in `diff_compressor`
and `search_compressor` that are now unused.
- The write-failure WARNING names the source and exception type. The
detail goes to DEBUG as `describe_exception(e)` (types and code
locations, no message or traceback), because store backends can echo the
payload. The import-failure warning uses it too. `explicit_hash` and
fail-open are unchanged.
- Tests:
- the three #816 bridge tests become one parametrized test over the
helper;
- they no longer skip without the Rust extension, because the helper is
pure Python;
- a new test formats every captured DEBUG and WARNING record and asserts
the original-content and credential canaries are absent. It fails on the
earlier `exc_info=True` version of this PR.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_ccr_rust_marker_hash_bridge.py tests/test_transforms_search_compressor.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/test_transforms/test_diff_compressor.py tests/test_transforms_log_compressor.py tests/test_log_compressor.py tests/test_search_compressor.py tests/test_transforms_content_router.py
280 passed in 15.69s
$ python -c "import sys; sys.modules['headroom._core'] = None; import pytest; pytest.main([... 'tests/test_ccr_rust_marker_hash_bridge.py'])"
3 passed
$ ruff check / format: All checks passed;  mypy base.py: Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
(head 1bd479d6) with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) a real `DiffCompressor.compress()` on a 60-file diff,
then a retrieval of its CCR entry under the marker key; (3) the real
`persist_rust_ccr_entry` with logging at DEBUG, against a rejecting
backend whose exception echoes the source (with a credential in its
cause) and a backend raising `OSError` with errno `10**10000`,
`HEADROOM_DEBUG_DUMP` unset.
- Observed result: The proxy serves both requests (200) with savings.
The diff compressor emits a cache_key, and the entry can be retrieved
under exactly that key. On a store failure the helper returns `None`
(fail-open); the WARNING names source, class and cache key, and the
DEBUG line names only types and code locations. Neither canary appears
in any rendered line (grep count 0). Output below.
- Not tested: A model-issued retrieval through a live proxy; the direct
check reads the same store.

```text
branch: cleanup/transforms-shared-ccr-persist @ 1bd479d6
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-57958.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ python diff_direct.py   # DiffCompressor(min_lines_for_ccr=10).compress(60-file diff); store.retrieve(cache_key)
cache_key: 76142b4e8a8086abb3709b9f
retrievable under marker key: True
```

```text
$ python persist_proof.py   # real persist_rust_ccr_entry, DEBUG on, two failing backends
WARNING headroom.transforms.base CCR store write failed (diff, RuntimeError); cache_key aaaaaaaaaaaaaaaaaaaaaaaa remains in-marker only
DEBUG headroom.transforms.base CCR store write failure detail: RuntimeError at persist_proof.py:12 in store <- headroom/transforms/base.py:40 in persist_rust_ccr_entry; caused by ValueError at persist_proof.py:10 in store
returned: None
WARNING headroom.transforms.base CCR store write failed (diff, OSError); cache_key aaaaaaaaaaaaaaaaaaaaaaaa remains in-marker only
DEBUG headroom.transforms.base CCR store write failure detail: OSError [Errno out of range] at persist_proof.py:15 in store <- headroom/transforms/base.py:40 in persist_rust_ccr_entry
returned: None
$ grep -c "CANARY\|sk-test" persist_proof.out
0
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: No, except that the warning text now
includes the compressor name and comes from the
`headroom.transforms.base` logger.
- Kill switch / disable path: Not applicable; the stored data and keys
are unchanged.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.

`DiffCompressor.compress_with_stats` is unchanged here. It is public, so
it is deprecated in its own PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-16 14:30:21 -07:00
Tejas Chopra 47f8710817 refactor(transforms): drop the Python mirror of the Rust search selector (#4103)
## Description

`SearchCompressor.compress()` runs entirely in Rust. The class still
carried a Python re-implementation of the scorer, selector and
formatter, plus a parser wrapper, kept only so old tests could call
them. Production never ran them, and the module's own docstring noted
the Python error-boost keywords had drifted from Rust. All removed names
are private (leading underscore).

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Remove `_parse_search_results`, `_score_matches`, `_select_matches`,
`_format_output`, `_cjk_bigrams` and `_is_cjk_char`.
- Parser tests now call `headroom._core.parse_search_lines`, the live
Rust parser, through a small test helper.
- These tests are converted to `compress()`-level tests, because Rust
only unit-tests error boosting and bigram extraction: context-word
overlap, error/warning/TODO boost, `context_keywords`, the short-word
filter, `boost_errors=False`, the CJK query bigram boost, and the
combined parse/score/select/format test.
- `test_score_capped_at_one` is dropped: the cap cannot be observed
through `compress()`.
- The `_cjk_bigrams` unit test is dropped: Rust covers it in
`cjk_bigrams_from_runs`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_search_compressor.py tests/test_transforms_search_compressor.py tests/test_search_compressor_cjk.py tests/test_ccr_rust_marker_hash_bridge.py tests/test_transforms_content_router.py tests/test_error_detection.py tests/test_signals_keyword_parity.py
170 passed in 7.31s
$ ruff check / format: All checks passed;  mypy search_compressor.py: Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) a real `SearchCompressor.compress()` keeping one match
per file, with a CJK query whose bigrams match only one row.
- Observed result: The proxy serves both requests (200) with the same
savings as main. The direct check keeps the CJK row ranked first by the
Rust scorer. Output below.
- Not tested: Nothing beyond the listed files.

```text
branch: cleanup/transforms-search-drop-python-mirror @ 0384d610
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-51340.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ python -c "<SearchCompressor(max_matches_per_file=1, ...).compress(3 rows, context=CJK query)>"
src/a.py:20:认证令牌已过期需要重新登录
[... and 2 more matches in src/a.py]
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: No. Only unused private methods are
removed.
- Kill switch / disable path: Not applicable; nothing that runs changes.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.

`adaptive_sizer.py` stays. The log-mirror PR still needs it until both
PRs land, and then it can go in a one-file follow-up.
`error_detection.PRIORITY_PATTERNS_SEARCH` is now unused in product code
but is in an `__all__`, so it is left for a later deprecation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-10-10 08:59:40 -07:00
Focused InstabilityandAsh Rhodes bfcb07d78e fix(ccr): key Rust search/diff/log markers with explicit_hash (#852)
Fixes #816

## What

The three `_persist_to_python_ccr` shims (`search_compressor.py`,
`diff_compressor.py`, `log_compressor.py`) called `store.store(original,
compressed)` with the default key — `SHA-256(original)[:24]` since PR
#395 — while the Rust side embeds `MD5(original)[:24]` in the emitted
`Retrieve more: hash=...` marker. Marker key and storage key never
matched, so **every retrieval of a Rust search/diff/log marker returned
"Entry not found or expired"** (inside any TTL — the symptom class
reported in #714).

Fix is exactly what #816 proposed: pass the marker's key via
`explicit_hash=cache_key` at all three call sites, the same contract
SmartCrusher has used since PR #395. No store changes needed — `store()`
already validates and honors `explicit_hash`. Also corrected the stale
comment in `search_compressor.py` that still claimed "both use
MD5(original)[:24]".

## Tests

`tests/test_ccr_rust_marker_hash_bridge.py` (companion to
`test_ccr_row_drop_store_bridge.py`, which pinned the same bug class for
SmartCrusher in #389): for each shim, the store entry must be
retrievable under the Rust marker key AND absent under the SHA-256
default key.

Verified red→green: all 3 tests fail on main with the exact issue
symptom ("store has no entry under the Rust marker key ...; the marker
dangles") and pass with the fix. `test_ccr_row_drop_store_bridge.py`
still green. `ruff check` + `ruff format --check` clean.

---------

Co-authored-by: Ash Rhodes <ashley.rhodes@king.com>
2026-06-11 13:08:05 -05:00
chopratejas 1de32b1437 style: ruff format tests/test_transforms_search_compressor.py 2026-04-29 18:56:31 -07:00
chopratejas 4d799d5264 feat(rust): port search_compressor to Rust + signals trait consumer (Phase 3e.2)
Ports `headroom.transforms.search_compressor` to Rust as the first
consumer of the `signals::LineImportanceDetector` trait shipped in
Phase 3e.1.

The Python regex registry (`_GREP_PATTERN`/`_RG_CONTEXT_PATTERN`)
silently misparsed two real-world inputs. The hand-rolled Rust parser
fixes both:

* **Windows paths.** `^([^:]+):(\d+):(.*)$` captured only the drive
  letter from `C:\Users\foo\bar.py:42:line`, then the `\d+` group
  failed on `\`. Result: every Windows-formatted line was silently
  dropped from `file_matches`. The Rust parser detects the drive
  prefix and starts the line-number scan after the drive colon.
* **Filenames with `-`.** `_RG_CONTEXT_PATTERN`'s `[^:-]+` excluded
  dashes from the path, so legitimate names like
  `pre-commit-config.yaml-42-line` parsed wrong. The Rust parser
  anchors on the *line-number marker* (`<sep>\d+<sep>`), so paths
  can contain dashes freely.

Two further hardening changes:

* CCR storage failures are loud (Python silently swallowed them).
* Per-file dedup is `O(n log n)` via `BTreeSet<(line_no, content_hash)>`
  (Python used linear `match not in file_selected`, worst-case
  quadratic for big files).

The Rust `SearchCompressor` owns a `Box<dyn LineImportanceDetector>`
defaulting to `KeywordDetector`. Priority scoring routes through the
trait instead of a hardcoded regex list, so a future BGE classifier
head (per the trait extension docs) can take over without touching
the compressor.

Sidecar `SearchCompressorStats` captures lines unparsed, files
dropped by `max_files`, matches dropped by per-file vs global caps,
and the CCR skip reason -- diagnostics Python never emitted.

`headroom.transforms.search_compressor` is now a thin shim that
delegates `compress()` to Rust end-to-end (so the parser bug fixes
land in production), and keeps the legacy `_parse_search_results`
helper routed through the same Rust parser. The other internal
helpers (`_score_matches`, `_select_matches`, `_format_output`)
stay Python -- they're heavily covered by existing direct-call tests
and Rust scoring is byte-equivalent.

The 4 public dataclasses are unchanged. Tests that monkeypatched the
old internal `_store_in_ccr` helper are updated to exercise the new
`_persist_to_python_ccr` boundary instead.

* 16 Rust unit tests (parser, scoring, selection, CCR round-trip) +
  3 explicit `fixed_in_3e2` markers for the bug-fix lines
* 53 Python tests (existing suite intact; 2 updated for new shape)
* `make ci-precheck` clean

Stacks on PR #317 (signals trait module).
2026-04-29 17:08:36 -07:00
GarmandClaude Opus 4.7 efd2ac1ca4 chore: renormalize line endings to LF
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but
74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings,
violating that contract. Every macOS/Linux clone reports these files as
"modified" on fresh checkout because git's diff engine sees the stored
bytes don't match the attribute contract, even though the working tree
and index match byte-for-byte.

Running `git add --renormalize .` rewrites each affected blob so the
stored form matches the attribute declaration. No semantic changes —
every affected file's diff is "N insertions, N deletions" with inserts
and deletes being the same lines modulo line endings.

Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub
blame skip this mechanical commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 15:33:30 +02:00
JerrettDavisandCopilot 38bf3e639c test: expand coverage across helper slices
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-23 07:39:52 -05:00