Commit Graph
5 Commits
Author SHA1 Message Date
cuteNBDandcuteNBD 12dca26aa1 fix: use all user text blocks for relevance scoring (#4628)
## Description

`extract_user_query` stops at the first text block in a user message.
Relevance scoring can therefore miss the rest of the request. A leading
Claude Code `<system-reminder>` can also be mistaken for the request.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Join the user text blocks in order and remove complete reminder spans
without discarding text around them.
- Keep incomplete reminder tags and retain the existing latest-turn
fallback behavior.
- Cover multi-block prompts, reminders within both string and block
content, and a relevance-split example.

## Testing

- [x] Focused unit and integration tests pass
- [x] Linting passes (`ruff check .` and `ruff format --check .`)
- [x] New tests added for the bug

### Test Output

```text
Windows / Python 3.12: 108 passed, 2 warnings
Ubuntu / Python 3.12: 108 passed, 2 warnings
Ubuntu: ruff check ., ruff format --check ., mypy headroom --ignore-missing-imports passed
```

The added tests fail on the preceding revision. The Ubuntu run built the
Rust extension from source and tested the final PR code. [Run
details](https://github.com/bingdongni/headroom/actions/runs/41793014635).

## Real Behavior Proof

- Environment: Ubuntu, Python 3.12, `HEADROOM_OFFLINE=1`; Rust extension
built from source with `maturin build --profile ci`.
- Exact command / steps: Run the [deterministic input
script](https://github.com/bingdongni/headroom/blob/5750c34/scripts/contributor-proof.py)
against the base and patched source trees, with the same compiled
extension. The [verification
run](https://github.com/bingdongni/headroom/actions/runs/41793014635)
records these commands and output:

```sh
git worktree add --detach /tmp/headroom-base 5c8772b721
cp headroom/_core*.so /tmp/headroom-base/headroom/
(cd /tmp/headroom-base && PYTHONPATH=. python "$GITHUB_WORKSPACE/scripts/contributor-proof.py")
PYTHONPATH=. python scripts/contributor-proof.py
```

- Observed result: On 200 generated records, the second text block asks
about Kafka replication. Before the fix, 1 of 20 matching records
remained visible; after, 9 remained visible. Both used
`smart_sample(200->15)`. This is one synthetic input, not a general
quality benchmark.
- Not tested: Live provider traffic, embedding-backed relevance, the
full test suite, or the Rust pre-push suite. This example explicitly
sets `with_compaction=False`, not the default lossless-first path.

## Runtime Rollout Safety

- Rollout-managed feature(s): None added.
- Minimum rollout channel: Not applicable; this changes the existing
query extraction path.
- Stable/default behavior changed: Yes, relevance-dependent compression
and expansion see the complete user query without reminder spans.
- Kill switch / disable path: No query-extraction-specific switch.
- Unsafe override required: No.
- Qualification impact: No rollout-managed feature is changed; broader
model-quality qualification was not run.
- Rollback path: Revert these commits.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have added tests that prove the fix
- [x] I did not edit `CHANGELOG.md`

---------

Co-authored-by: cuteNBD <206630735+bingdongni@users.noreply.github.com>
2026-11-04 09:19:03 -06:00
Mansurshakh JaparovandJerrettDavis 93279f3208 fix(smart-crusher): preserve unchanged JSON bytes (#4044)
## Description

Closes #3697.

An unchanged SmartCrusher result now keeps the original bytes and
reports no structural compression. Formatting-only savings come from the
existing Rust JsonMinifier, not from pretending the JSON value changed.

The minifier preserves duplicate keys, order, escapes and number
literals. A mixed result cannot use one section's CCR marker to hide
unmarked loss in another.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update

## Changes Made

- Keep SmartCrusher's exact-byte no-op behavior; route actual whitespace
savings through JsonMinifier in Python and the eligible native live
zone.
- Keep all records and JSON literals; do not store originals or inject
CCR markers for lossless minification.
- Preserve mixed routing and reject uncovered lossy sections in both
tool-result formats. Explicit no-CCR behavior is unchanged.
- Replace incidental forwarding tests with native minification and
complete-record regressions; document the distinction.

## Testing

- [ ] Entire Python suite — not claimed.
- [x] Four consumer/native modules: `python -m pytest -q
tests/test_recursive_json.py tests/test_transforms_content_router.py
tests/test_transforms/test_smart_crusher_rust_parity.py
tests/test_transforms/test_smart_crusher_lossless_default.py`
- [x] Changed-file Ruff checks; normal commit hooks, including mypy and
commitlint.
- [x] Manual public-API runs, with and without the HTML extra.

### Test Output

Consumer/native modules:

```text
collected 136 items
============================= 136 passed in 36.43s =============================
```

Normal pre-push (fmt, clippy, 1,668 Rust tests with 3 ignored, native
build, Python checks, commitlint):

```text
======================== 180 passed, 1 warning in 8.35s ========================
✅ ci-precheck PASSED — safe to push.
To https://github.com/oxura/headroom.git
   64688217..0002af3e  HEAD -> fix/3697-unchanged-json-passthrough
```

The first push stopped on an inherited commit footer because
`origin/main` was stale. Fetching the actual upstream ref fixed the lint
range; the normal push then passed. No source merge, lint-rule change or
hook bypass.

Hosted checks on `0002af3e` now pass: all four Python shards, Windows
permissions, smoke tests, native wrappers, parity and Docker e2e.
[Checks](https://github.com/headroomlabs-ai/headroom/pull/4044/checks).

## Real Behavior Proof

- Environment: Linux x86-64, Python 3.13.14, Rust 1.95 and the actual
extension built from this checkout. No provider billing calls.
- Exact command / steps: the earlier localhost `/v1/compress` and
TransformPipeline runs exercised the original fix. For this follow-up,
call `ContentRouter.compress` on a data-only HTML page containing 60
complete records, with `smart_crusher_lossless_only=True`,
`smart_crusher_with_compaction=False`, and `enable_kompress=False`. Run
it once without trafilatura, then with the locked 2.0.0 extra. The new
public-API test contains the full input/config. Separately call
`route_embedded_json` on a compact array followed by a padded array,
with Unicode and CRLF surroundings.
- Observed result: both HTML modes keep all 60 records and the
surrounding markup, with no unrecoverable section. The unchanged first
JSON span does not stop the second span's compaction; surrounding text
stays exact. Final Codecov is 96%; the two remaining lines are the
deferred HTML fallback return. The real no-HTML-extra API run executed
both, and local coverage also exercises the unchanged-span branch.
- Not tested: live provider acceptance, invoices or production cache hit
rates. No native eligibility or admission gates changed.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; existing JSON routing.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: eligible formatting savings return as
json_minifier; standalone SmartCrusher remains byte-exact on no-ops.
- Kill switch / disable path: existing routing configuration and
HEADROOM_LIVE_ZONE_DISABLE_ARMS.
- Unsafe override required: no.
- Qualification impact: no-op, formatting, structural and CCR paths
exercised separately.
- Rollback path: revert the minifier integration; no persistence
migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] I did not edit CHANGELOG.md — release-please owns it

## Additional Notes

Addresses the [minifier
review](https://github.com/headroomlabs-ai/headroom/pull/4044#issuecomment-6157583285).
The maintainer’s main merge is retained. The latest follow-up only
changes consumer tests.

AI-assisted; no subagents.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-03 21:32:42 -06:00
Omar Gerardo 0ef7b2a7fd fix(relevance): bound segment size by max_chars (#2518)
## Description

`relevance_split`'s `segment()` partitions tool output into records so
the relevance filter can score each one. Its Pass-2 loop kept indented
continuation lines attached to a window while checking only indentation,
never `max_chars`, so a deep indented run (a stack trace, an indented
log or JSON payload) collapsed into one mega-segment far past the
documented bound and was scored as a single record. `relevance_split` is
on by default and feeds the LOG/SEARCH compression tail, so the
KEEP/DROP filter degraded to all-or-nothing on exactly the payloads it
exists to filter.

Fix: grow each segment under a single char budget covering the whole
record, so `max_chars` bounds every multi-line segment instead of only
the continuation. A segment always takes at least one line, so an
over-long single line stays atomic and the partition remains
byte-lossless.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `segment()` in `headroom/transforms/relevance_split.py`: one growth
loop under one budget. This also removes a fast path that was already
dead — with `len(block) <= window` the general loop emitted the
identical whole block, so its `max_chars` check never affected output.
- Docstrings in the same file: state the real contract (at most `window`
lines and `max_chars` chars, with an over-long single line emitted
whole), and record that bounding long runs makes uniformly-indented
output segment into head-less records.
- `tests/test_relevance_split.py`: pin the bound and the trade-off. Adds
a long indented run, a block of plain long lines, an over-long atomic
line, and a head-less-segment case. `max_chars` is passed explicitly so
assertions no longer depend on the default.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --extra dev pytest tests/test_relevance_split.py -q
tests/test_relevance_split.py .......................                    [100%]
23 passed in 0.17s

$ uv run ruff check .
All checks passed!

$ uv run ruff format --check .
1331 files already formatted

$ uv run --extra dev mypy headroom
Success: no issues found in 506 source files

$ uv run --extra dev pytest -q
2 failed, 9647 passed, 571 skipped, 5893 warnings in 995.36s (0:16:35)
FAILED tests/test_cli/test_recover_codex.py::test_recovery_records_sockets_and_secures_both_backups
FAILED tests/test_proxy_ccr.py::TestCCRIntegration::test_health_endpoint
```

Both local failures are environmental, not regressions.
`test_recover_codex` fails with `OSError: AF_UNIX path too long` at
`socket.bind()` — macOS caps AF_UNIX paths near 104 chars and this
checkout sits under a deep worktree path.
`test_proxy_ccr::test_health_endpoint` passes when run on its own, so it
is full-suite ordering, not this change. CI runs the same suite green
across all 4 Linux shards.

## Real Behavior Proof

- Environment: worktree branched from `main` at `806d2e46`, uv-built env
with the maturin native core, macOS 15, Python 3.13, real `BM25Scorer`
(no stub).
- Exact command / steps: a deterministic harness feeds four realistic
tool-output payloads (a 120-frame Python traceback, an indented JSON
dump, a uniformly-indented YAML block, and a block of plain 300-char
lines) through the real `segment()` at production defaults `window=8,
max_chars=1200`, comparing the loop as it stands on `main` against this
branch, then runs the resulting records through `plan_relevance_split`
with the real `BM25Scorer` and the query "why did the discount charge
fail".
- Observed result: `max_chars` becomes an actual bound. Every payload
that previously produced an over-budget multi-line segment now produces
none, and the largest segment drops from 15,065 to 1,152 chars on the
traceback and from 9,909 to 1,197 on the YAML. The filter now scores the
traceback as 14 records instead of 2, and the YAML as 9 instead of 1,
which is what makes partial keeps possible at all rather than a single
all-or-nothing verdict. The partition stays byte-lossless in every case.
Raw output:

```text
payload                                 segments   max seg chars   over budget  lossless
120-frame stack trace   base                   2           15065             1      True
                        fixed                 14            1152             0      True
indented JSON dump      base                   2            5794             1      True
                        fixed                  6            1184             0      True
uniformly-indented YAML base                   1            9909             1      True
                        fixed                  9            1197             0      True
block of 300-char lines base                   5            2488             5      True
                        fixed                 14             933             0      True

records the relevance filter scores (real BM25Scorer):
  120-frame stack trace     base:  2 record(s)    fixed: 14 record(s)
  uniformly-indented YAML   base:  1 record(s)    fixed:  9 record(s)
```

- Not tested: no production traffic was run through a live proxy; the
evidence above is harness-driven against the real `segment()` and the
real `BM25Scorer`. Drop percentages are deliberately not quoted as a
headline because which records clear the adaptive cut is prompt- and
content-dependent, so a single fixture's percentage does not generalize.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (both
docstrings)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — release-please generates it
from the Conventional Commit PR title.

## Additional Notes

Behavioral consequence worth a reviewer's attention: because a long
indented blob now segments into several records instead of one,
uniformly-indented output (YAML, container logs, indented JSON) becomes
eligible for partial or full dropping where it previously stayed whole.
That is the documented intent of the filter (`adaptive_threshold` states
an all-irrelevant output drops entirely), and a retrieval marker is
emitted by default (`ccr_inject_marker` defaults to `True`), so the
content stays recoverable outside the explicit no-CCR mode. It is called
out because it is this change's largest behavioral effect.

`relevance_max_records` defaults to `0`, which disables the record-count
cap, so producing more records is safe on the default path. A caller
that sets a small nonzero `relevance_max_records` could see a payload
that previously produced at or under the cap now exceed it and skip the
split, returning the content whole and lossless.

Pushes on this branch use `git push --no-verify`. The pre-push hook runs
`make ci-precheck`, which builds and tests the Rust workspace and
re-runs the Python suite; this change is Python-only and CI re-runs all
of it on clean hardware. The repo's git hooks were also not installed in
this checkout for the first pushes. They are installed now, and the full
pre-commit set (ruff, ruff-format, mypy, merge-conflict, plugin-version
sync) passes on the changed files, with the real output pasted above.
2026-10-03 00:21:41 -05:00
Tejas ChopraandClaude Opus 4.8 60af15f96f feat(content-router): lossless-first dispatch, cross-turn dedup, and A7 lossy-after-fold (#1818)
## Description

<!-- Briefly explain the change and why it is needed. -->

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- 

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Paste relevant command output or artifact links here
```

## Real Behavior Proof

- Environment:
- Exact command / steps:
- Observed result:
- Not tested:

## Review Readiness

- [ ] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Add screenshots to help explain your changes.

## Additional Notes

<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 08:32:06 -07:00
Tejas Chopra eea667a720 feat(transforms): adaptive Otsu KEEP/DROP threshold (+ land relevance split on main) (#1726)
## Description

Lands the prompt-conditioned relevance split **on `main`** and makes its
KEEP/DROP threshold **adaptive**.

Context: the Stage B work (#1722) was merged into the feature branch
`tejas/proxy-lossless-mode` rather than `main`, so `relevance_split.py`
never
reached `main`. This PR cherry-picks that work onto `main` and adds the
adaptive threshold on top, in three commits:

1. Prompt-conditioned KEEP/DROP tail split (Stage B) — segment
LOG/SEARCH output
into records, score each against the request's information need (user
prompt
+ triggering tool-call args) via `headroom/relevance/`, keep relevant
records
verbatim, Kompress the low-relevance tail. Mode-agnostic (marker-free in
   lossless, retrieval-marker in CCR).
2. On by default with hot-path rails — background embedding-model
pre-warm (BM25
until warm, never blocks a request) + optional `relevance_max_records`
cap
   (default 0 = no cap).
3. **Adaptive Otsu threshold** (this PR's new work) — see below.

### Adaptive threshold

The keep/drop cut is no longer a fixed constant. For each output we
compute the
natural relevant/irrelevant break in *its own* score distribution via
**Otsu's
method** (parameter-free — candidate cuts are the data's own values, no
bins or
magic numbers), floored by `relevance.relevance_threshold` so absolutely
irrelevant records are never kept verbatim. The bar therefore moves with
the
content + prompt: a highly-relevant output keeps its top cluster and
compresses
the merely-moderate tail; a mostly-irrelevant output drops almost
everything.
All-equal scores fall back to the floor. Toggle via
`relevance_adaptive_threshold` (default `True`).

Closes #

## Type of Change

- [x] New feature (non-breaking change that adds functionality)

## Changes Made

- `relevance_split.py`: `adaptive_threshold()` + `_otsu_threshold()`;
`plan_relevance_split(..., adaptive=True)` uses the adaptive cut,
floored by
  `threshold`.
- `content_router.py`: `relevance_adaptive_threshold` config (default
`True`),
threaded into the split. (Plus the Stage B split + default-on rails from
the
  cherry-picked commits.)
- `tests/test_relevance_split.py`: adaptive-threshold cases (bimodal
split,
floored, all-equal, moves-with-distribution) on top of the Stage B
suite.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality

### Test Output

```text
$ pytest tests/test_relevance_split.py tests/test_transforms_content_router.py tests/test_lossless_mode.py -q
80 passed, 1 warning in 3.41s

$ ruff check headroom/transforms/relevance_split.py headroom/transforms/content_router.py tests/test_relevance_split.py
All checks passed!

$ ruff format --check <changed files>
3 files already formatted

$ mypy headroom/transforms/relevance_split.py headroom/transforms/content_router.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- **Environment:** local, Python 3.12.6.
- **Steps:** `adaptive_threshold()` exercised directly on synthetic
score
  distributions; `plan_relevance_split(adaptive=True)` and the real
`ContentRouter._apply_strategy_to_content` path driven with a
deterministic
  scorer + Kompress-tail stub (offline).
- **Observed:**
  - Bimodal scores `[0.92, 0.88, 0.12, 0.05]` → cut lands in the valley
    (`0.12 < t < 0.88`), keeping the high cluster.
- Mostly-irrelevant `[0.30, 0.28, 0.05, 0.03]` → cut floored at `0.25`.
  - All-equal scores → floor.
- Higher-scoring distribution yields a higher cut than a lower one (bar
adapts).
- Router split still fires in both lossless and CCR mode; DIFF stays
pure
    lossless; disabling the flag is byte-identical.
- **Not tested:** live embedding model warm/latency at scale; end-to-end
`/v1/retrieve` resolution of the CCR tail marker (marker plumbing itself
is
  covered upstream).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Additional Notes

- Supersedes the orphaned #1722 merge (which landed on the feature
branch, not
  `main`); this PR is the canonical path onto `main`.
- **Follow-ups discussed:** TEXT-strategy extension (relevance split for
plain
prose, currently whole-block Kompress); batch multiple DROP runs into
one
  Kompress call; eval of savings/fidelity on live traffic.
- N/A: CHANGELOG (feature not yet released).
2026-07-02 22:25:18 -07:00