mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
5c72d824f4 |
refactor(transforms): remove unused convenience wrappers and helpers (#4097)
## Description
This PR removes small helpers that nothing calls. None is in an
`__all__`, a lazy export map or the published docs, and none is imported
by a consumer repo. `align_for_cache` was in an earlier revision of this
PR. Its docstring calls it a stable public API, so it is now kept here
and deprecated in the deprecation PR instead.
Refs #4136
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- `content_router.route_and_compress`, `code_compressor.compress_code`,
`pipeline.create_pipeline` and `base.split_frozen`: one-off wrappers
with no callers.
- `image.trained_router.get_trained_router`: no callers.
- `content_detector.is_json_array_of_dicts`: test-only. The tests now
assert the `is_dict_array` metadata directly. The Rust
`_core.is_json_array_of_dicts` stays.
- `kompress_compressor._acquire_bounded`: test-only, superseded by the
live acquire path. Its two tests go too.
- `ContentRouter._freeze_pin_chars`: a write-only counter.
- Evidence: `rg -nw '<name>' headroom crates plugins sdk examples
benchmarks scripts e2e docs wiki`, plus the consumer repos, finds only
each definition (and, for the test-only items, the tests named above).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_transforms_content_detection.py tests/test_kompress_failsafe.py tests/test_transforms/test_content_router.py tests/test_transforms_content_router.py tests/test_transforms/test_code_compressor.py tests/test_image_compressor.py tests/test_transforms_package.py tests/test_cache_aligner_prefix_stability.py tests/test_cache_aligner_detector_only.py tests/test_pipeline.py tests/test_canonical_pipeline.py tests/test_transforms/test_pipeline_waste_signal_limit.py tests/test_pipeline_outcome_stage.py
405 passed in 19.78s
$ ruff check / format: All checks passed; mypy (4 modules): Success
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) the content-detector metadata check, `hasattr` checks
for the removed wrappers, and an import of the kept `align_for_cache`.
- Observed result: The proxy serves both requests (200) with the same
savings as main. The detector reports `is_dict_array`, the removed
wrappers are gone, and `align_for_cache` is still importable. Output
below.
- Not tested: Nothing beyond the listed files.
```text
branch: cleanup/transforms-remove-dead-helpers @
|
||
|
|
f085c47402 |
fix(transforms): classify grep -A/-B/-C context lines as search results (#3599)
## Description `grep -A`/`-B`/`-C` context lines (both the real GNU `path-NN-content` shape and the reported `path:NN-content` shape) were classified as `PLAIN_TEXT`, so code inside them was routed to the word-dropping Kompress prose compressor and silently corrupted (`=` dropped from assignments, whole lines removed) behind a plausible `[N words compressed to M]` marker. `content_detector._is_search_result_line` only accepted the `path:NN:content` match shape, and the mixed-content splitter had the same gap. This change recognizes both context-line shapes as `SEARCH_RESULTS` with path-shape guards (prefix must contain `/` or `.`, same `</>=` exclusions as the colon branch) so dates and dashed prose stay out. Closes #3580 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `_is_grep_context_line` predicate plus a `_GREP_COLON_DASH_PATTERN` branch in `headroom/transforms/content_detector.py`, sharing a `_prefix_looks_like_path` guard so the three grep-line shapes cannot drift. - Taught the mixed-content splitter (`headroom/transforms/mixed_content.py`) to carve the same shapes into `SEARCH_RESULTS` sections via the shared predicate. - Added one detection-signal table row in `docs/content/docs/text-and-logs.mdx`. - Added 6 regression tests (both separator shapes, mixed match/context blocks with `--` separators, date/dashed-prose/markup guard cases). - Thresholds, confidences, and the Rust production chain are untouched; the SearchCompressor already parses `-` context lines and keeps them verbatim. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text python -m pytest tests/test_transforms_content_detection.py tests/test_mixed_content_sections.py \ tests/test_mixed_content_scan_cache.py tests/test_transforms_content_router.py \ tests/test_transforms/test_content_router.py tests/test_content_router_detection_dedup.py \ tests/test_transforms/test_detect_fallback_1123.py tests/test_search_compressor.py \ tests/test_transforms_search_compressor.py tests/test_search_compressor_cjk.py \ tests/test_bash_search_lossless_fold.py -q 527 passed in ~19s ruff check: clean. ruff format --check: clean. git diff --check: clean. ``` `mypy` reports no errors in the two changed source files (10 pre-existing errors elsewhere, e.g. proxy/server.py), so the type-check box above stays unchecked. ## Real Behavior Proof - Environment: Linux, submitter worktree `.venv` (native `headroom._core` built via maturin). - Exact command / steps: fed real `grep -A` output through the content router and the SearchCompressor; the exact `if manifest.scope != ...` line from the issue was checked verbatim after compression. - Observed result: match lines and context lines both route to `search`; the SearchCompressor keeps the issue's line byte-identical (before the fix, context lines routed to Kompress and were corrupted). - Not tested: full `pytest tests/` suite, `cargo test --workspace`, parity suite, and the live native (Magika/ONNX) classification path — upstream CI still required. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A (not rollout-gated). - Stable/default behavior changed: Yes — grep `-A`/`-B`/`-C` context lines now classify as search results instead of prose (the intended fix for #3580); all other inputs unchanged. - Kill switch / disable path: Not applicable; no flag, no new dependency, no I/O. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes - Prior attempt #3582 (closed unmerged) diagnosed the same area; this fix additionally covers the reported `path:NN-content` shape, which a dash-dash-only predicate would miss. - Follow-up (not this PR): port the predicate to the Rust oracle `crates/headroom-core/src/transforms/content_detector.rs`; the locked production chain in `detection.rs` is intentionally untouched. No new parity fixture was added for the same reason. - Out of scope, unchanged pre-existing behavior: pathless single-file `grep` lines (`30:def...`) and drive-letter paths (`C:\...`) still classify as before. |
||
|
|
7784bb1846 |
fix(transforms): stop folding datetime-prefixed user messages as search results (#3221)
## Description Interactive `headroom wrap copilot` sessions intermittently lose the user's message: the model answers "How can I help you today?" to a real task prompt. Root cause: Copilot CLI prepends `<current_datetime>…</current_datetime>` to every interactive user turn; the ISO-8601 timestamp matches the grep `file:line:` detector, so a datetime + one-line prompt (1 match / 2 non-empty lines = 50% ≥ 30%) classifies as `SEARCH_RESULTS`, and `SearchCompressor` — which keeps only detector-matching lines — deletes the prompt before upstream. On the OpenAI chat streaming path there is no retrieval tool, so the loss is unrecoverable. Fix: `_try_detect_search` now (a) requires the pre-colon segment to look like a file path (no `<`, `>`, `=`), and (b) requires at least two matching lines, so one coincidental `word:digits:` line can no longer classify a whole payload. A genuine one-line grep result loses nothing: all its lines match, so the compressor would have kept it verbatim anyway. Closes #3220 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/content_detector.py`: new `_is_search_result_line` helper (path-like prefix gate); `_try_detect_search` gains a two-matching-line absolute floor. - `tests/test_transforms_content_detection.py`: regression tests — datetime-prefixed one-liner not search; two-line floor; tag-like / `key=value` prefixes rejected; genuine grep output still detected. - `tests/test_transforms_content_router.py`: router-level regression — the incident payload never routes to SEARCH and the prose survives `ContentRouter().compress()`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/python -m pytest tests/test_transforms_content_detection.py tests/test_transforms_content_router.py tests/test_mixed_content_sections.py tests/test_text_compressors.py tests/test_transforms_tabular.py -q 135 passed in 21.06s $ .venv/bin/ruff check headroom/transforms/content_detector.py tests/test_transforms_content_detection.py tests/test_transforms_content_router.py All checks passed! $ .venv/bin/mypy headroom/transforms/content_detector.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS 26.5 arm64, Python 3.13, editable source build 0.37.0-dev; upstream `api.githubcopilot.com`, cheapest subscription model `kimi-k2.7-code`. - Exact command / steps: standalone copilot-routed proxy (`OPENAI_TARGET_API_URL=https://api.githubcopilot.com headroom proxy --port 8899`) + `.overlay/e2e-copilot-content-probe.sh --port 8899 --model kimi-k2.7-code`, which sends the real interactive wire shape (`<current_datetime>…` + one-line sentinel prompt, streaming) and a multi-line control. - Observed result: BEFORE the fix, probe 1 FAIL — model replied "Hello! I see the current datetime is … How can I assist you today?" with proxy log `transforms=router:search:0.50` (prompt deleted). AFTER the fix, both probes PASS — the sentinel echoes verbatim, proving the user message reached upstream intact. - Not tested: other harnesses' interactive wrappers (claude/droid/auggie send different shapes; the detector fix is generic); the mixed-content section splitter has its own grep pattern (out of scope — its 1-line "search" sections are kept verbatim, no data loss). ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A (no flag). - Stable/default behavior changed: content with exactly one `path:line:`-shaped line no longer classifies as search results (stays uncompressed instead — safe direction; compression only ever engages on ≥2 matching lines now). - Kill switch / disable path: N/A. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert; prior behavior restores (with the bug). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — proxy transform change; no UI. ## Additional Notes Detection-precision tradeoff is documented in code comments: single-line genuine grep output is no longer folded (no data loss either way — the compressor keeps all-matching content verbatim). A residual edge (prose with ≥2 coincidental `x:1:` lines in ≤6 lines) is accepted and documented in the issue. |
||
|
|
7c2f0ea079 |
feat(cache): provider-agnostic cache-mode delta + cc-agnostic prefix comparison (#1868)
## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> |
||
|
|
5194bdc5a6 |
fix(content-detector): detect and compress space-separated JSON objects (#1742)
## Description
Headroom's `detect_content_type()` only recognizes content starting with
`[` as a `JSON array. Many web search tools (SerpAPI, Tavily, custom
backends) return space-separated JSON objects instead of a real array
like follows
```json
{"title": "Result 1", "url": "..."} {"title": "Result 2", "url": "..."} {"title": "Result 3", "url": "..."}
```
That shape is detected as `PLAIN_TEXT` (confidence 0.5), so SmartCrusher
never processes it and web-search results compress 0%.
Closes #1741
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `content_detector.py`: `_try_detect_json` now recognizes a run of ≥2
whitespace-separated (space- or newline-separated) JSON objects and
returns `JSON_ARRAY` with `metadata["concatenated"] = True`. The router
already falls back to the Python regex detector when the native detector
returns `PLAIN_TEXT` (`content_router.py`), so this fixes routing on the
default backend too.
- `content_detector.py`: added `normalize_concatenated_json()` (and a
`_decode_concatenated_json()` helper) that rewrites the space-separated
shape into a canonical `[{…}, {…}]` array string.
- `smart_crusher.py`: `SmartCrusher.crush()` normalizes concatenated
JSON to a real array before handing it to the Rust crusher, so it
actually compresses.
- The change is deliberately conservative: a single object stays
unclaimed (`_try_detect_json('{"id": 1}')` → `None`), and any non-JSON
token between objects disqualifies the run. Existing `[`-array detection
is unchanged.
- Added tests and a CHANGELOG entry.
## Testing
- [x] Unit tests pass (`pytest`) — affected suites (full suite has
network-dependent ML tests that can't run offline; see note)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check .
All checks passed!
$ pytest tests/test_transforms_content_detection.py -q
............ [100%]
12 passed
$ pytest tests/test_transforms_content_router.py \
tests/test_smart_crusher_toin_attachment.py \
tests/test_transforms_tabular.py -q
96 passed, 2 skipped
# + SmartCrusher passthrough tests in test_text_compressors.py: 2 passed
```
## Real Behavior Proof
- Environment: macOS 26.5, Python 3.12.11, editable source build (`uv
pip install -e .`) with the Rust `_core` compiled locally; default
detection backend (native Rust → Python-regex fallback on PLAIN_TEXT).
- Exact command / steps: ran a 100-object space-separated `web_search`
payload through `detect_content_type()` and
`ContentRouter().compress()`, before and after the patch (repro below).
- Observed result: detection flips `PLAIN_TEXT` (conf 0.5) →
`JSON_ARRAY` (conf 1.0) and SmartCrusher compression goes from 0.0% to
34.2% (10369 → 6819 bytes) on the identical payload.
- Not tested: the native Rust *detector* path in isolation (the fix
relies on the existing documented Python-regex fallback for
`PLAIN_TEXT`); separators other than whitespace
(comma-separated-without-brackets is intentionally not claimed).
Before:
```
detected : ContentType.PLAIN_TEXT conf 0.5
strategy : CompressionStrategy.SMART_CRUSHER
orig bytes: 10369
comp bytes: 10369
reduction : 0.0%
```
After:
```
detected : ContentType.JSON_ARRAY conf 1.0
strategy : CompressionStrategy.SMART_CRUSHER
orig bytes: 10369
comp bytes: 6819
reduction : 34.2%
```
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (CHANGELOG)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
c39d9fe13b |
test: align test_error_detection with Phase 3e.1 bug fixes
This test pinned the pre-3e.1 behavior on three lines that the new KeywordDetector intentionally changes: 1. `'token'` was asserted to be in SECURITY_KEYWORDS. It was dropped from the security set in 3e.1 because it false-positived on every LLM-token reference in our own product. Updated to assert the new set (`security|password|auth|secret`) and explicitly that `token` is gone. 2. SECURITY_PATTERN was tested via "rotate the auth token" (matched via the now-removed `token` keyword). Now tested via "rotate the auth header" (matches via `auth`, which is the real security signal) plus a negative assertion that LLM-metric strings no longer fire. 3. ERROR_PATTERN test added an assertion that "Connection timeout" now flags as an error (3e.1 fixed the keyword/regex drift). PRIORITY_PATTERNS_TEXT indices were also corrected: the Rust-supplied markdown_prefixes table is ordered `# `, `## `, `### `, `#### `, `**`, `> ` (six prefixes) so the bold/blockquote assertions live at indices 6 and 7, not 3 and 4. New `# ` and `## ` checks pin the lower indices. Each diverging assertion carries a `fixed_in_3e1` comment so the audit trail stays clear. |
||
|
|
efd2ac1ca4 |
chore: renormalize line endings to LF
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but 74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings, violating that contract. Every macOS/Linux clone reports these files as "modified" on fresh checkout because git's diff engine sees the stored bytes don't match the attribute contract, even though the working tree and index match byte-for-byte. Running `git add --renormalize .` rewrites each affected blob so the stored form matches the attribute declaration. No semantic changes — every affected file's diff is "N insertions, N deletions" with inserts and deletes being the same lines modulo line endings. Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub blame skip this mechanical commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
38bf3e639c |
test: expand coverage across helper slices
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |