mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
feat/https-compose-caddy
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
a8df6f919f |
fix(transforms): recognise code fences whose info string is not a bare word (#4291)
## Description
`split_into_sections()` in `headroom/transforms/mixed_content.py`
recognises an opening code fence with `^```(\w*)\s*$`. Common fence tags
with non-word characters don't match: ```` ```c++ ````, ```` ```c# ````,
```` ```objective-c ````, ```` ```js {1,3} ````, ```` ```python
title="app.py" ````. When the opener is missed:
1. the fence line and code are kept as plain text;
2. the block's **closing** ```` ``` ```` matches the pattern (empty
tag), so it is taken as an **opening** fence;
3. everything after it (prose, JSON, until the next ```` ``` ```` or end
of input) becomes one `SOURCE_CODE` section with language `unknown`,
goes to the code compressor, and is re-emitted wrapped in ````
```unknown ```` … ```` ``` ````.
So the code block loses its closing fence, and the prose and data that
follow it come back inside a fake code block. Through
`headroom.compress()` this turns a tool result's JSON into a
JSON-escaped string (`\n` sequences) inside that stray fence (see the
proof below). It only takes one C++ or C# snippet in a mixed tool result
or fetched page.
Closes: N/A (no matching issue found)
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/mixed_content.py`:
- `_CODE_FENCE_PATTERN` is now `^```([^\n`]*)$`. Group 1 is the whole
info string, any text without a backtick, so ```` ```c++ ````, ````
```js {1,3} ```` and ```` ``` c++ ```` (space before the tag, valid
Markdown) all open a fence. A closing ```` ``` ```` and a plain ````
```python ```` match as before. A line with a later backtick (````
```foo``` ````) still doesn't.
- New `_fence_language()` takes the info string's first word as the
language, and returns `"unknown"` for a bare fence, as before.
- It's a single quantifier, so a 200,000-character fence line takes
about 2 ms. While working on this I measured an overlapping variant
(`([^\s`]*)[^\n`]*`) at 3.3 s on a 20,000-character line, and avoided
it.
- `tests/test_mixed_content_sections.py`:
- `test_split_accepts_fence_info_strings_beyond_word_characters`,
parametrized over `c++`, `c#`, `objective-c`, `js {1,3}`, `python
title="app.py"`, ` c++` and `python\r`, checks the sections come out
PLAIN_TEXT / SOURCE_CODE(lang) / PLAIN_TEXT / JSON_ARRAY with the code
body intact.
- `test_fence_pattern_stays_linear_on_a_long_unclosed_fence_line` keeps
a 50,000-character fence line under 0.5 s. That's far above the linear
cost (under 1 ms), so a loaded runner won't trip it, but a quadratic
pattern (seconds) would. It passes on `main` too; it's there so any
future widening stays linear, as in
`test_multiline_regex_blank_runs.py`.
## Testing
- [x] Unit tests pass (`pytest`): `tests/test_mixed_content*.py`,
`tests/test_content_router*.py`,
`tests/test_transforms_content_router.py`,
`tests/test_code_compressor*.py`,
`tests/test_multiline_regex_blank_runs.py`, `tests/test_transforms/`,
not the full suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
New tests on `main` (fix reverted, tests kept):
```text
$ python -m pytest tests/test_mixed_content_sections.py -q
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[c++-c++]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[c#-c#]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[objective-c-objective-c]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[js {1,3}-js]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[python title="app.py"-python]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[ c++-c++]
========================= 6 failed, 6 passed in 1.53s ==========================
```
With the fix:
```text
$ python -m pytest tests/test_mixed_content*.py tests/test_content_router*.py tests/test_transforms_content_router.py \
tests/test_code_compressor*.py tests/test_multiline_regex_blank_runs.py tests/test_transforms -q
====================== 1044 passed, 88 skipped in 32.30s =======================
$ ruff check headroom/transforms/mixed_content.py tests/test_mixed_content_sections.py
All checks passed!
$ ruff format --check headroom/transforms/mixed_content.py tests/test_mixed_content_sections.py
2 files already formatted
```
## Real Behavior Proof
- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel),
`HEADROOM_OFFLINE=1`, public `headroom.compress()` API
- Exact command / steps: called `headroom.compress(messages,
model="gpt-4o", model_limit=128000, protect_recent=0)` on an
OpenAI-format conversation whose tool result is: one intro line, a ````
```c++ ```` block of two declarations, 40 prose lines, then a 400-row
JSON array (13.5K tokens). I printed the `transforms_applied`, every
fence line in the compressed tool result and its tail, on `main` and on
this branch. (Through the proxy, the same content only reaches the
mixed-content path once it is old enough to be compressed, so I used the
library entry point to keep the repro to one call.)
- Observed result: both runs take the mixed path (`router:mixed`). On
`main` the output has three fence lines (`c++`, a stray `unknown`, and a
closing one), the C++ block lost its closing fence, and the JSON tail
comes back as a JSON-escaped string inside the stray fence. On this
branch the C++ block keeps both fences, the prose stays prose, and the
JSON is compacted to CSV rows as the structured path does elsewhere.
```text
== main
tokens 13474 -> 4289 | transforms: [..., 'router:mixed:0.35']
fence lines: ['```c++', '```unknown', '```']
| Here is the Header file that we wrote for the project today.
| ```c++
| int add(int a, int b);
| int sub(int a, int b);
|
| ```unknown
| Paragraph 0 explains the Design choice that we made for this module
today.
tail:
'...397,item397,team-a,ok\\n398,item398,team-a,ok\\n399,item399,team-a,ok\\n"\n```'
== this branch
tokens 13474 -> 4285 | transforms: [..., 'router:mixed:0.29']
fence lines: ['```c++', '```']
| ```c++
| int add(int a, int b);
| int sub(int a, int b);
| ```
|
| Paragraph 0 explains the Design choice that we made for this module
today.
tail:
'...397,item397,team-a,ok\n398,item398,team-a,ok\n399,item399,team-a,ok\n'
```
- Not tested: the same content through the live proxy's mixed-content
path (in a short session the turn is protected or below the size gates,
so nothing is compressed on either branch); Rust — there is no Rust
mirror of `split_into_sections`.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: fences tagged with non-word
characters (or a space before the tag) are now parsed as code blocks.
Content with only `\w` tags (```` ```python ````, ```` ```json ````,
bare ```` ``` ````) splits exactly as before; the existing 565
router/transform tests pass unchanged.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- An **unclosed** fence still runs to the end of the input, as an
unclosed ```` ```python ```` already does on `main`. This PR only
changes which openers are recognised. Making unclosed fences fall back
to plain text would change behaviour for every tag, so it seems better
as its own PR.
- The router re-wraps a fenced section with ```` ```{section.language}
````, so trailing info-string metadata (`{1,3}`, `title=...`) is not
re-emitted. That was already true for every fence; this PR only fixes
the opener being missed.
- `_extract_fenced_command()` in `content_router.py` has its own
`[\w.-]+` fence class. It only extracts a shell command for
read-detection, so I left it alone to keep this PR to one change.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
abbade9b4c |
fix(mixed-content): do not carve ISO-timestamped log rows as search results (#4463)
## Description
The mixed-content splitter's `_is_search_line` ORs the legacy
`^\S+:\d+:` pattern with the detector's predicate. That pattern matches
ISO-timestamped log rows (`2026-09-23T10:05:07Z ...`), which the
detector has rejected since #3736, so the splitter carved them into a
SEARCH_RESULTS section and `_compress_mixed` sent them to the lossy
SearchCompressor. `_is_search_line` now returns False for rows matched
by `_TIMESTAMP_ROW_RE` (the regex the detector already uses) before the
OR; real `path:NN:content` grep lines still carve.
Fixes #4462
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/mixed_content.py`: import `_TIMESTAMP_ROW_RE`
from `lossless_compaction` (the detector already imports it from there)
and return False for such rows in `_is_search_line`; docstring note.
- `tests/test_mixed_content_sections.py`: new test
`test_split_does_not_carve_timestamped_log_rows_as_search` (timestamped
rows produce no SEARCH_RESULTS section; a `src/app.py:10:` grep line
after the same prose and fence still does).
## Testing
Rebased onto upstream/main `3fef15d3` (2026-10-24): the new tests pass
(5 passed); with only the source change reverted, 1 fails. The
real-behavior run below was done on `e995eb9f`.
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
(`ruff check` and `ruff format --check` were run on the two changed
files only. `mypy headroom/transforms/mixed_content.py` reports one
error in `headroom/providers/model_metadata.py:79`, a file this PR does
not touch.)
### Test Output
```text
# new test with only the mixed_content.py change reverted
FAILED tests/test_mixed_content_sections.py::test_split_does_not_carve_timestamped_log_rows_as_search
AssertionError: assert <ContentType.SEARCH_RESULTS: 'search'> not in [<ContentType.PLAIN_TEXT: 'text'>, <ContentType.SOURCE_CODE: 'source_code'>, <ContentType.SEARCH_RESULTS: 'search'>]
1 failed, 4 passed in 1.12s
# with the fix: the new file, test_mixed_content_scan_cache.py and the content-detector tests
285 passed in 3.31s
ruff check: All checks passed!
ruff format --check: 2 files already formatted
```
## Real Behavior Proof
- Environment: Ubuntu on WSL2, Python 3.12.15, upstream/main
|
||
|
|
f085c47402 |
fix(transforms): classify grep -A/-B/-C context lines as search results (#3599)
## Description `grep -A`/`-B`/`-C` context lines (both the real GNU `path-NN-content` shape and the reported `path:NN-content` shape) were classified as `PLAIN_TEXT`, so code inside them was routed to the word-dropping Kompress prose compressor and silently corrupted (`=` dropped from assignments, whole lines removed) behind a plausible `[N words compressed to M]` marker. `content_detector._is_search_result_line` only accepted the `path:NN:content` match shape, and the mixed-content splitter had the same gap. This change recognizes both context-line shapes as `SEARCH_RESULTS` with path-shape guards (prefix must contain `/` or `.`, same `</>=` exclusions as the colon branch) so dates and dashed prose stay out. Closes #3580 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `_is_grep_context_line` predicate plus a `_GREP_COLON_DASH_PATTERN` branch in `headroom/transforms/content_detector.py`, sharing a `_prefix_looks_like_path` guard so the three grep-line shapes cannot drift. - Taught the mixed-content splitter (`headroom/transforms/mixed_content.py`) to carve the same shapes into `SEARCH_RESULTS` sections via the shared predicate. - Added one detection-signal table row in `docs/content/docs/text-and-logs.mdx`. - Added 6 regression tests (both separator shapes, mixed match/context blocks with `--` separators, date/dashed-prose/markup guard cases). - Thresholds, confidences, and the Rust production chain are untouched; the SearchCompressor already parses `-` context lines and keeps them verbatim. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text python -m pytest tests/test_transforms_content_detection.py tests/test_mixed_content_sections.py \ tests/test_mixed_content_scan_cache.py tests/test_transforms_content_router.py \ tests/test_transforms/test_content_router.py tests/test_content_router_detection_dedup.py \ tests/test_transforms/test_detect_fallback_1123.py tests/test_search_compressor.py \ tests/test_transforms_search_compressor.py tests/test_search_compressor_cjk.py \ tests/test_bash_search_lossless_fold.py -q 527 passed in ~19s ruff check: clean. ruff format --check: clean. git diff --check: clean. ``` `mypy` reports no errors in the two changed source files (10 pre-existing errors elsewhere, e.g. proxy/server.py), so the type-check box above stays unchecked. ## Real Behavior Proof - Environment: Linux, submitter worktree `.venv` (native `headroom._core` built via maturin). - Exact command / steps: fed real `grep -A` output through the content router and the SearchCompressor; the exact `if manifest.scope != ...` line from the issue was checked verbatim after compression. - Observed result: match lines and context lines both route to `search`; the SearchCompressor keeps the issue's line byte-identical (before the fix, context lines routed to Kompress and were corrupted). - Not tested: full `pytest tests/` suite, `cargo test --workspace`, parity suite, and the live native (Magika/ONNX) classification path — upstream CI still required. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A (not rollout-gated). - Stable/default behavior changed: Yes — grep `-A`/`-B`/`-C` context lines now classify as search results instead of prose (the intended fix for #3580); all other inputs unchanged. - Kill switch / disable path: Not applicable; no flag, no new dependency, no I/O. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes - Prior attempt #3582 (closed unmerged) diagnosed the same area; this fix additionally covers the reported `path:NN-content` shape, which a dash-dash-only predicate would miss. - Follow-up (not this PR): port the predicate to the Rust oracle `crates/headroom-core/src/transforms/content_detector.rs`; the locked production chain in `detection.rs` is intentionally untouched. No new parity fixture was added for the same reason. - Out of scope, unchanged pre-existing behavior: pathless single-file `grep` lines (`30:def...`) and drive-letter paths (`C:\...`) still classify as before. |
||
|
|
9bacf4810f |
refactor(transforms): isolate mixed content parsing (#1939)
## Description Extracts mixed-content parsing out of the large `ContentRouter` module into a pure transform-domain module. The router still exports the existing compatibility names, but section typing, mixed-content indicators, section splitting, and JSON block extraction now live in a focused domain object/function layer. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `headroom.transforms.mixed_content` with `ContentSection`, `mixed_content_indicators`, `is_mixed_content`, `split_into_sections`, and JSON block extraction. - Updated `ContentRouter` to delegate mixed-content debug indicators and parsing to the new module while preserving legacy imports from `content_router.py`. - Added direct unit coverage for mixed-content detection, section boundaries, and JSON delimiters inside string literals. - Included the LiteLLM callback signature compatibility shim needed for repo-wide mypy while the earlier architecture PRs are still open. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_mixed_content_sections.py tests/test_transforms_content_router.py tests/test_litellm_callback.py tests/test_compress_api.py::TestLiteLLMCallback -q 50 passed in 6.82s python -m ruff check . All checks passed! python -m ruff format --check . 1095 files already formatted python -m mypy headroom --ignore-missing-imports headroom\proxy\server.py:1457: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom\proxy\server.py:1468: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] Success: no issues found in 409 source files ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, clean worktree `C:\git\headroom-pr-slice6` - Exact command / steps: ran the pytest, Ruff, format, and mypy commands listed above. - Observed result: mixed-content parsing behavior remains covered through existing router tests and new direct tests; repo-wide lint/type checks pass. - Not tested: full pytest suite and Docker/native CI jobs are left to GitHub Actions. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes - Documentation, changelog, and screenshots are N/A for this internal refactor. - Manual UI testing is N/A; this is pure transform parsing logic. - Comment checklist is unchecked because the extracted functions are small and covered by direct tests. |