Files
headroom/tests/test_bash_search_lossless_fold.py
Ayush Kumar JhaandJD Davis cf3284675e fix(lossless): never grep-fold Go, nginx, klog or access-log timestamps (#4901)
## Description

The lossless search fold splits each line as `path:line:content` and
factors a repeated `path` into a header. `_TIMESTAMP_ROW_RE` (#3419)
keeps timestamped log lines out of it, because a log line has the same
colon shape: folding it hoists the date and hour into a fake file header
and every row starts mid-clock. The fold round-trips byte-exactly, so
the inverse check in `compact_lossless` cannot catch it.

The guard knew ISO, `MM/DD/YYYY`, syslog and bare-clock prefixes, but
not these four common formats:

| Format | Line | Split as |
|---|---|---|
| Go `log` package | `2026/09/02 14:30:00 ERROR ...` | `2026/09/02 14` /
`30` / `00 ERROR ...` |
| nginx error log | `2026/09/02 14:30:00 [error] ...` | same |
| klog (Kubernetes components) | `E0902 14:30:00.000000 1
reflector.go:100] ...` | `E0902 14` / `30` / `00.000000 ...` |
| common/combined access log | `10.0.0.7 - - [02/Sep/2026:14:30:00
+0000] ...` | `10.0.0.7 - - [02/Sep/2026` / `14` / `30:00 ...` |

They reach the fold two ways: the bash-search fold (`grep ERROR
api.log`) and the router's lossless-first stage, which tries the search
fold on any tool output (`cat`, `tail`, `kubectl logs`).

This adds those three prefix shapes to `_TIMESTAMP_ROW_RE`, the same
place #3419 put the guard. The content detector (#3748) and the
mixed-content splitter (#4463) share the regex; neither claimed these
lines before, so for them only false positives could change, and there
are none on real grep output (below).

The access-log alternative differs from the one I suggested in #4900.
That one (`\S+ \S+ \S+ \[...`) also matched a real `grep -n` row over an
access log (`/var/log/nginx/access.log:12:10.0.0.7 - - [02/Sep/...`),
because `\S+` swallows `path:12:10.0.0.7`. The fields now exclude `:`,
which a mis-split line cannot contain before its first colon anyway. The
host field takes name/IPv4 characters only (`[\w.-]`), capped at 255 (a
DNS name is at most 253). With `[^\s:]` the check scanned every grep
row's whole path, and fold + detection on 10k grep rows took 10% and 32%
longer than main in two interleaved runs; with this class and cap it
stops early on a path and on a long unbroken line such as a base64 blob.

Closes #4900

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/lossless_compaction.py`: three alternatives in
`_TIMESTAMP_ROW_RE`: `YYYY/MM/DD hh:mm` (Go log, nginx error log),
`[IWEF]MMDD hh:mm` (klog/glog), and the access-log prefix `host ident
user [dd/Mon/yyyy:hh:mm` with a `[\w.-]{1,255}` host.
- `tests/test_lossless_timestamp_guard.py`: the four formats, plus an
access log with a 253-character host, join
`test_timestamped_logs_are_never_folded`. New
`test_grep_rows_over_those_logs_still_fold`: real `grep -n` rows over
the same logs (including a dated `2026/09/02/api.log` path) still fold
under their file header.
- `tests/test_bash_search_lossless_fold.py`:
`test_go_log_keeps_its_timestamps` runs a Go log through `ContentRouter`
as the output of `grep ERROR /var/log/api.log` (bash-search fold) and of
`cat /var/log/api.log` (lossless-first stage).

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# this branch's tests against main's lossless_compaction.py (main 66e00869):
$ pytest -q -rf tests/test_lossless_timestamp_guard.py "tests/test_bash_search_lossless_fold.py::test_go_log_keeps_its_timestamps"
FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[go-log]
FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[nginx-error]
FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[klog]
FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[access-log]
FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[access-log-longest-host]
FAILED tests/test_bash_search_lossless_fold.py::test_go_log_keeps_its_timestamps[grep ERROR /var/log/api.log]
FAILED tests/test_bash_search_lossless_fold.py::test_go_log_keeps_its_timestamps[cat /var/log/api.log]
======================== 7 failed, 12 passed in 19.98s ========================
# same command on this branch:
============================= 19 passed in 20.52s =============================

# mutations of the fix, each run against the same tests:
== M1 drop Go/nginx alt: ======================== 4 failed, 15 passed in 21.51s ========================
== M2 drop klog alt: ======================== 1 failed, 18 passed in 15.91s ========================
== M3 drop access-log alt: ======================== 2 failed, 17 passed in 13.28s ========================
== M4 access-log alt as proposed in issue (\S+): ======================== 1 failed, 18 passed in 14.13s ========================
    FAILED tests/test_lossless_timestamp_guard.py::test_grep_rows_over_those_logs_still_fold[access-log]
== M5 host cap too small {1,15}: ======================== 1 failed, 18 passed in 13.45s ========================
== M6 host cap removed: ============================= 19 passed in 12.49s =============================
== M7 host class widened to [^\s:]: ============================= 19 passed in 12.55s =============================

# false positives, the measure used on #3419: every line of every tracked text file in
# this repo as a real grep row (git grep -n -I -e ''), old guard vs this branch:
lines=977737 grep_rows=977737 rejected_by_old_guard=0 rejected_by_new_guard=0 newly_rejected=0

# cost (M6/M7 survive because the cap and class only bound the scan):
main   fold+detect+split, 10k real grep rows: median 37.4 ms (min 34.7)
final  fold+detect+split, 10k real grep rows: median 38.7 ms (min 37.1)
1MB word-char line main   0.000 ms per match
1MB word-char line final  0.001 ms per match
1MB base64 line    main   0.000 ms per match
1MB base64 line    final  0.000 ms per match
# an earlier run with the host as unbounded [^\s:]+ (not used):
1MB word-char line   unbounded        5.571 ms per match
1MB base64 line      unbounded        6.566 ms per match

# full suite on this machine: pytest tests scripts/tests -n 8 --continue-on-collection-errors
# main 5023c84e:
= 459 failed, 18027 passed, 1154 skipped, 42 warnings, 417 errors in 9256.57s (2:34:16) =
# this branch on 66e00869:
= 454 failed, 18085 passed, 1154 skipped, 39 warnings, 417 errors in 2921.98s (0:48:41) =
# 8 failures appear only in the branch run. Each rerun 3x serially on main 66e00869 and on the branch:
[wtmain] ppp  tests/test_fastembed_pinned_snapshot.py::test_token_fixture_matches_huggingface_hub[HF_TOKEN wins over the login file]
[wtmain] ppp  tests/test_image_compression_isolation.py::test_one_timeout_does_not_destroy_a_concurrent_request
[wtmain] ppp  tests/test_offline_egress_chokepoint.py::TestLiteLLMCostMapOffline::test_import_with_offline_set_after_package_load_opens_no_socket[headroom.backends.litellm]
[wtmain] ppp  tests/test_openai_chat_buffered_ccr.py::test_a_200_with_no_usable_choice_is_not_replayed_as_an_empty_stream[payload7]
[wtmain] ppp  tests/test_openai_chat_buffered_ccr.py::test_failed_continuation_fails_closed_with_its_status[429-rate_limit_error]
[wtmain] ppp  tests/test_read_maturation_handler_nobust.py::test_quiesce_turns_config_is_honored
[wtmain] FFF  tests/test_stage_timer.py::test_concurrent_measure_calls_are_independent
[wtmain] ppp  tests/test_stateless_no_disk.py::test_cli_stateless_flag_exports_env_for_children
[wt4900] ppp  tests/test_fastembed_pinned_snapshot.py::test_token_fixture_matches_huggingface_hub[HF_TOKEN wins over the login file]
[wt4900] ppp  tests/test_image_compression_isolation.py::test_one_timeout_does_not_destroy_a_concurrent_request
[wt4900] ppp  tests/test_offline_egress_chokepoint.py::TestLiteLLMCostMapOffline::test_import_with_offline_set_after_package_load_opens_no_socket[headroom.backends.litellm]
[wt4900] ppp  tests/test_openai_chat_buffered_ccr.py::test_a_200_with_no_usable_choice_is_not_replayed_as_an_empty_stream[payload7]
[wt4900] ppp  tests/test_openai_chat_buffered_ccr.py::test_failed_continuation_fails_closed_with_its_status[429-rate_limit_error]
[wt4900] ppp  tests/test_read_maturation_handler_nobust.py::test_quiesce_turns_config_is_honored
[wt4900] FFF  tests/test_stage_timer.py::test_concurrent_measure_calls_are_independent
[wt4900] ppp  tests/test_stateless_no_disk.py::test_cli_stateless_flag_exports_env_for_children

$ ruff check .   (0.17.4)
All checks passed!
$ ruff format --check .   (0.17.4)
1954 files already formatted
$ mypy headroom/transforms/lossless_compaction.py headroom/transforms/content_detector.py headroom/transforms/mixed_content.py --ignore-missing-imports --platform linux --follow-imports=silent   (1.20.2)
Success: no issues found in 3 source files
```

`mypy headroom` is unchecked because it is not clean on this machine for
main either. With mypy 1.20.2 from the pre-commit hook and `--platform
linux`, main and this branch both report the same single error,
`upstream_guard.py:162` (`no-any-return`), which comes from that
environment having no `httpx` installed. Without `--platform linux`,
both also report the Windows-only `os` attribute errors from #4520.

## Real Behavior Proof

- Environment: Windows 11, Python 3.11.9, this repo at 5023c84e (main)
and this branch on the same base via `PYTHONPATH=.` (the branch was then
rebased onto 66e00869, whose three new commits touch only OpenCode
files), Rust core taken from the released headroom-ai 0.44.0 wheel (main
is 0.44.0), `HEADROOM_OFFLINE=1`, pure-Python content detection (the
default on Windows).
- Exact command / steps: the reproduction script from #4900
(`compact_lossless(text, "search")` on 30 lines of each format), then an
Anthropic `Bash` `tool_use` plus a 40-line `tool_result` through
`ContentRouter(ContentRouterConfig()).apply(...)` for `kubectl logs ...`
(klog), `tail -n 200 /var/log/nginx/access.log`, `cat /var/log/api.log`
and `grep ERROR /var/log/api.log` (Go log).
- Observed result: on main each format folds into a fake header
(`2026/09/02 14`, `E0902 14`, `10.0.0.7 - - [02/Sep/2026`) and each
router case applies `lossless_search`; on this branch every case comes
back byte-identical with its timestamps, and the router reports
`router:noop`.
- Not tested: a live proxy session with a real client and model
(in-process `ContentRouter` only); Linux and macOS (the change is a
pure-Python regex; CI covers ubuntu); the Rust core has no copy of this
fold, so there is nothing to port.

```text
main (5023c84e):
--- go log: 1489 -> 1083 chars
2026/09/02 14
30:00 ERROR upstream timeout req=0
--- klog: 2359 -> 2098 chars
E0902 14
30:00.000000       1 reflector.go:0] failed to list *v1.Pod: timeout
--- access log: 2359 -> 1605 chars
10.0.0.7 - - [02/Sep/2026
14:30:00 +0000] "GET /api/items/0 HTTP/1.1" 500 512
klog     N=40 5399->5048 fake_header=True first='E0902 14'
          transforms: ['router:tool_result:lossless_search']
access   N=40 3829->2815 fake_header=True first='10.0.0.7 - - [02/Sep/2026'
          transforms: ['router:tool_result:lossless_search']
go       N=40 3079->2533 fake_header=True first='2026/09/02 14'
          transforms: ['router:tool_result:lossless_search']
go-grep  N=40 3079->2533 fake_header=True first='2026/09/02 14'
          transforms: ['router:bash:lossless_search']

this branch:
--- go log: 1489 -> 1489 chars
2026/09/02 14:30:00 ERROR upstream timeout req=0
--- klog: 2359 -> 2359 chars
E0902 14:30:00.000000       1 reflector.go:0] failed to list *v1.Pod: timeout
--- access log: 2359 -> 2359 chars
10.0.0.7 - - [02/Sep/2026:14:30:00 +0000] "GET /api/items/0 HTTP/1.1" 500 512
klog     N=40 5399->5399 fake_header=False first='E0902 14:30:00.000000       1 reflector.go:100] failed to li'
          transforms: ['router:noop']
access   N=40 3829->3829 fake_header=False first='10.0.0.7 - - [02/Sep/2026:14:30:00 +0000] "GET /api/items/0 '
          transforms: ['router:noop']
go       N=40 3079->3079 fake_header=False first='2026/09/02 14:30:00 ERROR handler: upstream timeout after 90'
          transforms: ['router:noop']
go-grep  N=40 3079->3079 fake_header=False first='2026/09/02 14:30:00 ERROR handler: upstream timeout after 90'
          transforms: ['router:noop']
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A (no feature flag involved).
- Stable/default behavior changed: Yes, narrowly. Lines that start with
a Go/nginx, klog or access-log timestamp no longer take the grep fold;
real grep output folds as before (0 of 977,737 real grep rows newly
rejected).
- Kill switch / disable path: `HEADROOM_LOSSLESS_COMPACTION=0` disables
every lossless fold, as before.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: N/A, the guard is internal and no docs describe its
formats.
- `Unit tests pass` and `New and existing unit tests pass locally` are
unchecked because the full suite is not clean on this Windows machine
for main either. Across the 28 test files that reference the fold, the
guard, the detector or the mixed-content splitter, the branch run fails
the same 8 tests as main (7 in `test_image_compressor.py`, 1 in
`test_garbled_compression_fixes.py`) and nothing else; the full-suite
comparison is in Test Output.
- Ruby's default `Logger` format (`E, [2026-09-02T14:30:00.123456 #4321]
ERROR -- : ...`) folds the same way (`I, [2026-09-02T14` header). It is
not in #4900, so I left it out; it is one more alternative if you want
it here.
- Logs with ANSI color codes before the timestamp still fold under the
`search` kind, for every format including ISO. That predates this
change.
- An access log whose client is an IPv6 address with an all-digit second
group (`2001:4860:4860::8888 - - [...]`) splits on the address itself,
not the timestamp, and folds under a `2001` header on main and on this
branch. Left for a separate change.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4901"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:20:01 -06:00

344 lines
13 KiB
Python

# ruff: noqa: E402 — test sections import after helper/setup code by design.
"""Bash-search lossless fold.
`bash` is not an excluded tool, so its output normally takes the lossy strategy
path. But a read-only search run through it (grep/rg/git grep) produces byte-
losslessly foldable output — the router detects the *command* and folds it with
the same ripgrep --heading transform excluded Grep gets, instead of lossy
compression. Non-search bash commands (cat/build/mutate) are untouched.
"""
from __future__ import annotations
import json
import pytest
from headroom.providers import OpenAIProvider
from headroom.tokenizer import Tokenizer
from headroom.transforms.content_router import (
ContentRouter,
ContentRouterConfig,
_bash_command_is_search,
_bash_program,
)
from headroom.transforms.lossless_compaction import search_unheading
SEARCH = frozenset({"grep", "egrep", "fgrep", "rg", "ripgrep", "ag", "ack"})
GREP = "".join(
f"src/module_{f}.py:{ln * 3}:matched occurrence with some real content here\n"
for f in range(6)
for ln in range(15)
)
CODE = "def foo(x):\n return x + 1\n\nclass Bar:\n value = 42\n" * 30
@pytest.fixture
def tokenizer():
provider = OpenAIProvider()
return Tokenizer(provider.get_token_counter("gpt-4o"), "gpt-4o")
# --- command parsing: peel wrappers, detect search programs ---
@pytest.mark.parametrize(
"command",
[
"grep -rn foo .",
"rtk grep def headroom/transforms", # the user's token-proxy wrapper
"rg --heading pattern src/",
"git grep -n TODO",
"sudo grep root /etc/passwd",
"timeout 30 rg foo", # wrapper takes a numeric arg
"FOO=1 BAR=2 grep foo", # env assignments
"/usr/bin/grep -rn foo .", # absolute path
'bash -lc "grep -rn foo ."', # Codex-style shell -c
"nice -n 5 ack pattern", # wrapper with option arg
],
)
def test_detects_search_commands(command):
assert _bash_command_is_search(command, SEARCH) is True
@pytest.mark.parametrize(
"command",
[
"cat headroom/server.py",
"cargo test",
"pytest tests/ -x",
"git diff HEAD~1", # diff, NOT search
"echo grep", # echo, not a real grep
"python script.py",
"rm -rf build",
"ls -la",
],
)
def test_ignores_non_search_commands(command):
assert _bash_command_is_search(command, SEARCH) is False
def test_bash_program_peels_wrappers():
assert _bash_program("rtk grep foo")[0] == "grep"
assert _bash_program("timeout 30 rg x")[0] == "rg"
assert _bash_program("FOO=1 /usr/bin/grep y")[0] == "grep"
assert _bash_program("")[0] == ""
# --- end-to-end through the router (both wire formats) ---
def _openai(command: str, content: str, tokenizer):
router = ContentRouter(ContentRouterConfig())
messages = [
{
"role": "assistant",
"tool_calls": [
{
"id": "c1",
"function": {"name": "bash", "arguments": json.dumps({"command": command})},
}
],
},
{"role": "tool", "tool_call_id": "c1", "content": content},
]
result = router.apply(messages, tokenizer, compress_user_messages=True)
return result.messages[1]["content"], result.transforms_applied
def _anthropic(command: str, content: str, tokenizer):
router = ContentRouter(ContentRouterConfig())
messages = [
{
"role": "assistant",
"content": [
{"type": "tool_use", "id": "t1", "name": "bash", "input": {"command": command}}
],
},
{
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": content}],
},
]
result = router.apply(messages, tokenizer, compress_user_messages=True)
return result.messages[1]["content"][0]["content"], result.transforms_applied
def test_openai_bash_grep_folds_and_recovers(tokenizer):
out, transforms = _openai("grep -rn foo .", GREP, tokenizer)
assert "router:bash:lossless_search" in transforms
assert len(out) < len(GREP)
assert search_unheading(out) == GREP # byte-exact
def test_anthropic_bash_rtk_grep_folds_and_recovers(tokenizer):
out, transforms = _anthropic("rtk grep foo headroom/", GREP, tokenizer)
assert "router:bash:lossless_search" in transforms
assert search_unheading(out) == GREP
def test_non_search_bash_command_not_folded(tokenizer):
# `cat` is not a search — must NOT take the bash-search fold.
_out, transforms = _openai("cat headroom/server.py", GREP, tokenizer)
assert "router:bash:lossless_search" not in transforms
def test_source_output_from_search_command_untouched(tokenizer):
# Command is a search, but the output isn't path:line:content — the
# reversibility guard makes compact_lossless return it unchanged.
out, transforms = _openai("grep -l foo", CODE, tokenizer)
assert "router:bash:lossless_search" not in transforms
assert out == CODE
GO_LOG = "".join(
f"2026/09/02 14:{30 + i // 60:02d}:{i % 60:02d} ERROR handler: upstream timeout req={i}\n"
for i in range(40)
)
@pytest.mark.parametrize("command", ["grep ERROR /var/log/api.log", "cat /var/log/api.log"])
def test_go_log_keeps_its_timestamps(command, tokenizer):
# Split as path:line:content, `2026/09/02 14:30:00 ...` would fold into a
# `2026/09/02 14` header over `30:00 ...` rows. `cat` reaches the search
# fold through the router's lossless-first stage, `grep` through the
# bash-search fold.
out, transforms = _anthropic(command, GO_LOG, tokenizer)
assert "router:bash:lossless_search" not in transforms
assert "router:tool_result:lossless_search" not in transforms
assert "2026/09/02 14" not in out.split("\n")
# ---- path-listing fold (find/ls -1/rg -l): fold repeated parent dirs ----
from headroom.transforms.lossless_compaction import (
compact_lossless as _cl,
)
from headroom.transforms.lossless_compaction import (
path_heading as _ph,
)
from headroom.transforms.lossless_compaction import (
path_unheading as _puh,
)
def test_path_fold_roundtrip_and_shrinks_pure_list():
c = "./suma/apps/ext/core.py\n./suma/apps/ext/dao.py\n./suma/apps/other/x.py"
folded = _cl(c, "paths")
assert _puh(_ph(c)) == c # exact inverse
assert len(folded) < len(c) # shrinks
assert folded != c
c = "src/app/main.rs\nsrc/app/lib.rs\nsrc/app/util.rs\ntests/a.rs\n"
assert _cl(c, "paths") == "src/app/\nmain.rs\nlib.rs\nutil.rs\ntests/\na.rs\n"
c = "app/[slug]/page.tsx\napp/[slug]/layout.tsx\n"
assert _cl(c, "paths") == "app/[slug]/\npage.tsx\nlayout.tsx\n"
def test_path_fold_safe_passthrough_on_non_path_shapes():
# grep path:line:content is the search fold's job, not paths -> unchanged
assert _cl("a/b.py:12:def f\na/b.py:15:x", "paths") == "a/b.py:12:def f\na/b.py:15:x"
# trailing-slash dir entries and single paths -> unchanged
assert _cl("./a/b/\n./a/c/", "paths") == "./a/b/\n./a/c/"
assert _cl("./only/one.py", "paths") == "./only/one.py"
def test_path_fold_mixed_content_roundtrips_or_passes_through():
# a non-path no-slash line among paths must never corrupt: compact_lossless
# verifies and returns original if the fold isn't exactly reversible.
c = "./a/b/f.py\n./a/b/g.py\nsome log line\n./a/b/h.py"
out = _cl(c, "paths")
assert _puh(_ph(out)) == out or out == c # never corrupts
# simplest invariant: decoding whatever we emit reconstructs the input
assert _puh(_ph(c)) == c or _cl(c, "paths") == c
def test_path_fold_leaves_html_closing_tags_alone():
# `</body>` is markup, not dir `</` + base `body>`. The fold round-trips,
# but the model only sees the folded side and reads broken HTML.
tail = " </script>\n</body>\n</html>\n"
assert _cl(tail, "paths") == tail
assert _cl("</body>\n</html>", "paths") == "</body>\n</html>"
@pytest.mark.parametrize(
"text",
[
"{{#each people}}\n{{#if active}}\n<p>{{name}}</p>\n{{/if}}\n{{/each}}\n",
"{#each items as item}\n <li>{item}</li>\n{/each}\n{/if}\n",
"<pre>\n&lt;/body&gt;\n&lt;/html&gt;\n</pre>\n",
"Results for 3 hosts:\nN/A\nN/A\nN/A\n",
# sections made only of closers
"{{/if}}\n{{/each}}\n",
"{/if}\n{/each}\n",
"[/quote]\n[/b]\n",
"&lt;/body&gt;\n&lt;/html&gt;\n",
],
)
def test_path_fold_skips_text_that_only_contains_path_shaped_lines(text):
# Closers and slashed values among other lines are not a path listing.
assert _cl(text, "paths") == text
def test_router_keeps_html_closing_tags_in_mixed_content(monkeypatch):
# A page tail (no DOCTYPE/<html> root, so no HTML override) with a JSON-LD
# block routes MIXED; every section gets the lossless folds tried on it.
monkeypatch.delenv("HEADROOM_LOSSLESS_COMPACTION", raising=False)
ld = json.dumps(
{
"@context": "https://schema.org",
"@type": "Product",
"name": "Example Widget",
"offers": {"@type": "Offer", "price": "19.00", "priceCurrency": "USD"},
},
indent=2,
)
body = "".join(
f" <p>Paragraph {i} of the example page with some ordinary text.</p>\n"
for i in range(20)
)
tail = (
" <main>\n" + body + " </main>\n"
' <script type="application/ld+json">\n' + ld + "\n </script>\n</body>\n</html>\n"
)
result = ContentRouter(ContentRouterConfig()).compress(tail)
# Fails loudly if routing changes, so the test cannot silently stop
# covering the mixed path.
assert result.strategy_used.value == "mixed"
assert result.compressed.endswith(" </script>\n</body>\n</html>\n")
assert "</\n" not in result.compressed
# ---- EXPERIMENT: HEADROOM_EXPERIMENTAL_READ_KEEP_RATIO (light Kompress on reads) ----
def test_experimental_read_keep_ratio_flag_and_gating(monkeypatch):
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
# OFF by default -> verbatim (helper returns None, no compression attempted)
monkeypatch.delenv("HEADROOM_EXPERIMENTAL_READ_KEEP_RATIO", raising=False)
r_off = ContentRouter(ContentRouterConfig())
assert r_off._exp_read_keep_ratio == 0.0
assert r_off._experimental_compress_read("x" * 500) is None
# ON -> calls Kompress at the ratio; keeps result only if it actually shrank
monkeypatch.setenv("HEADROOM_EXPERIMENTAL_READ_KEEP_RATIO", "0.9")
r_on = ContentRouter(ContentRouterConfig())
assert r_on._exp_read_keep_ratio == 0.9
seen = {}
def fake_ml(content, context, question=None, target_ratio=None):
seen["ratio"] = target_ratio
return content[: len(content) // 2], 10 # pretend it shrank
monkeypatch.setattr(r_on, "_try_ml_compressor", fake_ml)
out = r_on._experimental_compress_read("y" * 500, "ctx")
assert out is not None and len(out) < 500 # adopted (shrank)
assert seen["ratio"] == 0.9 # ratio threaded through
# no-shrink -> None (fall back to verbatim protection)
monkeypatch.setattr(
r_on, "_try_ml_compressor", lambda c, ctx, question=None, target_ratio=None: (c, 1)
)
assert r_on._experimental_compress_read("z" * 500) is None
# sub-floor content never attempted
assert r_on._experimental_compress_read("short") is None
# --- directory-prefix fold: grep -rn across many distinct files ---
from headroom.transforms.lossless_compaction import ( # noqa: E402
compact_lossless,
search_dir_heading,
search_dir_unheading,
)
def test_search_dir_fold_factors_directory_across_distinct_files() -> None:
# Sorted grep -rn output: same-dir files are consecutive, one match each, so
# the file-heading fold saves nothing but the shared directory repeats on
# every row. The dir fold factors it out — byte-losslessly.
grep = (
"\n".join(f"headroom/proxy/mod_{i:02d}.py:{i + 1}: x = compress(p)" for i in range(12))
+ "\n"
)
folded = compact_lossless(grep, "search")
assert len(folded) < len(grep) # actually shrank (0% before this fold)
assert "headroom/proxy/" in folded # directory factored to a header line
assert search_dir_unheading(folded) == grep # exact byte round-trip
assert search_dir_unheading(search_dir_heading(grep)) == grep
def test_search_dir_fold_roundtrips_mixed_and_passthrough() -> None:
mixed = (
"src/a/x.py:1:hit one\nsrc/a/y.py:2:hit two\n"
"== a plain banner ==\n"
"src/b/z.py:3:content with a colon: value\nnoslash.py:4:pathless row\n"
)
out = compact_lossless(mixed, "search")
assert search_dir_unheading(out) == mixed or search_unheading(out) == mixed or out == mixed
def test_search_file_fold_still_wins_for_many_matches_one_file() -> None:
# Many matches in ONE file: the file fold is smaller, and compact_lossless
# keeps whichever candidate round-trips and is smallest.
grep = "\n".join(f"headroom/proxy/server.py:{i}: line {i}" for i in range(1, 40)) + "\n"
out = compact_lossless(grep, "search")
assert len(out) < len(grep)
assert search_unheading(out) == grep