mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
## Description The lossless search fold splits each line as `path:line:content` and factors a repeated `path` into a header. `_TIMESTAMP_ROW_RE` (#3419) keeps timestamped log lines out of it, because a log line has the same colon shape: folding it hoists the date and hour into a fake file header and every row starts mid-clock. The fold round-trips byte-exactly, so the inverse check in `compact_lossless` cannot catch it. The guard knew ISO, `MM/DD/YYYY`, syslog and bare-clock prefixes, but not these four common formats: | Format | Line | Split as | |---|---|---| | Go `log` package | `2026/09/02 14:30:00 ERROR ...` | `2026/09/02 14` / `30` / `00 ERROR ...` | | nginx error log | `2026/09/02 14:30:00 [error] ...` | same | | klog (Kubernetes components) | `E0902 14:30:00.000000 1 reflector.go:100] ...` | `E0902 14` / `30` / `00.000000 ...` | | common/combined access log | `10.0.0.7 - - [02/Sep/2026:14:30:00 +0000] ...` | `10.0.0.7 - - [02/Sep/2026` / `14` / `30:00 ...` | They reach the fold two ways: the bash-search fold (`grep ERROR api.log`) and the router's lossless-first stage, which tries the search fold on any tool output (`cat`, `tail`, `kubectl logs`). This adds those three prefix shapes to `_TIMESTAMP_ROW_RE`, the same place #3419 put the guard. The content detector (#3748) and the mixed-content splitter (#4463) share the regex; neither claimed these lines before, so for them only false positives could change, and there are none on real grep output (below). The access-log alternative differs from the one I suggested in #4900. That one (`\S+ \S+ \S+ \[...`) also matched a real `grep -n` row over an access log (`/var/log/nginx/access.log:12:10.0.0.7 - - [02/Sep/...`), because `\S+` swallows `path:12:10.0.0.7`. The fields now exclude `:`, which a mis-split line cannot contain before its first colon anyway. The host field takes name/IPv4 characters only (`[\w.-]`), capped at 255 (a DNS name is at most 253). With `[^\s:]` the check scanned every grep row's whole path, and fold + detection on 10k grep rows took 10% and 32% longer than main in two interleaved runs; with this class and cap it stops early on a path and on a long unbroken line such as a base64 blob. Closes #4900 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/lossless_compaction.py`: three alternatives in `_TIMESTAMP_ROW_RE`: `YYYY/MM/DD hh:mm` (Go log, nginx error log), `[IWEF]MMDD hh:mm` (klog/glog), and the access-log prefix `host ident user [dd/Mon/yyyy:hh:mm` with a `[\w.-]{1,255}` host. - `tests/test_lossless_timestamp_guard.py`: the four formats, plus an access log with a 253-character host, join `test_timestamped_logs_are_never_folded`. New `test_grep_rows_over_those_logs_still_fold`: real `grep -n` rows over the same logs (including a dated `2026/09/02/api.log` path) still fold under their file header. - `tests/test_bash_search_lossless_fold.py`: `test_go_log_keeps_its_timestamps` runs a Go log through `ContentRouter` as the output of `grep ERROR /var/log/api.log` (bash-search fold) and of `cat /var/log/api.log` (lossless-first stage). ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # this branch's tests against main's lossless_compaction.py (main66e00869): $ pytest -q -rf tests/test_lossless_timestamp_guard.py "tests/test_bash_search_lossless_fold.py::test_go_log_keeps_its_timestamps" FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[go-log] FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[nginx-error] FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[klog] FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[access-log] FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[access-log-longest-host] FAILED tests/test_bash_search_lossless_fold.py::test_go_log_keeps_its_timestamps[grep ERROR /var/log/api.log] FAILED tests/test_bash_search_lossless_fold.py::test_go_log_keeps_its_timestamps[cat /var/log/api.log] ======================== 7 failed, 12 passed in 19.98s ======================== # same command on this branch: ============================= 19 passed in 20.52s ============================= # mutations of the fix, each run against the same tests: == M1 drop Go/nginx alt: ======================== 4 failed, 15 passed in 21.51s ======================== == M2 drop klog alt: ======================== 1 failed, 18 passed in 15.91s ======================== == M3 drop access-log alt: ======================== 2 failed, 17 passed in 13.28s ======================== == M4 access-log alt as proposed in issue (\S+): ======================== 1 failed, 18 passed in 14.13s ======================== FAILED tests/test_lossless_timestamp_guard.py::test_grep_rows_over_those_logs_still_fold[access-log] == M5 host cap too small {1,15}: ======================== 1 failed, 18 passed in 13.45s ======================== == M6 host cap removed: ============================= 19 passed in 12.49s ============================= == M7 host class widened to [^\s:]: ============================= 19 passed in 12.55s ============================= # false positives, the measure used on #3419: every line of every tracked text file in # this repo as a real grep row (git grep -n -I -e ''), old guard vs this branch: lines=977737 grep_rows=977737 rejected_by_old_guard=0 rejected_by_new_guard=0 newly_rejected=0 # cost (M6/M7 survive because the cap and class only bound the scan): main fold+detect+split, 10k real grep rows: median 37.4 ms (min 34.7) final fold+detect+split, 10k real grep rows: median 38.7 ms (min 37.1) 1MB word-char line main 0.000 ms per match 1MB word-char line final 0.001 ms per match 1MB base64 line main 0.000 ms per match 1MB base64 line final 0.000 ms per match # an earlier run with the host as unbounded [^\s:]+ (not used): 1MB word-char line unbounded 5.571 ms per match 1MB base64 line unbounded 6.566 ms per match # full suite on this machine: pytest tests scripts/tests -n 8 --continue-on-collection-errors # main5023c84e: = 459 failed, 18027 passed, 1154 skipped, 42 warnings, 417 errors in 9256.57s (2:34:16) = # this branch on66e00869: = 454 failed, 18085 passed, 1154 skipped, 39 warnings, 417 errors in 2921.98s (0:48:41) = # 8 failures appear only in the branch run. Each rerun 3x serially on main66e00869and on the branch: [wtmain] ppp tests/test_fastembed_pinned_snapshot.py::test_token_fixture_matches_huggingface_hub[HF_TOKEN wins over the login file] [wtmain] ppp tests/test_image_compression_isolation.py::test_one_timeout_does_not_destroy_a_concurrent_request [wtmain] ppp tests/test_offline_egress_chokepoint.py::TestLiteLLMCostMapOffline::test_import_with_offline_set_after_package_load_opens_no_socket[headroom.backends.litellm] [wtmain] ppp tests/test_openai_chat_buffered_ccr.py::test_a_200_with_no_usable_choice_is_not_replayed_as_an_empty_stream[payload7] [wtmain] ppp tests/test_openai_chat_buffered_ccr.py::test_failed_continuation_fails_closed_with_its_status[429-rate_limit_error] [wtmain] ppp tests/test_read_maturation_handler_nobust.py::test_quiesce_turns_config_is_honored [wtmain] FFF tests/test_stage_timer.py::test_concurrent_measure_calls_are_independent [wtmain] ppp tests/test_stateless_no_disk.py::test_cli_stateless_flag_exports_env_for_children [wt4900] ppp tests/test_fastembed_pinned_snapshot.py::test_token_fixture_matches_huggingface_hub[HF_TOKEN wins over the login file] [wt4900] ppp tests/test_image_compression_isolation.py::test_one_timeout_does_not_destroy_a_concurrent_request [wt4900] ppp tests/test_offline_egress_chokepoint.py::TestLiteLLMCostMapOffline::test_import_with_offline_set_after_package_load_opens_no_socket[headroom.backends.litellm] [wt4900] ppp tests/test_openai_chat_buffered_ccr.py::test_a_200_with_no_usable_choice_is_not_replayed_as_an_empty_stream[payload7] [wt4900] ppp tests/test_openai_chat_buffered_ccr.py::test_failed_continuation_fails_closed_with_its_status[429-rate_limit_error] [wt4900] ppp tests/test_read_maturation_handler_nobust.py::test_quiesce_turns_config_is_honored [wt4900] FFF tests/test_stage_timer.py::test_concurrent_measure_calls_are_independent [wt4900] ppp tests/test_stateless_no_disk.py::test_cli_stateless_flag_exports_env_for_children $ ruff check . (0.17.4) All checks passed! $ ruff format --check . (0.17.4) 1954 files already formatted $ mypy headroom/transforms/lossless_compaction.py headroom/transforms/content_detector.py headroom/transforms/mixed_content.py --ignore-missing-imports --platform linux --follow-imports=silent (1.20.2) Success: no issues found in 3 source files ``` `mypy headroom` is unchecked because it is not clean on this machine for main either. With mypy 1.20.2 from the pre-commit hook and `--platform linux`, main and this branch both report the same single error, `upstream_guard.py:162` (`no-any-return`), which comes from that environment having no `httpx` installed. Without `--platform linux`, both also report the Windows-only `os` attribute errors from #4520. ## Real Behavior Proof - Environment: Windows 11, Python 3.11.9, this repo at5023c84e(main) and this branch on the same base via `PYTHONPATH=.` (the branch was then rebased onto66e00869, whose three new commits touch only OpenCode files), Rust core taken from the released headroom-ai 0.44.0 wheel (main is 0.44.0), `HEADROOM_OFFLINE=1`, pure-Python content detection (the default on Windows). - Exact command / steps: the reproduction script from #4900 (`compact_lossless(text, "search")` on 30 lines of each format), then an Anthropic `Bash` `tool_use` plus a 40-line `tool_result` through `ContentRouter(ContentRouterConfig()).apply(...)` for `kubectl logs ...` (klog), `tail -n 200 /var/log/nginx/access.log`, `cat /var/log/api.log` and `grep ERROR /var/log/api.log` (Go log). - Observed result: on main each format folds into a fake header (`2026/09/02 14`, `E0902 14`, `10.0.0.7 - - [02/Sep/2026`) and each router case applies `lossless_search`; on this branch every case comes back byte-identical with its timestamps, and the router reports `router:noop`. - Not tested: a live proxy session with a real client and model (in-process `ContentRouter` only); Linux and macOS (the change is a pure-Python regex; CI covers ubuntu); the Rust core has no copy of this fold, so there is nothing to port. ```text main (5023c84e): --- go log: 1489 -> 1083 chars 2026/09/02 14 30:00 ERROR upstream timeout req=0 --- klog: 2359 -> 2098 chars E0902 14 30:00.000000 1 reflector.go:0] failed to list *v1.Pod: timeout --- access log: 2359 -> 1605 chars 10.0.0.7 - - [02/Sep/2026 14:30:00 +0000] "GET /api/items/0 HTTP/1.1" 500 512 klog N=40 5399->5048 fake_header=True first='E0902 14' transforms: ['router:tool_result:lossless_search'] access N=40 3829->2815 fake_header=True first='10.0.0.7 - - [02/Sep/2026' transforms: ['router:tool_result:lossless_search'] go N=40 3079->2533 fake_header=True first='2026/09/02 14' transforms: ['router:tool_result:lossless_search'] go-grep N=40 3079->2533 fake_header=True first='2026/09/02 14' transforms: ['router:bash:lossless_search'] this branch: --- go log: 1489 -> 1489 chars 2026/09/02 14:30:00 ERROR upstream timeout req=0 --- klog: 2359 -> 2359 chars E0902 14:30:00.000000 1 reflector.go:0] failed to list *v1.Pod: timeout --- access log: 2359 -> 2359 chars 10.0.0.7 - - [02/Sep/2026:14:30:00 +0000] "GET /api/items/0 HTTP/1.1" 500 512 klog N=40 5399->5399 fake_header=False first='E0902 14:30:00.000000 1 reflector.go:100] failed to li' transforms: ['router:noop'] access N=40 3829->3829 fake_header=False first='10.0.0.7 - - [02/Sep/2026:14:30:00 +0000] "GET /api/items/0 ' transforms: ['router:noop'] go N=40 3079->3079 fake_header=False first='2026/09/02 14:30:00 ERROR handler: upstream timeout after 90' transforms: ['router:noop'] go-grep N=40 3079->3079 fake_header=False first='2026/09/02 14:30:00 ERROR handler: upstream timeout after 90' transforms: ['router:noop'] ``` ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A (no feature flag involved). - Stable/default behavior changed: Yes, narrowly. Lines that start with a Go/nginx, klog or access-log timestamp no longer take the grep fold; real grep output folds as before (0 of 977,737 real grep rows newly rejected). - Kill switch / disable path: `HEADROOM_LOSSLESS_COMPACTION=0` disables every lossless fold, as before. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Documentation: N/A, the guard is internal and no docs describe its formats. - `Unit tests pass` and `New and existing unit tests pass locally` are unchecked because the full suite is not clean on this Windows machine for main either. Across the 28 test files that reference the fold, the guard, the detector or the mixed-content splitter, the branch run fails the same 8 tests as main (7 in `test_image_compressor.py`, 1 in `test_garbled_compression_fixes.py`) and nothing else; the full-suite comparison is in Test Output. - Ruby's default `Logger` format (`E, [2026-09-02T14:30:00.123456 #4321] ERROR -- : ...`) folds the same way (`I, [2026-09-02T14` header). It is not in #4900, so I left it out; it is one more alternative if you want it here. - Logs with ANSI color codes before the timestamp still fold under the `search` kind, for every format including ISO. That predates this change. - An access log whose client is an IPv6 address with an all-digit second group (`2001:4860:4860::8888 - - [...]`) splits on the address itself, not the timestamp, and folds under a `2001` header on main and on this branch. Left for a separate change. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4901" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JD Davis <mxjerrett@gmail.com>
344 lines
13 KiB
Python
344 lines
13 KiB
Python
# ruff: noqa: E402 — test sections import after helper/setup code by design.
|
|
"""Bash-search lossless fold.
|
|
|
|
`bash` is not an excluded tool, so its output normally takes the lossy strategy
|
|
path. But a read-only search run through it (grep/rg/git grep) produces byte-
|
|
losslessly foldable output — the router detects the *command* and folds it with
|
|
the same ripgrep --heading transform excluded Grep gets, instead of lossy
|
|
compression. Non-search bash commands (cat/build/mutate) are untouched.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
|
|
import pytest
|
|
|
|
from headroom.providers import OpenAIProvider
|
|
from headroom.tokenizer import Tokenizer
|
|
from headroom.transforms.content_router import (
|
|
ContentRouter,
|
|
ContentRouterConfig,
|
|
_bash_command_is_search,
|
|
_bash_program,
|
|
)
|
|
from headroom.transforms.lossless_compaction import search_unheading
|
|
|
|
SEARCH = frozenset({"grep", "egrep", "fgrep", "rg", "ripgrep", "ag", "ack"})
|
|
GREP = "".join(
|
|
f"src/module_{f}.py:{ln * 3}:matched occurrence with some real content here\n"
|
|
for f in range(6)
|
|
for ln in range(15)
|
|
)
|
|
CODE = "def foo(x):\n return x + 1\n\nclass Bar:\n value = 42\n" * 30
|
|
|
|
|
|
@pytest.fixture
|
|
def tokenizer():
|
|
provider = OpenAIProvider()
|
|
return Tokenizer(provider.get_token_counter("gpt-4o"), "gpt-4o")
|
|
|
|
|
|
# --- command parsing: peel wrappers, detect search programs ---
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"command",
|
|
[
|
|
"grep -rn foo .",
|
|
"rtk grep def headroom/transforms", # the user's token-proxy wrapper
|
|
"rg --heading pattern src/",
|
|
"git grep -n TODO",
|
|
"sudo grep root /etc/passwd",
|
|
"timeout 30 rg foo", # wrapper takes a numeric arg
|
|
"FOO=1 BAR=2 grep foo", # env assignments
|
|
"/usr/bin/grep -rn foo .", # absolute path
|
|
'bash -lc "grep -rn foo ."', # Codex-style shell -c
|
|
"nice -n 5 ack pattern", # wrapper with option arg
|
|
],
|
|
)
|
|
def test_detects_search_commands(command):
|
|
assert _bash_command_is_search(command, SEARCH) is True
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"command",
|
|
[
|
|
"cat headroom/server.py",
|
|
"cargo test",
|
|
"pytest tests/ -x",
|
|
"git diff HEAD~1", # diff, NOT search
|
|
"echo grep", # echo, not a real grep
|
|
"python script.py",
|
|
"rm -rf build",
|
|
"ls -la",
|
|
],
|
|
)
|
|
def test_ignores_non_search_commands(command):
|
|
assert _bash_command_is_search(command, SEARCH) is False
|
|
|
|
|
|
def test_bash_program_peels_wrappers():
|
|
assert _bash_program("rtk grep foo")[0] == "grep"
|
|
assert _bash_program("timeout 30 rg x")[0] == "rg"
|
|
assert _bash_program("FOO=1 /usr/bin/grep y")[0] == "grep"
|
|
assert _bash_program("")[0] == ""
|
|
|
|
|
|
# --- end-to-end through the router (both wire formats) ---
|
|
|
|
|
|
def _openai(command: str, content: str, tokenizer):
|
|
router = ContentRouter(ContentRouterConfig())
|
|
messages = [
|
|
{
|
|
"role": "assistant",
|
|
"tool_calls": [
|
|
{
|
|
"id": "c1",
|
|
"function": {"name": "bash", "arguments": json.dumps({"command": command})},
|
|
}
|
|
],
|
|
},
|
|
{"role": "tool", "tool_call_id": "c1", "content": content},
|
|
]
|
|
result = router.apply(messages, tokenizer, compress_user_messages=True)
|
|
return result.messages[1]["content"], result.transforms_applied
|
|
|
|
|
|
def _anthropic(command: str, content: str, tokenizer):
|
|
router = ContentRouter(ContentRouterConfig())
|
|
messages = [
|
|
{
|
|
"role": "assistant",
|
|
"content": [
|
|
{"type": "tool_use", "id": "t1", "name": "bash", "input": {"command": command}}
|
|
],
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": content}],
|
|
},
|
|
]
|
|
result = router.apply(messages, tokenizer, compress_user_messages=True)
|
|
return result.messages[1]["content"][0]["content"], result.transforms_applied
|
|
|
|
|
|
def test_openai_bash_grep_folds_and_recovers(tokenizer):
|
|
out, transforms = _openai("grep -rn foo .", GREP, tokenizer)
|
|
assert "router:bash:lossless_search" in transforms
|
|
assert len(out) < len(GREP)
|
|
assert search_unheading(out) == GREP # byte-exact
|
|
|
|
|
|
def test_anthropic_bash_rtk_grep_folds_and_recovers(tokenizer):
|
|
out, transforms = _anthropic("rtk grep foo headroom/", GREP, tokenizer)
|
|
assert "router:bash:lossless_search" in transforms
|
|
assert search_unheading(out) == GREP
|
|
|
|
|
|
def test_non_search_bash_command_not_folded(tokenizer):
|
|
# `cat` is not a search — must NOT take the bash-search fold.
|
|
_out, transforms = _openai("cat headroom/server.py", GREP, tokenizer)
|
|
assert "router:bash:lossless_search" not in transforms
|
|
|
|
|
|
def test_source_output_from_search_command_untouched(tokenizer):
|
|
# Command is a search, but the output isn't path:line:content — the
|
|
# reversibility guard makes compact_lossless return it unchanged.
|
|
out, transforms = _openai("grep -l foo", CODE, tokenizer)
|
|
assert "router:bash:lossless_search" not in transforms
|
|
assert out == CODE
|
|
|
|
|
|
GO_LOG = "".join(
|
|
f"2026/09/02 14:{30 + i // 60:02d}:{i % 60:02d} ERROR handler: upstream timeout req={i}\n"
|
|
for i in range(40)
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("command", ["grep ERROR /var/log/api.log", "cat /var/log/api.log"])
|
|
def test_go_log_keeps_its_timestamps(command, tokenizer):
|
|
# Split as path:line:content, `2026/09/02 14:30:00 ...` would fold into a
|
|
# `2026/09/02 14` header over `30:00 ...` rows. `cat` reaches the search
|
|
# fold through the router's lossless-first stage, `grep` through the
|
|
# bash-search fold.
|
|
out, transforms = _anthropic(command, GO_LOG, tokenizer)
|
|
assert "router:bash:lossless_search" not in transforms
|
|
assert "router:tool_result:lossless_search" not in transforms
|
|
assert "2026/09/02 14" not in out.split("\n")
|
|
|
|
|
|
# ---- path-listing fold (find/ls -1/rg -l): fold repeated parent dirs ----
|
|
from headroom.transforms.lossless_compaction import (
|
|
compact_lossless as _cl,
|
|
)
|
|
from headroom.transforms.lossless_compaction import (
|
|
path_heading as _ph,
|
|
)
|
|
from headroom.transforms.lossless_compaction import (
|
|
path_unheading as _puh,
|
|
)
|
|
|
|
|
|
def test_path_fold_roundtrip_and_shrinks_pure_list():
|
|
c = "./suma/apps/ext/core.py\n./suma/apps/ext/dao.py\n./suma/apps/other/x.py"
|
|
folded = _cl(c, "paths")
|
|
assert _puh(_ph(c)) == c # exact inverse
|
|
assert len(folded) < len(c) # shrinks
|
|
assert folded != c
|
|
c = "src/app/main.rs\nsrc/app/lib.rs\nsrc/app/util.rs\ntests/a.rs\n"
|
|
assert _cl(c, "paths") == "src/app/\nmain.rs\nlib.rs\nutil.rs\ntests/\na.rs\n"
|
|
c = "app/[slug]/page.tsx\napp/[slug]/layout.tsx\n"
|
|
assert _cl(c, "paths") == "app/[slug]/\npage.tsx\nlayout.tsx\n"
|
|
|
|
|
|
def test_path_fold_safe_passthrough_on_non_path_shapes():
|
|
# grep path:line:content is the search fold's job, not paths -> unchanged
|
|
assert _cl("a/b.py:12:def f\na/b.py:15:x", "paths") == "a/b.py:12:def f\na/b.py:15:x"
|
|
# trailing-slash dir entries and single paths -> unchanged
|
|
assert _cl("./a/b/\n./a/c/", "paths") == "./a/b/\n./a/c/"
|
|
assert _cl("./only/one.py", "paths") == "./only/one.py"
|
|
|
|
|
|
def test_path_fold_mixed_content_roundtrips_or_passes_through():
|
|
# a non-path no-slash line among paths must never corrupt: compact_lossless
|
|
# verifies and returns original if the fold isn't exactly reversible.
|
|
c = "./a/b/f.py\n./a/b/g.py\nsome log line\n./a/b/h.py"
|
|
out = _cl(c, "paths")
|
|
assert _puh(_ph(out)) == out or out == c # never corrupts
|
|
# simplest invariant: decoding whatever we emit reconstructs the input
|
|
assert _puh(_ph(c)) == c or _cl(c, "paths") == c
|
|
|
|
|
|
def test_path_fold_leaves_html_closing_tags_alone():
|
|
# `</body>` is markup, not dir `</` + base `body>`. The fold round-trips,
|
|
# but the model only sees the folded side and reads broken HTML.
|
|
tail = " </script>\n</body>\n</html>\n"
|
|
assert _cl(tail, "paths") == tail
|
|
assert _cl("</body>\n</html>", "paths") == "</body>\n</html>"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"text",
|
|
[
|
|
"{{#each people}}\n{{#if active}}\n<p>{{name}}</p>\n{{/if}}\n{{/each}}\n",
|
|
"{#each items as item}\n <li>{item}</li>\n{/each}\n{/if}\n",
|
|
"<pre>\n</body>\n</html>\n</pre>\n",
|
|
"Results for 3 hosts:\nN/A\nN/A\nN/A\n",
|
|
# sections made only of closers
|
|
"{{/if}}\n{{/each}}\n",
|
|
"{/if}\n{/each}\n",
|
|
"[/quote]\n[/b]\n",
|
|
"</body>\n</html>\n",
|
|
],
|
|
)
|
|
def test_path_fold_skips_text_that_only_contains_path_shaped_lines(text):
|
|
# Closers and slashed values among other lines are not a path listing.
|
|
assert _cl(text, "paths") == text
|
|
|
|
|
|
def test_router_keeps_html_closing_tags_in_mixed_content(monkeypatch):
|
|
# A page tail (no DOCTYPE/<html> root, so no HTML override) with a JSON-LD
|
|
# block routes MIXED; every section gets the lossless folds tried on it.
|
|
monkeypatch.delenv("HEADROOM_LOSSLESS_COMPACTION", raising=False)
|
|
ld = json.dumps(
|
|
{
|
|
"@context": "https://schema.org",
|
|
"@type": "Product",
|
|
"name": "Example Widget",
|
|
"offers": {"@type": "Offer", "price": "19.00", "priceCurrency": "USD"},
|
|
},
|
|
indent=2,
|
|
)
|
|
body = "".join(
|
|
f" <p>Paragraph {i} of the example page with some ordinary text.</p>\n"
|
|
for i in range(20)
|
|
)
|
|
tail = (
|
|
" <main>\n" + body + " </main>\n"
|
|
' <script type="application/ld+json">\n' + ld + "\n </script>\n</body>\n</html>\n"
|
|
)
|
|
result = ContentRouter(ContentRouterConfig()).compress(tail)
|
|
# Fails loudly if routing changes, so the test cannot silently stop
|
|
# covering the mixed path.
|
|
assert result.strategy_used.value == "mixed"
|
|
assert result.compressed.endswith(" </script>\n</body>\n</html>\n")
|
|
assert "</\n" not in result.compressed
|
|
|
|
|
|
# ---- EXPERIMENT: HEADROOM_EXPERIMENTAL_READ_KEEP_RATIO (light Kompress on reads) ----
|
|
def test_experimental_read_keep_ratio_flag_and_gating(monkeypatch):
|
|
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
|
|
|
|
# OFF by default -> verbatim (helper returns None, no compression attempted)
|
|
monkeypatch.delenv("HEADROOM_EXPERIMENTAL_READ_KEEP_RATIO", raising=False)
|
|
r_off = ContentRouter(ContentRouterConfig())
|
|
assert r_off._exp_read_keep_ratio == 0.0
|
|
assert r_off._experimental_compress_read("x" * 500) is None
|
|
|
|
# ON -> calls Kompress at the ratio; keeps result only if it actually shrank
|
|
monkeypatch.setenv("HEADROOM_EXPERIMENTAL_READ_KEEP_RATIO", "0.9")
|
|
r_on = ContentRouter(ContentRouterConfig())
|
|
assert r_on._exp_read_keep_ratio == 0.9
|
|
seen = {}
|
|
|
|
def fake_ml(content, context, question=None, target_ratio=None):
|
|
seen["ratio"] = target_ratio
|
|
return content[: len(content) // 2], 10 # pretend it shrank
|
|
|
|
monkeypatch.setattr(r_on, "_try_ml_compressor", fake_ml)
|
|
out = r_on._experimental_compress_read("y" * 500, "ctx")
|
|
assert out is not None and len(out) < 500 # adopted (shrank)
|
|
assert seen["ratio"] == 0.9 # ratio threaded through
|
|
|
|
# no-shrink -> None (fall back to verbatim protection)
|
|
monkeypatch.setattr(
|
|
r_on, "_try_ml_compressor", lambda c, ctx, question=None, target_ratio=None: (c, 1)
|
|
)
|
|
assert r_on._experimental_compress_read("z" * 500) is None
|
|
# sub-floor content never attempted
|
|
assert r_on._experimental_compress_read("short") is None
|
|
|
|
|
|
# --- directory-prefix fold: grep -rn across many distinct files ---
|
|
from headroom.transforms.lossless_compaction import ( # noqa: E402
|
|
compact_lossless,
|
|
search_dir_heading,
|
|
search_dir_unheading,
|
|
)
|
|
|
|
|
|
def test_search_dir_fold_factors_directory_across_distinct_files() -> None:
|
|
# Sorted grep -rn output: same-dir files are consecutive, one match each, so
|
|
# the file-heading fold saves nothing but the shared directory repeats on
|
|
# every row. The dir fold factors it out — byte-losslessly.
|
|
grep = (
|
|
"\n".join(f"headroom/proxy/mod_{i:02d}.py:{i + 1}: x = compress(p)" for i in range(12))
|
|
+ "\n"
|
|
)
|
|
folded = compact_lossless(grep, "search")
|
|
assert len(folded) < len(grep) # actually shrank (0% before this fold)
|
|
assert "headroom/proxy/" in folded # directory factored to a header line
|
|
assert search_dir_unheading(folded) == grep # exact byte round-trip
|
|
assert search_dir_unheading(search_dir_heading(grep)) == grep
|
|
|
|
|
|
def test_search_dir_fold_roundtrips_mixed_and_passthrough() -> None:
|
|
mixed = (
|
|
"src/a/x.py:1:hit one\nsrc/a/y.py:2:hit two\n"
|
|
"== a plain banner ==\n"
|
|
"src/b/z.py:3:content with a colon: value\nnoslash.py:4:pathless row\n"
|
|
)
|
|
out = compact_lossless(mixed, "search")
|
|
assert search_dir_unheading(out) == mixed or search_unheading(out) == mixed or out == mixed
|
|
|
|
|
|
def test_search_file_fold_still_wins_for_many_matches_one_file() -> None:
|
|
# Many matches in ONE file: the file fold is smaller, and compact_lossless
|
|
# keeps whichever candidate round-trips and is smallest.
|
|
grep = "\n".join(f"headroom/proxy/server.py:{i}: line {i}" for i in range(1, 40)) + "\n"
|
|
out = compact_lossless(grep, "search")
|
|
assert len(out) < len(grep)
|
|
assert search_unheading(out) == grep
|