mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
> **Severity: silent data loss.** Kompress deletes whole records from single-line JSON tool output and leaves *valid* JSON behind, so no parser, log line or model can tell. This hits the default `coding` profile with `accuracy_guard=strict`, on every release up to and including v0.39.1. In our deployment an agent repeatedly and confidently told a user that a data collection did not exist; it did. The only mitigation today is `HEADROOM_DISABLE_KOMPRESS=1`, which also turns off prose compression. ## Description Keeps JSON whose record structure a prose model can destroy out of Kompress, on every route that reaches it, while prose around that JSON still compresses. This carries #3680 to a mergeable state. It is the same four commits, authorship preserved: - `113b6451` by @iuiu-py: the whole-document guard. - `2ebf6b54`: the prefixed-line and relevance-split routes, which were the P1 review. - `15b04665` by @JerrettDavis: formatting. - `9386c994` by @Ayushraj06-bit (from iuiu-py/headroom#1, open since 22 Sep and not picked up): narrows the guard, as the last review on #3680 asked. Everything is rebased on current `main` (`ffc6edb4`). #3680 has been waiting on the author since 24 Sep (`needs author action`). If @iuiu-py would rather land it there, I will close this one; nothing here is meant to take the credit. Closes #3673 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `recursive_json.json_document_spans()` lists every top-level object or array span that parses. `carries_record_array()` replaces `_has_routable_json()` under the same semantics. - `content_router._protected_json_spans()` protects two shapes: - a span carrying an array of objects, because deleting `},{` there leaves valid JSON; - a block that is one JSON document end to end. - Separate top-level values (JSONL, space-joined search hits) are **not** protected. Welding two of them yields unparseable output, not a silently shorter document. That is what the last review asked for, and it is why `test_force_kompress_routes_anthropic_tool_result_to_targeted_kompress` and the three `test_kompress_deadline` tests pass again. - `_try_ml_compressor` skips Kompress when protected JSON is present. - `_relevance_split_compress` declines only when a protected span straddles a `segment()` cut, so line-contained JSONL still splits. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check <4 changed files> && ruff format --check <4 changed files> All checks passed! / 4 files already formatted $ mypy headroom/transforms/content_router.py headroom/transforms/recursive_json.py Success: no issues found $ pytest tests/test_transforms tests/test_transforms_content_router.py tests/test_recursive_json.py \ tests/test_relevance*.py tests/test_*smart_crusher*.py tests/test_*kompress*.py 668 passed, 84 skipped, 1 failed ``` The one failure is `test_smart_crusher_issue_3650::test_issue_3650_router_emits_visible_retrievable_sentinel`. It fails identically on `origin/main` in my environment, because I cannot build `_core` locally and ran against the `_core` from the 0.39.1 wheel. For the same reason I skipped the Rust-parity tests. CI will be the real check for both. The 4 tests that fail on #3680's head (`fec63a13`, CI shard `test (4)`) pass here. ## Real Behavior Proof - Environment: real Kompress ONNX model (`chopratejas/kompress-v2-base`, int8-wo), `coding` profile env applied via `get_agent_savings_profile("coding").apply_proxy_env_defaults`, `HEADROOM_OFFLINE=1`; the `payload_synthetic.json` attached to #3673 (5 records, 3721 bytes, one line) - Exact command / steps: `ContentRouter().compress()` over three routes (bare document, `Tool result: {…}` on the same line, document embedded in prose), 3 runs each, on `origin/main` (`ffc6edb4`) and on this branch, plus a pure-prose control - Observed result: `main` loses 1 of 5 records on all three routes in 3/3 runs (3721 → 3126 bytes on the bare document); this branch keeps 5/5 byte-exact on all three, the prose around the document still compresses (9651 → 9129) and the pure-prose control compresses identically on both (2964 → 2702). Table below - Not tested: a full proxy run end to end on this branch. The original report comes from a proxy deployment, where the same record loss was observed on the wire | route | `main` | this branch | |---|---|---| | bare document | 4/5 records, 3721 → 3126 bytes | 5/5, byte-exact | | `Tool result: {…}` on the same line | 4/5 | 5/5, byte-exact | | document inside prose | 4/5 | 5/5, byte-exact; prose around it still compresses (9651 → 9129) | | control: pure prose | 2964 → 2702 | 2964 → 2702 (unchanged) | ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a (stable) - Stable/default behavior changed: yes, narrowly: record-bearing JSON is no longer handed to Kompress and is forwarded byte-exact instead of lossy-compressed; structural compressors are unaffected - Kill switch / disable path: none new; revert the PR - Unsafe override required: no - Qualification impact: less lossy savings on JSON tool outputs that previously went through Kompress (those savings were partly the data loss itself) - Rollback path: revert this PR; no state, schema or config migration involved ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Documentation: no user-facing doc change; the behavior now matches what the JSON docs already promise ("keys, brackets … remain preserved"). - Local test caveat: `test_smart_crusher_issue_3650::test_issue_3650_router_emits_visible_retrievable_sentinel` fails identically on `origin/main` in my environment (I ran against the 0.39.1 wheel's `_core`, not a local build), and Rust-parity tests were skipped for the same reason; CI is the real check for both. - Workflows are awaiting maintainer approval (first PR from this fork). --------- Signed-off-by: wangzifei <wangzifei@cit.group.hk> Co-authored-by: iuiuiu <92971770+iuiu-py@users.noreply.github.com> Co-authored-by: wangzifei <wangzifei@cit.group.hk> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Ayushraj06-bit <ayush10d172390@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
iuiuiu
wangzifei
JerrettDavis
Ayushraj06-bit
Claude Opus 5.5
parent
91237cae70
commit
8dbbd1d896
@@ -91,7 +91,7 @@ from .lossless_provider import (
|
||||
get_lossless_verifier,
|
||||
)
|
||||
from .mixed_content import ContentSection, mixed_content_indicators
|
||||
from .relevance_split import build_relevance_query, plan_relevance_split
|
||||
from .relevance_split import build_relevance_query, plan_relevance_split, segment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -895,6 +895,73 @@ def _content_is_valid_json(content: str) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
def _protected_json_spans(content: str) -> list[tuple[int, int]]:
|
||||
"""JSON spans of ``content`` that a prose model must not be handed (#3673).
|
||||
|
||||
Kompress has no notion of JSON grammar, so JSON bytes in its input are at
|
||||
risk of a destructive span around ``},{``. The scan is the deterministic
|
||||
balanced-span walk embedded-JSON routing already uses, so prefixed tool
|
||||
output and relevance fragments are seen too. Two shapes qualify:
|
||||
|
||||
* a span carrying an array of objects — deleting the ``},{`` between two
|
||||
records leaves VALID JSON, so the loss is invisible to every parser, log
|
||||
line and model downstream, which is what made #3673 invisible;
|
||||
* a block that is one JSON document end to end, which covers welding two
|
||||
keys of a lone object by the same argument.
|
||||
|
||||
A stream of SEPARATE top-level values (search hits as space-joined objects,
|
||||
JSONL) is deliberately not protected. Welding two of those yields an
|
||||
unparseable record rather than a silently shorter document, so the
|
||||
invisible-loss argument does not apply — and protecting them costs a large,
|
||||
legitimate savings class: it turns forced Kompress into a no-op on a
|
||||
160-object search payload (see
|
||||
``test_force_kompress_routes_anthropic_tool_result_to_targeted_kompress``).
|
||||
"""
|
||||
from .recursive_json import carries_record_array, scan_json_documents
|
||||
|
||||
spans, complete = scan_json_documents(content)
|
||||
if not complete:
|
||||
# The scan stopped at its budget (input built to defeat the linear walk),
|
||||
# so a record array past that point cannot be ruled out. Protect the
|
||||
# whole block: declining a prose compression is recoverable, a silently
|
||||
# deleted record is not.
|
||||
return [(0, len(content))]
|
||||
if not spans:
|
||||
return []
|
||||
if (
|
||||
len(spans) == 1
|
||||
and not content[: spans[0][0]].strip()
|
||||
and not content[spans[0][1] :].strip()
|
||||
):
|
||||
return spans
|
||||
return [(a, b) for a, b in spans if carries_record_array(content[a:b])]
|
||||
|
||||
|
||||
def _contains_protected_json(content: str) -> bool:
|
||||
"""True when ``content`` is, or contains, a JSON span worth protecting."""
|
||||
return bool(_protected_json_spans(content))
|
||||
|
||||
|
||||
def _json_straddles_segments(content: str) -> bool:
|
||||
"""True when a protected JSON span crosses a boundary ``segment()`` cuts at.
|
||||
|
||||
The relevance split windows by line with no JSON awareness, so a payload
|
||||
misclassified as LOG/SEARCH can be cut mid-structure and the pieces handed
|
||||
to Kompress as fragments no span check can recognise. Declining only for a
|
||||
straddling span keeps the split working on line-contained JSON (JSONL
|
||||
logs), which is the shape it exists for.
|
||||
"""
|
||||
spans = _protected_json_spans(content)
|
||||
if not spans:
|
||||
return False
|
||||
cuts: list[int] = []
|
||||
offset = 0
|
||||
for piece in segment(content)[:-1]:
|
||||
offset += len(piece)
|
||||
cuts.append(offset)
|
||||
return any(a < cut < b for a, b in spans for cut in cuts)
|
||||
|
||||
|
||||
def _mixed_indicators(content: str) -> dict[str, bool]:
|
||||
return mixed_content_indicators(content)
|
||||
|
||||
@@ -4290,6 +4357,14 @@ class ContentRouter(Transform):
|
||||
Returns:
|
||||
Tuple of (compressed, token_count).
|
||||
"""
|
||||
# Kompress is a prose model: dropping a span around ``},{`` can remove
|
||||
# an entire JSON record while leaving the remaining document valid
|
||||
# (#3673). This boundary sees the final Kompress input, which may be a
|
||||
# prefixed block or a fragment rather than a whole JSON document, so
|
||||
# guard every parseable JSON span before the prose model runs.
|
||||
if _contains_protected_json(content):
|
||||
return content, _estimate_tokens(content)
|
||||
|
||||
from .tag_protector import protect_tags, restore_tags
|
||||
|
||||
# Protect custom tags before any ML compression
|
||||
@@ -4632,6 +4707,11 @@ class ContentRouter(Transform):
|
||||
normal path when the scorer is unavailable, the query is empty, nothing
|
||||
is dropped, or the split doesn't beat plain compaction. Never raises.
|
||||
|
||||
JSON the windows would cut is excluded from splitting. Without that
|
||||
route guard, a payload misclassified as LOG/SEARCH is cut at arbitrary
|
||||
offsets before the JSON guard sees it (#3673); line-contained JSON
|
||||
(JSONL) still splits, which is the shape this exists for.
|
||||
|
||||
Embedding cost is bounded two ways: the model is pre-warmed off the
|
||||
request thread (BM25 until it's ready, see _get_relevance_scorer) and
|
||||
outputs segmenting into more than ``relevance_max_records`` records skip
|
||||
@@ -4640,6 +4720,8 @@ class ContentRouter(Transform):
|
||||
scorer = self._get_relevance_scorer()
|
||||
if scorer is None or not query.strip():
|
||||
return None
|
||||
if _json_straddles_segments(content):
|
||||
return None
|
||||
from .lossless_compaction import compact_lossless
|
||||
|
||||
try:
|
||||
|
||||
@@ -70,26 +70,124 @@ def _match_span(text: str, start: int) -> int | None:
|
||||
return None
|
||||
|
||||
|
||||
def _spans(text: str) -> list[tuple[int, int]]:
|
||||
"""Deterministic list of ``(start, end)`` for top-level balanced JSON spans.
|
||||
Nested spans are not returned separately — the dispatch handles depth."""
|
||||
#: Backstop for ``_scan_spans``: characters it may walk, per character of input,
|
||||
#: before it stops looking for further spans. The memoized walk below is linear
|
||||
#: on real payloads (source, logs, JSON, prose), so this only bounds inputs built
|
||||
#: to defeat the memo. Callers that must not miss a span check ``complete``.
|
||||
_SCAN_BUDGET_PER_CHAR = 4
|
||||
_SCAN_BUDGET_FLOOR = 4096
|
||||
|
||||
|
||||
def _scan_from(text: str, start: int, known: dict[int, int | None]) -> tuple[int | None, int]:
|
||||
"""``_match_span(text, start)``, recording every nested container's verdict.
|
||||
|
||||
A walk from ``start`` also decides the result of ``_match_span`` for every
|
||||
``[``/``{`` it pushes outside a string: from that position the walk's string
|
||||
state is the same (not in a string), its stack is the suffix above that
|
||||
bracket, so it pops, hits a mismatch or runs off the end exactly when this
|
||||
walk does. Those verdicts go into ``known`` so the caller never walks the same
|
||||
bytes again for them — which is what made unmatched ``{`` in source and logs
|
||||
quadratic. Returns ``(end or None, characters walked)``.
|
||||
"""
|
||||
stack: list[int] = []
|
||||
in_str = esc = False
|
||||
n = len(text)
|
||||
for j in range(start, n):
|
||||
ch = text[j]
|
||||
if in_str:
|
||||
if esc:
|
||||
esc = False
|
||||
elif ch == "\\":
|
||||
esc = True
|
||||
elif ch == '"':
|
||||
in_str = False
|
||||
continue
|
||||
if ch == '"':
|
||||
in_str = True
|
||||
elif ch in _OPEN:
|
||||
stack.append(j)
|
||||
elif ch in _CLOSE:
|
||||
if not stack or text[stack[-1]] != _PAIR[ch]:
|
||||
for p in stack:
|
||||
known[p] = None
|
||||
return None, j - start + 1
|
||||
known[stack.pop()] = j + 1
|
||||
if not stack:
|
||||
return j + 1, j - start + 1
|
||||
for p in stack:
|
||||
known[p] = None
|
||||
return None, n - start
|
||||
|
||||
|
||||
def _scan_spans(text: str) -> tuple[list[tuple[int, int]], bool]:
|
||||
"""``(spans, complete)``: the top-level balanced spans ``_spans`` returns, and
|
||||
whether the walk covered the whole text within its budget.
|
||||
|
||||
Same left-to-right semantics as calling ``_match_span`` at every opening
|
||||
bracket, but each bracket's verdict is computed at most once (see
|
||||
``_scan_from``), so a block full of unmatched ``{`` costs one pass instead of
|
||||
one pass per brace.
|
||||
"""
|
||||
out: list[tuple[int, int]] = []
|
||||
i, n = 0, len(text)
|
||||
known: dict[int, int | None] = {}
|
||||
n = len(text)
|
||||
budget = _SCAN_BUDGET_PER_CHAR * n + _SCAN_BUDGET_FLOOR
|
||||
spent = 0
|
||||
i = 0
|
||||
while i < n:
|
||||
if text[i] in _OPEN:
|
||||
end = _match_span(text, i)
|
||||
if i in known:
|
||||
end = known[i]
|
||||
elif spent >= budget:
|
||||
return out, False
|
||||
else:
|
||||
end, walked = _scan_from(text, i, known)
|
||||
spent += walked
|
||||
if end is not None:
|
||||
out.append((i, end))
|
||||
i = end
|
||||
continue
|
||||
i += 1
|
||||
return out
|
||||
return out, True
|
||||
|
||||
|
||||
def _has_routable_json(span: str) -> bool:
|
||||
def _spans(text: str) -> list[tuple[int, int]]:
|
||||
"""Deterministic list of ``(start, end)`` for top-level balanced JSON spans.
|
||||
Nested spans are not returned separately — the dispatch handles depth."""
|
||||
return _scan_spans(text)[0]
|
||||
|
||||
|
||||
def json_document_spans(text: str) -> list[tuple[int, int]]:
|
||||
"""``(start, end)`` of every top-level balanced span of ``text`` that parses
|
||||
as a JSON object or array, left to right. Scalars are not documents; a
|
||||
whole-document ``text`` yields one span covering it (modulo whitespace)."""
|
||||
return scan_json_documents(text)[0]
|
||||
|
||||
|
||||
def scan_json_documents(text: str) -> tuple[list[tuple[int, int]], bool]:
|
||||
"""``json_document_spans`` plus whether the scan covered all of ``text``.
|
||||
|
||||
``complete`` is only ever False past the scan budget, i.e. on input built to
|
||||
defeat the linear walk; a caller that must not miss a document treats that
|
||||
as "may contain one".
|
||||
"""
|
||||
spans, complete = _scan_spans(text)
|
||||
out: list[tuple[int, int]] = []
|
||||
for a, b in spans:
|
||||
try:
|
||||
parsed = json.loads(text[a:b])
|
||||
except (ValueError, TypeError, RecursionError):
|
||||
continue
|
||||
if isinstance(parsed, dict | list):
|
||||
out.append((a, b))
|
||||
return out, complete
|
||||
|
||||
|
||||
def carries_record_array(span: str) -> bool:
|
||||
"""True if ``span`` parses and contains an array of objects somewhere — the
|
||||
shape the JSON compressors actually act on. Cheap structural check, no size
|
||||
threshold."""
|
||||
shape the JSON compressors act on, and the one whose record delimiters a
|
||||
prose model can delete while leaving valid JSON behind (#3673). Cheap
|
||||
structural check, no size threshold."""
|
||||
try:
|
||||
v = json.loads(span)
|
||||
except (ValueError, TypeError):
|
||||
@@ -141,7 +239,7 @@ def route_embedded_json(
|
||||
chunk = content[a:b]
|
||||
if "<<ccr:" in chunk: # R1: already compressed — never re-route
|
||||
continue
|
||||
if not _has_routable_json(chunk):
|
||||
if not carries_record_array(chunk):
|
||||
continue
|
||||
out = dispatch(chunk)
|
||||
if out is None or out == chunk:
|
||||
|
||||
@@ -6,7 +6,11 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from headroom.transforms.recursive_json import route_embedded_json
|
||||
from headroom.transforms.recursive_json import (
|
||||
carries_record_array,
|
||||
json_document_spans,
|
||||
route_embedded_json,
|
||||
)
|
||||
|
||||
|
||||
def _upper_dispatch(span: str) -> str | None:
|
||||
@@ -65,3 +69,121 @@ def test_scalar_array_not_routed() -> None:
|
||||
# array of scalars is not a "routable" JSON shape (no dict rows)
|
||||
content = "nums: [1,2,3,4,5,6,7,8] done"
|
||||
assert route_embedded_json(content, _upper_dispatch) is None
|
||||
|
||||
|
||||
def test_json_document_spans_finds_containers_anywhere() -> None:
|
||||
doc = json.dumps({"domains": [{"name": "a"}, {"name": "b"}]})
|
||||
arr = json.dumps([1, 2, 3])
|
||||
text = "Tool result: " + doc + " and a list " + arr + " done"
|
||||
assert [text[a:b] for a, b in json_document_spans(text)] == [doc, arr]
|
||||
|
||||
|
||||
def test_json_document_spans_ignores_scalars_prose_and_unbalanced_json() -> None:
|
||||
assert json_document_spans('"just a quoted sentence"') == []
|
||||
assert json_document_spans("42") == []
|
||||
assert json_document_spans("prose with a [note] and {braces} but no JSON") == []
|
||||
assert json_document_spans('{"truncated": [1, 2, 3') == []
|
||||
assert json_document_spans("{{HEADROOM_TAG_0}}") == []
|
||||
assert json_document_spans("") == []
|
||||
|
||||
|
||||
def test_json_document_spans_whole_document() -> None:
|
||||
doc = json.dumps({"a": [1, 2]})
|
||||
assert json_document_spans(doc) == [(0, len(doc))]
|
||||
assert json_document_spans(" " + doc + "\n") == [(2, 2 + len(doc))]
|
||||
|
||||
|
||||
def test_carries_record_array_separates_the_two_shapes() -> None:
|
||||
assert carries_record_array(json.dumps({"domains": [{"name": "a"}, {"name": "b"}]}))
|
||||
assert carries_record_array(json.dumps([{"id": 1}, {"id": 2}]))
|
||||
# A lone object, or an array of scalars, has no record delimiter whose
|
||||
# deletion leaves a valid-but-shorter document.
|
||||
assert not carries_record_array(json.dumps({"file": "src/mod.py", "line": 1}))
|
||||
assert not carries_record_array(json.dumps([1, 2, 3]))
|
||||
assert not carries_record_array(json.dumps([{"id": 1}]))
|
||||
|
||||
|
||||
# --- the span walk is linear on brace-heavy text -----------------------------
|
||||
#
|
||||
# ``_spans`` used to call ``_match_span`` at every opening bracket, and a ``{``
|
||||
# that never closes walks to the end of the text: source code and logs full of
|
||||
# unmatched braces cost one full pass per brace. ``_scan_spans`` memoizes each
|
||||
# bracket's verdict from the walk that already crossed it. These tests pin that
|
||||
# the result did not change and that the work did.
|
||||
|
||||
|
||||
def _reference_spans(text: str) -> list[tuple[int, int]]:
|
||||
"""The previous ``_spans``: ``_match_span`` at every opening bracket."""
|
||||
from headroom.transforms.recursive_json import _match_span
|
||||
|
||||
out: list[tuple[int, int]] = []
|
||||
i, n = 0, len(text)
|
||||
while i < n:
|
||||
if text[i] in "[{":
|
||||
end = _match_span(text, i)
|
||||
if end is not None:
|
||||
out.append((i, end))
|
||||
i = end
|
||||
continue
|
||||
i += 1
|
||||
return out
|
||||
|
||||
|
||||
def test_span_walk_matches_the_previous_semantics_exactly() -> None:
|
||||
import random
|
||||
|
||||
from headroom.transforms.recursive_json import _scan_spans
|
||||
|
||||
rng = random.Random(3673)
|
||||
alphabet = '{}[]"\\ a,:\n'
|
||||
samples = [
|
||||
"".join(rng.choice(alphabet) for _ in range(rng.randint(0, 120))) for _ in range(4000)
|
||||
]
|
||||
samples += [
|
||||
'noise {"a": [1, {"b": "}"}]} tail [1,2] {oops',
|
||||
'He said "a {quote" then {"k": "v"} and "another {one}"',
|
||||
'{"esc": "a \\" } still string"} {x',
|
||||
"[{]}] {[}] {{{{",
|
||||
]
|
||||
for text in samples:
|
||||
spans, complete = _scan_spans(text)
|
||||
assert complete, text
|
||||
assert spans == _reference_spans(text), text
|
||||
|
||||
|
||||
def test_span_walk_is_linear_on_unmatched_braces(monkeypatch) -> None:
|
||||
import headroom.transforms.recursive_json as rj
|
||||
|
||||
line = ' if (flags & MASK) { log.debug("state={}", state); retry(ctx, {timeout: 30, max\n'
|
||||
text = line * 800 # ~70 KB, about three unmatched `{` per line
|
||||
walked = 0
|
||||
real = rj._scan_from
|
||||
|
||||
def counting(t, start, known): # noqa: ANN001, ANN202
|
||||
nonlocal walked
|
||||
end, n = real(t, start, known)
|
||||
walked += n
|
||||
return end, n
|
||||
|
||||
monkeypatch.setattr(rj, "_scan_from", counting)
|
||||
spans, complete = rj._scan_spans(text)
|
||||
assert complete
|
||||
# One pass, give or take the per-line restarts inside quoted strings. The
|
||||
# previous walk re-read the rest of the text for every unmatched brace:
|
||||
# hundreds of times the input size here.
|
||||
assert walked <= 2 * len(text), (walked, len(text))
|
||||
# Same answer as before on the same shape (a slice keeps the quadratic
|
||||
# reference fast enough to run).
|
||||
small = line * 40
|
||||
assert rj._scan_spans(small)[0] == _reference_spans(small)
|
||||
|
||||
|
||||
def test_scan_stops_at_its_budget_and_says_so(monkeypatch) -> None:
|
||||
import headroom.transforms.recursive_json as rj
|
||||
|
||||
monkeypatch.setattr(rj, "_SCAN_BUDGET_PER_CHAR", 0)
|
||||
monkeypatch.setattr(rj, "_SCAN_BUDGET_FLOOR", 0)
|
||||
text = 'prose {"a": 1} then [{"id": 1}, {"id": 2}]'
|
||||
spans, complete = rj.scan_json_documents(text)
|
||||
assert complete is False
|
||||
assert spans == []
|
||||
|
||||
@@ -705,6 +705,378 @@ def test_smart_crusher_log_fallback_skipped_for_invalid_json(
|
||||
assert CompressionStrategy.KOMPRESS.value in strategy_chain
|
||||
|
||||
|
||||
def test_smart_crusher_fallback_skips_kompress_for_valid_json_object(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Kompress must not delete records from parseable single-line JSON (#3673).
|
||||
|
||||
SmartCrusher passes a compact JSON object through when it has no applicable
|
||||
array transform. The no-savings fallback used to hand the whole document to
|
||||
Kompress; when Kompress dropped a span around ``},{``, the result remained
|
||||
valid JSON but silently lost a record.
|
||||
"""
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
payload = json.dumps(
|
||||
{
|
||||
"domains": [
|
||||
{"name": "first", "description": "First collection description"},
|
||||
{"name": "second", "description": "Second collection description"},
|
||||
{"name": "third", "description": "Third collection description"},
|
||||
]
|
||||
},
|
||||
separators=(",", ":"),
|
||||
)
|
||||
kompress_calls: list[str] = []
|
||||
|
||||
class NoopSmartCrusher:
|
||||
def crush(self, content: str, query: str = "", bias: float = 1.0) -> SimpleNamespace:
|
||||
return SimpleNamespace(compressed=content)
|
||||
|
||||
class RecordEatingKompress:
|
||||
def is_ready(self) -> bool:
|
||||
return True
|
||||
|
||||
def compress(self, content: str, **_kwargs: object) -> SimpleNamespace:
|
||||
kompress_calls.append("kompress")
|
||||
return SimpleNamespace(
|
||||
compressed='{"domains":["record deleted"]}',
|
||||
compressed_tokens=1,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(router, "_get_smart_crusher", lambda: NoopSmartCrusher())
|
||||
monkeypatch.setattr(router, "_get_log_compressor", lambda: None)
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: RecordEatingKompress())
|
||||
|
||||
result = router.compress(payload)
|
||||
|
||||
assert result.compressed == payload
|
||||
assert kompress_calls == []
|
||||
|
||||
|
||||
def test_force_kompress_skips_valid_json_object(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The forced-Kompress fast path keeps the same parseable-JSON guard."""
|
||||
router = ContentRouter(ContentRouterConfig(force_kompress_all=True))
|
||||
router._runtime_force_kompress = True
|
||||
payload = json.dumps({"items": [{"id": 1}, {"id": 2}]}, separators=(",", ":"))
|
||||
calls: list[str] = []
|
||||
|
||||
class RecordEatingKompress:
|
||||
def is_ready(self) -> bool:
|
||||
return True
|
||||
|
||||
def compress(self, content: str, **_kwargs: object) -> SimpleNamespace:
|
||||
calls.append("kompress")
|
||||
return SimpleNamespace(compressed='{"items":[{"id":1}]}', compressed_tokens=1)
|
||||
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: RecordEatingKompress())
|
||||
|
||||
result = router.compress(payload)
|
||||
|
||||
assert calls == []
|
||||
assert result.compressed == payload
|
||||
|
||||
|
||||
def test_prefixed_same_line_json_skips_kompress(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Embedded JSON must be guarded even when the block itself is not JSON.
|
||||
|
||||
The mixed splitter gives a JSON span its own section only when a line starts
|
||||
with ``{``/``[``. With ``Tool result:`` on the same line, embedded routing
|
||||
finds the span but SmartCrusher may produce no savings; the full block then
|
||||
reaches the Kompress fallback. That final input still contains JSON and must
|
||||
not reach the prose model (#3673).
|
||||
"""
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
payload = 'Tool result: {"items":[{"id":1},{"id":2},{"id":3}]}'
|
||||
kompress_calls: list[str] = []
|
||||
|
||||
class NoopSmartCrusher:
|
||||
def crush(self, content: str, query: str = "", bias: float = 1.0) -> SimpleNamespace:
|
||||
return SimpleNamespace(compressed=content)
|
||||
|
||||
class RecordEatingKompress:
|
||||
def is_ready(self) -> bool:
|
||||
return True
|
||||
|
||||
def compress(self, content: str, **_kwargs: object) -> SimpleNamespace:
|
||||
kompress_calls.append("kompress")
|
||||
return SimpleNamespace(
|
||||
compressed='Tool result: {"items":[{"id":1}]}',
|
||||
compressed_tokens=1,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(router, "_get_smart_crusher", lambda: NoopSmartCrusher())
|
||||
monkeypatch.setattr(router, "_get_log_compressor", lambda: None)
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: RecordEatingKompress())
|
||||
|
||||
result = router.compress(payload)
|
||||
|
||||
assert kompress_calls == []
|
||||
assert result.compressed == payload
|
||||
|
||||
|
||||
def test_record_stream_of_separate_objects_still_compresses(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Separate top-level objects (search hits, JSONL) are NOT protected.
|
||||
|
||||
Welding two of them yields an unparseable record rather than a silently
|
||||
shorter document, so the #3673 invisible-loss argument does not apply.
|
||||
Protecting them turns forced Kompress into a no-op on real search payloads
|
||||
— ``test_force_kompress_routes_anthropic_tool_result_to_targeted_kompress``
|
||||
in tests/test_transforms/test_content_router.py depends on this line.
|
||||
"""
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
stream = " ".join(
|
||||
json.dumps({"file": f"src/mod_{i}.py", "line": i, "text": "repeated search payload"})
|
||||
for i in range(160)
|
||||
)
|
||||
kompress_inputs: list[str] = []
|
||||
|
||||
class HalvingKompress:
|
||||
def is_ready(self) -> bool:
|
||||
return True
|
||||
|
||||
def compress(self, content: str, **_kwargs: object) -> SimpleNamespace:
|
||||
kompress_inputs.append(content)
|
||||
out = " ".join(content.split()[::2])
|
||||
return SimpleNamespace(compressed=out, compressed_tokens=len(out.split()))
|
||||
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: HalvingKompress())
|
||||
|
||||
out, _tokens = router._try_ml_compressor(stream, context="")
|
||||
|
||||
assert kompress_inputs == [stream]
|
||||
assert out != stream
|
||||
|
||||
|
||||
def test_record_array_is_protected_even_beside_a_lone_object(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The protected shape is the record array, at any offset in the block."""
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
document = json.dumps({"domains": [{"name": "a"}, {"name": "b"}, {"name": "c"}]})
|
||||
block = (
|
||||
"The listing call returned two payloads on this line: "
|
||||
+ json.dumps({"file": "src/mod.py", "line": 1})
|
||||
+ " and then "
|
||||
+ document
|
||||
+ " done."
|
||||
)
|
||||
kompress_inputs: list[str] = []
|
||||
|
||||
class RecordEatingKompress:
|
||||
def is_ready(self) -> bool:
|
||||
return True
|
||||
|
||||
def compress(self, content: str, **_kwargs: object) -> SimpleNamespace:
|
||||
kompress_inputs.append(content)
|
||||
return SimpleNamespace(compressed="eaten", compressed_tokens=1)
|
||||
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: RecordEatingKompress())
|
||||
|
||||
out, _tokens = router._try_ml_compressor(block, context="")
|
||||
|
||||
assert out == block
|
||||
assert kompress_inputs == []
|
||||
|
||||
|
||||
def test_protection_scan_stays_linear_on_brace_heavy_text(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Source and logs are full of ``{`` that never close. The protection guard
|
||||
scans at every Kompress boundary; it must not re-read the rest of the block
|
||||
for each brace, or large tool output spends its compression time in scans."""
|
||||
import headroom.transforms.recursive_json as rj
|
||||
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
line = " if (flags & MASK) { log.debug(state); retry(ctx, {timeout: 30, max attempts\n"
|
||||
block = line * 800 # ~65 KB, two unmatched `{` per line, no JSON at all
|
||||
walked = 0
|
||||
real = rj._scan_from
|
||||
|
||||
def counting(text, start, known): # noqa: ANN001, ANN202
|
||||
nonlocal walked
|
||||
end, n = real(text, start, known)
|
||||
walked += n
|
||||
return end, n
|
||||
|
||||
monkeypatch.setattr(rj, "_scan_from", counting)
|
||||
kompress_inputs: list[str] = []
|
||||
|
||||
class HalvingKompress:
|
||||
def is_ready(self) -> bool:
|
||||
return True
|
||||
|
||||
def compress(self, content: str, **_kwargs: object) -> SimpleNamespace:
|
||||
kompress_inputs.append(content)
|
||||
out = " ".join(content.split()[::2])
|
||||
return SimpleNamespace(compressed=out, compressed_tokens=len(out.split()))
|
||||
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: HalvingKompress())
|
||||
|
||||
out, _tokens = router._try_ml_compressor(block, context="")
|
||||
|
||||
# No JSON to protect, so prose compression still runs...
|
||||
assert kompress_inputs
|
||||
assert out != block
|
||||
# ...and every scan of the call together stays a small multiple of one pass
|
||||
# (the quadratic walk read hundreds of times the block here).
|
||||
assert walked <= 8 * len(block), (walked, len(block))
|
||||
|
||||
|
||||
def test_unfinished_protection_scan_protects_the_whole_block(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""If the scan ever stops at its budget, a record array past that point
|
||||
cannot be ruled out, so the block is kept away from the prose model."""
|
||||
import headroom.transforms.recursive_json as rj
|
||||
|
||||
monkeypatch.setattr(rj, "_SCAN_BUDGET_PER_CHAR", 0)
|
||||
monkeypatch.setattr(rj, "_SCAN_BUDGET_FLOOR", 0)
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
block = "Results below. " + json.dumps([{"id": i, "v": "x" * 20} for i in range(5)])
|
||||
kompress_inputs: list[str] = []
|
||||
|
||||
class RecordEatingKompress:
|
||||
def is_ready(self) -> bool:
|
||||
return True
|
||||
|
||||
def compress(self, content: str, **_kwargs: object) -> SimpleNamespace:
|
||||
kompress_inputs.append(content)
|
||||
return SimpleNamespace(compressed="eaten", compressed_tokens=1)
|
||||
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: RecordEatingKompress())
|
||||
|
||||
out, _tokens = router._try_ml_compressor(block, context="")
|
||||
|
||||
assert out == block
|
||||
assert kompress_inputs == []
|
||||
|
||||
|
||||
def test_relevance_split_still_runs_for_line_contained_json(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""JSONL keeps each object inside one line, so no window cuts a document and
|
||||
the split — which exists for log payloads — proceeds as before."""
|
||||
router = ContentRouter(ContentRouterConfig(relevance_split=True))
|
||||
payload = "\n".join(
|
||||
json.dumps({"level": "info", "msg": f"event {i} with a prose-like description"})
|
||||
for i in range(40)
|
||||
)
|
||||
planned: list[str] = []
|
||||
|
||||
router._relevance_scorer = object()
|
||||
router._relevance_scorer_tried = True
|
||||
monkeypatch.setattr(
|
||||
content_router_module,
|
||||
"plan_relevance_split",
|
||||
lambda content, query, scorer, **_kw: planned.append(content) or [(True, content)],
|
||||
)
|
||||
|
||||
router._relevance_split_compress(payload, "log", "event 3")
|
||||
|
||||
assert planned == [payload]
|
||||
|
||||
|
||||
def test_relevance_split_runs_when_a_record_array_sits_inside_one_window(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Only a STRADDLING span blocks the split.
|
||||
|
||||
A log whose paragraphs happen to embed a small record array is windowed at
|
||||
the blank lines, so the array stays whole inside one window. That window is
|
||||
safe to score and drop: if it is dropped, the Kompress boundary sees the
|
||||
intact array and passes it through. Declining here instead would disable
|
||||
the relevance split for most log payloads.
|
||||
"""
|
||||
router = ContentRouter(ContentRouterConfig(relevance_split=True))
|
||||
array = json.dumps([{"id": 1, "state": "ok"}, {"id": 2, "state": "ok"}])
|
||||
paragraphs = [
|
||||
"worker started and connected to the queue, waiting for the first batch",
|
||||
"batch 1 finished; per-item results were reported as " + array,
|
||||
"worker idle for 30 seconds, then shut down cleanly on SIGTERM",
|
||||
]
|
||||
payload = (chr(10) + chr(10)).join(paragraphs)
|
||||
planned: list[str] = []
|
||||
kompress_inputs: list[str] = []
|
||||
|
||||
class HalvingKompress:
|
||||
def is_ready(self) -> bool:
|
||||
return True
|
||||
|
||||
def compress(self, content: str, **_kwargs: object) -> SimpleNamespace:
|
||||
kompress_inputs.append(content)
|
||||
out = " ".join(content.split()[::2])
|
||||
return SimpleNamespace(compressed=out, compressed_tokens=len(out.split()))
|
||||
|
||||
router._relevance_scorer = object()
|
||||
router._relevance_scorer_tried = True
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: HalvingKompress())
|
||||
monkeypatch.setattr(
|
||||
content_router_module,
|
||||
"plan_relevance_split",
|
||||
lambda content, query, scorer, **_kw: (
|
||||
planned.append(content)
|
||||
or [(False, piece) for piece in content_router_module.segment(content)]
|
||||
),
|
||||
)
|
||||
|
||||
router._relevance_split_compress(payload, "log", "worker shutdown")
|
||||
|
||||
# The split ran...
|
||||
assert planned == [payload]
|
||||
# ...and the window carrying the array reached the Kompress boundary, which
|
||||
# passed it through untouched rather than letting the prose model see it.
|
||||
carrying = [text for text in kompress_inputs if array in text]
|
||||
assert carrying == [], f"a record array reached the prose model: {carrying}"
|
||||
|
||||
|
||||
def test_relevance_split_does_not_fragment_json_payloads(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Relevance windowing must not cut JSON before the Kompress guard.
|
||||
|
||||
A detector can misclassify JSON as LOG/SEARCH. Windowing such a payload at
|
||||
blank-line-free boundaries produces fragments that no longer parse, so a
|
||||
whole-block JSON check cannot help. Skip the split and let the structural
|
||||
JSON boundary preserve the document (#3673).
|
||||
"""
|
||||
router = ContentRouter(ContentRouterConfig(relevance_split=True))
|
||||
payload = json.dumps(
|
||||
{
|
||||
"items": [
|
||||
{"id": index, "value": f"record {index} with prose-like description"}
|
||||
for index in range(8)
|
||||
]
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
kompress_inputs: list[str] = []
|
||||
|
||||
class DeterministicScorer:
|
||||
def score_batch(self, items: list[str], _context: str) -> list[SimpleNamespace]:
|
||||
return [SimpleNamespace(score=0.0) for _ in items]
|
||||
|
||||
def record_kompress(
|
||||
content: str,
|
||||
_context: str,
|
||||
question: str | None = None,
|
||||
) -> tuple[str, int]:
|
||||
kompress_inputs.append(content)
|
||||
return content, _estimate_tokens(content)
|
||||
|
||||
router._relevance_scorer = DeterministicScorer()
|
||||
router._relevance_scorer_tried = True
|
||||
monkeypatch.setattr(router, "_try_ml_compressor", record_kompress)
|
||||
|
||||
assert router._relevance_split_compress(payload, "log", "find record 3") is None
|
||||
assert kompress_inputs == []
|
||||
|
||||
|
||||
def test_smart_crusher_log_fallback_runs_for_valid_json(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
|
||||
Reference in New Issue
Block a user