mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat(router): elide dense machine-generated lines no compressor can shrink (#3685)
## Description Tool outputs that dump a fetched web page, a bundled asset or an encoded blob (minified JS/CSS, base64, RSC payloads) arrive as a few very long lines with almost no whitespace. No structural compressor can do anything with them: `code_aware` is disabled, `log`/`text` return them unchanged, and the HTML extractor returns either the whole thing or `""` (which the unit layer then rejects as an empty block and keeps all of it verbatim). The block rides through the router at ratio 1.0. On a Codex web-research session these were the four largest tool outputs (75k tokens) and the whole session compressed at 1.3% of new input. This adds a last-resort pass in `_apply_strategy_to_content`: after the strategy chain (and on the passthrough path), every line that is >= 300 chars with < 6% spaces keeps its head and tail and the middle becomes a marker. Prose, code, logs, indented JSON, CSV and markdown all have far more spaces and are never touched; JSON-shaped lines are left to SmartCrusher. The pre-elision block goes into the CCR store and a `Retrieve original: hash=` marker is appended, so the messages path's marker-less lossy gate (#1307) keeps the result and the agent can recover exact bytes. Runs only on strategies whose result is still plain text (`html`, `log`, `text`, `code_aware`, `search`, `passthrough`), never after SmartCrusher/tabular/config/diff/Kompress, and never in lossless mode. New config flag `enable_dense_line_elision`, default on. Also: an HTML extraction that comes back `""` on non-empty input is now treated as "nothing extracted" (fall through) rather than "compressed to zero tokens". ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Performance improvement ## Changes Made - `headroom/transforms/dense_line_elider.py`: `is_dense_line` / `elide_dense_lines`, byte-identical no-op when nothing is dense. - `headroom/transforms/content_router.py`: `enable_dense_line_elision` config, `_DENSE_ELIDE_AFTER` strategy set, `_elide_dense` (CCR store + marker), hook after the strategy chain and on the passthrough path, empty-HTML-extraction fallthrough. - `tests/test_transforms/test_dense_line_elider.py`: 8 tests (prose/code/JSON untouched, elision shape, router hook on/off, marker retrievable from the CCR store, survives the messages-path lossy gate, lossless mode never elides, search output keeps line structure). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --frozen --extra dev pytest tests/test_transforms/test_dense_line_elider.py -q -> 8 passed uv run --frozen --extra dev pytest tests/test_transforms/ tests/test_no_ccr_lossy.py tests/test_transforms_content_router.py tests/test_openai_responses_compression_units.py tests/test_code_aware_regressions.py tests/test_router_registry_dispatch.py tests/test_router_external_dispatch.py tests/test_compression_observability.py tests/test_openai_responses_context_compaction.py tests/test_agent_savings.py tests/test_openai_responses_t3_replay_regression.py -q -> 588 passed, 72 skipped uv run --frozen --extra dev pytest tests/ --ignore=tests/e2e -q -> 9 failed, 12619 passed, 597 skipped (see note below) uvx ruff check headroom/transforms tests/test_transforms/test_dense_line_elider.py -> All checks passed! uv run --frozen --extra dev mypy headroom/transforms/content_router.py headroom/transforms/dense_line_elider.py -> Success: no issues found in 2 source files ``` The 9 full-suite failures are unrelated to this change: `tests/test_provider_grok.py` (2) and `tests/test_cli/test_wrap_grok.py` fail because this shell exports `GROK_CLI_CHAT_PROXY_BASE_URL` (all pass with it unset); the other 7 (`test_proxy_health.py` readyz/kompress x4, `test_read_maturation_handler_nobust.py` x2, `test_providers/test_deepseek.py` x1) are order-dependent, pass when run in isolation, and fail identically with `upstream/main`'s `content_router.py` swapped in. ## Real Behavior Proof - Environment: this branch, `HeadroomProxy(ProxyConfig(stateless=True, mode="token"))`, `HEADROOM_DISABLE_KOMPRESS=1`, two real `~/.codex/sessions/*.jsonl` rollouts replayed through `_compress_openai_responses_payload` one request per tool-output item (`input=items[:i+1]`), novel `tokens_saved` (this pass minus previous pass) over tiktoken of the items added since the previous request. Same script with `elide_dense_lines` monkeypatched to a no-op for the "off" arm. - Exact command / steps: `HEADROOM_ELIDE_TEST=off|on uv run --frozen --extra dev python replay_onoff.py <rollout.jsonl> <n_outputs> <model>` - Observed result: see the block below (web-research session 1.3% -> 15.3% of new input, coding session unchanged at 34.6%). ```text 2026-09-09 gpt-5.6-sol elision off: novel=4377 new_input=344023 rate=1.3% 2026-09-09 gpt-5.6-sol elision on: novel=52737 new_input=344023 rate=15.3% 2026-09-07 gpt-6-astra elision off: novel=51520 new_input=148924 rate=34.6% 2026-09-07 gpt-6-astra elision on: novel=51520 new_input=148924 rate=34.6% ``` The web-research session (curl'd pages, truncated minified bundles) goes 1.3% -> 15.3%; the coding session (code-mode JSON chunk outputs, no dense lines) is byte-for-byte unchanged. Across 14 local Codex sessions, dense lines are 13.5% of all tool-output characters (0% to 54% per session), so the fleet effect will be uneven and largest for research-heavy users. What remains un-elided on the research session is inside `embedded_json` spans (SSE `data: {...}` lines) and is excluded on purpose: eliding inside those would corrupt the JSON. - Not tested: live proxy traffic with a real Codex or Claude Code client (replay only); Kompress enabled alongside elision (replays ran with `HEADROOM_DISABLE_KOMPRESS=1`, matching the desktop's production config); a CCR `/v1/retrieve` round trip over HTTP (the store-and-retrieve check is in-process in the tests). Desktop-side: measured 2026-09-20 on the `user_daily_savings` rollup, Codex-dominant users sit at a median 6.5% strict input rate (n=341 user-days) vs 25.7% for Claude Code (n=478) at identical cache coverage; this is the largest single lever found for that gap. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes. `ContentRouterConfig.enable_dense_line_elision` defaults to on, so blocks with dense machine-generated lines that previously passed through at ratio 1.0 are now elided with a CCR retrieval marker. Blocks with no such lines are byte-identical to before. - Kill switch / disable path: `ContentRouterConfig(enable_dense_line_elision=False)`; `lossless=True` also disables it. - Unsafe override required: No. - Qualification impact: None; the elision is a deterministic post-pass and does not touch routing decisions, strategy selection or the frozen-prefix logic. - Rollback path: Revert this PR, or set `enable_dense_line_elision=False`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Documentation: no docs page lists `ContentRouterConfig` flags today, so there is nothing to update; happy to add one if you want the flag documented. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
a7858fd2ae
commit
9263b42099
@@ -83,6 +83,7 @@ from .content_detector import (
|
||||
_try_detect_structured_config,
|
||||
)
|
||||
from .content_detector import detect_content_type as _regex_detect_content_type
|
||||
from .dense_line_elider import elide_dense_lines
|
||||
from .error_detection import content_has_strong_error_indicators
|
||||
from .lossless_provider import (
|
||||
get_lossless_generation,
|
||||
@@ -1602,6 +1603,10 @@ class ContentRouterConfig:
|
||||
enable_tabular_compressor: bool = True # CSV/TSV/markdown tables via SmartCrusher
|
||||
enable_config_compressor: bool = True # YAML/TOML/INI structural compression
|
||||
enable_html_extractor: bool = True # HTML content extraction
|
||||
# Last-resort fallback for blocks no compressor could shrink: elide the
|
||||
# middle of long, whitespace-free lines (minified JS/CSS, base64, RSC
|
||||
# payloads). See ``dense_line_elider``.
|
||||
enable_dense_line_elision: bool = True
|
||||
enable_image_optimizer: bool = True # Image token optimization
|
||||
|
||||
# Routing preferences
|
||||
@@ -1933,6 +1938,21 @@ class ContentRouter(Transform):
|
||||
# the memo O(1) and unlockable at the cost of an occasional cold start.
|
||||
_LOSSLESS_FIRST_MEMO_MAX = 256
|
||||
|
||||
# Strategies whose result is still plain text, so dense-line elision can
|
||||
# run on top of them (see ``_elide_dense``).
|
||||
_DENSE_ELIDE_AFTER = frozenset(
|
||||
{
|
||||
CompressionStrategy.HTML,
|
||||
CompressionStrategy.LOG,
|
||||
CompressionStrategy.TEXT,
|
||||
CompressionStrategy.CODE_AWARE,
|
||||
# grep-style output is line-oriented (path, then ``N:match``), so a
|
||||
# dense match line can be elided without breaking the format.
|
||||
CompressionStrategy.SEARCH,
|
||||
CompressionStrategy.PASSTHROUGH,
|
||||
}
|
||||
)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config: ContentRouterConfig | None = None,
|
||||
@@ -3766,6 +3786,13 @@ class ContentRouter(Transform):
|
||||
compressed = (
|
||||
output.content if output is not None and output.compressed else None
|
||||
)
|
||||
# A page that is all <script>/<style> extracts to "".
|
||||
# That is "nothing extracted" (fall through to the
|
||||
# passthrough / dense-line path), not a compression
|
||||
# to zero tokens: the unit layer rejects an empty
|
||||
# block and keeps the original verbatim.
|
||||
if compressed is not None and not compressed.strip():
|
||||
compressed = None
|
||||
# Estimate tokens from extracted text (simple word count)
|
||||
compressed_tokens = _estimate_tokens(compressed) if compressed else 0
|
||||
decision_reason = "html_extractor"
|
||||
@@ -3944,6 +3971,21 @@ class ContentRouter(Transform):
|
||||
# Re-narrow for mypy: all reassignments above produce str, but
|
||||
# mypy 1.14.x widens after nested try/except/else reassignments.
|
||||
assert compressed is not None
|
||||
# Dense-line elision: whatever survived the chain, long
|
||||
# whitespace-free lines (minified bundles, base64, RSC payloads)
|
||||
# are still in it, because no structural compressor understands
|
||||
# them. Runs on the RESULT so it composes with a partial win
|
||||
# (code_aware at 0.95 on minified JS) as well as a no-op.
|
||||
# Only after strategies that hand back the text AS TEXT. SmartCrusher,
|
||||
# tabular, config and diff emit their own structured (and
|
||||
# CCR-marked) forms; Kompress is lossy with its own marker. Eliding
|
||||
# inside those would corrupt output another compressor owns.
|
||||
if self.config.enable_dense_line_elision and actual_strategy in self._DENSE_ELIDE_AFTER:
|
||||
dense = self._elide_dense(compressed, context)
|
||||
if dense is not None and dense[1] < compressed_tokens:
|
||||
compressed, compressed_tokens = dense
|
||||
strategy_chain.append("dense_elide")
|
||||
decision_reason = f"{decision_reason}_dense_elide"
|
||||
if logger.isEnabledFor(logging.DEBUG):
|
||||
_log_router_debug(
|
||||
"content_router_strategy_result",
|
||||
@@ -3978,6 +4020,21 @@ class ContentRouter(Transform):
|
||||
return compressed, compressed_tokens, strategy_chain
|
||||
|
||||
# Fallback: return unchanged
|
||||
if self.config.enable_dense_line_elision:
|
||||
dense = self._elide_dense(content, context)
|
||||
if dense is not None and dense[1] < original_tokens:
|
||||
elided, elided_tokens = dense
|
||||
strategy_chain.append("dense_elide")
|
||||
self._record_to_toin(
|
||||
strategy=strategy,
|
||||
content=content,
|
||||
compressed=elided,
|
||||
original_tokens=original_tokens,
|
||||
compressed_tokens=elided_tokens,
|
||||
language=language,
|
||||
context=context,
|
||||
)
|
||||
return elided, elided_tokens, strategy_chain
|
||||
strategy_chain.append(CompressionStrategy.PASSTHROUGH.value)
|
||||
if logger.isEnabledFor(logging.DEBUG):
|
||||
_log_router_debug(
|
||||
@@ -4001,6 +4058,40 @@ class ContentRouter(Transform):
|
||||
)
|
||||
return content, original_tokens, strategy_chain
|
||||
|
||||
def _elide_dense(self, text: str, context: str) -> tuple[str, int] | None:
|
||||
"""Dense-line elision with a CCR retrieval marker; None when no line is dense.
|
||||
|
||||
The elided middle is lossy, so the pre-elision block is stored in the
|
||||
CCR store and a ``Retrieve original: hash=`` marker is appended, the
|
||||
same contract Kompress honours: the messages path discards a lossy
|
||||
result that carries no marker (#1307), and the agent can get the
|
||||
exact bytes back if it turns out to need them.
|
||||
"""
|
||||
# Lossy by nature: never in lossless mode, where nothing may be dropped.
|
||||
if self.config.lossless:
|
||||
return None
|
||||
elided, n_dense = elide_dense_lines(text)
|
||||
if not n_dense:
|
||||
return None
|
||||
if self.config.ccr_inject_marker:
|
||||
try:
|
||||
from ..cache.compression_store import get_compression_store
|
||||
|
||||
key = get_compression_store().store(
|
||||
text,
|
||||
elided,
|
||||
original_tokens=_estimate_tokens(text),
|
||||
compressed_tokens=_estimate_tokens(elided),
|
||||
query_context=context or None,
|
||||
compression_strategy="dense_elide",
|
||||
)
|
||||
noun = "line" if n_dense == 1 else "lines"
|
||||
elided += f"\n[{n_dense} dense machine-generated {noun} elided. Retrieve original: hash={key}]"
|
||||
except Exception as exc: # noqa: BLE001 - store optional; keep the block verbatim
|
||||
logger.debug("dense-line elision: CCR store unavailable (%s); skipping", exc)
|
||||
return None
|
||||
return elided, _estimate_tokens(elided)
|
||||
|
||||
def _try_ml_compressor(
|
||||
self,
|
||||
content: str,
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
"""Elide dense machine-generated lines (minified JS/CSS, base64, RSC payloads).
|
||||
|
||||
Tool outputs that dump a fetched web page, a bundled asset or an encoded blob
|
||||
arrive as a few very long lines with almost no whitespace. None of the
|
||||
structural compressors can do anything with them: they are not code an AST
|
||||
walker can prune, not a log, not a table, and the HTML extractor returns the
|
||||
whole thing (or nothing) once the page is mostly ``<script>``. The result is a
|
||||
multi-thousand-token block that rides through the router at ratio 1.0.
|
||||
|
||||
This module keeps a head and a tail of each such line and replaces the middle
|
||||
with a one-line marker. It is deliberately dumb: a line is "dense" when it is
|
||||
long AND has a tiny fraction of spaces. Prose, code, logs, JSON with any
|
||||
indentation, CSV and markdown all have far more than 6% spaces, so they are
|
||||
never touched, a JSON-shaped line is left to SmartCrusher, and a lone dense
|
||||
value (JWT, signed URL, PATH) is below the per-block minimum. Measured on a replayed Codex web-research session against the
|
||||
installed 0.37.0 wheel: novel savings went from 7.2% to 25.1% of new input, on
|
||||
top of (not instead of) what the router already removed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
MIN_LINE_CHARS = 300
|
||||
MAX_SPACE_RATIO = 0.06
|
||||
# A single dense line (a JWT, a signed URL, a PATH, an RSA modulus) is a value
|
||||
# the agent asked for, not a dump: a block is only elided when its dense lines
|
||||
# add up to at least this many chars. Real bundle dumps are tens of KB.
|
||||
MIN_DENSE_TOTAL_CHARS = 2000
|
||||
HEAD_CHARS = 160
|
||||
TAIL_CHARS = 80
|
||||
|
||||
|
||||
def is_dense_line(line: str) -> bool:
|
||||
"""True when ``line`` is long, nearly whitespace-free, and not JSON-shaped.
|
||||
|
||||
A compact JSON value (or a run of them) on one line is also nearly
|
||||
whitespace-free, but that is SmartCrusher's job and it does it losslessly;
|
||||
the elider must never pre-empt it.
|
||||
"""
|
||||
n = len(line)
|
||||
# Tabs: TSV / ``psql -A`` rows pass the space ratio but are data the agent
|
||||
# asked for; minified assets and encoded blobs never carry tabs.
|
||||
if n < MIN_LINE_CHARS or "\t" in line or (line.count(" ") / n) >= MAX_SPACE_RATIO:
|
||||
return False
|
||||
stripped = line.strip()
|
||||
return not (stripped[:1] in "{[" and stripped[-1:] in "}]")
|
||||
|
||||
|
||||
def elide_dense_lines(text: str) -> tuple[str, int]:
|
||||
"""Return ``(text_with_dense_lines_elided, lines_elided)``.
|
||||
|
||||
Byte-identical to the input (and ``0``) when no line is dense, so callers
|
||||
can use ``elided != text`` as the "did anything" signal. Line endings are
|
||||
preserved: the split is on ``"\\n"`` only, so ``"\\r"`` stays on its line.
|
||||
"""
|
||||
if len(text) < MIN_LINE_CHARS:
|
||||
return text, 0
|
||||
lines = text.split("\n")
|
||||
if sum(len(line) for line in lines if is_dense_line(line)) < MIN_DENSE_TOTAL_CHARS:
|
||||
return text, 0
|
||||
out: list[str] = []
|
||||
n_elided = 0
|
||||
for line in lines:
|
||||
if is_dense_line(line):
|
||||
omitted = len(line) - HEAD_CHARS - TAIL_CHARS
|
||||
out.append(
|
||||
f"{line[:HEAD_CHARS]} ...[{omitted} chars of dense "
|
||||
f"machine-generated content elided]... {line[-TAIL_CHARS:]}"
|
||||
)
|
||||
n_elided += 1
|
||||
else:
|
||||
out.append(line)
|
||||
if n_elided == 0:
|
||||
return text, 0
|
||||
return "\n".join(out), n_elided
|
||||
@@ -0,0 +1,139 @@
|
||||
"""Dense-line elision: minified/base64/RSC lines that no compressor can shrink."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
|
||||
from headroom.transforms.dense_line_elider import elide_dense_lines
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def tokenizer():
|
||||
from headroom.providers import OpenAIProvider
|
||||
from headroom.tokenizer import Tokenizer
|
||||
|
||||
return Tokenizer(OpenAIProvider().get_token_counter("gpt-4o"), "gpt-4o")
|
||||
|
||||
|
||||
MINIFIED_JS = (
|
||||
"!function(){var e=window,t=e.document,n=t.createElement('div');"
|
||||
"n.className='x';for(var r=0;r<1e3;r++){n.appendChild(t.createTextNode(r))}"
|
||||
"e.__x=n;var a=e.location.search.replace(/^\\?/,'').split('&');"
|
||||
) * 16
|
||||
BASE64 = (
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg=="
|
||||
* 12
|
||||
)
|
||||
PROSE = "This is an ordinary paragraph with plenty of spaces in it, " * 12
|
||||
CODE = "\n".join(f" value_{i} = compute(arg_{i}, other_{i}) # comment {i}" for i in range(40))
|
||||
|
||||
|
||||
def test_prose_code_and_short_lines_untouched():
|
||||
for text in (PROSE, CODE, "short", "a" * 299):
|
||||
assert elide_dense_lines(text) == (text, 0)
|
||||
|
||||
|
||||
def test_dense_lines_elided_others_kept():
|
||||
text = f"Script completed\nOutput:\n{MINIFIED_JS}\n{PROSE}\n{BASE64}\n"
|
||||
out, n = elide_dense_lines(text)
|
||||
assert n == 2
|
||||
lines = out.split("\n")
|
||||
assert lines[0] == "Script completed" and lines[3] == PROSE and lines[5] == ""
|
||||
assert lines[2].startswith(MINIFIED_JS[:160]) and lines[2].endswith(MINIFIED_JS[-80:])
|
||||
assert "chars of dense machine-generated content elided" in lines[2]
|
||||
assert len(out) < len(text) // 2
|
||||
|
||||
|
||||
def test_router_applies_elision_when_no_compressor_wins():
|
||||
html = (
|
||||
"<!doctype html><html><head><script>" + MINIFIED_JS + "</script></head><body></body></html>"
|
||||
)
|
||||
router = ContentRouter(ContentRouterConfig(enable_kompress=False, min_section_tokens=10))
|
||||
result = router.compress(html, context="tool_result")
|
||||
assert "content elided" in result.compressed
|
||||
assert len(result.compressed) < len(html) // 2
|
||||
assert result.compressed.strip(), "must never blank a non-empty block"
|
||||
|
||||
off = ContentRouter(
|
||||
ContentRouterConfig(
|
||||
enable_kompress=False, enable_dense_line_elision=False, min_section_tokens=10
|
||||
)
|
||||
)
|
||||
assert "content elided" not in off.compress(html, context="tool_result").compressed
|
||||
|
||||
|
||||
def test_elided_block_carries_a_retrievable_marker():
|
||||
from headroom.cache.compression_store import get_compression_store
|
||||
|
||||
html = "<html><head><script>" + MINIFIED_JS + "</script></head><body>hi there</body></html>"
|
||||
router = ContentRouter(ContentRouterConfig(enable_kompress=False, min_section_tokens=10))
|
||||
out = router.compress(html, context="tool_result").compressed
|
||||
assert "Retrieve original: hash=" in out
|
||||
key = out.rsplit("hash=", 1)[1].rstrip("]\n")
|
||||
assert get_compression_store().retrieve(key) is not None
|
||||
|
||||
|
||||
def test_messages_path_keeps_elided_tool_result(tokenizer):
|
||||
"""The Claude path discards marker-less lossy results (#1307); elision must survive it."""
|
||||
body = (
|
||||
"<!doctype html><html><head><script>"
|
||||
+ MINIFIED_JS * 3
|
||||
+ "</script></head><body><p>x</p></body></html>"
|
||||
)
|
||||
messages = [
|
||||
{"role": "user", "content": "check the site"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{"type": "tool_use", "id": "t1", "name": "Bash", "input": {"command": "curl -s x"}}
|
||||
],
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": body}],
|
||||
},
|
||||
{"role": "assistant", "content": "ok"},
|
||||
{"role": "user", "content": "and now?"},
|
||||
]
|
||||
router = ContentRouter(ContentRouterConfig(enable_kompress=False, min_section_tokens=10))
|
||||
result = router.apply(messages, tokenizer)
|
||||
block = result.messages[2]["content"][0]["content"]
|
||||
text = block if isinstance(block, str) else block[0]["text"]
|
||||
assert "content elided" in text and "Retrieve original: hash=" in text
|
||||
assert len(text) < len(body) // 2
|
||||
|
||||
|
||||
def test_single_dense_values_and_tsv_are_kept():
|
||||
"""A lone JWT / signed URL / PATH / modulus is a value the agent asked for; TSV is data."""
|
||||
jwt = "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9." + "eyJzdWIiOiIxMjM0NTY3ODkwIn0" * 40 + ".sig"
|
||||
url = "https://d1.cloudfront.net/v.mp4?Policy=" + "A" * 500 + "&Signature=" + "b" * 128
|
||||
path = "PATH=" + ":".join(f"/opt/homebrew/opt/package{i}/bin" for i in range(24))
|
||||
modulus = "Modulus: " + ":".join("ab" for _ in range(256))
|
||||
tsv = "\n".join("\t".join(f"column_{i}_value_{j}" for i in range(35)) for j in range(5))
|
||||
for text in (jwt, url, path, modulus, tsv, "Output:\n" + tsv * 3):
|
||||
assert elide_dense_lines(text) == (text, 0)
|
||||
|
||||
|
||||
def test_compact_json_lines_are_left_to_smart_crusher():
|
||||
objs = " ".join(
|
||||
f'{{"file":"src/m_{i}.py","line":{i},"text":"repeated search payload"}}' for i in range(160)
|
||||
)
|
||||
assert elide_dense_lines(objs) == (objs, 0)
|
||||
assert elide_dense_lines("[" + ",".join(f'{{"id":{i}}}' for i in range(200)) + "]")[1] == 0
|
||||
|
||||
|
||||
def test_lossless_mode_never_elides():
|
||||
html = "<html><head><script>" + MINIFIED_JS + "</script></head><body>hi</body></html>"
|
||||
router = ContentRouter(
|
||||
ContentRouterConfig(enable_kompress=False, lossless=True, min_section_tokens=10)
|
||||
)
|
||||
assert "content elided" not in router.compress(html, context="tool_result").compressed
|
||||
|
||||
|
||||
def test_search_output_keeps_line_structure():
|
||||
grep = "src/bundle.js\n1:" + MINIFIED_JS + "\n2:export default x\nsrc/app.py\n10:def main():\n"
|
||||
out, n = elide_dense_lines(grep)
|
||||
lines = out.split("\n")
|
||||
assert n == 1 and lines[0] == "src/bundle.js" and lines[1].startswith("1:!function")
|
||||
assert lines[2:] == ["2:export default x", "src/app.py", "10:def main():", ""]
|
||||
Reference in New Issue
Block a user