mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(transforms): detect fixed-width command output as tabular and keep it out of Kompress (#3786)
## Description Part of #3652. **The problem.** Commands like `ls -la`, `ps aux`, `df -h`, `docker ps` and `kubectl get pods` print a table whose columns are lined up with spaces. Headroom did not recognise that shape, so it treated the output as plain text and sent it to Kompress, the ML compressor meant for prose. Kompress drops words it thinks are unimportant. In a table that means some rows lose a field (an owner, a size, a date) and others keep it, and nothing in the output says which rows were changed. The agent then works from a file listing that is quietly wrong. **Still happening on `main`.** I ran the example from the issue on `main` at3aa50128with the Kompress model loaded. 60 lines of `ls -la` came back as 2 lines, and the owner column was missing from 17 of the 59 file rows. **What this PR changes.** 1. The content detector now recognises space-aligned columns and labels them as a table (`TABULAR`, format `fixed_width`) instead of plain text. 2. The router no longer passes a table to Kompress when the table compressor could not shrink it. Before this PR, a table the tabular compressor declined (for example a CSV with one uneven row, see #1652) still went to Kompress as a fallback. Now it is kept exactly as it was. With both changes, the issue's `ls -la` example comes back byte-for-byte unchanged: 60 lines, owner present on all 59 rows. **What this PR does not fix.** - The `git status` example from the same issue. It has only 2 columns under prose headings, and a rule loose enough to catch it would also catch bullet lists and `key: value` lines. It is still word-dropped, so this PR says "Part of" rather than "Closes", and #3652 should stay open. - Tables where columns are sometimes separated by a single space (`ls -l`, `ps aux`, `df -h`) split into uneven rows with the existing parser, so they are now kept verbatim. That is safe, but it saves nothing. Tables with wider gaps (`kubectl get`, `docker ps`) go through the existing lossless CSV-schema compaction that markdown tables already use, and every row is kept. In my runs that took a 2,805-byte `kubectl` listing to 1,803 bytes and a 4,194-byte `docker ps` listing to 3,628 bytes. - `kubectl` output that contains `Error` pods can be classified as build/log output earlier in the detector, so it never reaches the new check. - The opt-in `lossless_then_lossy` and `force_kompress_all` modes are unchanged. - Whether Bash output should be excluded from compression by default (#3590) is a separate call for maintainers. `DEFAULT_EXCLUDE_TOOLS` is not touched here. **How it relates to other open PRs.** - #3119 keeps line breaks inside Kompress output. It helps, but it does not fix this issue on its own: I applied it to `main` in a scratch copy and the `ls -la` example kept all 60 lines but still lost the owner from 17 of 59 rows. This PR keeps that kind of output away from Kompress entirely, so the two work together. - #3590 reports the same symptom and suggests excluding Bash by default. - #3514 edits `tabular_ingest.py` and the same test file in different places. Whichever lands second may need a small rebase of the test imports. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/content_detector.py`: new `_try_detect_fixed_width`. It runs last, right before the plain-text fallback, so it can only change content that is classified as plain text today. It needs at least 4 lines and at least 3 columns whose gaps line up on 90% of the lines. It backs off when the text looks like a bullet or numbered list, wrapped prose (lines ending in `.`, `!` or `?`), or code (lines ending in `{ } ; ( ) , : \` or starting with a comment marker). A one-line heading such as `total 480` above the `ls -l` rows is allowed. - `headroom/transforms/content_router.py`: - `TABULAR` is removed from the strategies that fall back to Kompress when they save nothing, so a declined table stays verbatim. A stale comment about that fallback is updated. - File reads (`cat`, `Read`) of space-aligned text stay protected (byte-exact), because hand-aligned files such as `/etc/fstab` or a block of C `#define`s look the same as command output. CSV and markdown reads are released as before. - `headroom/transforms/tabular_ingest.py`: `parse_tabular` returns `None` for a fixed-width table that parses into a single column. The detector finds columns by single-space gaps, but the existing parser splits on two or more spaces, so GNU `ls -l` can parse as one cell per line. That is not a table, so it is passed through verbatim. - `tests/test_transforms_tabular.py`: - detection tests for 5 real command outputs (the issue's `ls -la`, macOS `ls -la`, `kubectl get pods`, `ps aux`, `df -h`); - 7 look-alikes that must stay plain text (bullets, numbered list, SQL, C `#define`s, `git status`, `git log --oneline`, wrapped prose); - a parser test for the single-column case; - router tests showing that the issue's `ls -la` payload and an uneven CSV come back unchanged and never reach Kompress; - a test that a fixed-width file read stays protected while a CSV read is released. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed Type checking: `mypy headroom --ignore-missing-imports` (mypy 1.20.2) reports one error, in `headroom/fileperms.py` (`os.fchmod`, which does not exist on Windows). That file is not touched by this PR and the changed files are clean, so I left the box unchecked. Full suite on Windows: 13,001 passed, 57 failed, 57 errors. I reran every failing test on an untouched `main` checkout (3aa50128) and on this branch. The same 52 fail on both (Windows file modes, symlinks, POSIX paths). The rest passed when rerun on their own, so none of the failures come from this change. ### Test Output ```text $ pytest tests/test_transforms_tabular.py -q .................sssss............ [100%] ======================= 68 passed, 5 skipped in 54.55s ======================== $ ruff check . && ruff format --check . All checks passed! 1653 files already formatted $ pytest tests -q -n 8 # Windows, full suite on this branch = 57 failed, 13001 passed, 861 skipped, 2785 warnings, 57 errors in 903.33s (0:15:03) = $ pytest <every failing test above> -q -n 8 # same list, main vs this branch main (3aa50128): 50 failed, 32 passed, 14 warnings, 2 errors this branch: 50 failed, 32 passed, 14 warnings, 2 errors (identical failing set) ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.11, headroom installed from this branch, Kompress model `kompress-v2-base` loaded from the local cache, pure-Python content detection (the default on Windows). - Exact command / steps: built the `ls -la` payload from #3652 (a `total 480` line plus 59 file rows, sizes from `random.seed(1)`) and ran it through `ContentRouter(ContentRouterConfig()).compress(payload)`, once on `main` at3aa50128and once on this branch, with Kompress warmed up first. - Observed result: on `main` the payload is detected as `PLAIN_TEXT` (confidence 0.5), goes to Kompress, and comes back as 2 lines with the owner on only 42 of 59 rows. On this branch it is detected as `TABULAR` (`fixed_width`, confidence 0.7), the strategy is `TABULAR`, and the output is byte-for-byte identical to the input: 60 lines, owner on 59 of 59 rows. I also ran the new detector over every `.py`, `.md`, `.rs`, `.ts`, `.sh`, `.toml`, `.yml` and `.txt` file in this repo, cut into 60-line chunks (about 12,400 chunks). It matched the real table in `sbom/headroom-sbom-table.txt` and one block of four lined-up `assert` lines in a test file, which is now passed through unchanged instead of going to Kompress. - Not tested: Linux or macOS, where the Rust/Magika detector runs first and this Python check only runs if that detector also says plain text. Not tested in a live proxy session with a real agent. ## Runtime Rollout Safety - Rollout-managed feature(s): None. This is a detection and routing fix in the content router. - Minimum rollout channel: stable - Stable/default behavior changed: Yes. Space-aligned command output that used to be word-dropped by Kompress is now either kept verbatim or compacted losslessly (every row kept), and tables the tabular compressor declines are no longer sent to Kompress. Some of that output now saves fewer tokens than before, in exchange for not losing fields. - Kill switch / disable path: No dedicated switch. Turning off the tabular compressor (`--compressor` without `tabular`) keeps these outputs verbatim. There is no setting that restores the old word-dropping behavior; reverting this PR does. - Unsafe override required: No - Qualification impact: None expected. The change only affects content that was previously classified as plain text, or tables the tabular compressor had already declined. - Rollback path: Revert this PR. There is no stored state or migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`. It is generated by release-please from the Conventional Commit PR title. ## Additional Notes No documentation change: `tabular_ingest.py` and the `TABULAR` content type already say fixed-width tables are supported. Until now the detector never produced that format, so that path was only reachable in tests. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
23c2054552
commit
6c9aef1dec
@@ -245,7 +245,13 @@ def detect_content_type(content: str) -> DetectionResult:
|
||||
if code_result and code_result.confidence >= 0.5:
|
||||
return code_result
|
||||
|
||||
# 9. Fallback to plain text
|
||||
# 9. Space-aligned command output (`ls -l`, `ps aux`, `docker ps`). Last,
|
||||
# so it only claims content that would otherwise be plain text.
|
||||
fixed_width_result = _try_detect_fixed_width(content)
|
||||
if fixed_width_result:
|
||||
return fixed_width_result
|
||||
|
||||
# 10. Fallback to plain text
|
||||
return DetectionResult(ContentType.PLAIN_TEXT, 0.5, {})
|
||||
|
||||
|
||||
@@ -716,6 +722,82 @@ def _try_detect_tabular(content: str) -> DetectionResult | None:
|
||||
return _try_detect_delimited(lines)
|
||||
|
||||
|
||||
# Fixed-width (space-aligned) command output: `ls -l`, `ps aux`, `df -h`,
|
||||
# `docker ps`, `kubectl get`. These rows have no delimiter, so without this
|
||||
# check they fall through to PLAIN_TEXT and the prose compressor drops fields
|
||||
# out of individual rows with nothing marking which row lost what (#3652).
|
||||
_FIXED_WIDTH_LIST_RE = re.compile(r"^\s*(?:[-*+•]|\d{1,3}[.)])\s")
|
||||
_FIXED_WIDTH_PROSE_END_RE = re.compile(r"[A-Za-z][.!?][\"')\]]?$")
|
||||
_FIXED_WIDTH_CODE_ENDS = ("{", "}", ";", "(", ")", ",", ":", "\\")
|
||||
_FIXED_WIDTH_CODE_STARTS = ("#", "//", "/*", "--")
|
||||
_FIXED_WIDTH_MAX_COLS = 400
|
||||
|
||||
|
||||
def _fixed_width_gutters(lines: list[str]) -> int:
|
||||
"""Count the column gaps shared by at least 90% of ``lines``.
|
||||
|
||||
A gutter is a character position that holds a space strictly inside the
|
||||
text (after the first non-space character; lines are right-stripped) on
|
||||
nearly every line. Adjacent gutter positions count as one gap, so a table
|
||||
with N columns has N - 1 gaps. Prose and code line up by accident on one
|
||||
position at most, not on several.
|
||||
"""
|
||||
need = -(-9 * len(lines) // 10) # ceil(0.9 * n)
|
||||
hits = [0] * _FIXED_WIDTH_MAX_COLS
|
||||
for ln in lines:
|
||||
start = len(ln) - len(ln.lstrip(" "))
|
||||
for i in range(start, min(len(ln), _FIXED_WIDTH_MAX_COLS)):
|
||||
if ln[i] == " ":
|
||||
hits[i] += 1
|
||||
gaps = 0
|
||||
in_gap = False
|
||||
for count in hits:
|
||||
is_gutter = count >= need
|
||||
if is_gutter and not in_gap:
|
||||
gaps += 1
|
||||
in_gap = is_gutter
|
||||
return gaps
|
||||
|
||||
|
||||
def _try_detect_fixed_width(content: str) -> DetectionResult | None:
|
||||
"""Detect space-aligned columns (>= 3 of them) in command output.
|
||||
|
||||
Runs last, only for content that would otherwise be PLAIN_TEXT, so it
|
||||
never takes content away from another detector. A false positive costs
|
||||
savings, not correctness: the tabular compressor passes rows it cannot
|
||||
split cleanly through verbatim.
|
||||
"""
|
||||
lines = [ln.rstrip() for ln in content.split("\n") if ln.strip()][:50]
|
||||
if len(lines) < 4 or any("\t" in ln for ln in lines):
|
||||
return None
|
||||
|
||||
n = len(lines)
|
||||
if sum(1 for ln in lines if _FIXED_WIDTH_LIST_RE.match(ln)) / n >= 0.5:
|
||||
return None
|
||||
if sum(1 for ln in lines if _FIXED_WIDTH_PROSE_END_RE.search(ln)) / n >= 0.3:
|
||||
return None
|
||||
code_like = sum(
|
||||
1
|
||||
for ln in lines
|
||||
if ln.endswith(_FIXED_WIDTH_CODE_ENDS) or ln.lstrip().startswith(_FIXED_WIDTH_CODE_STARTS)
|
||||
)
|
||||
if code_like / n >= 0.3:
|
||||
return None
|
||||
|
||||
gaps = _fixed_width_gutters(lines)
|
||||
if gaps < 2 and n >= 5:
|
||||
# A one-line preamble (`total 480` above `ls -l` rows) has no gaps of
|
||||
# its own; measure the rows without it.
|
||||
gaps = _fixed_width_gutters(lines[1:])
|
||||
if gaps < 2:
|
||||
return None
|
||||
return DetectionResult(
|
||||
ContentType.TABULAR,
|
||||
0.7,
|
||||
{"format": "fixed_width", "columns": gaps + 1},
|
||||
)
|
||||
|
||||
|
||||
def _try_parse_toml(content: str) -> bool:
|
||||
"""True if `content` parses as TOML (stdlib tomllib, or the tomli backport)."""
|
||||
try:
|
||||
|
||||
@@ -1173,10 +1173,16 @@ def _read_output_should_be_protected(text: Any) -> bool:
|
||||
if not isinstance(text, str) or not text:
|
||||
return False
|
||||
try:
|
||||
return _detect_content(text).content_type not in _RELEASABLE_READ_TYPES
|
||||
detection = _detect_content(text)
|
||||
except Exception:
|
||||
# Detection failure → protect (preserve the byte-exact default).
|
||||
return True
|
||||
if detection.metadata.get("format") == "fixed_width":
|
||||
# Space-aligned columns are also what hand-aligned files look like
|
||||
# (/etc/fstab, a block of C #defines), so a READ of one stays
|
||||
# byte-exact. Only delimited and markdown tables are released.
|
||||
return True
|
||||
return detection.content_type not in _RELEASABLE_READ_TYPES
|
||||
|
||||
|
||||
def _create_content_signature(
|
||||
@@ -3941,10 +3947,13 @@ class ContentRouter(Transform):
|
||||
|
||||
# If compression succeeded, record to TOIN
|
||||
if compressed is not None and compressed_tokens is not None:
|
||||
# TABULAR is deliberately absent: a table the tabular compressor
|
||||
# declined (ragged rows, #1652) or could not shrink stays verbatim.
|
||||
# Kompress drops words inside rows, so it would remove fields from
|
||||
# some rows and not others with nothing marking which (#3652).
|
||||
fallback_eligible_strategy = strategy in {
|
||||
CompressionStrategy.SMART_CRUSHER,
|
||||
CompressionStrategy.CODE_AWARE,
|
||||
CompressionStrategy.TABULAR,
|
||||
CompressionStrategy.CONFIG,
|
||||
}
|
||||
fallback_no_savings = compressed == content or compressed_tokens >= original_tokens
|
||||
@@ -4024,8 +4033,8 @@ class ContentRouter(Transform):
|
||||
# actually shorter — never inflating, never doing worse than the
|
||||
# strategy output. DIFF is excluded (Kompress corrupts ``git
|
||||
# apply``); TEXT/KOMPRESS already ran Kompress; CODE_AWARE has its
|
||||
# own inline no-shrink fallback; SMART_CRUSHER/TABULAR use the
|
||||
# zero-savings fallback above.
|
||||
# own inline no-shrink fallback; SMART_CRUSHER uses the
|
||||
# zero-savings fallback above; TABULAR never goes to Kompress.
|
||||
if (
|
||||
self._lossless_then_lossy
|
||||
and compressed is not None
|
||||
|
||||
@@ -153,6 +153,11 @@ def parse_tabular(
|
||||
width = len(headers)
|
||||
if any(len(row) != width for row in rows):
|
||||
return None
|
||||
# The detector finds fixed-width columns by single-space gutters, but the
|
||||
# parser splits on 2+ spaces, so a table like GNU `ls -l` can come back as
|
||||
# one cell per line. That is not a table; leave it verbatim.
|
||||
if fmt == "fixed_width" and width < 2:
|
||||
return None
|
||||
return headers, rows, fmt
|
||||
|
||||
|
||||
|
||||
@@ -9,6 +9,7 @@ from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import importlib.util
|
||||
import random
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -25,6 +26,7 @@ from headroom.transforms.content_router import (
|
||||
CompressionStrategy,
|
||||
ContentRouter,
|
||||
ContentRouterConfig,
|
||||
_read_output_should_be_protected,
|
||||
)
|
||||
from headroom.transforms.tabular_ingest import (
|
||||
TabularCompressionResult,
|
||||
@@ -93,6 +95,107 @@ def test_does_not_misroute_to_tabular(content: str, expected: ContentType) -> No
|
||||
assert detect_content_type(content).content_type is expected
|
||||
|
||||
|
||||
# Detection: fixed-width command output (#3652) ------------------------------
|
||||
|
||||
|
||||
def _ls_issue_payload() -> str:
|
||||
# The exact payload from issue #3652.
|
||||
rng = random.Random(1)
|
||||
rows = [
|
||||
f"-rw-r--r-- 1 tejas staff {rng.randint(1000, 99999)} Sep {d} 09:{d:02d} file_{d}.py"
|
||||
for d in range(1, 60)
|
||||
]
|
||||
return "total 480\n" + "\n".join(rows)
|
||||
|
||||
|
||||
LS_MACOS = (
|
||||
"total 64\n"
|
||||
"drwxr-xr-x 12 tejas staff 384 Sep 18 09:01 .\n"
|
||||
"drwxr-xr-x 5 tejas staff 160 Sep 17 11:20 ..\n"
|
||||
"-rw-r--r-- 1 tejas staff 1834 Sep 18 09:01 README.md\n"
|
||||
"-rw-r--r-- 1 tejas staff 18611 Sep 18 09:01 setup.py\n"
|
||||
"drwxr-xr-x 8 tejas staff 256 Sep 18 09:01 src"
|
||||
)
|
||||
KUBECTL = (
|
||||
"NAME READY STATUS RESTARTS AGE\n"
|
||||
"api-7d9f8b6c4-2xkqp 1/1 Running 0 3d\n"
|
||||
"api-7d9f8b6c4-9wz7m 1/1 Running 0 3d\n"
|
||||
"worker-5c8d7f9b8-lq2vx 1/1 Running 2 5h\n"
|
||||
"redis-0 1/1 Running 0 12d"
|
||||
)
|
||||
PS_AUX = (
|
||||
"USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND\n"
|
||||
"root 1 0.0 0.1 168100 11520 ? Ss Sep17 0:04 /sbin/init\n"
|
||||
"root 2 0.0 0.0 0 0 ? S Sep17 0:00 [kthreadd]\n"
|
||||
"tejas 4121 1.2 2.3 912344 190220 pts/0 Sl+ 09:01 0:12 python app.py\n"
|
||||
"tejas 4188 0.0 0.0 10072 3300 pts/1 R+ 09:05 0:00 ps aux"
|
||||
)
|
||||
DF_H = (
|
||||
"Filesystem Size Used Avail Use% Mounted on\n"
|
||||
"/dev/nvme0n1p2 468G 201G 244G 46% /\n"
|
||||
"tmpfs 16G 1.2M 16G 1% /dev/shm\n"
|
||||
"/dev/nvme0n1p1 511M 6.1M 505M 2% /boot/efi\n"
|
||||
"tmpfs 3.2G 2.4M 3.2G 1% /run/user/1000"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"content",
|
||||
[_ls_issue_payload(), LS_MACOS, KUBECTL, PS_AUX, DF_H],
|
||||
ids=["ls_issue", "ls_macos", "kubectl", "ps_aux", "df_h"],
|
||||
)
|
||||
def test_detects_fixed_width_command_output(content: str) -> None:
|
||||
result = detect_content_type(content)
|
||||
assert result.content_type is ContentType.TABULAR
|
||||
assert result.metadata["format"] == "fixed_width"
|
||||
assert result.metadata["columns"] >= 3
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"content,expected",
|
||||
[
|
||||
pytest.param(
|
||||
"- first item in the list\n- second item in the list\n- third item in the list\n- fourth item in the list",
|
||||
ContentType.PLAIN_TEXT,
|
||||
id="bullets",
|
||||
),
|
||||
pytest.param(
|
||||
"1. install the package\n2. run the proxy\n3. wrap the agent\n4. check the stats",
|
||||
ContentType.PLAIN_TEXT,
|
||||
id="numbered",
|
||||
),
|
||||
pytest.param(
|
||||
"SELECT id, name\nFROM users\nWHERE active = 1\nORDER BY name;\n-- comment\nLIMIT 10;",
|
||||
ContentType.PLAIN_TEXT,
|
||||
id="sql",
|
||||
),
|
||||
pytest.param(
|
||||
"#define FOO 1\n#define BAR 2\n#define BAZ 3\n#define QUX 4",
|
||||
ContentType.PLAIN_TEXT,
|
||||
id="c_defines",
|
||||
),
|
||||
pytest.param(
|
||||
'On branch main\nChanges not staged for commit:\n (use "git add <file>..." to update what will be committed)\n'
|
||||
+ "\n".join(f"\tmodified: src/m_{i}.py" for i in range(10)),
|
||||
ContentType.PLAIN_TEXT,
|
||||
id="git_status",
|
||||
),
|
||||
pytest.param(
|
||||
"3aa5012 perf(memory/budget): precompute word sets once\nc81378c fix(grok): preserve xAI model context metadata\nb0c19a2 fix(security): reject unauthenticated public proxy binds\n871bbde fix(proxy): reject Anthropic batch operations on Copilot\na29162b fix(dashboard): separate rolling cache economics by owner",
|
||||
ContentType.PLAIN_TEXT,
|
||||
id="git_log",
|
||||
),
|
||||
pytest.param(
|
||||
"Headroom compresses tool output before it reaches the model, which saves\ntokens on long agent sessions. The router picks a compressor per content\ntype, and plain prose goes to Kompress, an ML model that drops words it\npredicts the reader can do without. That is fine for prose and wrong for\nrecords, where every field matters to whatever command runs next, so the\ndetector has to tell the two apart before anything is dropped at all.",
|
||||
ContentType.PLAIN_TEXT,
|
||||
id="wrapped_prose",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_fixed_width_does_not_claim_non_tables(content: str, expected: ContentType) -> None:
|
||||
assert detect_content_type(content).content_type is expected
|
||||
|
||||
|
||||
# Detection — edge branches --------------------------------------------------
|
||||
|
||||
|
||||
@@ -251,8 +354,8 @@ def test_parse_fixed_width_too_short_returns_empty() -> None:
|
||||
|
||||
|
||||
def test_parse_tabular_dispatches_fixed_width(monkeypatch) -> None:
|
||||
# The detector currently emits only csv/markdown, so drive the fixed_width
|
||||
# dispatch branch directly with a stubbed detection result.
|
||||
# Drive the fixed_width dispatch branch directly with a stubbed detection
|
||||
# result, independent of the detector's thresholds.
|
||||
import headroom.transforms.tabular_ingest as ti
|
||||
|
||||
monkeypatch.setattr(
|
||||
@@ -266,6 +369,20 @@ def test_parse_tabular_dispatches_fixed_width(monkeypatch) -> None:
|
||||
assert rows[0] == ["Alice", "30"]
|
||||
|
||||
|
||||
def test_parse_tabular_rejects_single_column_fixed_width(monkeypatch) -> None:
|
||||
import headroom.transforms.tabular_ingest as ti
|
||||
|
||||
monkeypatch.setattr(
|
||||
ti,
|
||||
"detect_content_type",
|
||||
lambda _c: DetectionResult(ContentType.TABULAR, 0.9, {"format": "fixed_width"}),
|
||||
)
|
||||
# Single-space rows split into one cell each; that is not a table.
|
||||
assert (
|
||||
ti.parse_tabular("-rw-r--r-- 1 a b 1 f\n-rw-r--r-- 1 a b 2 g\n-rw-r--r-- 1 a b 3 h") is None
|
||||
)
|
||||
|
||||
|
||||
def test_parse_tabular_none_when_no_data_rows_survive() -> None:
|
||||
# Detected as a markdown table, but it is header + separator rows only:
|
||||
# nothing survives as a data row, so parse_tabular bails to None.
|
||||
@@ -342,6 +459,47 @@ def test_router_respects_disable_flag() -> None:
|
||||
assert result.tokens_saved == 0
|
||||
|
||||
|
||||
# Router: tables never fall back to Kompress (#3652) -------------------------
|
||||
|
||||
|
||||
def _record_kompress_calls(monkeypatch) -> list[str]:
|
||||
calls: list[str] = []
|
||||
|
||||
def fake(self, content, context, question=None, target_ratio=None):
|
||||
calls.append(content)
|
||||
return "x", 1 # would "win" on savings if the router ever called it
|
||||
|
||||
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "python")
|
||||
monkeypatch.setattr(ContentRouter, "_try_ml_compressor", fake)
|
||||
return calls
|
||||
|
||||
|
||||
def test_router_keeps_ls_output_verbatim(monkeypatch) -> None:
|
||||
calls = _record_kompress_calls(monkeypatch)
|
||||
payload = _ls_issue_payload()
|
||||
result = ContentRouter(ContentRouterConfig()).compress(payload)
|
||||
assert result.compressed == payload
|
||||
assert calls == []
|
||||
assert result.strategy_used is CompressionStrategy.TABULAR
|
||||
|
||||
|
||||
def test_router_does_not_kompress_a_ragged_csv(monkeypatch) -> None:
|
||||
calls = _record_kompress_calls(monkeypatch)
|
||||
csv = "id,name,city\n" + "\n".join(f"{i},user_{i},city_{i % 5}" for i in range(30))
|
||||
csv += "\n99,extra,field,here"
|
||||
assert detect_content_type(csv).content_type is ContentType.TABULAR
|
||||
result = ContentRouter(ContentRouterConfig()).compress(csv)
|
||||
assert result.compressed == csv
|
||||
assert calls == []
|
||||
|
||||
|
||||
def test_fixed_width_read_stays_protected(monkeypatch) -> None:
|
||||
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "python")
|
||||
assert _read_output_should_be_protected(_ls_issue_payload()) is True
|
||||
csv = "id,name,city\n" + "\n".join(f"{i},user_{i},city_{i % 5}" for i in range(30))
|
||||
assert _read_output_should_be_protected(csv) is False
|
||||
|
||||
|
||||
# Binary spreadsheet ingestion -----------------------------------------------
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user