fix(transforms): detect fixed-width command output as tabular and keep it out of Kompress (#3786)

## Description

Part of #3652.

**The problem.** Commands like `ls -la`, `ps aux`, `df -h`, `docker ps`
and `kubectl get pods` print a table whose columns are lined up with
spaces. Headroom did not recognise that shape, so it treated the output
as plain text and sent it to Kompress, the ML compressor meant for
prose. Kompress drops words it thinks are unimportant. In a table that
means some rows lose a field (an owner, a size, a date) and others keep
it, and nothing in the output says which rows were changed. The agent
then works from a file listing that is quietly wrong.

**Still happening on `main`.** I ran the example from the issue on
`main` at 3aa50128 with the Kompress model loaded. 60 lines of `ls -la`
came back as 2 lines, and the owner column was missing from 17 of the 59
file rows.

**What this PR changes.**

1. The content detector now recognises space-aligned columns and labels
them as a table (`TABULAR`, format `fixed_width`) instead of plain text.
2. The router no longer passes a table to Kompress when the table
compressor could not shrink it. Before this PR, a table the tabular
compressor declined (for example a CSV with one uneven row, see #1652)
still went to Kompress as a fallback. Now it is kept exactly as it was.

With both changes, the issue's `ls -la` example comes back byte-for-byte
unchanged: 60 lines, owner present on all 59 rows.

**What this PR does not fix.**

- The `git status` example from the same issue. It has only 2 columns
under prose headings, and a rule loose enough to catch it would also
catch bullet lists and `key: value` lines. It is still word-dropped, so
this PR says "Part of" rather than "Closes", and #3652 should stay open.
- Tables where columns are sometimes separated by a single space (`ls
-l`, `ps aux`, `df -h`) split into uneven rows with the existing parser,
so they are now kept verbatim. That is safe, but it saves nothing.
Tables with wider gaps (`kubectl get`, `docker ps`) go through the
existing lossless CSV-schema compaction that markdown tables already
use, and every row is kept. In my runs that took a 2,805-byte `kubectl`
listing to 1,803 bytes and a 4,194-byte `docker ps` listing to 3,628
bytes.
- `kubectl` output that contains `Error` pods can be classified as
build/log output earlier in the detector, so it never reaches the new
check.
- The opt-in `lossless_then_lossy` and `force_kompress_all` modes are
unchanged.
- Whether Bash output should be excluded from compression by default
(#3590) is a separate call for maintainers. `DEFAULT_EXCLUDE_TOOLS` is
not touched here.

**How it relates to other open PRs.**

- #3119 keeps line breaks inside Kompress output. It helps, but it does
not fix this issue on its own: I applied it to `main` in a scratch copy
and the `ls -la` example kept all 60 lines but still lost the owner from
17 of 59 rows. This PR keeps that kind of output away from Kompress
entirely, so the two work together.
- #3590 reports the same symptom and suggests excluding Bash by default.
- #3514 edits `tabular_ingest.py` and the same test file in different
places. Whichever lands second may need a small rebase of the test
imports.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/content_detector.py`: new
`_try_detect_fixed_width`. It runs last, right before the plain-text
fallback, so it can only change content that is classified as plain text
today. It needs at least 4 lines and at least 3 columns whose gaps line
up on 90% of the lines. It backs off when the text looks like a bullet
or numbered list, wrapped prose (lines ending in `.`, `!` or `?`), or
code (lines ending in `{ } ; ( ) , : \` or starting with a comment
marker). A one-line heading such as `total 480` above the `ls -l` rows
is allowed.
- `headroom/transforms/content_router.py`:
- `TABULAR` is removed from the strategies that fall back to Kompress
when they save nothing, so a declined table stays verbatim. A stale
comment about that fallback is updated.
- File reads (`cat`, `Read`) of space-aligned text stay protected
(byte-exact), because hand-aligned files such as `/etc/fstab` or a block
of C `#define`s look the same as command output. CSV and markdown reads
are released as before.
- `headroom/transforms/tabular_ingest.py`: `parse_tabular` returns
`None` for a fixed-width table that parses into a single column. The
detector finds columns by single-space gaps, but the existing parser
splits on two or more spaces, so GNU `ls -l` can parse as one cell per
line. That is not a table, so it is passed through verbatim.
- `tests/test_transforms_tabular.py`:
- detection tests for 5 real command outputs (the issue's `ls -la`,
macOS `ls -la`, `kubectl get pods`, `ps aux`, `df -h`);
- 7 look-alikes that must stay plain text (bullets, numbered list, SQL,
C `#define`s, `git status`, `git log --oneline`, wrapped prose);
  - a parser test for the single-column case;
- router tests showing that the issue's `ls -la` payload and an uneven
CSV come back unchanged and never reach Kompress;
- a test that a fixed-width file read stays protected while a CSV read
is released.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Type checking: `mypy headroom --ignore-missing-imports` (mypy 1.20.2)
reports one error, in `headroom/fileperms.py` (`os.fchmod`, which does
not exist on Windows). That file is not touched by this PR and the
changed files are clean, so I left the box unchecked.

Full suite on Windows: 13,001 passed, 57 failed, 57 errors. I reran
every failing test on an untouched `main` checkout (3aa50128) and on
this branch. The same 52 fail on both (Windows file modes, symlinks,
POSIX paths). The rest passed when rerun on their own, so none of the
failures come from this change.

### Test Output

```text
$ pytest tests/test_transforms_tabular.py -q
.................sssss............                                       [100%]
======================= 68 passed, 5 skipped in 54.55s ========================

$ ruff check . && ruff format --check .
All checks passed!
1653 files already formatted

$ pytest tests -q -n 8   # Windows, full suite on this branch
= 57 failed, 13001 passed, 861 skipped, 2785 warnings, 57 errors in 903.33s (0:15:03) =

$ pytest <every failing test above> -q -n 8   # same list, main vs this branch
main (3aa50128): 50 failed, 32 passed, 14 warnings, 2 errors
this branch:     50 failed, 32 passed, 14 warnings, 2 errors   (identical failing set)
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.11, headroom installed from this
branch, Kompress model `kompress-v2-base` loaded from the local cache,
pure-Python content detection (the default on Windows).
- Exact command / steps: built the `ls -la` payload from #3652 (a `total
480` line plus 59 file rows, sizes from `random.seed(1)`) and ran it
through `ContentRouter(ContentRouterConfig()).compress(payload)`, once
on `main` at 3aa50128 and once on this branch, with Kompress warmed up
first.
- Observed result: on `main` the payload is detected as `PLAIN_TEXT`
(confidence 0.5), goes to Kompress, and comes back as 2 lines with the
owner on only 42 of 59 rows. On this branch it is detected as `TABULAR`
(`fixed_width`, confidence 0.7), the strategy is `TABULAR`, and the
output is byte-for-byte identical to the input: 60 lines, owner on 59 of
59 rows. I also ran the new detector over every `.py`, `.md`, `.rs`,
`.ts`, `.sh`, `.toml`, `.yml` and `.txt` file in this repo, cut into
60-line chunks (about 12,400 chunks). It matched the real table in
`sbom/headroom-sbom-table.txt` and one block of four lined-up `assert`
lines in a test file, which is now passed through unchanged instead of
going to Kompress.
- Not tested: Linux or macOS, where the Rust/Magika detector runs first
and this Python check only runs if that detector also says plain text.
Not tested in a live proxy session with a real agent.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. This is a detection and routing fix
in the content router.
- Minimum rollout channel: stable
- Stable/default behavior changed: Yes. Space-aligned command output
that used to be word-dropped by Kompress is now either kept verbatim or
compacted losslessly (every row kept), and tables the tabular compressor
declines are no longer sent to Kompress. Some of that output now saves
fewer tokens than before, in exchange for not losing fields.
- Kill switch / disable path: No dedicated switch. Turning off the
tabular compressor (`--compressor` without `tabular`) keeps these
outputs verbatim. There is no setting that restores the old
word-dropping behavior; reverting this PR does.
- Unsafe override required: No
- Qualification impact: None expected. The change only affects content
that was previously classified as plain text, or tables the tabular
compressor had already declined.
- Rollback path: Revert this PR. There is no stored state or migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`. It is generated by
release-please from the Conventional Commit PR title.

## Additional Notes

No documentation change: `tabular_ingest.py` and the `TABULAR` content
type already say fixed-width tables are supported. Until now the
detector never produced that format, so that path was only reachable in
tests.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Parideboy
2026-09-25 09:48:23 -05:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 23c2054552
commit 6c9aef1dec
4 changed files with 261 additions and 7 deletions
+83 -1
View File
@@ -245,7 +245,13 @@ def detect_content_type(content: str) -> DetectionResult:
if code_result and code_result.confidence >= 0.5:
return code_result
# 9. Fallback to plain text
# 9. Space-aligned command output (`ls -l`, `ps aux`, `docker ps`). Last,
# so it only claims content that would otherwise be plain text.
fixed_width_result = _try_detect_fixed_width(content)
if fixed_width_result:
return fixed_width_result
# 10. Fallback to plain text
return DetectionResult(ContentType.PLAIN_TEXT, 0.5, {})
@@ -716,6 +722,82 @@ def _try_detect_tabular(content: str) -> DetectionResult | None:
return _try_detect_delimited(lines)
# Fixed-width (space-aligned) command output: `ls -l`, `ps aux`, `df -h`,
# `docker ps`, `kubectl get`. These rows have no delimiter, so without this
# check they fall through to PLAIN_TEXT and the prose compressor drops fields
# out of individual rows with nothing marking which row lost what (#3652).
_FIXED_WIDTH_LIST_RE = re.compile(r"^\s*(?:[-*+•]|\d{1,3}[.)])\s")
_FIXED_WIDTH_PROSE_END_RE = re.compile(r"[A-Za-z][.!?][\"')\]]?$")
_FIXED_WIDTH_CODE_ENDS = ("{", "}", ";", "(", ")", ",", ":", "\\")
_FIXED_WIDTH_CODE_STARTS = ("#", "//", "/*", "--")
_FIXED_WIDTH_MAX_COLS = 400
def _fixed_width_gutters(lines: list[str]) -> int:
"""Count the column gaps shared by at least 90% of ``lines``.
A gutter is a character position that holds a space strictly inside the
text (after the first non-space character; lines are right-stripped) on
nearly every line. Adjacent gutter positions count as one gap, so a table
with N columns has N - 1 gaps. Prose and code line up by accident on one
position at most, not on several.
"""
need = -(-9 * len(lines) // 10) # ceil(0.9 * n)
hits = [0] * _FIXED_WIDTH_MAX_COLS
for ln in lines:
start = len(ln) - len(ln.lstrip(" "))
for i in range(start, min(len(ln), _FIXED_WIDTH_MAX_COLS)):
if ln[i] == " ":
hits[i] += 1
gaps = 0
in_gap = False
for count in hits:
is_gutter = count >= need
if is_gutter and not in_gap:
gaps += 1
in_gap = is_gutter
return gaps
def _try_detect_fixed_width(content: str) -> DetectionResult | None:
"""Detect space-aligned columns (>= 3 of them) in command output.
Runs last, only for content that would otherwise be PLAIN_TEXT, so it
never takes content away from another detector. A false positive costs
savings, not correctness: the tabular compressor passes rows it cannot
split cleanly through verbatim.
"""
lines = [ln.rstrip() for ln in content.split("\n") if ln.strip()][:50]
if len(lines) < 4 or any("\t" in ln for ln in lines):
return None
n = len(lines)
if sum(1 for ln in lines if _FIXED_WIDTH_LIST_RE.match(ln)) / n >= 0.5:
return None
if sum(1 for ln in lines if _FIXED_WIDTH_PROSE_END_RE.search(ln)) / n >= 0.3:
return None
code_like = sum(
1
for ln in lines
if ln.endswith(_FIXED_WIDTH_CODE_ENDS) or ln.lstrip().startswith(_FIXED_WIDTH_CODE_STARTS)
)
if code_like / n >= 0.3:
return None
gaps = _fixed_width_gutters(lines)
if gaps < 2 and n >= 5:
# A one-line preamble (`total 480` above `ls -l` rows) has no gaps of
# its own; measure the rows without it.
gaps = _fixed_width_gutters(lines[1:])
if gaps < 2:
return None
return DetectionResult(
ContentType.TABULAR,
0.7,
{"format": "fixed_width", "columns": gaps + 1},
)
def _try_parse_toml(content: str) -> bool:
"""True if `content` parses as TOML (stdlib tomllib, or the tomli backport)."""
try:
+13 -4
View File
@@ -1173,10 +1173,16 @@ def _read_output_should_be_protected(text: Any) -> bool:
if not isinstance(text, str) or not text:
return False
try:
return _detect_content(text).content_type not in _RELEASABLE_READ_TYPES
detection = _detect_content(text)
except Exception:
# Detection failure → protect (preserve the byte-exact default).
return True
if detection.metadata.get("format") == "fixed_width":
# Space-aligned columns are also what hand-aligned files look like
# (/etc/fstab, a block of C #defines), so a READ of one stays
# byte-exact. Only delimited and markdown tables are released.
return True
return detection.content_type not in _RELEASABLE_READ_TYPES
def _create_content_signature(
@@ -3941,10 +3947,13 @@ class ContentRouter(Transform):
# If compression succeeded, record to TOIN
if compressed is not None and compressed_tokens is not None:
# TABULAR is deliberately absent: a table the tabular compressor
# declined (ragged rows, #1652) or could not shrink stays verbatim.
# Kompress drops words inside rows, so it would remove fields from
# some rows and not others with nothing marking which (#3652).
fallback_eligible_strategy = strategy in {
CompressionStrategy.SMART_CRUSHER,
CompressionStrategy.CODE_AWARE,
CompressionStrategy.TABULAR,
CompressionStrategy.CONFIG,
}
fallback_no_savings = compressed == content or compressed_tokens >= original_tokens
@@ -4024,8 +4033,8 @@ class ContentRouter(Transform):
# actually shorter — never inflating, never doing worse than the
# strategy output. DIFF is excluded (Kompress corrupts ``git
# apply``); TEXT/KOMPRESS already ran Kompress; CODE_AWARE has its
# own inline no-shrink fallback; SMART_CRUSHER/TABULAR use the
# zero-savings fallback above.
# own inline no-shrink fallback; SMART_CRUSHER uses the
# zero-savings fallback above; TABULAR never goes to Kompress.
if (
self._lossless_then_lossy
and compressed is not None
+5
View File
@@ -153,6 +153,11 @@ def parse_tabular(
width = len(headers)
if any(len(row) != width for row in rows):
return None
# The detector finds fixed-width columns by single-space gutters, but the
# parser splits on 2+ spaces, so a table like GNU `ls -l` can come back as
# one cell per line. That is not a table; leave it verbatim.
if fmt == "fixed_width" and width < 2:
return None
return headers, rows, fmt
+160 -2
View File
@@ -9,6 +9,7 @@ from __future__ import annotations
import datetime
import importlib.util
import random
import pytest
@@ -25,6 +26,7 @@ from headroom.transforms.content_router import (
CompressionStrategy,
ContentRouter,
ContentRouterConfig,
_read_output_should_be_protected,
)
from headroom.transforms.tabular_ingest import (
TabularCompressionResult,
@@ -93,6 +95,107 @@ def test_does_not_misroute_to_tabular(content: str, expected: ContentType) -> No
assert detect_content_type(content).content_type is expected
# Detection: fixed-width command output (#3652) ------------------------------
def _ls_issue_payload() -> str:
# The exact payload from issue #3652.
rng = random.Random(1)
rows = [
f"-rw-r--r-- 1 tejas staff {rng.randint(1000, 99999)} Sep {d} 09:{d:02d} file_{d}.py"
for d in range(1, 60)
]
return "total 480\n" + "\n".join(rows)
LS_MACOS = (
"total 64\n"
"drwxr-xr-x 12 tejas staff 384 Sep 18 09:01 .\n"
"drwxr-xr-x 5 tejas staff 160 Sep 17 11:20 ..\n"
"-rw-r--r-- 1 tejas staff 1834 Sep 18 09:01 README.md\n"
"-rw-r--r-- 1 tejas staff 18611 Sep 18 09:01 setup.py\n"
"drwxr-xr-x 8 tejas staff 256 Sep 18 09:01 src"
)
KUBECTL = (
"NAME READY STATUS RESTARTS AGE\n"
"api-7d9f8b6c4-2xkqp 1/1 Running 0 3d\n"
"api-7d9f8b6c4-9wz7m 1/1 Running 0 3d\n"
"worker-5c8d7f9b8-lq2vx 1/1 Running 2 5h\n"
"redis-0 1/1 Running 0 12d"
)
PS_AUX = (
"USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND\n"
"root 1 0.0 0.1 168100 11520 ? Ss Sep17 0:04 /sbin/init\n"
"root 2 0.0 0.0 0 0 ? S Sep17 0:00 [kthreadd]\n"
"tejas 4121 1.2 2.3 912344 190220 pts/0 Sl+ 09:01 0:12 python app.py\n"
"tejas 4188 0.0 0.0 10072 3300 pts/1 R+ 09:05 0:00 ps aux"
)
DF_H = (
"Filesystem Size Used Avail Use% Mounted on\n"
"/dev/nvme0n1p2 468G 201G 244G 46% /\n"
"tmpfs 16G 1.2M 16G 1% /dev/shm\n"
"/dev/nvme0n1p1 511M 6.1M 505M 2% /boot/efi\n"
"tmpfs 3.2G 2.4M 3.2G 1% /run/user/1000"
)
@pytest.mark.parametrize(
"content",
[_ls_issue_payload(), LS_MACOS, KUBECTL, PS_AUX, DF_H],
ids=["ls_issue", "ls_macos", "kubectl", "ps_aux", "df_h"],
)
def test_detects_fixed_width_command_output(content: str) -> None:
result = detect_content_type(content)
assert result.content_type is ContentType.TABULAR
assert result.metadata["format"] == "fixed_width"
assert result.metadata["columns"] >= 3
@pytest.mark.parametrize(
"content,expected",
[
pytest.param(
"- first item in the list\n- second item in the list\n- third item in the list\n- fourth item in the list",
ContentType.PLAIN_TEXT,
id="bullets",
),
pytest.param(
"1. install the package\n2. run the proxy\n3. wrap the agent\n4. check the stats",
ContentType.PLAIN_TEXT,
id="numbered",
),
pytest.param(
"SELECT id, name\nFROM users\nWHERE active = 1\nORDER BY name;\n-- comment\nLIMIT 10;",
ContentType.PLAIN_TEXT,
id="sql",
),
pytest.param(
"#define FOO 1\n#define BAR 2\n#define BAZ 3\n#define QUX 4",
ContentType.PLAIN_TEXT,
id="c_defines",
),
pytest.param(
'On branch main\nChanges not staged for commit:\n (use "git add <file>..." to update what will be committed)\n'
+ "\n".join(f"\tmodified: src/m_{i}.py" for i in range(10)),
ContentType.PLAIN_TEXT,
id="git_status",
),
pytest.param(
"3aa5012 perf(memory/budget): precompute word sets once\nc81378c fix(grok): preserve xAI model context metadata\nb0c19a2 fix(security): reject unauthenticated public proxy binds\n871bbde fix(proxy): reject Anthropic batch operations on Copilot\na29162b fix(dashboard): separate rolling cache economics by owner",
ContentType.PLAIN_TEXT,
id="git_log",
),
pytest.param(
"Headroom compresses tool output before it reaches the model, which saves\ntokens on long agent sessions. The router picks a compressor per content\ntype, and plain prose goes to Kompress, an ML model that drops words it\npredicts the reader can do without. That is fine for prose and wrong for\nrecords, where every field matters to whatever command runs next, so the\ndetector has to tell the two apart before anything is dropped at all.",
ContentType.PLAIN_TEXT,
id="wrapped_prose",
),
],
)
def test_fixed_width_does_not_claim_non_tables(content: str, expected: ContentType) -> None:
assert detect_content_type(content).content_type is expected
# Detection — edge branches --------------------------------------------------
@@ -251,8 +354,8 @@ def test_parse_fixed_width_too_short_returns_empty() -> None:
def test_parse_tabular_dispatches_fixed_width(monkeypatch) -> None:
# The detector currently emits only csv/markdown, so drive the fixed_width
# dispatch branch directly with a stubbed detection result.
# Drive the fixed_width dispatch branch directly with a stubbed detection
# result, independent of the detector's thresholds.
import headroom.transforms.tabular_ingest as ti
monkeypatch.setattr(
@@ -266,6 +369,20 @@ def test_parse_tabular_dispatches_fixed_width(monkeypatch) -> None:
assert rows[0] == ["Alice", "30"]
def test_parse_tabular_rejects_single_column_fixed_width(monkeypatch) -> None:
import headroom.transforms.tabular_ingest as ti
monkeypatch.setattr(
ti,
"detect_content_type",
lambda _c: DetectionResult(ContentType.TABULAR, 0.9, {"format": "fixed_width"}),
)
# Single-space rows split into one cell each; that is not a table.
assert (
ti.parse_tabular("-rw-r--r-- 1 a b 1 f\n-rw-r--r-- 1 a b 2 g\n-rw-r--r-- 1 a b 3 h") is None
)
def test_parse_tabular_none_when_no_data_rows_survive() -> None:
# Detected as a markdown table, but it is header + separator rows only:
# nothing survives as a data row, so parse_tabular bails to None.
@@ -342,6 +459,47 @@ def test_router_respects_disable_flag() -> None:
assert result.tokens_saved == 0
# Router: tables never fall back to Kompress (#3652) -------------------------
def _record_kompress_calls(monkeypatch) -> list[str]:
calls: list[str] = []
def fake(self, content, context, question=None, target_ratio=None):
calls.append(content)
return "x", 1 # would "win" on savings if the router ever called it
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "python")
monkeypatch.setattr(ContentRouter, "_try_ml_compressor", fake)
return calls
def test_router_keeps_ls_output_verbatim(monkeypatch) -> None:
calls = _record_kompress_calls(monkeypatch)
payload = _ls_issue_payload()
result = ContentRouter(ContentRouterConfig()).compress(payload)
assert result.compressed == payload
assert calls == []
assert result.strategy_used is CompressionStrategy.TABULAR
def test_router_does_not_kompress_a_ragged_csv(monkeypatch) -> None:
calls = _record_kompress_calls(monkeypatch)
csv = "id,name,city\n" + "\n".join(f"{i},user_{i},city_{i % 5}" for i in range(30))
csv += "\n99,extra,field,here"
assert detect_content_type(csv).content_type is ContentType.TABULAR
result = ContentRouter(ContentRouterConfig()).compress(csv)
assert result.compressed == csv
assert calls == []
def test_fixed_width_read_stays_protected(monkeypatch) -> None:
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "python")
assert _read_output_should_be_protected(_ls_issue_payload()) is True
csv = "id,name,city\n" + "\n".join(f"{i},user_{i},city_{i % 5}" for i in range(30))
assert _read_output_should_be_protected(csv) is False
# Binary spreadsheet ingestion -----------------------------------------------