mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(tabular): split fixed-width rows on column boundaries, not on double spaces (#4541)
## Description `parse_fixed_width` split every row on runs of 2+ spaces, while the detector finds fixed-width columns by character position. When the two disagree inside one row, the cell count can still match the header, the ragged-row guard in `parse_tabular` passes, and a value is stored under the wrong column. The compressed output (`csv-schema`) carries no CCR marker, so the model cannot tell. See the issue for the exact input. This change cuts each row at the character positions that are blank on every line. If a row crosses a column boundary (a header cell comes out empty or holds a 2+ space gap), the table is not parsed and passes through verbatim. Fixes #4540 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/tabular_ingest.py`: `parse_fixed_width` now finds column spans from positions that are blank on every line, slices each row by those spans, and returns `([], [])` when the text has a tab, or when a header cell is empty or contains a 2+ space gap. A stale comment in `parse_tabular` about the parser splitting on 2+ spaces was updated. - `tests/test_transforms_tabular.py`: four tests for `parse_fixed_width` (the PID/USER/COMMAND shape, an empty cell next to a double-spaced cell, an ambiguous grid that must refuse, a row crossing a boundary) and one kubectl control through `parse_tabular`. Behaviour change to be aware of: a row with an empty cell is now parsed with `""` in that column when the grid is unambiguous (before, it counted as ragged and the table passed through). A table where one row does not line up with the others is still not parsed. Output for aligned tables (kubectl, `ps aux`, `df -h` fixtures) is unchanged. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output With the fix reverted (source file only, new tests kept), run from a copy of the tree with the 0.42.0 `_core.abi3.so`: ```text FAILED tests/test_transforms_tabular.py::test_parse_fixed_width_splits_on_column_positions_not_double_spaces FAILED tests/test_transforms_tabular.py::test_parse_fixed_width_keeps_an_empty_cell_empty FAILED tests/test_transforms_tabular.py::test_parse_fixed_width_never_shifts_a_value_when_the_grid_is_ambiguous FAILED tests/test_transforms_tabular.py::test_parse_fixed_width_refuses_a_row_that_crosses_a_column_boundary =================== 4 failed, 85 passed, 17 skipped in 2.40s =================== ``` With the fix, three files: ```text tests/test_transforms_tabular.py ....................................... [ 18%] ................................................sssssssssssssssss.. [ 49%] tests/test_transforms_content_detection.py .................... [ 58%] tests/test_transforms_content_router.py ................................ [ 73%] ..................................................ss.... [100%] ======================= 195 passed, 19 skipped in 5.54s ======================== ``` ```text $ ruff check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py All checks passed! $ ruff format --check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py 2 files already formatted ``` The skipped tests are the spreadsheet loader tests; they were skipped on `main` too. ## Real Behavior Proof - Environment: Ubuntu under WSL2 on Windows 11, Python 3.12.15, Python source of this branch on top of `main` `a2d4fe72`, `_core.abi3.so` copied from the `headroom_ai` 0.42.0 wheel into a copy of the tree (not into the branch), `HEADROOM_OFFLINE=1`. - Exact command / steps: ran `TabularCompressor().compress(text)` and `parse_tabular(text)` on the table from the issue (`PID USER COMMAND` header, rows `1 root /sbin/init`, `22 www-data nginx`, `4444 sysadmin1 sleep 60`, `5 root cron`, plus 40 rows `"%-5d root worker%d" % (100+i, i)`), once on `main` and once on this branch. - Observed result: on `main`, row 4444 parsed as `['4444', 'sysadmin1 sleep', '60']` and the output line was `60,4444,sysadmin1 sleep` under `{COMMAND,PID,USER}`. On this branch it parses as `['4444', 'sysadmin1', 'sleep 60']` and the output line is `sleep 60,4444,sysadmin1`. For the empty-cell shape (`NAME STATUS NOTE`, row `beta two words`) `main` returned `['beta', 'two', 'words']`; this branch returns `None` (passthrough). kubectl and `df -h` fixtures parse to the same columns as before. - Not tested: Windows or macOS runs; the full test suite (I ran the three tabular/detection/router test files); `mypy`; tables with wide (CJK) characters, where character positions and display columns differ, so such a table may be refused or grouped by code point; very large outputs for speed (the gutter scan is one pass over each line per column position, bounded by the longest line). ## Runtime Rollout Safety - Rollout-managed feature(s): None. This changes a parser used by the tabular compressor; no rollout-managed feature is touched. - Minimum rollout channel: N/A (not rollout-managed) - Stable/default behavior changed: Yes, narrowly. Fixed-width tables that did not line up used to be compressed with shifted values and now pass through; tables with an empty cell now parse with `""` instead of passing through as ragged. Aligned tables are unchanged. - Kill switch / disable path: the existing option that turns the tabular compressor off in the content router; no new switch. - Unsafe override required: No - Qualification impact: None expected; not checked against the qualification suite. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Mypy was not run (marked unchecked above). Documentation: none of the docs I know of describe the fixed-width splitting rule, so I did not change any (not searched exhaustively). - The ambiguous-grid case is a tradeoff: when every cell in a column is shorter than one double-spaced cell, the gap inside that cell looks like a column gutter. The parser refuses rather than guess. Produced with AI assistance (Claude Code) operated by me (HetCreep); the reproduction was run on my machine against a clone of the repository. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4541" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `b5d1bbf773bc6f4a9de8f6faec3af1e2b17f1562`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Re-reviewedfbfb58e52and the latest author response. Author now explicitly declares readiness and removed the stale draft note, clearing the sole remaining readiness request. Fixed-width production code is unchanged from the previously qualified implementation; intervening CSV changes originate from main. Reviewed current composed fixed-width/CSV code and test changes, including duplicate-header pass-through and interior blank rows. Clean integration with upstream/main2d3ee51846. 192 tabular/release/version/MCP tests pass with five skips; Ruff check/format and whitespace pass. Occupied-position linear scanning, two-wide gutters, covered multiword cells, and ambiguous-grid pass-through remain intact. No new source blocker found. Existing native extension supports Python startup, not a new native build. Formal approval awaits exact-head hosted qualification; no contributor logic rewritten and nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
3544cf5b88
commit
b80245ea73
@@ -115,14 +115,54 @@ def parse_markdown_table(content: str) -> tuple[list[str], list[list[str]]]:
|
||||
|
||||
|
||||
def parse_fixed_width(content: str) -> tuple[list[str], list[list[str]]]:
|
||||
"""Parse whitespace-aligned columns (best-effort, ≥ 2 spaces as a gap)."""
|
||||
lines = [ln for ln in content.split("\n") if ln.strip()]
|
||||
if len(lines) < 2:
|
||||
"""Parse whitespace-aligned columns by character position.
|
||||
|
||||
A gutter is a run of two or more positions that are blank on every line,
|
||||
so a column is a run of positions where some line has text, and a single
|
||||
space inside it (``First Name``, ``Alice Brown``) stays inside the cell.
|
||||
Splitting on runs of spaces instead would let a narrow gutter merge two
|
||||
cells while a double space inside another cell splits it, leaving the cell
|
||||
count right and the values under the wrong columns. Returns no rows when
|
||||
the text does not fit that grid: a header cell that is empty or holds a 2+
|
||||
space gap means two columns were only one space apart somewhere, so the
|
||||
caller passes the text through.
|
||||
|
||||
The occupied positions are collected from the characters actually present,
|
||||
so the cost is the input length, not rows x longest row.
|
||||
"""
|
||||
lines = [ln.rstrip() for ln in content.split("\n") if ln.strip()]
|
||||
if len(lines) < 2 or any("\t" in ln for ln in lines):
|
||||
return [], []
|
||||
splitter = re.compile(r"\s{2,}")
|
||||
headers = splitter.split(lines[0].strip())
|
||||
rows = [splitter.split(ln.strip()) for ln in lines[1:]]
|
||||
return headers, rows
|
||||
width = max(len(ln) for ln in lines)
|
||||
occupied = bytearray(width)
|
||||
for ln in lines:
|
||||
for i, ch in enumerate(ln):
|
||||
if ch != " ":
|
||||
occupied[i] = 1
|
||||
spans: list[tuple[int, int]] = []
|
||||
start = -1
|
||||
blank_run = 0
|
||||
for i in range(width + 2):
|
||||
filled = i < width and occupied[i]
|
||||
if filled:
|
||||
if start < 0:
|
||||
start = i
|
||||
blank_run = 0
|
||||
continue
|
||||
blank_run += 1
|
||||
if start >= 0 and blank_run == 2:
|
||||
spans.append((start, i - 1))
|
||||
start = -1
|
||||
if start >= 0:
|
||||
spans.append((start, width))
|
||||
|
||||
def cells(ln: str) -> list[str]:
|
||||
return [ln[a:b].strip() for a, b in spans]
|
||||
|
||||
headers = cells(lines[0])
|
||||
if any(not h or " " in h for h in headers):
|
||||
return [], []
|
||||
return headers, [cells(ln) for ln in lines[1:]]
|
||||
|
||||
|
||||
def to_records(headers: list[str], rows: list[list[str]]) -> list[dict[str, str]]:
|
||||
@@ -175,8 +215,7 @@ def parse_tabular(
|
||||
# column of each name, and the others are dropped. Pass through verbatim.
|
||||
if len(set(headers)) != width:
|
||||
return None
|
||||
# The detector finds fixed-width columns by single-space gutters, but the
|
||||
# parser splits on 2+ spaces, so a table like GNU `ls -l` can come back as
|
||||
# A table like GNU `ls -l` has no usable column grid and can come back as
|
||||
# one cell per line. That is not a table; leave it verbatim.
|
||||
if fmt == "fixed_width" and width < 2:
|
||||
return None
|
||||
|
||||
@@ -615,6 +615,69 @@ def test_parse_fixed_width() -> None:
|
||||
assert rows[0] == ["Alice", "30", "NYC"]
|
||||
|
||||
|
||||
def test_parse_fixed_width_never_splits_on_double_spaces_inside_a_cell() -> None:
|
||||
# One-space gutter between USER and COMMAND on the 4444 row, and a double
|
||||
# space inside that row's COMMAND cell. A 2+ space split returned
|
||||
# ['4444', 'sysadmin1 sleep', '60'] (COMMAND=60), still three cells. A
|
||||
# one-space gap cannot be told apart from a space inside a cell, so the
|
||||
# table is not parsed and passes through verbatim.
|
||||
table = (
|
||||
"PID USER COMMAND\n"
|
||||
"1 root /sbin/init\n"
|
||||
"22 www-data nginx\n"
|
||||
"4444 sysadmin1 sleep 60\n"
|
||||
"5 root cron"
|
||||
)
|
||||
assert parse_fixed_width(table) == ([], [])
|
||||
|
||||
|
||||
def test_parse_fixed_width_keeps_an_empty_cell_empty() -> None:
|
||||
# An empty STATUS next to a NOTE with a double space used to read as
|
||||
# STATUS='two', NOTE='words'. Here the other NOTE cells are longer, so the
|
||||
# grid is unambiguous and the empty cell stays empty.
|
||||
table = (
|
||||
"NAME STATUS NOTE\n"
|
||||
"alpha ok fine-and-dandy-things\n"
|
||||
"beta two words\n"
|
||||
"gamma ok fine-and-dandy-things"
|
||||
)
|
||||
headers, rows = parse_fixed_width(table)
|
||||
assert headers == ["NAME", "STATUS", "NOTE"]
|
||||
assert rows[1] == ["beta", "", "two words"]
|
||||
|
||||
|
||||
def test_parse_fixed_width_never_shifts_a_value_when_the_grid_is_ambiguous() -> None:
|
||||
# 'fine' covers one of the two spaces inside 'two words', so no 2-wide
|
||||
# gutter opens there and NOTE keeps the whole value. The one wrong answer
|
||||
# is putting 'two' under STATUS.
|
||||
table = (
|
||||
"NAME STATUS NOTE\nalpha ok fine\nbeta two words\ngamma ok fine"
|
||||
)
|
||||
headers, rows = parse_fixed_width(table)
|
||||
assert headers == ["NAME", "STATUS", "NOTE"]
|
||||
assert rows[1] == ["beta", "", "two words"]
|
||||
|
||||
|
||||
def test_parse_fixed_width_refuses_a_row_that_crosses_a_column_boundary() -> None:
|
||||
table = "NAME STATUS\nalphabetical ok\nbeta ok"
|
||||
assert parse_fixed_width(table) == ([], [])
|
||||
|
||||
|
||||
def test_parse_tabular_fixed_width_control_matches_kubectl_columns(monkeypatch) -> None:
|
||||
import headroom.transforms.tabular_ingest as ti
|
||||
|
||||
monkeypatch.setattr(
|
||||
ti,
|
||||
"detect_content_type",
|
||||
lambda _c: DetectionResult(ContentType.TABULAR, 0.9, {"format": "fixed_width"}),
|
||||
)
|
||||
parsed = ti.parse_tabular(KUBECTL)
|
||||
assert parsed is not None
|
||||
headers, rows, _fmt = parsed
|
||||
assert headers == ["NAME", "READY", "STATUS", "RESTARTS", "AGE"]
|
||||
assert rows[2] == ["worker-5c8d7f9b8-lq2vx", "1/1", "Running", "2", "5h"]
|
||||
|
||||
|
||||
def test_to_records_empty_headers_returns_empty() -> None:
|
||||
assert to_records([], [["a", "b"]]) == []
|
||||
|
||||
@@ -1126,3 +1189,30 @@ def test_load_spreadsheet_missing_file(tmp_path) -> None:
|
||||
|
||||
with pytest.raises(FileNotFoundError):
|
||||
load_spreadsheet(tmp_path / "nope.xlsx")
|
||||
|
||||
|
||||
def test_parse_fixed_width_keeps_multiword_headers_and_values_whole():
|
||||
text = (
|
||||
"First Name AGE CITY\n"
|
||||
"Alice Brown 30 LA\n"
|
||||
"Carol Green 25 SF\n"
|
||||
"David White 40 NY\n"
|
||||
"Frank Young 32 TX"
|
||||
)
|
||||
headers, rows = parse_fixed_width(text)
|
||||
assert headers == ["First Name", "AGE", "CITY"]
|
||||
assert rows[0] == ["Alice Brown", "30", "LA"]
|
||||
|
||||
|
||||
def test_parse_fixed_width_cost_follows_input_length():
|
||||
# Many short rows plus one very long cell: checking every row at every
|
||||
# position of the longest row was ~20M checks for 33 KB of input.
|
||||
import time
|
||||
|
||||
text = "KEY FLAG VALUE\n" + "a b c\n" * 1000 + "a b " + "z" * 20000
|
||||
t0 = time.perf_counter()
|
||||
headers, rows = parse_fixed_width(text)
|
||||
elapsed = time.perf_counter() - t0
|
||||
assert headers == ["KEY", "FLAG", "VALUE"]
|
||||
assert rows[-1][2] == "z" * 20000
|
||||
assert elapsed < 0.05, f"parse took {elapsed:.3f}s for {len(text)} chars"
|
||||
|
||||
Reference in New Issue
Block a user