fix(tabular): pass tables with repeated header names through verbatim (#4278)

## Description

`TabularCompressor` converts each row into a dict keyed by header name.
When a table repeats a header name, the later column overwrites the
earlier one and the earlier column's values are dropped from the
compressed output. This happens with SQL join output (`id,name,id,name`)
and with markdown tables that have two or more blank header cells. The
shorter result passes the `min_savings_chars` check, so the model reads
a table that is missing a column and has values under the wrong labels.

`parse_tabular` already returns `None` for ragged tables so they pass
through verbatim (#1652). This change applies the same rule when header
names are not unique.

Closes #4277

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/tabular_ingest.py`: `parse_tabular` returns
`None` when the header row has duplicate names, so the content passes
through unchanged.
- `tests/test_transforms_tabular.py`: covers a join-style CSV and a
markdown table with blank header cells at the `parse_tabular` level, and
a 40-row CSV through `TabularCompressor.compress`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Without the fix (new tests only):

```text
$ python -m pytest -q tests/test_transforms_tabular.py -k duplicate
FAILED tests/test_transforms_tabular.py::test_parse_tabular_rejects_duplicate_headers[id,name,id,name\n1,alice,10,acme\n...]
FAILED tests/test_transforms_tabular.py::test_parse_tabular_rejects_duplicate_headers[| id | | |\n| --- | --- | --- |\n...]
FAILED tests/test_transforms_tabular.py::test_compress_passes_through_duplicate_headers
3 failed, 73 deselected
```

With the fix:

```text
$ python -m pytest -q tests/test_transforms_tabular.py
76 passed

$ python -m pytest -q <every test file that mentions tabular>  # 14 files
1 failed, 281 passed, 4 skipped
```

The one failure,
`test_garbled_compression_fixes.py::test_banner_survives_with_live_kompress_model`,
needs a live Kompress model and fails the same way on unmodified `main`
in this environment.

```text
$ ruff check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
All checks passed!
$ ruff format --check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
2 files already formatted
```

## Real Behavior Proof

- Environment: Linux aarch64, Python 3.12, `main` at fbf7bc3 plus this
commit, default `ContentRouter` config, pure-Python content detection.
- Exact command / steps:

  ```python
  from headroom.transforms.content_router import ContentRouter

  table = "id,name,id,name\n" + "".join(
      f"{i},user_{i},{100 + i},org_{i % 3}\n" for i in range(40)
  )
  result = ContentRouter().compress(table)
  print("strategy:", result.strategy_used)
  print("user_7 kept:", "user_7" in result.compressed)
  print(result.compressed[:300])
  ```

- Observed result before the fix:

  ```text
  strategy: CompressionStrategy.TABULAR
  user_7 kept: False

"[40]{id:string,name:string}\n100,org_0\n101,org_1\n102,org_2\n103,org_0\n...
  ```

- Observed result after the fix:

  ```text
  strategy: CompressionStrategy.TABULAR
  user_7 kept: True
  id,name,id,name
  0,user_0,100,org_0
  ```

- Not tested: the full proxy path with a live provider, Windows and
macOS, and the Kompress ML model. The change is limited to
`parse_tabular`, which runs before any of those.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Yes, narrowly. Tables with repeated
header names are no longer compressed and pass through unchanged. All
other tables behave as before.
- Kill switch / disable path: `enable_tabular_compressor=False` already
exists; not needed for this change.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: no docs describe tabular header handling, so none
changed.
- `mypy headroom` was not run. The change adds one comparison on
existing `list[str]` values.
- Open PR #3514 also edits `parse_tabular` (multi-table documents). The
two changes are independent; whichever lands second needs a trivial
rebase.
- An alternative is to rename duplicates (`id`, `id_2`) in `to_records`
so these tables still compress. Passing through matches the existing
ragged-table rule and cannot change any value.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4278"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Raphael Fakhri
2026-10-24 08:37:41 -07:00
committed by GitHub
co-authored by JerrettDavis
parent 51e4ae36a5
commit ed3195beef
2 changed files with 28 additions and 0 deletions
+5
View File
@@ -157,6 +157,11 @@ def parse_tabular(
width = len(headers)
if any(len(row) != width for row in rows):
return None
# Repeated header names (a SQL join's `id,name,id,name`, or blank markdown
# header cells) collide as record keys: each row keeps only the last
# column of each name, and the others are dropped. Pass through verbatim.
if len(set(headers)) != width:
return None
# The detector finds fixed-width columns by single-space gutters, but the
# parser splits on 2+ spaces, so a table like GNU `ls -l` can come back as
# one cell per line. That is not a table; leave it verbatim.
+23
View File
@@ -313,6 +313,29 @@ def test_compress_passes_through_ragged_table(monkeypatch) -> None:
assert result.compressed == ragged
_JOIN_CSV = "id,name,id,name\n1,alice,10,acme\n2,bob,20,globex\n3,carol,30,initech\n"
_BLANK_HEADER_MARKDOWN = (
"| id | | |\n| --- | --- | --- |\n| 1 | alice | x |\n| 2 | bob | y |\n| 3 | carol | z |"
)
@pytest.mark.parametrize("content", [_JOIN_CSV, _BLANK_HEADER_MARKDOWN])
def test_parse_tabular_rejects_duplicate_headers(content: str) -> None:
# Records are keyed by header, so a repeated name keeps only its last
# column: `alice` and `bob` would vanish from the compressed table.
assert parse_tabular(content) is None
def test_compress_passes_through_duplicate_headers() -> None:
# Large enough that SmartCrusher would otherwise save bytes on it.
content = "id,name,id,name\n" + "".join(
f"{i},user_{i},{100 + i},org_{i % 3}\n" for i in range(40)
)
result = TabularCompressor().compress(content)
assert not result.was_modified
assert result.compressed == content
def _csv_with_an_oversized_cell() -> str:
# csv.field_size_limit is 128 KB per cell; one pasted document, log excerpt
# or base64 blob in a column goes past it.