mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(tabular): pass tables with repeated header names through verbatim (#4278)
## Description
`TabularCompressor` converts each row into a dict keyed by header name.
When a table repeats a header name, the later column overwrites the
earlier one and the earlier column's values are dropped from the
compressed output. This happens with SQL join output (`id,name,id,name`)
and with markdown tables that have two or more blank header cells. The
shorter result passes the `min_savings_chars` check, so the model reads
a table that is missing a column and has values under the wrong labels.
`parse_tabular` already returns `None` for ragged tables so they pass
through verbatim (#1652). This change applies the same rule when header
names are not unique.
Closes #4277
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/tabular_ingest.py`: `parse_tabular` returns
`None` when the header row has duplicate names, so the content passes
through unchanged.
- `tests/test_transforms_tabular.py`: covers a join-style CSV and a
markdown table with blank header cells at the `parse_tabular` level, and
a 40-row CSV through `TabularCompressor.compress`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
Without the fix (new tests only):
```text
$ python -m pytest -q tests/test_transforms_tabular.py -k duplicate
FAILED tests/test_transforms_tabular.py::test_parse_tabular_rejects_duplicate_headers[id,name,id,name\n1,alice,10,acme\n...]
FAILED tests/test_transforms_tabular.py::test_parse_tabular_rejects_duplicate_headers[| id | | |\n| --- | --- | --- |\n...]
FAILED tests/test_transforms_tabular.py::test_compress_passes_through_duplicate_headers
3 failed, 73 deselected
```
With the fix:
```text
$ python -m pytest -q tests/test_transforms_tabular.py
76 passed
$ python -m pytest -q <every test file that mentions tabular> # 14 files
1 failed, 281 passed, 4 skipped
```
The one failure,
`test_garbled_compression_fixes.py::test_banner_survives_with_live_kompress_model`,
needs a live Kompress model and fails the same way on unmodified `main`
in this environment.
```text
$ ruff check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
All checks passed!
$ ruff format --check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
2 files already formatted
```
## Real Behavior Proof
- Environment: Linux aarch64, Python 3.12, `main` at fbf7bc3 plus this
commit, default `ContentRouter` config, pure-Python content detection.
- Exact command / steps:
```python
from headroom.transforms.content_router import ContentRouter
table = "id,name,id,name\n" + "".join(
f"{i},user_{i},{100 + i},org_{i % 3}\n" for i in range(40)
)
result = ContentRouter().compress(table)
print("strategy:", result.strategy_used)
print("user_7 kept:", "user_7" in result.compressed)
print(result.compressed[:300])
```
- Observed result before the fix:
```text
strategy: CompressionStrategy.TABULAR
user_7 kept: False
"[40]{id:string,name:string}\n100,org_0\n101,org_1\n102,org_2\n103,org_0\n...
```
- Observed result after the fix:
```text
strategy: CompressionStrategy.TABULAR
user_7 kept: True
id,name,id,name
0,user_0,100,org_0
```
- Not tested: the full proxy path with a live provider, Windows and
macOS, and the Kompress ML model. The change is limited to
`parse_tabular`, which runs before any of those.
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Yes, narrowly. Tables with repeated
header names are no longer compressed and pass through unchanged. All
other tables behave as before.
- Kill switch / disable path: `enable_tabular_compressor=False` already
exists; not needed for this change.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- Documentation: no docs describe tabular header handling, so none
changed.
- `mypy headroom` was not run. The change adds one comparison on
existing `list[str]` values.
- Open PR #3514 also edits `parse_tabular` (multi-table documents). The
two changes are independent; whichever lands second needs a trivial
rebase.
- An alternative is to rename duplicates (`id`, `id_2`) in `to_records`
so these tables still compress. Passing through matches the existing
ragged-table rule and cannot change any value.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4278"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
51e4ae36a5
commit
ed3195beef
@@ -157,6 +157,11 @@ def parse_tabular(
|
||||
width = len(headers)
|
||||
if any(len(row) != width for row in rows):
|
||||
return None
|
||||
# Repeated header names (a SQL join's `id,name,id,name`, or blank markdown
|
||||
# header cells) collide as record keys: each row keeps only the last
|
||||
# column of each name, and the others are dropped. Pass through verbatim.
|
||||
if len(set(headers)) != width:
|
||||
return None
|
||||
# The detector finds fixed-width columns by single-space gutters, but the
|
||||
# parser splits on 2+ spaces, so a table like GNU `ls -l` can come back as
|
||||
# one cell per line. That is not a table; leave it verbatim.
|
||||
|
||||
@@ -313,6 +313,29 @@ def test_compress_passes_through_ragged_table(monkeypatch) -> None:
|
||||
assert result.compressed == ragged
|
||||
|
||||
|
||||
_JOIN_CSV = "id,name,id,name\n1,alice,10,acme\n2,bob,20,globex\n3,carol,30,initech\n"
|
||||
_BLANK_HEADER_MARKDOWN = (
|
||||
"| id | | |\n| --- | --- | --- |\n| 1 | alice | x |\n| 2 | bob | y |\n| 3 | carol | z |"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("content", [_JOIN_CSV, _BLANK_HEADER_MARKDOWN])
|
||||
def test_parse_tabular_rejects_duplicate_headers(content: str) -> None:
|
||||
# Records are keyed by header, so a repeated name keeps only its last
|
||||
# column: `alice` and `bob` would vanish from the compressed table.
|
||||
assert parse_tabular(content) is None
|
||||
|
||||
|
||||
def test_compress_passes_through_duplicate_headers() -> None:
|
||||
# Large enough that SmartCrusher would otherwise save bytes on it.
|
||||
content = "id,name,id,name\n" + "".join(
|
||||
f"{i},user_{i},{100 + i},org_{i % 3}\n" for i in range(40)
|
||||
)
|
||||
result = TabularCompressor().compress(content)
|
||||
assert not result.was_modified
|
||||
assert result.compressed == content
|
||||
|
||||
|
||||
def _csv_with_an_oversized_cell() -> str:
|
||||
# csv.field_size_limit is 128 KB per cell; one pasted document, log excerpt
|
||||
# or base64 blob in a column goes past it.
|
||||
|
||||
Reference in New Issue
Block a user