mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(tabular): keep interior blank CSV rows instead of dropping them (#4563)
## Description `parse_csv` (`headroom/transforms/tabular_ingest.py:77`) dropped every all-blank row, so separator rows inside a table vanished and the row count changed before the "lossless" csv-schema step, with no marker. `spreadsheet_ingest._rows_to_csv` trims only trailing blank rows. `parse_csv` now trims all-blank rows only at the start and the end and keeps interior ones as empty-cell rows; truly empty lines are still skipped, so a stray blank line cannot make the table ragged. Item 2 of #4546. Refs #4546 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/tabular_ingest.py`: rows with more than one cell or any non-blank cell are kept; all-blank rows are then trimmed from both ends in one pass (first and last non-blank index, one slice), and interior all-blank rows, whitespace-only ones included, are normalized to empty cells. - `tests/test_transforms_tabular.py`: `test_parse_csv_keeps_interior_blank_rows_and_trims_the_ends`, `test_parse_csv_normalizes_whitespace_only_interior_rows`, `test_parse_csv_trims_a_large_blank_prefix_in_linear_time`. Leading blank rows: unchanged. Before this PR they were dropped with every other blank row and the first non-blank row became the header; they still are. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # fail before (source ata2d4fe72, new test kept) 1 failed, 101 deselected # review round: the two new tests against the previous head5beddc1e2 failed, 102 deselected # pass after: tests/test_transforms_tabular.py 87 passed, 17 skipped # ruff check / ruff format --check: All checks passed! / 2 files already formatted ``` ## Real Behavior Proof - Environment: Ubuntu (WSL2) on Windows 11, Python 3.12.15, `headroom` Python source at `a2d4fe72` and at this branch, `_core` from the 0.42.0 wheel, offline. - Exact command / steps: `parse_csv("a,b\n1,2\n,\n3,4\n,\n5,6")` (the snippet in #4546). - Observed result: on `a2d4fe72`: 3 data rows (`[['1','2'],['3','4'],['5','6']]`). On this branch: 5 data rows, the blanks as `['','']`. - Leading-blank timing (the review repro, `",\n" * n + "a,b\n1,2\n"`): on this head 25k/50k/100k/200k rows took 0.018/0.062/0.108/0.193 s (one run each, same machine; the previous head took 0.41/1.17/5.11/23.46 s per the review). - Not tested: the full suite and `mypy`. `parse_csv`'s only caller is `parse_tabular`; any consumer that relied on interior blank rows disappearing now sees them as empty-cell rows. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A - Stable/default behavior changed: Yes, narrowly: interior all-blank CSV rows are now returned as empty-cell rows. - Kill switch / disable path: None added; revert. - Unsafe override required: No - Qualification impact: None expected - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Split out of #4560 (one logical change per PR, as CONTRIBUTING asks). The three items of #4546 are now #4560 (update check), #4562 (null thinking block) and #4563 (interior blank CSV rows). Produced with AI assistance (Claude Code) operated by me (HetCreep); the reproduction was run on my machine against a clone of the repository. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
cc1eb5402a
commit
8c360ca69f
@@ -77,8 +77,18 @@ def parse_csv(content: str, delimiter: str = ",") -> tuple[list[str], list[list[
|
||||
parsed = [row for row in reader if row and not (len(row) == 1 and not row[0].strip())]
|
||||
except csv.Error:
|
||||
return [], []
|
||||
if not parsed:
|
||||
# Trim all-blank rows at both ends, but keep interior ones so the row
|
||||
# count survives, like spreadsheet_ingest._rows_to_csv. One pass and one
|
||||
# slice: popping index 0 per leading blank row is quadratic. An interior
|
||||
# all-blank row is normalized to empty cells, so a whitespace-only row and
|
||||
# an empty row compress the same.
|
||||
filled = [i for i, row in enumerate(parsed) if any(cell.strip() for cell in row)]
|
||||
if not filled:
|
||||
return [], []
|
||||
parsed = [
|
||||
row if any(cell.strip() for cell in row) else [""] * len(row)
|
||||
for row in parsed[filled[0] : filled[-1] + 1]
|
||||
]
|
||||
headers = [h.strip() for h in parsed[0]]
|
||||
return headers, parsed[1:]
|
||||
|
||||
|
||||
@@ -403,6 +403,37 @@ def test_parse_csv_and_records() -> None:
|
||||
assert records[1] == {"name": "Bob", "age": "25", "city": "LA"}
|
||||
|
||||
|
||||
def test_parse_csv_keeps_interior_blank_rows_and_trims_the_ends() -> None:
|
||||
headers, rows = parse_csv("a,b\n1,2\n,\n3,4\n,\n5,6\n,\n")
|
||||
assert headers == ["a", "b"]
|
||||
assert rows == [["1", "2"], ["", ""], ["3", "4"], ["", ""], ["5", "6"]]
|
||||
# Empty and whitespace-only lines have no real cells and are still skipped.
|
||||
assert parse_csv("a,b\n1,2\n\n3,4\n")[1] == [["1", "2"], ["3", "4"]]
|
||||
assert parse_csv("a,b\n1,2\n \n3,4\n")[1] == [["1", "2"], ["3", "4"]]
|
||||
|
||||
|
||||
def test_parse_csv_normalizes_whitespace_only_interior_rows() -> None:
|
||||
headers, rows = parse_csv("a,b\n1,2\n , \n3,4\n")
|
||||
assert rows == [["1", "2"], ["", ""], ["3", "4"]]
|
||||
|
||||
|
||||
def test_parse_csv_trims_a_large_blank_prefix_in_linear_time() -> None:
|
||||
import time
|
||||
|
||||
def run(n: int) -> float:
|
||||
text = ",\n" * n + "a,b\n1,2\n"
|
||||
t0 = time.perf_counter()
|
||||
headers, rows = parse_csv(text)
|
||||
elapsed = time.perf_counter() - t0
|
||||
assert headers == ["a", "b"]
|
||||
assert rows == [["1", "2"]]
|
||||
return elapsed
|
||||
|
||||
small, large = run(25_000), run(200_000)
|
||||
# 8x the rows: linear stays near 8x; the old pop(0) loop was ~57x.
|
||||
assert large < max(small, 1e-3) * 25, (small, large)
|
||||
|
||||
|
||||
def test_parse_markdown_table_drops_separator() -> None:
|
||||
headers, rows = parse_markdown_table(MARKDOWN)
|
||||
assert headers == ["name", "age"]
|
||||
|
||||
Reference in New Issue
Block a user