fix(tabular): keep interior blank CSV rows instead of dropping them (#4563)

## Description

`parse_csv` (`headroom/transforms/tabular_ingest.py:77`) dropped every
all-blank row, so separator rows inside a table vanished and the row
count changed before the "lossless" csv-schema step, with no marker.
`spreadsheet_ingest._rows_to_csv` trims only trailing blank rows.
`parse_csv` now trims all-blank rows only at the start and the end and
keeps interior ones as empty-cell rows; truly empty lines are still
skipped, so a stray blank line cannot make the table ragged. Item 2 of
#4546.

Refs #4546

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/tabular_ingest.py`: rows with more than one cell
or any non-blank cell are kept; all-blank rows are then trimmed from
both ends in one pass (first and last non-blank index, one slice), and
interior all-blank rows, whitespace-only ones included, are normalized
to empty cells.
- `tests/test_transforms_tabular.py`:
`test_parse_csv_keeps_interior_blank_rows_and_trims_the_ends`,
`test_parse_csv_normalizes_whitespace_only_interior_rows`,
`test_parse_csv_trims_a_large_blank_prefix_in_linear_time`.

Leading blank rows: unchanged. Before this PR they were dropped with
every other blank row and the first non-blank row became the header;
they still are.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# fail before (source at a2d4fe72, new test kept)
1 failed, 101 deselected

# review round: the two new tests against the previous head 5beddc1e
2 failed, 102 deselected

# pass after: tests/test_transforms_tabular.py
87 passed, 17 skipped

# ruff check / ruff format --check: All checks passed! / 2 files already formatted
```

## Real Behavior Proof

- Environment: Ubuntu (WSL2) on Windows 11, Python 3.12.15, `headroom`
Python source at `a2d4fe72` and at this branch, `_core` from the 0.42.0
wheel, offline.
- Exact command / steps: `parse_csv("a,b\n1,2\n,\n3,4\n,\n5,6")` (the
snippet in #4546).
- Observed result: on `a2d4fe72`: 3 data rows
(`[['1','2'],['3','4'],['5','6']]`). On this branch: 5 data rows, the
blanks as `['','']`.
- Leading-blank timing (the review repro, `",\n" * n + "a,b\n1,2\n"`):
on this head 25k/50k/100k/200k rows took 0.018/0.062/0.108/0.193 s (one
run each, same machine; the previous head took 0.41/1.17/5.11/23.46 s
per the review).
- Not tested: the full suite and `mypy`. `parse_csv`'s only caller is
`parse_tabular`; any consumer that relied on interior blank rows
disappearing now sees them as empty-cell rows.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes, narrowly: interior all-blank CSV
rows are now returned as empty-cell rows.
- Kill switch / disable path: None added; revert.
- Unsafe override required: No
- Qualification impact: None expected
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Split out of #4560 (one logical change per PR, as CONTRIBUTING asks).
The three items of #4546 are now #4560 (update check), #4562 (null
thinking block) and #4563 (interior blank CSV rows).

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine against a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
HetCreep
2026-11-04 23:19:13 -06:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent cc1eb5402a
commit 8c360ca69f
2 changed files with 42 additions and 1 deletions
+11 -1
View File
@@ -77,8 +77,18 @@ def parse_csv(content: str, delimiter: str = ",") -> tuple[list[str], list[list[
parsed = [row for row in reader if row and not (len(row) == 1 and not row[0].strip())]
except csv.Error:
return [], []
if not parsed:
# Trim all-blank rows at both ends, but keep interior ones so the row
# count survives, like spreadsheet_ingest._rows_to_csv. One pass and one
# slice: popping index 0 per leading blank row is quadratic. An interior
# all-blank row is normalized to empty cells, so a whitespace-only row and
# an empty row compress the same.
filled = [i for i, row in enumerate(parsed) if any(cell.strip() for cell in row)]
if not filled:
return [], []
parsed = [
row if any(cell.strip() for cell in row) else [""] * len(row)
for row in parsed[filled[0] : filled[-1] + 1]
]
headers = [h.strip() for h in parsed[0]]
return headers, parsed[1:]
+31
View File
@@ -403,6 +403,37 @@ def test_parse_csv_and_records() -> None:
assert records[1] == {"name": "Bob", "age": "25", "city": "LA"}
def test_parse_csv_keeps_interior_blank_rows_and_trims_the_ends() -> None:
headers, rows = parse_csv("a,b\n1,2\n,\n3,4\n,\n5,6\n,\n")
assert headers == ["a", "b"]
assert rows == [["1", "2"], ["", ""], ["3", "4"], ["", ""], ["5", "6"]]
# Empty and whitespace-only lines have no real cells and are still skipped.
assert parse_csv("a,b\n1,2\n\n3,4\n")[1] == [["1", "2"], ["3", "4"]]
assert parse_csv("a,b\n1,2\n \n3,4\n")[1] == [["1", "2"], ["3", "4"]]
def test_parse_csv_normalizes_whitespace_only_interior_rows() -> None:
headers, rows = parse_csv("a,b\n1,2\n , \n3,4\n")
assert rows == [["1", "2"], ["", ""], ["3", "4"]]
def test_parse_csv_trims_a_large_blank_prefix_in_linear_time() -> None:
import time
def run(n: int) -> float:
text = ",\n" * n + "a,b\n1,2\n"
t0 = time.perf_counter()
headers, rows = parse_csv(text)
elapsed = time.perf_counter() - t0
assert headers == ["a", "b"]
assert rows == [["1", "2"]]
return elapsed
small, large = run(25_000), run(200_000)
# 8x the rows: linear stays near 8x; the old pop(0) loop was ~57x.
assert large < max(small, 1e-3) * 25, (small, large)
def test_parse_markdown_table_drops_separator() -> None:
headers, rows = parse_markdown_table(MARKDOWN)
assert headers == ["name", "age"]