Commit Graph
15 Commits
Author SHA1 Message Date
b80245ea73 fix(tabular): split fixed-width rows on column boundaries, not on double spaces (#4541)
## Description

`parse_fixed_width` split every row on runs of 2+ spaces, while the
detector finds fixed-width columns by character position. When the two
disagree inside one row, the cell count can still match the header, the
ragged-row guard in `parse_tabular` passes, and a value is stored under
the wrong column. The compressed output (`csv-schema`) carries no CCR
marker, so the model cannot tell. See the issue for the exact input.

This change cuts each row at the character positions that are blank on
every line. If a row crosses a column boundary (a header cell comes out
empty or holds a 2+ space gap), the table is not parsed and passes
through verbatim.

Fixes #4540

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/tabular_ingest.py`: `parse_fixed_width` now finds
column spans from positions that are blank on every line, slices each
row by those spans, and returns `([], [])` when the text has a tab, or
when a header cell is empty or contains a 2+ space gap. A stale comment
in `parse_tabular` about the parser splitting on 2+ spaces was updated.
- `tests/test_transforms_tabular.py`: four tests for `parse_fixed_width`
(the PID/USER/COMMAND shape, an empty cell next to a double-spaced cell,
an ambiguous grid that must refuse, a row crossing a boundary) and one
kubectl control through `parse_tabular`.

Behaviour change to be aware of: a row with an empty cell is now parsed
with `""` in that column when the grid is unambiguous (before, it
counted as ragged and the table passed through). A table where one row
does not line up with the others is still not parsed. Output for aligned
tables (kubectl, `ps aux`, `df -h` fixtures) is unchanged.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

With the fix reverted (source file only, new tests kept), run from a
copy of the tree with the 0.42.0 `_core.abi3.so`:

```text
FAILED tests/test_transforms_tabular.py::test_parse_fixed_width_splits_on_column_positions_not_double_spaces
FAILED tests/test_transforms_tabular.py::test_parse_fixed_width_keeps_an_empty_cell_empty
FAILED tests/test_transforms_tabular.py::test_parse_fixed_width_never_shifts_a_value_when_the_grid_is_ambiguous
FAILED tests/test_transforms_tabular.py::test_parse_fixed_width_refuses_a_row_that_crosses_a_column_boundary
=================== 4 failed, 85 passed, 17 skipped in 2.40s ===================
```

With the fix, three files:

```text
tests/test_transforms_tabular.py ....................................... [ 18%]
................................................sssssssssssssssss..      [ 49%]
tests/test_transforms_content_detection.py ....................          [ 58%]
tests/test_transforms_content_router.py ................................ [ 73%]
..................................................ss....                 [100%]

======================= 195 passed, 19 skipped in 5.54s ========================
```

```text
$ ruff check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
All checks passed!
$ ruff format --check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
2 files already formatted
```

The skipped tests are the spreadsheet loader tests; they were skipped on
`main` too.

## Real Behavior Proof

- Environment: Ubuntu under WSL2 on Windows 11, Python 3.12.15, Python
source of this branch on top of `main` `a2d4fe72`, `_core.abi3.so`
copied from the `headroom_ai` 0.42.0 wheel into a copy of the tree (not
into the branch), `HEADROOM_OFFLINE=1`.
- Exact command / steps: ran `TabularCompressor().compress(text)` and
`parse_tabular(text)` on the table from the issue (`PID USER COMMAND`
header, rows `1 root /sbin/init`, `22 www-data nginx`, `4444 sysadmin1
sleep 60`, `5 root cron`, plus 40 rows `"%-5d root worker%d" % (100+i,
i)`), once on `main` and once on this branch.
- Observed result: on `main`, row 4444 parsed as `['4444', 'sysadmin1
sleep', '60']` and the output line was `60,4444,sysadmin1 sleep` under
`{COMMAND,PID,USER}`. On this branch it parses as `['4444', 'sysadmin1',
'sleep 60']` and the output line is `sleep 60,4444,sysadmin1`. For the
empty-cell shape (`NAME STATUS NOTE`, row `beta two words`) `main`
returned `['beta', 'two', 'words']`; this branch returns `None`
(passthrough). kubectl and `df -h` fixtures parse to the same columns as
before.
- Not tested: Windows or macOS runs; the full test suite (I ran the
three tabular/detection/router test files); `mypy`; tables with wide
(CJK) characters, where character positions and display columns differ,
so such a table may be refused or grouped by code point; very large
outputs for speed (the gutter scan is one pass over each line per column
position, bounded by the longest line).

## Runtime Rollout Safety

- Rollout-managed feature(s): None. This changes a parser used by the
tabular compressor; no rollout-managed feature is touched.
- Minimum rollout channel: N/A (not rollout-managed)
- Stable/default behavior changed: Yes, narrowly. Fixed-width tables
that did not line up used to be compressed with shifted values and now
pass through; tables with an empty cell now parse with `""` instead of
passing through as ragged. Aligned tables are unchanged.
- Kill switch / disable path: the existing option that turns the tabular
compressor off in the content router; no new switch.
- Unsafe override required: No
- Qualification impact: None expected; not checked against the
qualification suite.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Mypy was not run (marked unchecked above). Documentation: none of the
docs I know of describe the fixed-width splitting rule, so I did not
change any (not searched exhaustively).
- The ambiguous-grid case is a tradeoff: when every cell in a column is
shorter than one double-spaced cell, the gap inside that cell looks like
a column gutter. The parser refuses rather than guess.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine against a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4541"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->


### Maintainer current-main requalification

Exact head `b5d1bbf773bc6f4a9de8f6faec3af1e2b17f1562`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Re-reviewed fbfb58e52 and the latest author response. Author now
explicitly declares readiness and removed the stale draft note, clearing
the sole remaining readiness request. Fixed-width production code is
unchanged from the previously qualified implementation; intervening CSV
changes originate from main. Reviewed current composed fixed-width/CSV
code and test changes, including duplicate-header pass-through and
interior blank rows.

Clean integration with upstream/main
2d3ee51846. 192
tabular/release/version/MCP tests pass with five skips; Ruff
check/format and whitespace pass. Occupied-position linear scanning,
two-wide gutters, covered multiword cells, and ambiguous-grid
pass-through remain intact. No new source blocker found. Existing native
extension supports Python startup, not a new native build. Formal
approval awaits exact-head hosted qualification; no contributor logic
rewritten and nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-05 20:13:40 -06:00
8c360ca69f fix(tabular): keep interior blank CSV rows instead of dropping them (#4563)
## Description

`parse_csv` (`headroom/transforms/tabular_ingest.py:77`) dropped every
all-blank row, so separator rows inside a table vanished and the row
count changed before the "lossless" csv-schema step, with no marker.
`spreadsheet_ingest._rows_to_csv` trims only trailing blank rows.
`parse_csv` now trims all-blank rows only at the start and the end and
keeps interior ones as empty-cell rows; truly empty lines are still
skipped, so a stray blank line cannot make the table ragged. Item 2 of
#4546.

Refs #4546

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/tabular_ingest.py`: rows with more than one cell
or any non-blank cell are kept; all-blank rows are then trimmed from
both ends in one pass (first and last non-blank index, one slice), and
interior all-blank rows, whitespace-only ones included, are normalized
to empty cells.
- `tests/test_transforms_tabular.py`:
`test_parse_csv_keeps_interior_blank_rows_and_trims_the_ends`,
`test_parse_csv_normalizes_whitespace_only_interior_rows`,
`test_parse_csv_trims_a_large_blank_prefix_in_linear_time`.

Leading blank rows: unchanged. Before this PR they were dropped with
every other blank row and the first non-blank row became the header;
they still are.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# fail before (source at a2d4fe72, new test kept)
1 failed, 101 deselected

# review round: the two new tests against the previous head 5beddc1e
2 failed, 102 deselected

# pass after: tests/test_transforms_tabular.py
87 passed, 17 skipped

# ruff check / ruff format --check: All checks passed! / 2 files already formatted
```

## Real Behavior Proof

- Environment: Ubuntu (WSL2) on Windows 11, Python 3.12.15, `headroom`
Python source at `a2d4fe72` and at this branch, `_core` from the 0.42.0
wheel, offline.
- Exact command / steps: `parse_csv("a,b\n1,2\n,\n3,4\n,\n5,6")` (the
snippet in #4546).
- Observed result: on `a2d4fe72`: 3 data rows
(`[['1','2'],['3','4'],['5','6']]`). On this branch: 5 data rows, the
blanks as `['','']`.
- Leading-blank timing (the review repro, `",\n" * n + "a,b\n1,2\n"`):
on this head 25k/50k/100k/200k rows took 0.018/0.062/0.108/0.193 s (one
run each, same machine; the previous head took 0.41/1.17/5.11/23.46 s
per the review).
- Not tested: the full suite and `mypy`. `parse_csv`'s only caller is
`parse_tabular`; any consumer that relied on interior blank rows
disappearing now sees them as empty-cell rows.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes, narrowly: interior all-blank CSV
rows are now returned as empty-cell rows.
- Kill switch / disable path: None added; revert.
- Unsafe override required: No
- Qualification impact: None expected
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Split out of #4560 (one logical change per PR, as CONTRIBUTING asks).
The three items of #4546 are now #4560 (update check), #4562 (null
thinking block) and #4563 (interior blank CSV rows).

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine against a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 23:19:13 -06:00
Vaibhav HajareandJerrettDavis ddbbb07746 fix(tabular): keep tables whose columns share a name (#4329)
## Description

`to_records` zips the headers with each row into a dict, so two columns
with the same name make the later one overwrite the earlier one:

```python
records.append({headers[i]: padded[i] for i in range(width)})
```

Every value of the earlier column is lost, and the remaining values are
relabelled as if they were the dropped ones. `parse_tabular` checked
that every row has as many cells as the header, but not that the header
names are unique, so such tables reached SmartCrusher. The compressed
table is shorter than the original, so the `min_savings_chars` check
accepts it.

Repeated names are common in tool output:

- SQL join results (`psql --csv`, sqlite `.mode csv`) repeat `id`,
`name`, `created_at`.
- Markdown tables with two or more blank header cells collapse to one
`""` key.

For the reproduction in the issue, the first `id` and `name` columns
(`0..39`, `user_0..user_39`) were gone, and `100,org_0` was presented as
the `id,name` of row 0.

Closes #4277

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `parse_tabular` now rejects a table whose header names are not unique,
alongside the existing ragged-row check. A table that cannot be turned
into records without losing columns passes through unchanged, as a
ragged table already does (#1652).

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_transforms_tabular.py -q -k duplicate -v
collected 77 items / 73 deselected / 4 selected

tests/test_transforms_tabular.py::test_parse_tabular_rejects_duplicate_headers PASSED [ 25%]
tests/test_transforms_tabular.py::test_parse_tabular_rejects_duplicate_blank_markdown_headers PASSED [ 50%]
tests/test_transforms_tabular.py::test_compress_passes_through_a_table_with_duplicate_headers PASSED [ 75%]
tests/test_transforms_tabular.py::test_router_keeps_every_column_of_a_table_with_duplicate_headers PASSED [100%]

================ 4 passed, 73 deselected, 1 warning in 1.43s =================
```

The whole file reports `2 failed, 58 passed, 17 skipped`; both failures
(`test_verbose_markdown_compresses`,
`test_compact_unique_csv_passes_through`) reproduce on the unmodified
tree — they need `httpx`, which is not installed here.

The wider transform/router suites report the same failure count before
and after the change (`36 failed, 111 passed` → `36 failed, 115 passed`;
the four extra passes are the new tests).

## Real Behavior Proof

- Environment: headroom source checkout on Windows, Python 3.12; no
third-party deps installed
- Exact command / steps: the new tests build the table from the issue's
reproduction (`id,name,id,name` plus 40 rows) and run it through
`parse_tabular`, `TabularCompressor` and `ContentRouter`
- Observed result: before the fix the parser returned the table and the
records lost the first two columns; after the fix the table is not
tabular, so it passes through unchanged with every column intact
- Not tested: the SmartCrusher path itself (no `httpx` here), so the
end-to-end router test could not be shown to fail on the unmodified tree
in this environment

Three of the four new tests fail on the unmodified tree:

```text
$ pytest tests/test_transforms_tabular.py -q -k duplicate   # fix reverted
FAILED ...::test_parse_tabular_rejects_duplicate_headers
FAILED ...::test_parse_tabular_rejects_duplicate_blank_markdown_headers
FAILED ...::test_compress_passes_through_a_table_with_duplicate_headers
3 failed, 1 passed, 73 deselected
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: a table with repeated column names is
no longer compressed; it passes through unchanged instead of being
compressed with a column silently dropped
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- The router test is the one from the issue's reproduction. In this
environment it passes on the unmodified tree as well, because `httpx` is
missing and the SmartCrusher import fails before the lossy zip — so it
only bites where SmartCrusher is available. The other three tests cover
the loss directly and do fail without the fix.
- `to_records` itself is unchanged: it stays the raw zipper it is
documented to be, and the guard sits in `parse_tabular`, the gate every
tabular payload passes through.
- `opentelemetry` is not installed on this machine, so `import headroom`
fails and the suite cannot be collected as-is. The test runs above used
a minimal local stub for the `opentelemetry` API surface that
`headroom/observability` imports; no product code was changed to
accommodate it, and the stub is not part of this PR.
- `ruff` and `mypy` were not run: neither is installed here.

## Model Used

MiniMax M3.1 Flash Free via the opencode agent harness
(`minimax-m3.1-flash-free`)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4329"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-04 22:52:49 -06:00
5cebb90db1 fix(tabular): keep empty records and separator-like Markdown body rows (#4582)
## Description

Tabular ingest silently dropped real data rows: a CSV/TSV record whose
fields are all empty (`,` or `"",""`) and any Markdown body row whose
values look like a separator (`| --- | --- |`). Records disappeared with
no marker.

Closes #4279

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `parse_csv`: keep every record; skip only truly blank lines (empty row
or a single whitespace-only cell).
- `parse_markdown_table`: drop only the header separator directly under
the header row; later separator-like rows are data.
- Regression tests for CSV, TSV, quoted empty CSV record and Markdown
`---` body row.

## Testing

- [x] Unit tests added/updated

```
$ python -m pytest tests/test_transforms_tabular.py -q
2 failed, 98 passed, 5 skipped
# the 2 failures (test_verbose_markdown_compresses, test_compact_unique_csv_passes_through) fail identically on main in this env; on main the new tests also fail (6 failed)
$ ruff check / ruff format --check (touched files)
All checks passed! / 2 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11, Python 3, local checkout rebased on
upstream/main
- Exact command / steps: `parse_csv("a,b\n1,2\n,\n3,4")` and
`parse_markdown_table("| a | b |\n|---|---|\n| 1 | 2 |\n| --- | --- |")`
via the new tests
- Observed result: CSV returns 3 rows including `['', '']`; Markdown
returns `[['1','2'],['---','---']]`
- Not tested: end-to-end proxy compression on real tabular payloads

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: yes, tabular parsers now keep rows
they previously dropped
- Kill switch / disable path: none needed; parser correctness fix
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 08:36:43 -06:00
Ayush Kumar JhaandJD Davis b66c9c7db6 fix(transforms): stop an embedded markdown table from dropping the rest of a tool result (#4216)
## Description

A tool result that contains a markdown table is rewritten as JSON of the
table's rows. Everything else in the result is dropped without a marker:
headings, prose, and the output of other commands in the same Bash call.
A later line with pipes is parsed as a row too.

Root cause: `_try_detect_markdown_table` claims the whole payload as
`TABULAR` (0.95) as soon as a header and separator pair appears anywhere
in its first 50 lines. The tabular compressor then parses it with
`parse_markdown_table`, which keeps only lines containing `|`. Its
savings check compares the JSON against the full original text, so
dropping the prose counts as compression. Markdown tables are not
detected by the Rust side, which only returns `Tabular` for git status.
On the native backend, plain text falls back to this same Python
detector, and `parse_tabular` re-runs it. So this function decides the
route on both backends.

The fix treats content as a markdown table only when it is one: the
header and separator open it and every remaining line is a row. A row is
written in the header's leading-pipe style, is neither another separator
nor a list item, and splits into exactly the header's cells the way the
tabular parser splits it, so the records built from it can't truncate or
pad it. A table with an escaped pipe (`\|`) is left alone, since the
parser splits on it. A blank line ends a markdown table, so anything
after one also disqualifies it. A table with anything else around it is
no longer `TABULAR` and falls through to the remaining detectors, which
is the issue's first expected behavior. CSV/TSV detection already
requires 85% of the sampled lines to share a column count, so this
brings markdown into line with it. Repeated headers, which records can't
hold either, are handled on `main` by #4278's check in `parse_tabular`.
This PR's regression test covers that check for markdown and
whitespace-normalized collisions.

Closes #4215

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/content_detector.py`:
- `_try_detect_tabular` keeps a markdown verdict only when the new
`_is_whole_markdown_table(content)` holds. That helper checks the whole
content, not the 50-line sample, so a summary line after a long table
also counts.
- Each row must have the header's cell count (`_md_cell_count`, the same
split as `parse_markdown_table`), and content with an escaped pipe is
rejected.
- A rejected markdown candidate still goes through CSV/TSV detection, so
CSV that holds markdown in a quoted field stays tabular. A `|` delimiter
is not accepted on that path, since those pipes belong to the markdown.
- `headroom/transforms/tabular_ingest.py`: unchanged. During review I
added a repeated-header check there, but #4278 landed the same check on
`main` first, so I dropped mine after merging `main`.
- `tests/test_transforms_tabular.py`:
- detection is no longer `TABULAR` for ten shapes: the reporter's
document, a heading line, a pipe line after the blank that ends the
table, a pipe line above the table, text above a separator row, a line
after a 60-row table, a list item straight after the rows, a note
without the leading pipe, a list item after a pipe-less table, and a
second separator row;
- a whole table is still `TABULAR` with a trailing newline, surrounding
blank lines, CRLF, or no outer pipes;
- CSV with a markdown table inside a quoted field is still `TABULAR`
with `,` and parses to 21 rows;
- `_is_whole_markdown_table` needs a header of at least two cells
followed by a separator;
- `TabularCompressor` returns the reporter's document unchanged, and the
router no longer picks `TABULAR` for it;
- end to end, a 40-row table followed by an over-width row (with and
without outer pipes) or by a row with an escaped pipe is not claimed as
markdown and comes back from `TabularCompressor` unchanged;
- end to end, a 40-row table with repeated headers (`| Name | Name |`,
`| Name | Name |`, `name,name`, `name, name `) is not parsed into
records and comes back from `TabularCompressor` unchanged (it exercises
#4278's check and fails without it).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`) - run on the changed files
- [x] Type checking passes (`mypy headroom`) - run on the changed module
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# the first commit's tests against main's content_detector.py
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[doc_with_table]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[heading_line]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[pipe_line_after_table]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[pipe_line_before_table]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[text_above_a_separator_row]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[line_after_a_long_table]
FAILED tests/test_transforms_tabular.py::test_compress_keeps_a_document_with_a_table_whole
FAILED tests/test_transforms_tabular.py::test_router_does_not_route_a_document_with_a_table_to_tabular
8 failed, 3 passed, 73 deselected
# (the 3 that pass are the whole-table cases, which must hold both ways)

# the cases added for the review (second commit), against the first commit's detector (22f4e1b6)
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[list_item_right_after_rows]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[note_without_leading_pipe]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[list_item_after_pipeless_rows]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[second_separator_row]
FAILED tests/test_transforms_tabular.py::test_csv_with_markdown_in_a_quoted_field_is_still_csv
5 failed, 12 passed, 73 deselected
# (against main the file no longer collects, since it imports the new _is_whole_markdown_table)

# the review's over-width rows (third commit), against 1fa07094
FAILED tests/test_transforms_tabular.py::test_table_with_a_row_records_cannot_hold_is_kept_whole[over_width_row_pipeless]
FAILED tests/test_transforms_tabular.py::test_table_with_a_row_records_cannot_hold_is_kept_whole[over_width_row_piped]
FAILED tests/test_transforms_tabular.py::test_table_with_a_row_records_cannot_hold_is_kept_whole[escaped_pipe]
3 failed, 1 passed, 90 deselected

# the review's repeated headers (fourth commit), against 477e9cca
FAILED tests/test_transforms_tabular.py::test_table_with_repeated_headers_is_kept_whole[markdown]
FAILED tests/test_transforms_tabular.py::test_table_with_repeated_headers_is_kept_whole[markdown_whitespace]
FAILED tests/test_transforms_tabular.py::test_table_with_repeated_headers_is_kept_whole[csv]
FAILED tests/test_transforms_tabular.py::test_table_with_repeated_headers_is_kept_whole[csv_whitespace]
4 failed, 94 deselected
# with main merged in (ed0f4531), the same four pass on #4278's check alone, and fail again if that check is removed

# at ed0f4531
tests/test_transforms_tabular.py: 84 passed, 17 skipped

# mutations of the fix: each must fail at least one test
M1 no whole-table check                    KILLED (16)
M2 any row instead of all rows             KILLED (8)
M3 drop separator-at-line-2                KILLED (1)
M4 drop two-column header                  KILLED (1)
M5 check only the first 50 lines           KILLED (1)
M6 ignore leading-pipe style               KILLED (1)
M7 allow a second separator row            KILLED (1)
M8 allow list items                        KILLED (1)
M9 no delimited fallback                   KILLED (1)
M10 accept pipe as delimiter               KILLED (8)
M11 any width instead of the header's      KILLED (2)
M12 allow escaped pipes                    KILLED (1)

# every test file that touches detection, tabular parsing, mixed-content splitting or the router (37 files)
main 3fef15d3:  6 failed, 1011 passed, 23 skipped
ed0f4531:       5 failed, 1037 passed, 23 skipped

ruff 0.17.0 check / format --check (changed files): All checks passed! / 3 files already formatted
mypy 1.19.1 headroom/transforms/content_detector.py: no errors in the changed module
```

The 5 failures are the same tests with and without this change:
`test_offline_egress_chokepoint.py` OTLP/Langfuse cases, which fail on
my machine either way. The sixth failure on `main` was
`test_content_router_ml_stage_deadline.py::test_saturated_remote_calls_fail_open_without_queueing_their_payloads`,
which passed 4 of 4 times on its own, so it's a timing flake under
parallel load. The 7 mypy errors are the Windows-only `os.fchmod` /
`os.O_*` ones in `fileperms.py` and `astgrep.py`, identical on `main`.

## Real Behavior Proof

- Environment: Windows 11, Python 3.11.9, headroom from this branch
(ed0f4531) and from `main` (76133cff, 0.41.1-dev), pure-Python detection
backend (the Windows default).
- Exact command / steps: the reporter's script from #4215, unchanged
(`seed_proxy_env_defaults()`, `create_app(_proxy_config_from_env())`,
then `proxy.anthropic_pipeline.apply(...)` on a Bash `tool_result`
holding a heading, prose, a table, a pipe-bearing bullet and another
command's output), run with `HEADROOM_OFFLINE=1 PYTHONPATH=.`.
- Observed result: on `main` the script prints the first block below, so
the whole tool result became the table's rows, including the bullet.
With this branch it prints the second block, followed by the original
text unchanged.
- Not tested: the native (Rust/Magika) detection backend locally; it
reaches this function through the plain-text fallback, and the router
test pins the Python backend. Linux/macOS, and real Claude Code traffic.

```text
# main (76133cff)
transforms: ['router:tool_result:tabular']
chars: 2318 -> 344
prose kept: False | other output kept: False
[{"Step":"Build","Owner":"CI","Command":"`make build`"},{"Step":"Test","Owner":"CI","Command":"`make test`"},{"Step":"Tag","Owner":"Maintainer","Command":"`git tag vX.Y.Z`"},{"Step":"Publish","Owner":"Maintainer","Command":"`make publish`"},{"Step":"- Create an issue: `tracker new --type=bug","Owner":"feature","Command":"task --priority=2`"}]

# this branch
transforms: ['router:noop']
chars: 2318 -> 2318
prose kept: True | other output kept: True
# Release checklist
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None
- Minimum rollout channel: N/A
- Stable/default behavior changed: Narrowly. Content that holds a
markdown table plus anything else is no longer routed to the tabular
compressor. A bare table routes and compresses exactly as before
(measured: `tabular`, 2019 -> 1580 chars on both `main` and this
branch).
- Kill switch / disable path: None added. `--compressor` /
`HEADROOM_COMPRESSORS` without `tabular` still disables the compressor.
- Unsafe override required: No
- Qualification impact: None
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A: the
docs list "markdown tables" as tabular, which still holds for content
that is a table)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- One trade-off to flag: a table wrapped in a
`<returncode>0</returncode>` + `<output>` envelope. The router detects
on the unwrapped body, but the compressor receives the wrapped text. On
`main` it was compressed (2065 -> 1580 chars) and the envelope,
including the return code, was dropped the same silent way. With this
change it passes through verbatim (2065 -> 2065). Compressing the body
inside the envelope would recover those savings; I left that out to keep
this fix to the detector.
- Also lossless in the same direction, from reading
`_RELEASABLE_READ_TYPES` rather than a separate test: a `Read` of a
markdown file that has a table plus prose is no longer a releasable
`TABULAR` read, so it stays protected like other plain text.
- Relation to #3514: that PR takes the richer route for the same
underlying line-drop in `parse_markdown_table`, splitting a document
into table and prose segments so each table is compacted and the text is
carried through. It is still in changes-requested. This PR is the small
lossless fix for #4215 in the meantime and doesn't change
`tabular_ingest.py`. If segmenting lands, `_is_whole_markdown_table` is
the one gate to relax so documents with tables reach it again.
- Out of scope: compacting only the table region and keeping the
surrounding lines (the issue's second option), which is what #3514 does.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4216"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-28 17:56:27 -05:00
Raphael FakhriandJerrettDavis ed3195beef fix(tabular): pass tables with repeated header names through verbatim (#4278)
## Description

`TabularCompressor` converts each row into a dict keyed by header name.
When a table repeats a header name, the later column overwrites the
earlier one and the earlier column's values are dropped from the
compressed output. This happens with SQL join output (`id,name,id,name`)
and with markdown tables that have two or more blank header cells. The
shorter result passes the `min_savings_chars` check, so the model reads
a table that is missing a column and has values under the wrong labels.

`parse_tabular` already returns `None` for ragged tables so they pass
through verbatim (#1652). This change applies the same rule when header
names are not unique.

Closes #4277

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/tabular_ingest.py`: `parse_tabular` returns
`None` when the header row has duplicate names, so the content passes
through unchanged.
- `tests/test_transforms_tabular.py`: covers a join-style CSV and a
markdown table with blank header cells at the `parse_tabular` level, and
a 40-row CSV through `TabularCompressor.compress`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Without the fix (new tests only):

```text
$ python -m pytest -q tests/test_transforms_tabular.py -k duplicate
FAILED tests/test_transforms_tabular.py::test_parse_tabular_rejects_duplicate_headers[id,name,id,name\n1,alice,10,acme\n...]
FAILED tests/test_transforms_tabular.py::test_parse_tabular_rejects_duplicate_headers[| id | | |\n| --- | --- | --- |\n...]
FAILED tests/test_transforms_tabular.py::test_compress_passes_through_duplicate_headers
3 failed, 73 deselected
```

With the fix:

```text
$ python -m pytest -q tests/test_transforms_tabular.py
76 passed

$ python -m pytest -q <every test file that mentions tabular>  # 14 files
1 failed, 281 passed, 4 skipped
```

The one failure,
`test_garbled_compression_fixes.py::test_banner_survives_with_live_kompress_model`,
needs a live Kompress model and fails the same way on unmodified `main`
in this environment.

```text
$ ruff check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
All checks passed!
$ ruff format --check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
2 files already formatted
```

## Real Behavior Proof

- Environment: Linux aarch64, Python 3.12, `main` at fbf7bc3 plus this
commit, default `ContentRouter` config, pure-Python content detection.
- Exact command / steps:

  ```python
  from headroom.transforms.content_router import ContentRouter

  table = "id,name,id,name\n" + "".join(
      f"{i},user_{i},{100 + i},org_{i % 3}\n" for i in range(40)
  )
  result = ContentRouter().compress(table)
  print("strategy:", result.strategy_used)
  print("user_7 kept:", "user_7" in result.compressed)
  print(result.compressed[:300])
  ```

- Observed result before the fix:

  ```text
  strategy: CompressionStrategy.TABULAR
  user_7 kept: False

"[40]{id:string,name:string}\n100,org_0\n101,org_1\n102,org_2\n103,org_0\n...
  ```

- Observed result after the fix:

  ```text
  strategy: CompressionStrategy.TABULAR
  user_7 kept: True
  id,name,id,name
  0,user_0,100,org_0
  ```

- Not tested: the full proxy path with a live provider, Windows and
macOS, and the Kompress ML model. The change is limited to
`parse_tabular`, which runs before any of those.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Yes, narrowly. Tables with repeated
header names are no longer compressed and pass through unchanged. All
other tables behave as before.
- Kill switch / disable path: `enable_tabular_compressor=False` already
exists; not needed for this change.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: no docs describe tabular header handling, so none
changed.
- `mypy headroom` was not run. The change adds one comparison on
existing `list[str]` values.
- Open PR #3514 also edits `parse_tabular` (multi-table documents). The
two changes are independent; whichever lands second needs a trivial
rebase.
- An alternative is to rename duplicates (`id`, `id_2`) in `to_records`
so these tables still compress. Passing through matches the existing
ragged-table rule and cannot change any value.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4278"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:37:41 -07:00
Mansurshakh JaparovandJD Davis a42fb2b9a1 fix(router): preserve human-readable Git status losslessly (#4020)
## Description

Addresses the still-reproducible Git status corruption in #3652. The
aligned `ls`/`ps`/`df` portions are already protected in current main;
this PR does not claim them as new fixes. Human-readable Git status is
still classified as prose and can lose staged/unstaged/untracked paths
through Kompress.

This is a narrow content-preservation bug fix, not the separate policy
proposal to exclude all `Bash` outputs by default.

## Type of Change

- [x] Bug fix
- [x] Documentation update

## Changes Made

- Recognize the standard English branch/detached-HEAD header plus Git
status sections/entries as structured tabular content.
- Preserve this non-rectangular `git_status` format verbatim instead of
forcing it into CSV or a prose model.
- Keep Python detection and the native Rust content/live-zone route
aligned; Rust detection uses iterator-based scans without
collection/cloning of the input.
- Leave ordinary prose, bullet lists, and other Bash output on their
existing routes.
- Guard Git status before native model classification and before
external CSV/wildcard compressor dispatch; retain the locked native
Magika production chain.
- Replace the stale `PLAIN_TEXT` status expectation with
`format=git_status` and `parse_tabular(status) is None`; keep
prose/git-log negative cases.
- Document the English-format scope and lossless/no-savings tradeoff.

## Testing

- [x] Unit tests pass (`pytest`), including the consumer-visible
lossless regression.
- [x] Native regression tests, changed-file lint, configured pre-commit
mypy and manual production runtime checks pass.

### Test Output

```text
Review correction:
Python router/external-dispatch regressions: 121 passed, 2 skipped
Existing tabular parser/detection suite: 73 passed in 7.76s
Native Git-status regressions: 3 passed
Final mixed-content correction (router, external dispatch, tabular suites):
184 passed, 2 skipped in 52.14s
```

The mandatory pre-push `make ci-precheck` passed: workspace Rust
fmt/clippy/tests, native extension build/import verification, the
complete configured Python gate, and Commitlint. The native gate used
the explicitly installed CPU `libonnxruntime.so.1.26.0` via
`ORT_DYLIB_PATH`, with cached-model assertions enabled.

```text
[build_rust_extension] headroom._core build + install + verify: OK
================== 177 passed, 4 skipped, 1 warning in 27.72s ==================
✅ ci-precheck PASSED — safe to push.
```

```sh
python -m pytest -q tests/test_transforms_content_router.py
cargo test -p headroom-core git_status
cargo test -p headroom-core ordinary_bullet
```

Python content-router suite: **84 passed, 2 skipped**, with an existing
Hugging Face deprecation warning. Native focused regressions: **2 Git
status tests passed**, **1 ordinary-bullet negative passed**. The native
extension was rebuilt and the build/import verification passed. The
final deterministic consumer regression uses a ready lossy prose
compressor so a cold model cannot accidentally mask missing protection;
it asserts direct and tool-result output bytes, not internal call
counts.

The deterministic regression fails against the unchanged baseline with
missing status rows. After rebasing onto current main
`1ad92f0cbc3dea27676e253588100f64a2af6994`, ran the status regression
plus main's six small-MCP result tests: **7 passed**, and reran the
native Git status cases: **2 passed**.

Changed-file Ruff and formatting passed. The installed pre-commit hooks,
including `mypy headroom` and Commitlint, passed for this commit.
Full-tree Ruff is locally blocked by executable filesystem permissions
on five unchanged OAuth2 plugin files (tracked 100644); they are not
modified. A separate full dev-environment mypy run found an unchanged
optional LangChain `truthy-function` error; the configured isolated
pre-commit environment passed. No unrelated source/permissions were
changed.

## Real Behavior Proof

- Environment: Linux x86_64, CPython 3.13.14; unchanged
`dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; actual Git
fixture, real content router and Anthropic token counter, rebuilt Rust
extension, **real ONNX Kompress fully preloaded before each run**, no
mocked compression backend in this smoke.
- Exact command / steps: initialize a throwaway Git repo; commit 120
files named `modified path NNN α.txt`; modify those, stage 120 new files
named `staged path NNN β.txt`, and add 120 untracked files named
`untracked path NNN γ.txt`. Capture:
  ```sh
LC_ALL=C git -c core.quotePath=false -c color.status=false status
--branch
  ```
Load the actual model with
`KompressCompressor().preload(allow_download=True)`, run
`ContentRouter(ContentRouterConfig()).compress(status)`, then run the
same router's `apply` on an Anthropic `Bash` tool-use/tool-result pair
using
`AnthropicProvider(warn=False).get_token_counter('claude-sonnet-4-5')`
and model limit 200000. Compare output bytes to the same captured
status.
- Same input in both runs: **12,469 bytes**, SHA256
`78ed6a39f2f994159099f8d4c0fe052ca09a37fa6d9f6248e8c5ebb655b19a32`.
- Before: detected as `text`; direct and Bash tool-result routes both
returned **10,748 bytes**, neither byte-exact; transform
`router:tool_result:text`.
- Observed result: after the fix, detected as `tabular`; both routes
returned **12,469 bytes**, both byte-exact; `router:noop`.
- External-compressor review proof against published head `3b035027`: a
real Git repository with 120 untracked paths produced the same
**3,415-byte** input (SHA256
`e0164d25f3f47428686d102a73018d0e00bcda53ff9674e658aa7128c038896b`).
With an operational row-trimming plugin selected for `text/csv` and
`*/*`, direct and tool-output paths both shrank to **318 bytes** before;
both preserve all **3,415 bytes** after. The same plugin still
compresses ordinary CSV. This separate plugin smoke uses the production
Tiktoken counter and disables Kompress to isolate external dispatch.
- Final mixed-content review: on published `539669c6`, appended JSON
selected `MIXED` and changed both direct and Bash tool-result bytes; the
fenced-code example was already preserved. The strategy selector now
honors `git_status` before mixed splitting. Both appended JSON and
fenced code preserve the complete payload on both routes with a ready
lossy compressor available.
- Not tested: localized Git status formats, every platform-specific Git
variant, or a paid upstream model response. The recognizer deliberately
covers standard English status output. The local real compression path
was exercised after model warmup; the initial cold-model no-op was not
treated as proof.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; content-routing data-preservation
fix.
- Minimum rollout channel: stable; no rollout-managed feature is
enabled.
- Stable/default behavior changed: recognized Git status is no longer
compressed lossily, sacrificing savings for path/state integrity.
- Kill switch / disable path: existing global Bash exclusion can bypass
compression; no new flag.
- Unsafe override required: no.
- Qualification impact: Python/Rust route parity for the recognized
status format only.
- Rollback path: revert this commit; no persistent schema migration.
Global Bash exclusions remain unchanged.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Existing tabular lossless route reused; compression docs updated.
- [x] Consumer regression and native positive/negative tests added.
- [x] `CHANGELOG.md` not edited; Conventional Commit title supplied.

This PR completes the remaining Git status defect, but intentionally
does not close #3652's separate Bash-default policy request. Remote CI
is reported by GitHub, not assumed from local tests.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-07 17:46:57 -05:00
ParideboyandClaude Opus 5.5 6c9aef1dec fix(transforms): detect fixed-width command output as tabular and keep it out of Kompress (#3786)
## Description

Part of #3652.

**The problem.** Commands like `ls -la`, `ps aux`, `df -h`, `docker ps`
and `kubectl get pods` print a table whose columns are lined up with
spaces. Headroom did not recognise that shape, so it treated the output
as plain text and sent it to Kompress, the ML compressor meant for
prose. Kompress drops words it thinks are unimportant. In a table that
means some rows lose a field (an owner, a size, a date) and others keep
it, and nothing in the output says which rows were changed. The agent
then works from a file listing that is quietly wrong.

**Still happening on `main`.** I ran the example from the issue on
`main` at 3aa50128 with the Kompress model loaded. 60 lines of `ls -la`
came back as 2 lines, and the owner column was missing from 17 of the 59
file rows.

**What this PR changes.**

1. The content detector now recognises space-aligned columns and labels
them as a table (`TABULAR`, format `fixed_width`) instead of plain text.
2. The router no longer passes a table to Kompress when the table
compressor could not shrink it. Before this PR, a table the tabular
compressor declined (for example a CSV with one uneven row, see #1652)
still went to Kompress as a fallback. Now it is kept exactly as it was.

With both changes, the issue's `ls -la` example comes back byte-for-byte
unchanged: 60 lines, owner present on all 59 rows.

**What this PR does not fix.**

- The `git status` example from the same issue. It has only 2 columns
under prose headings, and a rule loose enough to catch it would also
catch bullet lists and `key: value` lines. It is still word-dropped, so
this PR says "Part of" rather than "Closes", and #3652 should stay open.
- Tables where columns are sometimes separated by a single space (`ls
-l`, `ps aux`, `df -h`) split into uneven rows with the existing parser,
so they are now kept verbatim. That is safe, but it saves nothing.
Tables with wider gaps (`kubectl get`, `docker ps`) go through the
existing lossless CSV-schema compaction that markdown tables already
use, and every row is kept. In my runs that took a 2,805-byte `kubectl`
listing to 1,803 bytes and a 4,194-byte `docker ps` listing to 3,628
bytes.
- `kubectl` output that contains `Error` pods can be classified as
build/log output earlier in the detector, so it never reaches the new
check.
- The opt-in `lossless_then_lossy` and `force_kompress_all` modes are
unchanged.
- Whether Bash output should be excluded from compression by default
(#3590) is a separate call for maintainers. `DEFAULT_EXCLUDE_TOOLS` is
not touched here.

**How it relates to other open PRs.**

- #3119 keeps line breaks inside Kompress output. It helps, but it does
not fix this issue on its own: I applied it to `main` in a scratch copy
and the `ls -la` example kept all 60 lines but still lost the owner from
17 of 59 rows. This PR keeps that kind of output away from Kompress
entirely, so the two work together.
- #3590 reports the same symptom and suggests excluding Bash by default.
- #3514 edits `tabular_ingest.py` and the same test file in different
places. Whichever lands second may need a small rebase of the test
imports.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/content_detector.py`: new
`_try_detect_fixed_width`. It runs last, right before the plain-text
fallback, so it can only change content that is classified as plain text
today. It needs at least 4 lines and at least 3 columns whose gaps line
up on 90% of the lines. It backs off when the text looks like a bullet
or numbered list, wrapped prose (lines ending in `.`, `!` or `?`), or
code (lines ending in `{ } ; ( ) , : \` or starting with a comment
marker). A one-line heading such as `total 480` above the `ls -l` rows
is allowed.
- `headroom/transforms/content_router.py`:
- `TABULAR` is removed from the strategies that fall back to Kompress
when they save nothing, so a declined table stays verbatim. A stale
comment about that fallback is updated.
- File reads (`cat`, `Read`) of space-aligned text stay protected
(byte-exact), because hand-aligned files such as `/etc/fstab` or a block
of C `#define`s look the same as command output. CSV and markdown reads
are released as before.
- `headroom/transforms/tabular_ingest.py`: `parse_tabular` returns
`None` for a fixed-width table that parses into a single column. The
detector finds columns by single-space gaps, but the existing parser
splits on two or more spaces, so GNU `ls -l` can parse as one cell per
line. That is not a table, so it is passed through verbatim.
- `tests/test_transforms_tabular.py`:
- detection tests for 5 real command outputs (the issue's `ls -la`,
macOS `ls -la`, `kubectl get pods`, `ps aux`, `df -h`);
- 7 look-alikes that must stay plain text (bullets, numbered list, SQL,
C `#define`s, `git status`, `git log --oneline`, wrapped prose);
  - a parser test for the single-column case;
- router tests showing that the issue's `ls -la` payload and an uneven
CSV come back unchanged and never reach Kompress;
- a test that a fixed-width file read stays protected while a CSV read
is released.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Type checking: `mypy headroom --ignore-missing-imports` (mypy 1.20.2)
reports one error, in `headroom/fileperms.py` (`os.fchmod`, which does
not exist on Windows). That file is not touched by this PR and the
changed files are clean, so I left the box unchecked.

Full suite on Windows: 13,001 passed, 57 failed, 57 errors. I reran
every failing test on an untouched `main` checkout (3aa50128) and on
this branch. The same 52 fail on both (Windows file modes, symlinks,
POSIX paths). The rest passed when rerun on their own, so none of the
failures come from this change.

### Test Output

```text
$ pytest tests/test_transforms_tabular.py -q
.................sssss............                                       [100%]
======================= 68 passed, 5 skipped in 54.55s ========================

$ ruff check . && ruff format --check .
All checks passed!
1653 files already formatted

$ pytest tests -q -n 8   # Windows, full suite on this branch
= 57 failed, 13001 passed, 861 skipped, 2785 warnings, 57 errors in 903.33s (0:15:03) =

$ pytest <every failing test above> -q -n 8   # same list, main vs this branch
main (3aa50128): 50 failed, 32 passed, 14 warnings, 2 errors
this branch:     50 failed, 32 passed, 14 warnings, 2 errors   (identical failing set)
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.11, headroom installed from this
branch, Kompress model `kompress-v2-base` loaded from the local cache,
pure-Python content detection (the default on Windows).
- Exact command / steps: built the `ls -la` payload from #3652 (a `total
480` line plus 59 file rows, sizes from `random.seed(1)`) and ran it
through `ContentRouter(ContentRouterConfig()).compress(payload)`, once
on `main` at 3aa50128 and once on this branch, with Kompress warmed up
first.
- Observed result: on `main` the payload is detected as `PLAIN_TEXT`
(confidence 0.5), goes to Kompress, and comes back as 2 lines with the
owner on only 42 of 59 rows. On this branch it is detected as `TABULAR`
(`fixed_width`, confidence 0.7), the strategy is `TABULAR`, and the
output is byte-for-byte identical to the input: 60 lines, owner on 59 of
59 rows. I also ran the new detector over every `.py`, `.md`, `.rs`,
`.ts`, `.sh`, `.toml`, `.yml` and `.txt` file in this repo, cut into
60-line chunks (about 12,400 chunks). It matched the real table in
`sbom/headroom-sbom-table.txt` and one block of four lined-up `assert`
lines in a test file, which is now passed through unchanged instead of
going to Kompress.
- Not tested: Linux or macOS, where the Rust/Magika detector runs first
and this Python check only runs if that detector also says plain text.
Not tested in a live proxy session with a real agent.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. This is a detection and routing fix
in the content router.
- Minimum rollout channel: stable
- Stable/default behavior changed: Yes. Space-aligned command output
that used to be word-dropped by Kompress is now either kept verbatim or
compacted losslessly (every row kept), and tables the tabular compressor
declines are no longer sent to Kompress. Some of that output now saves
fewer tokens than before, in exchange for not losing fields.
- Kill switch / disable path: No dedicated switch. Turning off the
tabular compressor (`--compressor` without `tabular`) keeps these
outputs verbatim. There is no setting that restores the old
word-dropping behavior; reverting this PR does.
- Unsafe override required: No
- Qualification impact: None expected. The change only affects content
that was previously classified as plain text, or tables the tabular
compressor had already declined.
- Rollback path: Revert this PR. There is no stored state or migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`. It is generated by
release-please from the Conventional Commit PR title.

## Additional Notes

No documentation change: `tabular_ingest.py` and the `TABULAR` content
type already say fixed-width tables are supported. Until now the
detector never produced that format, so that path was only reachable in
tests.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 09:48:23 -05:00
e8a510982a fix(spreadsheet): stop .xls integer rendering fabricating digits above 2^53 (#3704)
Closes #3695. Follow-up to #3616, which fixed a real bug but introduced
a narrower one at the top of the numeric range.

## Problem

`_xls_cell` converted every whole-valued number to `int`. xlrd hands
back an IEEE-754 double, and above 2^53 consecutive integers are no
longer representable — so `int()` renders the double's *exact* value,
which is not the number the user typed:

```
user typed:  123456789012345678
stored as:   1.2345678901234568e+17
int() shows: 123456789012345680   <- last two digits fabricated
```

This text goes to an agent. An ID column rendered as
`123456789012345680` reads as an exact identifier; it isn't. That is the
main concern — the `.xlsx`/`.xls` divergence #3616 set out to remove is
the secondary one (both loaders showed the float repr before, and only
diverge above 2^53 now).

## Fix

One bounded condition:

```python
if kind == xlrd.XL_CELL_NUMBER and float(value).is_integer() and abs(value) < 2**53:
```

Below 2^53 the double is exact, so `int()` is lossless and the `12.0 ->
12` win that motivated #3616 is preserved. At or above, fall through to
the float repr — which signals approximation honestly *and* restores
`_load_xlsx` parity.

`< 2**53` rather than `<=`: 2^53 is representable but 2^53+1 is not, so
the strict bound is the conventional "integers are exact below here"
line.

## Verification

New parametrized tests call `_xls_cell` directly with a stub cell (no
`.xls` fixture needed). Verified fail-before / pass-after by reverting
only the source file and keeping the tests:

```
4 failed, 3 passed     <- e.g. assert isinstance(123456789012345680, float)
```

After the fix: **52 passed** in `tests/test_transforms_tabular.py`.

## Also in this PR

- Docstring note that sub-second precision is dropped (`xldate_as_tuple`
returns whole seconds, so `12:34:56.5` renders `12:34:56` where openpyxl
keeps microseconds).
- Removed the stale `# pragma: no cover` on the `.xls` dispatch — it
referenced a comment #3616 deleted, on a line the tests now exercise
(confirmed via coverage: the line is no longer in the missing set).

## Note for reviewers

These tests **silently skip** without `xlrd` installed. CI is fine
(`[dev]` pins `xlrd>=2.0.1`), but a local venv without it shows `s`
rather than failures.

Still open in #3695 and deliberately out of scope here: replacing the
`xlwt` dev dependency with a committed binary fixture, and covering
`XL_CELL_ERROR`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com>
2026-09-24 13:09:02 +02:00
sxhandsxh313 1a7c5bbf52 fix(spreadsheet): stop inventing digits for .xls cells above 2**53 (#3700)
## Description

Bounds the `int()` conversion added by #3616 to the range where a double
is exact, so a legacy `.xls` cell stops inventing digits (#3695).

`_xls_cell` rendered any integral double with `int(value)`. xlrd hands
back an IEEE-754 double, and above 2**53 consecutive integers are not
representable, so the conversion prints the double's *exact* value
rather than the number the sheet held:

```text
typed  123456789012345678   stored  1.2345678901234568e+17   rendered  123456789012345680
```

Two trailing digits, fabricated, delivered to an agent in a form that
reads as an exact identifier. Before #3616 both loaders rendered the
float and simply disagreed about `12.0` vs `12`; after it they agree on
small whole numbers and disagree again -- in the worse direction -- at
the top of the range, which is also where the PR's own goal ("render an
.xls cell the way the .xlsx loader renders it") stops holding.

Closes #3695

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/spreadsheet_ingest.py`: the numeric branch
becomes `if kind == xlrd.XL_CELL_NUMBER and float(value).is_integer()
and abs(value) <= 2**53:`, so values past the exactly-representable
range`return value` and render as the float. The comment records why the
bound exists so it is not "simplified" away again.
- `tests/test_transforms_tabular.py`: adds
`test_load_xls_and_xlsx_agree_above_the_exact_integer_range`, written
against the same xlwt/xlrd fixture style the three existing `.xls` tests
use. It pins both halves of the requirement: `12` still renders without
a decimal point (the #3616 win survives the bound), and
1.2345678901234568e+17 renders as `repr()` of the double rather than an
invented integer.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Type checking was run on the changed module rather than the whole
package, so that line means `mypy --config-file pyproject.toml
headroom/transforms/spreadsheet_ingest.py`, not a full-package pass.

### Test Output

Red, with the fix removed and the new test kept (this is also the state
of `main` for a cell in that range):

```text
$ python -m pytest tests/test_transforms_tabular.py -k exact_integer_range -q
E   AssertionError: assert '123456789012345680' == '1.2345678901234568e+17'
E
E     - 1.2345678901234568e+17
E     ?  -                ^^^^
E     + 123456789012345680
FAILED tests/test_transforms_tabular.py::test_load_xls_and_xlsx_agree_above_the_exact_integer_range
1 failed, 45 deselected in 4.10s
```

Green, with the fix:

```text
$ python -m pytest tests/test_transforms_tabular.py -q
46 passed in 5.48s

$ uvx ruff@0.16.7 check headroom/transforms/spreadsheet_ingest.py tests/test_transforms_tabular.py
All checks passed!
$ uvx ruff@0.16.7 format --check headroom/transforms/spreadsheet_ingest.py tests/test_transforms_tabular.py
2 files already formatted
$ uvx mypy@1.14.1 --config-file pyproject.toml headroom/transforms/spreadsheet_ingest.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11 (10.0.26200) x64, Python 3.12.13, working tree
at `origin/main` 85fac8c3 plus this one-file change. Optional extras
present for this path only: `xlrd 2.0.2`, `xlwt 1.3.0`, `openpyxl`. No
provider, no API key, no network -- the loader reads a local file.
- Exact command / steps: build a real legacy workbook with `xlwt`
holding one small whole number and one whole-valued double beyond 2**53,
save it as `.xls`, and read it through the shipped `load_spreadsheet()`;
then run the new test twice, once with the bound in place and once with
the pre-fix line restored in the working tree.
- Observed result: the same cell renders `123456789012345680` before
this change and `1.2345678901234568e+17` after it, while `12` renders as
`12` in both states.

```text
$ python -c "xlwt writes [12, 1.2345678901234568e17] into legacy.xls; print(load_spreadsheet(...)['Data'].splitlines()[1])"
post-fix row: 12,1.2345678901234568e+17

$ python -m pytest tests/test_transforms_tabular.py -k exact_integer_range -q   # fix reverted in the tree
E   AssertionError: assert '123456789012345680' == '1.2345678901234568e+17'
```

The same cell, both states: the pre-fix render carries two invented
digits
(`123456789012345680`), the post-fix render is the double the sheet
holds. The
`12` half is identical in both, which is the point of bounding rather
than
deleting the conversion -- whole numbers at or below 2**53 keep the
#3616 rendering,
and only the range where a double cannot hold an integer reverts to the
float.
- Not tested: a workbook authored in real Excel or LibreOffice (the
fixture is written by `xlwt`, which is also how the three existing
`.xls` tests build theirs), and `.xls` cells in the 2**53..2**1024 range
carrying `#`-formatted number styles, which `_xls_cell` does not see at
all. One measurement worth flagging for whoever picks this up: I first
asserted that the two loaders produce the same *text*, and it failed for
a reason unrelated to this fix -- a value written and read back through
`openpyxl` comes back as `1.234567890123457e+17`, because openpyxl
serializes the literal to 15 significant digits, so the `.xlsx` path
loses a digit on write while the `.xls` path loses one on render. The
test therefore asserts the property that is this fix's subject (no
invented integer; small integers still `12`) instead of cross-format
textual equality, which the loaders do not actually guarantee for values
in that range.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. One comparison in a pure function
inside the spreadsheet loader; no flag service or staged rollout is
involved.
- Minimum rollout channel: n/a -- ordinary release.
- Stable/default behavior changed: yes, narrowly. `.xls` cells holding a
whole-valued double at or above 9007199254740992 now render as
`1.2345678901234568e+17` instead of `123456789012345680`. Below that
bound, and for every non-numeric cell type, output is byte-identical to
today.
- Kill switch / disable path: none needed and none added. Reverting the
single commit restores the previous rendering exactly; there is no new
code path to turn off, and no cache or stored format is involved
(`load_spreadsheet` is a read path).
- Unsafe override required: no.
- Qualification impact: text of affected cells changes, so any test or
prompt-cache fixture that captured `.xls` output for a cell in that
range sees a diff. Existing tests: all 46 in
`tests/test_transforms_tabular.py` pass, including the three `.xls`
tests from #3616 whose expectation (`12`) is deliberately preserved.
- Rollback path: revert the one commit; no migration, no persisted
state, no store rewrite.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` -- it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable; the observable artifact is the rendered CSV row, quoted
in the Real Behavior Proof.

## Additional Notes

Two judgement calls, both flagged for review:

- The bound is `abs(value) <= 2**53`. #3695 proposed the exclusive form,
and the review here correctly pointed out that binary64 represents 2**53
and -2**53 exactly while openpyxl loads those as integers, so excluding
them left the two loaders disagreeing at the boundary itself. A stricter
bound (e.g. 1e15, where decimal-style output still lo A stricter bound
(e.g. 1e15, where decimal-style output still looks like a plain integer)
would change more cells than necessary, and a looser one would keep
inventing digits.
- A negative whole number is also rendered as an int when small; `abs()`
keeps that behavior symmetric, which is what the existing tests expect
for positives and what `repr()` would otherwise contradict for
negatives.

Note on CI: `test (3)` on this branch is expected to show the same
pre-existing `test_cost_per_token_resolves_deepseek_v4_flash` failure
described in #3683 (litellm's off-peak pricing window), which #3684
addresses. Nothing in this diff is on that test's path.

---------

Co-authored-by: sxh313 <250161920+sxh313@users.noreply.github.com>
2026-09-23 23:37:48 -05:00
LukasandJD Davis 261796f9be fix(spreadsheet): render an .xls cell the way the .xlsx loader renders it (#3616)
## Description

`_load_xls` reads a legacy workbook with `sheet.row_values(i)`, which
hands back xlrd's **raw storage** rather than the value. The `.xlsx`
loader goes through openpyxl and hands back the value. So the two
loaders disagree about the same workbook.

Measured on `main` (35b1156), the same three cells written to both
formats:

| cell | `.xls` today | `.xlsx` |
|---|---|---|
| `2024-01-01` | `45292.0` | `2024-01-01 00:00:00` |
| `TRUE` | `1` | `True` |
| `12` | `12.0` | `12` |

The date is the one that matters: a serial number is not recoverable
from the compressed text, so whatever reads the table downstream sees
`45292.0` and has no way to know it was a date. The boolean and the
whole number are the same class of drift — `1` and `12.0` are not what
the sheet says.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/spreadsheet_ingest.py`: new `_xls_cell(cell,
datemode)` converts each cell by its xlrd type, matching what openpyxl
already delivers — `XL_CELL_DATE` → `datetime` via `xldate_as_tuple`
with the workbook's `datemode` (the 1900/1904 epoch matters and only the
book knows which it is); `XL_CELL_BOOLEAN` → `bool`; a whole
`XL_CELL_NUMBER` → `int`, since xlrd stores every number as a double;
`XL_CELL_ERROR` → the text Excel shows (`#DIV/0!`), which is what
openpyxl with `data_only=True` gives.
- A cell whose `xldate_as_tuple` has no date part — a time such as
`12:00` is `(0, 0, 0, 12, 0, 0)` — becomes a `datetime.time`, again
matching openpyxl. Building a `datetime` from it raises `ValueError:
year 0 is out of range`.
- A float that is not whole, a string, and an empty cell are untouched:
`1.5` stays `1.5`, and a blank still renders as an empty field through
`_rows_to_csv`.
- `sheet.row_values(i)` becomes `sheet.row(i)`, because the conversion
needs the cell type that `row_values` throws away.
- The `# pragma: no cover - legacy .xls; needs optional xlrd + binary
fixture` comes off `_load_xls`: it is covered now.
- `pyproject.toml`: `xlrd>=2.0.1` (the `spreadsheet` floor) and
`xlwt>=1.3.0` join the dev extra beside `openpyxl`, with the same style
of comment. The CI test job installs only `[dev]`, and every new test
`importorskip`s both: `xlwt` is the only way to write a legacy `.xls`
fixture, and `xlrd` is what `_load_xls` reads it with. `uv.lock` is
regenerated to match (13 added lines, nothing else moves).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Three cases in `tests/test_transforms_tabular.py`, each building a real
`.xls` with `xlwt`:

- the row renders as `2024-01-01 00:00:00,True,12,1.5,ok` — it was
`45292.0,1,12.0,1.5,ok`;
- `load_spreadsheet` of the `.xls` and of the equivalent `.xlsx` return
**equal** dictionaries, which is the property the first test is really
about;
- a time-only cell renders as `12:00:00` in both formats, where it
rendered as `0.5` (and raised with only the first commit applied).

`mypy` is unchecked because it was not run locally; CI's lint job runs
it.

### Test Output

```text
$ ruff check headroom/transforms/spreadsheet_ingest.py tests/test_transforms_tabular.py
All checks passed!

$ ruff format --check headroom/transforms/spreadsheet_ingest.py tests/test_transforms_tabular.py
2 files already formatted

$ python -m pytest tests/test_transforms_tabular.py -rs
43 passed in 4.03s
```

Before `xlrd` joined the dev extra, the same command in the CI test
environment gave 40 passed, 3 skipped (`could not import 'xlrd'`).

## Real Behavior Proof

- Environment: macOS 26.6 (arm64), CPython 3.12.11, `xlrd 2.0.2`,
`openpyxl`, `xlwt 1.3.0`; the wheel built with `maturin build --profile
ci` (Rust 1.98.1) and installed as `"${WHEEL}[dev]"`, as the CI test job
does.
- Exact command / steps: build a one-cell `.xls` with `xlwt` and the
equivalent `.xlsx` with `openpyxl`, then call
`headroom.transforms.spreadsheet_ingest.load_spreadsheet` on each. That
module imports cleanly on its own; only the test file's other imports
pull in `headroom._core`.
- Observed result: on `main` the `.xls` row reads
`45292.0,1,12.0,1.5,ok` and a time-only cell reads `0.5`; with this PR
both match the `.xlsx` output:

```text
# before (main)
xls  -> ['Starts', '0.5']                           # the time-only cell
xls  -> ['When,Active,Units,Rate,Text',
         '45292.0,1,12.0,1.5,ok']                   # the value cells

# after
xls  -> ['Starts', '12:00:00']
xlsx -> ['Starts', '12:00:00']
xls  -> ['When,Active,Units,Rate,Text',
         '2024-01-01 00:00:00,True,12,1.5,ok']
```

- Not tested: `mypy headroom` locally (CI's lint job runs `mypy headroom
--ignore-missing-imports`; green at 466683c). Note on CI: its test job
installs only `"${WHEEL}[dev]"`, and `xlrd` was only in the
`spreadsheet` extra, so the three new cases were skipped there (shard 4:
`s`/`ss`) until `xlrd` joined `dev` in this PR.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. `spreadsheet_ingest` is not behind a
rollout channel.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, for `.xls` input only — a date,
a boolean, a whole number and an error cell now render as their value
instead of as xlrd's storage, which is what the `.xlsx` path has always
done. `.xlsx`, `.csv` and every other input are byte-identical.
- Kill switch / disable path: none needed; revert the commit.
- Unsafe override required: no.
- Qualification impact: any golden output that pinned `45292.0`, `1` or
`12.0` for a `.xls` would change. There is none in the repo —
`_load_xls` was `# pragma: no cover` before this PR.
- Rollback path: `git revert`; the change is confined to `_xls_cell` /
`_load_xls`, two dev-extra lines and their `uv.lock` entries.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (no
documented behaviour changes: the `.xls` path now matches the `.xlsx`
path the docs already describe)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes (full
suite locally: 12550 passed, 599 skipped, 1 failed; the failure,
`tests/test_cli/test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`,
needs the venv's `bin` on `PATH` and passes with it)
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

The time-only case was found after the first commit: `xldate_as_tuple`
returns `(0, 0, 0, 12, 0, 0)` for `0.5`, and `datetime(0, 0, 0, 12, 0,
0)` raises, so the whole workbook failed to load. Commit 466683c fixes
it and adds the third test.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-21 10:06:01 -07:00
Lukas 334317db5d fix(tabular): pass a table through when one cell is past the csv field limit (#3618)
## Description

`csv.field_size_limit` is **128 KB per cell** by default. One pasted
document, log excerpt or base64 blob in a column goes past it, and the
stdlib reader then raises for the **whole text**, not for the row:

```python
>>> csv.field_size_limit()
131072
>>> list(csv.reader(io.StringIO("id,body\nl," + "x" * 200_000)))
_csv.Error: field larger than field limit (131072)
```

`parse_csv` does not catch it, so `TabularCompressor.compress` raises
out of a content type it advertises support for:

```
>>> TabularCompressor().compress("id,title,body\nl,short,ok\n2,long," + "x" * 200_000)
Error: field larger than field limit (131072)
```

The module already has a policy for a table it cannot use —
`parse_tabular` returns `None` for a ragged one (#1652) and the text
passes through verbatim to the plain-text route. An oversized cell is
the same situation and should take the same exit.

This is not hypothetical for a context compressor: the content it sees
is tool output and pasted material, which is exactly where a single 128
KB+ cell comes from. Two other projects that ingest CSVs hit it and say
so in their own code — `onyx/backend/onyx/utils/csv_utils.py`:
*"real-world data (long descriptions, pasted docs, base64 blobs)
routinely exceeds"*, and `skyvern/forge/sdk/workflow/models/block.py`.
Both raise the limit process-globally, which is a reasonable call for an
application and the wrong call for a library running inside someone
else's process.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/tabular_ingest.py`: `parse_csv` catches
`csv.Error` and returns `([], [])`, which makes `parse_tabular` return
`None` and the compressor pass the content through unchanged. The
docstring records why the limit is not raised instead.
- No process-global state is touched, so a host application's own `csv`
behaviour is unaffected.
- Nothing else changes: a table that parses today parses identically,
and the `[]` return reuses the empty-input path that was already there.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Two cases in `tests/test_transforms_tabular.py`, next to the
ragged-table ones they mirror:

- `test_parse_csv_gives_up_on_a_cell_past_the_field_size_limit`
- `test_compress_passes_through_a_table_with_an_oversized_cell`

`pytest` is unchecked because the suite does not run in my checkout, not
because it failed — see **Not tested**.

### Test Output

```text
$ ruff check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
All checks passed!

$ ruff format --check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
2 files already formatted

$ python -m pytest tests/test_transforms_tabular.py
tests/test_transforms_tabular.py:24: in <module>
    from headroom.transforms.content_router import (
headroom/transforms/error_detection.py:41: in <module>
    from headroom._core import (
E   ModuleNotFoundError: No module named 'headroom._core'
```

The collection error is identical with my changes stashed: the file
imports the compiled Rust extension and there is no Rust toolchain on
this machine.

## Real Behavior Proof

- Environment: macOS 15 (arm64), CPython 3.12, no Rust toolchain.
- Exact command / steps: call
`headroom.transforms.tabular_ingest.parse_csv` and
`TabularCompressor().compress` on `"id,title,body\nl,short,ok\n2,long,"
+ "x" * 200_000`. That module imports cleanly on its own; only the test
file's other imports pull in `headroom._core`.
- Observed result: `parse_csv` raised `field larger than field limit`
before the change and returns `([], [])` after it, and `compress` passes
the text through byte-identical (output below).

```text
# before (source change stashed)
parse_csv -> Error : field larger than field limit (131072)

# after
parse_csv -> ([], [])
compress -> passthrough, byte-identical
```

- Not tested: `pytest tests/test_transforms_tabular.py` and `mypy
headroom`, both of which need `headroom._core`; CI builds it, so the two
new cases run there. Also **not** tested end to end through
`ContentRouter`, for the same reason — I verified the public
`TabularCompressor` / `parse_tabular` / `parse_csv` surface, and I do
not claim to know whether the router's dispatch would have swallowed the
exception or propagated it.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. `tabular_ingest` is not behind a
rollout channel.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: only for text that raises today. A
table that parses now parses identically; a table that raised now passes
through verbatim and is still compressed by the plain-text route.
- Kill switch / disable path: none needed; revert the commit.
- Unsafe override required: no.
- Qualification impact: none — no golden output can have pinned an
exception from this path.
- Rollback path: `git revert`; the change is four lines inside
`parse_csv`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (no
documented behaviour changes)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes (cannot
run the suite here — see **Not tested**)
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

If you would rather compress these tables than pass them through, the
other option is a `csv.field_size_limit` bump. I did not take it because
it is process-global and headroom is imported into someone else's
process: raising it there silently changes how that application's own
`csv` code behaves. If you want it anyway, a set/restore around the
parse keeps the blast radius to this module, at the cost of a race
between concurrent parses on separate threads — which is the trade-off
skyvern's comment describes when it chose to set the limit once at
import instead.
2026-09-21 10:05:42 -07:00
Abhay SinghandJD Davis 10d8f15bc7 fix(transforms): drop trailing empty spreadsheet rows and stray CR in CSV render (#3254)
## Description

`_rows_to_csv` in the binary spreadsheet ingester does not do what its
own docstring ("dropping fully empty trailing rows") says, and emits a
stray carriage return:

```python
def _rows_to_csv(rows: list[list[object]]) -> str:
    """Render rows to CSV text, dropping fully empty trailing rows."""
    buf = io.StringIO()
    writer = csv.writer(buf)                 # default lineterminator = "\r\n"
    for row in rows:                         # trailing empty rows are NOT dropped
        writer.writerow(["" if cell is None else cell for cell in row])
    return buf.getvalue().strip("\n")        # strips "\n" only, leaving a dangling "\r"
```

Two problems, both hit in normal use:

1. **Trailing empty rows are not dropped.** openpyxl's `iter_rows` walks
the sheet's *used range*, which routinely extends past the last data row
(leftover formatting, a cleared cell), so a real sheet commonly ends in
`(None, None, …)` tuples. Those get written out as blank `,` rows.
2. **A dangling `\r` is left on the output.** `csv.writer` defaults to a
`\r\n` terminator, and `.strip("\n")` removes only the `\n`, so the last
line ends in a bare `\r` (and every undropped trailing empty row leaves
`,\r` residue).

Concretely, a sheet `[["Name","Age"], ["Alice","30"], (None,None),
(None,None), (None,None)]` renders to
`'Name,Age\r\nAlice,30\r\n,\r\n,\r\n,\r'` instead of
`'Name,Age\nAlice,30'`. This noisy text then flows through tabular
detection → SmartCrusher and on to the model — extra rows and
control-character residue for every spreadsheet with a trailing
used-range.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/spreadsheet_ingest.py`: drop the trailing run of
fully-empty rows before writing (a row is empty when every cell is
`None` or blank-after-strip), and set `csv.writer(...,
lineterminator="\n")` so `.strip("\n")` leaves no dangling `\r`.
Interior empty rows are preserved (only the trailing run is dropped); a
fully empty sheet renders to `""`.
- `tests/test_transforms_tabular.py`: added
`test_rows_to_csv_drops_trailing_empty_rows_and_has_no_dangling_cr`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_transforms_tabular.py  ->  38 passed, 2 skipped in 1.76s
(the new test FAILS on pre-fix code, verified via git stash)
uvx ruff@0.16.2 check headroom/transforms/spreadsheet_ingest.py tests/test_transforms_tabular.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/transforms/spreadsheet_ingest.py  ->  Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps:
`_rows_to_csv([["Name","Age"],["Alice","30"],[None,None],["","
"],[None,None]])` returned `'Name,Age\r\nAlice,30\r\n,\r\n,\r...'`
before the fix and `'Name,Age\nAlice,30'` after;
`[["a","b"],[None,None],["c","d"],[None,None]]` → `'a,b\n,\nc,d'`
(interior empty kept, trailing dropped); an all-empty sheet → `''`.
- Observed result: trailing empty rows dropped, no stray `\r`, interior
rows and data unchanged.
- Not tested: exercised `_rows_to_csv` directly rather than
round-tripping a physical `.xlsx` (the existing
`test_load_and_compress_xlsx` covers the openpyxl path; `splitlines()`
there is agnostic to the terminator change).

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is the SDK-side spreadsheet
ingestion adapter, not a rollout-channel-gated runtime feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: only the incorrect output is
corrected — trailing empty rows and the stray `\r` are gone; data rows,
interior empty rows, and sheet skipping are unchanged. The line
terminator in the intermediate CSV text is now `\n` (it feeds text
compression, not a strict CSV consumer).
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-09 23:07:13 -05:00
ParideboyandClaude Fable 5 c7665ca088 fix(transforms): pass through ragged tables instead of misaligning columns (#1713)
## Description

Issue #1652 reports the proxy's compression layer surfacing an
"impossible mixed" status line — a row combining fields from two
different rows of a version-status table (Docker row `0.42.4 → 0.43.0
update available` blended with WSL row `0.42.4 → 0.42.4 up-to-date`).
The reporter's follow-up refined the claim: the stored canonical content
was intact, but the compression path presents a lossier view that
invites exactly this misattribution.

There is a concrete mechanism for that in the tabular bridge:
`parse_tabular` (`headroom/transforms/tabular_ingest.py`) hands parsed
rows to `to_records`, which **silently pads/truncates every row to the
header width**. For ragged tables — rows whose cell count differs from
the header row, exactly what mixed-shape status tables like the
reporter's produce (`✓` and `-` placeholder cells change the token count
per row) — this shifts values under the wrong column before SmartCrusher
compaction. The compressed output can then state column/value pairings
the original never contained.

Fix: `parse_tabular` now rejects ragged tables (any row width ≠ header
width) and returns `None`, so the content passes through verbatim, per
the issue's requirement that a lossy summary "must not create impossible
mixed facts". Aligned tables compress exactly as before. The Rust
`log_template` Drain miner was also examined; its template rendering
only emits tokens that are constant across all rows of a run, so no
defect was found there and it is left untouched.

Fixes #1652

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/tabular_ingest.py`: `parse_tabular` returns
`None` when any parsed row's cell count differs from the header count,
instead of letting `to_records` pad/truncate rows into the wrong
columns. `TabularCompressor.compress` then takes its existing
pass-through branch (`was_modified=False`).
- `tests/test_transforms_tabular.py`: three new tests — ragged
fixed-width table rejected (reproducing the issue's rtk version-status
shape), ragged markdown table rejected, and end-to-end
`TabularCompressor.compress` pass-through of a ragged table.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_transforms_tabular.py -q
39 passed

$ ruff check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
All checks passed!
$ ruff format --check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
2 files already formatted
$ mypy headroom --ignore-missing-imports
Success: no issues found (note-level messages only)
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13, local checkout branched from
`upstream/main` (9fbd47ba), Rust core built locally.
- Exact command / steps: constructed the issue's table shape (Docker row
with 4 cells, WSL row with 6 cells under 4 headers) and ran it through
`TabularCompressor().compress()` before and after the change; ran the
full `tests/test_transforms_tabular.py` suite.
- Observed result: before — `to_records` turned the WSL row into
`{'tool': 'rtk', 'installed': '✓', 'latest': '0.42.4', 'status':
'0.42.4'}`: the `up-to-date` status is dropped and a version number
lands under `status` — precisely the misattributed-fact class from the
issue. After — `parse_tabular` returns `None`, `compress` returns the
original text unmodified (`was_modified=False`, byte-identical
pass-through), and all 39 tests pass (36 pre-existing + 3 new).
- Not tested: the reporter's exact end-to-end session (OMP → headroom
proxy on 8787 → sticky-router on 4140); the Rust BuildOutput
`log_template` path, which was reviewed and found to only emit
run-constant tokens.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 12:45:46 -05:00
AshishandClaude Opus 4.8 d789a7c528 feat(transforms): tabular + spreadsheet (.xlsx/.xls) compression (#1128)
## Description

Adds a content-type-aware path for **tabular data** — CSV/TSV, markdown
tables, fixed-width text, and binary `.xlsx`/`.xls` spreadsheets — by
routing them through the existing, battle-tested `SmartCrusher` instead
of letting them fall through to `PLAIN_TEXT → Kompress`.

The pipeline already compressed tables losslessly when handed a JSON
array of records. This wires up the missing front door: detect tabular
text (and ingest binary spreadsheets), convert to JSON records, and
reuse `SmartCrusher.crush()`. No new compression algorithm.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Detection** (`content_detector.py`): new `ContentType.TABULAR` +
`_try_detect_tabular()` for CSV/TSV, markdown tables, and fixed-width
columns. Ordered after search/log (which also look "delimited") and
before code, with a prose-rejection guard so it never steals
`file:line:content` search output, `key: value` logs, or sentences with
incidental commas. Rust backend returns `plain_text` for unknown types
and the router already falls back to the Python detector, so **no Rust
change**.
- **Bridge** (`tabular_ingest.py`): stdlib parsers + `to_records()` + a
`TabularCompressor` that parses → JSON records → `SmartCrusher`
(lossless `csv-schema` first; lossy row-drop with reversible
`<<ccr:HASH>>` markers stays SmartCrusher's built-in fallback). Only
adopts a result when it actually saves bytes.
- **Spreadsheets** (`spreadsheet_ingest.py`): `.xlsx`/`.xls` → per-sheet
CSV text at the SDK boundary. Optional deps (`pip install
headroom-ai[spreadsheet]`) fail loudly with an install hint, never
silently degrade.
- **Routing** (`content_router.py`): `CompressionStrategy.TABULAR`,
`enable_tabular_compressor` flag, lazy getter, apply branch, strategy
maps, Kompress fallback eligibility.
- **SDK** (`compress.py`): `compress_spreadsheet(path, ...)` helper (one
message per sheet).
- **Packaging** (`pyproject.toml`): new `[spreadsheet]` extra;
`openpyxl` added to `[dev]` so the xlsx path is exercised in CI.
- **Docs/demo**: `examples/tabular_compression_demo.py` + README entry.

### Design note: lossless-only

Compact, all-unique tables with no query yield ~0 savings — this is
correct, not a bug. SmartCrusher returns
`skip:unique_entities_no_signal` and won't drop unique rows without a
duplicate/relevance signal. Real wins come from verbose/redundant tables
and query-driven selection. A pressure-driven lossy row sampler was
considered and intentionally not added.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_transforms_tabular.py -q
collected 20 items
tests/test_transforms_tabular.py ....................                    [100%]
============================== 20 passed in 7.15s ==============================

$ ruff check headroom/transforms/tabular_ingest.py headroom/transforms/spreadsheet_ingest.py
All checks passed!

$ mypy headroom/transforms/tabular_ingest.py headroom/transforms/spreadsheet_ingest.py
Success: no issues found in 2 source files
```

`tests/test_transforms_tabular.py` (20 tests): detection true positives
+ no-misroute negatives (search/log/JSON/prose), parser units (incl.
fixed-width), the CSV→SmartCrusher bridge, router routing + disable
flag, and `.xlsx` ingestion (skipif openpyxl missing) + error paths.
`spreadsheet_ingest` 100% / `tabular_ingest` 90% line coverage.

## Real Behavior Proof

- **Environment:** local checkout of `feat/tabular-compression`, Python
3.x, `pip install -e ".[dev]"`.
- **Exact command / steps:** `python
examples/tabular_compression_demo.py` (no API key required).
- **Observed result:**
  ```text
  === Raw tabular text (ContentRouter, char-level) ===
compact unique CSV strat=tabular chars 1306 -> 1072 ( 17.9% saved)
redundant CSV strat=tabular chars 2661 -> 1350 ( 49.3% saved)
verbose markdown strat=tabular chars 2019 -> 1580 ( 21.7% saved)

  === Full pipeline (real tokenizer) ===
  redundant CSV            tokens       768 ->    394  ( 48.7% saved)

  === Binary spreadsheet (.xlsx) ===
  2-sheet workbook         tokens      1092 ->    683  ( 37.5% saved)
  ```
- **Not tested:** legacy `.xls` binary path (needs optional `xlrd` +
binary fixture; `# pragma: no cover`); base64-embedded `.xlsx` inside
multimodal blocks (out of scope, noted as a follow-up).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- CHANGELOG/version are intentionally untouched: this repo uses
**release-please**, which bumps the version and CHANGELOG via automated
`chore: release main` PRs, not per-feature PRs.
- The `.xls` path is `# pragma: no cover` (legacy, needs optional `xlrd`
+ a binary fixture).
- Follow-up (out of scope): base64-embedded `.xlsx` inside
tool-result/multimodal blocks; porting tabular parsers into the Rust
core for parity.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-19 11:30:20 -05:00