mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat/https-compose-caddy
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
b80245ea73 |
fix(tabular): split fixed-width rows on column boundaries, not on double spaces (#4541)
## Description `parse_fixed_width` split every row on runs of 2+ spaces, while the detector finds fixed-width columns by character position. When the two disagree inside one row, the cell count can still match the header, the ragged-row guard in `parse_tabular` passes, and a value is stored under the wrong column. The compressed output (`csv-schema`) carries no CCR marker, so the model cannot tell. See the issue for the exact input. This change cuts each row at the character positions that are blank on every line. If a row crosses a column boundary (a header cell comes out empty or holds a 2+ space gap), the table is not parsed and passes through verbatim. Fixes #4540 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/tabular_ingest.py`: `parse_fixed_width` now finds column spans from positions that are blank on every line, slices each row by those spans, and returns `([], [])` when the text has a tab, or when a header cell is empty or contains a 2+ space gap. A stale comment in `parse_tabular` about the parser splitting on 2+ spaces was updated. - `tests/test_transforms_tabular.py`: four tests for `parse_fixed_width` (the PID/USER/COMMAND shape, an empty cell next to a double-spaced cell, an ambiguous grid that must refuse, a row crossing a boundary) and one kubectl control through `parse_tabular`. Behaviour change to be aware of: a row with an empty cell is now parsed with `""` in that column when the grid is unambiguous (before, it counted as ragged and the table passed through). A table where one row does not line up with the others is still not parsed. Output for aligned tables (kubectl, `ps aux`, `df -h` fixtures) is unchanged. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output With the fix reverted (source file only, new tests kept), run from a copy of the tree with the 0.42.0 `_core.abi3.so`: ```text FAILED tests/test_transforms_tabular.py::test_parse_fixed_width_splits_on_column_positions_not_double_spaces FAILED tests/test_transforms_tabular.py::test_parse_fixed_width_keeps_an_empty_cell_empty FAILED tests/test_transforms_tabular.py::test_parse_fixed_width_never_shifts_a_value_when_the_grid_is_ambiguous FAILED tests/test_transforms_tabular.py::test_parse_fixed_width_refuses_a_row_that_crosses_a_column_boundary =================== 4 failed, 85 passed, 17 skipped in 2.40s =================== ``` With the fix, three files: ```text tests/test_transforms_tabular.py ....................................... [ 18%] ................................................sssssssssssssssss.. [ 49%] tests/test_transforms_content_detection.py .................... [ 58%] tests/test_transforms_content_router.py ................................ [ 73%] ..................................................ss.... [100%] ======================= 195 passed, 19 skipped in 5.54s ======================== ``` ```text $ ruff check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py All checks passed! $ ruff format --check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py 2 files already formatted ``` The skipped tests are the spreadsheet loader tests; they were skipped on `main` too. ## Real Behavior Proof - Environment: Ubuntu under WSL2 on Windows 11, Python 3.12.15, Python source of this branch on top of `main` `a2d4fe72`, `_core.abi3.so` copied from the `headroom_ai` 0.42.0 wheel into a copy of the tree (not into the branch), `HEADROOM_OFFLINE=1`. - Exact command / steps: ran `TabularCompressor().compress(text)` and `parse_tabular(text)` on the table from the issue (`PID USER COMMAND` header, rows `1 root /sbin/init`, `22 www-data nginx`, `4444 sysadmin1 sleep 60`, `5 root cron`, plus 40 rows `"%-5d root worker%d" % (100+i, i)`), once on `main` and once on this branch. - Observed result: on `main`, row 4444 parsed as `['4444', 'sysadmin1 sleep', '60']` and the output line was `60,4444,sysadmin1 sleep` under `{COMMAND,PID,USER}`. On this branch it parses as `['4444', 'sysadmin1', 'sleep 60']` and the output line is `sleep 60,4444,sysadmin1`. For the empty-cell shape (`NAME STATUS NOTE`, row `beta two words`) `main` returned `['beta', 'two', 'words']`; this branch returns `None` (passthrough). kubectl and `df -h` fixtures parse to the same columns as before. - Not tested: Windows or macOS runs; the full test suite (I ran the three tabular/detection/router test files); `mypy`; tables with wide (CJK) characters, where character positions and display columns differ, so such a table may be refused or grouped by code point; very large outputs for speed (the gutter scan is one pass over each line per column position, bounded by the longest line). ## Runtime Rollout Safety - Rollout-managed feature(s): None. This changes a parser used by the tabular compressor; no rollout-managed feature is touched. - Minimum rollout channel: N/A (not rollout-managed) - Stable/default behavior changed: Yes, narrowly. Fixed-width tables that did not line up used to be compressed with shifted values and now pass through; tables with an empty cell now parse with `""` instead of passing through as ragged. Aligned tables are unchanged. - Kill switch / disable path: the existing option that turns the tabular compressor off in the content router; no new switch. - Unsafe override required: No - Qualification impact: None expected; not checked against the qualification suite. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Mypy was not run (marked unchecked above). Documentation: none of the docs I know of describe the fixed-width splitting rule, so I did not change any (not searched exhaustively). - The ambiguous-grid case is a tradeoff: when every cell in a column is shorter than one double-spaced cell, the gap inside that cell looks like a column gutter. The parser refuses rather than guess. Produced with AI assistance (Claude Code) operated by me (HetCreep); the reproduction was run on my machine against a clone of the repository. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4541" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `b5d1bbf773bc6f4a9de8f6faec3af1e2b17f1562`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Re-reviewed |
||
|
|
8c360ca69f |
fix(tabular): keep interior blank CSV rows instead of dropping them (#4563)
## Description `parse_csv` (`headroom/transforms/tabular_ingest.py:77`) dropped every all-blank row, so separator rows inside a table vanished and the row count changed before the "lossless" csv-schema step, with no marker. `spreadsheet_ingest._rows_to_csv` trims only trailing blank rows. `parse_csv` now trims all-blank rows only at the start and the end and keeps interior ones as empty-cell rows; truly empty lines are still skipped, so a stray blank line cannot make the table ragged. Item 2 of #4546. Refs #4546 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/tabular_ingest.py`: rows with more than one cell or any non-blank cell are kept; all-blank rows are then trimmed from both ends in one pass (first and last non-blank index, one slice), and interior all-blank rows, whitespace-only ones included, are normalized to empty cells. - `tests/test_transforms_tabular.py`: `test_parse_csv_keeps_interior_blank_rows_and_trims_the_ends`, `test_parse_csv_normalizes_whitespace_only_interior_rows`, `test_parse_csv_trims_a_large_blank_prefix_in_linear_time`. Leading blank rows: unchanged. Before this PR they were dropped with every other blank row and the first non-blank row became the header; they still are. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # fail before (source at |
||
|
|
ddbbb07746 |
fix(tabular): keep tables whose columns share a name (#4329)
## Description
`to_records` zips the headers with each row into a dict, so two columns
with the same name make the later one overwrite the earlier one:
```python
records.append({headers[i]: padded[i] for i in range(width)})
```
Every value of the earlier column is lost, and the remaining values are
relabelled as if they were the dropped ones. `parse_tabular` checked
that every row has as many cells as the header, but not that the header
names are unique, so such tables reached SmartCrusher. The compressed
table is shorter than the original, so the `min_savings_chars` check
accepts it.
Repeated names are common in tool output:
- SQL join results (`psql --csv`, sqlite `.mode csv`) repeat `id`,
`name`, `created_at`.
- Markdown tables with two or more blank header cells collapse to one
`""` key.
For the reproduction in the issue, the first `id` and `name` columns
(`0..39`, `user_0..user_39`) were gone, and `100,org_0` was presented as
the `id,name` of row 0.
Closes #4277
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `parse_tabular` now rejects a table whose header names are not unique,
alongside the existing ragged-row check. A table that cannot be turned
into records without losing columns passes through unchanged, as a
ragged table already does (#1652).
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_transforms_tabular.py -q -k duplicate -v
collected 77 items / 73 deselected / 4 selected
tests/test_transforms_tabular.py::test_parse_tabular_rejects_duplicate_headers PASSED [ 25%]
tests/test_transforms_tabular.py::test_parse_tabular_rejects_duplicate_blank_markdown_headers PASSED [ 50%]
tests/test_transforms_tabular.py::test_compress_passes_through_a_table_with_duplicate_headers PASSED [ 75%]
tests/test_transforms_tabular.py::test_router_keeps_every_column_of_a_table_with_duplicate_headers PASSED [100%]
================ 4 passed, 73 deselected, 1 warning in 1.43s =================
```
The whole file reports `2 failed, 58 passed, 17 skipped`; both failures
(`test_verbose_markdown_compresses`,
`test_compact_unique_csv_passes_through`) reproduce on the unmodified
tree — they need `httpx`, which is not installed here.
The wider transform/router suites report the same failure count before
and after the change (`36 failed, 111 passed` → `36 failed, 115 passed`;
the four extra passes are the new tests).
## Real Behavior Proof
- Environment: headroom source checkout on Windows, Python 3.12; no
third-party deps installed
- Exact command / steps: the new tests build the table from the issue's
reproduction (`id,name,id,name` plus 40 rows) and run it through
`parse_tabular`, `TabularCompressor` and `ContentRouter`
- Observed result: before the fix the parser returned the table and the
records lost the first two columns; after the fix the table is not
tabular, so it passes through unchanged with every column intact
- Not tested: the SmartCrusher path itself (no `httpx` here), so the
end-to-end router test could not be shown to fail on the unmodified tree
in this environment
Three of the four new tests fail on the unmodified tree:
```text
$ pytest tests/test_transforms_tabular.py -q -k duplicate # fix reverted
FAILED ...::test_parse_tabular_rejects_duplicate_headers
FAILED ...::test_parse_tabular_rejects_duplicate_blank_markdown_headers
FAILED ...::test_compress_passes_through_a_table_with_duplicate_headers
3 failed, 1 passed, 73 deselected
```
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: a table with repeated column names is
no longer compressed; it passes through unchanged instead of being
compressed with a column silently dropped
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
- The router test is the one from the issue's reproduction. In this
environment it passes on the unmodified tree as well, because `httpx` is
missing and the SmartCrusher import fails before the lossy zip — so it
only bites where SmartCrusher is available. The other three tests cover
the loss directly and do fail without the fix.
- `to_records` itself is unchanged: it stays the raw zipper it is
documented to be, and the guard sits in `parse_tabular`, the gate every
tabular payload passes through.
- `opentelemetry` is not installed on this machine, so `import headroom`
fails and the suite cannot be collected as-is. The test runs above used
a minimal local stub for the `opentelemetry` API surface that
`headroom/observability` imports; no product code was changed to
accommodate it, and the stub is not part of this PR.
- `ruff` and `mypy` were not run: neither is installed here.
## Model Used
MiniMax M3.1 Flash Free via the opencode agent harness
(`minimax-m3.1-flash-free`)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4329"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
5cebb90db1 |
fix(tabular): keep empty records and separator-like Markdown body rows (#4582)
## Description Tabular ingest silently dropped real data rows: a CSV/TSV record whose fields are all empty (`,` or `"",""`) and any Markdown body row whose values look like a separator (`| --- | --- |`). Records disappeared with no marker. Closes #4279 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `parse_csv`: keep every record; skip only truly blank lines (empty row or a single whitespace-only cell). - `parse_markdown_table`: drop only the header separator directly under the header row; later separator-like rows are data. - Regression tests for CSV, TSV, quoted empty CSV record and Markdown `---` body row. ## Testing - [x] Unit tests added/updated ``` $ python -m pytest tests/test_transforms_tabular.py -q 2 failed, 98 passed, 5 skipped # the 2 failures (test_verbose_markdown_compresses, test_compact_unique_csv_passes_through) fail identically on main in this env; on main the new tests also fail (6 failed) $ ruff check / ruff format --check (touched files) All checks passed! / 2 files already formatted ``` ## Real Behavior Proof - Environment: Windows 11, Python 3, local checkout rebased on upstream/main - Exact command / steps: `parse_csv("a,b\n1,2\n,\n3,4")` and `parse_markdown_table("| a | b |\n|---|---|\n| 1 | 2 |\n| --- | --- |")` via the new tests - Observed result: CSV returns 3 rows including `['', '']`; Markdown returns `[['1','2'],['---','---']]` - Not tested: end-to-end proxy compression on real tabular payloads ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: stable - Stable/default behavior changed: yes, tabular parsers now keep rows they previously dropped - Kill switch / disable path: none needed; parser correctness fix - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
b66c9c7db6 |
fix(transforms): stop an embedded markdown table from dropping the rest of a tool result (#4216)
## Description A tool result that contains a markdown table is rewritten as JSON of the table's rows. Everything else in the result is dropped without a marker: headings, prose, and the output of other commands in the same Bash call. A later line with pipes is parsed as a row too. Root cause: `_try_detect_markdown_table` claims the whole payload as `TABULAR` (0.95) as soon as a header and separator pair appears anywhere in its first 50 lines. The tabular compressor then parses it with `parse_markdown_table`, which keeps only lines containing `|`. Its savings check compares the JSON against the full original text, so dropping the prose counts as compression. Markdown tables are not detected by the Rust side, which only returns `Tabular` for git status. On the native backend, plain text falls back to this same Python detector, and `parse_tabular` re-runs it. So this function decides the route on both backends. The fix treats content as a markdown table only when it is one: the header and separator open it and every remaining line is a row. A row is written in the header's leading-pipe style, is neither another separator nor a list item, and splits into exactly the header's cells the way the tabular parser splits it, so the records built from it can't truncate or pad it. A table with an escaped pipe (`\|`) is left alone, since the parser splits on it. A blank line ends a markdown table, so anything after one also disqualifies it. A table with anything else around it is no longer `TABULAR` and falls through to the remaining detectors, which is the issue's first expected behavior. CSV/TSV detection already requires 85% of the sampled lines to share a column count, so this brings markdown into line with it. Repeated headers, which records can't hold either, are handled on `main` by #4278's check in `parse_tabular`. This PR's regression test covers that check for markdown and whitespace-normalized collisions. Closes #4215 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/content_detector.py`: - `_try_detect_tabular` keeps a markdown verdict only when the new `_is_whole_markdown_table(content)` holds. That helper checks the whole content, not the 50-line sample, so a summary line after a long table also counts. - Each row must have the header's cell count (`_md_cell_count`, the same split as `parse_markdown_table`), and content with an escaped pipe is rejected. - A rejected markdown candidate still goes through CSV/TSV detection, so CSV that holds markdown in a quoted field stays tabular. A `|` delimiter is not accepted on that path, since those pipes belong to the markdown. - `headroom/transforms/tabular_ingest.py`: unchanged. During review I added a repeated-header check there, but #4278 landed the same check on `main` first, so I dropped mine after merging `main`. - `tests/test_transforms_tabular.py`: - detection is no longer `TABULAR` for ten shapes: the reporter's document, a heading line, a pipe line after the blank that ends the table, a pipe line above the table, text above a separator row, a line after a 60-row table, a list item straight after the rows, a note without the leading pipe, a list item after a pipe-less table, and a second separator row; - a whole table is still `TABULAR` with a trailing newline, surrounding blank lines, CRLF, or no outer pipes; - CSV with a markdown table inside a quoted field is still `TABULAR` with `,` and parses to 21 rows; - `_is_whole_markdown_table` needs a header of at least two cells followed by a separator; - `TabularCompressor` returns the reporter's document unchanged, and the router no longer picks `TABULAR` for it; - end to end, a 40-row table followed by an over-width row (with and without outer pipes) or by a row with an escaped pipe is not claimed as markdown and comes back from `TabularCompressor` unchanged; - end to end, a 40-row table with repeated headers (`| Name | Name |`, `| Name | Name |`, `name,name`, `name, name `) is not parsed into records and comes back from `TabularCompressor` unchanged (it exercises #4278's check and fails without it). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - run on the changed files - [x] Type checking passes (`mypy headroom`) - run on the changed module - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # the first commit's tests against main's content_detector.py FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[doc_with_table] FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[heading_line] FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[pipe_line_after_table] FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[pipe_line_before_table] FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[text_above_a_separator_row] FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[line_after_a_long_table] FAILED tests/test_transforms_tabular.py::test_compress_keeps_a_document_with_a_table_whole FAILED tests/test_transforms_tabular.py::test_router_does_not_route_a_document_with_a_table_to_tabular 8 failed, 3 passed, 73 deselected # (the 3 that pass are the whole-table cases, which must hold both ways) # the cases added for the review (second commit), against the first commit's detector ( |
||
|
|
ed3195beef |
fix(tabular): pass tables with repeated header names through verbatim (#4278)
## Description
`TabularCompressor` converts each row into a dict keyed by header name.
When a table repeats a header name, the later column overwrites the
earlier one and the earlier column's values are dropped from the
compressed output. This happens with SQL join output (`id,name,id,name`)
and with markdown tables that have two or more blank header cells. The
shorter result passes the `min_savings_chars` check, so the model reads
a table that is missing a column and has values under the wrong labels.
`parse_tabular` already returns `None` for ragged tables so they pass
through verbatim (#1652). This change applies the same rule when header
names are not unique.
Closes #4277
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/tabular_ingest.py`: `parse_tabular` returns
`None` when the header row has duplicate names, so the content passes
through unchanged.
- `tests/test_transforms_tabular.py`: covers a join-style CSV and a
markdown table with blank header cells at the `parse_tabular` level, and
a 40-row CSV through `TabularCompressor.compress`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
Without the fix (new tests only):
```text
$ python -m pytest -q tests/test_transforms_tabular.py -k duplicate
FAILED tests/test_transforms_tabular.py::test_parse_tabular_rejects_duplicate_headers[id,name,id,name\n1,alice,10,acme\n...]
FAILED tests/test_transforms_tabular.py::test_parse_tabular_rejects_duplicate_headers[| id | | |\n| --- | --- | --- |\n...]
FAILED tests/test_transforms_tabular.py::test_compress_passes_through_duplicate_headers
3 failed, 73 deselected
```
With the fix:
```text
$ python -m pytest -q tests/test_transforms_tabular.py
76 passed
$ python -m pytest -q <every test file that mentions tabular> # 14 files
1 failed, 281 passed, 4 skipped
```
The one failure,
`test_garbled_compression_fixes.py::test_banner_survives_with_live_kompress_model`,
needs a live Kompress model and fails the same way on unmodified `main`
in this environment.
```text
$ ruff check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
All checks passed!
$ ruff format --check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
2 files already formatted
```
## Real Behavior Proof
- Environment: Linux aarch64, Python 3.12, `main` at
|
||
|
|
a42fb2b9a1 |
fix(router): preserve human-readable Git status losslessly (#4020)
## Description Addresses the still-reproducible Git status corruption in #3652. The aligned `ls`/`ps`/`df` portions are already protected in current main; this PR does not claim them as new fixes. Human-readable Git status is still classified as prose and can lose staged/unstaged/untracked paths through Kompress. This is a narrow content-preservation bug fix, not the separate policy proposal to exclude all `Bash` outputs by default. ## Type of Change - [x] Bug fix - [x] Documentation update ## Changes Made - Recognize the standard English branch/detached-HEAD header plus Git status sections/entries as structured tabular content. - Preserve this non-rectangular `git_status` format verbatim instead of forcing it into CSV or a prose model. - Keep Python detection and the native Rust content/live-zone route aligned; Rust detection uses iterator-based scans without collection/cloning of the input. - Leave ordinary prose, bullet lists, and other Bash output on their existing routes. - Guard Git status before native model classification and before external CSV/wildcard compressor dispatch; retain the locked native Magika production chain. - Replace the stale `PLAIN_TEXT` status expectation with `format=git_status` and `parse_tabular(status) is None`; keep prose/git-log negative cases. - Document the English-format scope and lossless/no-savings tradeoff. ## Testing - [x] Unit tests pass (`pytest`), including the consumer-visible lossless regression. - [x] Native regression tests, changed-file lint, configured pre-commit mypy and manual production runtime checks pass. ### Test Output ```text Review correction: Python router/external-dispatch regressions: 121 passed, 2 skipped Existing tabular parser/detection suite: 73 passed in 7.76s Native Git-status regressions: 3 passed Final mixed-content correction (router, external dispatch, tabular suites): 184 passed, 2 skipped in 52.14s ``` The mandatory pre-push `make ci-precheck` passed: workspace Rust fmt/clippy/tests, native extension build/import verification, the complete configured Python gate, and Commitlint. The native gate used the explicitly installed CPU `libonnxruntime.so.1.26.0` via `ORT_DYLIB_PATH`, with cached-model assertions enabled. ```text [build_rust_extension] headroom._core build + install + verify: OK ================== 177 passed, 4 skipped, 1 warning in 27.72s ================== ✅ ci-precheck PASSED — safe to push. ``` ```sh python -m pytest -q tests/test_transforms_content_router.py cargo test -p headroom-core git_status cargo test -p headroom-core ordinary_bullet ``` Python content-router suite: **84 passed, 2 skipped**, with an existing Hugging Face deprecation warning. Native focused regressions: **2 Git status tests passed**, **1 ordinary-bullet negative passed**. The native extension was rebuilt and the build/import verification passed. The final deterministic consumer regression uses a ready lossy prose compressor so a cold model cannot accidentally mask missing protection; it asserts direct and tool-result output bytes, not internal call counts. The deterministic regression fails against the unchanged baseline with missing status rows. After rebasing onto current main `1ad92f0cbc3dea27676e253588100f64a2af6994`, ran the status regression plus main's six small-MCP result tests: **7 passed**, and reran the native Git status cases: **2 passed**. Changed-file Ruff and formatting passed. The installed pre-commit hooks, including `mypy headroom` and Commitlint, passed for this commit. Full-tree Ruff is locally blocked by executable filesystem permissions on five unchanged OAuth2 plugin files (tracked 100644); they are not modified. A separate full dev-environment mypy run found an unchanged optional LangChain `truthy-function` error; the configured isolated pre-commit environment passed. No unrelated source/permissions were changed. ## Real Behavior Proof - Environment: Linux x86_64, CPython 3.13.14; unchanged `dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; actual Git fixture, real content router and Anthropic token counter, rebuilt Rust extension, **real ONNX Kompress fully preloaded before each run**, no mocked compression backend in this smoke. - Exact command / steps: initialize a throwaway Git repo; commit 120 files named `modified path NNN α.txt`; modify those, stage 120 new files named `staged path NNN β.txt`, and add 120 untracked files named `untracked path NNN γ.txt`. Capture: ```sh LC_ALL=C git -c core.quotePath=false -c color.status=false status --branch ``` Load the actual model with `KompressCompressor().preload(allow_download=True)`, run `ContentRouter(ContentRouterConfig()).compress(status)`, then run the same router's `apply` on an Anthropic `Bash` tool-use/tool-result pair using `AnthropicProvider(warn=False).get_token_counter('claude-sonnet-4-5')` and model limit 200000. Compare output bytes to the same captured status. - Same input in both runs: **12,469 bytes**, SHA256 `78ed6a39f2f994159099f8d4c0fe052ca09a37fa6d9f6248e8c5ebb655b19a32`. - Before: detected as `text`; direct and Bash tool-result routes both returned **10,748 bytes**, neither byte-exact; transform `router:tool_result:text`. - Observed result: after the fix, detected as `tabular`; both routes returned **12,469 bytes**, both byte-exact; `router:noop`. - External-compressor review proof against published head `3b035027`: a real Git repository with 120 untracked paths produced the same **3,415-byte** input (SHA256 `e0164d25f3f47428686d102a73018d0e00bcda53ff9674e658aa7128c038896b`). With an operational row-trimming plugin selected for `text/csv` and `*/*`, direct and tool-output paths both shrank to **318 bytes** before; both preserve all **3,415 bytes** after. The same plugin still compresses ordinary CSV. This separate plugin smoke uses the production Tiktoken counter and disables Kompress to isolate external dispatch. - Final mixed-content review: on published `539669c6`, appended JSON selected `MIXED` and changed both direct and Bash tool-result bytes; the fenced-code example was already preserved. The strategy selector now honors `git_status` before mixed splitting. Both appended JSON and fenced code preserve the complete payload on both routes with a ready lossy compressor available. - Not tested: localized Git status formats, every platform-specific Git variant, or a paid upstream model response. The recognizer deliberately covers standard English status output. The local real compression path was exercised after model warmup; the initial cold-model no-op was not treated as proof. ## Runtime Rollout Safety - Rollout-managed feature(s): none; content-routing data-preservation fix. - Minimum rollout channel: stable; no rollout-managed feature is enabled. - Stable/default behavior changed: recognized Git status is no longer compressed lossily, sacrificing savings for path/state integrity. - Kill switch / disable path: existing global Bash exclusion can bypass compression; no new flag. - Unsafe override required: no. - Qualification impact: Python/Rust route parity for the recognized status format only. - Rollback path: revert this commit; no persistent schema migration. Global Bash exclusions remain unchanged. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Existing tabular lossless route reused; compression docs updated. - [x] Consumer regression and native positive/negative tests added. - [x] `CHANGELOG.md` not edited; Conventional Commit title supplied. This PR completes the remaining Git status defect, but intentionally does not close #3652's separate Bash-default policy request. Remote CI is reported by GitHub, not assumed from local tests. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
6c9aef1dec |
fix(transforms): detect fixed-width command output as tabular and keep it out of Kompress (#3786)
## Description Part of #3652. **The problem.** Commands like `ls -la`, `ps aux`, `df -h`, `docker ps` and `kubectl get pods` print a table whose columns are lined up with spaces. Headroom did not recognise that shape, so it treated the output as plain text and sent it to Kompress, the ML compressor meant for prose. Kompress drops words it thinks are unimportant. In a table that means some rows lose a field (an owner, a size, a date) and others keep it, and nothing in the output says which rows were changed. The agent then works from a file listing that is quietly wrong. **Still happening on `main`.** I ran the example from the issue on `main` at |
||
|
|
e8a510982a |
fix(spreadsheet): stop .xls integer rendering fabricating digits above 2^53 (#3704)
Closes #3695. Follow-up to #3616, which fixed a real bug but introduced a narrower one at the top of the numeric range. ## Problem `_xls_cell` converted every whole-valued number to `int`. xlrd hands back an IEEE-754 double, and above 2^53 consecutive integers are no longer representable — so `int()` renders the double's *exact* value, which is not the number the user typed: ``` user typed: 123456789012345678 stored as: 1.2345678901234568e+17 int() shows: 123456789012345680 <- last two digits fabricated ``` This text goes to an agent. An ID column rendered as `123456789012345680` reads as an exact identifier; it isn't. That is the main concern — the `.xlsx`/`.xls` divergence #3616 set out to remove is the secondary one (both loaders showed the float repr before, and only diverge above 2^53 now). ## Fix One bounded condition: ```python if kind == xlrd.XL_CELL_NUMBER and float(value).is_integer() and abs(value) < 2**53: ``` Below 2^53 the double is exact, so `int()` is lossless and the `12.0 -> 12` win that motivated #3616 is preserved. At or above, fall through to the float repr — which signals approximation honestly *and* restores `_load_xlsx` parity. `< 2**53` rather than `<=`: 2^53 is representable but 2^53+1 is not, so the strict bound is the conventional "integers are exact below here" line. ## Verification New parametrized tests call `_xls_cell` directly with a stub cell (no `.xls` fixture needed). Verified fail-before / pass-after by reverting only the source file and keeping the tests: ``` 4 failed, 3 passed <- e.g. assert isinstance(123456789012345680, float) ``` After the fix: **52 passed** in `tests/test_transforms_tabular.py`. ## Also in this PR - Docstring note that sub-second precision is dropped (`xldate_as_tuple` returns whole seconds, so `12:34:56.5` renders `12:34:56` where openpyxl keeps microseconds). - Removed the stale `# pragma: no cover` on the `.xls` dispatch — it referenced a comment #3616 deleted, on a line the tests now exercise (confirmed via coverage: the line is no longer in the missing set). ## Note for reviewers These tests **silently skip** without `xlrd` installed. CI is fine (`[dev]` pins `xlrd>=2.0.1`), but a local venv without it shows `s` rather than failures. Still open in #3695 and deliberately out of scope here: replacing the `xlwt` dev dependency with a committed binary fixture, and covering `XL_CELL_ERROR`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com> |
||
|
|
1a7c5bbf52 |
fix(spreadsheet): stop inventing digits for .xls cells above 2**53 (#3700)
## Description
Bounds the `int()` conversion added by #3616 to the range where a double
is exact, so a legacy `.xls` cell stops inventing digits (#3695).
`_xls_cell` rendered any integral double with `int(value)`. xlrd hands
back an IEEE-754 double, and above 2**53 consecutive integers are not
representable, so the conversion prints the double's *exact* value
rather than the number the sheet held:
```text
typed 123456789012345678 stored 1.2345678901234568e+17 rendered 123456789012345680
```
Two trailing digits, fabricated, delivered to an agent in a form that
reads as an exact identifier. Before #3616 both loaders rendered the
float and simply disagreed about `12.0` vs `12`; after it they agree on
small whole numbers and disagree again -- in the worse direction -- at
the top of the range, which is also where the PR's own goal ("render an
.xls cell the way the .xlsx loader renders it") stops holding.
Closes #3695
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/spreadsheet_ingest.py`: the numeric branch
becomes `if kind == xlrd.XL_CELL_NUMBER and float(value).is_integer()
and abs(value) <= 2**53:`, so values past the exactly-representable
range`return value` and render as the float. The comment records why the
bound exists so it is not "simplified" away again.
- `tests/test_transforms_tabular.py`: adds
`test_load_xls_and_xlsx_agree_above_the_exact_integer_range`, written
against the same xlwt/xlrd fixture style the three existing `.xls` tests
use. It pins both halves of the requirement: `12` still renders without
a decimal point (the #3616 win survives the bound), and
1.2345678901234568e+17 renders as `repr()` of the double rather than an
invented integer.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Type checking was run on the changed module rather than the whole
package, so that line means `mypy --config-file pyproject.toml
headroom/transforms/spreadsheet_ingest.py`, not a full-package pass.
### Test Output
Red, with the fix removed and the new test kept (this is also the state
of `main` for a cell in that range):
```text
$ python -m pytest tests/test_transforms_tabular.py -k exact_integer_range -q
E AssertionError: assert '123456789012345680' == '1.2345678901234568e+17'
E
E - 1.2345678901234568e+17
E ? - ^^^^
E + 123456789012345680
FAILED tests/test_transforms_tabular.py::test_load_xls_and_xlsx_agree_above_the_exact_integer_range
1 failed, 45 deselected in 4.10s
```
Green, with the fix:
```text
$ python -m pytest tests/test_transforms_tabular.py -q
46 passed in 5.48s
$ uvx ruff@0.16.7 check headroom/transforms/spreadsheet_ingest.py tests/test_transforms_tabular.py
All checks passed!
$ uvx ruff@0.16.7 format --check headroom/transforms/spreadsheet_ingest.py tests/test_transforms_tabular.py
2 files already formatted
$ uvx mypy@1.14.1 --config-file pyproject.toml headroom/transforms/spreadsheet_ingest.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11 (10.0.26200) x64, Python 3.12.13, working tree
at `origin/main`
|
||
|
|
261796f9be |
fix(spreadsheet): render an .xls cell the way the .xlsx loader renders it (#3616)
## Description
`_load_xls` reads a legacy workbook with `sheet.row_values(i)`, which
hands back xlrd's **raw storage** rather than the value. The `.xlsx`
loader goes through openpyxl and hands back the value. So the two
loaders disagree about the same workbook.
Measured on `main` (
|
||
|
|
334317db5d |
fix(tabular): pass a table through when one cell is past the csv field limit (#3618)
## Description
`csv.field_size_limit` is **128 KB per cell** by default. One pasted
document, log excerpt or base64 blob in a column goes past it, and the
stdlib reader then raises for the **whole text**, not for the row:
```python
>>> csv.field_size_limit()
131072
>>> list(csv.reader(io.StringIO("id,body\nl," + "x" * 200_000)))
_csv.Error: field larger than field limit (131072)
```
`parse_csv` does not catch it, so `TabularCompressor.compress` raises
out of a content type it advertises support for:
```
>>> TabularCompressor().compress("id,title,body\nl,short,ok\n2,long," + "x" * 200_000)
Error: field larger than field limit (131072)
```
The module already has a policy for a table it cannot use —
`parse_tabular` returns `None` for a ragged one (#1652) and the text
passes through verbatim to the plain-text route. An oversized cell is
the same situation and should take the same exit.
This is not hypothetical for a context compressor: the content it sees
is tool output and pasted material, which is exactly where a single 128
KB+ cell comes from. Two other projects that ingest CSVs hit it and say
so in their own code — `onyx/backend/onyx/utils/csv_utils.py`:
*"real-world data (long descriptions, pasted docs, base64 blobs)
routinely exceeds"*, and `skyvern/forge/sdk/workflow/models/block.py`.
Both raise the limit process-globally, which is a reasonable call for an
application and the wrong call for a library running inside someone
else's process.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/tabular_ingest.py`: `parse_csv` catches
`csv.Error` and returns `([], [])`, which makes `parse_tabular` return
`None` and the compressor pass the content through unchanged. The
docstring records why the limit is not raised instead.
- No process-global state is touched, so a host application's own `csv`
behaviour is unaffected.
- Nothing else changes: a table that parses today parses identically,
and the `[]` return reuses the empty-input path that was already there.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Two cases in `tests/test_transforms_tabular.py`, next to the
ragged-table ones they mirror:
- `test_parse_csv_gives_up_on_a_cell_past_the_field_size_limit`
- `test_compress_passes_through_a_table_with_an_oversized_cell`
`pytest` is unchecked because the suite does not run in my checkout, not
because it failed — see **Not tested**.
### Test Output
```text
$ ruff check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
All checks passed!
$ ruff format --check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
2 files already formatted
$ python -m pytest tests/test_transforms_tabular.py
tests/test_transforms_tabular.py:24: in <module>
from headroom.transforms.content_router import (
headroom/transforms/error_detection.py:41: in <module>
from headroom._core import (
E ModuleNotFoundError: No module named 'headroom._core'
```
The collection error is identical with my changes stashed: the file
imports the compiled Rust extension and there is no Rust toolchain on
this machine.
## Real Behavior Proof
- Environment: macOS 15 (arm64), CPython 3.12, no Rust toolchain.
- Exact command / steps: call
`headroom.transforms.tabular_ingest.parse_csv` and
`TabularCompressor().compress` on `"id,title,body\nl,short,ok\n2,long,"
+ "x" * 200_000`. That module imports cleanly on its own; only the test
file's other imports pull in `headroom._core`.
- Observed result: `parse_csv` raised `field larger than field limit`
before the change and returns `([], [])` after it, and `compress` passes
the text through byte-identical (output below).
```text
# before (source change stashed)
parse_csv -> Error : field larger than field limit (131072)
# after
parse_csv -> ([], [])
compress -> passthrough, byte-identical
```
- Not tested: `pytest tests/test_transforms_tabular.py` and `mypy
headroom`, both of which need `headroom._core`; CI builds it, so the two
new cases run there. Also **not** tested end to end through
`ContentRouter`, for the same reason — I verified the public
`TabularCompressor` / `parse_tabular` / `parse_csv` surface, and I do
not claim to know whether the router's dispatch would have swallowed the
exception or propagated it.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. `tabular_ingest` is not behind a
rollout channel.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: only for text that raises today. A
table that parses now parses identically; a table that raised now passes
through verbatim and is still compressed by the plain-text route.
- Kill switch / disable path: none needed; revert the commit.
- Unsafe override required: no.
- Qualification impact: none — no golden output can have pinned an
exception from this path.
- Rollback path: `git revert`; the change is four lines inside
`parse_csv`.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (no
documented behaviour changes)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes (cannot
run the suite here — see **Not tested**)
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
If you would rather compress these tables than pass them through, the
other option is a `csv.field_size_limit` bump. I did not take it because
it is process-global and headroom is imported into someone else's
process: raising it there silently changes how that application's own
`csv` code behaves. If you want it anyway, a set/restore around the
parse keeps the blast radius to this module, at the cost of a race
between concurrent parses on separate threads — which is the trade-off
skyvern's comment describes when it chose to set the limit once at
import instead.
|
||
|
|
10d8f15bc7 |
fix(transforms): drop trailing empty spreadsheet rows and stray CR in CSV render (#3254)
## Description
`_rows_to_csv` in the binary spreadsheet ingester does not do what its
own docstring ("dropping fully empty trailing rows") says, and emits a
stray carriage return:
```python
def _rows_to_csv(rows: list[list[object]]) -> str:
"""Render rows to CSV text, dropping fully empty trailing rows."""
buf = io.StringIO()
writer = csv.writer(buf) # default lineterminator = "\r\n"
for row in rows: # trailing empty rows are NOT dropped
writer.writerow(["" if cell is None else cell for cell in row])
return buf.getvalue().strip("\n") # strips "\n" only, leaving a dangling "\r"
```
Two problems, both hit in normal use:
1. **Trailing empty rows are not dropped.** openpyxl's `iter_rows` walks
the sheet's *used range*, which routinely extends past the last data row
(leftover formatting, a cleared cell), so a real sheet commonly ends in
`(None, None, …)` tuples. Those get written out as blank `,` rows.
2. **A dangling `\r` is left on the output.** `csv.writer` defaults to a
`\r\n` terminator, and `.strip("\n")` removes only the `\n`, so the last
line ends in a bare `\r` (and every undropped trailing empty row leaves
`,\r` residue).
Concretely, a sheet `[["Name","Age"], ["Alice","30"], (None,None),
(None,None), (None,None)]` renders to
`'Name,Age\r\nAlice,30\r\n,\r\n,\r\n,\r'` instead of
`'Name,Age\nAlice,30'`. This noisy text then flows through tabular
detection → SmartCrusher and on to the model — extra rows and
control-character residue for every spreadsheet with a trailing
used-range.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/spreadsheet_ingest.py`: drop the trailing run of
fully-empty rows before writing (a row is empty when every cell is
`None` or blank-after-strip), and set `csv.writer(...,
lineterminator="\n")` so `.strip("\n")` leaves no dangling `\r`.
Interior empty rows are preserved (only the trailing run is dropped); a
fully empty sheet renders to `""`.
- `tests/test_transforms_tabular.py`: added
`test_rows_to_csv_drops_trailing_empty_rows_and_has_no_dangling_cr`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_transforms_tabular.py -> 38 passed, 2 skipped in 1.76s
(the new test FAILS on pre-fix code, verified via git stash)
uvx ruff@0.16.2 check headroom/transforms/spreadsheet_ingest.py tests/test_transforms_tabular.py -> All checks passed!
uvx mypy@1.20.2 headroom/transforms/spreadsheet_ingest.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps:
`_rows_to_csv([["Name","Age"],["Alice","30"],[None,None],["","
"],[None,None]])` returned `'Name,Age\r\nAlice,30\r\n,\r\n,\r...'`
before the fix and `'Name,Age\nAlice,30'` after;
`[["a","b"],[None,None],["c","d"],[None,None]]` → `'a,b\n,\nc,d'`
(interior empty kept, trailing dropped); an all-empty sheet → `''`.
- Observed result: trailing empty rows dropped, no stray `\r`, interior
rows and data unchanged.
- Not tested: exercised `_rows_to_csv` directly rather than
round-tripping a physical `.xlsx` (the existing
`test_load_and_compress_xlsx` covers the openpyxl path; `splitlines()`
there is agnostic to the terminator change).
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is the SDK-side spreadsheet
ingestion adapter, not a rollout-channel-gated runtime feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: only the incorrect output is
corrected — trailing empty rows and the stray `\r` are gone; data rows,
interior empty rows, and sheet skipping are unchanged. The line
terminator in the intermediate CSV text is now `\n` (it feeds text
compression, not a strict CSV consumer).
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
c7665ca088 |
fix(transforms): pass through ragged tables instead of misaligning columns (#1713)
## Description
Issue #1652 reports the proxy's compression layer surfacing an
"impossible mixed" status line — a row combining fields from two
different rows of a version-status table (Docker row `0.42.4 → 0.43.0
update available` blended with WSL row `0.42.4 → 0.42.4 up-to-date`).
The reporter's follow-up refined the claim: the stored canonical content
was intact, but the compression path presents a lossier view that
invites exactly this misattribution.
There is a concrete mechanism for that in the tabular bridge:
`parse_tabular` (`headroom/transforms/tabular_ingest.py`) hands parsed
rows to `to_records`, which **silently pads/truncates every row to the
header width**. For ragged tables — rows whose cell count differs from
the header row, exactly what mixed-shape status tables like the
reporter's produce (`✓` and `-` placeholder cells change the token count
per row) — this shifts values under the wrong column before SmartCrusher
compaction. The compressed output can then state column/value pairings
the original never contained.
Fix: `parse_tabular` now rejects ragged tables (any row width ≠ header
width) and returns `None`, so the content passes through verbatim, per
the issue's requirement that a lossy summary "must not create impossible
mixed facts". Aligned tables compress exactly as before. The Rust
`log_template` Drain miner was also examined; its template rendering
only emits tokens that are constant across all rows of a run, so no
defect was found there and it is left untouched.
Fixes #1652
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/tabular_ingest.py`: `parse_tabular` returns
`None` when any parsed row's cell count differs from the header count,
instead of letting `to_records` pad/truncate rows into the wrong
columns. `TabularCompressor.compress` then takes its existing
pass-through branch (`was_modified=False`).
- `tests/test_transforms_tabular.py`: three new tests — ragged
fixed-width table rejected (reproducing the issue's rtk version-status
shape), ragged markdown table rejected, and end-to-end
`TabularCompressor.compress` pass-through of a ragged table.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_transforms_tabular.py -q
39 passed
$ ruff check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
All checks passed!
$ ruff format --check headroom/transforms/tabular_ingest.py tests/test_transforms_tabular.py
2 files already formatted
$ mypy headroom --ignore-missing-imports
Success: no issues found (note-level messages only)
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13, local checkout branched from
`upstream/main` (
|
||
|
|
d789a7c528 |
feat(transforms): tabular + spreadsheet (.xlsx/.xls) compression (#1128)
## Description Adds a content-type-aware path for **tabular data** — CSV/TSV, markdown tables, fixed-width text, and binary `.xlsx`/`.xls` spreadsheets — by routing them through the existing, battle-tested `SmartCrusher` instead of letting them fall through to `PLAIN_TEXT → Kompress`. The pipeline already compressed tables losslessly when handed a JSON array of records. This wires up the missing front door: detect tabular text (and ingest binary spreadsheets), convert to JSON records, and reuse `SmartCrusher.crush()`. No new compression algorithm. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Detection** (`content_detector.py`): new `ContentType.TABULAR` + `_try_detect_tabular()` for CSV/TSV, markdown tables, and fixed-width columns. Ordered after search/log (which also look "delimited") and before code, with a prose-rejection guard so it never steals `file:line:content` search output, `key: value` logs, or sentences with incidental commas. Rust backend returns `plain_text` for unknown types and the router already falls back to the Python detector, so **no Rust change**. - **Bridge** (`tabular_ingest.py`): stdlib parsers + `to_records()` + a `TabularCompressor` that parses → JSON records → `SmartCrusher` (lossless `csv-schema` first; lossy row-drop with reversible `<<ccr:HASH>>` markers stays SmartCrusher's built-in fallback). Only adopts a result when it actually saves bytes. - **Spreadsheets** (`spreadsheet_ingest.py`): `.xlsx`/`.xls` → per-sheet CSV text at the SDK boundary. Optional deps (`pip install headroom-ai[spreadsheet]`) fail loudly with an install hint, never silently degrade. - **Routing** (`content_router.py`): `CompressionStrategy.TABULAR`, `enable_tabular_compressor` flag, lazy getter, apply branch, strategy maps, Kompress fallback eligibility. - **SDK** (`compress.py`): `compress_spreadsheet(path, ...)` helper (one message per sheet). - **Packaging** (`pyproject.toml`): new `[spreadsheet]` extra; `openpyxl` added to `[dev]` so the xlsx path is exercised in CI. - **Docs/demo**: `examples/tabular_compression_demo.py` + README entry. ### Design note: lossless-only Compact, all-unique tables with no query yield ~0 savings — this is correct, not a bug. SmartCrusher returns `skip:unique_entities_no_signal` and won't drop unique rows without a duplicate/relevance signal. Real wins come from verbose/redundant tables and query-driven selection. A pressure-driven lossy row sampler was considered and intentionally not added. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_transforms_tabular.py -q collected 20 items tests/test_transforms_tabular.py .................... [100%] ============================== 20 passed in 7.15s ============================== $ ruff check headroom/transforms/tabular_ingest.py headroom/transforms/spreadsheet_ingest.py All checks passed! $ mypy headroom/transforms/tabular_ingest.py headroom/transforms/spreadsheet_ingest.py Success: no issues found in 2 source files ``` `tests/test_transforms_tabular.py` (20 tests): detection true positives + no-misroute negatives (search/log/JSON/prose), parser units (incl. fixed-width), the CSV→SmartCrusher bridge, router routing + disable flag, and `.xlsx` ingestion (skipif openpyxl missing) + error paths. `spreadsheet_ingest` 100% / `tabular_ingest` 90% line coverage. ## Real Behavior Proof - **Environment:** local checkout of `feat/tabular-compression`, Python 3.x, `pip install -e ".[dev]"`. - **Exact command / steps:** `python examples/tabular_compression_demo.py` (no API key required). - **Observed result:** ```text === Raw tabular text (ContentRouter, char-level) === compact unique CSV strat=tabular chars 1306 -> 1072 ( 17.9% saved) redundant CSV strat=tabular chars 2661 -> 1350 ( 49.3% saved) verbose markdown strat=tabular chars 2019 -> 1580 ( 21.7% saved) === Full pipeline (real tokenizer) === redundant CSV tokens 768 -> 394 ( 48.7% saved) === Binary spreadsheet (.xlsx) === 2-sheet workbook tokens 1092 -> 683 ( 37.5% saved) ``` - **Not tested:** legacy `.xls` binary path (needs optional `xlrd` + binary fixture; `# pragma: no cover`); base64-embedded `.xlsx` inside multimodal blocks (out of scope, noted as a follow-up). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG/version are intentionally untouched: this repo uses **release-please**, which bumps the version and CHANGELOG via automated `chore: release main` PRs, not per-feature PRs. - The `.xls` path is `# pragma: no cover` (legacy, needs optional `xlrd` + a binary fixture). - Follow-up (out of scope): base64-embedded `.xlsx` inside tool-result/multimodal blocks; porting tabular parsers into the Rust core for parity. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |