fix(transforms): stop an embedded markdown table from dropping the rest of a tool result (#4216)

## Description

A tool result that contains a markdown table is rewritten as JSON of the
table's rows. Everything else in the result is dropped without a marker:
headings, prose, and the output of other commands in the same Bash call.
A later line with pipes is parsed as a row too.

Root cause: `_try_detect_markdown_table` claims the whole payload as
`TABULAR` (0.95) as soon as a header and separator pair appears anywhere
in its first 50 lines. The tabular compressor then parses it with
`parse_markdown_table`, which keeps only lines containing `|`. Its
savings check compares the JSON against the full original text, so
dropping the prose counts as compression. Markdown tables are not
detected by the Rust side, which only returns `Tabular` for git status.
On the native backend, plain text falls back to this same Python
detector, and `parse_tabular` re-runs it. So this function decides the
route on both backends.

The fix treats content as a markdown table only when it is one: the
header and separator open it and every remaining line is a row. A row is
written in the header's leading-pipe style, is neither another separator
nor a list item, and splits into exactly the header's cells the way the
tabular parser splits it, so the records built from it can't truncate or
pad it. A table with an escaped pipe (`\|`) is left alone, since the
parser splits on it. A blank line ends a markdown table, so anything
after one also disqualifies it. A table with anything else around it is
no longer `TABULAR` and falls through to the remaining detectors, which
is the issue's first expected behavior. CSV/TSV detection already
requires 85% of the sampled lines to share a column count, so this
brings markdown into line with it. Repeated headers, which records can't
hold either, are handled on `main` by #4278's check in `parse_tabular`.
This PR's regression test covers that check for markdown and
whitespace-normalized collisions.

Closes #4215

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/content_detector.py`:
- `_try_detect_tabular` keeps a markdown verdict only when the new
`_is_whole_markdown_table(content)` holds. That helper checks the whole
content, not the 50-line sample, so a summary line after a long table
also counts.
- Each row must have the header's cell count (`_md_cell_count`, the same
split as `parse_markdown_table`), and content with an escaped pipe is
rejected.
- A rejected markdown candidate still goes through CSV/TSV detection, so
CSV that holds markdown in a quoted field stays tabular. A `|` delimiter
is not accepted on that path, since those pipes belong to the markdown.
- `headroom/transforms/tabular_ingest.py`: unchanged. During review I
added a repeated-header check there, but #4278 landed the same check on
`main` first, so I dropped mine after merging `main`.
- `tests/test_transforms_tabular.py`:
- detection is no longer `TABULAR` for ten shapes: the reporter's
document, a heading line, a pipe line after the blank that ends the
table, a pipe line above the table, text above a separator row, a line
after a 60-row table, a list item straight after the rows, a note
without the leading pipe, a list item after a pipe-less table, and a
second separator row;
- a whole table is still `TABULAR` with a trailing newline, surrounding
blank lines, CRLF, or no outer pipes;
- CSV with a markdown table inside a quoted field is still `TABULAR`
with `,` and parses to 21 rows;
- `_is_whole_markdown_table` needs a header of at least two cells
followed by a separator;
- `TabularCompressor` returns the reporter's document unchanged, and the
router no longer picks `TABULAR` for it;
- end to end, a 40-row table followed by an over-width row (with and
without outer pipes) or by a row with an escaped pipe is not claimed as
markdown and comes back from `TabularCompressor` unchanged;
- end to end, a 40-row table with repeated headers (`| Name | Name |`,
`| Name | Name |`, `name,name`, `name, name `) is not parsed into
records and comes back from `TabularCompressor` unchanged (it exercises
#4278's check and fails without it).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`) - run on the changed files
- [x] Type checking passes (`mypy headroom`) - run on the changed module
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# the first commit's tests against main's content_detector.py
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[doc_with_table]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[heading_line]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[pipe_line_after_table]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[pipe_line_before_table]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[text_above_a_separator_row]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[line_after_a_long_table]
FAILED tests/test_transforms_tabular.py::test_compress_keeps_a_document_with_a_table_whole
FAILED tests/test_transforms_tabular.py::test_router_does_not_route_a_document_with_a_table_to_tabular
8 failed, 3 passed, 73 deselected
# (the 3 that pass are the whole-table cases, which must hold both ways)

# the cases added for the review (second commit), against the first commit's detector (22f4e1b6)
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[list_item_right_after_rows]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[note_without_leading_pipe]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[list_item_after_pipeless_rows]
FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[second_separator_row]
FAILED tests/test_transforms_tabular.py::test_csv_with_markdown_in_a_quoted_field_is_still_csv
5 failed, 12 passed, 73 deselected
# (against main the file no longer collects, since it imports the new _is_whole_markdown_table)

# the review's over-width rows (third commit), against 1fa07094
FAILED tests/test_transforms_tabular.py::test_table_with_a_row_records_cannot_hold_is_kept_whole[over_width_row_pipeless]
FAILED tests/test_transforms_tabular.py::test_table_with_a_row_records_cannot_hold_is_kept_whole[over_width_row_piped]
FAILED tests/test_transforms_tabular.py::test_table_with_a_row_records_cannot_hold_is_kept_whole[escaped_pipe]
3 failed, 1 passed, 90 deselected

# the review's repeated headers (fourth commit), against 477e9cca
FAILED tests/test_transforms_tabular.py::test_table_with_repeated_headers_is_kept_whole[markdown]
FAILED tests/test_transforms_tabular.py::test_table_with_repeated_headers_is_kept_whole[markdown_whitespace]
FAILED tests/test_transforms_tabular.py::test_table_with_repeated_headers_is_kept_whole[csv]
FAILED tests/test_transforms_tabular.py::test_table_with_repeated_headers_is_kept_whole[csv_whitespace]
4 failed, 94 deselected
# with main merged in (ed0f4531), the same four pass on #4278's check alone, and fail again if that check is removed

# at ed0f4531
tests/test_transforms_tabular.py: 84 passed, 17 skipped

# mutations of the fix: each must fail at least one test
M1 no whole-table check                    KILLED (16)
M2 any row instead of all rows             KILLED (8)
M3 drop separator-at-line-2                KILLED (1)
M4 drop two-column header                  KILLED (1)
M5 check only the first 50 lines           KILLED (1)
M6 ignore leading-pipe style               KILLED (1)
M7 allow a second separator row            KILLED (1)
M8 allow list items                        KILLED (1)
M9 no delimited fallback                   KILLED (1)
M10 accept pipe as delimiter               KILLED (8)
M11 any width instead of the header's      KILLED (2)
M12 allow escaped pipes                    KILLED (1)

# every test file that touches detection, tabular parsing, mixed-content splitting or the router (37 files)
main 3fef15d3:  6 failed, 1011 passed, 23 skipped
ed0f4531:       5 failed, 1037 passed, 23 skipped

ruff 0.17.0 check / format --check (changed files): All checks passed! / 3 files already formatted
mypy 1.19.1 headroom/transforms/content_detector.py: no errors in the changed module
```

The 5 failures are the same tests with and without this change:
`test_offline_egress_chokepoint.py` OTLP/Langfuse cases, which fail on
my machine either way. The sixth failure on `main` was
`test_content_router_ml_stage_deadline.py::test_saturated_remote_calls_fail_open_without_queueing_their_payloads`,
which passed 4 of 4 times on its own, so it's a timing flake under
parallel load. The 7 mypy errors are the Windows-only `os.fchmod` /
`os.O_*` ones in `fileperms.py` and `astgrep.py`, identical on `main`.

## Real Behavior Proof

- Environment: Windows 11, Python 3.11.9, headroom from this branch
(ed0f4531) and from `main` (76133cff, 0.41.1-dev), pure-Python detection
backend (the Windows default).
- Exact command / steps: the reporter's script from #4215, unchanged
(`seed_proxy_env_defaults()`, `create_app(_proxy_config_from_env())`,
then `proxy.anthropic_pipeline.apply(...)` on a Bash `tool_result`
holding a heading, prose, a table, a pipe-bearing bullet and another
command's output), run with `HEADROOM_OFFLINE=1 PYTHONPATH=.`.
- Observed result: on `main` the script prints the first block below, so
the whole tool result became the table's rows, including the bullet.
With this branch it prints the second block, followed by the original
text unchanged.
- Not tested: the native (Rust/Magika) detection backend locally; it
reaches this function through the plain-text fallback, and the router
test pins the Python backend. Linux/macOS, and real Claude Code traffic.

```text
# main (76133cff)
transforms: ['router:tool_result:tabular']
chars: 2318 -> 344
prose kept: False | other output kept: False
[{"Step":"Build","Owner":"CI","Command":"`make build`"},{"Step":"Test","Owner":"CI","Command":"`make test`"},{"Step":"Tag","Owner":"Maintainer","Command":"`git tag vX.Y.Z`"},{"Step":"Publish","Owner":"Maintainer","Command":"`make publish`"},{"Step":"- Create an issue: `tracker new --type=bug","Owner":"feature","Command":"task --priority=2`"}]

# this branch
transforms: ['router:noop']
chars: 2318 -> 2318
prose kept: True | other output kept: True
# Release checklist
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None
- Minimum rollout channel: N/A
- Stable/default behavior changed: Narrowly. Content that holds a
markdown table plus anything else is no longer routed to the tabular
compressor. A bare table routes and compresses exactly as before
(measured: `tabular`, 2019 -> 1580 chars on both `main` and this
branch).
- Kill switch / disable path: None added. `--compressor` /
`HEADROOM_COMPRESSORS` without `tabular` still disables the compressor.
- Unsafe override required: No
- Qualification impact: None
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A: the
docs list "markdown tables" as tabular, which still holds for content
that is a table)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- One trade-off to flag: a table wrapped in a
`<returncode>0</returncode>` + `<output>` envelope. The router detects
on the unwrapped body, but the compressor receives the wrapped text. On
`main` it was compressed (2065 -> 1580 chars) and the envelope,
including the return code, was dropped the same silent way. With this
change it passes through verbatim (2065 -> 2065). Compressing the body
inside the envelope would recover those savings; I left that out to keep
this fix to the detector.
- Also lossless in the same direction, from reading
`_RELEASABLE_READ_TYPES` rather than a separate test: a `Read` of a
markdown file that has a table plus prose is no longer a releasable
`TABULAR` read, so it stays protected like other plain text.
- Relation to #3514: that PR takes the richer route for the same
underlying line-drop in `parse_markdown_table`, splitting a document
into table and prose segments so each table is compacted and the text is
carried through. It is still in changes-requested. This PR is the small
lossless fix for #4215 in the meantime and doesn't change
`tabular_ingest.py`. If segmenting lands, `_is_whole_markdown_table` is
the one gate to relax so documents with tables reach it again.
- Out of scope: compacting only the table region and keeping the
surrounding lines (the issue's second option), which is what #3514 does.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4216"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Ayush Kumar Jha
2026-10-28 17:56:27 -05:00
committed by GitHub
co-authored by JD Davis
parent 18a588b3f0
commit b66c9c7db6
2 changed files with 201 additions and 1 deletions
+42 -1
View File
@@ -770,11 +770,52 @@ def _try_detect_tabular(content: str) -> DetectionResult | None:
md_result = _try_detect_markdown_table(lines)
if md_result:
return md_result
# The tabular compressor replaces the whole payload with the table's
# rows, so a table with a heading, prose or other output around it is
# not tabular content: every line outside the table would be dropped.
if _is_whole_markdown_table(content):
return md_result
# It can still be CSV holding markdown in a quoted field. Its pipes
# belong to that markdown, so they are never the delimiter here.
delimited = _try_detect_delimited(lines)
if delimited and delimited.metadata.get("delimiter") != "|":
return delimited
return None
return _try_detect_delimited(lines)
def _is_whole_markdown_table(content: str) -> bool:
"""True when ``content`` is one markdown table and nothing else.
A blank line ends a markdown table, so every line after the header and
separator has to be a row: written in the header's leading-pipe style,
and neither another separator nor a list item. A note or a second table
straight after the rows means the content is not just the table.
Every row must also split into exactly the header's cells, the way the
tabular parser splits it, or the records built from it would truncate or
pad the row. The parser does not unescape ``\\|``, so a table that uses it
is left alone too.
"""
if "\\|" in content:
return False
rows = content.strip().split("\n")
if len(rows) < 3 or not _is_md_separator(rows[1]):
return False
cols = _md_cell_count(rows[0])
if cols < 2:
return False
leading_pipe = rows[0].lstrip().startswith("|")
return all(
_md_cell_count(row) == cols
and row.lstrip().startswith("|") == leading_pipe
and not _is_md_separator(row)
and not _FIXED_WIDTH_LIST_RE.match(row)
for row in rows[2:]
)
# Fixed-width (space-aligned) command output: `ls -l`, `ps aux`, `df -h`,
# `docker ps`, `kubectl get`. These rows have no delimiter, so without this
# check they fall through to PLAIN_TEXT and the prose compressor drops fields
+159
View File
@@ -17,6 +17,7 @@ from headroom.transforms.content_detector import (
ContentType,
DetectionResult,
_is_md_separator,
_is_whole_markdown_table,
_looks_like_prose,
_try_detect_delimited,
_try_detect_markdown_table,
@@ -95,6 +96,143 @@ def test_does_not_misroute_to_tabular(content: str, expected: ContentType) -> No
assert detect_content_type(content).content_type is expected
# Detection: a markdown table inside other content (#4215) -------------------
_PROSE = "\n".join(
f"Paragraph {i}: this sentence is ordinary prose that an agent needs to read verbatim."
for i in range(12)
)
_TABLE = (
"| Step | Owner | Command |\n"
"|------|-------|---------|\n"
"| Build | CI | `make build` |\n"
"| Test | CI | `make test` |\n"
"| Tag | Maintainer | `git tag vX.Y.Z` |\n"
"| Publish | Maintainer | `make publish` |"
)
_PIPE_BULLET = "- Create an issue: `tracker new --type=bug|feature|task --priority=2`"
_PIPELESS_TABLE = (
"Step | Owner | Command\n--- | --- | ---\nBuild | CI | make build\nTest | CI | make test"
)
# `cat` of a doc with a table, plus another command's output in the same result.
_DOC_WITH_TABLE = (
f"# Release checklist\n\n{_PROSE}\n\n{_TABLE}\n\n{_PIPE_BULLET}\n\n{_PROSE}\n"
"----\n412 other-command-output.txt\n"
)
@pytest.mark.parametrize(
"content",
[
_DOC_WITH_TABLE,
f"Results:\n{_TABLE}",
# A blank line ends the table; a later line with pipes is not a row.
f"{_TABLE}\n\n{_PIPE_BULLET}",
# A line with pipes above the table is not its header.
f"{_PIPE_BULLET}\n{_TABLE}",
# Nor is text sitting on top of a separator row.
f"Title\n|---|---|\n{_TABLE}",
# Past the 50 lines the detector samples.
f"{_verbose_markdown(60)}\n\nTotal: 60 users",
# Lines with as many pipe fields as the header, straight after the rows:
# a list item, a note in another style, a second separator.
f"{_TABLE}\n{_PIPE_BULLET}",
f"{_TABLE}\nNote: pick one of bug|feature|task",
f"{_PIPELESS_TABLE}\n- Next: bug|feature|task",
f"{_TABLE}\n|---|---|---|\n| Ship | CI | `make ship` |",
],
ids=[
"doc_with_table",
"heading_line",
"pipe_line_after_table",
"pipe_line_before_table",
"text_above_a_separator_row",
"line_after_a_long_table",
"list_item_right_after_rows",
"note_without_leading_pipe",
"list_item_after_pipeless_rows",
"second_separator_row",
],
)
def test_markdown_table_inside_other_content_is_not_tabular(content: str) -> None:
assert detect_content_type(content).content_type is not ContentType.TABULAR
@pytest.mark.parametrize(
"content",
[f"{_TABLE}\n", f"\n\n{_TABLE}\n\n", _TABLE.replace("\n", "\r\n"), _PIPELESS_TABLE],
ids=["trailing_newline", "surrounding_blank_lines", "crlf", "pipeless"],
)
def test_whole_markdown_table_is_still_tabular(content: str) -> None:
result = detect_content_type(content)
assert result.content_type is ContentType.TABULAR
assert result.metadata.get("format") == "markdown"
_LONG_ROWS = [(f"step_{i}", f"ok_{i}") for i in range(40)]
@pytest.mark.parametrize(
"content",
[
"Name | Value\n--- | ---\n"
+ "\n".join(f"{a} | {b}" for a, b in _LONG_ROWS)
+ "\nImportant: use x | y | DO_NOT_DELETE",
"| Name | Value |\n| --- | --- |\n"
+ "\n".join(f"| {a} | {b} |" for a, b in _LONG_ROWS)
+ "\n| Important: use x | y | DO_NOT_DELETE |",
# Splitting on the escaped pipe gives the header's three cells, but the
# literal `|` would be lost.
"| Name | Value | Note |\n| --- | --- | --- |\n"
+ "\n".join(f"| {a} | {b} | none |" for a, b in _LONG_ROWS)
+ "\n| `a \\| b` | DO_NOT_DELETE |",
],
ids=["over_width_row_pipeless", "over_width_row_piped", "escaped_pipe"],
)
def test_table_with_a_row_records_cannot_hold_is_kept_whole(content: str) -> None:
# Not claimed as markdown. Aligned rows may still read as fixed-width, which
# splits only on whitespace runs and passes ragged rows through.
assert detect_content_type(content).metadata.get("format") != "markdown"
result = TabularCompressor().compress(content)
assert not result.was_modified
assert result.compressed == content
_KEEP_ROWS = range(40)
@pytest.mark.parametrize(
"content",
[
"| Name | Name |\n| --- | --- |\n"
+ "\n".join(f"| DO_NOT_DELETE_{i} | value_{i} |" for i in _KEEP_ROWS),
# Equal once the parser strips each cell.
"| Name | Name |\n| --- | --- |\n"
+ "\n".join(f"| DO_NOT_DELETE_{i} | value_{i} |" for i in _KEEP_ROWS),
"name,name\n" + "\n".join(f"DO_NOT_DELETE_{i},value_{i}" for i in _KEEP_ROWS),
"name, name \n" + "\n".join(f"DO_NOT_DELETE_{i},value_{i}" for i in _KEEP_ROWS),
],
ids=["markdown", "markdown_whitespace", "csv", "csv_whitespace"],
)
def test_table_with_repeated_headers_is_kept_whole(content: str) -> None:
# Records key cells by header, so the second column would overwrite the first.
assert parse_tabular(content) is None
result = TabularCompressor().compress(content)
assert not result.was_modified
assert result.compressed == content
def test_csv_with_markdown_in_a_quoted_field_is_still_csv() -> None:
rows = "\n".join(f"{i},value_{i}" for i in range(1, 20))
csv = f'id,notes\n{rows}\n20,"| x | y |\n| --- | --- |\n| foo | bar |"\n21,value_21'
result = detect_content_type(csv)
assert result.content_type is ContentType.TABULAR
assert result.metadata.get("delimiter") == ","
headers, data, _fmt = parse_tabular(csv)
assert headers == ["id", "notes"]
assert len(data) == 21
# Detection: fixed-width command output (#3652) ------------------------------
@@ -213,6 +351,13 @@ def test_is_md_separator_needs_two_columns() -> None:
assert not _is_md_separator("| a | b |") # cells must be dashes
def test_whole_markdown_table_needs_a_piped_header_then_a_separator() -> None:
assert _is_whole_markdown_table("| a | b |\n|---|---|\n| 1 | 2 |")
assert not _is_whole_markdown_table("| a | b |\n| 1 | 2 |\n| 3 | 4 |")
assert not _is_whole_markdown_table("Title\n|---|---|\na | b")
assert not _is_whole_markdown_table("Title\n|---|---|\nfoo\nbar")
def test_markdown_table_needs_multiple_columns() -> None:
# Valid separator below, but the header is a single column -> not a table.
assert _try_detect_markdown_table(["x|", "---|---", "y|"]) is None
@@ -445,6 +590,14 @@ def test_compact_unique_csv_passes_through() -> None:
assert result.compressed == CSV
def test_compress_keeps_a_document_with_a_table_whole() -> None:
# Only the table's rows used to come back, as JSON, with the prose and the
# other command's output silently gone.
result = TabularCompressor().compress(_DOC_WITH_TABLE)
assert not result.was_modified
assert result.compressed == _DOC_WITH_TABLE
def test_non_tabular_passes_through_unmodified() -> None:
# Unparseable prose returns the original content untouched.
text = "just a normal paragraph here"
@@ -514,6 +667,12 @@ def test_router_keeps_ls_output_verbatim(monkeypatch) -> None:
assert result.strategy_used is CompressionStrategy.TABULAR
def test_router_does_not_route_a_document_with_a_table_to_tabular(monkeypatch) -> None:
_record_kompress_calls(monkeypatch)
result = ContentRouter(ContentRouterConfig()).compress(_DOC_WITH_TABLE)
assert result.strategy_used is not CompressionStrategy.TABULAR
def test_router_does_not_kompress_a_ragged_csv(monkeypatch) -> None:
calls = _record_kompress_calls(monkeypatch)
csv = "id,name,city\n" + "\n".join(f"{i},user_{i},city_{i % 5}" for i in range(30))