mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(transforms): stop an embedded markdown table from dropping the rest of a tool result (#4216)
## Description A tool result that contains a markdown table is rewritten as JSON of the table's rows. Everything else in the result is dropped without a marker: headings, prose, and the output of other commands in the same Bash call. A later line with pipes is parsed as a row too. Root cause: `_try_detect_markdown_table` claims the whole payload as `TABULAR` (0.95) as soon as a header and separator pair appears anywhere in its first 50 lines. The tabular compressor then parses it with `parse_markdown_table`, which keeps only lines containing `|`. Its savings check compares the JSON against the full original text, so dropping the prose counts as compression. Markdown tables are not detected by the Rust side, which only returns `Tabular` for git status. On the native backend, plain text falls back to this same Python detector, and `parse_tabular` re-runs it. So this function decides the route on both backends. The fix treats content as a markdown table only when it is one: the header and separator open it and every remaining line is a row. A row is written in the header's leading-pipe style, is neither another separator nor a list item, and splits into exactly the header's cells the way the tabular parser splits it, so the records built from it can't truncate or pad it. A table with an escaped pipe (`\|`) is left alone, since the parser splits on it. A blank line ends a markdown table, so anything after one also disqualifies it. A table with anything else around it is no longer `TABULAR` and falls through to the remaining detectors, which is the issue's first expected behavior. CSV/TSV detection already requires 85% of the sampled lines to share a column count, so this brings markdown into line with it. Repeated headers, which records can't hold either, are handled on `main` by #4278's check in `parse_tabular`. This PR's regression test covers that check for markdown and whitespace-normalized collisions. Closes #4215 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/content_detector.py`: - `_try_detect_tabular` keeps a markdown verdict only when the new `_is_whole_markdown_table(content)` holds. That helper checks the whole content, not the 50-line sample, so a summary line after a long table also counts. - Each row must have the header's cell count (`_md_cell_count`, the same split as `parse_markdown_table`), and content with an escaped pipe is rejected. - A rejected markdown candidate still goes through CSV/TSV detection, so CSV that holds markdown in a quoted field stays tabular. A `|` delimiter is not accepted on that path, since those pipes belong to the markdown. - `headroom/transforms/tabular_ingest.py`: unchanged. During review I added a repeated-header check there, but #4278 landed the same check on `main` first, so I dropped mine after merging `main`. - `tests/test_transforms_tabular.py`: - detection is no longer `TABULAR` for ten shapes: the reporter's document, a heading line, a pipe line after the blank that ends the table, a pipe line above the table, text above a separator row, a line after a 60-row table, a list item straight after the rows, a note without the leading pipe, a list item after a pipe-less table, and a second separator row; - a whole table is still `TABULAR` with a trailing newline, surrounding blank lines, CRLF, or no outer pipes; - CSV with a markdown table inside a quoted field is still `TABULAR` with `,` and parses to 21 rows; - `_is_whole_markdown_table` needs a header of at least two cells followed by a separator; - `TabularCompressor` returns the reporter's document unchanged, and the router no longer picks `TABULAR` for it; - end to end, a 40-row table followed by an over-width row (with and without outer pipes) or by a row with an escaped pipe is not claimed as markdown and comes back from `TabularCompressor` unchanged; - end to end, a 40-row table with repeated headers (`| Name | Name |`, `| Name | Name |`, `name,name`, `name, name `) is not parsed into records and comes back from `TabularCompressor` unchanged (it exercises #4278's check and fails without it). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - run on the changed files - [x] Type checking passes (`mypy headroom`) - run on the changed module - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # the first commit's tests against main's content_detector.py FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[doc_with_table] FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[heading_line] FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[pipe_line_after_table] FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[pipe_line_before_table] FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[text_above_a_separator_row] FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[line_after_a_long_table] FAILED tests/test_transforms_tabular.py::test_compress_keeps_a_document_with_a_table_whole FAILED tests/test_transforms_tabular.py::test_router_does_not_route_a_document_with_a_table_to_tabular 8 failed, 3 passed, 73 deselected # (the 3 that pass are the whole-table cases, which must hold both ways) # the cases added for the review (second commit), against the first commit's detector (22f4e1b6) FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[list_item_right_after_rows] FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[note_without_leading_pipe] FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[list_item_after_pipeless_rows] FAILED tests/test_transforms_tabular.py::test_markdown_table_inside_other_content_is_not_tabular[second_separator_row] FAILED tests/test_transforms_tabular.py::test_csv_with_markdown_in_a_quoted_field_is_still_csv 5 failed, 12 passed, 73 deselected # (against main the file no longer collects, since it imports the new _is_whole_markdown_table) # the review's over-width rows (third commit), against1fa07094FAILED tests/test_transforms_tabular.py::test_table_with_a_row_records_cannot_hold_is_kept_whole[over_width_row_pipeless] FAILED tests/test_transforms_tabular.py::test_table_with_a_row_records_cannot_hold_is_kept_whole[over_width_row_piped] FAILED tests/test_transforms_tabular.py::test_table_with_a_row_records_cannot_hold_is_kept_whole[escaped_pipe] 3 failed, 1 passed, 90 deselected # the review's repeated headers (fourth commit), against477e9ccaFAILED tests/test_transforms_tabular.py::test_table_with_repeated_headers_is_kept_whole[markdown] FAILED tests/test_transforms_tabular.py::test_table_with_repeated_headers_is_kept_whole[markdown_whitespace] FAILED tests/test_transforms_tabular.py::test_table_with_repeated_headers_is_kept_whole[csv] FAILED tests/test_transforms_tabular.py::test_table_with_repeated_headers_is_kept_whole[csv_whitespace] 4 failed, 94 deselected # with main merged in (ed0f4531), the same four pass on #4278's check alone, and fail again if that check is removed # ated0f4531tests/test_transforms_tabular.py: 84 passed, 17 skipped # mutations of the fix: each must fail at least one test M1 no whole-table check KILLED (16) M2 any row instead of all rows KILLED (8) M3 drop separator-at-line-2 KILLED (1) M4 drop two-column header KILLED (1) M5 check only the first 50 lines KILLED (1) M6 ignore leading-pipe style KILLED (1) M7 allow a second separator row KILLED (1) M8 allow list items KILLED (1) M9 no delimited fallback KILLED (1) M10 accept pipe as delimiter KILLED (8) M11 any width instead of the header's KILLED (2) M12 allow escaped pipes KILLED (1) # every test file that touches detection, tabular parsing, mixed-content splitting or the router (37 files) main3fef15d3: 6 failed, 1011 passed, 23 skippeded0f4531: 5 failed, 1037 passed, 23 skipped ruff 0.17.0 check / format --check (changed files): All checks passed! / 3 files already formatted mypy 1.19.1 headroom/transforms/content_detector.py: no errors in the changed module ``` The 5 failures are the same tests with and without this change: `test_offline_egress_chokepoint.py` OTLP/Langfuse cases, which fail on my machine either way. The sixth failure on `main` was `test_content_router_ml_stage_deadline.py::test_saturated_remote_calls_fail_open_without_queueing_their_payloads`, which passed 4 of 4 times on its own, so it's a timing flake under parallel load. The 7 mypy errors are the Windows-only `os.fchmod` / `os.O_*` ones in `fileperms.py` and `astgrep.py`, identical on `main`. ## Real Behavior Proof - Environment: Windows 11, Python 3.11.9, headroom from this branch (ed0f4531) and from `main` (76133cff, 0.41.1-dev), pure-Python detection backend (the Windows default). - Exact command / steps: the reporter's script from #4215, unchanged (`seed_proxy_env_defaults()`, `create_app(_proxy_config_from_env())`, then `proxy.anthropic_pipeline.apply(...)` on a Bash `tool_result` holding a heading, prose, a table, a pipe-bearing bullet and another command's output), run with `HEADROOM_OFFLINE=1 PYTHONPATH=.`. - Observed result: on `main` the script prints the first block below, so the whole tool result became the table's rows, including the bullet. With this branch it prints the second block, followed by the original text unchanged. - Not tested: the native (Rust/Magika) detection backend locally; it reaches this function through the plain-text fallback, and the router test pins the Python backend. Linux/macOS, and real Claude Code traffic. ```text # main (76133cff) transforms: ['router:tool_result:tabular'] chars: 2318 -> 344 prose kept: False | other output kept: False [{"Step":"Build","Owner":"CI","Command":"`make build`"},{"Step":"Test","Owner":"CI","Command":"`make test`"},{"Step":"Tag","Owner":"Maintainer","Command":"`git tag vX.Y.Z`"},{"Step":"Publish","Owner":"Maintainer","Command":"`make publish`"},{"Step":"- Create an issue: `tracker new --type=bug","Owner":"feature","Command":"task --priority=2`"}] # this branch transforms: ['router:noop'] chars: 2318 -> 2318 prose kept: True | other output kept: True # Release checklist ``` ## Runtime Rollout Safety - Rollout-managed feature(s): None - Minimum rollout channel: N/A - Stable/default behavior changed: Narrowly. Content that holds a markdown table plus anything else is no longer routed to the tabular compressor. A bare table routes and compresses exactly as before (measured: `tabular`, 2019 -> 1580 chars on both `main` and this branch). - Kill switch / disable path: None added. `--compressor` / `HEADROOM_COMPRESSORS` without `tabular` still disables the compressor. - Unsafe override required: No - Qualification impact: None - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A: the docs list "markdown tables" as tabular, which still holds for content that is a table) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - One trade-off to flag: a table wrapped in a `<returncode>0</returncode>` + `<output>` envelope. The router detects on the unwrapped body, but the compressor receives the wrapped text. On `main` it was compressed (2065 -> 1580 chars) and the envelope, including the return code, was dropped the same silent way. With this change it passes through verbatim (2065 -> 2065). Compressing the body inside the envelope would recover those savings; I left that out to keep this fix to the detector. - Also lossless in the same direction, from reading `_RELEASABLE_READ_TYPES` rather than a separate test: a `Read` of a markdown file that has a table plus prose is no longer a releasable `TABULAR` read, so it stays protected like other plain text. - Relation to #3514: that PR takes the richer route for the same underlying line-drop in `parse_markdown_table`, splitting a document into table and prose segments so each table is compacted and the text is carried through. It is still in changes-requested. This PR is the small lossless fix for #4215 in the meantime and doesn't change `tabular_ingest.py`. If segmenting lands, `_is_whole_markdown_table` is the one gate to relax so documents with tables reach it again. - Out of scope: compacting only the table region and keeping the surrounding lines (the issue's second option), which is what #3514 does. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4216" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JD Davis
parent
18a588b3f0
commit
b66c9c7db6
@@ -770,11 +770,52 @@ def _try_detect_tabular(content: str) -> DetectionResult | None:
|
||||
|
||||
md_result = _try_detect_markdown_table(lines)
|
||||
if md_result:
|
||||
return md_result
|
||||
# The tabular compressor replaces the whole payload with the table's
|
||||
# rows, so a table with a heading, prose or other output around it is
|
||||
# not tabular content: every line outside the table would be dropped.
|
||||
if _is_whole_markdown_table(content):
|
||||
return md_result
|
||||
# It can still be CSV holding markdown in a quoted field. Its pipes
|
||||
# belong to that markdown, so they are never the delimiter here.
|
||||
delimited = _try_detect_delimited(lines)
|
||||
if delimited and delimited.metadata.get("delimiter") != "|":
|
||||
return delimited
|
||||
return None
|
||||
|
||||
return _try_detect_delimited(lines)
|
||||
|
||||
|
||||
def _is_whole_markdown_table(content: str) -> bool:
|
||||
"""True when ``content`` is one markdown table and nothing else.
|
||||
|
||||
A blank line ends a markdown table, so every line after the header and
|
||||
separator has to be a row: written in the header's leading-pipe style,
|
||||
and neither another separator nor a list item. A note or a second table
|
||||
straight after the rows means the content is not just the table.
|
||||
|
||||
Every row must also split into exactly the header's cells, the way the
|
||||
tabular parser splits it, or the records built from it would truncate or
|
||||
pad the row. The parser does not unescape ``\\|``, so a table that uses it
|
||||
is left alone too.
|
||||
"""
|
||||
if "\\|" in content:
|
||||
return False
|
||||
rows = content.strip().split("\n")
|
||||
if len(rows) < 3 or not _is_md_separator(rows[1]):
|
||||
return False
|
||||
cols = _md_cell_count(rows[0])
|
||||
if cols < 2:
|
||||
return False
|
||||
leading_pipe = rows[0].lstrip().startswith("|")
|
||||
return all(
|
||||
_md_cell_count(row) == cols
|
||||
and row.lstrip().startswith("|") == leading_pipe
|
||||
and not _is_md_separator(row)
|
||||
and not _FIXED_WIDTH_LIST_RE.match(row)
|
||||
for row in rows[2:]
|
||||
)
|
||||
|
||||
|
||||
# Fixed-width (space-aligned) command output: `ls -l`, `ps aux`, `df -h`,
|
||||
# `docker ps`, `kubectl get`. These rows have no delimiter, so without this
|
||||
# check they fall through to PLAIN_TEXT and the prose compressor drops fields
|
||||
|
||||
@@ -17,6 +17,7 @@ from headroom.transforms.content_detector import (
|
||||
ContentType,
|
||||
DetectionResult,
|
||||
_is_md_separator,
|
||||
_is_whole_markdown_table,
|
||||
_looks_like_prose,
|
||||
_try_detect_delimited,
|
||||
_try_detect_markdown_table,
|
||||
@@ -95,6 +96,143 @@ def test_does_not_misroute_to_tabular(content: str, expected: ContentType) -> No
|
||||
assert detect_content_type(content).content_type is expected
|
||||
|
||||
|
||||
# Detection: a markdown table inside other content (#4215) -------------------
|
||||
|
||||
_PROSE = "\n".join(
|
||||
f"Paragraph {i}: this sentence is ordinary prose that an agent needs to read verbatim."
|
||||
for i in range(12)
|
||||
)
|
||||
_TABLE = (
|
||||
"| Step | Owner | Command |\n"
|
||||
"|------|-------|---------|\n"
|
||||
"| Build | CI | `make build` |\n"
|
||||
"| Test | CI | `make test` |\n"
|
||||
"| Tag | Maintainer | `git tag vX.Y.Z` |\n"
|
||||
"| Publish | Maintainer | `make publish` |"
|
||||
)
|
||||
_PIPE_BULLET = "- Create an issue: `tracker new --type=bug|feature|task --priority=2`"
|
||||
_PIPELESS_TABLE = (
|
||||
"Step | Owner | Command\n--- | --- | ---\nBuild | CI | make build\nTest | CI | make test"
|
||||
)
|
||||
# `cat` of a doc with a table, plus another command's output in the same result.
|
||||
_DOC_WITH_TABLE = (
|
||||
f"# Release checklist\n\n{_PROSE}\n\n{_TABLE}\n\n{_PIPE_BULLET}\n\n{_PROSE}\n"
|
||||
"----\n412 other-command-output.txt\n"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"content",
|
||||
[
|
||||
_DOC_WITH_TABLE,
|
||||
f"Results:\n{_TABLE}",
|
||||
# A blank line ends the table; a later line with pipes is not a row.
|
||||
f"{_TABLE}\n\n{_PIPE_BULLET}",
|
||||
# A line with pipes above the table is not its header.
|
||||
f"{_PIPE_BULLET}\n{_TABLE}",
|
||||
# Nor is text sitting on top of a separator row.
|
||||
f"Title\n|---|---|\n{_TABLE}",
|
||||
# Past the 50 lines the detector samples.
|
||||
f"{_verbose_markdown(60)}\n\nTotal: 60 users",
|
||||
# Lines with as many pipe fields as the header, straight after the rows:
|
||||
# a list item, a note in another style, a second separator.
|
||||
f"{_TABLE}\n{_PIPE_BULLET}",
|
||||
f"{_TABLE}\nNote: pick one of bug|feature|task",
|
||||
f"{_PIPELESS_TABLE}\n- Next: bug|feature|task",
|
||||
f"{_TABLE}\n|---|---|---|\n| Ship | CI | `make ship` |",
|
||||
],
|
||||
ids=[
|
||||
"doc_with_table",
|
||||
"heading_line",
|
||||
"pipe_line_after_table",
|
||||
"pipe_line_before_table",
|
||||
"text_above_a_separator_row",
|
||||
"line_after_a_long_table",
|
||||
"list_item_right_after_rows",
|
||||
"note_without_leading_pipe",
|
||||
"list_item_after_pipeless_rows",
|
||||
"second_separator_row",
|
||||
],
|
||||
)
|
||||
def test_markdown_table_inside_other_content_is_not_tabular(content: str) -> None:
|
||||
assert detect_content_type(content).content_type is not ContentType.TABULAR
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"content",
|
||||
[f"{_TABLE}\n", f"\n\n{_TABLE}\n\n", _TABLE.replace("\n", "\r\n"), _PIPELESS_TABLE],
|
||||
ids=["trailing_newline", "surrounding_blank_lines", "crlf", "pipeless"],
|
||||
)
|
||||
def test_whole_markdown_table_is_still_tabular(content: str) -> None:
|
||||
result = detect_content_type(content)
|
||||
assert result.content_type is ContentType.TABULAR
|
||||
assert result.metadata.get("format") == "markdown"
|
||||
|
||||
|
||||
_LONG_ROWS = [(f"step_{i}", f"ok_{i}") for i in range(40)]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"content",
|
||||
[
|
||||
"Name | Value\n--- | ---\n"
|
||||
+ "\n".join(f"{a} | {b}" for a, b in _LONG_ROWS)
|
||||
+ "\nImportant: use x | y | DO_NOT_DELETE",
|
||||
"| Name | Value |\n| --- | --- |\n"
|
||||
+ "\n".join(f"| {a} | {b} |" for a, b in _LONG_ROWS)
|
||||
+ "\n| Important: use x | y | DO_NOT_DELETE |",
|
||||
# Splitting on the escaped pipe gives the header's three cells, but the
|
||||
# literal `|` would be lost.
|
||||
"| Name | Value | Note |\n| --- | --- | --- |\n"
|
||||
+ "\n".join(f"| {a} | {b} | none |" for a, b in _LONG_ROWS)
|
||||
+ "\n| `a \\| b` | DO_NOT_DELETE |",
|
||||
],
|
||||
ids=["over_width_row_pipeless", "over_width_row_piped", "escaped_pipe"],
|
||||
)
|
||||
def test_table_with_a_row_records_cannot_hold_is_kept_whole(content: str) -> None:
|
||||
# Not claimed as markdown. Aligned rows may still read as fixed-width, which
|
||||
# splits only on whitespace runs and passes ragged rows through.
|
||||
assert detect_content_type(content).metadata.get("format") != "markdown"
|
||||
result = TabularCompressor().compress(content)
|
||||
assert not result.was_modified
|
||||
assert result.compressed == content
|
||||
|
||||
|
||||
_KEEP_ROWS = range(40)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"content",
|
||||
[
|
||||
"| Name | Name |\n| --- | --- |\n"
|
||||
+ "\n".join(f"| DO_NOT_DELETE_{i} | value_{i} |" for i in _KEEP_ROWS),
|
||||
# Equal once the parser strips each cell.
|
||||
"| Name | Name |\n| --- | --- |\n"
|
||||
+ "\n".join(f"| DO_NOT_DELETE_{i} | value_{i} |" for i in _KEEP_ROWS),
|
||||
"name,name\n" + "\n".join(f"DO_NOT_DELETE_{i},value_{i}" for i in _KEEP_ROWS),
|
||||
"name, name \n" + "\n".join(f"DO_NOT_DELETE_{i},value_{i}" for i in _KEEP_ROWS),
|
||||
],
|
||||
ids=["markdown", "markdown_whitespace", "csv", "csv_whitespace"],
|
||||
)
|
||||
def test_table_with_repeated_headers_is_kept_whole(content: str) -> None:
|
||||
# Records key cells by header, so the second column would overwrite the first.
|
||||
assert parse_tabular(content) is None
|
||||
result = TabularCompressor().compress(content)
|
||||
assert not result.was_modified
|
||||
assert result.compressed == content
|
||||
|
||||
|
||||
def test_csv_with_markdown_in_a_quoted_field_is_still_csv() -> None:
|
||||
rows = "\n".join(f"{i},value_{i}" for i in range(1, 20))
|
||||
csv = f'id,notes\n{rows}\n20,"| x | y |\n| --- | --- |\n| foo | bar |"\n21,value_21'
|
||||
result = detect_content_type(csv)
|
||||
assert result.content_type is ContentType.TABULAR
|
||||
assert result.metadata.get("delimiter") == ","
|
||||
headers, data, _fmt = parse_tabular(csv)
|
||||
assert headers == ["id", "notes"]
|
||||
assert len(data) == 21
|
||||
|
||||
|
||||
# Detection: fixed-width command output (#3652) ------------------------------
|
||||
|
||||
|
||||
@@ -213,6 +351,13 @@ def test_is_md_separator_needs_two_columns() -> None:
|
||||
assert not _is_md_separator("| a | b |") # cells must be dashes
|
||||
|
||||
|
||||
def test_whole_markdown_table_needs_a_piped_header_then_a_separator() -> None:
|
||||
assert _is_whole_markdown_table("| a | b |\n|---|---|\n| 1 | 2 |")
|
||||
assert not _is_whole_markdown_table("| a | b |\n| 1 | 2 |\n| 3 | 4 |")
|
||||
assert not _is_whole_markdown_table("Title\n|---|---|\na | b")
|
||||
assert not _is_whole_markdown_table("Title\n|---|---|\nfoo\nbar")
|
||||
|
||||
|
||||
def test_markdown_table_needs_multiple_columns() -> None:
|
||||
# Valid separator below, but the header is a single column -> not a table.
|
||||
assert _try_detect_markdown_table(["x|", "---|---", "y|"]) is None
|
||||
@@ -445,6 +590,14 @@ def test_compact_unique_csv_passes_through() -> None:
|
||||
assert result.compressed == CSV
|
||||
|
||||
|
||||
def test_compress_keeps_a_document_with_a_table_whole() -> None:
|
||||
# Only the table's rows used to come back, as JSON, with the prose and the
|
||||
# other command's output silently gone.
|
||||
result = TabularCompressor().compress(_DOC_WITH_TABLE)
|
||||
assert not result.was_modified
|
||||
assert result.compressed == _DOC_WITH_TABLE
|
||||
|
||||
|
||||
def test_non_tabular_passes_through_unmodified() -> None:
|
||||
# Unparseable prose returns the original content untouched.
|
||||
text = "just a normal paragraph here"
|
||||
@@ -514,6 +667,12 @@ def test_router_keeps_ls_output_verbatim(monkeypatch) -> None:
|
||||
assert result.strategy_used is CompressionStrategy.TABULAR
|
||||
|
||||
|
||||
def test_router_does_not_route_a_document_with_a_table_to_tabular(monkeypatch) -> None:
|
||||
_record_kompress_calls(monkeypatch)
|
||||
result = ContentRouter(ContentRouterConfig()).compress(_DOC_WITH_TABLE)
|
||||
assert result.strategy_used is not CompressionStrategy.TABULAR
|
||||
|
||||
|
||||
def test_router_does_not_kompress_a_ragged_csv(monkeypatch) -> None:
|
||||
calls = _record_kompress_calls(monkeypatch)
|
||||
csv = "id,name,city\n" + "\n".join(f"{i},user_{i},city_{i % 5}" for i in range(30))
|
||||
|
||||
Reference in New Issue
Block a user