mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description
A string cell in a `json` column whose text is a JSON **container**
(`"[]"`, `"{}"`, `"[1,2]"`) is
classified as `StringifiedJson` and un-escaped into
`CellValue::Scalar(parsed)`, so the cell renders
byte-identically to the real value it spells. The csv-schema decoder
cannot tell the two apart
afterwards: `expand_compacted` returns `[]` where the original had
`"[]"`, and with
`verify_lossless=True` (also forced by `mode="agent"`) the round-trip
check fails and the whole
message is reverted, silently dropping its compression.
This is the container-shaped remainder of #4053, which fixed the
scalar-looking case (`"null"`,
`"true"`, `"1"`). The formatter already knows how to render these
strings unambiguously
(`csv_formatter_json_column_quotes_strings_that_parse_as_json` covers
`"[1]"` and `'{"a":1}'`), but
`compact()` never lets them reach it as strings.
Closes #4874
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `compactor::cell_from_value`: for `StringifiedJson` that is not an
array of objects, keep the
original string (`CellValue::Scalar(v.clone())`) instead of the parsed
value. The formatter's
existing ambiguous-text rule JSON-quotes it, so the decoder returns the
string.
- Nested behaviour is unchanged: a stringified **array of objects**
still becomes a nested table, so
`test_nested_table_markers_resolve_to_source_bytes` and its CCR markers
keep working.
- Rust regression test pinning the exact wire cells (`"[]"` for the
array, `"""[1]"""`-style for the
string).
- Python regression test at the caller boundary (`_crush` +
`expand_compacted`).
## Testing
- [x] Unit tests pass (`cargo test -p headroom-core --lib` -> 1044
passed, 1 ignored)
- [x] `cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo test -p headroom-core --lib compaction::
test result: ok. 74 passed; 0 failed; 0 ignored
$ cargo test -p headroom-core --lib
test result: ok. 1044 passed; 0 failed; 1 ignored
# with the fix reverted, the new Rust test fails:
left: "\"[]\""
right: "\"\"\"[]\"\"\""
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, Rust 1.100
`x86_64-pc-windows-gnu`, repo `main` at
`fda75de5`
- Exact command / steps: `cargo test -p headroom-core --lib
compaction::compactor::tests::stringified_container_strings_keep_their_identity`;
the test builds
`[{id:0,v:[]},{id:1,v:"[]"},{id:2,v:[1,2]},{id:3,v:"[1,2]"},{id:4,v:{}},{id:5,v:"{}"}]`,
compacts
it, and asserts the rendered `v` cells.
- Observed result: without the change the array and the string render
the same cell (`"[]"`); with
it the string renders as the JSON string literal (`"""[]"""`), which
`expand_compacted` maps back to `"[]"`.
- Not tested: the Python regression test could not be executed locally
because this machine cannot
build the native extension (Rust build stopped in `icu_segmenter` under
the GNU toolchain); it is
included for CI, where the wheel is built normally.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; the change is in the lossless table
codec only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: a stringified JSON container in a
table cell now stays a string
in the densified text (it was displayed as the parsed value). This is
required for the documented
reversibility of `csv-schema`; the model sees a JSON string literal
instead of nested JSON.
- Kill switch / disable path: n/a (lossless path)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the single `cell_from_value` branch.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4875"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: JD Davis <mxjerrett@gmail.com>