Files
zhengqiuyangandJD Davis 303f521812 fix(smart-crusher): keep container-shaped strings as strings in lossless tables (#4875)
## Description

A string cell in a `json` column whose text is a JSON **container**
(`"[]"`, `"{}"`, `"[1,2]"`) is
classified as `StringifiedJson` and un-escaped into
`CellValue::Scalar(parsed)`, so the cell renders
byte-identically to the real value it spells. The csv-schema decoder
cannot tell the two apart
afterwards: `expand_compacted` returns `[]` where the original had
`"[]"`, and with
`verify_lossless=True` (also forced by `mode="agent"`) the round-trip
check fails and the whole
message is reverted, silently dropping its compression.

This is the container-shaped remainder of #4053, which fixed the
scalar-looking case (`"null"`,
`"true"`, `"1"`). The formatter already knows how to render these
strings unambiguously
(`csv_formatter_json_column_quotes_strings_that_parse_as_json` covers
`"[1]"` and `'{"a":1}'`), but
`compact()` never lets them reach it as strings.

Closes #4874

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `compactor::cell_from_value`: for `StringifiedJson` that is not an
array of objects, keep the
original string (`CellValue::Scalar(v.clone())`) instead of the parsed
value. The formatter's
existing ambiguous-text rule JSON-quotes it, so the decoder returns the
string.
- Nested behaviour is unchanged: a stringified **array of objects**
still becomes a nested table, so
`test_nested_table_markers_resolve_to_source_bytes` and its CCR markers
keep working.
- Rust regression test pinning the exact wire cells (`"[]"` for the
array, `"""[1]"""`-style for the
  string).
- Python regression test at the caller boundary (`_crush` +
`expand_compacted`).

## Testing

- [x] Unit tests pass (`cargo test -p headroom-core --lib` -> 1044
passed, 1 ignored)
- [x] `cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cargo test -p headroom-core --lib compaction::
test result: ok. 74 passed; 0 failed; 0 ignored

$ cargo test -p headroom-core --lib
test result: ok. 1044 passed; 0 failed; 1 ignored

# with the fix reverted, the new Rust test fails:
left: "\"[]\""
right: "\"\"\"[]\"\"\""
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, Rust 1.100
`x86_64-pc-windows-gnu`, repo `main` at
  `fda75de5`
- Exact command / steps: `cargo test -p headroom-core --lib

compaction::compactor::tests::stringified_container_strings_keep_their_identity`;
the test builds

`[{id:0,v:[]},{id:1,v:"[]"},{id:2,v:[1,2]},{id:3,v:"[1,2]"},{id:4,v:{}},{id:5,v:"{}"}]`,
compacts
  it, and asserts the rendered `v` cells.
- Observed result: without the change the array and the string render
the same cell (`"[]"`); with
  it the string renders as the JSON string literal (`"""[]"""`), which
  `expand_compacted` maps back to `"[]"`.
- Not tested: the Python regression test could not be executed locally
because this machine cannot
build the native extension (Rust build stopped in `icu_segmenter` under
the GNU toolchain); it is
  included for CI, where the wheel is built normally.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; the change is in the lossless table
codec only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: a stringified JSON container in a
table cell now stays a string
in the densified text (it was displayed as the parsed value). This is
required for the documented
reversibility of `csv-schema`; the model sees a JSON string literal
instead of nested JSON.
- Kill switch / disable path: n/a (lossless path)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the single `cell_from_value` branch.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4875"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:50:47 -06:00
..