refactor(smart-crusher): drop the dead crush_object clone from the object path (#3705)

Closes #3697 — **part 1 only**. Parts 2, 3 and 4 of that issue are
separate decisions and are deliberately untouched here (see "Not in this
PR").

## Problem

After #3648 correctly removed the key-dropping algorithm, `crush_object`
became a no-op:

```rust
pub fn crush_object(obj: &Map<String, Value>, _config: &..., _bias: f64)
    -> (Map<String, Value>, String) {
    (obj.clone(), "object:passthrough".to_string())
}
```

but it was still called for **every object with >=
`min_items_to_analyze` (5) keys, at every nesting depth**, and the only
branch reading its result was `if strategy != "object:passthrough"` —
statically dead. So every qualifying object was deep-cloned and the
clone discarded on the next line. On a deeply nested tool payload that
is a full recursive clone of the already-processed map at each level.

## Change

Removed the dead second-pass block, the function, the import, and a `pub
use` re-export in `mod.rs` that the issue did not mention. The seven
unit tests that exercised the deleted function went with it.

The explanatory comment is **kept and moved** onto the object arm, so
the institutional knowledge survives the code: object fields are
independent named properties rather than interchangeable sampled
records, and the old key-dropping implementation deleted keys with no
CCR retrieval marker, violating reversibility (#3650 / #3648).

## Verification

Deleting dead code should change *nothing*, so the evidence is the
inverse of a normal fix:

| check | result |
|---|---|
| `parity-run run --only smart_crusher` | **total=17 matched=17
skipped=0 diffed=0** |
| `cargo test -p headroom-core --lib` | **929 passed, 0 failed, 1
ignored** (−7 deleted, +3 new) |
| `cargo clippy --all-targets -- -D warnings` | clean |
| `cargo fmt --check` | clean |
| Python smart-crusher suites | 25 passed; broader sweep 94 passed / 1
skipped |

The 17/17 parity match is the load-bearing result: no fixture output
moved, confirming the branch was genuinely unreachable.

Three new Rust tests assert the invariant explicitly through the real
`SmartCrusher::crush` path — a 5-key object (the old gate boundary), a
15+-key wide mixed-type object with the expensive nested field at
begin/middle/end under both default and `lossless_only` configs, and a
40-key object — so "objects keep every key" rests on a test rather than
on the absence of code.

`min_items_to_analyze` is **not** orphaned: still read at
`crusher.rs:511`, `crusher.rs:1020`, `analyzer.rs:656` plus config
plumbing. No config field removed.

## Note on the Python test run

`headroom/transforms/smart_crusher.py` imports `headroom._core`, so a
stale prebuilt extension would not have exercised this change. The
extension was rebuilt via `cargo rustc` and swapped in for the test run
rather than `maturin develop`, because the dev worktree's
`_core.abi3.so` is a symlink into the main checkout and a build would
have written through it.

## Not in this PR

- **Part 2** — the `has_json_insignificant_whitespace` guard in
`smart_crush_content_with_hook`. Untouched.
- **Part 3** — key-level CCR offload to restore savings on wide objects.
- **Part 4** — TOIN comparability note.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
2026-09-23 23:14:30 -05:00
committed by GitHub
co-authored by Claude Opus 5
parent 46ac52d39d
commit 72217242c1
3 changed files with 181 additions and 220 deletions
@@ -45,7 +45,7 @@ use super::compaction::{
CompactConfig, Compaction, CompactionStage,
};
use super::config::SmartCrusherConfig;
use super::crushers::{compute_k_split, crush_number_array, crush_object, crush_string_array};
use super::crushers::{compute_k_split, crush_number_array, crush_string_array};
use super::planning::SmartCrusherPlanner;
use super::traits::{Constraint, CrushEvent, Observer};
use super::types::{CompressionPlan, CompressionStrategy, CrushResult};
@@ -634,7 +634,18 @@ impl SmartCrusher {
(Value::Array(processed), info_parts.join(","))
}
Value::Object(map) => {
// First pass: recurse into values to compress nested arrays.
// Object keys are never dropped. Object fields are
// independent named properties, not interchangeable
// sampled records, and the previous key-dropping
// implementation violated the CCR reversibility
// contract: it deleted keys (e.g. `tags`, `subtasks`
// on MCP task payloads) with no retrieval marker, so
// the data was unrecoverable (see #3650 / PR #3648).
//
// Recursing into the values is the whole job here:
// nested arrays still get large-array compaction and
// CCR-backed row offload, which stay reversible,
// without deleting their enclosing keys.
let mut processed = serde_json::Map::new();
for (k, v) in map {
let (p_val, p_info) =
@@ -645,19 +656,6 @@ impl SmartCrusher {
}
}
// Second pass: crush_object preserves every property
// (object fields are not interchangeable sampled
// records). Nested arrays were already compacted above,
// so large-array compaction and CCR row offload still
// run without deleting their enclosing keys.
if processed.len() >= self.config.min_items_to_analyze {
let (crushed_dict, strategy) = crush_object(&processed, &self.config, bias);
if strategy != "object:passthrough" {
info_parts.push(strategy);
return (Value::Object(crushed_dict), info_parts.join(","));
}
}
(Value::Object(processed), info_parts.join(","))
}
// Strings: walker-equivalent handling. Delegates to
@@ -1522,6 +1520,171 @@ mod tests {
assert_eq!(c.config.max_items_after_crush, 15);
}
// ---------- object key preservation (#3650 / #3697) ----------
//
// Object fields are independent named properties, not interchangeable
// sampled records, so no object key may be dropped: the object path
// emits no CCR retrieval marker, and dropping a key there would make
// the value unrecoverable. These guard the invariant explicitly, so
// it does not rest on the mere absence of key-dropping code.
/// Wide object above every historical key-selection gate: >= 5 keys,
/// long string values, a nested dict, nulls, booleans, empty arrays.
/// `expensive_index` places the nested dict at begin / middle / end,
/// since boundary/stride selection used to drop the middle one.
fn wide_object(expensive_index: usize) -> Value {
let n = 15;
let expensive_index = expensive_index.min(n - 1);
let mut obj = serde_json::Map::new();
for i in 0..n {
if i == expensive_index {
obj.insert(
format!("expensive_{i}"),
json!({
"nested": {"label": format!("nested-{i}"), "note": "x".repeat(120)},
"items": [],
"ok": true,
"missing": null
}),
);
} else {
obj.insert(
format!("field_{i:02}"),
json!(format!(
"this is a relatively long value string for entry number {i} with content"
)),
);
}
}
obj.insert("flag".to_string(), json!(false));
obj.insert("empty".to_string(), json!([]));
obj.insert("nada".to_string(), json!(null));
obj.insert(
"msg1".to_string(),
json!(format!("FATAL: {}", "x".repeat(200))),
);
obj.insert("tiny".to_string(), json!(1));
Value::Object(obj)
}
fn assert_every_key_round_trips(value: &Value, config: SmartCrusherConfig) {
let input = serde_json::to_string(value).expect("serialize");
let result = SmartCrusher::new(config).crush(&input, "", 1.0);
let out: Value = serde_json::from_str(&result.compressed)
.unwrap_or_else(|e| panic!("output is not JSON ({e}): {}", result.compressed));
assert_eq!(
&out, value,
"object keys must survive the crush path untouched"
);
let (expected, got) = (
value.as_object().expect("object"),
out.as_object().expect("object"),
);
assert_eq!(got.len(), expected.len(), "key count changed");
for (k, v) in expected {
assert_eq!(got.get(k), Some(v), "key {k} was dropped or altered");
}
assert!(
!result.strategy.contains("object:"),
"no object-level crush strategy may be reported: {}",
result.strategy
);
}
#[test]
fn crush_object_keeps_every_key_for_five_key_object() {
// Exactly at the old `min_items_to_analyze` gate: the smallest
// object the removed second pass used to touch.
let mut obj = serde_json::Map::new();
for i in 0..5 {
obj.insert(
format!("k{i}"),
json!(format!("value number {i} for the five key object")),
);
}
assert_every_key_round_trips(&Value::Object(obj), SmartCrusherConfig::default());
}
#[test]
fn crush_object_keeps_every_key_for_wide_mixed_object() {
for expensive_index in [0, 7, 14] {
assert_every_key_round_trips(
&wide_object(expensive_index),
SmartCrusherConfig::default(),
);
assert_every_key_round_trips(
&wide_object(expensive_index),
SmartCrusherConfig {
lossless_only: true,
..SmartCrusherConfig::default()
},
);
}
}
#[test]
fn crush_object_keeps_every_key_when_nested_arrays_compact() {
// The behaviour this refactor relies on: recursing into object VALUES
// still compacts a nested array, and the enclosing key survives that
// compaction. The array has to actually shrink for the test to mean
// anything -- an all-scalar object would pass even if recursion were
// removed from the object branch, which is the whole thing being
// guarded here.
let mut obj = serde_json::Map::new();
for i in 0..40 {
obj.insert(
format!("k{i:02}"),
json!(format!(
"long description for entry {i}, above the small-value floor"
)),
);
}
let rows: Vec<Value> = (0..100)
.map(|i| {
json!({
"id": i,
"name": format!("row-{i}"),
"status": if i % 3 == 0 { "error" } else { "ok" },
"detail": format!("detail text for row {i} padded out a bit"),
})
})
.collect();
let row_count = rows.len();
obj.insert("records".to_string(), Value::Array(rows));
let value = Value::Object(obj);
let input = serde_json::to_string(&value).expect("serialize");
let result = crusher().crush(&input, "", 1.0);
let out: Value = serde_json::from_str(&result.compressed).expect("valid JSON out");
let got = out.as_object().expect("object out");
assert_eq!(got.len(), 41, "all 40 scalar keys + `records` must survive");
for k in value.as_object().unwrap().keys() {
assert!(got.contains_key(k), "dropped key {k}");
}
// The nested array took a compaction path...
assert!(
result.was_modified,
"nested array should have been compacted; strategy={} out={}",
result.strategy, result.compressed
);
let compacted = got.get("records").expect("`records` key survived");
assert_ne!(
compacted,
value.as_object().unwrap().get("records").unwrap(),
"`records` should have been transformed, not passed through verbatim"
);
// ...and shrank, while its enclosing key stayed put.
if let Some(arr) = compacted.as_array() {
assert!(
arr.len() < row_count,
"compacted array should hold fewer than {row_count} items, got {}",
arr.len()
);
}
}
// ---------- top-level crush ----------
#[test]
@@ -1,13 +1,9 @@
//! Three universal crushers for non-dict-array JSON shapes.
//! Two universal crushers for non-dict-array JSON shapes.
//!
//! Direct ports from `headroom/transforms/smart_crusher.py`:
//!
//! - `crush_string_array` ← `_crush_string_array` (line 2727)
//! - `crush_number_array` ← `_crush_number_array` (line 2810) — has BUG #1
//! - `crush_object` ← `_crush_object` (line 3015)
//! Object fields are independent properties, not sampled records.
//! `crush_object` preserves every key; nested arrays are compacted
//! by the recursive `process_value` pass before this function runs.
//!
//! Each takes a `&SmartCrusherConfig`, a `bias` multiplier, and returns
//! `(crushed_items, strategy_string)`. Schema-preserving: the output
@@ -33,7 +29,7 @@
//! the bug-doc test below flips to pin the corrected behavior and the
//! fixtures are regenerated.
use serde_json::{Map, Value};
use serde_json::Value;
use std::collections::{BTreeSet, HashSet};
use super::config::SmartCrusherConfig;
@@ -382,27 +378,6 @@ pub fn crush_number_array(
(kept_values, strategy)
}
/// Preserve every property of a JSON object.
///
/// Object fields are independent named properties, not interchangeable
/// sampled records. The previous key-selection algorithm dropped keys
/// (no CCR marker, no recovery hash) which silently deleted structured
/// fields such as `tags` and `subtasks` from MCP task objects. Nested
/// arrays are still compacted by the recursive `process_value` pass
/// *before* this function runs, so legitimate large-array compaction
/// and CCR-backed row offload remain available without deleting their
/// enclosing keys.
///
/// `config` and `bias` are retained so the production call site in
/// `SmartCrusher::process_value_with_hook` stays unchanged.
pub fn crush_object(
obj: &Map<String, Value>,
_config: &SmartCrusherConfig,
_bias: f64,
) -> (Map<String, Value>, String) {
(obj.clone(), "object:passthrough".to_string())
}
// ---------- helpers ----------
/// Linear-interpolation percentile (numpy "linear" method).
@@ -634,165 +609,6 @@ mod tests {
assert!(strat.contains("p75="));
}
// ---------- crush_object ----------
/// Wide object above the historical n>8 / token-floor gate: arbitrary
/// names, long strings, a nested dict, nulls, booleans, empty arrays.
/// `expensive_index` places the nested dict at begin / middle / end.
fn wide_object(expensive_index: usize) -> Map<String, Value> {
let n = 15;
let expensive_index = expensive_index.min(n - 1);
let mut obj = Map::new();
for i in 0..n {
if i == expensive_index {
obj.insert(
format!("expensive_{i}"),
json!({
"nested": {
"label": format!("nested-{i}"),
"note": "x".repeat(120)
},
"items": [],
"ok": true,
"missing": null
}),
);
} else {
obj.insert(
format!("field_{i:02}"),
json!(format!(
"this is a relatively long value string for entry number {i} with content"
)),
);
}
}
obj.insert("flag".to_string(), json!(false));
obj.insert("empty".to_string(), json!([]));
obj.insert("nada".to_string(), json!(null));
obj
}
fn assert_object_fully_preserved(obj: &Map<String, Value>, config: &SmartCrusherConfig) {
let (out, strat) = crush_object(obj, config, 1.0);
assert_eq!(out, *obj, "crush_object must keep every property");
assert_eq!(out.len(), obj.len());
for key in obj.keys() {
assert!(out.contains_key(key), "dropped key {key}");
assert_eq!(out.get(key), obj.get(key));
}
assert_eq!(strat, "object:passthrough");
assert!(
!strat.contains("adaptive"),
"must not report a key-drop strategy: {strat}"
);
assert!(
!strat.contains("ccr"),
"must not invent a CCR hash: {strat}"
);
}
#[test]
fn object_passthrough_when_few_keys() {
let mut obj = Map::new();
for i in 0..5 {
obj.insert(format!("k{}", i), json!(i));
}
let (out, strat) = crush_object(&obj, &cfg(), 1.0);
assert_eq!(out.len(), 5);
assert_eq!(out, obj);
assert_eq!(strat, "object:passthrough");
}
#[test]
fn object_passthrough_when_total_tokens_below_min() {
// Many tiny keys/values: historically total_tokens stayed below
// min_tokens_to_crush=200 and skipped selection. Preservation is
// now unconditional, so every key still survives.
let mut obj = Map::new();
for i in 0..30 {
obj.insert(format!("k{}", i), json!(i));
}
let (out, strat) = crush_object(&obj, &cfg(), 1.0);
assert_eq!(out, obj);
assert_eq!(strat, "object:passthrough");
}
#[test]
fn object_preserves_every_key_when_token_budget_exceeded() {
// 30 keys, each with a long string value → historically above
// the token floor and subject to key-selection. Every property
// must survive; no adaptive key-drop strategy.
let mut obj = Map::new();
for i in 0..30 {
obj.insert(
format!("k{:02}", i),
json!(format!(
"this is a relatively long value string for entry number {} with content",
i
)),
);
}
assert_object_fully_preserved(&obj, &cfg());
}
#[test]
fn object_preserves_wide_object_with_mixed_value_types() {
assert_object_fully_preserved(&wide_object(7), &cfg());
}
#[test]
fn object_preserves_expensive_fields_in_any_position() {
// Reorder the expensive nested field into beginning, middle, and
// end. Boundary/stride selection used to drop the middle one.
for expensive_index in [0, 7, 14] {
assert_object_fully_preserved(&wide_object(expensive_index), &cfg());
assert_object_fully_preserved(&wide_object(expensive_index), &lossless_only_cfg());
}
}
#[test]
fn object_keeps_small_values() {
// Mix of small + large values; every key survives, including tiny.
let mut obj = Map::new();
obj.insert("tiny".to_string(), json!(1));
for i in 0..30 {
obj.insert(
format!("big{:02}", i),
json!(format!(
"this is a long string with content for entry number {} that exceeds the small threshold",
i
)),
);
}
let (out, _) = crush_object(&obj, &cfg(), 1.0);
assert_eq!(out, obj);
assert!(
out.contains_key("tiny"),
"tiny key (small value) must survive"
);
}
#[test]
fn object_keeps_error_keywords() {
let mut obj = Map::new();
obj.insert(
"msg1".to_string(),
json!(format!("FATAL: {}", "x".repeat(200))),
);
for i in 0..30 {
obj.insert(
format!("k{:02}", i),
json!(format!("padding content for entry {} with text", i)),
);
}
let (out, _) = crush_object(&obj, &cfg(), 1.0);
assert_eq!(out, obj);
assert!(
out.contains_key("msg1"),
"key with error-keyword value must survive"
);
}
// ---------- lossless_only (#3625) ----------
//
// These crushers drop items with no CCR marker at all, so strict
@@ -832,24 +648,6 @@ mod tests {
assert_eq!(strat, "number:lossless_only");
}
#[test]
fn object_lossless_only_keeps_every_key() {
let mut obj = Map::new();
for i in 0..40 {
obj.insert(
format!("k{i:02}"),
json!(format!(
"long description for entry {i}, above the small-value floor"
)),
);
}
// Default mode used to drop keys with no CCR marker. Preservation
// is now unconditional, so both configs keep every property.
assert_object_fully_preserved(&obj, &cfg());
assert_object_fully_preserved(&obj, &lossless_only_cfg());
}
// ---------- BUG #1 documentation test ----------
#[test]
@@ -62,7 +62,7 @@ pub use constraints::{
default_oss_constraints, KeepErrorsConstraint, KeepStructuralOutliersConstraint,
};
pub use crusher::{CrushArrayResult, SmartCrusher};
pub use crushers::{compute_k_split, crush_number_array, crush_object, crush_string_array};
pub use crushers::{compute_k_split, crush_number_array, crush_string_array};
pub use error_keywords::ERROR_KEYWORDS;
pub use field_detect::{detect_id_field_statistically, detect_score_field_statistically};
pub use hashing::hash_field_name;