mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
refactor(smart-crusher): drop the dead crush_object clone from the object path (#3705)
Closes #3697 — **part 1 only**. Parts 2, 3 and 4 of that issue are separate decisions and are deliberately untouched here (see "Not in this PR"). ## Problem After #3648 correctly removed the key-dropping algorithm, `crush_object` became a no-op: ```rust pub fn crush_object(obj: &Map<String, Value>, _config: &..., _bias: f64) -> (Map<String, Value>, String) { (obj.clone(), "object:passthrough".to_string()) } ``` but it was still called for **every object with >= `min_items_to_analyze` (5) keys, at every nesting depth**, and the only branch reading its result was `if strategy != "object:passthrough"` — statically dead. So every qualifying object was deep-cloned and the clone discarded on the next line. On a deeply nested tool payload that is a full recursive clone of the already-processed map at each level. ## Change Removed the dead second-pass block, the function, the import, and a `pub use` re-export in `mod.rs` that the issue did not mention. The seven unit tests that exercised the deleted function went with it. The explanatory comment is **kept and moved** onto the object arm, so the institutional knowledge survives the code: object fields are independent named properties rather than interchangeable sampled records, and the old key-dropping implementation deleted keys with no CCR retrieval marker, violating reversibility (#3650 / #3648). ## Verification Deleting dead code should change *nothing*, so the evidence is the inverse of a normal fix: | check | result | |---|---| | `parity-run run --only smart_crusher` | **total=17 matched=17 skipped=0 diffed=0** | | `cargo test -p headroom-core --lib` | **929 passed, 0 failed, 1 ignored** (−7 deleted, +3 new) | | `cargo clippy --all-targets -- -D warnings` | clean | | `cargo fmt --check` | clean | | Python smart-crusher suites | 25 passed; broader sweep 94 passed / 1 skipped | The 17/17 parity match is the load-bearing result: no fixture output moved, confirming the branch was genuinely unreachable. Three new Rust tests assert the invariant explicitly through the real `SmartCrusher::crush` path — a 5-key object (the old gate boundary), a 15+-key wide mixed-type object with the expensive nested field at begin/middle/end under both default and `lossless_only` configs, and a 40-key object — so "objects keep every key" rests on a test rather than on the absence of code. `min_items_to_analyze` is **not** orphaned: still read at `crusher.rs:511`, `crusher.rs:1020`, `analyzer.rs:656` plus config plumbing. No config field removed. ## Note on the Python test run `headroom/transforms/smart_crusher.py` imports `headroom._core`, so a stale prebuilt extension would not have exercised this change. The extension was rebuilt via `cargo rustc` and swapped in for the test run rather than `maturin develop`, because the dev worktree's `_core.abi3.so` is a symlink into the main checkout and a build would have written through it. ## Not in this PR - **Part 2** — the `has_json_insignificant_whitespace` guard in `smart_crush_content_with_hook`. Untouched. - **Part 3** — key-level CCR offload to restore savings on wide objects. - **Part 4** — TOIN comparability note. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
46ac52d39d
commit
72217242c1
@@ -45,7 +45,7 @@ use super::compaction::{
|
||||
CompactConfig, Compaction, CompactionStage,
|
||||
};
|
||||
use super::config::SmartCrusherConfig;
|
||||
use super::crushers::{compute_k_split, crush_number_array, crush_object, crush_string_array};
|
||||
use super::crushers::{compute_k_split, crush_number_array, crush_string_array};
|
||||
use super::planning::SmartCrusherPlanner;
|
||||
use super::traits::{Constraint, CrushEvent, Observer};
|
||||
use super::types::{CompressionPlan, CompressionStrategy, CrushResult};
|
||||
@@ -634,7 +634,18 @@ impl SmartCrusher {
|
||||
(Value::Array(processed), info_parts.join(","))
|
||||
}
|
||||
Value::Object(map) => {
|
||||
// First pass: recurse into values to compress nested arrays.
|
||||
// Object keys are never dropped. Object fields are
|
||||
// independent named properties, not interchangeable
|
||||
// sampled records, and the previous key-dropping
|
||||
// implementation violated the CCR reversibility
|
||||
// contract: it deleted keys (e.g. `tags`, `subtasks`
|
||||
// on MCP task payloads) with no retrieval marker, so
|
||||
// the data was unrecoverable (see #3650 / PR #3648).
|
||||
//
|
||||
// Recursing into the values is the whole job here:
|
||||
// nested arrays still get large-array compaction and
|
||||
// CCR-backed row offload, which stay reversible,
|
||||
// without deleting their enclosing keys.
|
||||
let mut processed = serde_json::Map::new();
|
||||
for (k, v) in map {
|
||||
let (p_val, p_info) =
|
||||
@@ -645,19 +656,6 @@ impl SmartCrusher {
|
||||
}
|
||||
}
|
||||
|
||||
// Second pass: crush_object preserves every property
|
||||
// (object fields are not interchangeable sampled
|
||||
// records). Nested arrays were already compacted above,
|
||||
// so large-array compaction and CCR row offload still
|
||||
// run without deleting their enclosing keys.
|
||||
if processed.len() >= self.config.min_items_to_analyze {
|
||||
let (crushed_dict, strategy) = crush_object(&processed, &self.config, bias);
|
||||
if strategy != "object:passthrough" {
|
||||
info_parts.push(strategy);
|
||||
return (Value::Object(crushed_dict), info_parts.join(","));
|
||||
}
|
||||
}
|
||||
|
||||
(Value::Object(processed), info_parts.join(","))
|
||||
}
|
||||
// Strings: walker-equivalent handling. Delegates to
|
||||
@@ -1522,6 +1520,171 @@ mod tests {
|
||||
assert_eq!(c.config.max_items_after_crush, 15);
|
||||
}
|
||||
|
||||
// ---------- object key preservation (#3650 / #3697) ----------
|
||||
//
|
||||
// Object fields are independent named properties, not interchangeable
|
||||
// sampled records, so no object key may be dropped: the object path
|
||||
// emits no CCR retrieval marker, and dropping a key there would make
|
||||
// the value unrecoverable. These guard the invariant explicitly, so
|
||||
// it does not rest on the mere absence of key-dropping code.
|
||||
|
||||
/// Wide object above every historical key-selection gate: >= 5 keys,
|
||||
/// long string values, a nested dict, nulls, booleans, empty arrays.
|
||||
/// `expensive_index` places the nested dict at begin / middle / end,
|
||||
/// since boundary/stride selection used to drop the middle one.
|
||||
fn wide_object(expensive_index: usize) -> Value {
|
||||
let n = 15;
|
||||
let expensive_index = expensive_index.min(n - 1);
|
||||
let mut obj = serde_json::Map::new();
|
||||
for i in 0..n {
|
||||
if i == expensive_index {
|
||||
obj.insert(
|
||||
format!("expensive_{i}"),
|
||||
json!({
|
||||
"nested": {"label": format!("nested-{i}"), "note": "x".repeat(120)},
|
||||
"items": [],
|
||||
"ok": true,
|
||||
"missing": null
|
||||
}),
|
||||
);
|
||||
} else {
|
||||
obj.insert(
|
||||
format!("field_{i:02}"),
|
||||
json!(format!(
|
||||
"this is a relatively long value string for entry number {i} with content"
|
||||
)),
|
||||
);
|
||||
}
|
||||
}
|
||||
obj.insert("flag".to_string(), json!(false));
|
||||
obj.insert("empty".to_string(), json!([]));
|
||||
obj.insert("nada".to_string(), json!(null));
|
||||
obj.insert(
|
||||
"msg1".to_string(),
|
||||
json!(format!("FATAL: {}", "x".repeat(200))),
|
||||
);
|
||||
obj.insert("tiny".to_string(), json!(1));
|
||||
Value::Object(obj)
|
||||
}
|
||||
|
||||
fn assert_every_key_round_trips(value: &Value, config: SmartCrusherConfig) {
|
||||
let input = serde_json::to_string(value).expect("serialize");
|
||||
let result = SmartCrusher::new(config).crush(&input, "", 1.0);
|
||||
let out: Value = serde_json::from_str(&result.compressed)
|
||||
.unwrap_or_else(|e| panic!("output is not JSON ({e}): {}", result.compressed));
|
||||
assert_eq!(
|
||||
&out, value,
|
||||
"object keys must survive the crush path untouched"
|
||||
);
|
||||
let (expected, got) = (
|
||||
value.as_object().expect("object"),
|
||||
out.as_object().expect("object"),
|
||||
);
|
||||
assert_eq!(got.len(), expected.len(), "key count changed");
|
||||
for (k, v) in expected {
|
||||
assert_eq!(got.get(k), Some(v), "key {k} was dropped or altered");
|
||||
}
|
||||
assert!(
|
||||
!result.strategy.contains("object:"),
|
||||
"no object-level crush strategy may be reported: {}",
|
||||
result.strategy
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn crush_object_keeps_every_key_for_five_key_object() {
|
||||
// Exactly at the old `min_items_to_analyze` gate: the smallest
|
||||
// object the removed second pass used to touch.
|
||||
let mut obj = serde_json::Map::new();
|
||||
for i in 0..5 {
|
||||
obj.insert(
|
||||
format!("k{i}"),
|
||||
json!(format!("value number {i} for the five key object")),
|
||||
);
|
||||
}
|
||||
assert_every_key_round_trips(&Value::Object(obj), SmartCrusherConfig::default());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn crush_object_keeps_every_key_for_wide_mixed_object() {
|
||||
for expensive_index in [0, 7, 14] {
|
||||
assert_every_key_round_trips(
|
||||
&wide_object(expensive_index),
|
||||
SmartCrusherConfig::default(),
|
||||
);
|
||||
assert_every_key_round_trips(
|
||||
&wide_object(expensive_index),
|
||||
SmartCrusherConfig {
|
||||
lossless_only: true,
|
||||
..SmartCrusherConfig::default()
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn crush_object_keeps_every_key_when_nested_arrays_compact() {
|
||||
// The behaviour this refactor relies on: recursing into object VALUES
|
||||
// still compacts a nested array, and the enclosing key survives that
|
||||
// compaction. The array has to actually shrink for the test to mean
|
||||
// anything -- an all-scalar object would pass even if recursion were
|
||||
// removed from the object branch, which is the whole thing being
|
||||
// guarded here.
|
||||
let mut obj = serde_json::Map::new();
|
||||
for i in 0..40 {
|
||||
obj.insert(
|
||||
format!("k{i:02}"),
|
||||
json!(format!(
|
||||
"long description for entry {i}, above the small-value floor"
|
||||
)),
|
||||
);
|
||||
}
|
||||
let rows: Vec<Value> = (0..100)
|
||||
.map(|i| {
|
||||
json!({
|
||||
"id": i,
|
||||
"name": format!("row-{i}"),
|
||||
"status": if i % 3 == 0 { "error" } else { "ok" },
|
||||
"detail": format!("detail text for row {i} padded out a bit"),
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
let row_count = rows.len();
|
||||
obj.insert("records".to_string(), Value::Array(rows));
|
||||
|
||||
let value = Value::Object(obj);
|
||||
let input = serde_json::to_string(&value).expect("serialize");
|
||||
let result = crusher().crush(&input, "", 1.0);
|
||||
let out: Value = serde_json::from_str(&result.compressed).expect("valid JSON out");
|
||||
let got = out.as_object().expect("object out");
|
||||
|
||||
assert_eq!(got.len(), 41, "all 40 scalar keys + `records` must survive");
|
||||
for k in value.as_object().unwrap().keys() {
|
||||
assert!(got.contains_key(k), "dropped key {k}");
|
||||
}
|
||||
|
||||
// The nested array took a compaction path...
|
||||
assert!(
|
||||
result.was_modified,
|
||||
"nested array should have been compacted; strategy={} out={}",
|
||||
result.strategy, result.compressed
|
||||
);
|
||||
let compacted = got.get("records").expect("`records` key survived");
|
||||
assert_ne!(
|
||||
compacted,
|
||||
value.as_object().unwrap().get("records").unwrap(),
|
||||
"`records` should have been transformed, not passed through verbatim"
|
||||
);
|
||||
// ...and shrank, while its enclosing key stayed put.
|
||||
if let Some(arr) = compacted.as_array() {
|
||||
assert!(
|
||||
arr.len() < row_count,
|
||||
"compacted array should hold fewer than {row_count} items, got {}",
|
||||
arr.len()
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- top-level crush ----------
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -1,13 +1,9 @@
|
||||
//! Three universal crushers for non-dict-array JSON shapes.
|
||||
//! Two universal crushers for non-dict-array JSON shapes.
|
||||
//!
|
||||
//! Direct ports from `headroom/transforms/smart_crusher.py`:
|
||||
//!
|
||||
//! - `crush_string_array` ← `_crush_string_array` (line 2727)
|
||||
//! - `crush_number_array` ← `_crush_number_array` (line 2810) — has BUG #1
|
||||
//! - `crush_object` ← `_crush_object` (line 3015)
|
||||
//! Object fields are independent properties, not sampled records.
|
||||
//! `crush_object` preserves every key; nested arrays are compacted
|
||||
//! by the recursive `process_value` pass before this function runs.
|
||||
//!
|
||||
//! Each takes a `&SmartCrusherConfig`, a `bias` multiplier, and returns
|
||||
//! `(crushed_items, strategy_string)`. Schema-preserving: the output
|
||||
@@ -33,7 +29,7 @@
|
||||
//! the bug-doc test below flips to pin the corrected behavior and the
|
||||
//! fixtures are regenerated.
|
||||
|
||||
use serde_json::{Map, Value};
|
||||
use serde_json::Value;
|
||||
use std::collections::{BTreeSet, HashSet};
|
||||
|
||||
use super::config::SmartCrusherConfig;
|
||||
@@ -382,27 +378,6 @@ pub fn crush_number_array(
|
||||
(kept_values, strategy)
|
||||
}
|
||||
|
||||
/// Preserve every property of a JSON object.
|
||||
///
|
||||
/// Object fields are independent named properties, not interchangeable
|
||||
/// sampled records. The previous key-selection algorithm dropped keys
|
||||
/// (no CCR marker, no recovery hash) which silently deleted structured
|
||||
/// fields such as `tags` and `subtasks` from MCP task objects. Nested
|
||||
/// arrays are still compacted by the recursive `process_value` pass
|
||||
/// *before* this function runs, so legitimate large-array compaction
|
||||
/// and CCR-backed row offload remain available without deleting their
|
||||
/// enclosing keys.
|
||||
///
|
||||
/// `config` and `bias` are retained so the production call site in
|
||||
/// `SmartCrusher::process_value_with_hook` stays unchanged.
|
||||
pub fn crush_object(
|
||||
obj: &Map<String, Value>,
|
||||
_config: &SmartCrusherConfig,
|
||||
_bias: f64,
|
||||
) -> (Map<String, Value>, String) {
|
||||
(obj.clone(), "object:passthrough".to_string())
|
||||
}
|
||||
|
||||
// ---------- helpers ----------
|
||||
|
||||
/// Linear-interpolation percentile (numpy "linear" method).
|
||||
@@ -634,165 +609,6 @@ mod tests {
|
||||
assert!(strat.contains("p75="));
|
||||
}
|
||||
|
||||
// ---------- crush_object ----------
|
||||
|
||||
/// Wide object above the historical n>8 / token-floor gate: arbitrary
|
||||
/// names, long strings, a nested dict, nulls, booleans, empty arrays.
|
||||
/// `expensive_index` places the nested dict at begin / middle / end.
|
||||
fn wide_object(expensive_index: usize) -> Map<String, Value> {
|
||||
let n = 15;
|
||||
let expensive_index = expensive_index.min(n - 1);
|
||||
let mut obj = Map::new();
|
||||
for i in 0..n {
|
||||
if i == expensive_index {
|
||||
obj.insert(
|
||||
format!("expensive_{i}"),
|
||||
json!({
|
||||
"nested": {
|
||||
"label": format!("nested-{i}"),
|
||||
"note": "x".repeat(120)
|
||||
},
|
||||
"items": [],
|
||||
"ok": true,
|
||||
"missing": null
|
||||
}),
|
||||
);
|
||||
} else {
|
||||
obj.insert(
|
||||
format!("field_{i:02}"),
|
||||
json!(format!(
|
||||
"this is a relatively long value string for entry number {i} with content"
|
||||
)),
|
||||
);
|
||||
}
|
||||
}
|
||||
obj.insert("flag".to_string(), json!(false));
|
||||
obj.insert("empty".to_string(), json!([]));
|
||||
obj.insert("nada".to_string(), json!(null));
|
||||
obj
|
||||
}
|
||||
|
||||
fn assert_object_fully_preserved(obj: &Map<String, Value>, config: &SmartCrusherConfig) {
|
||||
let (out, strat) = crush_object(obj, config, 1.0);
|
||||
assert_eq!(out, *obj, "crush_object must keep every property");
|
||||
assert_eq!(out.len(), obj.len());
|
||||
for key in obj.keys() {
|
||||
assert!(out.contains_key(key), "dropped key {key}");
|
||||
assert_eq!(out.get(key), obj.get(key));
|
||||
}
|
||||
assert_eq!(strat, "object:passthrough");
|
||||
assert!(
|
||||
!strat.contains("adaptive"),
|
||||
"must not report a key-drop strategy: {strat}"
|
||||
);
|
||||
assert!(
|
||||
!strat.contains("ccr"),
|
||||
"must not invent a CCR hash: {strat}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn object_passthrough_when_few_keys() {
|
||||
let mut obj = Map::new();
|
||||
for i in 0..5 {
|
||||
obj.insert(format!("k{}", i), json!(i));
|
||||
}
|
||||
let (out, strat) = crush_object(&obj, &cfg(), 1.0);
|
||||
assert_eq!(out.len(), 5);
|
||||
assert_eq!(out, obj);
|
||||
assert_eq!(strat, "object:passthrough");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn object_passthrough_when_total_tokens_below_min() {
|
||||
// Many tiny keys/values: historically total_tokens stayed below
|
||||
// min_tokens_to_crush=200 and skipped selection. Preservation is
|
||||
// now unconditional, so every key still survives.
|
||||
let mut obj = Map::new();
|
||||
for i in 0..30 {
|
||||
obj.insert(format!("k{}", i), json!(i));
|
||||
}
|
||||
let (out, strat) = crush_object(&obj, &cfg(), 1.0);
|
||||
assert_eq!(out, obj);
|
||||
assert_eq!(strat, "object:passthrough");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn object_preserves_every_key_when_token_budget_exceeded() {
|
||||
// 30 keys, each with a long string value → historically above
|
||||
// the token floor and subject to key-selection. Every property
|
||||
// must survive; no adaptive key-drop strategy.
|
||||
let mut obj = Map::new();
|
||||
for i in 0..30 {
|
||||
obj.insert(
|
||||
format!("k{:02}", i),
|
||||
json!(format!(
|
||||
"this is a relatively long value string for entry number {} with content",
|
||||
i
|
||||
)),
|
||||
);
|
||||
}
|
||||
assert_object_fully_preserved(&obj, &cfg());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn object_preserves_wide_object_with_mixed_value_types() {
|
||||
assert_object_fully_preserved(&wide_object(7), &cfg());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn object_preserves_expensive_fields_in_any_position() {
|
||||
// Reorder the expensive nested field into beginning, middle, and
|
||||
// end. Boundary/stride selection used to drop the middle one.
|
||||
for expensive_index in [0, 7, 14] {
|
||||
assert_object_fully_preserved(&wide_object(expensive_index), &cfg());
|
||||
assert_object_fully_preserved(&wide_object(expensive_index), &lossless_only_cfg());
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn object_keeps_small_values() {
|
||||
// Mix of small + large values; every key survives, including tiny.
|
||||
let mut obj = Map::new();
|
||||
obj.insert("tiny".to_string(), json!(1));
|
||||
for i in 0..30 {
|
||||
obj.insert(
|
||||
format!("big{:02}", i),
|
||||
json!(format!(
|
||||
"this is a long string with content for entry number {} that exceeds the small threshold",
|
||||
i
|
||||
)),
|
||||
);
|
||||
}
|
||||
let (out, _) = crush_object(&obj, &cfg(), 1.0);
|
||||
assert_eq!(out, obj);
|
||||
assert!(
|
||||
out.contains_key("tiny"),
|
||||
"tiny key (small value) must survive"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn object_keeps_error_keywords() {
|
||||
let mut obj = Map::new();
|
||||
obj.insert(
|
||||
"msg1".to_string(),
|
||||
json!(format!("FATAL: {}", "x".repeat(200))),
|
||||
);
|
||||
for i in 0..30 {
|
||||
obj.insert(
|
||||
format!("k{:02}", i),
|
||||
json!(format!("padding content for entry {} with text", i)),
|
||||
);
|
||||
}
|
||||
let (out, _) = crush_object(&obj, &cfg(), 1.0);
|
||||
assert_eq!(out, obj);
|
||||
assert!(
|
||||
out.contains_key("msg1"),
|
||||
"key with error-keyword value must survive"
|
||||
);
|
||||
}
|
||||
|
||||
// ---------- lossless_only (#3625) ----------
|
||||
//
|
||||
// These crushers drop items with no CCR marker at all, so strict
|
||||
@@ -832,24 +648,6 @@ mod tests {
|
||||
assert_eq!(strat, "number:lossless_only");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn object_lossless_only_keeps_every_key() {
|
||||
let mut obj = Map::new();
|
||||
for i in 0..40 {
|
||||
obj.insert(
|
||||
format!("k{i:02}"),
|
||||
json!(format!(
|
||||
"long description for entry {i}, above the small-value floor"
|
||||
)),
|
||||
);
|
||||
}
|
||||
|
||||
// Default mode used to drop keys with no CCR marker. Preservation
|
||||
// is now unconditional, so both configs keep every property.
|
||||
assert_object_fully_preserved(&obj, &cfg());
|
||||
assert_object_fully_preserved(&obj, &lossless_only_cfg());
|
||||
}
|
||||
|
||||
// ---------- BUG #1 documentation test ----------
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -62,7 +62,7 @@ pub use constraints::{
|
||||
default_oss_constraints, KeepErrorsConstraint, KeepStructuralOutliersConstraint,
|
||||
};
|
||||
pub use crusher::{CrushArrayResult, SmartCrusher};
|
||||
pub use crushers::{compute_k_split, crush_number_array, crush_object, crush_string_array};
|
||||
pub use crushers::{compute_k_split, crush_number_array, crush_string_array};
|
||||
pub use error_keywords::ERROR_KEYWORDS;
|
||||
pub use field_detect::{detect_id_field_statistically, detect_score_field_statistically};
|
||||
pub use hashing::hash_field_name;
|
||||
|
||||
Reference in New Issue
Block a user