fix(smart-crusher): disclose dropped scalar-array items with a retrievable ccr sentinel (#3692)

## Description

On default config, `SmartCrusher` silently truncated long
string/number/mixed arrays down to `max_items_after_crush` and emitted
no disclosure at all — valid JSON with no marker, no sentinel, and
untouched sibling count fields. This contradicts the repo's own "Safety
first: never drop user/assistant content" principle.

The dict-array path already disclosed drops via a `{"_ccr_dropped":
...}` sentinel; scalar arrays had no equivalent. This PR gives them one:
when the `StringArray`/`NumberArray`/`MixedArray` arms drop items and
`enable_ccr_marker` is on, it appends `"… N more items <<ccr:HASH
N_items_offloaded>>"` and stores the full original array in the CCR
store (same SHA256-12 hashing the Python bridge and retrieval use), so
the marker round-trips through `headroom_retrieve`. No-op when nothing
is dropped or the marker gate is off.

Closes #3650

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

| File | What changed |
|---|---|
| `crates/headroom-core/src/transforms/smart_crusher/crusher.rs` | New
`append_scalar_drop_sentinel()`: when
`StringArray`/`NumberArray`/`MixedArray` arms drop items and
`enable_ccr_marker` is on, append the sentinel and store the full
original array in the CCR store. No-op when nothing is dropped or the
marker gate is off. |
| `headroom/transforms/smart_crusher.py` | `is_ccr_sentinel()` now
recognizes the string sentinel shape (previously only the dict shape),
so downstream code can filter it like the dict sentinel; docstring
updates. |
| `tests/test_transforms/test_smart_crusher_issue_3650.py` | New
regression tests: sentinel is emitted with the right drop count, carries
a retrievable CCR marker whose hash resolves to the full original array,
and is suppressed when the marker gate is off. |
| `crates/.../smart_crusher/crusher.rs` (tests module) | Rust regression
tests mirroring the above (sentinel emission, marker retrieval of the
full 120-item array, gate-off suppression). |
| `tests/parity/fixtures/smart_crusher/*.json` (3 files) | Recorded
outputs updated — strategy strings now reflect the trailing sentinel. |

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
cargo test --lib transforms::smart_crusher  → 337 passed, 0 failed
  (includes the new #3650 Rust regression tests)

pytest tests/test_transforms/test_smart_crusher_issue_3650.py \
       tests/test_transforms/test_smart_crusher_bugs.py \
       tests/test_transforms/test_smart_crusher_ccr_roundtrip.py \
       tests/test_transforms/test_smart_crusher_rust_parity.py \
       tests/test_transforms/test_content_router.py  → 106 passed
  (includes 18 Rust-parity fixture tests)

ruff check and ruff format --check on the touched Python files → clean
```

## Real Behavior Proof

- Environment: Linux container, Python 3.12, editable install from
`main` with the maturin-built native `_core` module (no ONNX model
involved — this path is pure Rust crusher).
- Exact command / steps: the repro snippet from #3650, verbatim
(`ContentRouter(ContentRouterConfig())`, 120 slugs + `"total": 120`).
- Observed result: **before** → 16 of 120 kept, `"&lt;&lt;ccr:" in out`
False; **after** → 16 kept + `'… 104 more items &lt;&lt;ccr:4f03caae30bb
104_items_offloaded&gt;&gt;'`, marker present, and the marker's hash
retrieves all 120 originals from the CCR store. The `total` sibling
field is left untouched — the truncation is now disclosed in-band next
to it rather than silently contradicting it.
- Not tested: the ONNX/Kompress model path from #3673 (a separate issue;
this fix is model-independent), Windows/macOS install flows, and the
`lossless_only=True` path (unchanged: nothing dropped there, so no
sentinel is emitted).

## Runtime Rollout Safety

- Rollout-managed feature(s): CCR marker gate (`enable_ccr_marker`)
- Minimum rollout channel: n/a (open-source library; applies to default
config where the marker gate is on)
- Stable/default behavior changed: Yes — truncated scalar arrays now
carry a trailing sentinel string instead of being silently shortened;
sibling count fields untouched
- Kill switch / disable path: set `enable_ccr_marker=False` — no
sentinel emitted, gate-off path unchanged
- Unsafe override required: No
- Qualification impact: None expected
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — library behavior; observed output captured under Real Behavior
Proof.

## Additional Notes

Raphael Malikian — rtmalikian@gmail.com

If you'd like similar help with your own project or codebase, I'm open
to collaborate — feel free to reach out.

*This PR was prepared with AI assistance and reviewed by the author
before submission.*

---------

Signed-off-by: Raphael Malikian <rtmalikian@gmail.com>
This commit is contained in:
Raphael Malikian
2026-09-24 11:19:23 -05:00
committed by GitHub
parent ff60d5764c
commit c1fc84ade6
6 changed files with 310 additions and 11 deletions
@@ -595,20 +595,31 @@ impl SmartCrusher {
let strs: Vec<&str> = arr.iter().filter_map(|v| v.as_str()).collect();
let (crushed, strategy) = crush_string_array(&strs, &self.config, bias);
info_parts.push(format!("{}({}->{})", strategy, n, crushed.len()));
let crushed_values: Vec<Value> =
let mut crushed_values: Vec<Value> =
crushed.into_iter().map(Value::String).collect();
// #3650: dropped strings get a retrievable
// sentinel — silent truncation is a data-loss
// bug even when the marker gate is on.
self.append_scalar_drop_sentinel(&mut crushed_values, arr);
return (Value::Array(crushed_values), info_parts.join(","));
}
ArrayType::NumberArray => {
let (crushed, strategy) = crush_number_array(arr, &self.config, bias);
info_parts.push(format!("{}({}->{})", strategy, n, crushed.len()));
return (Value::Array(crushed), info_parts.join(","));
let mut crushed_values = crushed;
// #3650: same sentinel contract as StringArray.
self.append_scalar_drop_sentinel(&mut crushed_values, arr);
return (Value::Array(crushed_values), info_parts.join(","));
}
ArrayType::MixedArray => {
let (crushed, strategy) =
self.crush_mixed_array(arr, query_context, bias);
info_parts.push(format!("{}({}->{})", strategy, n, crushed.len()));
return (Value::Array(crushed), info_parts.join(","));
let mut crushed_values = crushed;
// #3650: same sentinel contract as the
// StringArray/NumberArray arms.
self.append_scalar_drop_sentinel(&mut crushed_values, arr);
return (Value::Array(crushed_values), info_parts.join(","));
}
// NestedArray, BoolArray, Empty → fall through
// to recursive descent.
@@ -1107,6 +1118,30 @@ impl SmartCrusher {
);
(result, strategy)
}
/// Appends a visible, retrievable CCR sentinel to `kept` when items were
/// dropped from `original` and the marker gate is on.
///
/// Mirrors the dict-array path: the full original array is serialized
/// once (`canonical_array_json`), hashed with the shared
/// `hash_canonical` (SHA256, first 12 hex chars — the same scheme the
/// Python bridge and retrieval use), and stored in the CCR store so
/// the marker round-trips through retrieval. No-op when nothing was
/// dropped or `enable_ccr_marker` is false.
fn append_scalar_drop_sentinel(&self, kept: &mut Vec<Value>, original: &[Value]) {
let dropped = original.len().saturating_sub(kept.len());
if dropped == 0 || !self.config.enable_ccr_marker {
return;
}
let canonical = canonical_array_json(original);
let hash = hash_canonical(&canonical);
if let Some(store) = &self.ccr_store {
store.put(&hash, &canonical);
}
kept.push(Value::String(format!(
"… {dropped} more items <<ccr:{hash} {dropped}_items_offloaded>>"
)));
}
}
// ---------- helpers ----------
@@ -1275,6 +1310,166 @@ mod tests {
use super::*;
use serde_json::json;
/// Extracts the CCR hash from a scalar drop sentinel produced by
/// `append_scalar_drop_sentinel`.
fn scalar_sentinel_hash(s: &str) -> &str {
let start = s.find("<<ccr:").expect("sentinel must carry a CCR marker") + 6;
let end = s[start..].find(' ').expect("marker must have a tag") + start;
&s[start..end]
}
/// Crushes a JSON array string and returns the parsed compressed array.
fn crush_array_values(crusher: &SmartCrusher, items: &[Value]) -> Vec<Value> {
let out = crusher.crush(&serde_json::to_string(items).unwrap(), "", 1.0);
serde_json::from_str::<Value>(&out.compressed)
.expect("compressed output is JSON")
.as_array()
.expect("output stays an array")
.clone()
}
/// Looks up the stored original array for a sentinel's CCR hash.
fn stored_original(crusher: &SmartCrusher, sentinel: &str) -> Vec<Value> {
let hash = scalar_sentinel_hash(sentinel);
let raw = crusher
.ccr_store()
.and_then(|store| store.get(hash))
.expect("sentinel hash must be in the CCR store");
serde_json::from_str(&raw).expect("stored payload is a JSON array")
}
// Regression tests for headroomlabs-ai/headroom#3650: scalar
// (string/number/mixed) array crushers dropped items silently. They
// must now emit a visible, retrievable CCR sentinel.
#[test]
fn string_array_drops_emit_retrievable_sentinel() {
// The issue's repro shape: 120 short slugs.
let items: Vec<Value> = (0..120).map(|i| json!(format!("r{i}"))).collect();
let config = SmartCrusherConfig {
enable_ccr_marker: true,
..Default::default()
};
let crusher = SmartCrusher::new(config);
let kept = crush_array_values(&crusher, &items);
// `kept` includes the trailing sentinel, so the crusher's drop
// count is original minus the data items (all but the last).
let dropped = items.len() - (kept.len() - 1);
assert!(
kept.len() < items.len(),
"expected drops for a 120-item string array"
);
let sentinel = kept
.last()
.and_then(|v| v.as_str())
.expect("last element is the string sentinel");
assert!(
sentinel.starts_with(&format!("… {dropped} more items ")),
"sentinel shows the visible drop count, got: {sentinel}"
);
assert!(
sentinel.contains("<<ccr:")
&& sentinel.ends_with(&format!("{dropped}_items_offloaded>>")),
"sentinel carries a CCR marker, got: {sentinel}"
);
// The marker hash must retrieve the FULL original array.
let stored = stored_original(&crusher, sentinel);
assert_eq!(stored.len(), 120);
assert_eq!(stored[119], json!("r119"));
}
#[test]
fn string_array_sentinel_suppressed_when_marker_gate_off() {
let items: Vec<Value> = (0..120).map(|i| json!(format!("r{i}"))).collect();
let config = SmartCrusherConfig {
enable_ccr_marker: false,
..Default::default()
};
let crusher = SmartCrusher::new(config);
let kept = crush_array_values(&crusher, &items);
assert!(
!kept
.iter()
.any(|v| v.as_str().is_some_and(|s| s.contains("<<ccr:"))),
"no sentinel when the marker gate is off"
);
}
#[test]
fn string_array_no_sentinel_when_nothing_dropped() {
let items: Vec<Value> = (0..5).map(|i| json!(format!("r{i}"))).collect();
let crusher = SmartCrusher::new(SmartCrusherConfig::default());
let kept = crush_array_values(&crusher, &items);
assert_eq!(kept.len(), items.len());
assert!(
!kept
.iter()
.any(|v| v.as_str().is_some_and(|s| s.contains("<<ccr:"))),
"no sentinel when nothing was dropped"
);
}
#[test]
fn number_array_drops_emit_retrievable_sentinel() {
let items: Vec<Value> = (0..120).map(|i| json!(i)).collect();
let crusher = SmartCrusher::new(SmartCrusherConfig::default());
let kept = crush_array_values(&crusher, &items);
// `kept` includes the trailing sentinel, so the crusher's drop
// count is original minus the data items (all but the last).
let dropped = items.len() - (kept.len() - 1);
assert!(
kept.len() < items.len(),
"expected drops for a 120-item number array"
);
let sentinel = kept
.last()
.and_then(|v| v.as_str())
.expect("last element is the string sentinel");
assert!(sentinel.starts_with(&format!("… {dropped} more items ")));
let stored = stored_original(&crusher, sentinel);
assert_eq!(stored.len(), 120);
}
#[test]
fn mixed_array_drops_emit_single_sentinel() {
let items: Vec<Value> = (0..120)
.map(|i| {
if i % 2 == 0 {
json!(format!("entry-{i}"))
} else {
json!(i)
}
})
.collect();
let crusher = SmartCrusher::new(SmartCrusherConfig::default());
let kept = crush_array_values(&crusher, &items);
// `kept` includes the trailing sentinel, so the crusher's drop
// count is original minus the data items (all but the last).
let dropped = items.len() - (kept.len() - 1);
assert!(
kept.len() < items.len(),
"expected drops for a 120-item mixed array"
);
let sentinels: Vec<&str> = kept
.iter()
.filter_map(|v| v.as_str())
.filter(|s| s.contains("_items_offloaded>>"))
.collect();
assert_eq!(
sentinels.len(),
1,
"exactly one sentinel, got {sentinels:?}"
);
assert!(sentinels[0].starts_with(&format!("… {dropped} more items ")));
let stored = stored_original(&crusher, sentinels[0]);
assert_eq!(stored.len(), 120);
}
#[test]
fn default_crush_ignores_opt_in_prose_hook() {
let crusher = SmartCrusher::new(SmartCrusherConfig::default());
+30 -5
View File
@@ -79,18 +79,43 @@ _SUPPORTED_COMPACTION_FORMATS = ("csv-schema", "json", "markdown-kv")
CCR_SENTINEL_KEY = "_ccr_dropped"
# Prefix/infix identifying the scalar-array drop sentinel emitted by the
# Rust crusher when string/number/mixed array items are dropped with
# `enable_ccr_marker` on (headroomlabs-ai/headroom#3650). Unlike the
# dict sentinel object above, this one is a plain string so string
# arrays keep a uniform element type.
SCALAR_SENTINEL_PREFIX = "… "
SCALAR_SENTINEL_INFIX = " more items "
def is_ccr_sentinel(item: Any) -> bool:
"""True if `item` is a CCR-dropped sentinel object."""
return isinstance(item, dict) and CCR_SENTINEL_KEY in item
"""True if `item` is a CCR-dropped sentinel.
Covers both sentinel shapes: the dict object (``{"_ccr_dropped": ...}``)
emitted for dict arrays, and the plain-string sentinel
(``"… N more items <<ccr:HASH N_items_offloaded>>"``) appended to
string/number/mixed arrays when items are dropped
(headroomlabs-ai/headroom#3650).
"""
if isinstance(item, dict):
return CCR_SENTINEL_KEY in item
return (
isinstance(item, str)
and item.startswith(SCALAR_SENTINEL_PREFIX)
and SCALAR_SENTINEL_INFIX in item
and "<<ccr:" in item
)
def strip_ccr_sentinels(items: Any) -> Any:
"""Return `items` with any CCR-dropped sentinel objects filtered out.
Pass this through any iteration over a compressed array's contents
when your code expects a uniform-schema list of records. The sentinel
carries a `<<ccr:HASH ...>>` marker for the LLM and shouldn't be
confused for a record — it has only the `_ccr_dropped` key.
when your code expects a uniform-schema list of records. The dict
sentinel carries a `<<ccr:HASH ...>>` marker for the LLM and shouldn't
be confused for a record — it has only the `_ccr_dropped` key. The
string sentinel (scalar arrays) likewise marks dropped items without
pretending to be data.
Non-list inputs pass through unchanged so callers can wrap whatever
`json.loads` returned without first checking the shape.
@@ -24,7 +24,7 @@
"input_sha256": "3f929019b62fa97b372e2d8aad705dbef029ddb7934acdb237083fdead8162c0",
"label": "mixed_array",
"output": {
"compressed": "[\"start\",0,1,2,3,18,19,\"middle\",\"end\",\"end\",\"end\",\"end\",\"end\"]",
"compressed": "[\"start\",0,1,2,3,18,19,\"middle\",\"end\",\"end\",\"end\",\"end\",\"end\",\"\u2026 14 more items <<ccr:c698823f3fa0 14_items_offloaded>>\"]",
"original": "[\"start\", 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, \"middle\", \"end\", \"end\", \"end\", \"end\", \"end\"]",
"strategy": "mixed:adaptive(27->13,str:7->7,num:20)(27->13)",
"was_modified": true
@@ -24,7 +24,7 @@
"input_sha256": "8cd416c24d53a5c61ebf4ad2e06963169de552cf163331df276773459af00275",
"label": "number_array_40_changepoint",
"output": {
"compressed": "[10,11,12,10,10,10,10,10,10,51,54,57,60,68,69]",
"compressed": "[10,11,12,10,10,10,10,10,10,51,54,57,60,68,69,\"\u2026 25 more items <<ccr:905235220a7b 25_items_offloaded>>\"]",
"original": "[10, 11, 12, 10, 11, 12, 10, 11, 12, 10, 11, 12, 10, 11, 12, 10, 11, 12, 10, 11, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69]",
"strategy": "number:adaptive(40->15,min=10,max=69,mean=35.23,median=31,stddev=24.94,p25=11,p75=59.25)(40->15)",
"was_modified": true
@@ -24,7 +24,7 @@
"input_sha256": "6fe4a307de817df20507df7fc88f0237c203a785c550696abb1f6b8dd413b40e",
"label": "string_array_25",
"output": {
"compressed": "[\"event 0: something happened at index 0\",\"event 1: something happened at index 1\",\"event 2: something happened at index 2\",\"event 3: something happened at index 3\",\"event 4: something happened at index 4\",\"event 6: something happened at index 6\",\"event 8: something happened at index 8\",\"event 10: something happened at index 10\",\"event 12: something happened at index 12\",\"event 14: something happened at index 14\",\"event 16: something happened at index 16\",\"event 18: something happened at index 18\",\"event 20: something happened at index 20\",\"event 23: something happened at index 23\",\"event 24: something happened at index 24\"]",
"compressed": "[\"event 0: something happened at index 0\",\"event 1: something happened at index 1\",\"event 2: something happened at index 2\",\"event 3: something happened at index 3\",\"event 4: something happened at index 4\",\"event 6: something happened at index 6\",\"event 8: something happened at index 8\",\"event 10: something happened at index 10\",\"event 12: something happened at index 12\",\"event 14: something happened at index 14\",\"event 16: something happened at index 16\",\"event 18: something happened at index 18\",\"event 20: something happened at index 20\",\"event 23: something happened at index 23\",\"event 24: something happened at index 24\",\"\u2026 10 more items <<ccr:9b64aca4dbf5 10_items_offloaded>>\"]",
"original": "[\"event 0: something happened at index 0\", \"event 1: something happened at index 1\", \"event 2: something happened at index 2\", \"event 3: something happened at index 3\", \"event 4: something happened at index 4\", \"event 5: something happened at index 5\", \"event 6: something happened at index 6\", \"event 7: something happened at index 7\", \"event 8: something happened at index 8\", \"event 9: something happened at index 9\", \"event 10: something happened at index 10\", \"event 11: something happened at index 11\", \"event 12: something happened at index 12\", \"event 13: something happened at index 13\", \"event 14: something happened at index 14\", \"event 15: something happened at index 15\", \"event 16: something happened at index 16\", \"event 17: something happened at index 17\", \"event 18: something happened at index 18\", \"event 19: something happened at index 19\", \"event 20: something happened at index 20\", \"event 21: something happened at index 21\", \"event 22: something happened at index 22\", \"event 23: something happened at index 23\", \"event 24: something happened at index 24\"]",
"strategy": "string:adaptive(25->15)(25->15)",
"was_modified": true
@@ -0,0 +1,79 @@
"""Regression tests for headroomlabs-ai/headroom#3650.
Scalar (string/number/mixed) arrays used to drop items silently: the
crusher returned only the retained items with no marker, so e.g. 120
slugs came back as 16 with no trace of the other 104. Dropped scalar
items now get a visible, retrievable CCR sentinel appended to the
array.
"""
from __future__ import annotations
import json
import re
import pytest
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
from headroom.transforms.smart_crusher import (
is_ccr_sentinel,
strip_ccr_sentinels,
)
_CCR_RE = re.compile(r"<<ccr:([0-9a-f]{12}) ")
def _issue_payload() -> str:
# The exact shape from the issue report.
return json.dumps({"slugs": [f"r{i}" for i in range(120)], "total": 120})
def test_scalar_sentinel_recognized_by_python_shim() -> None:
"""Pure-Python coverage: the new string sentinel is a CCR sentinel."""
sentinel = "… 104 more items <<ccr:4f03caae30bb 104_items_offloaded>>"
assert is_ccr_sentinel(sentinel)
# The dict sentinel keeps working.
assert is_ccr_sentinel({"_ccr_dropped": "<<ccr:abc123 5_rows_offloaded>>"})
# Real data is untouched.
assert not is_ccr_sentinel("r42")
assert not is_ccr_sentinel({"slugs": ["r1"]})
assert not is_ccr_sentinel(42)
items = ["r0", sentinel, "r1", {"_ccr_dropped": "<<ccr:abc123 1_rows_offloaded>>"}]
assert strip_ccr_sentinels(items) == ["r0", "r1"]
assert strip_ccr_sentinels("not-a-list") == "not-a-list"
def test_issue_3650_router_emits_visible_retrievable_sentinel() -> None:
"""End-to-end: the issue's payload truncates visibly and retrievably."""
pytest.importorskip("headroom._core") # needs the native extension
router = ContentRouter(ContentRouterConfig())
result = router.compress(_issue_payload())
compressed = result.compressed
# 1. The truncation is visible: a sentinel names the drop count and
# carries a CCR marker. (Before the fix, 104 slugs vanished with
# no marker at all.)
assert "more items" in compressed, compressed[:300]
m = _CCR_RE.search(compressed)
assert m, f"no CCR marker in compressed output: {compressed[:300]}"
ccr_hash = m.group(1)
# 2. The marker retrieves the FULL original array: the Python
# SmartCrusher mirrors Rust CCR markers into the process
# compression store.
from headroom.cache.compression_store import get_compression_store
entry = get_compression_store().retrieve(ccr_hash)
assert entry is not None, "CCR marker hash must resolve in the compression store"
original = json.loads(entry.original_content)
assert len(original) == 120
assert original[0] == "r0"
assert original[119] == "r119"
# 3. The sibling field is untouched and the output still parses.
doc = json.loads(compressed)
assert doc["total"] == 120
assert isinstance(doc["slugs"], list)
assert len(doc["slugs"]) < 120 # some were dropped…
assert any(is_ccr_sentinel(x) for x in doc["slugs"]) # …but marked