mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(pipeline): store the pipeline input in CCR behind offload markers (#4576)
## Description
Fixes #4550. Two contract mismatches in the Rust `CompressionPipeline`
(`crates/headroom-core/src/transforms/pipeline/`):
1. Offloads ran on the post-reformat buffer and stored it in CCR, so
with `LogTemplate` in front of `LogOffload` the entry behind the marker
was the templated text, not the pipeline input. That contradicts
`traits.rs` ("original is in supplied store") and `ccr/mod.rs`
("lossless end-to-end").
2. `LogTemplate` was documented as lossless, but
`split_whitespace`/`lines()` drop indentation, tabs, whitespace runs and
`\r`, and a literal `<*>` constant printed exactly like the wildcard.
`CompressionPipeline` has no production caller yet, so this fixes the
contract before it is wired.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `OffloadTransform::apply` takes `ccr_source: &str`; the orchestrator
passes the original pipeline input. Same approach as the SmartCrusher
`ccr_source` fix (#2694).
- All six offloads (`log_offload`, `json_offload`, `diff_offload`,
`diff_noise`, `search_offload`, `prose_field`) key and store
`ccr_source` and still compress `content`. `json_offload`'s prose hook
passes each leaf as its own source, since its marker refers to that
leaf.
- `LogCompressor`, `DiffCompressor` and `SearchCompressor` gain
`compress_with_ccr_source`; the existing entry points delegate with
`content` as the source, so their behavior is unchanged.
- I did not overwrite the existing key with the original instead:
`CcrStore::put` assumes same hash = same content, and two inputs that
differ only in whitespace reformat to the same text.
- `LogTemplate`: the doc now says whitespace is normalized and the exact
original is recoverable only through CCR when an offload runs. A
constant token that is `<*>` behind zero or more backslashes gets one
more leading backslash, so the header is unambiguous.
- Tests: `ccr_stores_pipeline_input_not_reformatted_text` (issue repro:
LogTemplate + LogOffload, `store.get(key) == input`),
`literal_wildcard_constant_header_differs_from_variable_position`.
`lossless_round_trip_via_template_and_variants` is renamed to
`token_round_trip_via_template_and_variants` and compares token lists.
Not done: a whitespace-preserving `LogTemplate` (separators stored in
the variant table). Possible follow-up.
## Testing
- [x] Unit tests pass
- [x] Lint/format checks pass
```
$ cargo fmt -p headroom-core -- --check # exit 0
$ cargo clippy -p headroom-core --all-targets -- -D warnings # exit 0
$ cargo test -p headroom-core --lib
test result: FAILED. 981 passed; 1 failed; 1 ignored
(failure: tokenizer::hf_impl::tests::from_pretrained_invalid_repo_returns_hub_error, unrelated, HF hub network)
$ cargo test -p headroom-core --lib from_pretrained_invalid_repo
test result: ok. 1 passed; 0 failed
```
## Real Behavior Proof
- Environment: Windows 11, Rust 1.95.0, headroom-core default features,
branch based on upstream main a2d4fe72
- Exact command / steps: cargo test -p headroom-core --lib
ccr_stores_pipeline_input_not_reformatted_text (pipeline LogTemplate +
LogOffload + InMemoryCcrStore on 300 INFO + 300 DEBUG + 1 ERROR log
lines, the repro from the issue)
- Observed result: steps_applied contains log_template and log_offload,
and the stored CCR value equals the input byte for byte (before the fix
it started with "[Template T1: ...")
- Not tested: a live proxy/SDK run. `CompressionPipeline` has no
production caller yet (outside `transforms/pipeline/`, tests and benches
it is only re-exported from `transforms/mod.rs`; headroom-proxy,
headroom-py and the Python package do not use it), so there is no
runtime path to observe and the in-process pipeline run above is the
closest real behavior available. Also not run: integration tests that
reach Kompress (live_zone_dispatch, live_zone_kompress_async_init,
live_zone_disable_arms, live_zone_arm_properties plain-text arm,
kompress_parity) hang locally without the cached model; the other
integration test files pass
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, CompressionPipeline has no production
caller
- Stable/default behavior changed: no; compressor entry points
unchanged, pipeline unused in proxy/py
- Kill switch / disable path: n/a, no runtime path uses it
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4576"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
053925b03c
commit
4d7765c84b
@@ -277,6 +277,20 @@ impl DiffCompressor {
|
||||
content: &str,
|
||||
context: &str,
|
||||
store: Option<&dyn CcrStore>,
|
||||
) -> (DiffCompressionResult, DiffCompressorStats) {
|
||||
self.compress_with_ccr_source(content, context, store, content)
|
||||
}
|
||||
|
||||
/// Like [`Self::compress_with_store`], but the CCR entry and its
|
||||
/// hash come from `ccr_source` rather than `content`. Use when
|
||||
/// `content` is a derived form of the original (e.g. after a
|
||||
/// reformat) and retrieval must return the original.
|
||||
pub fn compress_with_ccr_source(
|
||||
&self,
|
||||
content: &str,
|
||||
context: &str,
|
||||
store: Option<&dyn CcrStore>,
|
||||
ccr_source: &str,
|
||||
) -> (DiffCompressionResult, DiffCompressorStats) {
|
||||
let start = Instant::now();
|
||||
let mut stats = DiffCompressorStats::default();
|
||||
@@ -473,7 +487,7 @@ impl DiffCompressor {
|
||||
if self.config.enable_ccr
|
||||
&& (compressed_line_count as f64) < (original_line_count as f64) * savings_threshold
|
||||
{
|
||||
let key = md5_hex_24(content);
|
||||
let key = md5_hex_24(ccr_source);
|
||||
compressed_output.push('\n');
|
||||
compressed_output.push_str(&format!(
|
||||
"[{} lines compressed to {}. Retrieve full diff: hash={}]",
|
||||
@@ -485,7 +499,7 @@ impl DiffCompressor {
|
||||
// (typically the Python shim) is responsible — see the
|
||||
// method-level docs.
|
||||
if let Some(s) = store {
|
||||
s.put(&key, content);
|
||||
s.put(&key, ccr_source);
|
||||
}
|
||||
cache_key = Some(key);
|
||||
stats.cache_key_emitted = true;
|
||||
|
||||
@@ -1139,6 +1139,20 @@ impl LogCompressor {
|
||||
content: &str,
|
||||
bias: f64,
|
||||
store: Option<&dyn CcrStore>,
|
||||
) -> (LogCompressionResult, LogCompressorStats) {
|
||||
self.compress_with_ccr_source(content, bias, store, content)
|
||||
}
|
||||
|
||||
/// Like [`Self::compress_with_store`], but the CCR entry and its
|
||||
/// hash come from `ccr_source` rather than `content`. Use when
|
||||
/// `content` is a derived form of the original (e.g. after a
|
||||
/// reformat) and retrieval must return the original.
|
||||
pub fn compress_with_ccr_source(
|
||||
&self,
|
||||
content: &str,
|
||||
bias: f64,
|
||||
store: Option<&dyn CcrStore>,
|
||||
ccr_source: &str,
|
||||
) -> (LogCompressionResult, LogCompressorStats) {
|
||||
let mut stats = LogCompressorStats::default();
|
||||
let lines: Vec<&str> = content.split('\n').collect();
|
||||
@@ -1177,8 +1191,8 @@ impl LogCompressor {
|
||||
if ratio >= self.config.min_compression_ratio_for_ccr {
|
||||
stats.ccr_skip_reason = Some("compression ratio too high");
|
||||
} else if let Some(store) = store {
|
||||
let key = md5_hex_24(content);
|
||||
store.put(&key, content);
|
||||
let key = md5_hex_24(ccr_source);
|
||||
store.put(&key, ccr_source);
|
||||
// Descriptive so the reader/agent can tell whether the
|
||||
// dropped content is relevant before deciding whether to
|
||||
// retrieve it (rather than just "how much" vanished).
|
||||
|
||||
@@ -52,8 +52,10 @@
|
||||
//! Reformats:
|
||||
//! - [`reformats::JsonMinifier`] — JSON whitespace stripping.
|
||||
//! - [`reformats::LogTemplate`] — Drain-style template miner for
|
||||
//! build/log output. Lossless — emits `[Template Tn: ...] (Nx)` +
|
||||
//! variant table, every original line reconstructible.
|
||||
//! build/log output. Tokens preserved, whitespace normalized —
|
||||
//! emits `[Template Tn: ...] (Nx)` + variant table; the exact
|
||||
//! original is recoverable only through CCR when an offload runs
|
||||
//! after the reformat.
|
||||
//!
|
||||
//! Offloads:
|
||||
//! - [`offloads::JsonOffload`] — wraps `SmartCrusher` for JSON arrays
|
||||
|
||||
@@ -117,6 +117,7 @@ impl OffloadTransform for DiffNoise {
|
||||
content: &str,
|
||||
_ctx: &CompressionContext,
|
||||
store: &dyn CcrStore,
|
||||
ccr_source: &str,
|
||||
) -> Result<OffloadOutput, TransformError> {
|
||||
let segments = parse_segments(content);
|
||||
if segments.is_empty() {
|
||||
@@ -173,8 +174,8 @@ impl OffloadTransform for DiffNoise {
|
||||
}
|
||||
|
||||
// CCR: hash original, stash, append marker.
|
||||
let key = md5_hex_24(content);
|
||||
store.put(&key, content);
|
||||
let key = md5_hex_24(ccr_source);
|
||||
store.put(&key, ccr_source);
|
||||
output.push_str(&format!("\n[diff_noise CCR: hash={key}]"));
|
||||
|
||||
Ok(OffloadOutput::from_lengths(content.len(), output, key))
|
||||
@@ -432,7 +433,7 @@ mod tests {
|
||||
]);
|
||||
let store = InMemoryCcrStore::new();
|
||||
let r = offload()
|
||||
.apply(&diff, &CompressionContext::default(), &store)
|
||||
.apply(&diff, &CompressionContext::default(), &store, &diff)
|
||||
.expect("must compress");
|
||||
assert!(r.bytes_saved > 0);
|
||||
assert!(r.output.contains("[diff_noise: lockfile hunks dropped"));
|
||||
@@ -451,7 +452,7 @@ mod tests {
|
||||
let diff = build_diff(&[("src/main.rs", &body_refs)]);
|
||||
let store = InMemoryCcrStore::new();
|
||||
let r = offload()
|
||||
.apply(&diff, &CompressionContext::default(), &store)
|
||||
.apply(&diff, &CompressionContext::default(), &store, &diff)
|
||||
.expect("must compress");
|
||||
assert!(r
|
||||
.output
|
||||
@@ -468,7 +469,7 @@ mod tests {
|
||||
let diff = build_diff(&[("src/main.rs", &body_refs)]);
|
||||
let store = InMemoryCcrStore::new();
|
||||
let err = offload()
|
||||
.apply(&diff, &CompressionContext::default(), &store)
|
||||
.apply(&diff, &CompressionContext::default(), &store, &diff)
|
||||
.expect_err("nothing droppable");
|
||||
match err {
|
||||
TransformError::Skipped { .. } => {}
|
||||
@@ -502,7 +503,7 @@ mod tests {
|
||||
diff.push_str(&build_diff(&[("yarn.lock", &lock_refs)]));
|
||||
let store = InMemoryCcrStore::new();
|
||||
let r = offload()
|
||||
.apply(&diff, &CompressionContext::default(), &store)
|
||||
.apply(&diff, &CompressionContext::default(), &store, &diff)
|
||||
.expect("compresses");
|
||||
// Pre-diff prelude must survive.
|
||||
assert!(r.output.contains("Subject: bump deps"));
|
||||
@@ -514,7 +515,7 @@ mod tests {
|
||||
assert_eq!(offload().estimate_bloat(""), 0.0);
|
||||
let store = InMemoryCcrStore::new();
|
||||
let err = offload()
|
||||
.apply("", &CompressionContext::default(), &store)
|
||||
.apply("", &CompressionContext::default(), &store, "")
|
||||
.expect_err("must skip");
|
||||
match err {
|
||||
TransformError::Skipped { .. } => {}
|
||||
|
||||
@@ -132,15 +132,14 @@ impl OffloadTransform for DiffOffload {
|
||||
content: &str,
|
||||
ctx: &CompressionContext,
|
||||
store: &dyn CcrStore,
|
||||
ccr_source: &str,
|
||||
) -> Result<OffloadOutput, TransformError> {
|
||||
// `compress_with_store` (added in the audit-cleanup PR) writes
|
||||
// the original to `store` under the same `cache_key` it embeds
|
||||
// in the marker. Earlier versions of this offload had to
|
||||
// double-store post-hoc because the compressor lacked a store
|
||||
// parameter — that hack is gone.
|
||||
let (result, _) = self
|
||||
.compressor
|
||||
.compress_with_store(content, &ctx.query, Some(store));
|
||||
// The compressor writes `ccr_source` to `store` under the same
|
||||
// `cache_key` it embeds in the marker, so retrieval returns the
|
||||
// pipeline input rather than the reformatted `content`.
|
||||
let (result, _) =
|
||||
self.compressor
|
||||
.compress_with_ccr_source(content, &ctx.query, Some(store), ccr_source);
|
||||
|
||||
let Some(key) = result.cache_key else {
|
||||
return Err(TransformError::skipped(
|
||||
@@ -254,7 +253,7 @@ mod tests {
|
||||
let diff = build_diff(1, 200, 5);
|
||||
let store = InMemoryCcrStore::new();
|
||||
let r = offload()
|
||||
.apply(&diff, &CompressionContext::default(), &store)
|
||||
.apply(&diff, &CompressionContext::default(), &store, &diff)
|
||||
.expect("should compress");
|
||||
assert!(!r.cache_key.is_empty());
|
||||
// Bug-fix verification: DiffCompressor itself wouldn't have
|
||||
@@ -268,7 +267,7 @@ mod tests {
|
||||
let diff = build_diff(1, 5, 2);
|
||||
let store = InMemoryCcrStore::new();
|
||||
let err = offload()
|
||||
.apply(&diff, &CompressionContext::default(), &store)
|
||||
.apply(&diff, &CompressionContext::default(), &store, &diff)
|
||||
.expect_err("must skip");
|
||||
match err {
|
||||
TransformError::Skipped { transform, .. } => assert_eq!(transform, "diff_offload"),
|
||||
|
||||
@@ -147,12 +147,15 @@ impl OffloadTransform for JsonOffload {
|
||||
content: &str,
|
||||
ctx: &CompressionContext,
|
||||
store: &dyn CcrStore,
|
||||
ccr_source: &str,
|
||||
) -> Result<OffloadOutput, TransformError> {
|
||||
let prose = &self.prose;
|
||||
let prose_hook = |leaf: &str, query: &str| {
|
||||
let leaf_ctx = CompressionContext::with_query(query);
|
||||
// Each leaf is its own CCR source: its marker resolves to the
|
||||
// leaf text the crusher saw.
|
||||
prose
|
||||
.apply(leaf, &leaf_ctx, store)
|
||||
.apply(leaf, &leaf_ctx, store, leaf)
|
||||
.ok()
|
||||
.map(|output| (output.output, output.cache_key))
|
||||
};
|
||||
@@ -173,9 +176,10 @@ impl OffloadTransform for JsonOffload {
|
||||
// through the orchestrator-supplied store, append a marker so
|
||||
// the LLM sees a resolvable handle. SmartCrusher's per-array
|
||||
// markers in `result.compressed` stay informational — the LLM
|
||||
// retrieves the full original via this outer hash.
|
||||
let key = md5_hex_24(content);
|
||||
store.put(&key, content);
|
||||
// retrieves the full original via this outer hash. The hash and
|
||||
// stored payload come from `ccr_source` (the pipeline input).
|
||||
let key = md5_hex_24(ccr_source);
|
||||
store.put(&key, ccr_source);
|
||||
let mut output = result.compressed;
|
||||
output.push_str("\n[json_offload CCR: hash=");
|
||||
output.push_str(&key);
|
||||
@@ -298,7 +302,7 @@ mod tests {
|
||||
let input = serde_json::json!([{"description": diff}]).to_string();
|
||||
let store = InMemoryCcrStore::new();
|
||||
let output = offload()
|
||||
.apply(&input, &CompressionContext::default(), &store)
|
||||
.apply(&input, &CompressionContext::default(), &store, &input)
|
||||
.expect("diff-shaped leaf should still process through the direct offload path");
|
||||
let marker = output
|
||||
.output
|
||||
@@ -318,7 +322,12 @@ mod tests {
|
||||
let input = serde_json::json!([{"summary": html}]).to_string();
|
||||
let store = InMemoryCcrStore::new();
|
||||
let result = offload()
|
||||
.apply(&input, &CompressionContext::with_query("recovery"), &store)
|
||||
.apply(
|
||||
&input,
|
||||
&CompressionContext::with_query("recovery"),
|
||||
&store,
|
||||
&input,
|
||||
)
|
||||
.expect("structured html should still process");
|
||||
assert!(result.output.contains(",html,"));
|
||||
}
|
||||
@@ -424,7 +433,7 @@ mod tests {
|
||||
let arr = build_tabular_array(500);
|
||||
let store = InMemoryCcrStore::new();
|
||||
let r = offload()
|
||||
.apply(&arr, &CompressionContext::default(), &store)
|
||||
.apply(&arr, &CompressionContext::default(), &store, &arr)
|
||||
.expect("smart crusher should compress");
|
||||
assert!(r.bytes_saved > 0);
|
||||
assert!(!r.cache_key.is_empty());
|
||||
@@ -441,7 +450,7 @@ mod tests {
|
||||
let arr = build_tabular_array(2);
|
||||
let store = InMemoryCcrStore::new();
|
||||
let err = offload()
|
||||
.apply(&arr, &CompressionContext::default(), &store)
|
||||
.apply(&arr, &CompressionContext::default(), &store, &arr)
|
||||
.expect_err("must skip");
|
||||
match err {
|
||||
TransformError::Skipped { transform, .. } => assert_eq!(transform, "json_offload"),
|
||||
@@ -454,7 +463,12 @@ mod tests {
|
||||
fn apply_skipped_for_non_json_input() {
|
||||
let store = InMemoryCcrStore::new();
|
||||
let err = offload()
|
||||
.apply("not json at all", &CompressionContext::default(), &store)
|
||||
.apply(
|
||||
"not json at all",
|
||||
&CompressionContext::default(),
|
||||
&store,
|
||||
"not json at all",
|
||||
)
|
||||
.expect_err("must skip non-json");
|
||||
match err {
|
||||
TransformError::Skipped { .. } => {}
|
||||
@@ -487,7 +501,7 @@ mod tests {
|
||||
let store = InMemoryCcrStore::new();
|
||||
let ctx = CompressionContext::with_query("needle");
|
||||
let r = offload()
|
||||
.apply(&s, &ctx, &store)
|
||||
.apply(&s, &ctx, &store, &s)
|
||||
.expect("crusher should run");
|
||||
// Output should reference "needle" — SmartCrusher's anchor logic
|
||||
// should keep that row.
|
||||
@@ -503,10 +517,10 @@ mod tests {
|
||||
let store_a = InMemoryCcrStore::new();
|
||||
let store_b = InMemoryCcrStore::new();
|
||||
let r_a = offload()
|
||||
.apply(&arr, &CompressionContext::default(), &store_a)
|
||||
.apply(&arr, &CompressionContext::default(), &store_a, &arr)
|
||||
.expect("ok");
|
||||
let r_b = offload()
|
||||
.apply(&arr, &CompressionContext::default(), &store_b)
|
||||
.apply(&arr, &CompressionContext::default(), &store_b, &arr)
|
||||
.expect("ok");
|
||||
assert_eq!(
|
||||
r_a.cache_key, r_b.cache_key,
|
||||
|
||||
@@ -145,10 +145,11 @@ impl OffloadTransform for LogOffload {
|
||||
content: &str,
|
||||
_ctx: &CompressionContext,
|
||||
store: &dyn CcrStore,
|
||||
ccr_source: &str,
|
||||
) -> Result<OffloadOutput, TransformError> {
|
||||
let (result, stats) = self
|
||||
.compressor
|
||||
.compress_with_store(content, self.bias, Some(store));
|
||||
let (result, stats) =
|
||||
self.compressor
|
||||
.compress_with_ccr_source(content, self.bias, Some(store), ccr_source);
|
||||
|
||||
// The trait contract says `cache_key` is required. If the
|
||||
// underlying compressor decided post-hoc that the offload
|
||||
@@ -258,7 +259,7 @@ mod tests {
|
||||
let log: String = line.repeat(200);
|
||||
let store = InMemoryCcrStore::new();
|
||||
let r = offload()
|
||||
.apply(&log, &CompressionContext::default(), &store)
|
||||
.apply(&log, &CompressionContext::default(), &store, &log)
|
||||
.expect("offload should produce a key on bloaty input");
|
||||
assert!(!r.cache_key.is_empty());
|
||||
assert_eq!(store.get(&r.cache_key).as_deref(), Some(log.as_str()));
|
||||
@@ -273,7 +274,7 @@ mod tests {
|
||||
let log = "INFO: a\nINFO: b\nINFO: c\nINFO: d\nINFO: e";
|
||||
let store = InMemoryCcrStore::new();
|
||||
let err = offload()
|
||||
.apply(log, &CompressionContext::default(), &store)
|
||||
.apply(log, &CompressionContext::default(), &store, log)
|
||||
.expect_err("must skip");
|
||||
match err {
|
||||
TransformError::Skipped { transform, .. } => assert_eq!(transform, "log_offload"),
|
||||
|
||||
@@ -34,7 +34,9 @@ impl ProseFieldOffload {
|
||||
&& detect_content_type(content).content_type == ContentType::PlainText
|
||||
}
|
||||
|
||||
fn compress(&self, content: &str, query: &str) -> Option<(String, String)> {
|
||||
/// Returns `(output, key)`. The key hashes `ccr_source`, the payload
|
||||
/// the caller will store; `content` is only the text compressed.
|
||||
fn compress(&self, content: &str, query: &str, ccr_source: &str) -> Option<(String, String)> {
|
||||
if !self.eligible(content) {
|
||||
return None;
|
||||
}
|
||||
@@ -45,7 +47,7 @@ impl ProseFieldOffload {
|
||||
return None;
|
||||
}
|
||||
|
||||
let key = compute_key(content.as_bytes());
|
||||
let key = compute_key(ccr_source.as_bytes());
|
||||
let output = format!("{}\n{}", result.compressed, marker_for(&key));
|
||||
(output.len() < content.len()).then_some((output, key))
|
||||
}
|
||||
@@ -80,14 +82,15 @@ impl OffloadTransform for ProseFieldOffload {
|
||||
content: &str,
|
||||
ctx: &CompressionContext,
|
||||
store: &dyn CcrStore,
|
||||
ccr_source: &str,
|
||||
) -> Result<OffloadOutput, TransformError> {
|
||||
let Some((output, key)) = self.compress(content, &ctx.query) else {
|
||||
let Some((output, key)) = self.compress(content, &ctx.query, ccr_source) else {
|
||||
return Err(TransformError::skipped(
|
||||
NAME,
|
||||
"prose compression not worth it",
|
||||
));
|
||||
};
|
||||
store.put(&key, content);
|
||||
store.put(&key, ccr_source);
|
||||
Ok(OffloadOutput::from_lengths(content.len(), output, key))
|
||||
}
|
||||
|
||||
@@ -123,7 +126,7 @@ mod tests {
|
||||
fn short_plain_text_is_byte_identical() {
|
||||
let input = "A short note.";
|
||||
let store = InMemoryCcrStore::new();
|
||||
let result = offload().apply(input, &CompressionContext::default(), &store);
|
||||
let result = offload().apply(input, &CompressionContext::default(), &store, input);
|
||||
assert!(result.is_err());
|
||||
assert!(store.is_empty());
|
||||
}
|
||||
@@ -132,7 +135,7 @@ mod tests {
|
||||
fn long_low_segment_text_is_byte_identical() {
|
||||
let input = "A".repeat(300);
|
||||
let store = InMemoryCcrStore::new();
|
||||
let result = offload().apply(&input, &CompressionContext::default(), &store);
|
||||
let result = offload().apply(&input, &CompressionContext::default(), &store, &input);
|
||||
assert!(result.is_err());
|
||||
assert!(store.is_empty());
|
||||
}
|
||||
@@ -159,8 +162,8 @@ mod tests {
|
||||
fn query_changes_selection_deterministically() {
|
||||
let input = prose();
|
||||
let crusher = offload();
|
||||
let a = crusher.compress(&input, "recovery").unwrap();
|
||||
let b = crusher.compress(&input, "recovery").unwrap();
|
||||
let a = crusher.compress(&input, "recovery", &input).unwrap();
|
||||
let b = crusher.compress(&input, "recovery", &input).unwrap();
|
||||
assert_eq!(a, b);
|
||||
assert!(a.0.contains("recovery"));
|
||||
assert!(!a
|
||||
|
||||
@@ -116,10 +116,15 @@ impl OffloadTransform for SearchOffload {
|
||||
content: &str,
|
||||
ctx: &CompressionContext,
|
||||
store: &dyn CcrStore,
|
||||
ccr_source: &str,
|
||||
) -> Result<OffloadOutput, TransformError> {
|
||||
let (result, stats) =
|
||||
self.compressor
|
||||
.compress_with_store(content, &ctx.query, self.bias, Some(store));
|
||||
let (result, stats) = self.compressor.compress_with_ccr_source(
|
||||
content,
|
||||
&ctx.query,
|
||||
self.bias,
|
||||
Some(store),
|
||||
ccr_source,
|
||||
);
|
||||
|
||||
let Some(key) = result.cache_key else {
|
||||
let reason = stats.ccr_skip_reason.unwrap_or("no cache_key emitted");
|
||||
@@ -309,7 +314,7 @@ mod tests {
|
||||
.join("\n");
|
||||
let store = InMemoryCcrStore::new();
|
||||
let r = offload()
|
||||
.apply(&s, &CompressionContext::default(), &store)
|
||||
.apply(&s, &CompressionContext::default(), &store, &s)
|
||||
.expect("offload should produce a key on clustered input");
|
||||
assert!(!r.cache_key.is_empty());
|
||||
assert_eq!(store.get(&r.cache_key).as_deref(), Some(s.as_str()));
|
||||
@@ -321,7 +326,7 @@ mod tests {
|
||||
let s = "only.py:1:trivial";
|
||||
let store = InMemoryCcrStore::new();
|
||||
let err = offload()
|
||||
.apply(s, &CompressionContext::default(), &store)
|
||||
.apply(s, &CompressionContext::default(), &store, s)
|
||||
.expect_err("must skip");
|
||||
match err {
|
||||
TransformError::Skipped { transform, .. } => assert_eq!(transform, "search_offload"),
|
||||
|
||||
@@ -153,7 +153,9 @@ impl CompressionPipeline {
|
||||
);
|
||||
continue;
|
||||
}
|
||||
match offload.apply(¤t, ctx, store) {
|
||||
// CCR stores the pipeline input (`content`), not the
|
||||
// reformatted `current`, so retrieval returns the original.
|
||||
match offload.apply(¤t, ctx, store, content) {
|
||||
Ok(out) => {
|
||||
if out.bytes_saved == 0 {
|
||||
tracing::trace!(
|
||||
@@ -422,11 +424,12 @@ mod tests {
|
||||
content: &str,
|
||||
_ctx: &CompressionContext,
|
||||
store: &dyn CcrStore,
|
||||
ccr_source: &str,
|
||||
) -> Result<OffloadOutput, TransformError> {
|
||||
// Always halve; emit a cache_key derived from name.
|
||||
let half = &content[..content.len() / 2];
|
||||
let key = format!("test_{}_key", self.name);
|
||||
store.put(&key, content);
|
||||
store.put(&key, ccr_source);
|
||||
Ok(OffloadOutput::from_lengths(
|
||||
content.len(),
|
||||
half.to_string(),
|
||||
@@ -644,6 +647,7 @@ mod tests {
|
||||
_content: &str,
|
||||
_ctx: &CompressionContext,
|
||||
_store: &dyn CcrStore,
|
||||
_ccr_source: &str,
|
||||
) -> Result<OffloadOutput, TransformError> {
|
||||
Err(TransformError::internal("always_internal_err", "by design"))
|
||||
}
|
||||
@@ -699,7 +703,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn end_to_end_log_template_collapses_then_log_offload_can_run() {
|
||||
// LogTemplate runs first (lossless reformat), then LogOffload
|
||||
// LogTemplate runs first (token-preserving reformat), then LogOffload
|
||||
// sees the collapsed output and may further drop low-priority
|
||||
// lines. Both should appear in steps_applied if both fire.
|
||||
let cfg = PipelineConfig::default();
|
||||
@@ -733,6 +737,42 @@ mod tests {
|
||||
assert!(r.output.len() < log.len());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ccr_stores_pipeline_input_not_reformatted_text() {
|
||||
// LogTemplate rewrites the log before LogOffload runs. The CCR
|
||||
// entry behind LogOffload's marker must still be the original
|
||||
// pipeline input, so retrieval gives back the untouched log.
|
||||
let cfg = PipelineConfig::default();
|
||||
let p = CompressionPipeline::builder()
|
||||
.with_reformat(LogTemplate::new(cfg.reformat.log_template))
|
||||
.with_offload(LogOffload::new(cfg.bloat.log))
|
||||
.with_config(cfg)
|
||||
.build();
|
||||
let s = store();
|
||||
let mut input = String::new();
|
||||
for i in 0..300 {
|
||||
input.push_str(&format!(
|
||||
"2025-01-15T12:{:02}:{:02} INFO processing job {i}\n",
|
||||
i / 60 % 60,
|
||||
i % 60
|
||||
));
|
||||
}
|
||||
for i in 0..300 {
|
||||
input.push_str(&format!(
|
||||
"2025-01-15T13:{:02}:{:02} DEBUG heartbeat ok\n",
|
||||
i / 60 % 60,
|
||||
i % 60
|
||||
));
|
||||
}
|
||||
input.push_str("2025-01-15T15:00:00 ERROR database connection lost\n");
|
||||
|
||||
let r = p.run(&input, ContentType::BuildOutput, &ctx(), &s);
|
||||
assert!(r.steps_applied.iter().any(|n| n == "log_template"));
|
||||
assert!(r.steps_applied.iter().any(|n| n == "log_offload"));
|
||||
let key = r.cache_keys.first().expect("log_offload emits a cache key");
|
||||
assert_eq!(s.get(key).as_deref(), Some(input.as_str()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn end_to_end_diff_noise_drops_lockfile_then_diff_offload_handles_rest() {
|
||||
let cfg = PipelineConfig::default();
|
||||
|
||||
@@ -23,9 +23,17 @@
|
||||
//! ...
|
||||
//! ```
|
||||
//!
|
||||
//! Every original line is reconstructible from `template + variants`,
|
||||
//! so this is **lossless** — no CCR retrieval needed. The win is the
|
||||
//! template prefix (often 30+ chars) emitted once instead of N times.
|
||||
//! Every line is reconstructible from `template + variants` at token
|
||||
//! level: the tokens are kept, but whitespace is normalized. Runs of
|
||||
//! spaces, tabs, indentation, and `\r` are not preserved, so the
|
||||
//! output is not a byte-exact copy of the input. The exact original is
|
||||
//! recoverable only through CCR, when an offload runs after this
|
||||
//! reformat. The win is the template prefix (often 30+ chars) emitted
|
||||
//! once instead of N times.
|
||||
//!
|
||||
//! Constant tokens that are `<*>` behind zero or more backslashes are
|
||||
//! escaped with one more leading backslash (`<*>` → `\<*>`), so a
|
||||
//! header position reads as a wildcard only when it is exactly `<*>`.
|
||||
//!
|
||||
//! Order is preserved: only *consecutive* runs collapse, so the
|
||||
//! temporal flow of the log stays intact for the LLM.
|
||||
@@ -244,7 +252,7 @@ impl LogTemplate {
|
||||
out.push(' ');
|
||||
}
|
||||
match slot {
|
||||
Some(constant) => out.push_str(constant),
|
||||
Some(constant) => push_constant(out, constant),
|
||||
None => out.push_str(WILDCARD),
|
||||
}
|
||||
}
|
||||
@@ -289,6 +297,17 @@ impl Run {
|
||||
}
|
||||
}
|
||||
|
||||
/// Write a constant template token. A literal `<*>` would print the
|
||||
/// same as the wildcard, and a literal `\<*>` (any number of leading
|
||||
/// backslashes) would read as an escape, so each gets one extra
|
||||
/// leading backslash.
|
||||
fn push_constant(out: &mut String, token: &str) {
|
||||
if token.trim_start_matches('\\') == WILDCARD {
|
||||
out.push('\\');
|
||||
}
|
||||
out.push_str(token);
|
||||
}
|
||||
|
||||
/// Whitespace-split tokenizer. Empty result = blank line.
|
||||
///
|
||||
/// Uses `str::split_whitespace` semantics: collapses runs of whitespace,
|
||||
@@ -357,7 +376,7 @@ mod tests {
|
||||
r.output.chars().take(200).collect::<String>()
|
||||
);
|
||||
assert!(r.output.contains("(50 occurrences)"));
|
||||
// Variants should still be in output (lossless guarantee).
|
||||
// Variants should still be in output (token-level guarantee).
|
||||
assert!(r.output.contains("worker-7"));
|
||||
}
|
||||
|
||||
@@ -384,9 +403,10 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lossless_round_trip_via_template_and_variants() {
|
||||
// The flushed output must be reconstructible: each template
|
||||
// line + its variant rows reproduces the original input lines.
|
||||
fn token_round_trip_via_template_and_variants() {
|
||||
// The flushed output must be reconstructible at token level:
|
||||
// each template line + its variant rows reproduces the original
|
||||
// tokens. Whitespace is normalized, so compare token lists.
|
||||
let mut log = String::new();
|
||||
for i in 0..25 {
|
||||
log.push_str(&format!("TOK1 TOK2 var{i} TOK3\n"));
|
||||
@@ -420,7 +440,14 @@ mod tests {
|
||||
full[var_pos] = var_tokens[0];
|
||||
reconstructed.push(full.join(" "));
|
||||
}
|
||||
let original: Vec<String> = log.lines().map(|s| s.to_string()).collect();
|
||||
let original: Vec<Vec<String>> = log
|
||||
.lines()
|
||||
.map(|s| s.split_whitespace().map(str::to_string).collect())
|
||||
.collect();
|
||||
let reconstructed: Vec<Vec<String>> = reconstructed
|
||||
.iter()
|
||||
.map(|s: &String| s.split_whitespace().map(str::to_string).collect())
|
||||
.collect();
|
||||
assert_eq!(reconstructed, original);
|
||||
}
|
||||
|
||||
@@ -463,7 +490,7 @@ mod tests {
|
||||
}
|
||||
// These all share token-count and "event-" prefix is varying.
|
||||
// Similarity might trigger collapse — that's fine if it does
|
||||
// (still lossless), but we mainly assert no panic + no
|
||||
// (still token-level preserving), but we mainly assert no panic + no
|
||||
// information loss.
|
||||
let r = reformat().apply(&log).expect("processes");
|
||||
// Either way: every variant value must survive.
|
||||
@@ -542,4 +569,29 @@ mod tests {
|
||||
let r = reformat().apply(&log).expect("processes");
|
||||
assert!(!r.output.contains("[Template"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn literal_wildcard_constant_header_differs_from_variable_position() {
|
||||
// "<*>" as a literal constant and "x{i}" as a varying token must
|
||||
// not print the same template header.
|
||||
let mut cfg = cfg();
|
||||
cfg.min_lines = 10;
|
||||
let r = LogTemplate::new(cfg);
|
||||
let mut literal = String::new();
|
||||
let mut varying = String::new();
|
||||
for i in 0..30 {
|
||||
literal.push_str(&format!("svc up <*> ok {i}\n"));
|
||||
varying.push_str(&format!("svc up x{i} ok {i}\n"));
|
||||
}
|
||||
let header = |log: &str| {
|
||||
let out = r.apply(log).expect("collapses").output;
|
||||
out.lines().next().expect("has header").to_string()
|
||||
};
|
||||
let literal_header = header(&literal);
|
||||
let varying_header = header(&varying);
|
||||
assert!(literal_header.starts_with("[Template T1:"));
|
||||
assert!(varying_header.starts_with("[Template T1:"));
|
||||
assert_ne!(literal_header, varying_header);
|
||||
assert!(literal_header.contains("\\<*>"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -211,13 +211,20 @@ pub trait OffloadTransform: Send + Sync {
|
||||
|
||||
/// Run the offload. The orchestrator only calls this when
|
||||
/// `estimate_bloat(content) ≥ threshold`. On success, the
|
||||
/// returned `cache_key` MUST resolve to the original payload
|
||||
/// in `store`.
|
||||
/// returned `cache_key` MUST resolve to `ccr_source` in `store`.
|
||||
///
|
||||
/// `content` is the text to compress: the output of the prior
|
||||
/// reformat/offload stages. `ccr_source` is the original pipeline
|
||||
/// input, before any reformat ran. Offloads key and store
|
||||
/// `ccr_source`, not `content`, so a retrieval marker resolves to
|
||||
/// exactly what the caller passed in. Storing `content` would
|
||||
/// capture reformatted text instead.
|
||||
fn apply(
|
||||
&self,
|
||||
content: &str,
|
||||
ctx: &CompressionContext,
|
||||
store: &dyn CcrStore,
|
||||
ccr_source: &str,
|
||||
) -> Result<OffloadOutput, TransformError>;
|
||||
|
||||
/// Calibrated 0.0–1.0 quality score for telemetry. Future PR4
|
||||
@@ -264,9 +271,10 @@ mod tests {
|
||||
content: &str,
|
||||
_ctx: &CompressionContext,
|
||||
store: &dyn CcrStore,
|
||||
ccr_source: &str,
|
||||
) -> Result<OffloadOutput, TransformError> {
|
||||
let key = format!("test_key_{:024x}", content.len());
|
||||
store.put(&key, content);
|
||||
let key = format!("test_key_{:024x}", ccr_source.len());
|
||||
store.put(&key, ccr_source);
|
||||
Ok(OffloadOutput::from_lengths(
|
||||
content.len(),
|
||||
content.to_string(),
|
||||
@@ -322,7 +330,7 @@ mod tests {
|
||||
let store = InMemoryCcrStore::new();
|
||||
let t = TestOffload { bloat: 0.9 };
|
||||
let r = t
|
||||
.apply("hello", &CompressionContext::default(), &store)
|
||||
.apply("hello", &CompressionContext::default(), &store, "hello")
|
||||
.expect("offload writes");
|
||||
// Trait contract: cache_key is required and resolves in the store.
|
||||
assert!(!r.cache_key.is_empty());
|
||||
|
||||
@@ -295,6 +295,21 @@ impl SearchCompressor {
|
||||
context: &str,
|
||||
bias: f64,
|
||||
store: Option<&dyn CcrStore>,
|
||||
) -> (SearchCompressionResult, SearchCompressorStats) {
|
||||
self.compress_with_ccr_source(content, context, bias, store, content)
|
||||
}
|
||||
|
||||
/// Like [`Self::compress_with_store`], but the CCR entry and its
|
||||
/// hash come from `ccr_source` rather than `content`. Use when
|
||||
/// `content` is a derived form of the original (e.g. after a
|
||||
/// reformat) and retrieval must return the original.
|
||||
pub fn compress_with_ccr_source(
|
||||
&self,
|
||||
content: &str,
|
||||
context: &str,
|
||||
bias: f64,
|
||||
store: Option<&dyn CcrStore>,
|
||||
ccr_source: &str,
|
||||
) -> (SearchCompressionResult, SearchCompressorStats) {
|
||||
let mut stats = SearchCompressorStats::default();
|
||||
let parsed = self.parse_search_results(content, &mut stats);
|
||||
@@ -334,8 +349,8 @@ impl SearchCompressor {
|
||||
} else if ratio >= self.config.min_compression_ratio_for_ccr {
|
||||
stats.ccr_skip_reason = Some("compression ratio too high");
|
||||
} else if let Some(store) = store {
|
||||
let key = md5_hex_24(content);
|
||||
store.put(&key, content);
|
||||
let key = md5_hex_24(ccr_source);
|
||||
store.put(&key, ccr_source);
|
||||
let marker = format!(
|
||||
"\n[{} matches compressed to {}. Retrieve more: hash={}]",
|
||||
original_count, compressed_count, key
|
||||
|
||||
Reference in New Issue
Block a user