fix(live-zone): leave source code unchanged when the code compressor declines

When CodeAwareCompressor returned a SourceCode block unchanged, the Rust
live-zone dispatcher handed it to Kompress. Kompress is a prose model that
drops low-salience words, which for code means keywords such as `for`, `in`
and `if` (#4214). This is the Rust counterpart of the Python router change:
a decline is now a NoOp, so the block is forwarded byte-for-byte.

Tests: a single-test binary proving the Kompress slot is never consulted for
a declined block (fails before, passes after, needs no model), a dispatch
test with the model warm, and a guard for the script quoted in #4214.
This commit is contained in:
Tao Hansen
2026-11-11 14:29:09 +01:00
parent 0c4a5e63bc
commit 9df60c7c35
5 changed files with 204 additions and 38 deletions
@@ -865,10 +865,11 @@ impl CodeAwareCompressor {
// Unknown language → fallback (Kompress) or passthrough.
if detected_lang == CodeLanguage::Unknown {
// Kompress fallback is delegated to the live-zone dispatcher in
// the Rust port (the engine doesn't own the model); fixtures are
// recorded with fallback_to_kompress=False, so this is the
// passthrough branch.
// The Rust engine does not own the Kompress model, so this is
// always the passthrough branch (fixtures are recorded with
// fallback_to_kompress=False). The live-zone dispatcher leaves
// the declined block unchanged too: code is never handed to the
// prose model, which drops keywords (#4214).
return CodeCompressionResult {
language: CodeLanguage::Unknown,
language_confidence: 0.0,
@@ -58,9 +58,9 @@
//! per-content-type byte thresholds. The gate and the threshold
//! plumbing already shipped; **this PR** completes the item by wiring
//! the two remaining compressor arms — `SourceCode` →
//! CodeAwareCompressor (falling back to cached Kompress when the
//! compressor returns its input unchanged) and `PlainText` → cached
//! Kompress — and by moving the `SourceCode` / `PlainText` thresholds
//! CodeAwareCompressor (left unchanged when the compressor returns its
//! input unchanged: code is never handed to the prose model, #4214) and
//! `PlainText` → cached Kompress — and by moving the `SourceCode` / `PlainText` thresholds
//! off their placeholder 512 to the specified 2048 / 5120. `Html`
//! stays no-op.
//! - **PR-B7** wires CCR retrieval-marker injection.
@@ -133,10 +133,8 @@ const STRATEGY_SEARCH_COMPRESSOR: &str = "search_compressor";
const STRATEGY_DIFF_COMPRESSOR: &str = "diff_compressor";
/// Strategy tag emitted when CodeAwareCompressor rewrote a source-code block.
const STRATEGY_CODE_COMPRESSOR: &str = "code_compressor";
/// Strategy tag emitted when Kompress rewrote a plain-text block (also
/// used by the SourceCode arm's passthrough fallback — see
/// [`dispatch_compressor`]). Only the `ml` build has a Kompress arm to
/// emit it.
/// Strategy tag emitted when Kompress rewrote a plain-text block. Only
/// the `ml` build has a Kompress arm to emit it.
#[cfg(feature = "ml")]
const STRATEGY_KOMPRESS: &str = "kompress";
@@ -704,14 +702,9 @@ fn kompress_cached() -> Option<&'static Kompress> {
}
/// Route `text` through Kompress when the model is cache-resident;
/// otherwise fall back to a deterministic no-op. Shared by the
/// `PlainText` arm and the `SourceCode` arm's passthrough fallback.
///
/// Not a full port of the Python router's fallback chain: that one also
/// re-runs Kompress when the code compressor returns *changed but not
/// smaller* output, whereas this arm sends only an exact passthrough
/// here and leaves the changed-but-not-smaller case to the
/// tokenizer-rejection gate in `compress_one_block`.
/// otherwise fall back to a deterministic no-op. Only the `PlainText`
/// arm calls it: source code that the code compressor declines is
/// deliberately not routed here (Kompress drops keywords from code).
#[cfg(feature = "ml")]
fn kompress_or_noop(text: &str, content_type: ContentType) -> DispatchResult {
if let Some(k) = kompress_cached() {
@@ -805,10 +798,9 @@ fn warm_kompress_blocking() -> bool {
// arms to a deterministic no-op, bypassing their compressor entirely.
//
// One operator-facing caveat: granularity is per ARM, not per
// compressor. `plain_text` alone does not stop Kompress runs reached
// through the SourceCode arm's passthrough fallback; disabling
// `source_code` as well is what removes Kompress from traffic entirely
// (at the cost of also disabling the code compressor).
// compressor. Only the `PlainText` arm reaches Kompress, so disabling
// `plain_text` removes Kompress from traffic; `source_code` governs
// the code compressor alone.
/// Parse the comma-separated arm list. Blank entries are skipped;
/// unknown names are ignored with a warning rather than failing the
@@ -1637,8 +1629,8 @@ enum DispatchResult {
/// - `BuildOutput` → LogCompressor
/// - `SearchResults` → SearchCompressor
/// - `GitDiff` → DiffCompressor
/// - `SourceCode` → CodeAwareCompressor, falling back to cached
/// Kompress on exact passthrough (PR-B4)
/// - `SourceCode` → CodeAwareCompressor; an exact passthrough stays a
/// no-op, never Kompress (PR-B4, #4214)
/// - `PlainText` → cached Kompress, cache-cold → no-op (PR-B4)
/// - `Html` → no-op (no compressor)
///
@@ -1726,21 +1718,20 @@ fn dispatch_compressor(text: &str, content_type: ContentType) -> DispatchResult
// syntax errors, a min-token floor, and a compression-ratio
// guard), so this arm only has to compare its output against
// the input; the tokenizer-rejection gate in
// `compress_one_block` applies on top of that. When the code
// compressor doesn't shrink the block (e.g. it's already
// dense, or tree-sitter reverted on a parse error), fall back
// to Kompress rather than giving up — mirrors the Python
// router's no-shrink → Kompress fallback.
// `compress_one_block` applies on top of that.
//
// When the code compressor declines (exact passthrough), the block
// is left UNCHANGED. It is deliberately NOT handed to Kompress:
// that is a prose model that drops low-salience words, and for code
// those are keywords such as `for`, `in` and `if` (#4214). Mirrors
// the Python router, which no longer falls back from code to prose
// ML compression either.
ContentType::SourceCode => {
let result = code_compressor().compress(text);
if result.compressed == text {
// Exact passthrough only: the compressor declined
// (min-token floor, unknown language, parse failure,
// ratio guard), so hand the block to Kompress. A
// rewrite that changed bytes without shrinking tokens
// is NOT retried here — `compress_one_block`'s
// tokenizer gate rejects it and forwards the original.
return kompress_or_noop(text, content_type);
return DispatchResult::NoOp {
content_type: content_type.as_str(),
};
}
DispatchResult::Compressed {
strategy: STRATEGY_CODE_COMPRESSOR,
+20
View File
@@ -122,6 +122,26 @@ pub fn python_module_source(n: usize) -> String {
code
}
/// Go source the tree-sitter code compressor declines (exact
/// passthrough: the language has no elision rules), yet which the
/// detector still classifies as `SourceCode` and which clears the
/// 2048-byte threshold. This is the shape that used to fall through to
/// Kompress (#4214).
pub fn go_handlers_source(n: usize) -> String {
use std::fmt::Write as _;
let mut code = String::new();
for i in 0..n {
let _ = write!(
code,
"func Handle{i}(w http.ResponseWriter, r *http.Request) {{\n\t\
id := r.URL.Query().Get(\"id\")\n\t\
fmt.Fprintf(w, \"%s-{i}\", id)\n}}\n\n"
);
}
code
}
/// Repetitive plain prose of at least `min_bytes`, varied enough to
/// classify as `PlainText`.
pub fn plain_prose(min_bytes: usize) -> String {
@@ -14,10 +14,47 @@
mod common;
use common::{body_with_tool_result, dispatch, kompress_available, python_module_source, sha256};
use common::{
body_with_tool_result, dispatch, go_handlers_source, kompress_available, python_module_source,
sha256, tool_result_action,
};
use headroom_core::transforms::{BlockAction, LiveZoneOutcome};
use serde_json::{json, Value};
/// The script from #4214 (a fenced Python snippet whose keywords Kompress
/// deleted). The reporter's own fence is not published; this reconstruction
/// keeps the lines quoted in the issue and parses as Python.
const ISSUE_4214_SCRIPT: &str = r#"import json, collections
d = json.load(open('turns.json'))
segs=[s for s in open('segs.txt').read().split() if s]
def load(path):
try:
return json.load(open(path))
except Exception as e: return None
for i,s in enumerate(segs):
t0=(d or {}).get('turns') or []
if not t0 or len(t0) < 2:
continue
else:
label = s if s else 'unknown'
counts = collections.Counter(x.get('role') for x in t0)
for role, n in counts.items():
if n > 1 or role == 'tool':
print(i, label, role, n)
else:
print(i, label, role)
total = sum(counts.values())
if total > 10 or label == 'big':
print('large session', label, total)
out = [k for k in counts if counts[k] > 0]
print(json.dumps(out))
summary = {s: i for i, s in enumerate(segs)}
for k, v in summary.items():
if v % 2 == 0 or k.startswith('a'):
print(k, v)
print('done', len(segs))"#;
// ─── Routing tests ─────────────────────────────────────────────────────
#[test]
@@ -249,6 +286,70 @@ fn source_code_tool_result_routes_to_code_compressor() {
}
}
#[test]
fn source_code_declined_by_code_compressor_is_never_kompressed() {
// The code compressor leaves Go source untouched (exact passthrough).
// That must stay a NoOp: handing it to Kompress, a prose model, drops
// keywords such as `func`, `if` and `for` from the code (#4214). The
// warmup makes Kompress live when the model is cache-resident, so on
// such a host the old fallback would have rewritten this block.
let _ = kompress_available();
let code = go_handlers_source(30);
assert!(
code.len() > 2048,
"fixture must clear the SourceCode byte threshold (2048); got {} bytes",
code.len()
);
let (body, _) = body_with_tool_result(&code);
match dispatch(&body) {
LiveZoneOutcome::NoChange { manifest } => {
let action = manifest
.block_outcomes
.iter()
.find(|b| b.block_type == "tool_result")
.expect("tool_result block present")
.action
.clone();
assert!(
matches!(&action, BlockAction::NoCompressionApplied { content_type } if content_type == "source_code"),
"expected a source_code NoOp, got {action:?}"
);
}
LiveZoneOutcome::Modified { manifest, .. } => panic!(
"a declined source-code block must be forwarded unchanged, not rewritten. manifest: {manifest:?}"
),
}
}
#[test]
fn issue_4214_script_is_never_kompressed() {
// The Python script quoted in #4214, repeated until the block clears the
// SourceCode threshold. The code compressor may shrink it (it keeps the
// keywords) or leave it alone, but the prose model must never rewrite it.
// A guard rather than a discriminator: the code compressor already handles
// Python, so this passes before the fix too; the Go test above is the one
// that fails without it.
let _ = kompress_available();
let code = format!("{ISSUE_4214_SCRIPT}\n").repeat(3);
assert!(
code.len() > 2048,
"fixture must clear the SourceCode byte threshold"
);
let (body, _) = body_with_tool_result(&code);
match dispatch(&body) {
LiveZoneOutcome::NoChange { .. } => {}
LiveZoneOutcome::Modified { manifest, .. } => match tool_result_action(&manifest) {
BlockAction::Compressed { strategy, .. } => assert_eq!(
strategy, "code_compressor",
"only the code compressor may rewrite the #4214 script"
),
other => panic!("unexpected action {other:?}"),
},
}
}
#[test]
fn tiny_source_code_below_threshold_no_op() {
// Detector classifies this as SourceCode, but it's well under the
@@ -0,0 +1,53 @@
//! A declined source-code block never reaches the Kompress slot — dedicated
//! integration test file.
//!
//! When the code compressor declines a block (exact passthrough), the
//! dispatcher used to hand it to Kompress, a prose model that drops
//! keywords such as `for`, `in` and `if` (#4214). The Kompress slot is
//! process-global and latches on first use, so the cleanest observable of
//! "Kompress was consulted" is that the model construction never starts.
//! That holds on any host, warm cache or not, which is why this does not
//! need the model. It needs a virgin process, so this is a single-test
//! binary like `live_zone_kompress_async_init.rs`. Do NOT add more tests
//! to this file.
//!
//! Vacuity note: without the `ml` feature there is no Kompress slot and
//! `kompress_init_runs()` is always 0, so the init assertion is trivially
//! true there; the byte-identical assertion still holds.
mod common;
use std::time::{Duration, Instant};
use common::{body_with_tool_result, dispatch, go_handlers_source};
use headroom_core::transforms::live_zone;
use headroom_core::transforms::LiveZoneOutcome;
#[test]
fn declined_source_code_never_starts_kompress() {
let code = go_handlers_source(30);
assert!(
code.len() > 2048,
"fixture must clear the SourceCode byte threshold (2048); got {} bytes",
code.len()
);
let (body, _) = body_with_tool_result(&code);
assert!(
matches!(dispatch(&body), LiveZoneOutcome::NoChange { .. }),
"a block the code compressor declines must be forwarded unchanged"
);
// The slot's initializer runs on a background thread started by the
// dispatch that first consults it. Give a would-be start ample time to
// show up before asserting it never did.
let deadline = Instant::now() + Duration::from_secs(2);
while Instant::now() < deadline && live_zone::kompress_init_runs() == 0 {
std::thread::sleep(Duration::from_millis(20));
}
assert_eq!(
live_zone::kompress_init_runs(),
0,
"Kompress must not be consulted for code the code compressor declined"
);
}