fix(logs): count omitted levels from dropped lines (#4141)

## Description

Closes #4070.

The omission marker previously counted severities in the whole input,
including retained errors. It now counts ERROR/FAIL/WARN/INFO only on
dropped line identities. Selection and whole-input result.stats are
unchanged. If no dropped line has a recognized severity, the marker is
simply `[N lines omitted]`, not a severity breakdown. CCR retains the
exact original.

## Type of Change

- [x] Bug fix (non-breaking)
- [x] Documentation update

## Changes Made

- Keep the existing retained-line identity set and a single-pass fixed
stack histogram; no omitted-line collection or repeated full scans.
- Restore complete structured-marker assertions in Rust and Python,
including bounded failure names, repeated ids, whole-input totals, and
non-severity omissions.
- Restore parity fixture `3bc015edc0a36387.json` using its original
input, explicit configuration and input digest. Output and compression
ratio were re-recorded through the freshly built native extension rather
than manually copied. New runtime-frame defaults remain implicit in the
legacy configuration.
- Remove the constructor-only result-copy test; retain consumer behavior
regressions.
- Document omitted-versus-input counts and the exact bare-marker form in
the existing text/log guide. No manual changelog change.

## Testing

- [x] Scoped Rust and Python tests passed
- [x] Normal touched-file Ruff and repository mypy hooks passed
- [x] Actual native recorder and parity CLI exercised

Head `6c472f6b` includes upstream main `f1bc2609`.

- Focused Rust log-compressor tests: **41 passed**.
- Python public-transform tests: **11 passed**, using the freshly built
native extension.
- Actual parity CLI: **20 matched, 0 skipped, 0 differences**, including
the restored original-input fixture:

```text
cargo run --locked -p headroom-parity --bin parity-run -- run --fixtures tests/parity/fixtures --only log_compressor
[log_compressor  ] total=20 matched=20 skipped=0 diffed=0
```

## Real Behavior Proof

- Environment: Linux x86_64, Python 3.13.14, native extension freshly
built from this worktree.
- Exact command / steps: Re-record the original log fixture through
LogCompressor.compress with its original explicit configuration, then
run the actual parity CLI shown above.
- Observed result: The real native recorder produced `[297 lines
omitted: 297 INFO]` while whole-input statistics remained 300 INFO / one
ERROR / 305 total / eight selected. The exact original was unchanged and
the measured compression ratio is `0.024777201057682892`. The original
fixture digest remains
`3bc015edc0a36387194d075139845c120a308afb314bae436ab454906b431d84`.

Normal commit and full pre-push hooks passed without bypass: Rust
formatting, warnings-as-errors Clippy, workspace tests, native editable
rebuild/import verification, **181 critical Python tests**, and
branch-range commitlint. The critical Python run reported five
dependency/provider warnings, no failures. These local results do not
claim that hosted checks have finished.

- Not tested: New Windows/macOS runs, live-provider sessions and the
full Python suite.

Earlier before/after actual-native CCR qualification showed the retained
ERROR was not counted as omitted, whole-input totals stayed unchanged,
and retrieval recovered the exact original. That historical
implementation evidence is in the review discussion; the explicit
results above qualify this follow-up head. No new Windows/macOS or
live-provider qualification is claimed.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing LogCompressor and CCR; no new
feature or gate.
- Minimum rollout channel: Unchanged (stable); no channel promotion.
- Stable/default behavior changed: Yes, omitted-severity accounting is
corrected; selection and stored originals remain unchanged.
- Kill switch / disable path:
ContentRouterConfig(enable_log_compressor=False) disables the log route;
existing CCR settings disable retrieval separately.
- Unsafe override required: No.
- Qualification impact: The scoped native/Python/parity proof and
required normal push gate above cover this marker path. No new
platform/live-provider qualification is claimed.
- Rollback path: Revert the fix; no schema or stored-original migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review
- [x] Existing style and documentation conventions followed
- [x] Consumer-visible regressions and actual native/CLI paths exercised
- [x] Normal hooks passed; no bypass
- [x] No manual CHANGELOG edit

AI-assisted: omp / GPT-6.1 Sol. This review follow-up was completed
without subagents. No manual human-validation or full Python-suite
claim; scoped tests and the required push gate are reported separately.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Mansurshakh Japarov
2026-10-24 08:45:24 -07:00
committed by GitHub
co-authored by JerrettDavis
parent 89545fbdf3
commit 023dd1bc54
4 changed files with 161 additions and 73 deletions
@@ -1532,35 +1532,50 @@ impl LogCompressor {
let omitted = all_lines.len().saturating_sub(selected.len());
if omitted > 0 {
// Keep public stats as input totals; describe only dropped identities here.
let mut omitted_levels = [0u64; 4];
for line in all_lines {
if selected_numbers.contains(&line.line_number) {
continue;
}
match line.level {
LogLevel::Error => omitted_levels[0] += 1,
LogLevel::Fail => omitted_levels[1] += 1,
LogLevel::Warn => omitted_levels[2] += 1,
LogLevel::Info => omitted_levels[3] += 1,
_ => {}
}
}
let mut summary_parts: Vec<String> = Vec::new();
for (label, key) in [
("ERROR", "errors"),
("FAIL", "fails"),
("WARN", "warnings"),
("INFO", "info"),
for (label, n) in [
("ERROR", omitted_levels[0]),
("FAIL", omitted_levels[1]),
("WARN", omitted_levels[2]),
("INFO", omitted_levels[3]),
] {
let n = stats.get(key).copied().unwrap_or(0);
if n > 0 {
summary_parts.push(format!("{} {}", n, label));
}
}
if !summary_parts.is_empty() {
let omitted_names = if omitted_short_summary_ids.is_empty() {
String::new()
let omitted_names = if omitted_short_summary_ids.is_empty() {
String::new()
} else {
let shown = omitted_short_summary_ids
.iter()
.take(5)
.copied()
.collect::<Vec<_>>()
.join(", ");
let overflow = omitted_short_summary_ids.len().saturating_sub(5);
if overflow > 0 {
format!("; omitted: {shown}, +{overflow} more")
} else {
let shown = omitted_short_summary_ids
.iter()
.take(5)
.copied()
.collect::<Vec<_>>()
.join(", ");
let overflow = omitted_short_summary_ids.len().saturating_sub(5);
if overflow > 0 {
format!("; omitted: {shown}, +{overflow} more")
} else {
format!("; omitted: {shown}")
}
};
format!("; omitted: {shown}")
}
};
if summary_parts.is_empty() {
output.push(format!("[{} lines omitted{}]", omitted, omitted_names));
} else {
output.push(format!(
"[{} lines omitted: {}{}]",
omitted,
@@ -1963,12 +1978,6 @@ mod tests {
let (output, _) = c.format_output(&selected, &parsed);
assert!(!output.contains("; omitted: "));
if !keep_summary_lines {
assert_eq!(
output,
"FAILED outside.py::test_0\nFAILED outside.py::test_9\nERROR outside.py::test_10\nERROR outside.py::test_19\n[67 lines omitted: 10 ERROR, 10 FAIL, 51 INFO]"
);
}
}
}
@@ -2366,7 +2375,79 @@ mod tests {
}
#[test]
fn format_output_emits_summary_with_omitted_count() {
fn omitted_marker_counts_only_dropped_lines_on_live_path() {
let mut lines = (0..400)
.map(|i| format!("INFO worker processed item {i}"))
.collect::<Vec<_>>();
let error = "ERROR worker failed: connection reset";
lines.insert(200, error.to_string());
let c = LogCompressor::new(LogCompressorConfig {
enable_ccr: false,
..Default::default()
});
let (result, _) = c.compress(&lines.join("\n"), 1.0);
let marker = result
.compressed
.lines()
.find(|line| line.contains("lines omitted:"))
.expect("large input must have omitted lines");
let pattern = Regex::new(r"(\d+) (ERROR|FAIL|WARN|INFO)").unwrap();
let mut level_counts = pattern.captures_iter(marker);
let count = level_counts.next().expect("omitted INFO count");
assert!(result.compressed.lines().any(|line| line == error));
assert_eq!(&count[2], "INFO");
assert_eq!(
count[1].parse::<usize>().unwrap(),
result.original_line_count - result.compressed_line_count
);
assert!(level_counts.next().is_none());
assert_eq!(
marker,
format!("[{} lines omitted: {} INFO]", &count[1], &count[1])
);
assert_eq!(result.stats["errors"], 1);
assert_eq!(result.stats["info"], 400);
}
#[test]
fn omitted_non_severity_lines_keep_count_notice_on_live_path() {
use std::fmt::Write as _;
let count_pattern = Regex::new(r"(?m)^\[(\d+) lines omitted[^\]]*\]$").unwrap();
let level_pattern = Regex::new(r"\d+ (ERROR|FAIL|WARN|INFO)\b").unwrap();
let error = "ERROR retained failure";
for prefix in ["DEBUG ", "TRACE ", ""] {
let mut input = String::with_capacity(4096);
for i in 0..100 {
writeln!(input, "{prefix}progress step {i}").unwrap();
}
input.push_str(error);
let c = LogCompressor::new(LogCompressorConfig {
enable_ccr: false,
..Default::default()
});
let (result, _) = c.compress(&input, 1.0);
let marker = count_pattern
.captures(&result.compressed)
.expect("dropped lines must retain an omission count");
assert_eq!(
marker[1].parse::<usize>().unwrap(),
result.original_line_count - result.compressed_line_count
);
assert!(!level_pattern.is_match(marker.get(0).unwrap().as_str()));
assert_eq!(
marker.get(0).unwrap().as_str(),
format!("[{} lines omitted]", &marker[1])
);
assert!(result.compressed.lines().any(|line| line == error));
assert_eq!(result.stats["errors"], 1);
assert_eq!(result.stats["total"], 101);
}
}
#[test]
fn format_output_preserves_input_totals() {
let c = cmp();
let all_lines = vec![
LogLine::new(0, "ERROR a"),
@@ -2388,9 +2469,9 @@ mod tests {
.collect::<Vec<_>>();
let selected = vec![all_lines[0].clone()];
let (output, stats) = c.format_output(&selected, &all_lines);
assert!(output.contains("[3 lines omitted: 1 ERROR, 1 WARN, 2 INFO]"));
assert_eq!(stats["errors"], 1);
assert_eq!(stats["info"], 2);
assert_eq!(output, "ERROR a\n[3 lines omitted: 1 WARN, 2 INFO]");
}
#[test]
@@ -2415,11 +2496,9 @@ mod tests {
} else {
String::new()
};
assert!(
output.ends_with(&format!(
"[{count} lines omitted: {count} FAIL, 1 INFO; omitted: {shown}{overflow}]"
)),
"{output}"
assert_eq!(
output.lines().last().unwrap(),
format!("[{count} lines omitted: {count} FAIL; omitted: {shown}{overflow}]")
);
}
}
@@ -2442,11 +2521,9 @@ mod tests {
];
let (output, stats) = cmp().format_output(&selected, &all_lines);
assert!(
output.ends_with(
"[3 lines omitted: 1 ERROR, 3 FAIL, 1 INFO; omitted: tests/test_ids.py::test_error, tests/test_ids.py::test_repeat, tests/test_ids.py::test_repeat]"
),
"{output}"
assert_eq!(
output.lines().last().unwrap(),
"[3 lines omitted: 1 ERROR, 2 FAIL; omitted: tests/test_ids.py::test_error, tests/test_ids.py::test_repeat, tests/test_ids.py::test_repeat]"
);
assert_eq!(stats["errors"], 1);
assert_eq!(stats["fails"], 3);
@@ -2482,26 +2559,12 @@ mod tests {
);
let (output, _) = c.format_output(&selected, &all_lines);
assert!(
output.ends_with(
"[4 lines omitted: 4 FAIL, 1 INFO; omitted: tests/test_ids.py::test_1, tests/test_ids.py::test_2]"
),
"{output}"
assert_eq!(
output.lines().last().unwrap(),
"[4 lines omitted: 2 FAIL, 1 INFO; omitted: tests/test_ids.py::test_1, tests/test_ids.py::test_2]"
);
}
#[test]
fn format_output_marker_is_unchanged_without_omitted_short_summary_entries() {
let c = cmp();
let mut error = LogLine::new(0, "ERROR a");
error.level = LogLevel::Error;
let mut info = LogLine::new(1, "INFO b");
info.level = LogLevel::Info;
let (output, _) = c.format_output(&[error.clone()], &[error, info]);
assert_eq!(output, "ERROR a\n[1 lines omitted: 1 ERROR, 1 INFO]");
}
#[test]
fn score_line_caps_at_one_point_zero() {
let line = LogLine {
+5
View File
@@ -93,6 +93,11 @@ print(f"Compression ratio: {result.compression_ratio:.1%}")
- Verbose success output
- Repeated patterns
Omission markers count `ERROR`, `FAIL`, `WARN`, and `INFO` levels only among the dropped lines, not the full input. An error retained in the compressed output is therefore not advertised as omitted. Debug, trace, and unclassified lines are not listed in the level breakdown.
When only debug, trace, or unclassified lines are dropped, the marker is simply `[N lines omitted]`, without a severity breakdown.
`result.stats` continues to report whole-input totals; the marker's narrower counts do not change those statistics. CCR still retains the exact original log when enabled.
## DiffCompressor
Compresses unified diffs while keeping the actual changes and enough context to understand them.
File diff suppressed because one or more lines are too long
+33 -13
View File
@@ -1,9 +1,10 @@
from __future__ import annotations
import re
import pytest
from headroom.transforms.log_compressor import (
LogCompressionResult,
LogCompressor,
LogCompressorConfig,
LogFormat,
@@ -185,18 +186,37 @@ def test_real_compress_ignores_malformed_short_summary_diagnostic() -> None:
"FAILED outside.py::test_9\n"
"ERROR outside.py::test_10\n"
"ERROR outside.py::test_19\n"
"[67 lines omitted: 10 ERROR, 10 FAIL, 51 INFO]"
"[67 lines omitted: 8 ERROR, 8 FAIL, 51 INFO]"
)
def test_result_properties() -> None:
result = LogCompressionResult(
compressed="small",
original="this is a substantially longer log body",
original_line_count=20,
compressed_line_count=5,
format_detected=LogFormat.GENERIC,
compression_ratio=0.25,
)
assert result.tokens_saved_estimate > 0
assert result.lines_omitted == 15
def test_omitted_marker_counts_only_dropped_lines_on_native_path() -> None:
lines = [f"INFO worker processed item {i}" for i in range(400)]
error = "ERROR worker failed: connection reset"
lines.insert(200, error)
result = LogCompressor(LogCompressorConfig(enable_ccr=False)).compress("\n".join(lines))
marker = next(line for line in result.compressed.splitlines() if "lines omitted:" in line)
counts = {
level: int(count) for count, level in re.findall(r"(\d+) (ERROR|FAIL|WARN|INFO)", marker)
}
assert error in result.compressed.splitlines()
assert counts == {"INFO": result.lines_omitted}
assert marker == f"[{result.lines_omitted} lines omitted: {result.lines_omitted} INFO]"
assert result.stats["errors"] == 1
assert result.stats["info"] == 400
@pytest.mark.parametrize("prefix", ["DEBUG ", "TRACE ", ""])
def test_non_severity_omissions_keep_count_notice_on_native_path(prefix: str) -> None:
error = "ERROR retained failure"
original = "\n".join([f"{prefix}progress step {i}" for i in range(100)] + [error])
result = LogCompressor(LogCompressorConfig(enable_ccr=False)).compress(original)
marker = re.search(r"(?m)^\[(\d+) lines omitted[^\]]*\]$", result.compressed)
assert marker is not None
assert int(marker[1]) == result.lines_omitted
assert not re.search(r"\d+ (ERROR|FAIL|WARN|INFO)\b", marker[0])
assert marker[0] == f"[{result.lines_omitted} lines omitted]"
assert error in result.compressed.splitlines()
assert result.stats["errors"] == 1
assert result.stats["total"] == 101