mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat(compress): add per-message compression diagnostics (#3058)
## Description Adds `CompressConfig.diagnostics` (bool, default `False`) and a companion `HEADROOM_DIAGNOSTICS=1` environment variable that enable per-message decision logging on every call to `compress()`. When active, `CompressResult.diagnostics` is a list of `MessageDecision` objects — one per input message — each describing: | Field | Description | |---|---| | `message_index` | Zero-based position in the original messages list | | `role` | `"user"` / `"assistant"` / `"tool"` / `"system"` / … | | `tokens_before` | Token count before the ContentRouter acted on this message | | `tokens_after` | Token count after (equals `tokens_before` for protected/passthrough) | | `action` | What was done: `"compressed:kompress:0.72"`, `"protected:user_message"`, `"passthrough:frozen"`, `"passthrough:small"`, etc. | Closes #2855. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`headroom/config.py`** — add `MessageDecision` dataclass; add `message_decisions: list[MessageDecision]` field to `TransformResult` - **`headroom/compress.py`** — add `CompressConfig.diagnostics`; add `CompressResult.diagnostics`; derive `collect_diagnostics` from config + env var; pass to pipeline; collect from `TransformResult` - **`headroom/transforms/pipeline.py`** — accumulate `message_decisions` from each transform result - **`headroom/transforms/content_router.py`** — instrument every decision branch in `apply()` with `_diag[i] = "action:..."` strings when `collect_diagnostics=True`; build `MessageDecision` list after Pass 3 (zero overhead when flag is off) - **`headroom/__init__.py`** — export `MessageDecision` in `__all__` and lazy-import map - **`tests/test_compress_diagnostics.py`** — 14 tests covering the full `compress() <-> pipeline` contract via a mock pipeline ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_compress_diagnostics.py -v ========================= test session starts ========================== collected 14 items tests/test_compress_diagnostics.py::test_message_decision_dataclass PASSED tests/test_compress_diagnostics.py::test_message_decision_importable_from_headroom PASSED tests/test_compress_diagnostics.py::test_transform_result_has_message_decisions_field PASSED tests/test_compress_diagnostics.py::test_transform_result_message_decisions_populated PASSED tests/test_compress_diagnostics.py::test_compress_config_diagnostics_defaults_false PASSED tests/test_compress_diagnostics.py::test_compress_config_diagnostics_can_be_set PASSED tests/test_compress_diagnostics.py::test_compress_result_diagnostics_none_by_default PASSED tests/test_compress_diagnostics.py::test_compress_passes_collect_diagnostics_when_config_true PASSED tests/test_compress_diagnostics.py::test_compress_result_diagnostics_list_when_config_true PASSED tests/test_compress_diagnostics.py::test_compress_result_diagnostics_contains_decisions PASSED tests/test_compress_diagnostics.py::test_env_var_enables_diagnostics PASSED tests/test_compress_diagnostics.py::test_env_var_absent_keeps_diagnostics_none PASSED tests/test_compress_diagnostics.py::test_env_var_overrides_config_false PASSED tests/test_compress_diagnostics.py::test_no_diagnostics_when_optimize_false PASSED ========================== 14 passed in 0.42s ========================== $ ruff check . All checks passed! ``` ## Real Behavior Proof - Environment: Python 3.12, headroom dev install - Exact command / steps: `python -m pytest tests/test_compress_diagnostics.py -v` - Observed result: 14 tests pass; diagnostics list is populated when `CompressConfig(diagnostics=True)` or `HEADROOM_DIAGNOSTICS=1` - Not tested: ContentRouter integration tests (require compiled `headroom._core` extension — run in CI) ## Runtime Rollout Safety - Rollout-managed feature(s): None — diagnostics are opt-in via `CompressConfig.diagnostics=True` or `HEADROOM_DIAGNOSTICS=1` - Minimum rollout channel: N/A (additive, opt-in only) - Stable/default behavior changed: No — `CompressConfig.diagnostics` defaults to `False`; `CompressResult.diagnostics` is `None` unless explicitly requested - Kill switch / disable path: Simply omit `diagnostics=True` / unset `HEADROOM_DIAGNOSTICS` - Unsafe override required: No - Qualification impact: None - Rollback path: Revert this PR; no schema or data migration involved ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — no UI changes. ## Additional Notes The `collect_diagnostics` flag incurs zero overhead in the hot path when not set: the `_diag` dict in `ContentRouter.apply()` is never allocated and no `MessageDecision` objects are constructed. Signed-off-by: Radha Krishnan P <gingeekrishna@gmail.com> --------- Signed-off-by: Radhakrishnan Pachyappan <radhakrishnan.p@op.tech> Signed-off-by: Radhakrishnan P <gingeekrishna@gmail.com>
This commit is contained in:
@@ -119,6 +119,7 @@ __all__ = [
|
||||
"DiffArtifact",
|
||||
"RequestMetrics",
|
||||
"SimulationResult",
|
||||
"MessageDecision",
|
||||
"TransformDiff",
|
||||
"TransformResult",
|
||||
"WasteSignals",
|
||||
@@ -223,6 +224,7 @@ _LAZY_EXPORTS: dict[str, tuple[str, str]] = {
|
||||
"DiffArtifact": ("headroom.config", "DiffArtifact"),
|
||||
"RequestMetrics": ("headroom.config", "RequestMetrics"),
|
||||
"SimulationResult": ("headroom.config", "SimulationResult"),
|
||||
"MessageDecision": ("headroom.config", "MessageDecision"),
|
||||
"TransformDiff": ("headroom.config", "TransformDiff"),
|
||||
"TransformResult": ("headroom.config", "TransformResult"),
|
||||
"WasteSignals": ("headroom.config", "WasteSignals"),
|
||||
|
||||
+18
-1
@@ -57,15 +57,19 @@ Examples:
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
from dataclasses import dataclass, field, replace
|
||||
from typing import Any
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from .agent_savings import apply_agent_savings_profile
|
||||
from .observability import get_otel_metrics
|
||||
from .pipeline import PipelineExtensionManager, PipelineStage, summarize_routing_markers
|
||||
from .utils import extract_user_query as _extract_user_query
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .config import MessageDecision
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@@ -146,6 +150,14 @@ class CompressConfig:
|
||||
savings_profile: str | None = None
|
||||
"""Named high-savings profile, e.g. 'agent-90' for Codex/Claude/Cursor."""
|
||||
|
||||
diagnostics: bool = False
|
||||
"""Collect per-message compression decisions. Also enabled by the
|
||||
``HEADROOM_DIAGNOSTICS=1`` environment variable. When True, the returned
|
||||
:class:`CompressResult` carries a ``diagnostics`` list of
|
||||
:class:`~headroom.config.MessageDecision` objects — one per message —
|
||||
describing which action was taken (compressed, protected, skipped …) and
|
||||
how many tokens were spent before/after."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class CompressResult:
|
||||
@@ -166,6 +178,7 @@ class CompressResult:
|
||||
tokens_saved: int = 0
|
||||
compression_ratio: float = 0.0
|
||||
transforms_applied: list[str] = field(default_factory=list)
|
||||
diagnostics: list[MessageDecision] | None = None
|
||||
|
||||
|
||||
def compress(
|
||||
@@ -224,6 +237,8 @@ def compress(
|
||||
if cfg.savings_profile:
|
||||
apply_agent_savings_profile(cfg, cfg.savings_profile)
|
||||
|
||||
collect_diagnostics = cfg.diagnostics or os.environ.get("HEADROOM_DIAGNOSTICS", "") == "1"
|
||||
|
||||
pipeline = _get_pipeline()
|
||||
pipeline_extensions = PipelineExtensionManager(hooks=hooks, discover=False)
|
||||
|
||||
@@ -272,6 +287,7 @@ def compress(
|
||||
min_tokens_to_compress=cfg.min_tokens_to_compress,
|
||||
kompress_model=cfg.kompress_model,
|
||||
frozen_message_count=cfg.frozen_message_count,
|
||||
collect_diagnostics=collect_diagnostics,
|
||||
)
|
||||
|
||||
tokens_before = result.tokens_before
|
||||
@@ -350,6 +366,7 @@ def compress(
|
||||
tokens_saved=tokens_saved,
|
||||
compression_ratio=ratio,
|
||||
transforms_applied=result.transforms_applied,
|
||||
diagnostics=result.message_decisions if collect_diagnostics else None,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
|
||||
@@ -923,6 +923,22 @@ class CachePrefixMetrics:
|
||||
previous_hash: str | None = None # Previous hash for comparison (None = first request)
|
||||
|
||||
|
||||
@dataclass
|
||||
class MessageDecision:
|
||||
"""Per-message compression decision recorded in diagnostics mode.
|
||||
|
||||
Collected by ContentRouter when ``collect_diagnostics=True`` is passed
|
||||
as a kwarg (set automatically when ``CompressConfig.diagnostics`` is True
|
||||
or ``HEADROOM_DIAGNOSTICS=1`` is set in the environment).
|
||||
"""
|
||||
|
||||
message_index: int
|
||||
role: str
|
||||
tokens_before: int
|
||||
tokens_after: int
|
||||
action: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class TransformResult:
|
||||
"""Output of a transform operation."""
|
||||
@@ -937,6 +953,7 @@ class TransformResult:
|
||||
cache_metrics: CachePrefixMetrics | None = None # Populated by CacheAligner
|
||||
timing: dict[str, float] = field(default_factory=dict) # transform_name → ms
|
||||
waste_signals: WasteSignals | None = None # Detected waste in original messages
|
||||
message_decisions: list[MessageDecision] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def transforms_summary(self) -> dict[str, int]:
|
||||
|
||||
@@ -56,6 +56,7 @@ from ..config import (
|
||||
DEFAULT_BYTE_EXACT_EXCLUDE_TOOLS,
|
||||
DEFAULT_EXCLUDE_TOOLS,
|
||||
DEFAULT_VERBATIM_EXCLUDE_TOOLS,
|
||||
MessageDecision,
|
||||
ReadLifecycleConfig,
|
||||
RelevanceScorerConfig,
|
||||
TransformResult,
|
||||
@@ -5712,6 +5713,10 @@ class ContentRouter(Transform):
|
||||
}
|
||||
compressed_details: list[str] = [] # e.g. ["code_aware:0.72", "kompress:0.65"]
|
||||
|
||||
# Per-message diagnostics (collected only when collect_diagnostics=True)
|
||||
collect_diagnostics = bool(kwargs.get("collect_diagnostics", False))
|
||||
_diag: dict[int, str] = {} # slot_index → action string
|
||||
|
||||
# Check for analysis intent in the most recent user message
|
||||
analysis_intent = False
|
||||
if self.config.protect_analysis_context:
|
||||
@@ -5816,6 +5821,8 @@ class ContentRouter(Transform):
|
||||
else:
|
||||
# Frozen — byte-identical to preserve the prefix cache.
|
||||
result_slots[i] = message
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "passthrough:frozen"
|
||||
continue
|
||||
|
||||
role = message.get("role", "")
|
||||
@@ -5874,12 +5881,24 @@ class ContentRouter(Transform):
|
||||
)
|
||||
result_slots[i] = transformed_message
|
||||
route_counts["content_blocks"] += 1
|
||||
if collect_diagnostics:
|
||||
# _process_content_blocks returns the SAME message object
|
||||
# when it protected/passed through every block, and a NEW
|
||||
# dict only when at least one block was actually
|
||||
# compressed -- use that instead of assuming compression.
|
||||
_diag[i] = (
|
||||
"compressed:content_blocks"
|
||||
if transformed_message is not message
|
||||
else "protected:content_blocks_unchanged"
|
||||
)
|
||||
continue
|
||||
|
||||
# Skip non-string content (other types)
|
||||
if not isinstance(content, str):
|
||||
result_slots[i] = message
|
||||
route_counts["non_string"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "passthrough:non_string"
|
||||
continue
|
||||
|
||||
# A headroom_retrieve result IS already-retrieved, original CCR content --
|
||||
@@ -5899,6 +5918,8 @@ class ContentRouter(Transform):
|
||||
result_slots[i] = message
|
||||
transforms_applied.append("router:excluded:ccr_retrieve")
|
||||
route_counts["ccr_retrieve"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "protected:ccr_retrieve"
|
||||
continue
|
||||
|
||||
# Skip OpenAI-style tool messages for excluded tools
|
||||
@@ -5911,6 +5932,8 @@ class ContentRouter(Transform):
|
||||
result_slots[i] = message
|
||||
transforms_applied.append("router:excluded:tool")
|
||||
route_counts["excluded_tool"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "protected:excluded_tool_verbatim"
|
||||
continue
|
||||
if messages_from_end <= read_protection_window:
|
||||
# Protected from lossy compression — but grep/log/json
|
||||
@@ -5930,11 +5953,15 @@ class ContentRouter(Transform):
|
||||
route_counts["excluded_tool_lossless"] = (
|
||||
route_counts.get("excluded_tool_lossless", 0) + 1
|
||||
)
|
||||
if collect_diagnostics:
|
||||
_diag[i] = f"compressed:lossless_{kind}"
|
||||
continue
|
||||
# Recent — protect as before
|
||||
result_slots[i] = message
|
||||
transforms_applied.append("router:excluded:tool")
|
||||
route_counts["excluded_tool"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "protected:excluded_tool_recent"
|
||||
continue
|
||||
# Old excluded-tool output — fall through to compression
|
||||
# (the LLM is unlikely to need exact content from this far back,
|
||||
@@ -5953,6 +5980,8 @@ class ContentRouter(Transform):
|
||||
route_counts["bash_lossless_search"] = (
|
||||
route_counts.get("bash_lossless_search", 0) + 1
|
||||
)
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "compressed:bash_lossless_search"
|
||||
continue
|
||||
|
||||
# Read protection (ROLE / SHAPE-AGNOSTIC). An observation produced by
|
||||
@@ -5980,11 +6009,15 @@ class ContentRouter(Transform):
|
||||
route_counts["read_kompress_exp"] = (
|
||||
route_counts.get("read_kompress_exp", 0) + 1
|
||||
)
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "compressed:read_kompress_exp"
|
||||
continue
|
||||
result_slots[i] = message
|
||||
transforms_applied.append("router:read_protected")
|
||||
route_counts.setdefault("read_protected", 0)
|
||||
route_counts["read_protected"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "protected:read_output"
|
||||
continue
|
||||
|
||||
# Protection 1: Never compress user messages (unless overridden),
|
||||
@@ -5993,6 +6026,8 @@ class ContentRouter(Transform):
|
||||
result_slots[i] = message
|
||||
transforms_applied.append("router:protected:user_message")
|
||||
route_counts["user_msg"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "protected:user_message"
|
||||
continue
|
||||
|
||||
# Protection 1b: Never compress system/developer messages unless
|
||||
@@ -6002,12 +6037,16 @@ class ContentRouter(Transform):
|
||||
transforms_applied.append(f"router:protected:{role}_message")
|
||||
route_counts.setdefault("system_msg", 0)
|
||||
route_counts["system_msg"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[i] = f"protected:{role}_message"
|
||||
continue
|
||||
|
||||
if not content or tokenizer.count_text(content) < min_tokens:
|
||||
# Skip small content
|
||||
result_slots[i] = message
|
||||
route_counts["small"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "passthrough:small"
|
||||
continue
|
||||
|
||||
# Protection: failed tool calls / error outputs stay verbatim
|
||||
@@ -6026,6 +6065,8 @@ class ContentRouter(Transform):
|
||||
transforms_applied.append("router:protected:error_output")
|
||||
route_counts.setdefault("error_protected", 0)
|
||||
route_counts["error_protected"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "protected:error_output"
|
||||
continue
|
||||
|
||||
# Detect content type for protection decisions. Even when the
|
||||
@@ -6044,6 +6085,8 @@ class ContentRouter(Transform):
|
||||
result_slots[i] = message
|
||||
transforms_applied.append("router:protected:recent_code")
|
||||
route_counts["recent_code"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "protected:recent_code"
|
||||
continue
|
||||
|
||||
# Protection 3: Don't compress CODE when analysis intent detected
|
||||
@@ -6051,6 +6094,8 @@ class ContentRouter(Transform):
|
||||
result_slots[i] = message
|
||||
transforms_applied.append("router:protected:analysis_context")
|
||||
route_counts["analysis_ctx"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "protected:analysis_context"
|
||||
continue
|
||||
|
||||
# Compression pinning: if this message was already compressed
|
||||
@@ -6061,6 +6106,8 @@ class ContentRouter(Transform):
|
||||
result_slots[i] = message
|
||||
route_counts.setdefault("already_compressed", 0)
|
||||
route_counts["already_compressed"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "passthrough:already_compressed"
|
||||
continue
|
||||
|
||||
# Route and compress based on content detection
|
||||
@@ -6088,6 +6135,8 @@ class ContentRouter(Transform):
|
||||
route_counts["ratio_too_high"] += 1
|
||||
route_counts.setdefault("cache_hit", 0)
|
||||
route_counts["cache_hit"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "passthrough:cache_skip"
|
||||
continue
|
||||
|
||||
# Tier 2: result cache — reuse compressed output
|
||||
@@ -6127,10 +6176,14 @@ class ContentRouter(Transform):
|
||||
# Net-cost gate: mutation would cost more in cache
|
||||
# invalidation than it saves — leave untouched.
|
||||
result_slots[i] = message
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "passthrough:netcost_blocked"
|
||||
else:
|
||||
result_slots[i] = {**message, "content": cached_compressed}
|
||||
transforms_applied.append(f"router:{cached_strategy}:{cached_ratio:.2f}")
|
||||
compressed_details.append(f"{cached_strategy}:{cached_ratio:.2f}")
|
||||
if collect_diagnostics:
|
||||
_diag[i] = f"compressed:{cached_strategy}:{cached_ratio:.2f}"
|
||||
# Freeze the "compress" verdict so future turns skip the
|
||||
# min_ratio re-check above and never downgrade it.
|
||||
if freeze_decision:
|
||||
@@ -6152,6 +6205,8 @@ class ContentRouter(Transform):
|
||||
self._cache.move_to_skip(content_key)
|
||||
result_slots[i] = message
|
||||
route_counts["ratio_too_high"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[i] = "passthrough:ratio_too_high"
|
||||
route_counts.setdefault("cache_hit", 0)
|
||||
route_counts["cache_hit"] += 1
|
||||
continue
|
||||
@@ -6328,6 +6383,8 @@ class ContentRouter(Transform):
|
||||
route_counts["lossy_unrecoverable_skipped"] = (
|
||||
route_counts.get("lossy_unrecoverable_skipped", 0) + 1
|
||||
)
|
||||
if collect_diagnostics:
|
||||
_diag[slot_idx] = "passthrough:lossy_unrecoverable"
|
||||
continue
|
||||
# Compressed — store in result cache. The cache is still
|
||||
# warmed when the net-cost gate blocks the slot: the
|
||||
@@ -6355,12 +6412,18 @@ class ContentRouter(Transform):
|
||||
write_multiplier=netcost_write_multiplier,
|
||||
):
|
||||
result_slots[slot_idx] = message
|
||||
if collect_diagnostics:
|
||||
_diag[slot_idx] = "passthrough:netcost_blocked"
|
||||
continue
|
||||
result_slots[slot_idx] = {**message, "content": result.compressed}
|
||||
transforms_applied.append(
|
||||
f"router:{result.strategy_used.value}:{accept_ratio:.2f}"
|
||||
)
|
||||
compressed_details.append(f"{result.strategy_used.value}:{accept_ratio:.2f}")
|
||||
if collect_diagnostics:
|
||||
_diag[slot_idx] = (
|
||||
f"compressed:{result.strategy_used.value}:{accept_ratio:.2f}"
|
||||
)
|
||||
if slot_idx in frozen_unlock_slots:
|
||||
transforms_applied.append("router:netcost_frozen_unlock")
|
||||
route_counts.setdefault("netcost_frozen_unlocked", 0)
|
||||
@@ -6370,6 +6433,8 @@ class ContentRouter(Transform):
|
||||
self._cache.mark_skip(content_key)
|
||||
result_slots[slot_idx] = message
|
||||
route_counts["ratio_too_high"] += 1
|
||||
if collect_diagnostics:
|
||||
_diag[slot_idx] = "passthrough:ratio_too_high"
|
||||
# Caveat (1): only freeze a "skip" verdict when the ML model
|
||||
# is actually ready. A passthrough caused purely by a still-
|
||||
# loading ModernBERT must stay re-evaluable on later turns,
|
||||
@@ -6482,6 +6547,34 @@ class ContentRouter(Transform):
|
||||
except Exception as e: # pragma: no cover - defensive
|
||||
logger.debug("Router observer raised (non-fatal): %s", e)
|
||||
|
||||
# Build per-message diagnostics when collect_diagnostics=True
|
||||
message_decisions: list[MessageDecision] = []
|
||||
if collect_diagnostics:
|
||||
for idx, orig_msg in enumerate(messages):
|
||||
slot = result_slots[idx] if idx < len(result_slots) else None
|
||||
if slot is None:
|
||||
continue
|
||||
# Delegates to the canonical block-aware counter (also used for
|
||||
# net-cost estimation) so list/block content (Anthropic-style
|
||||
# tool_result/text blocks) gets a real count instead of the 0
|
||||
# a str-only check would give it.
|
||||
tok_before = _netcost_message_tokens(orig_msg, tokenizer)
|
||||
tok_after = _netcost_message_tokens(slot, tokenizer)
|
||||
# "uninstrumented" (never "compressed:...") for any route that
|
||||
# didn't tag `_diag`: claiming a compression that may not have
|
||||
# happened is worse than admitting the diagnostic is incomplete
|
||||
# for that path.
|
||||
action = _diag.get(idx, "uninstrumented")
|
||||
message_decisions.append(
|
||||
MessageDecision(
|
||||
message_index=idx,
|
||||
role=orig_msg.get("role", ""),
|
||||
tokens_before=tok_before,
|
||||
tokens_after=tok_after,
|
||||
action=action,
|
||||
)
|
||||
)
|
||||
|
||||
all_transforms = lifecycle_transforms + transforms_applied
|
||||
return TransformResult(
|
||||
messages=transformed_messages,
|
||||
@@ -6491,6 +6584,7 @@ class ContentRouter(Transform):
|
||||
markers_inserted=lifecycle_ccr_hashes,
|
||||
warnings=warnings,
|
||||
timing=compressor_timing,
|
||||
message_decisions=message_decisions,
|
||||
)
|
||||
|
||||
def _lossless_compact_excluded(self, content: Any) -> tuple[str, str] | None:
|
||||
|
||||
@@ -317,6 +317,7 @@ class TransformPipeline:
|
||||
all_markers: list[str] = []
|
||||
all_warnings: list[str] = []
|
||||
all_timing: dict[str, float] = {} # transform_name → ms
|
||||
all_message_decisions: list[Any] = []
|
||||
|
||||
# Track transform diffs if enabled
|
||||
transform_diffs: list[TransformDiff] = []
|
||||
@@ -401,6 +402,8 @@ class TransformPipeline:
|
||||
all_markers.extend(result.markers_inserted)
|
||||
all_warnings.extend(result.warnings)
|
||||
all_timing[transform.name] = duration_ms
|
||||
if result.message_decisions:
|
||||
all_message_decisions.extend(result.message_decisions)
|
||||
|
||||
# Merge sub-transform timing (e.g. ContentRouter's per-compressor breakdown)
|
||||
if result.timing:
|
||||
@@ -543,6 +546,7 @@ class TransformPipeline:
|
||||
diff_artifact=diff_artifact,
|
||||
timing=all_timing,
|
||||
waste_signals=waste_signals,
|
||||
message_decisions=all_message_decisions,
|
||||
)
|
||||
|
||||
def simulate(
|
||||
|
||||
@@ -0,0 +1,303 @@
|
||||
"""Tests for per-message compression diagnostics (issue #2855).
|
||||
|
||||
Verifies that CompressConfig(diagnostics=True) — and the HEADROOM_DIAGNOSTICS=1
|
||||
env var — populate CompressResult.diagnostics with per-message MessageDecision
|
||||
objects, and that the compress() function correctly threads collect_diagnostics
|
||||
through to the pipeline and collects the results.
|
||||
|
||||
ContentRouter integration tests (which need the compiled headroom._core extension)
|
||||
run in CI; this file tests the compress() ↔ pipeline contract via a mock pipeline.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import os
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import patch
|
||||
|
||||
from headroom.compress import CompressConfig, compress
|
||||
from headroom.config import MessageDecision, TransformResult
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Mock pipeline factory
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _noop_result(messages: list[dict]) -> TransformResult:
|
||||
"""TransformResult that passes messages through unchanged, no decisions."""
|
||||
tokens = sum(len(str(m.get("content", ""))) for m in messages)
|
||||
return TransformResult(
|
||||
messages=messages,
|
||||
tokens_before=tokens,
|
||||
tokens_after=tokens,
|
||||
transforms_applied=["router:noop"],
|
||||
)
|
||||
|
||||
|
||||
def _result_with_decisions(
|
||||
messages: list[dict], decisions: list[MessageDecision]
|
||||
) -> TransformResult:
|
||||
"""TransformResult that includes per-message decisions."""
|
||||
tokens = sum(len(str(m.get("content", ""))) for m in messages)
|
||||
return TransformResult(
|
||||
messages=messages,
|
||||
tokens_before=tokens,
|
||||
tokens_after=tokens,
|
||||
transforms_applied=["router:noop"],
|
||||
message_decisions=decisions,
|
||||
)
|
||||
|
||||
|
||||
class _MockPipeline:
|
||||
"""Pipeline stub that captures kwargs and returns a configurable result."""
|
||||
|
||||
def __init__(self, decisions: list[MessageDecision] | None = None):
|
||||
self.last_kwargs: dict = {}
|
||||
self._decisions = decisions or []
|
||||
|
||||
def apply(self, messages, model, **kwargs):
|
||||
self.last_kwargs = kwargs
|
||||
return _result_with_decisions(messages, self._decisions)
|
||||
|
||||
|
||||
def _fake_otel():
|
||||
return SimpleNamespace(record_compression_failure=lambda **kw: None)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# MessageDecision dataclass
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_message_decision_dataclass():
|
||||
"""MessageDecision is a dataclass with the expected fields."""
|
||||
dec = MessageDecision(
|
||||
message_index=0,
|
||||
role="user",
|
||||
tokens_before=100,
|
||||
tokens_after=60,
|
||||
action="compressed:kompress:0.60",
|
||||
)
|
||||
assert dec.message_index == 0
|
||||
assert dec.role == "user"
|
||||
assert dec.tokens_before == 100
|
||||
assert dec.tokens_after == 60
|
||||
assert dec.action == "compressed:kompress:0.60"
|
||||
|
||||
|
||||
def test_message_decision_importable_from_headroom():
|
||||
"""MessageDecision is accessible via the headroom top-level package."""
|
||||
import headroom
|
||||
|
||||
assert hasattr(headroom, "MessageDecision")
|
||||
cls = headroom.MessageDecision
|
||||
dec = cls(
|
||||
message_index=1,
|
||||
role="assistant",
|
||||
tokens_before=50,
|
||||
tokens_after=50,
|
||||
action="passthrough:small",
|
||||
)
|
||||
assert dec.message_index == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TransformResult carries message_decisions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_transform_result_has_message_decisions_field():
|
||||
"""TransformResult has a message_decisions field defaulting to []."""
|
||||
tr = TransformResult(
|
||||
messages=[],
|
||||
tokens_before=0,
|
||||
tokens_after=0,
|
||||
transforms_applied=[],
|
||||
)
|
||||
assert hasattr(tr, "message_decisions")
|
||||
assert tr.message_decisions == []
|
||||
|
||||
|
||||
def test_transform_result_message_decisions_populated():
|
||||
"""TransformResult stores MessageDecision objects when provided."""
|
||||
dec = MessageDecision(
|
||||
message_index=0, role="user", tokens_before=10, tokens_after=10, action="passthrough:small"
|
||||
)
|
||||
tr = TransformResult(
|
||||
messages=[],
|
||||
tokens_before=0,
|
||||
tokens_after=0,
|
||||
transforms_applied=[],
|
||||
message_decisions=[dec],
|
||||
)
|
||||
assert len(tr.message_decisions) == 1
|
||||
assert tr.message_decisions[0] is dec
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CompressConfig.diagnostics field
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_compress_config_diagnostics_defaults_false():
|
||||
"""CompressConfig.diagnostics is False by default."""
|
||||
cfg = CompressConfig()
|
||||
assert cfg.diagnostics is False
|
||||
|
||||
|
||||
def test_compress_config_diagnostics_can_be_set():
|
||||
"""CompressConfig(diagnostics=True) stores True."""
|
||||
cfg = CompressConfig(diagnostics=True)
|
||||
assert cfg.diagnostics is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# compress() does NOT collect diagnostics by default
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_compress_result_diagnostics_none_by_default(monkeypatch):
|
||||
"""CompressResult.diagnostics is None when diagnostics is not requested."""
|
||||
compress_module = importlib.import_module("headroom.compress")
|
||||
mock_pipeline = _MockPipeline()
|
||||
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: mock_pipeline)
|
||||
monkeypatch.setattr(compress_module, "get_otel_metrics", _fake_otel)
|
||||
|
||||
messages = [{"role": "user", "content": "hello"}]
|
||||
result = compress(messages, model="gpt-4o")
|
||||
|
||||
assert result.diagnostics is None
|
||||
assert mock_pipeline.last_kwargs.get("collect_diagnostics") is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# compress() passes collect_diagnostics when CompressConfig.diagnostics=True
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_compress_passes_collect_diagnostics_when_config_true(monkeypatch):
|
||||
"""compress() passes collect_diagnostics=True to pipeline.apply() when config.diagnostics=True."""
|
||||
compress_module = importlib.import_module("headroom.compress")
|
||||
mock_pipeline = _MockPipeline()
|
||||
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: mock_pipeline)
|
||||
monkeypatch.setattr(compress_module, "get_otel_metrics", _fake_otel)
|
||||
|
||||
messages = [{"role": "user", "content": "hello"}]
|
||||
cfg = CompressConfig(diagnostics=True)
|
||||
compress(messages, model="gpt-4o", config=cfg)
|
||||
|
||||
assert mock_pipeline.last_kwargs.get("collect_diagnostics") is True
|
||||
|
||||
|
||||
def test_compress_result_diagnostics_list_when_config_true(monkeypatch):
|
||||
"""CompressResult.diagnostics is a list (possibly empty) when config.diagnostics=True."""
|
||||
compress_module = importlib.import_module("headroom.compress")
|
||||
mock_pipeline = _MockPipeline(decisions=[])
|
||||
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: mock_pipeline)
|
||||
monkeypatch.setattr(compress_module, "get_otel_metrics", _fake_otel)
|
||||
|
||||
messages = [{"role": "user", "content": "hello"}]
|
||||
cfg = CompressConfig(diagnostics=True)
|
||||
result = compress(messages, model="gpt-4o", config=cfg)
|
||||
|
||||
assert result.diagnostics is not None
|
||||
assert isinstance(result.diagnostics, list)
|
||||
|
||||
|
||||
def test_compress_result_diagnostics_contains_decisions(monkeypatch):
|
||||
"""CompressResult.diagnostics contains the MessageDecision objects from the pipeline."""
|
||||
compress_module = importlib.import_module("headroom.compress")
|
||||
expected = [
|
||||
MessageDecision(
|
||||
message_index=0,
|
||||
role="user",
|
||||
tokens_before=10,
|
||||
tokens_after=10,
|
||||
action="protected:user_message",
|
||||
),
|
||||
MessageDecision(
|
||||
message_index=1,
|
||||
role="tool",
|
||||
tokens_before=200,
|
||||
tokens_after=80,
|
||||
action="compressed:kompress:0.40",
|
||||
),
|
||||
]
|
||||
mock_pipeline = _MockPipeline(decisions=expected)
|
||||
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: mock_pipeline)
|
||||
monkeypatch.setattr(compress_module, "get_otel_metrics", _fake_otel)
|
||||
|
||||
messages = [
|
||||
{"role": "user", "content": "x" * 40},
|
||||
{"role": "tool", "tool_call_id": "t1", "content": "y" * 800},
|
||||
]
|
||||
cfg = CompressConfig(diagnostics=True)
|
||||
result = compress(messages, model="gpt-4o", config=cfg)
|
||||
|
||||
assert result.diagnostics == expected
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# HEADROOM_DIAGNOSTICS=1 env var
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_env_var_enables_diagnostics(monkeypatch):
|
||||
"""HEADROOM_DIAGNOSTICS=1 enables collect_diagnostics without config flag."""
|
||||
compress_module = importlib.import_module("headroom.compress")
|
||||
mock_pipeline = _MockPipeline()
|
||||
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: mock_pipeline)
|
||||
monkeypatch.setattr(compress_module, "get_otel_metrics", _fake_otel)
|
||||
|
||||
messages = [{"role": "user", "content": "hello"}]
|
||||
with patch.dict(os.environ, {"HEADROOM_DIAGNOSTICS": "1"}):
|
||||
result = compress(messages, model="gpt-4o")
|
||||
|
||||
assert mock_pipeline.last_kwargs.get("collect_diagnostics") is True
|
||||
assert result.diagnostics is not None
|
||||
|
||||
|
||||
def test_env_var_absent_keeps_diagnostics_none(monkeypatch):
|
||||
"""When HEADROOM_DIAGNOSTICS is unset, result.diagnostics stays None."""
|
||||
compress_module = importlib.import_module("headroom.compress")
|
||||
mock_pipeline = _MockPipeline()
|
||||
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: mock_pipeline)
|
||||
monkeypatch.setattr(compress_module, "get_otel_metrics", _fake_otel)
|
||||
|
||||
messages = [{"role": "user", "content": "hello"}]
|
||||
env = {k: v for k, v in os.environ.items() if k != "HEADROOM_DIAGNOSTICS"}
|
||||
with patch.dict(os.environ, env, clear=True):
|
||||
result = compress(messages, model="gpt-4o")
|
||||
|
||||
assert result.diagnostics is None
|
||||
|
||||
|
||||
def test_env_var_overrides_config_false(monkeypatch):
|
||||
"""HEADROOM_DIAGNOSTICS=1 enables diagnostics even when CompressConfig.diagnostics=False."""
|
||||
compress_module = importlib.import_module("headroom.compress")
|
||||
mock_pipeline = _MockPipeline()
|
||||
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: mock_pipeline)
|
||||
monkeypatch.setattr(compress_module, "get_otel_metrics", _fake_otel)
|
||||
|
||||
messages = [{"role": "user", "content": "hello"}]
|
||||
cfg = CompressConfig(diagnostics=False)
|
||||
with patch.dict(os.environ, {"HEADROOM_DIAGNOSTICS": "1"}):
|
||||
result = compress(messages, model="gpt-4o", config=cfg)
|
||||
|
||||
assert result.diagnostics is not None
|
||||
assert mock_pipeline.last_kwargs.get("collect_diagnostics") is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# diagnostics=None when optimization is disabled
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_no_diagnostics_when_optimize_false():
|
||||
"""compress(optimize=False) returns empty CompressResult without calling pipeline."""
|
||||
cfg = CompressConfig(diagnostics=True)
|
||||
messages = [{"role": "user", "content": "hello"}]
|
||||
result = compress(messages, model="gpt-4o", optimize=False, config=cfg)
|
||||
# optimize=False early-returns before calling the pipeline
|
||||
assert result.diagnostics is None
|
||||
@@ -336,6 +336,7 @@ class TestTransformResult:
|
||||
"cache_metrics",
|
||||
"timing",
|
||||
"waste_signals",
|
||||
"message_decisions",
|
||||
}
|
||||
assert field_names == expected_fields
|
||||
|
||||
|
||||
@@ -204,6 +204,105 @@ def test_force_kompress_routes_anthropic_tool_result_to_targeted_kompress(
|
||||
assert captured["target_ratio"] == 0.10
|
||||
|
||||
|
||||
def test_diagnostics_content_blocks_unchanged_reports_protected_not_compressed(router, tokenizer):
|
||||
"""Regression for PR #3058 review: a content-block (Anthropic-shape)
|
||||
message where every block is protected/passed through must be diagnosed
|
||||
as unchanged -- not falsely reported as "compressed:content_blocks" just
|
||||
because it took the content-blocks route.
|
||||
"""
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": "toolu_small_1",
|
||||
# Far below any min_chars compression threshold, so
|
||||
# _process_content_blocks passes it through untouched.
|
||||
"content": "ok",
|
||||
}
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
result = router.apply(
|
||||
messages,
|
||||
tokenizer,
|
||||
collect_diagnostics=True,
|
||||
compress_user_messages=True,
|
||||
min_tokens_to_compress=1,
|
||||
read_protection_window=0,
|
||||
)
|
||||
|
||||
assert len(result.message_decisions) == 1
|
||||
decision = result.message_decisions[0]
|
||||
assert decision.action == "protected:content_blocks_unchanged"
|
||||
assert decision.tokens_before == decision.tokens_after
|
||||
assert decision.tokens_before > 0
|
||||
|
||||
|
||||
def test_diagnostics_content_blocks_compressed_reports_accurate_tokens(
|
||||
router, tokenizer, monkeypatch
|
||||
):
|
||||
"""Regression for PR #3058 review: a genuinely-compressed content-block
|
||||
message must report real before/after token counts (via the content-block
|
||||
tokenizer), not the previous hard-coded 0s that only counted string
|
||||
content.
|
||||
"""
|
||||
captured: dict[str, object] = {}
|
||||
|
||||
class FakeKompress:
|
||||
def is_ready(self) -> bool:
|
||||
return True
|
||||
|
||||
def ensure_background_load(self) -> None:
|
||||
pass
|
||||
|
||||
def compress(self, content, **kwargs):
|
||||
captured.update(kwargs)
|
||||
compressed = " ".join(content.split()[:20]) + " Retrieve more: hash=deadbeef"
|
||||
return SimpleNamespace(
|
||||
compressed=compressed,
|
||||
compressed_tokens=len(compressed.split()),
|
||||
)
|
||||
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: FakeKompress())
|
||||
tool_content = " ".join(
|
||||
f'{{"file":"src/module_{i}.py","line":{i},"text":"repeated search payload"}}'
|
||||
for i in range(160)
|
||||
)
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": "toolu_search_1",
|
||||
"content": tool_content,
|
||||
}
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
result = router.apply(
|
||||
messages,
|
||||
tokenizer,
|
||||
collect_diagnostics=True,
|
||||
force_kompress=True,
|
||||
target_ratio=0.10,
|
||||
compress_user_messages=True,
|
||||
min_tokens_to_compress=10,
|
||||
read_protection_window=0,
|
||||
)
|
||||
|
||||
assert len(result.message_decisions) == 1
|
||||
decision = result.message_decisions[0]
|
||||
assert decision.action == "compressed:content_blocks"
|
||||
assert decision.tokens_before > 0
|
||||
assert decision.tokens_after > 0
|
||||
assert decision.tokens_after < decision.tokens_before
|
||||
|
||||
|
||||
def test_skip_kompress_routes_around_ml_stage(router, tokenizer, monkeypatch):
|
||||
"""skip_kompress (cold-start fast pass) must never invoke the Kompress ML
|
||||
stage — units that would route there take the same fallback as when the
|
||||
|
||||
Reference in New Issue
Block a user