mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Adds `CompressConfig.diagnostics` (bool, default `False`) and a companion `HEADROOM_DIAGNOSTICS=1` environment variable that enable per-message decision logging on every call to `compress()`. When active, `CompressResult.diagnostics` is a list of `MessageDecision` objects — one per input message — each describing: | Field | Description | |---|---| | `message_index` | Zero-based position in the original messages list | | `role` | `"user"` / `"assistant"` / `"tool"` / `"system"` / … | | `tokens_before` | Token count before the ContentRouter acted on this message | | `tokens_after` | Token count after (equals `tokens_before` for protected/passthrough) | | `action` | What was done: `"compressed:kompress:0.72"`, `"protected:user_message"`, `"passthrough:frozen"`, `"passthrough:small"`, etc. | Closes #2855. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`headroom/config.py`** — add `MessageDecision` dataclass; add `message_decisions: list[MessageDecision]` field to `TransformResult` - **`headroom/compress.py`** — add `CompressConfig.diagnostics`; add `CompressResult.diagnostics`; derive `collect_diagnostics` from config + env var; pass to pipeline; collect from `TransformResult` - **`headroom/transforms/pipeline.py`** — accumulate `message_decisions` from each transform result - **`headroom/transforms/content_router.py`** — instrument every decision branch in `apply()` with `_diag[i] = "action:..."` strings when `collect_diagnostics=True`; build `MessageDecision` list after Pass 3 (zero overhead when flag is off) - **`headroom/__init__.py`** — export `MessageDecision` in `__all__` and lazy-import map - **`tests/test_compress_diagnostics.py`** — 14 tests covering the full `compress() <-> pipeline` contract via a mock pipeline ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_compress_diagnostics.py -v ========================= test session starts ========================== collected 14 items tests/test_compress_diagnostics.py::test_message_decision_dataclass PASSED tests/test_compress_diagnostics.py::test_message_decision_importable_from_headroom PASSED tests/test_compress_diagnostics.py::test_transform_result_has_message_decisions_field PASSED tests/test_compress_diagnostics.py::test_transform_result_message_decisions_populated PASSED tests/test_compress_diagnostics.py::test_compress_config_diagnostics_defaults_false PASSED tests/test_compress_diagnostics.py::test_compress_config_diagnostics_can_be_set PASSED tests/test_compress_diagnostics.py::test_compress_result_diagnostics_none_by_default PASSED tests/test_compress_diagnostics.py::test_compress_passes_collect_diagnostics_when_config_true PASSED tests/test_compress_diagnostics.py::test_compress_result_diagnostics_list_when_config_true PASSED tests/test_compress_diagnostics.py::test_compress_result_diagnostics_contains_decisions PASSED tests/test_compress_diagnostics.py::test_env_var_enables_diagnostics PASSED tests/test_compress_diagnostics.py::test_env_var_absent_keeps_diagnostics_none PASSED tests/test_compress_diagnostics.py::test_env_var_overrides_config_false PASSED tests/test_compress_diagnostics.py::test_no_diagnostics_when_optimize_false PASSED ========================== 14 passed in 0.42s ========================== $ ruff check . All checks passed! ``` ## Real Behavior Proof - Environment: Python 3.12, headroom dev install - Exact command / steps: `python -m pytest tests/test_compress_diagnostics.py -v` - Observed result: 14 tests pass; diagnostics list is populated when `CompressConfig(diagnostics=True)` or `HEADROOM_DIAGNOSTICS=1` - Not tested: ContentRouter integration tests (require compiled `headroom._core` extension — run in CI) ## Runtime Rollout Safety - Rollout-managed feature(s): None — diagnostics are opt-in via `CompressConfig.diagnostics=True` or `HEADROOM_DIAGNOSTICS=1` - Minimum rollout channel: N/A (additive, opt-in only) - Stable/default behavior changed: No — `CompressConfig.diagnostics` defaults to `False`; `CompressResult.diagnostics` is `None` unless explicitly requested - Kill switch / disable path: Simply omit `diagnostics=True` / unset `HEADROOM_DIAGNOSTICS` - Unsafe override required: No - Qualification impact: None - Rollback path: Revert this PR; no schema or data migration involved ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — no UI changes. ## Additional Notes The `collect_diagnostics` flag incurs zero overhead in the hot path when not set: the `_diag` dict in `ContentRouter.apply()` is never allocated and no `MessageDecision` objects are constructed. Signed-off-by: Radha Krishnan P <gingeekrishna@gmail.com> --------- Signed-off-by: Radhakrishnan Pachyappan <radhakrishnan.p@op.tech> Signed-off-by: Radhakrishnan P <gingeekrishna@gmail.com>
304 lines
11 KiB
Python
304 lines
11 KiB
Python
"""Tests for per-message compression diagnostics (issue #2855).
|
|
|
|
Verifies that CompressConfig(diagnostics=True) — and the HEADROOM_DIAGNOSTICS=1
|
|
env var — populate CompressResult.diagnostics with per-message MessageDecision
|
|
objects, and that the compress() function correctly threads collect_diagnostics
|
|
through to the pipeline and collects the results.
|
|
|
|
ContentRouter integration tests (which need the compiled headroom._core extension)
|
|
run in CI; this file tests the compress() ↔ pipeline contract via a mock pipeline.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import importlib
|
|
import os
|
|
from types import SimpleNamespace
|
|
from unittest.mock import patch
|
|
|
|
from headroom.compress import CompressConfig, compress
|
|
from headroom.config import MessageDecision, TransformResult
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Mock pipeline factory
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _noop_result(messages: list[dict]) -> TransformResult:
|
|
"""TransformResult that passes messages through unchanged, no decisions."""
|
|
tokens = sum(len(str(m.get("content", ""))) for m in messages)
|
|
return TransformResult(
|
|
messages=messages,
|
|
tokens_before=tokens,
|
|
tokens_after=tokens,
|
|
transforms_applied=["router:noop"],
|
|
)
|
|
|
|
|
|
def _result_with_decisions(
|
|
messages: list[dict], decisions: list[MessageDecision]
|
|
) -> TransformResult:
|
|
"""TransformResult that includes per-message decisions."""
|
|
tokens = sum(len(str(m.get("content", ""))) for m in messages)
|
|
return TransformResult(
|
|
messages=messages,
|
|
tokens_before=tokens,
|
|
tokens_after=tokens,
|
|
transforms_applied=["router:noop"],
|
|
message_decisions=decisions,
|
|
)
|
|
|
|
|
|
class _MockPipeline:
|
|
"""Pipeline stub that captures kwargs and returns a configurable result."""
|
|
|
|
def __init__(self, decisions: list[MessageDecision] | None = None):
|
|
self.last_kwargs: dict = {}
|
|
self._decisions = decisions or []
|
|
|
|
def apply(self, messages, model, **kwargs):
|
|
self.last_kwargs = kwargs
|
|
return _result_with_decisions(messages, self._decisions)
|
|
|
|
|
|
def _fake_otel():
|
|
return SimpleNamespace(record_compression_failure=lambda **kw: None)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# MessageDecision dataclass
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_message_decision_dataclass():
|
|
"""MessageDecision is a dataclass with the expected fields."""
|
|
dec = MessageDecision(
|
|
message_index=0,
|
|
role="user",
|
|
tokens_before=100,
|
|
tokens_after=60,
|
|
action="compressed:kompress:0.60",
|
|
)
|
|
assert dec.message_index == 0
|
|
assert dec.role == "user"
|
|
assert dec.tokens_before == 100
|
|
assert dec.tokens_after == 60
|
|
assert dec.action == "compressed:kompress:0.60"
|
|
|
|
|
|
def test_message_decision_importable_from_headroom():
|
|
"""MessageDecision is accessible via the headroom top-level package."""
|
|
import headroom
|
|
|
|
assert hasattr(headroom, "MessageDecision")
|
|
cls = headroom.MessageDecision
|
|
dec = cls(
|
|
message_index=1,
|
|
role="assistant",
|
|
tokens_before=50,
|
|
tokens_after=50,
|
|
action="passthrough:small",
|
|
)
|
|
assert dec.message_index == 1
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# TransformResult carries message_decisions
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_transform_result_has_message_decisions_field():
|
|
"""TransformResult has a message_decisions field defaulting to []."""
|
|
tr = TransformResult(
|
|
messages=[],
|
|
tokens_before=0,
|
|
tokens_after=0,
|
|
transforms_applied=[],
|
|
)
|
|
assert hasattr(tr, "message_decisions")
|
|
assert tr.message_decisions == []
|
|
|
|
|
|
def test_transform_result_message_decisions_populated():
|
|
"""TransformResult stores MessageDecision objects when provided."""
|
|
dec = MessageDecision(
|
|
message_index=0, role="user", tokens_before=10, tokens_after=10, action="passthrough:small"
|
|
)
|
|
tr = TransformResult(
|
|
messages=[],
|
|
tokens_before=0,
|
|
tokens_after=0,
|
|
transforms_applied=[],
|
|
message_decisions=[dec],
|
|
)
|
|
assert len(tr.message_decisions) == 1
|
|
assert tr.message_decisions[0] is dec
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# CompressConfig.diagnostics field
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_compress_config_diagnostics_defaults_false():
|
|
"""CompressConfig.diagnostics is False by default."""
|
|
cfg = CompressConfig()
|
|
assert cfg.diagnostics is False
|
|
|
|
|
|
def test_compress_config_diagnostics_can_be_set():
|
|
"""CompressConfig(diagnostics=True) stores True."""
|
|
cfg = CompressConfig(diagnostics=True)
|
|
assert cfg.diagnostics is True
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# compress() does NOT collect diagnostics by default
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_compress_result_diagnostics_none_by_default(monkeypatch):
|
|
"""CompressResult.diagnostics is None when diagnostics is not requested."""
|
|
compress_module = importlib.import_module("headroom.compress")
|
|
mock_pipeline = _MockPipeline()
|
|
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: mock_pipeline)
|
|
monkeypatch.setattr(compress_module, "get_otel_metrics", _fake_otel)
|
|
|
|
messages = [{"role": "user", "content": "hello"}]
|
|
result = compress(messages, model="gpt-4o")
|
|
|
|
assert result.diagnostics is None
|
|
assert mock_pipeline.last_kwargs.get("collect_diagnostics") is False
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# compress() passes collect_diagnostics when CompressConfig.diagnostics=True
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_compress_passes_collect_diagnostics_when_config_true(monkeypatch):
|
|
"""compress() passes collect_diagnostics=True to pipeline.apply() when config.diagnostics=True."""
|
|
compress_module = importlib.import_module("headroom.compress")
|
|
mock_pipeline = _MockPipeline()
|
|
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: mock_pipeline)
|
|
monkeypatch.setattr(compress_module, "get_otel_metrics", _fake_otel)
|
|
|
|
messages = [{"role": "user", "content": "hello"}]
|
|
cfg = CompressConfig(diagnostics=True)
|
|
compress(messages, model="gpt-4o", config=cfg)
|
|
|
|
assert mock_pipeline.last_kwargs.get("collect_diagnostics") is True
|
|
|
|
|
|
def test_compress_result_diagnostics_list_when_config_true(monkeypatch):
|
|
"""CompressResult.diagnostics is a list (possibly empty) when config.diagnostics=True."""
|
|
compress_module = importlib.import_module("headroom.compress")
|
|
mock_pipeline = _MockPipeline(decisions=[])
|
|
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: mock_pipeline)
|
|
monkeypatch.setattr(compress_module, "get_otel_metrics", _fake_otel)
|
|
|
|
messages = [{"role": "user", "content": "hello"}]
|
|
cfg = CompressConfig(diagnostics=True)
|
|
result = compress(messages, model="gpt-4o", config=cfg)
|
|
|
|
assert result.diagnostics is not None
|
|
assert isinstance(result.diagnostics, list)
|
|
|
|
|
|
def test_compress_result_diagnostics_contains_decisions(monkeypatch):
|
|
"""CompressResult.diagnostics contains the MessageDecision objects from the pipeline."""
|
|
compress_module = importlib.import_module("headroom.compress")
|
|
expected = [
|
|
MessageDecision(
|
|
message_index=0,
|
|
role="user",
|
|
tokens_before=10,
|
|
tokens_after=10,
|
|
action="protected:user_message",
|
|
),
|
|
MessageDecision(
|
|
message_index=1,
|
|
role="tool",
|
|
tokens_before=200,
|
|
tokens_after=80,
|
|
action="compressed:kompress:0.40",
|
|
),
|
|
]
|
|
mock_pipeline = _MockPipeline(decisions=expected)
|
|
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: mock_pipeline)
|
|
monkeypatch.setattr(compress_module, "get_otel_metrics", _fake_otel)
|
|
|
|
messages = [
|
|
{"role": "user", "content": "x" * 40},
|
|
{"role": "tool", "tool_call_id": "t1", "content": "y" * 800},
|
|
]
|
|
cfg = CompressConfig(diagnostics=True)
|
|
result = compress(messages, model="gpt-4o", config=cfg)
|
|
|
|
assert result.diagnostics == expected
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# HEADROOM_DIAGNOSTICS=1 env var
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_env_var_enables_diagnostics(monkeypatch):
|
|
"""HEADROOM_DIAGNOSTICS=1 enables collect_diagnostics without config flag."""
|
|
compress_module = importlib.import_module("headroom.compress")
|
|
mock_pipeline = _MockPipeline()
|
|
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: mock_pipeline)
|
|
monkeypatch.setattr(compress_module, "get_otel_metrics", _fake_otel)
|
|
|
|
messages = [{"role": "user", "content": "hello"}]
|
|
with patch.dict(os.environ, {"HEADROOM_DIAGNOSTICS": "1"}):
|
|
result = compress(messages, model="gpt-4o")
|
|
|
|
assert mock_pipeline.last_kwargs.get("collect_diagnostics") is True
|
|
assert result.diagnostics is not None
|
|
|
|
|
|
def test_env_var_absent_keeps_diagnostics_none(monkeypatch):
|
|
"""When HEADROOM_DIAGNOSTICS is unset, result.diagnostics stays None."""
|
|
compress_module = importlib.import_module("headroom.compress")
|
|
mock_pipeline = _MockPipeline()
|
|
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: mock_pipeline)
|
|
monkeypatch.setattr(compress_module, "get_otel_metrics", _fake_otel)
|
|
|
|
messages = [{"role": "user", "content": "hello"}]
|
|
env = {k: v for k, v in os.environ.items() if k != "HEADROOM_DIAGNOSTICS"}
|
|
with patch.dict(os.environ, env, clear=True):
|
|
result = compress(messages, model="gpt-4o")
|
|
|
|
assert result.diagnostics is None
|
|
|
|
|
|
def test_env_var_overrides_config_false(monkeypatch):
|
|
"""HEADROOM_DIAGNOSTICS=1 enables diagnostics even when CompressConfig.diagnostics=False."""
|
|
compress_module = importlib.import_module("headroom.compress")
|
|
mock_pipeline = _MockPipeline()
|
|
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: mock_pipeline)
|
|
monkeypatch.setattr(compress_module, "get_otel_metrics", _fake_otel)
|
|
|
|
messages = [{"role": "user", "content": "hello"}]
|
|
cfg = CompressConfig(diagnostics=False)
|
|
with patch.dict(os.environ, {"HEADROOM_DIAGNOSTICS": "1"}):
|
|
result = compress(messages, model="gpt-4o", config=cfg)
|
|
|
|
assert result.diagnostics is not None
|
|
assert mock_pipeline.last_kwargs.get("collect_diagnostics") is True
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# diagnostics=None when optimization is disabled
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_no_diagnostics_when_optimize_false():
|
|
"""compress(optimize=False) returns empty CompressResult without calling pipeline."""
|
|
cfg = CompressConfig(diagnostics=True)
|
|
messages = [{"role": "user", "content": "hello"}]
|
|
result = compress(messages, model="gpt-4o", optimize=False, config=cfg)
|
|
# optimize=False early-returns before calling the pipeline
|
|
assert result.diagnostics is None
|