refactor(transforms): remove the unused summarize_dropped_items (#4093)

## Description

`compression_summary.summarize_dropped_items` was SmartCrusher's
dropped-item categoriser. SmartCrusher now runs in Rust and emits no
dropped-item summary (`smart_crusher/crusher.rs`), and nothing in Python
called the function. Only tests did, including two LLM eval files. The
module has no `__all__`, is not re-exported, and the function is not
documented or imported by any consumer repo.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Remove `summarize_dropped_items`, its private helpers (`_item_key`,
`_categorize_by_fields`, `_find_notable_items`, `_common_keys`) and
their constants.
- Fix the module docstring, which still said SmartCrusher used it.
`summarize_compressed_code` (used by CodeCompressor) stays.
- Delete `test_compression_summary_hard_eval.py`,
`test_compression_summary_tool_eval.py` and
`test_compression_summary_eval.py`. The last file's one code-summary
test moves into `test_compression_summary.py`.
- Drop the dropped-item cases from the other two test files.
- Evidence: `rg -n summarize_dropped_items headroom crates plugins sdk
examples benchmarks scripts docs wiki` plus the consumer repos finds
only the definition and a Rust doc comment, now reworded.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression_summary.py tests/test_compression_summary_integration.py tests/test_transforms/test_code_compressor.py
109 passed, 1 skipped in 11.24s
$ ruff check / format: All checks passed;  mypy compression_summary.py: Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) import check: `summarize_compressed_code` still works
and `summarize_dropped_items` is gone.
- Observed result: The proxy serves both requests (200) with the same
savings as main. The live summary helper works, and the removed one
raises ImportError. Output below.
- Not tested: The LLM integration test (it skips without
ANTHROPIC_API_KEY).

```text
branch: cleanup/transforms-remove-summarize-dropped-items @ bc0dc2f6
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-51402.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ python -c "from headroom.transforms.compression_summary import summarize_compressed_code; print(summarize_compressed_code([...2 bodies...], 2)); from headroom.transforms.compression_summary import summarize_dropped_items"
2 bodies compressed: authenticate(), refresh()
ImportError: cannot import name 'summarize_dropped_items' from 'headroom.transforms.compression_summary' (headroom/transforms/compression_summary.py)
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: No. The removed function was not
called at runtime.
- Kill switch / disable path: Not applicable; nothing that runs changes.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
2026-10-10 08:58:19 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 08fce18310
commit eaa7308a52
7 changed files with 24 additions and 1105 deletions
@@ -27,7 +27,7 @@
//! - **Telemetry**: no-op.
//! - **`_compress_text_within_items`**: pass-through (returns input
//! unchanged) since text compression has its own port pipeline.
//! - **`summarize_dropped_items`**: empty string.
//! - **Dropped-item summary**: empty string.
//!
//! Parity fixtures will be recorded with all four disabled on the
//! Python side, locking byte-equal output. The TOIN/CCR/feedback
+4 -159
View File
@@ -1,80 +1,17 @@
"""Compression summary generator — describes what was dropped.
When content is compressed, the LLM needs to know what it's missing.
Instead of just "[480 items omitted]", we generate a categorical summary:
"[480 items omitted: 150 log entries (3 with errors), 200 test results (12 failures)]"
When code is compressed, the LLM needs to know what it's missing. Instead of
just "[5 bodies compressed]", we list the removed function/class names (from
the AST, language-agnostic): "5 bodies compressed: authenticate(), ...".
This helps the LLM decide whether to call headroom_retrieve and what to search for.
Used by:
- SmartCrusher: categorizes dropped JSON items by field values
- CodeCompressor: lists removed function/class names (from AST, language-agnostic)
Used by CodeCompressor.
"""
from __future__ import annotations
import re
from collections import Counter
def summarize_dropped_items(
all_items: list[dict],
kept_items: list[dict],
kept_indices: set[int] | None = None,
max_categories: int = 5,
max_notable: int = 3,
) -> str:
"""Generate a categorical summary of items that were dropped.
Args:
all_items: The original full list of items.
kept_items: The items that were kept after compression (used for count).
kept_indices: Indices of kept items (preferred over identity comparison).
max_categories: Maximum number of categories to show.
max_notable: Maximum number of notable items to call out.
Returns:
Summary string or empty string if no useful summary can be generated.
"""
if not all_items or len(kept_items) >= len(all_items):
return ""
# Determine which items were dropped
if kept_indices is not None:
dropped = [item for i, item in enumerate(all_items) if i not in kept_indices]
else:
# Fallback: index-based comparison using JSON equality
kept_json = {_item_key(item) for item in kept_items}
dropped = [item for item in all_items if _item_key(item) not in kept_json]
if not dropped:
return ""
# Strategy 1: Categorize by type/status/kind fields
categories = _categorize_by_fields(dropped)
# Strategy 2: Find notable items (errors, failures, warnings)
notable = _find_notable_items(dropped, max_notable)
# Build summary
parts = []
if categories:
cat_strs = []
for field_val, count in categories.most_common(max_categories):
cat_strs.append(f"{count} {field_val}")
parts.append(", ".join(cat_strs))
if notable:
parts.append(f"notable: {'; '.join(notable)}")
if not parts:
# Fallback: just describe the data shape
keys = _common_keys(dropped)
if keys:
parts.append(f"fields: {', '.join(keys[:5])}")
return "; ".join(parts)
def summarize_compressed_code(
@@ -117,98 +54,6 @@ def summarize_compressed_code(
# ---- Internal helpers ----
# Fields that commonly indicate item category/type
_CATEGORY_FIELDS = (
"type",
"status",
"kind",
"category",
"level",
"severity",
"state",
"phase",
"action",
"event_type",
"log_level",
"result",
"outcome",
)
# Values that indicate something notable/important
_NOTABLE_PATTERNS = re.compile(
r"error|fail|critical|warning|exception|crash|timeout|denied|rejected|invalid",
re.IGNORECASE,
)
# Values that look like URLs or paths (not useful as categories)
_URL_PATTERN = re.compile(r"^https?://|^/[a-z]", re.IGNORECASE)
def _item_key(item: dict) -> str:
"""Create a hashable key for an item (for dropped detection without id())."""
# Use first few field values as a fingerprint
parts = []
for k, v in list(item.items())[:4]:
parts.append(f"{k}={v}")
return "|".join(parts)
def _categorize_by_fields(items: list[dict]) -> Counter:
"""Categorize items by their type/status/kind field values."""
categories: Counter = Counter()
for item in items:
categorized = False
for field in _CATEGORY_FIELDS:
val = item.get(field)
if val and isinstance(val, str) and len(val) < 50:
clean_val = val.replace("\n", " ").replace("\r", "").strip()
if clean_val:
categories[clean_val] += 1
categorized = True
break
if not categorized:
# Try to infer from the item's first short string field
for key, val in item.items():
if (
isinstance(val, str)
and 2 < len(val) < 30
and key not in ("id", "name", "path", "url", "href", "email")
and not _URL_PATTERN.match(val)
):
clean_val = val.replace("\n", " ").replace("\r", "").strip()
categories[f"{key}={clean_val}"] += 1
break
return categories
def _find_notable_items(items: list[dict], max_notable: int) -> list[str]:
"""Find items that contain error/failure/warning indicators."""
notable = []
for item in items:
item_str = str(item)[:500]
matches = _NOTABLE_PATTERNS.findall(item_str)
if matches:
name = item.get("name", item.get("id", item.get("path", "")))
if name:
clean_name = str(name).replace("\n", " ").strip()[:50]
notable.append(f"{clean_name} ({matches[0]})")
else:
notable.append(matches[0])
if len(notable) >= max_notable:
break
return notable
def _common_keys(items: list[dict]) -> list[str]:
"""Get the most common keys across items."""
key_counts: Counter = Counter()
for item in items[:50]:
for key in item.keys():
key_counts[key] += 1
return [k for k, _ in key_counts.most_common(8)]
def _extract_name_from_signature(sig: str) -> str:
"""Extract the function/method name from a signature string.
+16 -78
View File
@@ -3,87 +3,9 @@
from headroom.transforms.compression_summary import (
_extract_name_from_signature,
summarize_compressed_code,
summarize_dropped_items,
)
class TestSummarizeDroppedItems:
def test_items_with_status_field(self):
all_items = [{"id": i, "name": f"item-{i}", "status": "active"} for i in range(50)] + [
{"id": i, "name": f"err-{i}", "status": "error"} for i in range(5)
]
kept = all_items[:3]
summary = summarize_dropped_items(all_items, kept, kept_indices={0, 1, 2})
assert "active" in summary
assert summary
def test_items_with_type_field(self):
all_items = [{"type": "log", "message": f"entry {i}"} for i in range(30)] + [
{"type": "metric", "value": i} for i in range(20)
]
kept = all_items[:5]
summary = summarize_dropped_items(all_items, kept, kept_indices={0, 1, 2, 3, 4})
assert "log" in summary or "metric" in summary
def test_notable_items_with_errors(self):
all_items = [{"name": f"test-{i}", "result": "pass"} for i in range(40)] + [
{"name": "test-auth", "result": "fail", "error": "authentication failed"},
]
kept = all_items[:5]
summary = summarize_dropped_items(all_items, kept, kept_indices=set(range(5)))
assert summary
def test_no_dropped_items(self):
items = [{"id": 1}, {"id": 2}]
summary = summarize_dropped_items(items, items)
assert summary == ""
def test_empty_input(self):
summary = summarize_dropped_items([], [])
assert summary == ""
def test_all_items_dropped(self):
items = [{"status": "active", "name": f"item-{i}"} for i in range(20)]
summary = summarize_dropped_items(items, [], kept_indices=set())
assert "active" in summary
def test_mixed_category_fields(self):
all_items = [{"level": "info", "msg": "something"} for _ in range(10)] + [
{"level": "error", "msg": "bad thing"} for _ in range(3)
]
kept = all_items[:2]
summary = summarize_dropped_items(all_items, kept, kept_indices={0, 1})
assert summary
def test_items_without_category_fields(self):
all_items = [{"code": 200, "count": i} for i in range(30)]
kept = all_items[:3]
summary = summarize_dropped_items(all_items, kept, kept_indices={0, 1, 2})
assert summary # Should produce field-based fallback
def test_summary_not_too_long(self):
all_items = [{"type": f"type_{i % 20}", "data": "x"} for i in range(100)]
kept = all_items[:5]
summary = summarize_dropped_items(all_items, kept, kept_indices=set(range(5)))
assert len(summary) < 500
def test_url_values_excluded_from_categories(self):
"""URL-like values should not be used as category labels."""
all_items = [
{"url": f"https://api.example.com/v1/items/{i}", "method": "GET"} for i in range(30)
]
kept = all_items[:3]
summary = summarize_dropped_items(all_items, kept, kept_indices={0, 1, 2})
assert "https://" not in summary
def test_fallback_without_kept_indices(self):
"""Works correctly without kept_indices (uses _item_key fallback)."""
all_items = [{"status": "active", "id": i} for i in range(20)]
kept = [all_items[0], all_items[1]] # Copies from same list
summary = summarize_dropped_items(all_items, kept)
assert summary
class TestSummarizeCompressedCode:
def test_python_function_bodies(self):
bodies = [
@@ -134,6 +56,22 @@ class TestSummarizeCompressedCode:
summary = summarize_compressed_code(bodies, 20)
assert "+14 more" in summary # 20 - 6 shown
def test_real_python_module(self):
bodies = [
("def __init__(self, url: str, pool_size: int = 10):", "...", 8),
("def connect(self) -> Any:", "...", 15),
("def _create_new(self) -> Any:", "...", 22),
("def release(self, conn: Any) -> None:", "...", 28),
("def close_all(self) -> None:", "...", 33),
("def create_engine(url: str) -> DatabaseConnection:", "...", 38),
]
summary = summarize_compressed_code(bodies, 6)
assert "6 bodies compressed" in summary
has_names = any(
name in summary for name in ["connect()", "release()", "close_all()", "create_engine()"]
)
assert has_names, f"Summary missing function names: {summary}"
class TestExtractNameFromSignature:
def test_python_def(self):
-217
View File
@@ -1,217 +0,0 @@
"""Eval: Compression summary quality — generic, unbiased.
Tests that compression summaries are:
1. Accurate (categories match actual dropped items)
2. Useful (contain information that would help retrieval)
3. Not misleading (don't hallucinate categories)
These are NOT skewed to show summaries as amazing — they test
real-world data patterns and verify correctness.
"""
from headroom.transforms.compression_summary import (
summarize_compressed_code,
summarize_dropped_items,
)
# ============================================================================
# Realistic test data (modeled on actual tool outputs)
# ============================================================================
def _make_github_issues(n: int) -> list[dict]:
"""Realistic GitHub issues list."""
statuses = ["open"] * (n // 2) + ["closed"] * (n // 4) + ["in_progress"] * (n // 4)
issues = []
for i in range(n):
issue = {
"id": i + 1,
"title": f"Issue #{i + 1}: {'Fix auth bug' if i == 42 else 'General issue'}",
"status": statuses[i % len(statuses)],
"labels": ["bug"] if i % 10 == 0 else ["enhancement"],
"assignee": f"user-{i % 5}",
}
if i in (42, 87):
issue["status"] = "open"
issue["labels"] = ["critical", "bug"]
issue["title"] = f"CRITICAL: Auth failure in production (issue #{i + 1})"
issues.append(issue)
return issues
def _make_test_results(n: int) -> list[dict]:
"""Realistic test suite results."""
results = []
for i in range(n):
result = {
"name": f"test_{'auth' if i < 10 else 'general'}_{i}",
"status": "pass",
"duration_ms": 50 + i * 2,
}
if i in (3, 7, 45, 88):
result["status"] = "fail"
result["error"] = "AssertionError: expected True, got False"
if i in (12, 67):
result["status"] = "error"
result["error"] = "TimeoutError: test exceeded 30s limit"
results.append(result)
return results
def _make_log_entries(n: int) -> list[dict]:
"""Realistic log entries."""
entries = []
for i in range(n):
entry = {
"timestamp": f"2024-01-15T10:{i:02d}:00Z",
"level": "info",
"message": f"Request processed in {10 + i}ms",
"service": "api-gateway",
}
if i in (15, 45, 89):
entry["level"] = "error"
entry["message"] = "Connection refused: database pool exhausted"
if i in (20, 50):
entry["level"] = "warning"
entry["message"] = "High memory usage: 85% threshold exceeded"
entries.append(entry)
return entries
def _make_api_endpoints(n: int) -> list[dict]:
"""Realistic API endpoint list."""
return [
{
"path": f"/api/v1/{'users' if i < n // 3 else 'orders' if i < 2 * n // 3 else 'products'}/{i}",
"method": "GET" if i % 3 else "POST",
"status_code": 200 if i % 20 else 500,
"latency_ms": 50 + i,
}
for i in range(n)
]
# ============================================================================
# Eval: Summary accuracy
# ============================================================================
class TestSummaryAccuracy:
"""Verify summaries accurately reflect what was dropped."""
def test_github_issues_categories_correct(self):
"""Summary mentions actual status values from dropped items."""
issues = _make_github_issues(100)
kept = issues[:5]
summary = summarize_dropped_items(issues, kept)
# Should mention the status values present in dropped items
assert summary # Non-empty
# At minimum, should contain some status category info
has_category = any(s in summary.lower() for s in ["open", "closed", "in_progress"])
assert has_category, f"Summary missing status categories: {summary}"
def test_test_results_mentions_failures(self):
"""Summary mentions failures when test results are compressed."""
results = _make_test_results(100)
kept = results[:5]
summary = summarize_dropped_items(results, kept)
assert summary
# Should mention pass/fail somewhere
has_result = any(s in summary.lower() for s in ["pass", "fail", "error"])
assert has_result, f"Summary missing test result info: {summary}"
def test_log_entries_mentions_errors(self):
"""Summary mentions error log entries."""
logs = _make_log_entries(100)
kept = logs[:3]
summary = summarize_dropped_items(logs, kept)
assert summary
# Should categorize by log level
has_level = any(s in summary.lower() for s in ["info", "error", "warning"])
assert has_level, f"Summary missing log level info: {summary}"
def test_no_hallucinated_categories(self):
"""Summary should NOT mention categories that don't exist."""
items = [{"status": "active", "id": i} for i in range(50)]
kept = items[:3]
summary = summarize_dropped_items(items, kept)
# Should NOT mention statuses that don't exist in the data
assert "error" not in summary.lower() or "notable" in summary.lower()
assert "fail" not in summary.lower()
assert "critical" not in summary.lower()
def test_summary_proportional_to_data(self):
"""Category counts in summary should roughly match actual data."""
items = (
[{"type": "log", "data": "x"}] * 100
+ [{"type": "metric", "data": "y"}] * 50
+ [{"type": "alert", "data": "z"}] * 10
)
kept = items[:3]
summary = summarize_dropped_items(items, kept)
# "log" should appear with a higher count than "alert"
# (We can't verify exact counts from the summary string,
# but we verify the summary is non-empty and reasonable)
assert summary
assert len(summary) < 300
class TestSummaryUsefulness:
"""Verify summaries contain information useful for retrieval."""
def test_enough_info_to_search(self):
"""Summary should contain terms the LLM could use as search queries."""
results = _make_test_results(100)
kept = results[:5]
summary = summarize_dropped_items(results, kept)
# The LLM should be able to extract search terms from the summary
# At minimum, it should know WHAT KIND of items are in the compressed data
assert len(summary) > 10, "Summary too short to be useful"
def test_notable_items_actionable(self):
"""Notable items should contain enough info to act on."""
logs = _make_log_entries(100)
kept = logs[:2]
summary = summarize_dropped_items(logs, kept)
# If there are errors in the logs, the summary should help
# the LLM decide to retrieve them
assert summary
# Just verify it's substantive enough
assert len(summary.split()) > 3
def test_api_endpoints_described(self):
"""API endpoint data should produce some useful description."""
endpoints = _make_api_endpoints(60)
kept = endpoints[:5]
summary = summarize_dropped_items(endpoints, kept)
assert summary # Should produce SOMETHING, even without type/status fields
class TestCodeSummaryAccuracy:
"""Verify code summaries accurately describe removed sections."""
def test_real_python_module(self):
"""Summary of a realistic Python module compression."""
# Use AST-based summary (language-agnostic)
bodies = [
("def __init__(self, url: str, pool_size: int = 10):", "...", 8),
("def connect(self) -> Any:", "...", 15),
("def _create_new(self) -> Any:", "...", 22),
("def release(self, conn: Any) -> None:", "...", 28),
("def close_all(self) -> None:", "...", 33),
("def create_engine(url: str) -> DatabaseConnection:", "...", 38),
]
summary = summarize_compressed_code(bodies, 6)
assert "6 bodies compressed" in summary
has_names = any(
name in summary for name in ["connect()", "release()", "close_all()", "create_engine()"]
)
assert has_names, f"Summary missing function names: {summary}"
-251
View File
@@ -1,251 +0,0 @@
"""Hard eval: Cases where the LLM has NO reason to check compressed data.
The previous eval asked "are there failures?" — that's too easy, the LLM
will proactively check regardless of summary.
This eval tests the SUBTLE case: the user asks a DIFFERENT question,
but the answer is in the compressed data. The summary is the only hint.
Requires: ANTHROPIC_API_KEY in environment or .env file.
"""
from __future__ import annotations
import json
import os
import pytest
from tests._dotenv import autouse_apply_env, load_env_overrides
_env_overrides = load_env_overrides()
ANTHROPIC_KEY = os.environ.get("ANTHROPIC_API_KEY") or _env_overrides.get("ANTHROPIC_API_KEY", "")
apply_dotenv = autouse_apply_env(_env_overrides)
pytestmark = pytest.mark.skipif(
not ANTHROPIC_KEY,
reason="ANTHROPIC_API_KEY not set",
)
HEADROOM_RETRIEVE_TOOL = {
"name": "headroom_retrieve",
"description": "Retrieve uncompressed content. Pass a query to search within it.",
"input_schema": {
"type": "object",
"properties": {
"hash": {"type": "string"},
"query": {"type": "string"},
},
"required": ["hash"],
},
}
def _call_claude(messages, tools, max_tokens=300):
import httpx
resp = httpx.post(
"https://api.anthropic.com/v1/messages",
headers={
"X-Api-Key": ANTHROPIC_KEY,
"anthropic-version": "2023-06-01",
"Content-Type": "application/json",
},
json={
"model": "claude-sonnet-4-5-20250929",
"max_tokens": max_tokens,
"messages": messages,
"tools": tools,
},
timeout=30,
)
return resp.json()
def _get_tool_calls(resp):
return [
{"name": b["name"], "input": b.get("input", {})}
for b in resp.get("content", [])
if b.get("type") == "tool_use"
]
def _get_text(resp):
return " ".join(b.get("text", "") for b in resp.get("content", []) if b.get("type") == "text")
class TestHardCases:
"""Cases where the LLM wouldn't naturally check compressed data."""
def test_config_lookup_with_summary(self):
"""User asks about a config value that's in compressed data.
The visible items are all about 'production' env.
The compressed items include 'staging' configs.
Summary mentions this. LLM should retrieve.
"""
visible = [
{"env": "production", "key": "DATABASE_URL", "value": "postgres://prod-db:5432/app"},
{"env": "production", "key": "REDIS_URL", "value": "redis://prod-cache:6379"},
{"env": "production", "key": "API_RATE_LIMIT", "value": "1000"},
]
# Hidden in compressed: staging configs
all_items = (
visible
+ [
{
"env": "staging",
"key": "DATABASE_URL",
"value": "postgres://staging-db:5432/app",
},
{"env": "staging", "key": "REDIS_URL", "value": "redis://staging-cache:6379"},
{"env": "staging", "key": "DEBUG_MODE", "value": "true"},
{"env": "staging", "key": "LOG_LEVEL", "value": "debug"},
]
* 10
+ [
{
"env": "development",
"key": "DATABASE_URL",
"value": "postgres://localhost:5432/dev",
},
]
* 5
)
from headroom.transforms.compression_summary import summarize_dropped_items
summary = summarize_dropped_items(all_items, visible)
compressed_output = json.dumps(visible, indent=2)
compressed_output += (
f"\n[{len(all_items) - len(visible)} items compressed to {len(visible)}."
f" Omitted: {summary}."
f' Retrieve specific items: headroom_retrieve(hash="config_hash", query="search")]'
)
messages = [
{
"role": "user",
"content": (
f"Here are the application configs:\n\n{compressed_output}\n\n"
"What is the staging database URL?"
),
}
]
resp = _call_claude(messages, [HEADROOM_RETRIEVE_TOOL])
tool_calls = _get_tool_calls(resp)
text = _get_text(resp)
print(f"\n Summary: {summary}")
print(f" Stop reason: {resp.get('stop_reason')}")
print(f" Tool calls: {tool_calls}")
if text:
print(f" Text: {text[:200]}")
# WITH summary mentioning "staging" → should retrieve
if resp.get("stop_reason") == "tool_use":
assert tool_calls[0]["name"] == "headroom_retrieve"
query = tool_calls[0]["input"].get("query", "").lower()
assert "staging" in query or "database" in query
print(" RESULT: Retrieved staging config ✓")
else:
# If LLM didn't retrieve, it should at least mention the data is compressed
assert "compressed" in text.lower() or "staging" in text.lower()
print(" RESULT: Mentioned compressed data but didn't retrieve")
def test_config_lookup_without_summary(self):
"""Same question, but NO summary. LLM only sees production configs."""
visible = [
{"env": "production", "key": "DATABASE_URL", "value": "postgres://prod-db:5432/app"},
{"env": "production", "key": "REDIS_URL", "value": "redis://prod-cache:6379"},
{"env": "production", "key": "API_RATE_LIMIT", "value": "1000"},
]
compressed_output = json.dumps(visible, indent=2)
compressed_output += "\n[45 items compressed to 3. Retrieve more: hash=config_hash]"
messages = [
{
"role": "user",
"content": (
f"Here are the application configs:\n\n{compressed_output}\n\n"
"What is the staging database URL?"
),
}
]
resp = _call_claude(messages, [HEADROOM_RETRIEVE_TOOL])
tool_calls = _get_tool_calls(resp)
text = _get_text(resp)
print(f"\n Stop reason: {resp.get('stop_reason')}")
print(f" Tool calls: {tool_calls}")
if text:
print(f" Text: {text[:200]}")
if resp.get("stop_reason") == "tool_use":
print(" RESULT: LLM proactively retrieved (smart)")
else:
print(" RESULT: LLM did NOT retrieve staging config")
def test_specific_user_in_large_list_with_summary(self):
"""Find a specific user in a compressed user list.
Summary mentions user roles. User asks about admins.
"""
visible = [
{"id": i, "name": f"user_{i}", "role": "member", "email": f"user{i}@co.com"}
for i in range(5)
]
all_items = (
visible
+ [
{"id": i, "name": f"user_{i}", "role": "member", "email": f"user{i}@co.com"}
for i in range(5, 95)
]
+ [
{"id": 96, "name": "admin_sarah", "role": "admin", "email": "sarah@co.com"},
{"id": 97, "name": "admin_mike", "role": "admin", "email": "mike@co.com"},
{"id": 98, "name": "superadmin_jane", "role": "superadmin", "email": "jane@co.com"},
]
)
from headroom.transforms.compression_summary import summarize_dropped_items
summary = summarize_dropped_items(all_items, visible)
compressed_output = json.dumps(visible, indent=2)
compressed_output += (
f"\n[{len(all_items) - len(visible)} items compressed to {len(visible)}."
f" Omitted: {summary}."
f' Retrieve: headroom_retrieve(hash="users_hash", query="search")]'
)
messages = [
{
"role": "user",
"content": (
f"Here's our user list:\n\n{compressed_output}\n\n"
"Who are the admin users? I need to contact them."
),
}
]
resp = _call_claude(messages, [HEADROOM_RETRIEVE_TOOL])
tool_calls = _get_tool_calls(resp)
text = _get_text(resp)
print(f"\n Summary: {summary}")
print(f" Stop reason: {resp.get('stop_reason')}")
print(f" Tool calls: {tool_calls}")
if text:
print(f" Text: {text[:200]}")
if resp.get("stop_reason") == "tool_use":
query = tool_calls[0]["input"].get("query", "").lower()
assert "admin" in query
print(f" RESULT: Retrieved admin users (query='{query}') ✓")
else:
print(" RESULT: Did not retrieve admin users")
+3 -110
View File
@@ -1,7 +1,7 @@
"""Integration eval: Compression summaries with real LLM calls.
Tests whether compression summaries actually help the LLM find information
in compressed data. Compares behavior with and without summaries.
Tests whether the code-compression summary actually helps the LLM tell which
functions were removed from compressed code.
Requires: ANTHROPIC_API_KEY in environment or .env file.
@@ -10,7 +10,6 @@ Run: python -m pytest tests/test_compression_summary_integration.py -v -s
from __future__ import annotations
import json
import os
import pytest
@@ -48,114 +47,8 @@ def _call_claude(messages: list[dict], max_tokens: int = 200) -> dict:
return resp.json()
# ============================================================================
# Test data: realistic tool output that gets compressed
# ============================================================================
def _make_test_suite_output(n: int = 100) -> list[dict]:
"""Simulate a large test suite result (like from a CI/CD tool)."""
results = []
for i in range(n):
result = {
"test_name": f"test_module_{i // 10}.test_case_{i}",
"status": "passed",
"duration_ms": 50 + i * 3,
"file": f"tests/test_module_{i // 10}.py",
}
# Inject specific failures that the LLM should find
if i == 42:
result["status"] = "failed"
result["error"] = "AssertionError: expected status 200, got 401 in auth_middleware"
result["test_name"] = "test_auth.test_login_with_expired_token"
if i == 67:
result["status"] = "failed"
result["error"] = "TimeoutError: database connection pool exhausted after 30s"
result["test_name"] = "test_database.test_concurrent_connections"
if i == 88:
result["status"] = "error"
result["error"] = "ImportError: cannot import name 'NewFeature' from 'app.features'"
result["test_name"] = "test_features.test_new_feature_integration"
results.append(result)
return results
class TestSummaryHelpfulness:
"""Compare LLM accuracy with vs without compression summaries."""
def test_find_failures_with_summary(self):
"""LLM can identify failure types from the summary alone."""
test_results = _make_test_suite_output(100)
# Simulate compression: keep first 10, compress rest with summary
kept = test_results[:10]
from headroom.transforms.compression_summary import summarize_dropped_items
summary = summarize_dropped_items(test_results, kept)
compressed_output = json.dumps(kept, indent=2)
compressed_output += f"\n[90 items compressed to 10. Omitted: {summary}. "
compressed_output += (
'Retrieve specific items: headroom_retrieve(hash="abc123", query="your search")]'
)
messages = [
{
"role": "user",
"content": (
"Here are the test results from CI:\n\n"
f"{compressed_output}\n\n"
"Are there any test failures? What types of failures are there? "
"Answer concisely."
),
},
]
resp = _call_claude(messages)
text = resp.get("content", [{}])[0].get("text", "").lower()
# The LLM should mention failures (from the summary info)
has_failure_info = any(
word in text for word in ["fail", "error", "timeout", "assert", "import"]
)
print(f"\n Summary: {summary}")
print(f" LLM response: {text[:200]}")
print(f" Detected failure info: {has_failure_info}")
assert has_failure_info, f"LLM didn't detect failures from summary. Response: {text[:300]}"
def test_find_failures_without_summary(self):
"""Baseline: LLM with NO summary — just '[90 items compressed]'."""
test_results = _make_test_suite_output(100)
kept = test_results[:10]
compressed_output = json.dumps(kept, indent=2)
compressed_output += "\n[90 items compressed to 10. Retrieve more: hash=abc123]"
messages = [
{
"role": "user",
"content": (
"Here are the test results from CI:\n\n"
f"{compressed_output}\n\n"
"Are there any test failures? What types of failures are there? "
"Answer concisely."
),
},
]
resp = _call_claude(messages)
text = resp.get("content", [{}])[0].get("text", "").lower()
# The LLM may or may not detect failures (it only sees 10 passing tests)
has_failure_info = any(
word in text for word in ["fail", "error", "timeout", "assert", "import"]
)
print(f"\n LLM response (no summary): {text[:200]}")
print(f" Detected failure info: {has_failure_info}")
# We're NOT asserting here — this is the baseline.
# We expect this to often MISS failures since the summary is generic.
"""Check that the code summary carries information the LLM can use."""
def test_code_summary_helps_identify_functions(self):
"""LLM can identify which functions were removed from compressed code."""
-289
View File
@@ -1,289 +0,0 @@
"""Eval: Does the LLM invoke headroom_retrieve when summaries are present?
The REAL test — it's not enough for the LLM to know something is missing.
It must actually call the tool to fetch it.
Compares:
- WITH summary: LLM sees "2 failed, 1 error" → should call headroom_retrieve
- WITHOUT summary: LLM sees "[90 items compressed]" → likely does NOT call tool
Requires: ANTHROPIC_API_KEY in environment or .env file.
Run: python -m pytest tests/test_compression_summary_tool_eval.py -v -s
"""
from __future__ import annotations
import json
import os
import pytest
from tests._dotenv import autouse_apply_env, load_env_overrides
_env_overrides = load_env_overrides()
ANTHROPIC_KEY = os.environ.get("ANTHROPIC_API_KEY") or _env_overrides.get("ANTHROPIC_API_KEY", "")
apply_dotenv = autouse_apply_env(_env_overrides)
pytestmark = pytest.mark.skipif(
not ANTHROPIC_KEY,
reason="ANTHROPIC_API_KEY not set — skipping integration tests",
)
# The headroom_retrieve tool definition (same as what CCR injects)
HEADROOM_RETRIEVE_TOOL = {
"name": "headroom_retrieve",
"description": (
"Retrieve original uncompressed content from Headroom's compression cache. "
"Use this when you need more details from compressed data. "
"You can pass a query to search within the compressed content."
),
"input_schema": {
"type": "object",
"properties": {
"hash": {
"type": "string",
"description": "The hash key from the compression marker",
},
"query": {
"type": "string",
"description": "Optional search query to find specific items within the compressed data",
},
},
"required": ["hash"],
},
}
def _call_claude_with_tools(messages: list[dict], tools: list[dict], max_tokens: int = 300) -> dict:
"""Make a real Anthropic API call with tool use."""
import httpx
resp = httpx.post(
"https://api.anthropic.com/v1/messages",
headers={
"X-Api-Key": ANTHROPIC_KEY,
"anthropic-version": "2023-06-01",
"Content-Type": "application/json",
},
json={
"model": "claude-sonnet-4-5-20250929",
"max_tokens": max_tokens,
"messages": messages,
"tools": tools,
},
timeout=30,
)
return resp.json()
def _make_test_results(n: int = 100) -> list[dict]:
"""Test suite output with hidden failures in the compressed portion."""
results = []
for i in range(n):
result = {
"test_name": f"test_module_{i // 10}.test_case_{i}",
"status": "passed",
"duration_ms": 50 + i * 3,
}
if i == 42:
result["status"] = "failed"
result["error"] = "AssertionError: expected 200, got 401 in auth_middleware"
result["test_name"] = "test_auth.test_login_expired_token"
if i == 67:
result["status"] = "failed"
result["error"] = "TimeoutError: database pool exhausted after 30s"
result["test_name"] = "test_database.test_concurrent_connections"
if i == 88:
result["status"] = "error"
result["error"] = "ImportError: cannot import 'NewFeature'"
result["test_name"] = "test_features.test_new_feature_integration"
results.append(result)
return results
def _has_tool_use(response: dict) -> bool:
"""Check if the response contains a tool_use block."""
for block in response.get("content", []):
if block.get("type") == "tool_use":
return True
return False
def _get_tool_calls(response: dict) -> list[dict]:
"""Extract all tool_use blocks from response."""
calls = []
for block in response.get("content", []):
if block.get("type") == "tool_use":
calls.append(
{
"name": block.get("name"),
"input": block.get("input", {}),
}
)
return calls
class TestToolInvocationWithSummary:
"""The real eval: does the LLM call headroom_retrieve?"""
def test_with_summary_triggers_tool_call(self):
"""WITH compression summary → LLM should call headroom_retrieve."""
test_results = _make_test_results(100)
kept = test_results[:10] # All passing
from headroom.transforms.compression_summary import summarize_dropped_items
summary = summarize_dropped_items(test_results, kept)
compressed = json.dumps(kept, indent=2)
compressed += (
f"\n[90 items compressed to 10. Omitted: {summary}."
f' Retrieve specific items: headroom_retrieve(hash="ccr_test_abc123", query="your search")]'
)
messages = [
{
"role": "user",
"content": (
"Here are the test results from our CI pipeline:\n\n"
f"{compressed}\n\n"
"Tell me about any test failures. What went wrong?"
),
},
]
resp = _call_claude_with_tools(messages, [HEADROOM_RETRIEVE_TOOL])
tool_calls = _get_tool_calls(resp)
stop_reason = resp.get("stop_reason", "")
print(f"\n Summary: {summary}")
print(f" Stop reason: {stop_reason}")
print(f" Tool calls: {tool_calls}")
# With a summary showing failures, the LLM SHOULD call the tool
if stop_reason == "tool_use":
assert len(tool_calls) > 0
call = tool_calls[0]
assert call["name"] == "headroom_retrieve"
assert call["input"].get("hash") == "ccr_test_abc123"
# The query should be about failures/errors
query = call["input"].get("query", "").lower()
print(f" Query used: {query}")
has_relevant_query = any(
term in query for term in ["fail", "error", "issue", "problem", "broken", "test"]
)
assert has_relevant_query, f"Tool was called but query isn't relevant: {query}"
print(" RESULT: LLM invoked headroom_retrieve with relevant query ✓")
else:
# LLM responded with text — check if it at least mentions the failures
text = ""
for block in resp.get("content", []):
if block.get("type") == "text":
text += block.get("text", "")
print(f" LLM text response: {text[:200]}")
# It's acceptable if the LLM mentions it WANTS to retrieve
mentions_retrieval = any(
term in text.lower()
for term in ["retrieve", "headroom_retrieve", "fetch", "see more", "compressed"]
)
print(f" Mentions retrieval: {mentions_retrieval}")
def test_without_summary_baseline(self):
"""WITHOUT compression summary → LLM likely does NOT call tool."""
test_results = _make_test_results(100)
kept = test_results[:10] # All passing
compressed = json.dumps(kept, indent=2)
compressed += "\n[90 items compressed to 10. Retrieve more: hash=ccr_test_abc123]"
messages = [
{
"role": "user",
"content": (
"Here are the test results from our CI pipeline:\n\n"
f"{compressed}\n\n"
"Tell me about any test failures. What went wrong?"
),
},
]
resp = _call_claude_with_tools(messages, [HEADROOM_RETRIEVE_TOOL])
tool_calls = _get_tool_calls(resp)
stop_reason = resp.get("stop_reason", "")
print(f"\n Stop reason: {stop_reason}")
print(f" Tool calls: {tool_calls}")
if stop_reason == "tool_use":
call = tool_calls[0]
print(f" Query used: {call['input'].get('query', 'none')}")
print(" RESULT: LLM DID invoke tool (may check proactively)")
else:
text = ""
for block in resp.get("content", []):
if block.get("type") == "text":
text += block.get("text", "")
print(f" LLM text response: {text[:200]}")
print(" RESULT: LLM did NOT invoke tool — assumed all tests passed")
def test_code_summary_triggers_retrieval(self):
"""Code compression summary → LLM should retrieve specific function."""
compressed_code = '''class PaymentProcessor:
"""Processes payments via Stripe."""
def __init__(self, api_key: str):
# [2 lines omitted]
pass
def charge(self, amount: float, currency: str, token: str) -> dict:
# [8 lines omitted]
pass
def refund(self, charge_id: str, amount: float = None) -> dict:
# [3 lines omitted]
pass
def get_balance(self) -> float:
# [2 lines omitted]
pass
# [180 tokens compressed. removed: def charge (12 lines), def refund (6 lines). Retrieve full code: headroom_retrieve(hash="ccr_code_xyz", query="function name")]'''
messages = [
{
"role": "user",
"content": (
"Here's the payment processor code:\n\n"
f"```python\n{compressed_code}\n```\n\n"
"There's a bug in the retry logic for failed charges. "
"Can you find and fix it?"
),
},
]
resp = _call_claude_with_tools(messages, [HEADROOM_RETRIEVE_TOOL])
tool_calls = _get_tool_calls(resp)
stop_reason = resp.get("stop_reason", "")
print(f"\n Stop reason: {stop_reason}")
print(f" Tool calls: {tool_calls}")
if stop_reason == "tool_use":
call = tool_calls[0]
assert call["name"] == "headroom_retrieve"
query = call["input"].get("query", "").lower()
print(f" Query: {query}")
# Should be asking for the charge function specifically
has_charge = any(term in query for term in ["charge", "retry", "payment", "stripe"])
print(f" Targets charge/retry: {has_charge}")
print(" RESULT: LLM invoked tool to get the charge() implementation ✓")
else:
text = ""
for block in resp.get("content", []):
if block.get("type") == "text":
text += block.get("text", "")
print(f" LLM text: {text[:200]}")
print(" RESULT: LLM did NOT invoke tool")