mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description
`compression_summary.summarize_dropped_items` was SmartCrusher's
dropped-item categoriser. SmartCrusher now runs in Rust and emits no
dropped-item summary (`smart_crusher/crusher.rs`), and nothing in Python
called the function. Only tests did, including two LLM eval files. The
module has no `__all__`, is not re-exported, and the function is not
documented or imported by any consumer repo.
Refs #4136
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- Remove `summarize_dropped_items`, its private helpers (`_item_key`,
`_categorize_by_fields`, `_find_notable_items`, `_common_keys`) and
their constants.
- Fix the module docstring, which still said SmartCrusher used it.
`summarize_compressed_code` (used by CodeCompressor) stays.
- Delete `test_compression_summary_hard_eval.py`,
`test_compression_summary_tool_eval.py` and
`test_compression_summary_eval.py`. The last file's one code-summary
test moves into `test_compression_summary.py`.
- Drop the dropped-item cases from the other two test files.
- Evidence: `rg -n summarize_dropped_items headroom crates plugins sdk
examples benchmarks scripts docs wiki` plus the consumer repos finds
only the definition and a Rust doc comment, now reworded.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_compression_summary.py tests/test_compression_summary_integration.py tests/test_transforms/test_code_compressor.py
109 passed, 1 skipped in 11.24s
$ ruff check / format: All checks passed; mypy compression_summary.py: Success
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) import check: `summarize_compressed_code` still works
and `summarize_dropped_items` is gone.
- Observed result: The proxy serves both requests (200) with the same
savings as main. The live summary helper works, and the removed one
raises ImportError. Output below.
- Not tested: The LLM integration test (it skips without
ANTHROPIC_API_KEY).
```text
branch: cleanup/transforms-remove-summarize-dropped-items @ bc0dc2f6
health: 200
POST /v1/messages -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-51402.log):
2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```
```text
$ python -c "from headroom.transforms.compression_summary import summarize_compressed_code; print(summarize_compressed_code([...2 bodies...], 2)); from headroom.transforms.compression_summary import summarize_dropped_items"
2 bodies compressed: authenticate(), refresh()
ImportError: cannot import name 'summarize_dropped_items' from 'headroom.transforms.compression_summary' (headroom/transforms/compression_summary.py)
```
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: No. The removed function was not
called at runtime.
- Kill switch / disable path: Not applicable; nothing that runs changes.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
Not applicable.
## Additional Notes
Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.
Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
99 lines
3.2 KiB
Python
99 lines
3.2 KiB
Python
"""Integration eval: Compression summaries with real LLM calls.
|
|
|
|
Tests whether the code-compression summary actually helps the LLM tell which
|
|
functions were removed from compressed code.
|
|
|
|
Requires: ANTHROPIC_API_KEY in environment or .env file.
|
|
|
|
Run: python -m pytest tests/test_compression_summary_integration.py -v -s
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
|
|
import pytest
|
|
|
|
from tests._dotenv import autouse_apply_env, load_env_overrides
|
|
|
|
_env_overrides = load_env_overrides()
|
|
ANTHROPIC_KEY = os.environ.get("ANTHROPIC_API_KEY") or _env_overrides.get("ANTHROPIC_API_KEY", "")
|
|
apply_dotenv = autouse_apply_env(_env_overrides)
|
|
|
|
pytestmark = pytest.mark.skipif(
|
|
not ANTHROPIC_KEY,
|
|
reason="ANTHROPIC_API_KEY not set — skipping integration tests",
|
|
)
|
|
|
|
|
|
def _call_claude(messages: list[dict], max_tokens: int = 200) -> dict:
|
|
"""Make a real Anthropic API call."""
|
|
import httpx
|
|
|
|
resp = httpx.post(
|
|
"https://api.anthropic.com/v1/messages",
|
|
headers={
|
|
"X-Api-Key": ANTHROPIC_KEY,
|
|
"anthropic-version": "2023-06-01",
|
|
"Content-Type": "application/json",
|
|
},
|
|
json={
|
|
"model": "claude-sonnet-4-5-20250929",
|
|
"max_tokens": max_tokens,
|
|
"messages": messages,
|
|
},
|
|
timeout=30,
|
|
)
|
|
return resp.json()
|
|
|
|
|
|
class TestSummaryHelpfulness:
|
|
"""Check that the code summary carries information the LLM can use."""
|
|
|
|
def test_code_summary_helps_identify_functions(self):
|
|
"""LLM can identify which functions were removed from compressed code."""
|
|
compressed_code = '''
|
|
class PaymentProcessor:
|
|
"""Processes payments via Stripe."""
|
|
|
|
def __init__(self, api_key: str):
|
|
# [2 lines omitted]
|
|
pass
|
|
|
|
def charge(self, amount: float, currency: str, token: str) -> dict:
|
|
# [8 lines omitted]
|
|
pass
|
|
|
|
def refund(self, charge_id: str, amount: float = None) -> dict:
|
|
# [3 lines omitted]
|
|
pass
|
|
|
|
def get_balance(self) -> float:
|
|
# [2 lines omitted]
|
|
pass
|
|
'''
|
|
from headroom.transforms.compression_summary import summarize_compressed_code
|
|
|
|
# Use AST-based summary (language-agnostic)
|
|
bodies = [
|
|
("def charge(self, amount: float, currency: str, token: str) -> dict:", "...", 10),
|
|
("def refund(self, charge_id: str, amount: float = None) -> dict:", "...", 20),
|
|
("def get_balance(self) -> float:", "...", 30),
|
|
]
|
|
code_summary = summarize_compressed_code(bodies, 3)
|
|
|
|
prompt = f"Here is a compressed Python file:\n\n```python\n{compressed_code}\n```\n\n"
|
|
if code_summary:
|
|
prompt += f"[Compression info: {code_summary}]\n\n"
|
|
prompt += "I need to understand the retry logic. Which function should I look at? Answer in one sentence."
|
|
|
|
messages = [{"role": "user", "content": prompt}]
|
|
resp = _call_claude(messages, max_tokens=100)
|
|
text = resp.get("content", [{}])[0].get("text", "").lower()
|
|
|
|
print(f"\n Code summary: {code_summary}")
|
|
print(f" LLM response: {text[:200]}")
|
|
|
|
# The LLM should identify the charge() function
|
|
assert "charge" in text, f"LLM didn't identify charge() function. Response: {text}"
|