Commit Graph
2 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5.5 eaa7308a52 refactor(transforms): remove the unused summarize_dropped_items (#4093)
## Description

`compression_summary.summarize_dropped_items` was SmartCrusher's
dropped-item categoriser. SmartCrusher now runs in Rust and emits no
dropped-item summary (`smart_crusher/crusher.rs`), and nothing in Python
called the function. Only tests did, including two LLM eval files. The
module has no `__all__`, is not re-exported, and the function is not
documented or imported by any consumer repo.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Remove `summarize_dropped_items`, its private helpers (`_item_key`,
`_categorize_by_fields`, `_find_notable_items`, `_common_keys`) and
their constants.
- Fix the module docstring, which still said SmartCrusher used it.
`summarize_compressed_code` (used by CodeCompressor) stays.
- Delete `test_compression_summary_hard_eval.py`,
`test_compression_summary_tool_eval.py` and
`test_compression_summary_eval.py`. The last file's one code-summary
test moves into `test_compression_summary.py`.
- Drop the dropped-item cases from the other two test files.
- Evidence: `rg -n summarize_dropped_items headroom crates plugins sdk
examples benchmarks scripts docs wiki` plus the consumer repos finds
only the definition and a Rust doc comment, now reworded.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression_summary.py tests/test_compression_summary_integration.py tests/test_transforms/test_code_compressor.py
109 passed, 1 skipped in 11.24s
$ ruff check / format: All checks passed;  mypy compression_summary.py: Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) import check: `summarize_compressed_code` still works
and `summarize_dropped_items` is gone.
- Observed result: The proxy serves both requests (200) with the same
savings as main. The live summary helper works, and the removed one
raises ImportError. Output below.
- Not tested: The LLM integration test (it skips without
ANTHROPIC_API_KEY).

```text
branch: cleanup/transforms-remove-summarize-dropped-items @ bc0dc2f6
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-51402.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ python -c "from headroom.transforms.compression_summary import summarize_compressed_code; print(summarize_compressed_code([...2 bodies...], 2)); from headroom.transforms.compression_summary import summarize_dropped_items"
2 bodies compressed: authenticate(), refresh()
ImportError: cannot import name 'summarize_dropped_items' from 'headroom.transforms.compression_summary' (headroom/transforms/compression_summary.py)
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: No. The removed function was not
called at runtime.
- Kill switch / disable path: Not applicable; nothing that runs changes.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 08:58:19 -07:00
chopratejas 729cc035a4 Add compression summaries, multi-provider headers, Dockerfile fix
Compression Summaries:
- New: headroom/transforms/compression_summary.py
  - summarize_dropped_items(): categorizes compressed JSON items by
    field values (status, type, level, etc.), highlights errors/failures
  - summarize_compressed_code(): extracts function names from AST
    signatures (language-agnostic: Python, JS, Go, Rust, Java)
  - Newline-safe: strips \n from field values to keep markers single-line
- SmartCrusher: CCR markers include categorical summary of dropped items
  e.g. "[500 items compressed to 20. Omitted: 87 passed, 2 failed.
  Retrieve more: hash=abc123. Expires in 5m.]"
- CodeCompressor: CCR markers list compressed function names from AST
  e.g. "[180 tokens compressed. 5 bodies compressed: authenticate().
  Retrieve more: hash=abc123. Expires in 5m.]"
- Markers include TTL so LLM knows retrieval window
- Summary escapes { } to prevent .format() crashes
- Uses index-based dropped detection (not id()) for .copy() correctness

Proxy Response Headers:
- Anthropic, OpenAI, and Gemini handlers inject x-headroom-tokens-*
  headers for SaaS metering

Multi-Provider Passthrough Routing:
- Detect x-goog-api-key (Gemini) and api-key (Azure OpenAI)
- X-Headroom-Base-URL for explicit upstream URL override

Dockerfile: add build-essential + g++ for hnswlib compilation
Bump version to 0.3.5

Tests: 27 new tests (unit, eval, integration with real API, tool invocation)
2026-02-18 16:54:20 -08:00