Commit Graph
3 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5.5 e4a1ec19be refactor(transforms): deprecate unused compress_with_stats, extract_batch and align_for_cache (#4098)
## Description

`DiffCompressor.compress_with_stats`, `HTMLExtractor.extract_batch` and
`cache_aligner.align_for_cache` are public. The first two are methods on
classes exported from `headroom.transforms`, and the third's docstring
called it a stable public API. None is called anywhere in Headroom, the
plugins or the consumer repos; only tests call them. Each now emits a
`DeprecationWarning` that names the replacement.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `DiffCompressor.compress_with_stats` warns and points to
`DiffCompressor.compress`. Unlike `compress()`, it never wrote its CCR
entry to the compression store, so a retrieval marker in its output
could dangle. The docstring now says so.
- `HTMLExtractor.extract_batch` warns and points to calling
`HTMLExtractor.extract` per page.
- `align_for_cache` warns and points to
`CacheAligner(config).apply(messages, tokenizer)` and the
`stable_prefix_hash:` marker.
- The existing tests now assert the warnings. A new test checks that
`align_for_cache` warns at the caller and still returns the messages and
the hash.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_cache_aligner_detector_only.py tests/test_cache_aligner_prefix_stability.py tests/test_transforms/test_html_extractor.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/test_transforms/test_diff_compressor.py
118 passed in 3.87s
$ ruff check / format: All checks passed;  mypy: Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) `python -W always -c 'align_for_cache(...);
DiffCompressor().compress_with_stats(...)'`.
- Observed result: The proxy serves both requests (200) with the same
savings as main. Each deprecated call emits one DeprecationWarning
attributed to the caller and still returns its result. Output below.
- Not tested: Third-party callers; none is known.

```text
branch: cleanup/transforms-deprecate-unused-methods @ a3e57303
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-51205.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ python -W always -c "align_for_cache([...]); DiffCompressor().compress_with_stats(...)"
<string>:4: DeprecationWarning: align_for_cache is deprecated and will be removed in a future release; use CacheAligner(config).apply(messages, tokenizer) and read the stable_prefix_hash: marker from markers_inserted.
<string>:5: DeprecationWarning: DiffCompressor.compress_with_stats is deprecated and will be removed in a future release; use DiffCompressor.compress instead.
ok
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: Only a DeprecationWarning when the
deprecated methods are called.
- Kill switch / disable path: Not applicable; the warning is a standard
DeprecationWarning and can be filtered with `warnings`.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.

Deprecated (not removed): `DiffCompressor.compress_with_stats`,
`HTMLExtractor.extract_batch`, `cache_aligner.align_for_cache`. The
warnings fire on every call, which is standard. Python's default warning
filters show each one once per call site.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 09:03:03 -07:00
Claude Code BotandClaude Opus 4.5 9ff67fbd1b fix(tests): skip HTML extractor tests when trafilatura not installed
Add pytest.importorskip("trafilatura") to HTML extractor test modules
to skip tests gracefully when the optional trafilatura dependency is
not installed. This fixes CI failures in the base test matrix that
doesn't include the html extras.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 15:39:55 -08:00
chopratejasandClaude Opus 4.5 d1a28322cc Add HTMLExtractor for web content extraction with OSS benchmarks
HTMLExtractor uses trafilatura to extract main content from HTML pages,
removing scripts, styles, navigation, and ads. This achieves 94.9%
compression while preserving 98.2% recall on the Scrapinghub benchmark.

Key features:
- Automatic HTML detection in content router
- Configurable output format (markdown or text)
- Metadata extraction (title, author, date, description)
- Batch extraction support

Evaluation framework:
- OSS benchmark integration (Scrapinghub Article Extraction Benchmark)
- LLM-as-judge evaluation for QA accuracy preservation
- F1 score: 0.919 on 181-sample benchmark (baseline: 0.958)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 10:08:29 -08:00