mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:06:14 +08:00
feat/https-compose-caddy
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
e4a1ec19be |
refactor(transforms): deprecate unused compress_with_stats, extract_batch and align_for_cache (#4098)
## Description
`DiffCompressor.compress_with_stats`, `HTMLExtractor.extract_batch` and
`cache_aligner.align_for_cache` are public. The first two are methods on
classes exported from `headroom.transforms`, and the third's docstring
called it a stable public API. None is called anywhere in Headroom, the
plugins or the consumer repos; only tests call them. Each now emits a
`DeprecationWarning` that names the replacement.
Refs #4136
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- `DiffCompressor.compress_with_stats` warns and points to
`DiffCompressor.compress`. Unlike `compress()`, it never wrote its CCR
entry to the compression store, so a retrieval marker in its output
could dangle. The docstring now says so.
- `HTMLExtractor.extract_batch` warns and points to calling
`HTMLExtractor.extract` per page.
- `align_for_cache` warns and points to
`CacheAligner(config).apply(messages, tokenizer)` and the
`stable_prefix_hash:` marker.
- The existing tests now assert the warnings. A new test checks that
`align_for_cache` warns at the caller and still returns the messages and
the hash.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_cache_aligner_detector_only.py tests/test_cache_aligner_prefix_stability.py tests/test_transforms/test_html_extractor.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/test_transforms/test_diff_compressor.py
118 passed in 3.87s
$ ruff check / format: All checks passed; mypy: Success
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) `python -W always -c 'align_for_cache(...);
DiffCompressor().compress_with_stats(...)'`.
- Observed result: The proxy serves both requests (200) with the same
savings as main. Each deprecated call emits one DeprecationWarning
attributed to the caller and still returns its result. Output below.
- Not tested: Third-party callers; none is known.
```text
branch: cleanup/transforms-deprecate-unused-methods @
|
||
|
|
9ff67fbd1b |
fix(tests): skip HTML extractor tests when trafilatura not installed
Add pytest.importorskip("trafilatura") to HTML extractor test modules
to skip tests gracefully when the optional trafilatura dependency is
not installed. This fixes CI failures in the base test matrix that
doesn't include the html extras.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
||
|
|
d1a28322cc |
Add HTMLExtractor for web content extraction with OSS benchmarks
HTMLExtractor uses trafilatura to extract main content from HTML pages, removing scripts, styles, navigation, and ads. This achieves 94.9% compression while preserving 98.2% recall on the Scrapinghub benchmark. Key features: - Automatic HTML detection in content router - Configurable output format (markdown or text) - Metadata extraction (title, author, date, description) - Batch extraction support Evaluation framework: - OSS benchmark integration (Scrapinghub Article Extraction Benchmark) - LLM-as-judge evaluation for QA accuracy preservation - F1 score: 0.919 on 181-sample benchmark (baseline: 0.958) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |