mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
`DiffCompressor.compress_with_stats`, `HTMLExtractor.extract_batch` and
`cache_aligner.align_for_cache` are public. The first two are methods on
classes exported from `headroom.transforms`, and the third's docstring
called it a stable public API. None is called anywhere in Headroom, the
plugins or the consumer repos; only tests call them. Each now emits a
`DeprecationWarning` that names the replacement.
Refs #4136
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- `DiffCompressor.compress_with_stats` warns and points to
`DiffCompressor.compress`. Unlike `compress()`, it never wrote its CCR
entry to the compression store, so a retrieval marker in its output
could dangle. The docstring now says so.
- `HTMLExtractor.extract_batch` warns and points to calling
`HTMLExtractor.extract` per page.
- `align_for_cache` warns and points to
`CacheAligner(config).apply(messages, tokenizer)` and the
`stable_prefix_hash:` marker.
- The existing tests now assert the warnings. A new test checks that
`align_for_cache` warns at the caller and still returns the messages and
the hash.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_cache_aligner_detector_only.py tests/test_cache_aligner_prefix_stability.py tests/test_transforms/test_html_extractor.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/test_transforms/test_diff_compressor.py
118 passed in 3.87s
$ ruff check / format: All checks passed; mypy: Success
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) `python -W always -c 'align_for_cache(...);
DiffCompressor().compress_with_stats(...)'`.
- Observed result: The proxy serves both requests (200) with the same
savings as main. Each deprecated call emits one DeprecationWarning
attributed to the caller and still returns its result. Output below.
- Not tested: Third-party callers; none is known.
```text
branch: cleanup/transforms-deprecate-unused-methods @ a3e57303
health: 200
POST /v1/messages -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-51205.log):
2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```
```text
$ python -W always -c "align_for_cache([...]); DiffCompressor().compress_with_stats(...)"
<string>:4: DeprecationWarning: align_for_cache is deprecated and will be removed in a future release; use CacheAligner(config).apply(messages, tokenizer) and read the stable_prefix_hash: marker from markers_inserted.
<string>:5: DeprecationWarning: DiffCompressor.compress_with_stats is deprecated and will be removed in a future release; use DiffCompressor.compress instead.
ok
```
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: Only a DeprecationWarning when the
deprecated methods are called.
- Kill switch / disable path: Not applicable; the warning is a standard
DeprecationWarning and can be filtered with `warnings`.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
Not applicable.
## Additional Notes
Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.
Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.
Deprecated (not removed): `DiffCompressor.compress_with_stats`,
`HTMLExtractor.extract_batch`, `cache_aligner.align_for_cache`. The
warnings fire on every call, which is standard. Python's default warning
filters show each one once per call site.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
597 lines
20 KiB
Python
597 lines
20 KiB
Python
"""Tests for HTMLExtractor.
|
|
|
|
These are real tests using actual HTML content - no mocks.
|
|
Tests verify that trafilatura correctly extracts main content
|
|
and removes structural noise.
|
|
"""
|
|
|
|
import pytest
|
|
|
|
# Skip entire module if trafilatura not installed
|
|
pytest.importorskip("trafilatura")
|
|
|
|
from headroom.transforms.html_extractor import (
|
|
HTMLExtractionResult,
|
|
HTMLExtractor,
|
|
HTMLExtractorConfig,
|
|
is_html_content,
|
|
)
|
|
|
|
|
|
class TestIsHtmlContent:
|
|
"""Tests for the is_html_content detection function."""
|
|
|
|
def test_detects_doctype_html(self):
|
|
"""Detects HTML with DOCTYPE declaration."""
|
|
html = "<!DOCTYPE html><html><body>Content</body></html>"
|
|
assert is_html_content(html) is True
|
|
|
|
def test_detects_html_tag(self):
|
|
"""Detects HTML with html tag."""
|
|
html = "<html><head></head><body>Content</body></html>"
|
|
assert is_html_content(html) is True
|
|
|
|
def test_detects_structural_tags(self):
|
|
"""Detects HTML with multiple structural tags (needs doctype or html tag)."""
|
|
# Note: is_html_content requires DOCTYPE or <html> tag, or 3+ structural tags
|
|
# Just structural tags alone may not trigger detection
|
|
html = "<html><div><nav>Menu</nav><article>Content</article><footer>Footer</footer></div></html>"
|
|
assert is_html_content(html) is True
|
|
|
|
def test_rejects_plain_text(self):
|
|
"""Rejects plain text."""
|
|
text = "This is just plain text with no HTML."
|
|
assert is_html_content(text) is False
|
|
|
|
def test_rejects_json(self):
|
|
"""Rejects JSON content."""
|
|
json_content = '{"name": "test", "value": 123}'
|
|
assert is_html_content(json_content) is False
|
|
|
|
def test_rejects_markdown(self):
|
|
"""Rejects markdown content."""
|
|
markdown = "# Heading\n\nParagraph with **bold** text."
|
|
assert is_html_content(markdown) is False
|
|
|
|
def test_rejects_code(self):
|
|
"""Rejects source code."""
|
|
code = "def hello():\n print('world')"
|
|
assert is_html_content(code) is False
|
|
|
|
def test_rejects_empty(self):
|
|
"""Rejects empty content."""
|
|
assert is_html_content("") is False
|
|
assert is_html_content(None) is False # type: ignore
|
|
|
|
def test_case_insensitive(self):
|
|
"""Detection is case insensitive."""
|
|
html = "<!DOCTYPE HTML><HTML><BODY>Content</BODY></HTML>"
|
|
assert is_html_content(html) is True
|
|
|
|
|
|
class TestHTMLExtractor:
|
|
"""Tests for HTMLExtractor content extraction."""
|
|
|
|
@pytest.fixture
|
|
def extractor(self):
|
|
"""Create a default HTMLExtractor."""
|
|
return HTMLExtractor()
|
|
|
|
def test_extracts_article_content(self, extractor):
|
|
"""Extracts main article content."""
|
|
html = """<!DOCTYPE html>
|
|
<html>
|
|
<head><title>Test Article</title></head>
|
|
<body>
|
|
<nav><a href="/">Home</a></nav>
|
|
<article>
|
|
<h1>Article Title</h1>
|
|
<p>This is the main content of the article.</p>
|
|
<p>It contains important information for the reader.</p>
|
|
</article>
|
|
<footer>Copyright 2024</footer>
|
|
</body>
|
|
</html>"""
|
|
|
|
result = extractor.extract(html)
|
|
|
|
assert "Article Title" in result.extracted
|
|
assert "main content" in result.extracted
|
|
assert "important information" in result.extracted
|
|
# Navigation and footer should be removed
|
|
assert "Home" not in result.extracted or "Copyright" not in result.extracted
|
|
|
|
def test_removes_script_tags(self, extractor):
|
|
"""Removes JavaScript content."""
|
|
html = """<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<script>
|
|
function malicious() {
|
|
alert('This should not appear');
|
|
console.log('Script content');
|
|
}
|
|
</script>
|
|
</head>
|
|
<body>
|
|
<article>
|
|
<p>Actual content that matters.</p>
|
|
</article>
|
|
<script>anotherScript();</script>
|
|
</body>
|
|
</html>"""
|
|
|
|
result = extractor.extract(html)
|
|
|
|
assert "malicious" not in result.extracted
|
|
assert "alert" not in result.extracted
|
|
assert "anotherScript" not in result.extracted
|
|
assert "Actual content" in result.extracted
|
|
|
|
def test_removes_style_tags(self, extractor):
|
|
"""Removes CSS content."""
|
|
html = """<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<style>
|
|
body { color: red; }
|
|
.hidden { display: none; }
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<article>
|
|
<p>Real content here.</p>
|
|
</article>
|
|
</body>
|
|
</html>"""
|
|
|
|
result = extractor.extract(html)
|
|
|
|
assert "color: red" not in result.extracted
|
|
assert "display: none" not in result.extracted
|
|
assert "Real content" in result.extracted
|
|
|
|
def test_compression_ratio(self, extractor):
|
|
"""Verifies significant compression ratio."""
|
|
html = """<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<title>Page Title</title>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width">
|
|
<script src="analytics.js"></script>
|
|
<script>
|
|
var config = {tracking: true, debug: false};
|
|
function init() { console.log('initialized'); }
|
|
</script>
|
|
<style>
|
|
* { margin: 0; padding: 0; }
|
|
body { font-family: Arial; }
|
|
nav { background: #333; }
|
|
footer { background: #666; }
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<header>
|
|
<nav>
|
|
<a href="/">Home</a>
|
|
<a href="/about">About</a>
|
|
<a href="/contact">Contact</a>
|
|
</nav>
|
|
</header>
|
|
<main>
|
|
<article>
|
|
<h1>Main Article Heading</h1>
|
|
<p>This is the first paragraph of actual content.</p>
|
|
<p>This is the second paragraph with more details.</p>
|
|
</article>
|
|
</main>
|
|
<aside>
|
|
<h3>Related Links</h3>
|
|
<ul>
|
|
<li><a href="/link1">Link 1</a></li>
|
|
<li><a href="/link2">Link 2</a></li>
|
|
</ul>
|
|
</aside>
|
|
<footer>
|
|
<p>Copyright 2024</p>
|
|
<p>Privacy Policy | Terms of Service</p>
|
|
</footer>
|
|
</body>
|
|
</html>"""
|
|
|
|
result = extractor.extract(html)
|
|
|
|
# Should achieve significant reduction
|
|
assert result.compression_ratio < 0.5 # At least 50% reduction
|
|
assert result.reduction_percent > 50
|
|
# Main content should be preserved
|
|
assert "Main Article Heading" in result.extracted
|
|
assert "first paragraph" in result.extracted
|
|
|
|
def test_extracts_metadata(self, extractor):
|
|
"""Extracts page metadata."""
|
|
html = """<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<title>Page Title for Testing</title>
|
|
<meta name="author" content="John Doe">
|
|
<meta name="description" content="A test page description">
|
|
</head>
|
|
<body>
|
|
<article>
|
|
<h1>Page Title for Testing</h1>
|
|
<p>Content here.</p>
|
|
</article>
|
|
</body>
|
|
</html>"""
|
|
|
|
result = extractor.extract(html)
|
|
|
|
assert result.title == "Page Title for Testing"
|
|
assert result.metadata.get("title") == "Page Title for Testing"
|
|
|
|
def test_handles_empty_html(self, extractor):
|
|
"""Handles empty HTML gracefully."""
|
|
result = extractor.extract("")
|
|
|
|
assert result.extracted == ""
|
|
assert result.original_length == 0
|
|
assert result.extracted_length == 0
|
|
|
|
def test_handles_whitespace_only(self, extractor):
|
|
"""Handles whitespace-only input."""
|
|
result = extractor.extract(" \n\t ")
|
|
|
|
assert result.extracted == ""
|
|
assert result.compression_ratio == 0.0
|
|
|
|
def test_handles_minimal_html(self, extractor):
|
|
"""Handles minimal HTML structure."""
|
|
html = "<p>Just a paragraph.</p>"
|
|
result = extractor.extract(html)
|
|
|
|
# trafilatura may or may not extract minimal content
|
|
# Just verify it doesn't crash
|
|
assert isinstance(result.extracted, str)
|
|
assert result.original_length > 0
|
|
|
|
def test_preserves_paragraphs(self, extractor):
|
|
"""Preserves paragraph structure."""
|
|
html = """<!DOCTYPE html>
|
|
<html>
|
|
<body>
|
|
<article>
|
|
<p>First paragraph.</p>
|
|
<p>Second paragraph.</p>
|
|
<p>Third paragraph.</p>
|
|
</article>
|
|
</body>
|
|
</html>"""
|
|
|
|
result = extractor.extract(html)
|
|
|
|
assert "First paragraph" in result.extracted
|
|
assert "Second paragraph" in result.extracted
|
|
assert "Third paragraph" in result.extracted
|
|
|
|
|
|
class TestHTMLExtractorConfig:
|
|
"""Tests for HTMLExtractor configuration options."""
|
|
|
|
def test_markdown_output_format(self):
|
|
"""Tests markdown output format."""
|
|
config = HTMLExtractorConfig(output_format="markdown")
|
|
extractor = HTMLExtractor(config)
|
|
|
|
html = """<!DOCTYPE html>
|
|
<html>
|
|
<body>
|
|
<article>
|
|
<h1>Heading</h1>
|
|
<p>Paragraph text.</p>
|
|
</article>
|
|
</body>
|
|
</html>"""
|
|
|
|
result = extractor.extract(html)
|
|
# Markdown format should include # for heading
|
|
assert "Heading" in result.extracted
|
|
|
|
def test_text_output_format(self):
|
|
"""Tests plain text output format."""
|
|
config = HTMLExtractorConfig(output_format="txt")
|
|
extractor = HTMLExtractor(config)
|
|
|
|
html = """<!DOCTYPE html>
|
|
<html>
|
|
<body>
|
|
<article>
|
|
<h1>Heading</h1>
|
|
<p>Paragraph text.</p>
|
|
</article>
|
|
</body>
|
|
</html>"""
|
|
|
|
result = extractor.extract(html)
|
|
assert "Heading" in result.extracted
|
|
assert "Paragraph" in result.extracted
|
|
|
|
def test_disable_metadata_extraction(self):
|
|
"""Tests disabling metadata extraction."""
|
|
config = HTMLExtractorConfig(extract_metadata=False)
|
|
extractor = HTMLExtractor(config)
|
|
|
|
html = """<!DOCTYPE html>
|
|
<html>
|
|
<head><title>Test Title</title></head>
|
|
<body><article><p>Content.</p></article></body>
|
|
</html>"""
|
|
|
|
result = extractor.extract(html)
|
|
|
|
# Metadata should be empty when disabled
|
|
assert result.title is None
|
|
assert result.metadata == {}
|
|
|
|
|
|
class TestHTMLExtractionResult:
|
|
"""Tests for HTMLExtractionResult dataclass."""
|
|
|
|
def test_reduction_percent_calculation(self):
|
|
"""Tests reduction percent calculation."""
|
|
result = HTMLExtractionResult(
|
|
extracted="short",
|
|
original="much longer content here",
|
|
original_length=100,
|
|
extracted_length=25,
|
|
compression_ratio=0.25,
|
|
)
|
|
|
|
assert result.reduction_percent == 75.0
|
|
|
|
def test_reduction_percent_with_zero_original(self):
|
|
"""Tests reduction percent with zero original length."""
|
|
result = HTMLExtractionResult(
|
|
extracted="",
|
|
original="",
|
|
original_length=0,
|
|
extracted_length=0,
|
|
compression_ratio=0.0,
|
|
)
|
|
|
|
assert result.reduction_percent == 0.0
|
|
|
|
|
|
class TestRealWorldHTML:
|
|
"""Tests with realistic HTML content."""
|
|
|
|
@pytest.fixture
|
|
def extractor(self):
|
|
return HTMLExtractor()
|
|
|
|
def test_news_article_structure(self, extractor):
|
|
"""Tests extraction from news-article-like structure."""
|
|
html = """<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="UTF-8">
|
|
<title>Breaking News: Important Event Happens</title>
|
|
<script>window.analytics = {};</script>
|
|
<style>.ad { display: block; }</style>
|
|
</head>
|
|
<body>
|
|
<header>
|
|
<nav class="main-nav">
|
|
<a href="/">Home</a>
|
|
<a href="/news">News</a>
|
|
<a href="/sports">Sports</a>
|
|
</nav>
|
|
</header>
|
|
<div class="ad-banner">Advertisement Here</div>
|
|
<main>
|
|
<article class="news-article">
|
|
<h1>Breaking News: Important Event Happens</h1>
|
|
<p class="byline">By Jane Reporter | January 15, 2024</p>
|
|
<p>In a surprising turn of events, something important happened today
|
|
that will affect millions of people around the world.</p>
|
|
<p>Experts say this development represents a major shift in how
|
|
we think about the topic at hand.</p>
|
|
<p>"This is truly unprecedented," said Dr. Expert, a leading
|
|
authority in the field.</p>
|
|
<p>The implications of this event are still being analyzed, but
|
|
early reports suggest significant changes ahead.</p>
|
|
</article>
|
|
</main>
|
|
<aside class="sidebar">
|
|
<h3>Trending Stories</h3>
|
|
<ul>
|
|
<li><a href="/story1">Story 1</a></li>
|
|
<li><a href="/story2">Story 2</a></li>
|
|
</ul>
|
|
</aside>
|
|
<footer>
|
|
<p>© 2024 News Site</p>
|
|
<a href="/privacy">Privacy Policy</a>
|
|
</footer>
|
|
<script>trackPageView();</script>
|
|
</body>
|
|
</html>"""
|
|
|
|
result = extractor.extract(html)
|
|
|
|
# Main article content should be preserved
|
|
assert "Important Event Happens" in result.extracted
|
|
assert "surprising turn of events" in result.extracted
|
|
assert "Dr. Expert" in result.extracted
|
|
|
|
# Noise should be removed or minimized
|
|
assert "trackPageView" not in result.extracted
|
|
assert "window.analytics" not in result.extracted
|
|
|
|
# Significant reduction
|
|
assert result.compression_ratio < 0.4
|
|
|
|
def test_documentation_page(self, extractor):
|
|
"""Tests extraction from documentation-like page."""
|
|
html = """<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<title>API Documentation - MyService</title>
|
|
</head>
|
|
<body>
|
|
<nav class="docs-nav">
|
|
<a href="/docs">Docs</a>
|
|
<a href="/api">API</a>
|
|
<a href="/guides">Guides</a>
|
|
</nav>
|
|
<div class="sidebar">
|
|
<h3>Table of Contents</h3>
|
|
<ul>
|
|
<li><a href="#intro">Introduction</a></li>
|
|
<li><a href="#auth">Authentication</a></li>
|
|
</ul>
|
|
</div>
|
|
<main class="content">
|
|
<h1>API Documentation</h1>
|
|
<section id="intro">
|
|
<h2>Introduction</h2>
|
|
<p>Welcome to the MyService API documentation. This guide will
|
|
help you integrate our service into your application.</p>
|
|
</section>
|
|
<section id="auth">
|
|
<h2>Authentication</h2>
|
|
<p>All API requests require authentication using an API key.
|
|
Include your key in the Authorization header.</p>
|
|
<pre><code>Authorization: Bearer YOUR_API_KEY</code></pre>
|
|
</section>
|
|
</main>
|
|
<footer>Built with Docs Generator</footer>
|
|
</body>
|
|
</html>"""
|
|
|
|
result = extractor.extract(html)
|
|
|
|
# Documentation content should be preserved
|
|
assert "API Documentation" in result.extracted
|
|
assert "Authentication" in result.extracted
|
|
assert "API key" in result.extracted
|
|
|
|
def test_blog_post(self, extractor):
|
|
"""Tests extraction from blog-post-like structure."""
|
|
html = """<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<title>My Blog Post Title - Personal Blog</title>
|
|
<meta name="author" content="Blog Author">
|
|
</head>
|
|
<body>
|
|
<header>
|
|
<h1 class="site-title">My Personal Blog</h1>
|
|
<nav>Home | About | Contact</nav>
|
|
</header>
|
|
<article class="blog-post">
|
|
<h1>My Blog Post Title</h1>
|
|
<time>Posted on March 15, 2024</time>
|
|
<p>Today I want to share my thoughts on an interesting topic
|
|
that I've been thinking about for a while.</p>
|
|
<p>The key insight is that we often overlook the simple things
|
|
in life that bring us joy.</p>
|
|
<p>In conclusion, I believe we should all take more time to
|
|
appreciate the small moments.</p>
|
|
</article>
|
|
<section class="comments">
|
|
<h3>Comments</h3>
|
|
<div class="comment">Great post!</div>
|
|
</section>
|
|
<footer>© 2024 Blog Author</footer>
|
|
</body>
|
|
</html>"""
|
|
|
|
result = extractor.extract(html)
|
|
|
|
# Blog content should be preserved
|
|
assert "Blog Post Title" in result.extracted
|
|
assert "interesting topic" in result.extracted
|
|
assert "small moments" in result.extracted
|
|
|
|
|
|
class TestBatchExtraction:
|
|
"""Tests for batch extraction."""
|
|
|
|
def test_extract_batch(self):
|
|
"""Tests batch extraction of multiple HTML pages."""
|
|
extractor = HTMLExtractor()
|
|
|
|
pages = [
|
|
(
|
|
"<html><body><article><p>Page one content.</p></article></body></html>",
|
|
"http://example.com/page1",
|
|
),
|
|
(
|
|
"<html><body><article><p>Page two content.</p></article></body></html>",
|
|
"http://example.com/page2",
|
|
),
|
|
(
|
|
"<html><body><article><p>Page three content.</p></article></body></html>",
|
|
None,
|
|
),
|
|
]
|
|
|
|
with pytest.warns(DeprecationWarning, match="extract_batch is deprecated"):
|
|
results = extractor.extract_batch(pages)
|
|
|
|
assert len(results) == 3
|
|
assert all(isinstance(r, HTMLExtractionResult) for r in results)
|
|
|
|
|
|
class TestContentDetectorIntegration:
|
|
"""Tests for integration with content_detector."""
|
|
|
|
def test_detector_identifies_html(self):
|
|
"""Tests that content_detector correctly identifies HTML."""
|
|
from headroom.transforms.content_detector import ContentType, detect_content_type
|
|
|
|
html = """<!DOCTYPE html>
|
|
<html>
|
|
<head><title>Test</title></head>
|
|
<body>
|
|
<div><p>Content</p></div>
|
|
</body>
|
|
</html>"""
|
|
|
|
result = detect_content_type(html)
|
|
|
|
assert result.content_type == ContentType.HTML
|
|
assert result.confidence >= 0.7
|
|
|
|
def test_detector_rejects_non_html(self):
|
|
"""Tests that content_detector doesn't misidentify non-HTML."""
|
|
from headroom.transforms.content_detector import ContentType, detect_content_type
|
|
|
|
# Plain text
|
|
result = detect_content_type("Just some plain text without HTML.")
|
|
assert result.content_type != ContentType.HTML
|
|
|
|
# JSON
|
|
result = detect_content_type('[{"id": 1}, {"id": 2}]')
|
|
assert result.content_type == ContentType.JSON_ARRAY
|
|
|
|
# Code (needs enough patterns to trigger detection)
|
|
code = """
|
|
import os
|
|
import sys
|
|
|
|
def hello():
|
|
print('world')
|
|
|
|
class Foo:
|
|
def __init__(self):
|
|
pass
|
|
|
|
if __name__ == '__main__':
|
|
hello()
|
|
"""
|
|
result = detect_content_type(code)
|
|
assert result.content_type == ContentType.SOURCE_CODE
|