mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
refactor(compression): deprecate the test-only UniversalCompressor package (#4095)
## Description
`headroom.compression` (UniversalCompressor, StructureMask and the
JSON/code structure handlers) is not called by any product code; only
`tests/test_compression/` uses it. The one-function API people use is
`headroom.compress()`, which does not touch it.
`headroom.compression.*` was an undocumented import path. It was never
in the published docs, only in the unpublished `wiki/compression.md`,
and no consumer repo imports it. Because it is an importable package
with an `__all__`, this PR deprecates it instead of deleting it.
Refs #4136
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- The package exports (`compress`, `UniversalCompressor`,
`UniversalCompressorConfig`, `CompressionResult`, `MagikaDetector`,
`ContentType`, `StructureMask`) are now lazy. The first access emits one
`DeprecationWarning` at the caller, pointing to `headroom.compress()` or
`ContentRouter`.
- The resolved object is cached in the package globals, so a `from`
import (which looks the name up twice) warns once, not twice.
- On `importlib.reload()` the package drops its cached exports, so they
resolve against the current submodules again.
- Importing `headroom.compression.universal` directly also warns.
- `import headroom.compression.detector` stays quiet, because
ContentRouter's startup Magika preload imports it.
- `tests/test_compression/test_deprecation.py` covers attribute, `from`
and wildcard imports, reload, the universal import and the quiet
detector import.
- It restores `sys.modules`, the parent-package attribute and the export
cache through monkeypatch, so test order does not matter.
- Three of these tests fail on the previous commit, which warned twice.
- Evidence: `rg -n
'headroom\.compression|UniversalCompressor|StructureMask|MagikaDetector'
headroom crates plugins sdk examples benchmarks scripts`, plus the
consumer repos, finds only the package itself, `content_router.py`'s
detector import and unrelated OTel metric names.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_compression tests/test_kompress_preload_deferral.py tests/test_transforms_content_router.py tests/test_compress_api.py tests/test_proxy_warmup.py
262 passed, 8 skipped, 1 warning in 39.82s
$ ruff check / format: All checks passed; mypy: Success
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) `python -W always -c 'from headroom.compression import
compress; import headroom.compression.detector'`.
- Observed result: The proxy serves both requests (200) with savings and
logs no new warnings. The from-import emits exactly one
DeprecationWarning, attributed to the caller. The detector import is
silent. Output below.
- Not tested: Third-party code using `headroom.compression`; none is
known.
```text
branch: cleanup/compression-deprecate-universal-compressor @ 4a430674
health: 200
POST /v1/messages -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-52286.log):
2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```
```text
$ python -W always -c "from headroom.compression import compress; import headroom.compression.detector; print(\"ok\")"
<string>:2: DeprecationWarning: headroom.compression.compress is deprecated and will be removed in a future release; use headroom.compress() for one-call compression, or ContentRouter from headroom.transforms.
ok
```
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: Only a DeprecationWarning when the
deprecated names are used.
- Kill switch / disable path: Not applicable; the warning is a standard
DeprecationWarning and can be filtered with `warnings`.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
Not applicable.
## Additional Notes
Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.
Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.
Deprecated (not removed): the seven `headroom.compression` package
exports and the module `headroom.compression.universal`. The handlers
and masks modules warn only through the package. In
`tests/test_proxy_debug_endpoints.py`,
`test_debug_endpoints_do_not_mutate_state` failed once in a combined run
and passed on two reruns (29/29). It looks flaky and is unrelated to
this change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
9b42fcc4fc
commit
fc8ff5a761
@@ -1,32 +1,31 @@
|
||||
"""Universal compression with ML-based content detection.
|
||||
|
||||
This module provides intelligent, automatic compression that:
|
||||
1. Detects content type using ML (Magika)
|
||||
2. Preserves structure (keys, signatures, templates)
|
||||
3. Compresses content with Kompress
|
||||
4. Enables retrieval via CCR
|
||||
.. deprecated::
|
||||
The package API (``compress``, ``UniversalCompressor``,
|
||||
``UniversalCompressorConfig``, ``CompressionResult``, ``MagikaDetector``,
|
||||
``ContentType``, ``StructureMask``) is unused inside Headroom and will be
|
||||
removed in a future release. Use ``headroom.compress()`` for one-call
|
||||
compression, or ``ContentRouter`` from ``headroom.transforms``.
|
||||
|
||||
Quick Start:
|
||||
# One-liner for simple use
|
||||
from headroom.compression import compress
|
||||
result = compress(content)
|
||||
|
||||
# Or with configuration
|
||||
from headroom.compression import UniversalCompressor, UniversalCompressorConfig
|
||||
|
||||
config = UniversalCompressorConfig(compression_ratio_target=0.5)
|
||||
compressor = UniversalCompressor(config=config)
|
||||
result = compressor.compress(content)
|
||||
``headroom.compression.detector`` is still used by the proxy (Magika preload
|
||||
at startup) and importing it does not warn.
|
||||
"""
|
||||
|
||||
from headroom.compression.detector import ContentType, MagikaDetector
|
||||
from headroom.compression.masks import StructureMask
|
||||
from headroom.compression.universal import (
|
||||
CompressionResult,
|
||||
UniversalCompressor,
|
||||
UniversalCompressorConfig,
|
||||
compress,
|
||||
)
|
||||
from __future__ import annotations
|
||||
|
||||
import warnings
|
||||
from importlib import import_module
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from headroom.compression.detector import ContentType, MagikaDetector
|
||||
from headroom.compression.masks import StructureMask
|
||||
from headroom.compression.universal import (
|
||||
CompressionResult,
|
||||
UniversalCompressor,
|
||||
UniversalCompressorConfig,
|
||||
compress,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
# Simple API
|
||||
@@ -40,3 +39,45 @@ __all__ = [
|
||||
"ContentType",
|
||||
"StructureMask",
|
||||
]
|
||||
|
||||
_DEPRECATED_EXPORTS: dict[str, tuple[str, str]] = {
|
||||
"compress": ("headroom.compression.universal", "compress"),
|
||||
"UniversalCompressor": ("headroom.compression.universal", "UniversalCompressor"),
|
||||
"UniversalCompressorConfig": ("headroom.compression.universal", "UniversalCompressorConfig"),
|
||||
"CompressionResult": ("headroom.compression.universal", "CompressionResult"),
|
||||
"MagikaDetector": ("headroom.compression.detector", "MagikaDetector"),
|
||||
"ContentType": ("headroom.compression.detector", "ContentType"),
|
||||
"StructureMask": ("headroom.compression.masks", "StructureMask"),
|
||||
}
|
||||
|
||||
# On importlib.reload() the module dict is reused; drop cached exports so the
|
||||
# next access resolves (and warns) against the current submodules.
|
||||
for _name in _DEPRECATED_EXPORTS:
|
||||
globals().pop(_name, None)
|
||||
del _name
|
||||
|
||||
|
||||
def __getattr__(name: str) -> object:
|
||||
try:
|
||||
module_name, attr_name = _DEPRECATED_EXPORTS[name]
|
||||
except KeyError as exc:
|
||||
raise AttributeError(f"module {__name__!r} has no attribute {name!r}") from exc
|
||||
warnings.warn(
|
||||
f"headroom.compression.{name} is deprecated and will be removed in a future release; "
|
||||
"use headroom.compress() for one-call compression, or ContentRouter from headroom.transforms.",
|
||||
DeprecationWarning,
|
||||
stacklevel=2,
|
||||
)
|
||||
# Warn once, here, rather than again from universal.py's import-time warning.
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("ignore", DeprecationWarning)
|
||||
module = import_module(module_name)
|
||||
value = getattr(module, attr_name)
|
||||
# Cache it, so `from headroom.compression import X` (which looks the name
|
||||
# up twice) warns once, and later accesses stay quiet.
|
||||
globals()[name] = value
|
||||
return value
|
||||
|
||||
|
||||
def __dir__() -> list[str]:
|
||||
return sorted(set(globals()) | set(__all__))
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
"""Universal compressor with ML-based detection and structure preservation.
|
||||
|
||||
.. deprecated::
|
||||
Nothing in Headroom uses this module. It will be removed in a future
|
||||
release. Use ``headroom.compress()`` for one-call compression, or
|
||||
``ContentRouter`` from ``headroom.transforms``.
|
||||
|
||||
This is the main entry point for compression. It:
|
||||
1. Detects content type using Magika (ML)
|
||||
2. Extracts structure using appropriate handler
|
||||
@@ -20,6 +25,7 @@ Usage:
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import warnings
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
@@ -44,6 +50,13 @@ from headroom.compression.masks import (
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
warnings.warn(
|
||||
"headroom.compression.universal is deprecated and will be removed in a future release; "
|
||||
"use headroom.compress() for one-call compression, or ContentRouter from headroom.transforms.",
|
||||
DeprecationWarning,
|
||||
stacklevel=2,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class UniversalCompressorConfig:
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
"""The headroom.compression package API is deprecated; detector stays quiet."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import sys
|
||||
import warnings
|
||||
from collections.abc import Iterator
|
||||
|
||||
import pytest
|
||||
|
||||
import headroom.compression as compression
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fresh(monkeypatch: pytest.MonkeyPatch) -> Iterator[None]:
|
||||
"""Drop cached exports and the universal module for the test.
|
||||
|
||||
Exports cached during the test are dropped again afterwards; monkeypatch
|
||||
then restores whatever was cached before.
|
||||
"""
|
||||
for name in compression.__all__:
|
||||
monkeypatch.delitem(compression.__dict__, name, raising=False)
|
||||
monkeypatch.delitem(sys.modules, "headroom.compression.universal", raising=False)
|
||||
monkeypatch.delattr(compression, "universal", raising=False)
|
||||
yield
|
||||
for name in compression.__all__:
|
||||
compression.__dict__.pop(name, None)
|
||||
|
||||
|
||||
@pytest.mark.usefixtures("fresh")
|
||||
def test_attribute_access_warns_once() -> None:
|
||||
with pytest.warns(
|
||||
DeprecationWarning, match="compression.UniversalCompressor is deprecated"
|
||||
) as record:
|
||||
compressor_cls = compression.UniversalCompressor
|
||||
assert compression.UniversalCompressor is compressor_cls # cached: no second warning
|
||||
assert compressor_cls.__name__ == "UniversalCompressor"
|
||||
# One warning, attributed to this file rather than to importlib.
|
||||
assert len(record) == 1
|
||||
assert record[0].filename == __file__
|
||||
|
||||
|
||||
@pytest.mark.usefixtures("fresh")
|
||||
def test_from_import_warns_once() -> None:
|
||||
with warnings.catch_warnings(record=True) as record:
|
||||
warnings.simplefilter("always")
|
||||
from headroom.compression import ContentType # noqa: F401
|
||||
deprecations = [w for w in record if issubclass(w.category, DeprecationWarning)]
|
||||
assert len(deprecations) == 1
|
||||
assert "use headroom.compress" in str(deprecations[0].message)
|
||||
|
||||
|
||||
@pytest.mark.usefixtures("fresh")
|
||||
def test_wildcard_import_warns_once_per_name() -> None:
|
||||
namespace: dict[str, object] = {}
|
||||
with warnings.catch_warnings(record=True) as record:
|
||||
warnings.simplefilter("always")
|
||||
exec("from headroom.compression import *", namespace)
|
||||
messages = [str(w.message) for w in record if issubclass(w.category, DeprecationWarning)]
|
||||
assert len(messages) == len(compression.__all__)
|
||||
assert set(compression.__all__) <= set(namespace)
|
||||
|
||||
|
||||
@pytest.mark.usefixtures("fresh")
|
||||
def test_universal_module_import_warns() -> None:
|
||||
with pytest.warns(DeprecationWarning, match="compression.universal is deprecated"):
|
||||
importlib.import_module("headroom.compression.universal")
|
||||
|
||||
|
||||
def test_detector_import_stays_quiet(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.delitem(sys.modules, "headroom.compression.detector", raising=False)
|
||||
monkeypatch.delattr(compression, "detector", raising=False)
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("error", DeprecationWarning)
|
||||
detector = importlib.import_module("headroom.compression.detector")
|
||||
assert callable(detector._magika_available)
|
||||
|
||||
|
||||
@pytest.mark.usefixtures("fresh")
|
||||
def test_reload_drops_cached_exports() -> None:
|
||||
import headroom.compression.detector as detector
|
||||
|
||||
real = detector.ContentType
|
||||
with pytest.warns(DeprecationWarning):
|
||||
assert compression.ContentType is real
|
||||
stand_in = type("ReloadedContentType", (), {})
|
||||
with pytest.MonkeyPatch.context() as patch:
|
||||
patch.setattr(detector, "ContentType", stand_in)
|
||||
importlib.reload(compression)
|
||||
with pytest.warns(DeprecationWarning):
|
||||
assert compression.ContentType is stand_in
|
||||
# Back on the real class: a second reload must drop the cached stand-in.
|
||||
importlib.reload(compression)
|
||||
with pytest.warns(DeprecationWarning):
|
||||
assert compression.ContentType is real
|
||||
Reference in New Issue
Block a user