mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
refactor(transforms): remove unused convenience wrappers and helpers (#4097)
## Description
This PR removes small helpers that nothing calls. None is in an
`__all__`, a lazy export map or the published docs, and none is imported
by a consumer repo. `align_for_cache` was in an earlier revision of this
PR. Its docstring calls it a stable public API, so it is now kept here
and deprecated in the deprecation PR instead.
Refs #4136
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- `content_router.route_and_compress`, `code_compressor.compress_code`,
`pipeline.create_pipeline` and `base.split_frozen`: one-off wrappers
with no callers.
- `image.trained_router.get_trained_router`: no callers.
- `content_detector.is_json_array_of_dicts`: test-only. The tests now
assert the `is_dict_array` metadata directly. The Rust
`_core.is_json_array_of_dicts` stays.
- `kompress_compressor._acquire_bounded`: test-only, superseded by the
live acquire path. Its two tests go too.
- `ContentRouter._freeze_pin_chars`: a write-only counter.
- Evidence: `rg -nw '<name>' headroom crates plugins sdk examples
benchmarks scripts e2e docs wiki`, plus the consumer repos, finds only
each definition (and, for the test-only items, the tests named above).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_transforms_content_detection.py tests/test_kompress_failsafe.py tests/test_transforms/test_content_router.py tests/test_transforms_content_router.py tests/test_transforms/test_code_compressor.py tests/test_image_compressor.py tests/test_transforms_package.py tests/test_cache_aligner_prefix_stability.py tests/test_cache_aligner_detector_only.py tests/test_pipeline.py tests/test_canonical_pipeline.py tests/test_transforms/test_pipeline_waste_signal_limit.py tests/test_pipeline_outcome_stage.py
405 passed in 19.78s
$ ruff check / format: All checks passed; mypy (4 modules): Success
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) the content-detector metadata check, `hasattr` checks
for the removed wrappers, and an import of the kept `align_for_cache`.
- Observed result: The proxy serves both requests (200) with the same
savings as main. The detector reports `is_dict_array`, the removed
wrappers are gone, and `align_for_cache` is still importable. Output
below.
- Not tested: Nothing beyond the listed files.
```text
branch: cleanup/transforms-remove-dead-helpers @ 5fa12a0c
health: 200
POST /v1/messages -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-51054.log):
2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```
```text
$ python -c "<detect_content_type on a dict array; hasattr checks for the removed wrappers; import align_for_cache>"
ContentType.JSON_ARRAY {'item_count': 2, 'is_dict_array': True}
[False, False, False]
align_for_cache kept: True
```
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: No.
- Kill switch / disable path: Not applicable; nothing that runs changes.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
Not applicable.
## Additional Notes
Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.
Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.
`ContentRouterConfig.min_section_tokens` (never read, but a public
dataclass field) is left for the maintainer, as asked.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
a20c76c26b
commit
5c72d824f4
@@ -359,13 +359,3 @@ class TrainedRouter:
|
||||
query_prediction=technique.value,
|
||||
query_confidence=query_confidence,
|
||||
)
|
||||
|
||||
|
||||
def get_trained_router(model_path: str | None = None) -> TrainedRouter:
|
||||
"""Get a trained router instance.
|
||||
|
||||
Args:
|
||||
model_path: Optional path to model (local or HF hub).
|
||||
If None, uses local model if available, else HF hub.
|
||||
"""
|
||||
return TrainedRouter(model_path=model_path)
|
||||
|
||||
@@ -9,24 +9,6 @@ from ..config import TransformResult
|
||||
from ..tokenizer import Tokenizer
|
||||
|
||||
|
||||
def split_frozen(
|
||||
messages: list[dict[str, Any]],
|
||||
frozen_message_count: int,
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
"""Split messages into frozen (cached prefix) and mutable portions.
|
||||
|
||||
Args:
|
||||
messages: All messages.
|
||||
frozen_message_count: Number of leading messages to freeze.
|
||||
|
||||
Returns:
|
||||
(frozen, mutable) — frozen messages must not be modified.
|
||||
"""
|
||||
if frozen_message_count <= 0 or frozen_message_count >= len(messages):
|
||||
return [], messages
|
||||
return messages[:frozen_message_count], messages[frozen_message_count:]
|
||||
|
||||
|
||||
class Transform(ABC):
|
||||
"""Abstract base class for message transforms."""
|
||||
|
||||
|
||||
@@ -2686,33 +2686,3 @@ def _has_syntax_issues(node: Any) -> bool:
|
||||
if _has_syntax_issues(child):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def compress_code(
|
||||
code: str,
|
||||
language: str | None = None,
|
||||
target_rate: float = 0.2,
|
||||
context: str = "",
|
||||
) -> str:
|
||||
"""Convenience function for one-off code compression.
|
||||
|
||||
Args:
|
||||
code: Source code to compress.
|
||||
language: Language hint (auto-detected if None).
|
||||
target_rate: Target compression rate (0.2 = keep 20%).
|
||||
context: Optional context for relevance.
|
||||
|
||||
Returns:
|
||||
Compressed code string.
|
||||
|
||||
Example:
|
||||
>>> compressed = compress_code(large_python_file)
|
||||
>>> print(compressed) # Valid Python code
|
||||
"""
|
||||
config = CodeCompressorConfig(
|
||||
target_compression_rate=target_rate,
|
||||
language_hint=language,
|
||||
)
|
||||
compressor = CodeAwareCompressor(config)
|
||||
result = compressor.compress(code, language=language, context=context)
|
||||
return result.compressed
|
||||
|
||||
@@ -1009,20 +1009,3 @@ def _try_detect_code(content: str) -> DetectionResult | None:
|
||||
confidence,
|
||||
{"language": best_lang, "pattern_matches": best_score},
|
||||
)
|
||||
|
||||
|
||||
def is_json_array_of_dicts(content: str) -> bool:
|
||||
"""Quick check if content is a JSON array of dictionaries.
|
||||
|
||||
This is the format SmartCrusher can handle natively.
|
||||
|
||||
Args:
|
||||
content: The content to check.
|
||||
|
||||
Returns:
|
||||
True if content is a JSON array where all items are dicts.
|
||||
"""
|
||||
result = detect_content_type(content)
|
||||
return result.content_type == ContentType.JSON_ARRAY and result.metadata.get(
|
||||
"is_dict_array", False
|
||||
)
|
||||
|
||||
@@ -2489,7 +2489,6 @@ class ContentRouter(Transform):
|
||||
# skip — i.e. revert it to original text and bust the provider prefix
|
||||
# cache. Counting pins isolates the freeze's attributable payoff.
|
||||
self._freeze_pin_hits = 0
|
||||
self._freeze_pin_chars = 0
|
||||
|
||||
# ── #3486: per-request runtime state accessors ──────────────────────
|
||||
#
|
||||
@@ -2624,7 +2623,6 @@ class ContentRouter(Transform):
|
||||
preserved = max(0, int(len(content) * (1.0 - cached_ratio)))
|
||||
with self._frozen_lock:
|
||||
self._freeze_pin_hits += 1
|
||||
self._freeze_pin_chars += preserved
|
||||
hits = self._freeze_pin_hits
|
||||
logger.info(
|
||||
f"FREEZE-PIN: pins={hits} cached_ratio={cached_ratio:.3f} "
|
||||
@@ -7885,24 +7883,3 @@ class ContentRouter(Transform):
|
||||
Always returns True - the router handles all content types.
|
||||
"""
|
||||
return True
|
||||
|
||||
|
||||
def route_and_compress(
|
||||
content: str,
|
||||
context: str = "",
|
||||
) -> str:
|
||||
"""Convenience function for one-off routing and compression.
|
||||
|
||||
Args:
|
||||
content: Content to compress.
|
||||
context: Optional context for relevance-aware compression.
|
||||
|
||||
Returns:
|
||||
Compressed content.
|
||||
|
||||
Example:
|
||||
>>> compressed = route_and_compress(mixed_content)
|
||||
"""
|
||||
router = ContentRouter()
|
||||
result = router.compress(content, context=context)
|
||||
return result.compressed
|
||||
|
||||
@@ -549,25 +549,6 @@ def _execution_semaphore(backend: str, device_type: str) -> threading.BoundedSem
|
||||
return semaphore
|
||||
|
||||
|
||||
def _acquire_bounded(
|
||||
semaphore: threading.BoundedSemaphore,
|
||||
acquire_timeout: float | None,
|
||||
remaining_budget: float | None,
|
||||
) -> bool:
|
||||
"""Acquire ``semaphore``, waiting at most the tighter of the two bounds.
|
||||
|
||||
Returns False on timeout. An unbounded wait here is how one stuck
|
||||
inference (e.g. degraded ONNX on Windows) used to wedge every other
|
||||
compression in the process: abandoned executor threads queued forever
|
||||
on the semaphore. With both bounds disabled (<=0) this degrades to the
|
||||
legacy blocking acquire.
|
||||
"""
|
||||
bounds = [b for b in (acquire_timeout, remaining_budget) if b is not None]
|
||||
if not bounds:
|
||||
return semaphore.acquire()
|
||||
return semaphore.acquire(timeout=max(0.0, min(bounds)))
|
||||
|
||||
|
||||
def _batch_size() -> int:
|
||||
return _env_int(KOMPRESS_BATCH_SIZE_ENV) or 32
|
||||
|
||||
|
||||
@@ -12,7 +12,6 @@ from contextlib import nullcontext
|
||||
from typing import TYPE_CHECKING, Any, TypeVar
|
||||
|
||||
from ..config import (
|
||||
CacheAlignerConfig,
|
||||
DiffArtifact,
|
||||
HeadroomConfig,
|
||||
TransformDiff,
|
||||
@@ -597,23 +596,3 @@ class TransformPipeline:
|
||||
"""
|
||||
# apply() already works on a copy, so this is safe
|
||||
return self.apply(messages, model, record_metrics=False, **kwargs)
|
||||
|
||||
|
||||
def create_pipeline(
|
||||
cache_aligner_config: CacheAlignerConfig | None = None,
|
||||
) -> TransformPipeline:
|
||||
"""
|
||||
Create a pipeline with specific configurations.
|
||||
|
||||
Args:
|
||||
cache_aligner_config: Cache aligner configuration.
|
||||
|
||||
Returns:
|
||||
Configured TransformPipeline.
|
||||
"""
|
||||
config = HeadroomConfig()
|
||||
|
||||
if cache_aligner_config is not None:
|
||||
config.cache_aligner = cache_aligner_config
|
||||
|
||||
return TransformPipeline(config)
|
||||
|
||||
@@ -369,21 +369,6 @@ def test_carried_deadline_reaches_sequential_fallback(monkeypatch):
|
||||
assert model.calls == 1
|
||||
|
||||
|
||||
def test_acquire_bounded_unbounded_when_both_disabled():
|
||||
semaphore = kc._execution_semaphore("onnx", "onnx")
|
||||
assert kc._acquire_bounded(semaphore, None, None) is True
|
||||
semaphore.release()
|
||||
|
||||
|
||||
def test_acquire_bounded_negative_remaining_does_not_raise():
|
||||
semaphore = kc._execution_semaphore("onnx", "onnx")
|
||||
assert semaphore.acquire(timeout=0)
|
||||
try:
|
||||
assert kc._acquire_bounded(semaphore, 5.0, -1.0) is False
|
||||
finally:
|
||||
semaphore.release()
|
||||
|
||||
|
||||
# ── Wall-clock budget: give up before the proxy's stage timeout ───────
|
||||
|
||||
|
||||
|
||||
@@ -13,7 +13,6 @@ from headroom.transforms.content_detector import (
|
||||
_try_detect_log,
|
||||
_try_detect_search,
|
||||
detect_content_type,
|
||||
is_json_array_of_dicts,
|
||||
normalize_concatenated_json,
|
||||
)
|
||||
from headroom.transforms.error_detection import (
|
||||
@@ -65,8 +64,6 @@ def test_json_detection_distinguishes_dict_arrays_and_other_lists() -> None:
|
||||
assert object_result.metadata == {"is_dict_array": False, "is_object": True}
|
||||
|
||||
assert _try_detect_json("[not valid json") is None
|
||||
assert is_json_array_of_dicts('[{"id": 1}]') is True
|
||||
assert is_json_array_of_dicts('["value"]') is False
|
||||
|
||||
|
||||
def test_space_separated_json_objects_detected_as_array() -> None:
|
||||
@@ -81,8 +78,9 @@ def test_space_separated_json_objects_detected_as_array() -> None:
|
||||
assert result.metadata == {"item_count": 3, "is_dict_array": True, "concatenated": True}
|
||||
|
||||
# Reaches the same verdict through the top-level detector (not PLAIN_TEXT).
|
||||
assert detect_content_type(content).content_type is ContentType.JSON_ARRAY
|
||||
assert is_json_array_of_dicts(content) is True
|
||||
top_level = detect_content_type(content)
|
||||
assert top_level.content_type is ContentType.JSON_ARRAY
|
||||
assert top_level.metadata.get("is_dict_array") is True
|
||||
|
||||
# Newline separation is just as common and must also be recognized.
|
||||
newline_sep = "\n".join(json.dumps({"id": i, "snippet": "x"}) for i in range(2))
|
||||
|
||||
Reference in New Issue
Block a user