mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description `headroom proxy --compressor` / `HEADROOM_COMPRESSORS` lists `image` and says unselected built-ins are disabled. The selection only sets `ContentRouterConfig.enable_image_optimizer`, which the proxy does not consult. The Anthropic and OpenAI handlers gate their image stage on `ProxyConfig.image_optimize`, and nothing set it from a selection. So `--compressor smart_crusher` still sent every screenshot through the image stage. This PR makes the image gate honor the selection. `ImageCompressionDecision`, which both handlers already call, reads `config.compressors` on each request and skips the image stage with a new reason, `image_not_selected`, when a selection leaves out `image`. The operator's `image_optimize` is never written, and `image_optimize=False` still wins. This is Gap 1 from #4613, agreed with the reporter there. Gap 2 (Windows self-update while a proxy is running) will be a separate PR. Refs #4613 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/image_compression_policy.py`: - `image_compressor_selected(compressors)` follows the same rules as `_apply_compressor_selection`: no selection, an empty one or `*` keeps images; otherwise `image` must be named. - `decide_image_compression` takes `image_selected` (default `True`) and adds `image_not_selected`, which comes after `image_optimize_disabled` and before `no_messages`. - `headroom/proxy/image_compression_decision.py`: `decide` passes the selection from `config.compressors` and exposes `image_selected` next to the other inputs it saw. The new reason appears in `image_skip_reason`. - `tests/test_compressor_selection_image_stage.py`: sends a screenshot request through the real proxy app with only the upstream faked. It runs on two routes: Anthropic in token mode, and OpenAI chat in cache mode, which still compresses images in cache mode. The cases are: - a selection without `image` never enters the image stage; - a selection with `image`, `*`, and no selection each enter it once; - a config reused across two proxies follows the current selection; - `image_optimize=False` with `image` selected still skips the stage. - `tests/test_image_compression_decision.py`: covers the new reason, its precedence, the wildcard and blank selections, and a config with no `compressors` attribute. - `docs/content/docs/proxy.mdx`: the `--compressor` row says that leaving out `image` also turns off the image stage. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`): see Additional Notes - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output The new tests against `main`'s source (b21479e0): ```text FAILED tests/test_compressor_selection_image_stage.py::TestImageStageFollowsSelection::test_selection_without_image_skips_the_image_stage[anthropic-token] FAILED tests/test_compressor_selection_image_stage.py::TestImageStageFollowsSelection::test_selection_without_image_skips_the_image_stage[openai-cache] FAILED tests/test_compressor_selection_image_stage.py::TestImageStageFollowsSelection::test_reused_config_follows_the_current_selection[anthropic-token] FAILED tests/test_compressor_selection_image_stage.py::TestImageStageFollowsSelection::test_reused_config_follows_the_current_selection[openai-cache] 4 failed, 8 passed ``` At this head: ```text $ pytest tests/test_compressor_selection.py tests/test_compressor_selection_image_stage.py tests/test_image_compression_decision.py tests/test_image_compression_policy.py tests/test_router_external_dispatch.py tests/test_proxy_eager_preload_bind.py -q 99 passed $ pytest <every test file that mentions image_skip_reason, ImageCompressionDecision, image_optimize or compressors> -m "not live" -q 12 failed, 1994 passed, 26 skipped $ ruff check <changed files> && ruff format --check <changed files> All checks passed! ``` The 12 failures are in `tests/test_image_compressor.py`, `tests/test_ccr_batch_processor.py`, `tests/test_garbled_compression_fixes.py` and `tests/test_transforms/test_content_router.py`. The same 12 fail on `main` in my environment (the image ones because the venv has Pillow 12), so they are not from this change. ## Real Behavior Proof - Environment: macOS (Darwin 24.6.0, arm64), Python 3.12.3, Pillow 13.0.0. Older Pillow makes the proxy forward images unchanged. - Exact command / steps: the real proxy app in token mode on the Anthropic route, with the real image compressor and image worker; only the upstream model is faked, and it answers instantly. Each setting sends the same 2560x1600 PNG screenshot ("what is the exact error text in this screenshot?") 5 times. Per request, the script records image-stage entries, what reached the upstream, and the time the proxy took. Run once on `main` (b21479e0) and once on this branch. - Observed result: on `main`, `--compressor smart_crusher` still ran the image stage on 5 of 5 requests and downscaled every screenshot (`full_low`, 1,870 → 255 tokens) at about 45 ms each. With this PR it ran on 0 of 5, the original image went upstream unchanged, and each request took 4.6 ms. With no selection or with `image` selected, both builds behave the same. `image_optimize` stays `True` in every run. | Build | Setting | Image-stage runs | Image sent upstream | Proxy time p50 | |---|---|---|---|---| | `main` | no `--compressor` | 5 of 5 | downscaled (`full_low`, 1,870 → 255 tokens) | 45.4 ms | | `main` | `--compressor smart_crusher` | **5 of 5** | **downscaled** | **45.2 ms** | | `main` | `--compressor smart_crusher,image` | 5 of 5 | downscaled | 46.9 ms | | this PR | no `--compressor` | 5 of 5 | downscaled | 48.2 ms | | this PR | `--compressor smart_crusher` | **0 of 5** | **original, unchanged** | **4.6 ms** | | this PR | `--compressor smart_crusher,image` | 5 of 5 | downscaled | 47.6 ms | The first image request of each process also loads the image models (about 1.0–1.3 s here). With `image` left out, that load doesn't happen. - Not tested: a live upstream, the OCR (`transcode`) technique (the router picked `full_low` for this screenshot), and Windows. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: only for a `--compressor` / `HEADROOM_COMPRESSORS` selection that leaves out `image`. Those proxies now forward images unchanged, as the help text already says. With no selection, a selection that includes `image`, or `*`, nothing changes. - Kill switch / disable path: add `image` to the selection, or remove the selection - Unsafe override required: no - Qualification impact: none - Rollback path: revert the PR's commits ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) n/a ## Additional Notes - Review follow-up: the first revision (619ea4ad) wrote `image_optimize=False` into the caller's `ProxyConfig`. Now the decision reads the selection on each request, so a reused config follows its current selection, and an explicit `image_optimize=False` is kept. - #1327 proposes a separate `HEADROOM_DISABLE_IMAGE_OPTIMIZER` variable. This PR doesn't add one; it makes the existing option behave as its help text says. - `mypy headroom/proxy/server.py` stops in my local environment on a `numpy` stub (`Type statement is only supported in Python 3.12 and greater`) before reaching project code. CI runs mypy.
290 lines
11 KiB
Python
290 lines
11 KiB
Python
"""Tests for :class:`headroom.proxy.image_compression_decision.ImageCompressionDecision`.
|
|
|
|
Image compression today is gated at two sites (``openai.py:1203`` +
|
|
``anthropic.py:868``) by inline conjunctions. Both already check
|
|
``_bypass`` (the drift problem CompressionDecision fixed is NOT
|
|
present here), but consolidating into a value type still pays off:
|
|
|
|
* test-lockable contract (no future site can forget bypass)
|
|
* ``apply_to_tags()`` surfaces ``image_skip_reason`` in
|
|
``RequestOutcome.tags`` — dashboards can slice image-skipped
|
|
traffic by cause (same observability surface as
|
|
``passthrough_reason`` and ``memory_skip_reason``)
|
|
* Rust-portable shape mirrors :class:`CompressionDecision` exactly
|
|
|
|
Precedence (highest first):
|
|
1. ``bypass_header`` — user opt-out
|
|
2. ``image_optimize_disabled`` — operator config off
|
|
3. ``no_messages`` — nothing to inspect
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from dataclasses import FrozenInstanceError
|
|
from types import SimpleNamespace
|
|
from typing import Any
|
|
|
|
from headroom.proxy.image_compression_decision import ImageCompressionDecision
|
|
|
|
# ── Helpers ───────────────────────────────────────────────────────────
|
|
|
|
|
|
def _config(*, image_optimize: bool = True, compressors: set[str] | None = None) -> Any:
|
|
"""Minimal stand-in for ``HeadroomConfig`` — only the fields the
|
|
decision reads."""
|
|
return SimpleNamespace(image_optimize=image_optimize, compressors=compressors)
|
|
|
|
|
|
def _msgs(n: int = 1) -> list[dict[str, str]]:
|
|
return [{"role": "user", "content": f"hi-{i}"} for i in range(n)]
|
|
|
|
|
|
# ── Value-type contract ───────────────────────────────────────────────
|
|
|
|
|
|
def test_decision_is_frozen() -> None:
|
|
d = ImageCompressionDecision.decide(headers={}, config=_config(), messages=_msgs())
|
|
try:
|
|
d.should_compress = False # type: ignore[misc]
|
|
except FrozenInstanceError:
|
|
pass
|
|
else:
|
|
raise AssertionError("ImageCompressionDecision must be frozen")
|
|
|
|
|
|
def test_decision_is_value_equal() -> None:
|
|
a = ImageCompressionDecision.decide(headers={}, config=_config(), messages=_msgs())
|
|
b = ImageCompressionDecision.decide(headers={}, config=_config(), messages=_msgs())
|
|
assert a == b
|
|
|
|
|
|
# ── Precedence ────────────────────────────────────────────────────────
|
|
|
|
|
|
def test_compresses_when_every_gate_open() -> None:
|
|
d = ImageCompressionDecision.decide(
|
|
headers={}, config=_config(image_optimize=True), messages=_msgs()
|
|
)
|
|
assert d.should_compress is True
|
|
assert d.passthrough_reason is None
|
|
|
|
|
|
def test_bypass_header_wins() -> None:
|
|
"""Bypass is the user's explicit "don't touch my bytes" signal —
|
|
image compression mutates bytes (tile-aligns / re-encodes), so
|
|
bypass must skip it. Mirror of CompressionDecision."""
|
|
d = ImageCompressionDecision.decide(
|
|
headers={"x-headroom-bypass": "true"},
|
|
config=_config(image_optimize=True),
|
|
messages=_msgs(),
|
|
)
|
|
assert d.should_compress is False
|
|
assert d.passthrough_reason == "bypass_header"
|
|
|
|
|
|
def test_passthrough_mode_header_also_triggers_bypass_skip() -> None:
|
|
"""``x-headroom-mode: passthrough`` alt spelling — mirrors
|
|
``_headroom_bypass_enabled`` semantics across all proxy gates."""
|
|
d = ImageCompressionDecision.decide(
|
|
headers={"x-headroom-mode": "passthrough"},
|
|
config=_config(image_optimize=True),
|
|
messages=_msgs(),
|
|
)
|
|
assert d.should_compress is False
|
|
assert d.passthrough_reason == "bypass_header"
|
|
|
|
|
|
def test_image_optimize_disabled_is_skip() -> None:
|
|
"""Operator config — ``config.image_optimize = False``. Distinct
|
|
reason from ``compression_disabled`` (text compression's gate);
|
|
operators can enable text + disable image independently."""
|
|
d = ImageCompressionDecision.decide(
|
|
headers={}, config=_config(image_optimize=False), messages=_msgs()
|
|
)
|
|
assert d.should_compress is False
|
|
assert d.passthrough_reason == "image_optimize_disabled"
|
|
|
|
|
|
def test_selection_without_image_is_skip() -> None:
|
|
"""A ``--compressor`` selection that leaves out ``image`` (#4613)."""
|
|
d = ImageCompressionDecision.decide(
|
|
headers={}, config=_config(compressors={"smart_crusher"}), messages=_msgs()
|
|
)
|
|
assert d.should_compress is False
|
|
assert d.passthrough_reason == "image_not_selected"
|
|
assert d.image_selected is False
|
|
assert d.image_optimize_enabled is True
|
|
|
|
|
|
def test_selection_with_image_or_wildcard_compresses() -> None:
|
|
for compressors in ({"image"}, {"smart_crusher", " image "}, {"*"}, set(), {" "}):
|
|
d = ImageCompressionDecision.decide(
|
|
headers={}, config=_config(compressors=compressors), messages=_msgs()
|
|
)
|
|
assert d.should_compress is True, compressors
|
|
assert d.image_selected is True
|
|
|
|
|
|
def test_config_without_compressors_attribute_compresses() -> None:
|
|
d = ImageCompressionDecision.decide(
|
|
headers={}, config=SimpleNamespace(image_optimize=True), messages=_msgs()
|
|
)
|
|
assert d.should_compress is True
|
|
|
|
|
|
def test_image_optimize_disabled_beats_image_not_selected() -> None:
|
|
d = ImageCompressionDecision.decide(
|
|
headers={},
|
|
config=_config(image_optimize=False, compressors={"smart_crusher"}),
|
|
messages=_msgs(),
|
|
)
|
|
assert d.passthrough_reason == "image_optimize_disabled"
|
|
|
|
|
|
def test_image_not_selected_beats_no_messages() -> None:
|
|
d = ImageCompressionDecision.decide(
|
|
headers={}, config=_config(compressors={"smart_crusher"}), messages=[]
|
|
)
|
|
assert d.passthrough_reason == "image_not_selected"
|
|
|
|
|
|
def test_no_messages_is_skip() -> None:
|
|
"""Empty or missing messages — nothing to look at. Same shape as
|
|
CompressionDecision's no_messages reason."""
|
|
d = ImageCompressionDecision.decide(headers={}, config=_config(), messages=[])
|
|
assert d.should_compress is False
|
|
assert d.passthrough_reason == "no_messages"
|
|
|
|
|
|
def test_messages_none_is_skip() -> None:
|
|
"""``messages=None`` is treated identically to empty list."""
|
|
d = ImageCompressionDecision.decide(headers={}, config=_config(), messages=None)
|
|
assert d.should_compress is False
|
|
assert d.passthrough_reason == "no_messages"
|
|
|
|
|
|
# ── Precedence ordering ──────────────────────────────────────────────
|
|
|
|
|
|
def test_bypass_beats_image_optimize_disabled() -> None:
|
|
"""User signal beats operator signal — bypass is the more
|
|
informative dashboard slice."""
|
|
d = ImageCompressionDecision.decide(
|
|
headers={"x-headroom-bypass": "true"},
|
|
config=_config(image_optimize=False),
|
|
messages=_msgs(),
|
|
)
|
|
assert d.passthrough_reason == "bypass_header"
|
|
|
|
|
|
def test_bypass_beats_no_messages() -> None:
|
|
"""Bypass+no-messages surfaces bypass — user opted out, the
|
|
empty body is incidental."""
|
|
d = ImageCompressionDecision.decide(
|
|
headers={"x-headroom-bypass": "true"},
|
|
config=_config(),
|
|
messages=[],
|
|
)
|
|
assert d.passthrough_reason == "bypass_header"
|
|
|
|
|
|
def test_image_optimize_disabled_beats_no_messages() -> None:
|
|
"""When config is off AND messages empty, surface the operator
|
|
decision — more meaningful for dashboards."""
|
|
d = ImageCompressionDecision.decide(
|
|
headers={}, config=_config(image_optimize=False), messages=[]
|
|
)
|
|
assert d.passthrough_reason == "image_optimize_disabled"
|
|
|
|
|
|
# ── Observability fields ─────────────────────────────────────────────
|
|
|
|
|
|
def test_observability_booleans_populated_when_compressing() -> None:
|
|
d = ImageCompressionDecision.decide(
|
|
headers={}, config=_config(image_optimize=True), messages=_msgs(2)
|
|
)
|
|
assert d.bypass_header_set is False
|
|
assert d.image_optimize_enabled is True
|
|
assert d.has_messages is True
|
|
|
|
|
|
def test_observability_booleans_populated_when_passthrough() -> None:
|
|
d = ImageCompressionDecision.decide(
|
|
headers={"x-headroom-bypass": "true"},
|
|
config=_config(image_optimize=True),
|
|
messages=_msgs(),
|
|
)
|
|
assert d.bypass_header_set is True
|
|
assert d.image_optimize_enabled is True
|
|
assert d.has_messages is True
|
|
|
|
|
|
# ── apply_to_tags ────────────────────────────────────────────────────
|
|
|
|
|
|
def test_apply_to_tags_stamps_reason_when_passthrough() -> None:
|
|
d = ImageCompressionDecision.decide(
|
|
headers={"x-headroom-bypass": "true"}, config=_config(), messages=_msgs()
|
|
)
|
|
tags: dict[str, str] = {}
|
|
d.apply_to_tags(tags)
|
|
assert tags == {"image_skip_reason": "bypass_header"}
|
|
|
|
|
|
def test_apply_to_tags_is_a_noop_when_compressing() -> None:
|
|
d = ImageCompressionDecision.decide(headers={}, config=_config(), messages=_msgs())
|
|
tags: dict[str, str] = {"client": "codex"}
|
|
d.apply_to_tags(tags)
|
|
assert tags == {"client": "codex"}
|
|
|
|
|
|
def test_apply_to_tags_preserves_pre_existing_entries() -> None:
|
|
"""Image skip reason coexists with other slicing tags (client,
|
|
passthrough_reason, memory_skip_reason) — they all live in the
|
|
same RequestOutcome.tags dict."""
|
|
d = ImageCompressionDecision.decide(
|
|
headers={}, config=_config(image_optimize=False), messages=_msgs()
|
|
)
|
|
tags: dict[str, str] = {
|
|
"client": "claude-code",
|
|
"passthrough_reason": "compression_disabled",
|
|
"memory_skip_reason": "no_user_id",
|
|
}
|
|
d.apply_to_tags(tags)
|
|
assert tags["client"] == "claude-code"
|
|
assert tags["passthrough_reason"] == "compression_disabled"
|
|
assert tags["memory_skip_reason"] == "no_user_id"
|
|
assert tags["image_skip_reason"] == "image_optimize_disabled"
|
|
|
|
|
|
def test_apply_to_tags_for_every_skip_reason() -> None:
|
|
"""Every reason name round-trips cleanly into the tag dict."""
|
|
cases: dict[str, dict[str, Any]] = {
|
|
"bypass_header": {
|
|
"headers": {"x-headroom-bypass": "true"},
|
|
"config": _config(),
|
|
"messages": _msgs(),
|
|
},
|
|
"image_optimize_disabled": {
|
|
"headers": {},
|
|
"config": _config(image_optimize=False),
|
|
"messages": _msgs(),
|
|
},
|
|
"image_not_selected": {
|
|
"headers": {},
|
|
"config": _config(compressors={"smart_crusher"}),
|
|
"messages": _msgs(),
|
|
},
|
|
"no_messages": {
|
|
"headers": {},
|
|
"config": _config(),
|
|
"messages": [],
|
|
},
|
|
}
|
|
for expected_reason, kwargs in cases.items():
|
|
d = ImageCompressionDecision.decide(**kwargs)
|
|
tags: dict[str, str] = {}
|
|
d.apply_to_tags(tags)
|
|
assert tags.get("image_skip_reason") == expected_reason, expected_reason
|