Files
headroom/tests/test_image_compression_decision.py
SD 8f04221b2d fix(proxy): turn off the image stage when --compressor leaves out image (#4723)
## Description

`headroom proxy --compressor` / `HEADROOM_COMPRESSORS` lists `image` and
says unselected built-ins are disabled. The selection only sets
`ContentRouterConfig.enable_image_optimizer`, which the proxy does not
consult. The Anthropic and OpenAI handlers gate their image stage on
`ProxyConfig.image_optimize`, and nothing set it from a selection. So
`--compressor smart_crusher` still sent every screenshot through the
image stage.

This PR makes the image gate honor the selection.
`ImageCompressionDecision`, which both handlers already call, reads
`config.compressors` on each request and skips the image stage with a
new reason, `image_not_selected`, when a selection leaves out `image`.
The operator's `image_optimize` is never written, and
`image_optimize=False` still wins. This is Gap 1 from #4613, agreed with
the reporter there. Gap 2 (Windows self-update while a proxy is running)
will be a separate PR.

Refs #4613

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/image_compression_policy.py`:
- `image_compressor_selected(compressors)` follows the same rules as
`_apply_compressor_selection`: no selection, an empty one or `*` keeps
images; otherwise `image` must be named.
- `decide_image_compression` takes `image_selected` (default `True`) and
adds `image_not_selected`, which comes after `image_optimize_disabled`
and before `no_messages`.
- `headroom/proxy/image_compression_decision.py`: `decide` passes the
selection from `config.compressors` and exposes `image_selected` next to
the other inputs it saw. The new reason appears in `image_skip_reason`.
- `tests/test_compressor_selection_image_stage.py`: sends a screenshot
request through the real proxy app with only the upstream faked. It runs
on two routes: Anthropic in token mode, and OpenAI chat in cache mode,
which still compresses images in cache mode. The cases are:
  - a selection without `image` never enters the image stage;
  - a selection with `image`, `*`, and no selection each enter it once;
  - a config reused across two proxies follows the current selection;
  - `image_optimize=False` with `image` selected still skips the stage.
- `tests/test_image_compression_decision.py`: covers the new reason, its
precedence, the wildcard and blank selections, and a config with no
`compressors` attribute.
- `docs/content/docs/proxy.mdx`: the `--compressor` row says that
leaving out `image` also turns off the image stage.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): see Additional Notes
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

The new tests against `main`'s source (b21479e0):

```text
FAILED tests/test_compressor_selection_image_stage.py::TestImageStageFollowsSelection::test_selection_without_image_skips_the_image_stage[anthropic-token]
FAILED tests/test_compressor_selection_image_stage.py::TestImageStageFollowsSelection::test_selection_without_image_skips_the_image_stage[openai-cache]
FAILED tests/test_compressor_selection_image_stage.py::TestImageStageFollowsSelection::test_reused_config_follows_the_current_selection[anthropic-token]
FAILED tests/test_compressor_selection_image_stage.py::TestImageStageFollowsSelection::test_reused_config_follows_the_current_selection[openai-cache]
4 failed, 8 passed
```

At this head:

```text
$ pytest tests/test_compressor_selection.py tests/test_compressor_selection_image_stage.py tests/test_image_compression_decision.py tests/test_image_compression_policy.py tests/test_router_external_dispatch.py tests/test_proxy_eager_preload_bind.py -q
99 passed

$ pytest <every test file that mentions image_skip_reason, ImageCompressionDecision, image_optimize or compressors> -m "not live" -q
12 failed, 1994 passed, 26 skipped

$ ruff check <changed files> && ruff format --check <changed files>
All checks passed!
```

The 12 failures are in `tests/test_image_compressor.py`,
`tests/test_ccr_batch_processor.py`,
`tests/test_garbled_compression_fixes.py` and
`tests/test_transforms/test_content_router.py`. The same 12 fail on
`main` in my environment (the image ones because the venv has Pillow
12), so they are not from this change.

## Real Behavior Proof

- Environment: macOS (Darwin 24.6.0, arm64), Python 3.12.3, Pillow
13.0.0. Older Pillow makes the proxy forward images unchanged.
- Exact command / steps: the real proxy app in token mode on the
Anthropic route, with the real image compressor and image worker; only
the upstream model is faked, and it answers instantly. Each setting
sends the same 2560x1600 PNG screenshot ("what is the exact error text
in this screenshot?") 5 times. Per request, the script records
image-stage entries, what reached the upstream, and the time the proxy
took. Run once on `main` (b21479e0) and once on this branch.
- Observed result: on `main`, `--compressor smart_crusher` still ran the
image stage on 5 of 5 requests and downscaled every screenshot
(`full_low`, 1,870 → 255 tokens) at about 45 ms each. With this PR it
ran on 0 of 5, the original image went upstream unchanged, and each
request took 4.6 ms. With no selection or with `image` selected, both
builds behave the same. `image_optimize` stays `True` in every run.

| Build | Setting | Image-stage runs | Image sent upstream | Proxy time
p50 |
|---|---|---|---|---|
| `main` | no `--compressor` | 5 of 5 | downscaled (`full_low`, 1,870 →
255 tokens) | 45.4 ms |
| `main` | `--compressor smart_crusher` | **5 of 5** | **downscaled** |
**45.2 ms** |
| `main` | `--compressor smart_crusher,image` | 5 of 5 | downscaled |
46.9 ms |
| this PR | no `--compressor` | 5 of 5 | downscaled | 48.2 ms |
| this PR | `--compressor smart_crusher` | **0 of 5** | **original,
unchanged** | **4.6 ms** |
| this PR | `--compressor smart_crusher,image` | 5 of 5 | downscaled |
47.6 ms |

The first image request of each process also loads the image models
(about 1.0–1.3 s here). With `image` left out, that load doesn't happen.
- Not tested: a live upstream, the OCR (`transcode`) technique (the
router picked `full_low` for this screenshot), and Windows.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only for a `--compressor` /
`HEADROOM_COMPRESSORS` selection that leaves out `image`. Those proxies
now forward images unchanged, as the help text already says. With no
selection, a selection that includes `image`, or `*`, nothing changes.
- Kill switch / disable path: add `image` to the selection, or remove
the selection
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the PR's commits

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

n/a

## Additional Notes

- Review follow-up: the first revision (619ea4ad) wrote
`image_optimize=False` into the caller's `ProxyConfig`. Now the decision
reads the selection on each request, so a reused config follows its
current selection, and an explicit `image_optimize=False` is kept.
- #1327 proposes a separate `HEADROOM_DISABLE_IMAGE_OPTIMIZER` variable.
This PR doesn't add one; it makes the existing option behave as its help
text says.
- `mypy headroom/proxy/server.py` stops in my local environment on a
`numpy` stub (`Type statement is only supported in Python 3.12 and
greater`) before reaching project code. CI runs mypy.
2026-11-10 10:39:18 -06:00

290 lines
11 KiB
Python

"""Tests for :class:`headroom.proxy.image_compression_decision.ImageCompressionDecision`.
Image compression today is gated at two sites (``openai.py:1203`` +
``anthropic.py:868``) by inline conjunctions. Both already check
``_bypass`` (the drift problem CompressionDecision fixed is NOT
present here), but consolidating into a value type still pays off:
* test-lockable contract (no future site can forget bypass)
* ``apply_to_tags()`` surfaces ``image_skip_reason`` in
``RequestOutcome.tags`` — dashboards can slice image-skipped
traffic by cause (same observability surface as
``passthrough_reason`` and ``memory_skip_reason``)
* Rust-portable shape mirrors :class:`CompressionDecision` exactly
Precedence (highest first):
1. ``bypass_header`` — user opt-out
2. ``image_optimize_disabled`` — operator config off
3. ``no_messages`` — nothing to inspect
"""
from __future__ import annotations
from dataclasses import FrozenInstanceError
from types import SimpleNamespace
from typing import Any
from headroom.proxy.image_compression_decision import ImageCompressionDecision
# ── Helpers ───────────────────────────────────────────────────────────
def _config(*, image_optimize: bool = True, compressors: set[str] | None = None) -> Any:
"""Minimal stand-in for ``HeadroomConfig`` — only the fields the
decision reads."""
return SimpleNamespace(image_optimize=image_optimize, compressors=compressors)
def _msgs(n: int = 1) -> list[dict[str, str]]:
return [{"role": "user", "content": f"hi-{i}"} for i in range(n)]
# ── Value-type contract ───────────────────────────────────────────────
def test_decision_is_frozen() -> None:
d = ImageCompressionDecision.decide(headers={}, config=_config(), messages=_msgs())
try:
d.should_compress = False # type: ignore[misc]
except FrozenInstanceError:
pass
else:
raise AssertionError("ImageCompressionDecision must be frozen")
def test_decision_is_value_equal() -> None:
a = ImageCompressionDecision.decide(headers={}, config=_config(), messages=_msgs())
b = ImageCompressionDecision.decide(headers={}, config=_config(), messages=_msgs())
assert a == b
# ── Precedence ────────────────────────────────────────────────────────
def test_compresses_when_every_gate_open() -> None:
d = ImageCompressionDecision.decide(
headers={}, config=_config(image_optimize=True), messages=_msgs()
)
assert d.should_compress is True
assert d.passthrough_reason is None
def test_bypass_header_wins() -> None:
"""Bypass is the user's explicit "don't touch my bytes" signal —
image compression mutates bytes (tile-aligns / re-encodes), so
bypass must skip it. Mirror of CompressionDecision."""
d = ImageCompressionDecision.decide(
headers={"x-headroom-bypass": "true"},
config=_config(image_optimize=True),
messages=_msgs(),
)
assert d.should_compress is False
assert d.passthrough_reason == "bypass_header"
def test_passthrough_mode_header_also_triggers_bypass_skip() -> None:
"""``x-headroom-mode: passthrough`` alt spelling — mirrors
``_headroom_bypass_enabled`` semantics across all proxy gates."""
d = ImageCompressionDecision.decide(
headers={"x-headroom-mode": "passthrough"},
config=_config(image_optimize=True),
messages=_msgs(),
)
assert d.should_compress is False
assert d.passthrough_reason == "bypass_header"
def test_image_optimize_disabled_is_skip() -> None:
"""Operator config — ``config.image_optimize = False``. Distinct
reason from ``compression_disabled`` (text compression's gate);
operators can enable text + disable image independently."""
d = ImageCompressionDecision.decide(
headers={}, config=_config(image_optimize=False), messages=_msgs()
)
assert d.should_compress is False
assert d.passthrough_reason == "image_optimize_disabled"
def test_selection_without_image_is_skip() -> None:
"""A ``--compressor`` selection that leaves out ``image`` (#4613)."""
d = ImageCompressionDecision.decide(
headers={}, config=_config(compressors={"smart_crusher"}), messages=_msgs()
)
assert d.should_compress is False
assert d.passthrough_reason == "image_not_selected"
assert d.image_selected is False
assert d.image_optimize_enabled is True
def test_selection_with_image_or_wildcard_compresses() -> None:
for compressors in ({"image"}, {"smart_crusher", " image "}, {"*"}, set(), {" "}):
d = ImageCompressionDecision.decide(
headers={}, config=_config(compressors=compressors), messages=_msgs()
)
assert d.should_compress is True, compressors
assert d.image_selected is True
def test_config_without_compressors_attribute_compresses() -> None:
d = ImageCompressionDecision.decide(
headers={}, config=SimpleNamespace(image_optimize=True), messages=_msgs()
)
assert d.should_compress is True
def test_image_optimize_disabled_beats_image_not_selected() -> None:
d = ImageCompressionDecision.decide(
headers={},
config=_config(image_optimize=False, compressors={"smart_crusher"}),
messages=_msgs(),
)
assert d.passthrough_reason == "image_optimize_disabled"
def test_image_not_selected_beats_no_messages() -> None:
d = ImageCompressionDecision.decide(
headers={}, config=_config(compressors={"smart_crusher"}), messages=[]
)
assert d.passthrough_reason == "image_not_selected"
def test_no_messages_is_skip() -> None:
"""Empty or missing messages — nothing to look at. Same shape as
CompressionDecision's no_messages reason."""
d = ImageCompressionDecision.decide(headers={}, config=_config(), messages=[])
assert d.should_compress is False
assert d.passthrough_reason == "no_messages"
def test_messages_none_is_skip() -> None:
"""``messages=None`` is treated identically to empty list."""
d = ImageCompressionDecision.decide(headers={}, config=_config(), messages=None)
assert d.should_compress is False
assert d.passthrough_reason == "no_messages"
# ── Precedence ordering ──────────────────────────────────────────────
def test_bypass_beats_image_optimize_disabled() -> None:
"""User signal beats operator signal — bypass is the more
informative dashboard slice."""
d = ImageCompressionDecision.decide(
headers={"x-headroom-bypass": "true"},
config=_config(image_optimize=False),
messages=_msgs(),
)
assert d.passthrough_reason == "bypass_header"
def test_bypass_beats_no_messages() -> None:
"""Bypass+no-messages surfaces bypass — user opted out, the
empty body is incidental."""
d = ImageCompressionDecision.decide(
headers={"x-headroom-bypass": "true"},
config=_config(),
messages=[],
)
assert d.passthrough_reason == "bypass_header"
def test_image_optimize_disabled_beats_no_messages() -> None:
"""When config is off AND messages empty, surface the operator
decision — more meaningful for dashboards."""
d = ImageCompressionDecision.decide(
headers={}, config=_config(image_optimize=False), messages=[]
)
assert d.passthrough_reason == "image_optimize_disabled"
# ── Observability fields ─────────────────────────────────────────────
def test_observability_booleans_populated_when_compressing() -> None:
d = ImageCompressionDecision.decide(
headers={}, config=_config(image_optimize=True), messages=_msgs(2)
)
assert d.bypass_header_set is False
assert d.image_optimize_enabled is True
assert d.has_messages is True
def test_observability_booleans_populated_when_passthrough() -> None:
d = ImageCompressionDecision.decide(
headers={"x-headroom-bypass": "true"},
config=_config(image_optimize=True),
messages=_msgs(),
)
assert d.bypass_header_set is True
assert d.image_optimize_enabled is True
assert d.has_messages is True
# ── apply_to_tags ────────────────────────────────────────────────────
def test_apply_to_tags_stamps_reason_when_passthrough() -> None:
d = ImageCompressionDecision.decide(
headers={"x-headroom-bypass": "true"}, config=_config(), messages=_msgs()
)
tags: dict[str, str] = {}
d.apply_to_tags(tags)
assert tags == {"image_skip_reason": "bypass_header"}
def test_apply_to_tags_is_a_noop_when_compressing() -> None:
d = ImageCompressionDecision.decide(headers={}, config=_config(), messages=_msgs())
tags: dict[str, str] = {"client": "codex"}
d.apply_to_tags(tags)
assert tags == {"client": "codex"}
def test_apply_to_tags_preserves_pre_existing_entries() -> None:
"""Image skip reason coexists with other slicing tags (client,
passthrough_reason, memory_skip_reason) — they all live in the
same RequestOutcome.tags dict."""
d = ImageCompressionDecision.decide(
headers={}, config=_config(image_optimize=False), messages=_msgs()
)
tags: dict[str, str] = {
"client": "claude-code",
"passthrough_reason": "compression_disabled",
"memory_skip_reason": "no_user_id",
}
d.apply_to_tags(tags)
assert tags["client"] == "claude-code"
assert tags["passthrough_reason"] == "compression_disabled"
assert tags["memory_skip_reason"] == "no_user_id"
assert tags["image_skip_reason"] == "image_optimize_disabled"
def test_apply_to_tags_for_every_skip_reason() -> None:
"""Every reason name round-trips cleanly into the tag dict."""
cases: dict[str, dict[str, Any]] = {
"bypass_header": {
"headers": {"x-headroom-bypass": "true"},
"config": _config(),
"messages": _msgs(),
},
"image_optimize_disabled": {
"headers": {},
"config": _config(image_optimize=False),
"messages": _msgs(),
},
"image_not_selected": {
"headers": {},
"config": _config(compressors={"smart_crusher"}),
"messages": _msgs(),
},
"no_messages": {
"headers": {},
"config": _config(),
"messages": [],
},
}
for expected_reason, kwargs in cases.items():
d = ImageCompressionDecision.decide(**kwargs)
tags: dict[str, str] = {}
d.apply_to_tags(tags)
assert tags.get("image_skip_reason") == expected_reason, expected_reason