mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(image): flatten transparent images onto white before re-encoding as JPEG (#4296)
## Description
When Headroom resizes an image it re-encodes it as JPEG. That happens in
both `headroom/image/tile_optimizer.py:_resize_image_bytes`
(tile-boundary step) and
`headroom/image/compressor.py:ImageCompressor._resize_image`
(provider-format resize). Both prepare the image the same way:
```python
if resized.mode in ("RGBA", "P"):
resized = resized.convert("RGB")
```
`convert("RGB")` drops the alpha channel rather than compositing it, so
every fully transparent pixel shows whatever RGB value it was stored
with, which is usually `(0, 0, 0)`. A transparent-background PNG, such
as a diagram, chart export, logo or UI asset with dark text, comes out
as a **solid black image**. Through `ImageCompressor.compress(...,
provider="openai")` the model receives one colour and the text is gone
(proof below).
The same thing happens with the other kinds of PNG transparency: a
transparent palette entry (`P`), and a `tRNS` colour key on an `RGB` or
`L` image. Separately, `LA` (grayscale + alpha) raises `OSError: cannot
write mode LA as JPEG`, so `compress()` catches it and skips the
optimization.
Closes: N/A (no matching issue found)
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/image/tile_optimizer.py`: new `_jpeg_ready(img)` helper.
- Any image with transparency data (Pillow's `has_transparency_data`: an
alpha band, a palette `transparency` entry, or `tRNS` on RGB/L) is
composited onto a white RGB canvas using its alpha as the mask.
- `1`, `P`, `CMYK` and `YCbCr` images convert to RGB; RGB and L pass
through unchanged.
- High-bit-depth modes (`I;16`, `I`, `F`) are returned unconverted, so
the JPEG save fails as it does today and callers keep the original
image. A plain `convert("RGB")` would clip them to white.
- `_resize_image_bytes` uses the helper before `save(format="JPEG")`.
- `headroom/image/compressor.py`: `_resize_image` uses the same helper
in place of its copy of the `RGBA`/`P` conversion.
- `tests/test_image_compression.py` (runs without torch;
`tests/test_image_compressor.py` skips without it). The pixel checks
decode the JPEG that would be sent:
- `test_transparent_png_is_flattened_onto_white` (RGBA);
- `test_trns_and_palette_transparency_are_flattened_onto_white[RGB, L,
P]`;
- `test_grayscale_alpha_is_converted_not_raised` (LA);
- `test_compressor_resize_flattens_transparency_onto_white`
(`ImageCompressor._resize_image`);
- `test_high_bit_depth_is_not_squashed_to_8_bit`, which guards that
`I;16` is left unconverted.
## Testing
- [x] Unit tests pass (`pytest`): `tests/test_image_compression*.py`,
`tests/test_image_memo.py`, `tests/test_image_log_redaction.py`,
`tests/test_provider_openai_images.py`,
`tests/test_provider_codex_images.py`, not the full suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
New tests on `main` (fix reverted, tests kept):
```text
$ python -m pytest tests/test_image_compression.py -q -k "transparen or trns or grayscale_alpha or high_bit or flattens"
E OSError: cannot write mode LA as JPEG
E ImportError: cannot import name '_jpeg_ready' from 'headroom.image.tile_optimizer'
FAILED tests/test_image_compression.py::TestMessageOptimization::test_transparent_png_is_flattened_onto_white
FAILED tests/test_image_compression.py::TestMessageOptimization::test_trns_and_palette_transparency_are_flattened_onto_white[RGB]
FAILED tests/test_image_compression.py::TestMessageOptimization::test_trns_and_palette_transparency_are_flattened_onto_white[L]
FAILED tests/test_image_compression.py::TestMessageOptimization::test_trns_and_palette_transparency_are_flattened_onto_white[P]
FAILED tests/test_image_compression.py::TestMessageOptimization::test_grayscale_alpha_is_converted_not_raised
FAILED tests/test_image_compression.py::TestMessageOptimization::test_high_bit_depth_is_not_squashed_to_8_bit
FAILED tests/test_image_compression.py::TestMessageOptimization::test_compressor_resize_flattens_transparency_onto_white
======================= 7 failed, 29 deselected in 2.89s =======================
```
(The `I;16` guard fails on `main` only because the helper doesn't exist
yet. It's there to keep high-bit-depth images from being clipped.)
With the fix:
```text
$ python -m pytest tests/test_image_compression*.py tests/test_image_memo.py tests/test_image_log_redaction.py \
tests/test_provider_openai_images.py tests/test_provider_codex_images.py -q
======================== 95 passed, 4 skipped in 11.18s ========================
$ ruff check headroom/image tests/test_image_compression.py
All checks passed!
$ ruff format --check headroom/image/tile_optimizer.py tests/test_image_compression.py
2 files already formatted
```
## Real Behavior Proof
- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel), Pillow 13.0.0,
`HEADROOM_OFFLINE=1` (no torch, so ML technique routing falls back as it
does in a default proxy install)
- Exact command / steps: drew a 1920×1080 RGBA PNG with a transparent
background and a black label bar, put it in an OpenAI-format user
message (`image_url` data URL) with the text "Describe this diagram",
ran `ImageCompressor().compress(messages, provider="openai")` (what the
proxy's image step calls), decoded the image it would send, and printed
the background pixel, a label pixel and the number of distinct colours.
Same script on `main` and on this branch.
- Observed result: both runs resize to a 910×512 JPEG. On `main` the
image sent is a single colour: background `(0, 0, 0)`, the same as the
label, so the label is invisible. On this branch the background is white
`(255, 255, 255)` and the label stays black.
```text
== main
sent: data:image/jpeg (910, 512) | background pixel: (0, 0, 0) | label
pixel: (0, 0, 0)
distinct colours: 1
== this branch
sent: data:image/jpeg (910, 512) | background pixel: (255, 255, 255) |
label pixel: (0, 0, 0)
distinct colours: 43
```
- Not tested: a live model call; Anthropic and Google payloads through
`ImageCompressor` (they use the same `_resize_image`); semi-transparent
pixels are composited onto white, which matches how most viewers show a
PNG on a light page but isn't the only possible choice.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for images with transparency
(now flattened onto white instead of turning black) and LA images (now
resized instead of skipped). Opaque RGB / L images are encoded exactly
as before, and high-bit-depth images still fall back to the original as
before.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
`_jpeg_ready` lives in `tile_optimizer.py`, which has no heavy imports,
and `compressor.py` imports it lazily inside `_resize_image`, the same
way it already imports PIL there.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
75cfbba91d
commit
9f5cfcfd2b
@@ -334,13 +334,11 @@ class ImageCompressor:
|
||||
# Resize
|
||||
resized = img.resize((new_width, new_height), Image.Resampling.LANCZOS)
|
||||
|
||||
# Convert to RGB if needed (for JPEG)
|
||||
if resized.mode in ("RGBA", "P"):
|
||||
resized = resized.convert("RGB")
|
||||
|
||||
# Save as JPEG for best compression
|
||||
from .tile_optimizer import _jpeg_ready
|
||||
|
||||
buf = io.BytesIO()
|
||||
resized.save(buf, format="JPEG", quality=quality, optimize=True)
|
||||
_jpeg_ready(resized).save(buf, format="JPEG", quality=quality, optimize=True)
|
||||
return buf.getvalue(), "image/jpeg"
|
||||
|
||||
def _estimate_tokens(self, image_data: bytes, detail: str = "high") -> int:
|
||||
|
||||
@@ -167,6 +167,29 @@ def find_optimal_anthropic_dimensions(width: int, height: int) -> tuple[int, int
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _jpeg_ready(img: Any) -> Any:
|
||||
"""Return ``img`` in a mode JPEG can store.
|
||||
|
||||
JPEG has no alpha, and ``convert("RGB")`` simply drops it, so transparent
|
||||
pixels show their hidden colour (usually black). Composite them onto
|
||||
white instead: alpha bands, palette transparency and PNG ``tRNS`` on RGB
|
||||
or grayscale images alike. Palette, bilevel and CMYK/YCbCr images convert
|
||||
to RGB. High-bit-depth modes (``I;16``, ``I``, ``F``) have no faithful
|
||||
8-bit form, so they are returned as-is and the JPEG save fails as before,
|
||||
which makes callers keep the original image.
|
||||
"""
|
||||
from PIL import Image
|
||||
|
||||
if img.has_transparency_data:
|
||||
rgba = img.convert("RGBA")
|
||||
flat = Image.new("RGB", rgba.size, (255, 255, 255))
|
||||
flat.paste(rgba, mask=rgba.getchannel("A"))
|
||||
return flat
|
||||
if img.mode in ("1", "P", "CMYK", "YCbCr"):
|
||||
return img.convert("RGB")
|
||||
return img
|
||||
|
||||
|
||||
def _resize_image_bytes(
|
||||
image_data: bytes, target_width: int, target_height: int
|
||||
) -> tuple[bytes, str]:
|
||||
@@ -182,12 +205,8 @@ def _resize_image_bytes(
|
||||
|
||||
resized = img.resize((target_width, target_height), Image.Resampling.LANCZOS)
|
||||
|
||||
# Convert RGBA to RGB for JPEG
|
||||
if resized.mode in ("RGBA", "P"):
|
||||
resized = resized.convert("RGB")
|
||||
|
||||
buf = io.BytesIO()
|
||||
resized.save(buf, format="JPEG", quality=85, optimize=True)
|
||||
_jpeg_ready(resized).save(buf, format="JPEG", quality=85, optimize=True)
|
||||
return buf.getvalue(), "image/jpeg"
|
||||
|
||||
|
||||
|
||||
@@ -203,6 +203,102 @@ class TestMessageOptimization:
|
||||
assert len(text_blocks) == 1
|
||||
assert text_blocks[0]["text"] == "What is this?"
|
||||
|
||||
@staticmethod
|
||||
def _png_message(img) -> list[dict]:
|
||||
buf = io.BytesIO()
|
||||
img.save(buf, format="PNG")
|
||||
b64 = base64.b64encode(buf.getvalue()).decode()
|
||||
return [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}}
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
@staticmethod
|
||||
def _decoded(optimized: list[dict]):
|
||||
from PIL import Image
|
||||
|
||||
url = optimized[0]["content"][0]["image_url"]["url"]
|
||||
return Image.open(io.BytesIO(base64.b64decode(url.split(",", 1)[1]))).convert("RGB")
|
||||
|
||||
def test_transparent_png_is_flattened_onto_white(self):
|
||||
"""JPEG has no alpha: transparent pixels must not turn black (their hidden RGB)."""
|
||||
from PIL import Image
|
||||
|
||||
img = Image.new("RGBA", (1920, 1080), (0, 0, 0, 0))
|
||||
img.paste((0, 0, 0, 255), (100, 500, 1800, 560)) # black "text" bar
|
||||
optimized, results = optimize_images_in_messages(self._png_message(img), "openai")
|
||||
|
||||
assert len(results) == 1 and results[0].resized
|
||||
out = self._decoded(optimized)
|
||||
assert min(out.getpixel((5, 5))) > 245
|
||||
assert max(out.getpixel((out.width // 2, out.height * 530 // 1080))) < 10
|
||||
|
||||
@pytest.mark.parametrize("mode", ["RGB", "L", "P"])
|
||||
def test_trns_and_palette_transparency_are_flattened_onto_white(self, mode):
|
||||
"""PNG tRNS on RGB/L images and a transparent palette entry are transparency too."""
|
||||
from PIL import Image
|
||||
|
||||
img = Image.new("RGBA", (1920, 1080), (0, 0, 0, 0))
|
||||
img.paste((0, 0, 0, 255), (100, 500, 1800, 560))
|
||||
if mode == "P":
|
||||
img = Image.new("P", (1920, 1080), 0)
|
||||
img.putpalette([0, 0, 0, 0, 0, 0]) # index 0 transparent, index 1 opaque black
|
||||
img.paste(1, (100, 500, 1800, 560))
|
||||
img.info["transparency"] = 0
|
||||
else:
|
||||
# tRNS: one colour is transparent; the bar is a different, still dark colour.
|
||||
img = Image.new(mode, (1920, 1080), 0)
|
||||
img.paste(1 if mode == "L" else (1, 1, 1), (100, 500, 1800, 560))
|
||||
img.info["transparency"] = 0 if mode == "L" else (0, 0, 0)
|
||||
optimized, results = optimize_images_in_messages(self._png_message(img), "openai")
|
||||
|
||||
assert len(results) == 1 and results[0].resized
|
||||
out = self._decoded(optimized)
|
||||
assert min(out.getpixel((5, 5))) > 245
|
||||
assert max(out.getpixel((out.width // 2, out.height * 530 // 1080))) < 10
|
||||
|
||||
def test_grayscale_alpha_is_converted_not_raised(self):
|
||||
"""LA used to raise "cannot write mode LA as JPEG", skipping the optimization."""
|
||||
from PIL import Image
|
||||
|
||||
img = Image.new("LA", (1920, 1080), (0, 0))
|
||||
img.paste((0, 255), (100, 500, 1800, 560))
|
||||
optimized, results = optimize_images_in_messages(self._png_message(img), "openai")
|
||||
|
||||
assert len(results) == 1 and results[0].resized
|
||||
out = self._decoded(optimized)
|
||||
assert min(out.getpixel((5, 5))) > 245
|
||||
|
||||
def test_high_bit_depth_is_not_squashed_to_8_bit(self):
|
||||
"""I;16 has no faithful 8-bit form: leave it unconverted rather than clip to white."""
|
||||
from PIL import Image
|
||||
|
||||
from headroom.image.tile_optimizer import _jpeg_ready
|
||||
|
||||
img = Image.new("I;16", (64, 64), 1000)
|
||||
assert _jpeg_ready(img) is img
|
||||
|
||||
def test_compressor_resize_flattens_transparency_onto_white(self):
|
||||
"""ImageCompressor's provider-format resize also writes JPEG."""
|
||||
from PIL import Image
|
||||
|
||||
from headroom.image.compressor import ImageCompressor
|
||||
|
||||
img = Image.new("RGBA", (1600, 1000), (0, 0, 0, 0))
|
||||
img.paste((0, 0, 0, 255), (100, 480, 1500, 520))
|
||||
buf = io.BytesIO()
|
||||
img.save(buf, format="PNG")
|
||||
|
||||
data, media_type = ImageCompressor()._resize_image(buf.getvalue(), max_dimension=512)
|
||||
|
||||
assert media_type == "image/jpeg"
|
||||
out = Image.open(io.BytesIO(data)).convert("RGB")
|
||||
assert min(out.getpixel((5, 5))) > 245
|
||||
|
||||
def test_small_image_not_resized(self):
|
||||
"""Image already at optimal size is not changed."""
|
||||
msgs = _make_openai_image_message(512, 512)
|
||||
|
||||
Reference in New Issue
Block a user