Commit Graph
4 Commits
Author SHA1 Message Date
9f5cfcfd2b fix(image): flatten transparent images onto white before re-encoding as JPEG (#4296)
## Description

When Headroom resizes an image it re-encodes it as JPEG. That happens in
both `headroom/image/tile_optimizer.py:_resize_image_bytes`
(tile-boundary step) and
`headroom/image/compressor.py:ImageCompressor._resize_image`
(provider-format resize). Both prepare the image the same way:

```python
if resized.mode in ("RGBA", "P"):
    resized = resized.convert("RGB")
```

`convert("RGB")` drops the alpha channel rather than compositing it, so
every fully transparent pixel shows whatever RGB value it was stored
with, which is usually `(0, 0, 0)`. A transparent-background PNG, such
as a diagram, chart export, logo or UI asset with dark text, comes out
as a **solid black image**. Through `ImageCompressor.compress(...,
provider="openai")` the model receives one colour and the text is gone
(proof below).

The same thing happens with the other kinds of PNG transparency: a
transparent palette entry (`P`), and a `tRNS` colour key on an `RGB` or
`L` image. Separately, `LA` (grayscale + alpha) raises `OSError: cannot
write mode LA as JPEG`, so `compress()` catches it and skips the
optimization.

Closes: N/A (no matching issue found)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/image/tile_optimizer.py`: new `_jpeg_ready(img)` helper.
- Any image with transparency data (Pillow's `has_transparency_data`: an
alpha band, a palette `transparency` entry, or `tRNS` on RGB/L) is
composited onto a white RGB canvas using its alpha as the mask.
- `1`, `P`, `CMYK` and `YCbCr` images convert to RGB; RGB and L pass
through unchanged.
- High-bit-depth modes (`I;16`, `I`, `F`) are returned unconverted, so
the JPEG save fails as it does today and callers keep the original
image. A plain `convert("RGB")` would clip them to white.
  - `_resize_image_bytes` uses the helper before `save(format="JPEG")`.
- `headroom/image/compressor.py`: `_resize_image` uses the same helper
in place of its copy of the `RGBA`/`P` conversion.
- `tests/test_image_compression.py` (runs without torch;
`tests/test_image_compressor.py` skips without it). The pixel checks
decode the JPEG that would be sent:
  - `test_transparent_png_is_flattened_onto_white` (RGBA);
- `test_trns_and_palette_transparency_are_flattened_onto_white[RGB, L,
P]`;
  - `test_grayscale_alpha_is_converted_not_raised` (LA);
- `test_compressor_resize_flattens_transparency_onto_white`
(`ImageCompressor._resize_image`);
- `test_high_bit_depth_is_not_squashed_to_8_bit`, which guards that
`I;16` is left unconverted.

## Testing

- [x] Unit tests pass (`pytest`): `tests/test_image_compression*.py`,
`tests/test_image_memo.py`, `tests/test_image_log_redaction.py`,
`tests/test_provider_openai_images.py`,
`tests/test_provider_codex_images.py`, not the full suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New tests on `main` (fix reverted, tests kept):

```text
$ python -m pytest tests/test_image_compression.py -q -k "transparen or trns or grayscale_alpha or high_bit or flattens"
E   OSError: cannot write mode LA as JPEG
E   ImportError: cannot import name '_jpeg_ready' from 'headroom.image.tile_optimizer'
FAILED tests/test_image_compression.py::TestMessageOptimization::test_transparent_png_is_flattened_onto_white
FAILED tests/test_image_compression.py::TestMessageOptimization::test_trns_and_palette_transparency_are_flattened_onto_white[RGB]
FAILED tests/test_image_compression.py::TestMessageOptimization::test_trns_and_palette_transparency_are_flattened_onto_white[L]
FAILED tests/test_image_compression.py::TestMessageOptimization::test_trns_and_palette_transparency_are_flattened_onto_white[P]
FAILED tests/test_image_compression.py::TestMessageOptimization::test_grayscale_alpha_is_converted_not_raised
FAILED tests/test_image_compression.py::TestMessageOptimization::test_high_bit_depth_is_not_squashed_to_8_bit
FAILED tests/test_image_compression.py::TestMessageOptimization::test_compressor_resize_flattens_transparency_onto_white
======================= 7 failed, 29 deselected in 2.89s =======================
```

(The `I;16` guard fails on `main` only because the helper doesn't exist
yet. It's there to keep high-bit-depth images from being clipped.)

With the fix:

```text
$ python -m pytest tests/test_image_compression*.py tests/test_image_memo.py tests/test_image_log_redaction.py \
    tests/test_provider_openai_images.py tests/test_provider_codex_images.py -q
======================== 95 passed, 4 skipped in 11.18s ========================

$ ruff check headroom/image tests/test_image_compression.py
All checks passed!
$ ruff format --check headroom/image/tile_optimizer.py tests/test_image_compression.py
2 files already formatted
```

## Real Behavior Proof

- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel), Pillow 13.0.0,
`HEADROOM_OFFLINE=1` (no torch, so ML technique routing falls back as it
does in a default proxy install)
- Exact command / steps: drew a 1920×1080 RGBA PNG with a transparent
background and a black label bar, put it in an OpenAI-format user
message (`image_url` data URL) with the text "Describe this diagram",
ran `ImageCompressor().compress(messages, provider="openai")` (what the
proxy's image step calls), decoded the image it would send, and printed
the background pixel, a label pixel and the number of distinct colours.
Same script on `main` and on this branch.
- Observed result: both runs resize to a 910×512 JPEG. On `main` the
image sent is a single colour: background `(0, 0, 0)`, the same as the
label, so the label is invisible. On this branch the background is white
`(255, 255, 255)` and the label stays black.

  ```text
  == main
sent: data:image/jpeg (910, 512) | background pixel: (0, 0, 0) | label
pixel: (0, 0, 0)
  distinct colours: 1
  == this branch
sent: data:image/jpeg (910, 512) | background pixel: (255, 255, 255) |
label pixel: (0, 0, 0)
  distinct colours: 43
  ```
- Not tested: a live model call; Anthropic and Google payloads through
`ImageCompressor` (they use the same `_resize_image`); semi-transparent
pixels are composited onto white, which matches how most viewers show a
PNG on a light page but isn't the only possible choice.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for images with transparency
(now flattened onto white instead of turning black) and LA images (now
resized instead of skipped). Opaque RGB / L images are encoded exactly
as before, and high-bit-depth images still fall back to the original as
before.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

`_jpeg_ready` lives in `tile_optimizer.py`, which has no heavy imports,
and `compressor.py` imports it lazily inside `_resize_image`, the same
way it already imports PIL there.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:38:19 -07:00
chopratejas 13081f4647 Skip image tests when Pillow not installed (CI fix)
Tests that create PIL images now have @needs_pillow decorator.
Token estimation and tile math tests still run without Pillow.
2026-04-09 16:24:15 -07:00
chopratejas 65e4a35546 Add OCR routing via RapidOCR + measure actual tokens after compression
Transcode technique now runs RapidOCR to extract text from images.
Falls back to full_low if OCR confidence < 70% or no text detected.

Token counting is now done AFTER compression by measuring the actual
output — no hardcoded estimates. OCR text counted by char length,
resized images re-estimated from new dimensions.

RapidOCR added to headroom-ai[image] extra (~15MB ONNX models, ~180ms CPU).
4 new OCR tests (extraction, blank image, confidence threshold, full pipeline).
29 total image compression tests passing.
2026-04-09 16:22:18 -07:00
chopratejas 7984bd2389 Re-enable image compression: tile optimizer + ONNX router, no PyTorch
Image compression pipeline:
1. Tile-boundary alignment (pure math, 0ms) — saves 340-680 tokens/image
   on OpenAI by snapping to 512px tile boundaries
2. ONNX technique router (32MB, 6ms) — classifies query intent
3. ONNX SigLIP image encoder (95MB, 49ms) — analyzes image properties

Total: ~127MB models, ~55ms latency, no PyTorch dependency.
Re-enabled in both Anthropic and OpenAI proxy handlers.
25 tests.
2026-04-09 16:11:00 -07:00