mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Every turn resends the whole conversation, so every screenshot in history reaches the image stack again on every request: - A turn routed to `transcode` runs RapidOCR on every image in the conversation (`_apply_compression` loops over all messages). With N screenshots in history, that's N OCR runs per transcode turn. - Every turn runs the SigLIP encoder (`OnnxTechniqueRouter.analyze_image`) on the first image, which is usually the same one. Both depend only on the image bytes, so the repeats are pure waste. On one macOS proxy the same screenshot was OCR'd 114 times in two hours, and the isolation worker sat near a full core for hours. `ImageCompressor` and `OnnxTechniqueRouter` now keep a bounded LRU (4096 images) of those results, keyed on the image bytes' sha256. The OCR memo is also bounded at 32 MiB of retained text. The query half of routing still runs every turn, so the chosen technique and the forwarded messages are unchanged. The memo lives on the instance, which the isolation worker already reuses across calls (#2513). Only deterministic results are cached: a `None` for no confident text is remembered, while an OCR engine failure (`OcrEngineError`) or a failed SigLIP analysis is not, so that image is forwarded as-is this turn and retried on the next. The memo only helps while the isolation worker lives, and a cold transcode of a long history outruns the 30s isolation timeout: about 11 screenshots, at 2–3s of OCR each on a busy machine, plus the model load. The timeout resets the pool, the worker and its memo go with it, and the next turn's new worker starts over from the first screenshot. The history never gets compressed, and each turn burns the full timeout. On the same machine this happened 32 times in one day. So `run_image_compression_isolated` now passes the worker a deadline at two thirds of its timeout. Past it, no new OCR starts. An image not reached takes the existing no-OCR-text fallback this turn, and the next turn continues where this one stopped. The rest of the budget is the margin for the OCR still running when the deadline hits. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/image/image_types.py`: `ImageMemo[T]`, an `OrderedDict` LRU keyed on `sha256(image_bytes)`, with `get(image_data, compute)`. It evicts until both `max_entries` (default 4096) and `max_bytes` hold, measuring values with `sys.getsizeof`. A value larger than the whole byte budget is returned without being cached. Only values `compute` returns are cached, so a raise leaves the key uncached. It lives here because this module is already the dependency-free one shared by the compressor and the router. - `headroom/image/compressor.py`: `ImageCompressor._ocr_memo` (32 MiB). The transcode branch of `_apply_compression` reads OCR text through it. `_ocr_extract` now raises `OcrEngineError` when engine init or the OCR call fails; deterministic outcomes (no text, low confidence, unrecognized result shape, no backend installed) still return `None`. `_apply_compression` handles the raise like the old `None` path. - `headroom/image/onnx_router.py`: `OnnxTechniqueRouter._signals_memo`. `analyze_image` reads the SigLIP signals through it, inside its existing try, with the encoder moved to `_encode_image`, so a failed analysis is logged and retried rather than memoized. - `headroom/image/compressor.py`: `compress()` and `_apply_compression()` take an optional `deadline` (a `time.time()` value). Past it, the transcode branch only reads OCR text the memo already has. Default `None` means no limit, so direct and in-process callers are unchanged. - `headroom/proxy/image_isolation.py`: `run_image_compression_isolated` passes `deadline = time.time() + timeout * 2/3` to the worker, and the worker passes it to `compress()`. The test worker hooks, including #3940's `_hang_worker`, take the extra argument; #3940's targeted `_reset_image_pool(pool)` and worker kill are kept. - `tests/test_image_memo.py`: a two-turn transcode conversation OCRs each image once and produces the same text blocks on both turns; past the deadline a memoized image is still transcoded, an unseen one is not OCR'd and stays an image, and the next call OCRs it; a long history reuses OCR across turns; an OCR engine failure is retried while a no-text result is remembered; `classify` with a changing query analyzes each image once; LRU eviction order; the byte budget holds. Direct callers of `_ocr_extract` now get `OcrEngineError` for an engine failure where they used to get `None`; the only in-tree caller is `_apply_compression`, and the existing OCR API-compat tests pass unchanged. The PyTorch `TrainedRouter` fallback is not memoized. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output Before the change (upstream/maind5318ac2), new tests only: ```text FAILED tests/test_image_memo.py::test_transcode_ocrs_each_image_once_across_turns FAILED tests/test_image_memo.py::test_classify_analyzes_each_image_once FAILED tests/test_image_memo.py::test_image_memo_evicts_least_recently_used ============================== 3 failed in 0.71s =============================== ``` The transcode test fails on the repeat itself: `assert [b'a', b'a', b'b'] == [b'a', b'b']`. With the change, all image tests with the `image` extra installed (real RapidOCR and ONNX models, nothing skipped): ```text $ uv run --extra dev --extra image pytest tests/test_image_memo.py tests/test_image_compression.py tests/test_image_ocr_api_compat.py tests/test_image_compressor_singleton_reuse.py tests/test_image_compression_isolation.py tests/test_image_compression_offload.py ============================= 56 passed in 12.41s ============================== $ uv run ruff check headroom/image headroom/proxy/image_isolation.py tests/test_image_memo.py All checks passed! $ uv run mypy headroom/image headroom/proxy/image_isolation.py Success: no issues found in 7 source files ``` Every test file that touches the image stack or the proxy's image path, with the `image` extra: ```text $ uv run --extra dev --extra image pytest tests/test_image_*.py tests/test_onnx_router_threads.py tests/test_builtin_compressor_adapters.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_handler_helpers.py 228 passed, 2 skipped, 1 warning in 47.23s ``` The two skips are the tree-sitter and trafilatura adapters, which aren't installed here. The full suite did not finish locally within 25 minutes, so CI covers the rest. ## Real Behavior Proof - Environment: macOS 27 (M2, 8 cores), Python 3.11.13, `uv run --extra dev --extra image` (rapidocr-onnxruntime, real technique-router and SigLIP ONNX models), upstream/maind5318ac2vs this branch. - Exact command / steps: a script builds a 10-turn conversation from a real 1200x800 screenshot, adding one new screenshot (a different 500px band of it) per user turn. Each turn calls the real `ImageCompressor().compress(messages, provider="anthropic")` on one compressor instance, as the isolation worker does. The query classifier is pinned to `transcode` so every turn takes the OCR path; SigLIP, OCR and `_apply_compression` are real. `_ocr_extract` and `analyze_image` are wrapped with counters, and the final turn's output is hashed. - Observed result: upstream OCRs 55 times and runs SigLIP 10 times over 10 turns; the branch OCRs each screenshot once and runs SigLIP once, in under a third of the time, with byte-identical output (output below). ```text upstream: {"ocr": 55, "siglip": 10, "seconds": 94.4, "final_sha": "74db63599ff27958"} branch: {"ocr": 10, "siglip": 1, "seconds": 29.0, "final_sha": "74db63599ff27958"} ``` Upstream OCRs 1+2+...+10 = 55 times; the branch OCRs each screenshot once. The compressed messages are byte-identical. - Cold long history, through the real isolation worker: one user message with 12 real screenshots (500px bands of the same 1200x800 capture), sent six times through `run_image_compression_isolated(..., timeout=10.0)`. The query classifier is pinned to `transcode` in the spawned worker as well (a scratch `sitecustomize` on `PYTHONPATH`). Everything else is the real worker: model load, SigLIP, RapidOCR. ```text upstream/main this branch turn 1: 10.0s timeout images_as_text 0 turn 1: 6.9s transcode images_as_text 4 turn 2: 10.0s timeout images_as_text 0 turn 2: 7.5s transcode images_as_text 10 turn 3: 10.0s timeout images_as_text 0 turn 3: 2.4s transcode images_as_text 12 turn 4: 10.0s timeout images_as_text 0 turn 4: 0.0s transcode images_as_text 12 turn 5: 10.0s timeout images_as_text 0 turn 5: 0.0s transcode images_as_text 12 turn 6: 10.0s timeout images_as_text 0 turn 6: 0.0s transcode images_as_text 12 ``` Upstream never compresses this history, and every turn costs the full timeout plus a worker restart. The branch stays inside the budget and finishes the history on turn 3. After that, each turn costs nothing. - Live traffic: I ran the same memoization as a runtime patch on headroom-ai 0.39.0 behind a Claude Code proxy. In two hours before the patch there were 308 OCR runs for 234 image compressions, the most-repeated screenshot was OCR'd 21 times, and there were 24 worker model cold starts. In the 20 minutes after, there were 2 OCR runs for 53 image compressions, at most 1 per screenshot, and 1 cold start. The machine's load average went from 75–143 to 3 (the worker-timeout half of that is #3940). After a proxy restart, the memo alone was defeated by the cold-history timeout described above: a timeout every 1–2 minutes, the same screenshots OCR'd again by each new worker. That is what the deadline is for. - Live proxy, deadline included: the same change vendored into a desktop build on headroom-ai 0.39.0. I POSTed 30 synthetic text screenshots to the running proxy's `/v1/messages` with an invalid API key, so Anthropic answered 401 after the image pass and nothing was billed, and asked the same transcode question three turns in a row. Turn 1: 19 OCR runs, then the deadline stopped new OCR; transcode returned in 20.7s, inside the 30s timeout, with 11 images downscaled. Turn 2: 11 OCR runs (only the images not done yet); all 30 went out as text (33 KB body). Turn 3: 0 OCR runs, 0.4s. One worker process served every call, with no timeout or crash. Before the change, the same proxy logged 24 image worker timeouts in 30 minutes of real use. - Not tested: OpenAI and Google image formats on real traffic (they go through the same per-block loop, and the memo key is the decoded bytes). Memory growth over days: the OCR memo is capped at 32 MiB of text and 4096 entries, the SigLIP memo at 4096 entries of four floats each. The PyTorch router fallback. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a, not rollout-managed. - Stable/default behavior changed: the memo changes neither the forwarded bytes nor the technique chosen. The deadline changes what a transcode turn that would have timed out forwards. Before, the whole call timed out and every image went out original-size. Now the images OCR'd in time go out as text, and the rest take the existing no-text fallback (downscaled, the same as when OCR finds no confident text). Turns that finish inside the deadline are unchanged. An OCR engine failure is not memoized: the image takes the no-text fallback that turn and is retried on the next, the same per-turn retry as before this change. - Kill switch / disable path: none. A result is only reused for byte-identical images. Raising `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` raises the OCR budget with it. - Unsafe override required: no. - Qualification impact: CPU drops. Compression rises only on turns that used to time out. The prompt cache is unaffected: history images inside the provider-confirmed prefix are replayed byte-identical by `finalize_turn`/`overlay_cached_prefix`, whatever this turn's image pass produced. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` - it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
199 lines
7.7 KiB
Python
199 lines
7.7 KiB
Python
"""OCR and SigLIP run once per image, not once per turn.
|
|
|
|
Every turn resends the whole conversation, so each screenshot in history
|
|
reached the image stack again on every request: one proxy OCR'd the same
|
|
screenshot 114 times in two hours.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import base64
|
|
import sys
|
|
import types
|
|
from typing import Any
|
|
|
|
import pytest
|
|
|
|
from headroom.image import compressor as compressor_module
|
|
from headroom.image.compressor import ImageCompressor
|
|
from headroom.image.image_types import ImageMemo, ImageSignals, Technique
|
|
from headroom.image.onnx_router import OnnxTechniqueRouter
|
|
|
|
|
|
def _image(data: bytes) -> dict[str, Any]:
|
|
return {
|
|
"type": "image",
|
|
"source": {
|
|
"type": "base64",
|
|
"media_type": "image/png",
|
|
"data": base64.b64encode(data).decode(),
|
|
},
|
|
}
|
|
|
|
|
|
def test_transcode_ocrs_each_image_once_across_turns(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
calls: list[bytes] = []
|
|
|
|
class _RapidOCR:
|
|
def __call__(self, image_data: bytes) -> Any:
|
|
calls.append(image_data)
|
|
return [(None, f"text of {image_data.decode()}", 0.99)], 0.0
|
|
|
|
fake = types.ModuleType("rapidocr_onnxruntime")
|
|
fake.RapidOCR = _RapidOCR # type: ignore[attr-defined]
|
|
monkeypatch.setitem(sys.modules, "rapidocr_onnxruntime", fake)
|
|
monkeypatch.setattr(compressor_module, "_RESOLVED_OCR", None)
|
|
|
|
compressor = ImageCompressor(use_siglip=False)
|
|
turn1 = [{"role": "user", "content": [_image(b"a")]}]
|
|
turn2 = [
|
|
*turn1,
|
|
{"role": "assistant", "content": "ok"},
|
|
{"role": "user", "content": [_image(b"b")]},
|
|
]
|
|
|
|
first = compressor._apply_compression(turn1, Technique.TRANSCODE, "anthropic")
|
|
second = compressor._apply_compression(turn2, Technique.TRANSCODE, "anthropic")
|
|
|
|
assert calls == [b"a", b"b"]
|
|
assert second[0] == first[0]
|
|
assert first[0]["content"] == [{"type": "text", "text": "[OCR from image]\ntext of a"}]
|
|
assert second[2]["content"] == [{"type": "text", "text": "[OCR from image]\ntext of b"}]
|
|
|
|
|
|
def test_past_deadline_ocrs_nothing_new_but_reuses_the_memo(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
# A cold history that would outrun the isolation timeout stops starting
|
|
# OCR at the deadline, so the call returns, the worker and its memo
|
|
# survive, and the next turn picks up where this one stopped.
|
|
calls: list[bytes] = []
|
|
compressor = ImageCompressor(use_siglip=False)
|
|
monkeypatch.setattr(
|
|
compressor, "_ocr_extract", lambda data: calls.append(data) or f"text of {data.decode()}"
|
|
)
|
|
seen = [{"role": "user", "content": [_image(b"a")]}]
|
|
compressor._apply_compression(seen, Technique.TRANSCODE, "anthropic")
|
|
history = [{"role": "user", "content": [_image(b"a"), _image(b"b")]}]
|
|
calls.clear()
|
|
|
|
late = compressor._apply_compression(history, Technique.TRANSCODE, "anthropic", deadline=0.0)
|
|
on_time = compressor._apply_compression(history, Technique.TRANSCODE, "anthropic")
|
|
|
|
assert late[0]["content"][0] == {"type": "text", "text": "[OCR from image]\ntext of a"}
|
|
assert late[0]["content"][1]["type"] == "image"
|
|
assert on_time[0]["content"][1] == {"type": "text", "text": "[OCR from image]\ntext of b"}
|
|
assert calls == [b"b"]
|
|
|
|
|
|
def test_long_history_reuses_ocr_across_turns(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
# Each transcode turn scans the history oldest-first. With fewer slots
|
|
# than images, every miss evicts the very image the scan reaches next, so
|
|
# nothing was ever reused; 300 screenshots must fit.
|
|
calls: list[bytes] = []
|
|
compressor = ImageCompressor(use_siglip=False)
|
|
monkeypatch.setattr(compressor, "_ocr_extract", lambda data: calls.append(data) or "text")
|
|
images = [_image(str(i).encode()) for i in range(300)]
|
|
history = [{"role": "user", "content": images}]
|
|
|
|
for _ in range(2):
|
|
compressor._apply_compression(history, Technique.TRANSCODE, "anthropic")
|
|
|
|
assert len(calls) == 300
|
|
|
|
|
|
def test_ocr_engine_failure_is_retried_but_no_text_is_remembered(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
# "No confident text" is a property of the image; an engine that raised
|
|
# says nothing about it, so remembering that would skip the image for the
|
|
# worker's lifetime.
|
|
calls: list[bytes] = []
|
|
engine_up = False
|
|
|
|
class _RapidOCR:
|
|
def __call__(self, image_data: bytes) -> Any:
|
|
calls.append(image_data)
|
|
if not engine_up:
|
|
raise RuntimeError("onnxruntime session lost")
|
|
return ([], 0.0) if image_data == b"blank" else ([(None, "hi", 0.99)], 0.0)
|
|
|
|
fake = types.ModuleType("rapidocr_onnxruntime")
|
|
fake.RapidOCR = _RapidOCR # type: ignore[attr-defined]
|
|
monkeypatch.setitem(sys.modules, "rapidocr_onnxruntime", fake)
|
|
monkeypatch.setattr(compressor_module, "_RESOLVED_OCR", None)
|
|
|
|
compressor = ImageCompressor(use_siglip=False)
|
|
history = [{"role": "user", "content": [_image(b"blank"), _image(b"text")]}]
|
|
|
|
down = compressor._apply_compression(history, Technique.TRANSCODE, "anthropic")
|
|
engine_up = True
|
|
up = compressor._apply_compression(history, Technique.TRANSCODE, "anthropic")
|
|
again = compressor._apply_compression(history, Technique.TRANSCODE, "anthropic")
|
|
|
|
assert [part["type"] for part in down[0]["content"]] == ["image", "image"]
|
|
assert up[0]["content"][1] == {"type": "text", "text": "[OCR from image]\nhi"}
|
|
assert again == up
|
|
assert calls == [b"blank", b"text", b"blank", b"text"]
|
|
|
|
|
|
def test_classify_analyzes_each_image_once(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
router = OnnxTechniqueRouter()
|
|
analyzed: list[bytes] = []
|
|
|
|
def encode(image_data: bytes) -> ImageSignals:
|
|
analyzed.append(image_data)
|
|
if len(analyzed) == 1:
|
|
raise RuntimeError("siglip session lost")
|
|
return ImageSignals(has_text=0.9, is_document=0.1, is_complex=0.1, has_small_details=0.1)
|
|
|
|
monkeypatch.setattr(router, "classify_query", lambda query: (Technique.PRESERVE, 0.9))
|
|
monkeypatch.setattr(router, "_load_siglip", lambda: None)
|
|
monkeypatch.setattr(router, "_encode_image", encode)
|
|
|
|
# The query changes every turn; the image analysis does not need to. The
|
|
# first analysis of "a" fails, so it is retried rather than remembered.
|
|
turns = ((b"a", "what"), (b"a", "and now?"), (b"a", "and?"), (b"b", "what"))
|
|
for image_data, query in turns:
|
|
router.classify(image_data, query)
|
|
|
|
assert analyzed == [b"a", b"a", b"b"]
|
|
|
|
|
|
def test_image_memo_evicts_least_recently_used() -> None:
|
|
memo = ImageMemo[str](max_entries=2)
|
|
computed: list[bytes] = []
|
|
|
|
def get(data: bytes) -> str:
|
|
return memo.get(data, lambda: computed.append(data) or data.decode())
|
|
|
|
for data in (b"a", b"b", b"a", b"c", b"a", b"b"):
|
|
assert get(data) == data.decode()
|
|
|
|
# "a" stayed hot, so "c" evicted "b", and "b" evicted "c".
|
|
assert computed == [b"a", b"b", b"c", b"b"]
|
|
|
|
|
|
def test_image_memo_holds_at_most_max_bytes_of_values() -> None:
|
|
# Entry count alone let a few thousand text-heavy screenshots pin hundreds
|
|
# of MB of OCR text in a worker that outlives the conversation.
|
|
text = "x" * 1000
|
|
memo = ImageMemo[str](max_bytes=3 * sys.getsizeof(text))
|
|
computed: list[bytes] = []
|
|
|
|
def get(data: bytes, value: str = text) -> str:
|
|
return memo.get(data, lambda: computed.append(data) or value)
|
|
|
|
for data in (b"a", b"b", b"c", b"d", b"b"):
|
|
get(data)
|
|
assert computed == [b"a", b"b", b"c", b"d"] # "d" evicted "a"; "b" still held
|
|
get(b"a")
|
|
assert computed[-1] == b"a"
|
|
|
|
# A value bigger than the whole budget is returned but evicts nothing.
|
|
huge = "y" * 10_000
|
|
assert get(b"huge", huge) == huge
|
|
get(b"huge", huge)
|
|
assert computed[-2:] == [b"huge", b"huge"]
|
|
assert b"b" in memo and b"a" in memo
|