Commit Graph
3 Commits
Author SHA1 Message Date
aebb729ef3 fix(tokenizers): don't cache token-count inputs larger than max_chars (#4469)
## Description

`TokenCountCache.put` only checked the size budget before admitting a
new key, never the size of the key itself. A single text longer than
`max_chars` (8M chars by default) was stored and stayed alive until the
next new-key put cleared the cache, pinning a large request body in
memory for the tokenizer's lifetime.

Fixes #4264

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/tokenizers/base.py`: `put` returns early when `len(text) >
max_chars`. The check sits before the eviction clear, so warm entries
are kept. Re-count branch and soft budget for ordinary entries are
unchanged.
- `tests/test_token_count_cache.py`: oversized text is not cached, warm
entry survives an oversized put, text of exactly `max_chars` is still
admitted.

## Testing

- [x] Unit tests added/updated
- [x] Ran tests locally

```
$ python -m pytest tests/test_token_count_cache.py -q
============================= 21 passed in 2.57s ==============================
$ ruff check headroom/tokenizers/base.py tests/test_token_count_cache.py
All checks passed!
$ ruff format --check headroom/tokenizers/base.py tests/test_token_count_cache.py
2 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13, real `EstimatingTokenCounter`,
branch at 1ffe0993 vs upstream main `base.py`
- Exact command / steps: count a 9,600,000-char string via
`EstimatingTokenCounter().count_text(s)` under `tracemalloc`, then `del
s; gc.collect()` and read retained bytes
- Observed result: upstream main: count 2400000, retained_bytes 9600105
(string pinned by the cache). This branch: count 2400000, retained_bytes
32. Same token count, oversized body released.
- Not tested: full proxy end-to-end memory profile under live traffic

```
$ python repro.py   # upstream main base.py
count 2400000 retained_bytes 9600105
$ python repro.py   # this branch
count 2400000 retained_bytes 32
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: only oversized texts skip the count
cache; returned counts are identical
- Kill switch / disable path: none needed; revert the commit
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4469"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-28 23:33:57 -05:00
c873d13fe0 perf(proxy): lighter request path: alias the message snapshot, keep the token cache on re-counts, decompress off the event loop (#3909)
## Description

Hi! Three small costs on every proxied request, found while running
headroom in front of a busy agent loop:

- **Snapshot copy.** Both handlers `copy.deepcopy` the client messages
before the pipeline, which already makes its own deep copy, so long
conversations were copied twice per request. The snapshot now aliases
the list (`HEADROOM_SNAPSHOT_DEEPCOPY=1` brings the copy back).
- **Token cache wiped on re-counts.** `TokenCountCache.put` ran the
admission check even for a key already cached, so a full cache cleared
itself when an agent re-counted a stable prefix. A put on a known key
now just updates it; a new key past the cap still clears, as before.
- **Decompression on the event loop.** zstd/gzip/deflate/br bodies were
decompressed inline. They now go through one bounded helper on
`asyncio.to_thread`; the #3284 size caps and the 413/400 errors are
unchanged.

Two small things ride along: both shared body readers strip
streaming-only `index` keys from content blocks (well-formed requests
pass through byte-identical), and a dead `except` (ruff B025) that let a
missing codec escape as a raw ImportError instead of a 400.

**Happy to split this.** It is three independent changes plus two small
fixes, and CONTRIBUTING asks for one logical change per PR. If you'd
rather review them separately, say which and I'll open them one at a
time. Opening as a draft until you've had a look.

## Type of Change

- [x] Performance improvement
- [x] Bug fix (non-breaking change that fixes an issue): the B025 codec
path and the `index` strip

## Changes Made

- `proxy/handlers/anthropic.py`, `openai.py`: snapshot alias + env
switch
- `tokenizers/base.py`: early return in `TokenCountCache.put` for a
cached key
- `proxy/helpers.py`: `_decompress_bounded` on a worker thread; readers
call the `index` strip
- `utils.py`: `strip_streaming_only_content_fields_in_place`
- tests: 8 new (`test_output_only_request_blocks.py`,
`test_token_count_cache.py`)

## Testing

- [x] Unit tests pass (`pytest`): the proxy suite and the changed-area
files, not the whole suite
- [x] Linting passes (`ruff check`, `ruff format --check`) on the
touched files
- [ ] Type checking (`mypy headroom`): not run
- [x] New tests added

### Test Output

```text
# Linux, Python 3.12.13, source checkout + the compiled core from the published wheel (not built from source)
$ pytest -q tests/test_proxy*.py tests/test_openai_message_level_cache_control.py tests/test_request_body_decompression_limits.py \
    tests/test_output_only_request_blocks.py tests/test_token_count_cache.py tests/test_utils.py tests/test_tokenizer.py tests/test_tokenizers.py
this branch: 1095 passed, 123 skipped, 2 failed
main:        1087 passed, 123 skipped, 2 failed
# the same two fail on main in this environment (probably the prebuilt core vs. source), untouched by this PR:
#   tests/test_proxy_openai_cache_stability.py::test_openai_chat_outcome_provider_uses_the_fixed_taxonomy
#   tests/test_proxy_openai_cache_stability.py::test_openai_chat_custom_base_flood_cannot_grow_the_provider_set
$ ruff check <7 files>            All checks passed!
$ ruff format --check <7 files>   7 files already formatted
```

## Real Behavior Proof

- **Environment:** Linux, Python 3.12.13. `headroom proxy --host
127.0.0.1 --port 18787 --no-telemetry` run from this branch,
`ANTHROPIC_TARGET_API_URL` pointed at a small local stand-in for
`/v1/messages` that records what the proxy forwards. Same run repeated
on main for comparison.
- **Exact steps:** (1) after an 8 s warm-up, POST `/v1/messages` with a
gzip-encoded body (228 kB on the wire, 77.4 MB decompressed, 79
messages) while polling `/livez` every 10 ms; (2) POST a message whose
text block carries a stray `"index": 0`.
- **Observed result (this branch):**
  ```text
gzip body: 228 kB on the wire, 77.4 MB decompressed -> HTTP 200 in 1.29s
  upstream received: 77,422,858 bytes, 79 messages
  /livez while idle:          355 polls, max 2.2 ms, over 50 ms: 0
/livez during that request: 28 polls, max 288.3 ms, over 50 ms: 5, p50
2.3 ms
content block with a stray `index`: HTTP 200; upstream saw no `index`
key
  ```
Main, same steps: HTTP 200 in 1.35s; `/livez` during the request max
303.3 ms, 6 polls over 50 ms; no `index` forwarded either (the Anthropic
handler already strips it on that path).
- **Honest reading:** the decompression path works end to end off the
loop, with the caps intact. At this size the loop stall is dominated by
JSON parsing and the pipeline, not decompression, so this run shows **no
measurable latency difference**. I'm not claiming a latency win from it.
The snapshot alias and the cache change remove work, but they weren't
benchmarked here.
- **Not tested:** a real provider under production load; mypy; the full
suite; the OpenAI and passthrough paths live (unit tests only).

## Runtime Rollout Safety

- Stable/default behavior changed: snapshot aliases by default; cache
keeps entries on re-count; decompression on a worker thread; `index`
stripped at parse
- Kill switch: `HEADROOM_SNAPSHOT_DEEPCOPY=1` for the snapshot; the rest
have none
- Rollback path: revert the PR

## Review Readiness

- [x] I have performed a self-review
- [ ] This PR is ready for human review (draft: waiting on whether you'd
like it split)

## Additional Notes

Worth a reviewer's eye: with the alias, the snapshot shares the live
`messages` list. Nothing between the snapshot and its readers mutates
that list in place today, and a code comment warns against starting to.

Co-authored-by: bill3wits <8632714+bill3wits@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-04 13:58:28 -07:00
Tejas Chopra 53af90d68c perf: cut hot-path latency 27% (token-count memo, startup preloads, JSON scan memo) (#2838)
## Description

Four independent latency fixes on the request hot path, found by
profiling and each measured in isolation. No behaviour changes: every
commit is either a memo of a pure function, work moved to startup, or
work that was computed and discarded.

**End to end: 287ms → 210ms (−27%) on a 68k-token mixed payload, with
byte-identical output** (68,514 → 48,725 tokens both before and after).

Plus one-off costs removed that don't show in steady-state numbers:
~4.9s of lazy imports that were firing *inside* user requests, and
~750ms of HuggingFace round-trips per process start.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

**1. Memoise `count_text` (`ac369277`)** — tiktoken's `CoreBPE.encode`
was 0.243s of a 0.30s profiled request. It dominates because the same
string is counted repeatedly: a 103KB payload drove 600KB of encoding,
~6x the content, across six call sites (`tokenizers/base.py:196`,
`content_router.py:4704` and `:5474`, `parser.py:185/192/298`). 35% of
encode calls and 22% of encoded characters were an exact repeat *within
one request*.

`count_text` is a pure function of its text, so replaying a stored count
returns the same integer. That is the whole safety argument, and it is
what makes this safe at the sites whose count feeds a routing decision
(`context_pressure` → `min_ratio`) rather than a log line — an
*estimate* there would change which blocks compress; a memo cannot.

Keyed on the text itself, not a hash: a collision would hand back a
wrong count for real content and silently change compression. The cost
is holding the strings, so entries and total characters are both capped.
Clear-on-full rather than LRU eviction — the pipeline runs on a thread
pool, `dict` get/set/clear are atomic under the GIL but
`OrderedDict.move_to_end` is not.

**2. Preload what was importing mid-request (`2921a15b`)** — `litellm`
(2.9–3.8s) was imported lazily *on the event loop* during the first
request: `emit_request_outcome` → `record_request` →
`_estimate_compression_savings_usd` calls the loader before its own
`tokens_saved <= 0` early return, so even a request that saved nothing
paid it. `trafilatura` (978ms, pulling `htmldate` → `dateparser` and its
timezone tables) is the most expensive lazy import in the transform tree
— every other compressor module is 1–20ms — and fires on the first
request carrying an HTML-ish or mixed-content block. The TOIN singleton
reads ~5MB of learned patterns on construction (~150ms); a stale comment
claimed the SmartCrusher preload covered it, and it does not.

All three now load in `_eager_preload_transforms`, which already runs
under `asyncio.to_thread` and so cannot delay the port bind.

Same commit, two Kompress cold-path fixes: `_load_modernbert_tokenizer`
always used `local_files_only=False`, which makes transformers
re-validate against the Hub on every load — a tree listing plus a HEAD
per file — even when fully cached (~900ms warm-cache vs ~150ms
local-only). And `ensure_background_download` re-spawned a
finished-or-failed thread on the next call, so an unreachable Hub meant
one fresh download thread *per request* for the life of the process,
each importing transformers and holding the GIL against the event loop.
Consecutive failures now back off; success clears it, so the happy path
and the transient-failure path are unchanged.

**3. Memoise the JSON-block scan (`039c9735`)** —
`_has_valid_json_block_with_text` tries every `{`/`[`-leading line as a
possible block start. When a candidate never balances,
`_extract_json_block` scans character-by-character to the end of the
content and returns nothing — then the next candidate does it again.
Quadratic, on the request path, growing exactly 4x per doubling.

**4. `CostTracker.totals()` (`286b97e4`)** —
`_current_savings_tracker_totals` called `stats()` once per request and
read two of its fields. Building the rest includes
`period_cost_breakdown()`, which walks up to 100k cost records over 31
days, on the event loop, holding the metrics lock. It degrades with
proxy **uptime**, not load, which is why no short benchmark would
surface it.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/ tests/ --exclude headroom/dashboard/templates
All checks passed!

$ mypy --python-version 3.12 headroom/
Found 1 error in 1 file (checked 515 source files)
  headroom/release_version.py:235: error: Name "tomllib" already defined (by an import)
  # pre-existing on main, in a file this PR does not touch — verified by
  # running the same command on a clean main checkout.

$ python -m pytest tests/test_token_count_cache.py tests/test_mixed_content_scan_cache.py \
    tests/test_kompress_download_backoff.py tests/test_cost_tracker_totals.py -q
306 passed

$ python -m pytest tests/ -q -k "token or tokenizer or count or estimator or provider"
1303 passed, 105 skipped in 423.42s

$ python -m pytest tests/ -q -k "cost or budget or metrics or savings or stats"
683 passed, 127 skipped, 1 failed
  # tests/test_proxy_memory_integration.py::TestMemoryStats::test_health_endpoint_works_with_memory
  # Order-dependent and pre-existing: it SKIPS in isolation, and fails identically
  # on a clean main checkout under the same -k selection (681 passed, 1 failed).
```

## Real Behavior Proof

- **Environment:** macOS, Python 3.12.6, local CPU, remote Kompress
disabled. Profiled with `cProfile` on `anthropic_pipeline.apply`.
- **Exact command / steps:** a 68k-token payload of four `tool_result`
blocks (900-item pretty JSON, 60KB of Python source, 500 lines of
JS-style object logs, 500 plain log lines), six reps, **content unique
per rep so every run is router-cache-cold**, run on this branch and on
main in alternation.
- **Observed result:**

  | | median | min | tokens |
  |---|---|---|---|
  | main | 287ms | 286ms | 68,514 → 48,725 |
  | this branch | 210ms | 208ms | 68,514 → 48,725 |

  Per-change, measured in isolation:

  | change | before | after |
  |---|---|---|
| `count_text` memo | — | −25% pipeline wall; 44% of counted chars from
cache on new content, 100% when history repeats |
| litellm / trafilatura / TOIN | 3829 / 978 / 150ms mid-request | at
startup, off the event loop |
  | Kompress tokenizer | ~900ms | ~150ms |
  | JS-style object logs (1200 lines) | 4643ms | 183ms |
  | truncated JSONL (1200 lines) | 3737ms | 116ms |
| `cost_tracker` per request | 2.8ms @20k records, 13.6ms @100k | loop
over models, not records |

Output equality: 18/18 payloads byte-identical on `tokens_before`,
`tokens_after` and a sha256 of the resulting messages, with the memo
forced on vs off.

- **Not tested:** Windows and Linux (the ORT dylib and CPU-arena paths
differ); multi-worker deployments; a proxy with a genuinely large live
cost ledger (the 100k figure is from a synthetic ledger); real
HTML-heavy traffic through the preloaded trafilatura path.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

**Docs:** N/A — no user-facing surface changes. The reasoning lives in
the code, at the sites where someone debugging would look.

**A regression I introduced and caught.** The scan memo initially made
pretty-printed JSON ~2x **slower**: content that balances on the first
scan has nothing to reuse and just pays the per-line dict traffic. The
cache is now built only *after* a scan has run to the end without
balancing, which is the actual signal that later candidates will re-walk
the same tail. Every shape now improves and none regress:

```
                      before     after
js object logs       4642.9ms    182.7ms   25x
JSONL truncated      3736.8ms    115.9ms   32x
pretty JSON             5.6ms      3.5ms
JSONL valid             5.6ms      3.2ms
plain logs              1.0ms      0.6ms
python source           1.0ms      0.5ms
markdown prose          0.9ms      0.5ms
```

Worth stating plainly: had I only benchmarked the shape I was fixing,
I'd have shipped a win on rare content and a loss on the common case.

**The scan fix is constant-factor, not asymptotic.** The walk over
remaining lines is still O(candidates × lines), so 3200 lines of the
pathological shape is still ~1.4s. The tests assert scan-call counts
rather than implying linearity. True linearity needs a prefix-sum
rewrite with a string-state fallback; that seemed like the wrong risk
for this PR.

**How the parser change is proven safe.** `_extract_json_block` is a
parser, so golden values would only encode whatever the new code does.
Instead the pre-memo implementation is kept verbatim in the test file as
an oracle, and every candidate index of a 139-document corpus — escapes,
unterminated strings, delimiters inside strings, code fences, truncated
JSON, randomised mixtures — is asserted equal, with a cold cache, with
the shared cache the real callers use, and replayed.

**Measurement trap, for anyone re-running these numbers.** Give each arm
its own content. Reusing one payload across arms lets the second arm hit
the router's result cache, which reads as a speedup having nothing to do
with the change under test. I hit this twice while working on it: it
manufactured a fake "INFO logging costs 21.8%" finding (real answer:
0.3%) and it *understated* the memo win.

**Deliberately not in this PR:**
- **ONNX thread tuning** — measured zero gain, and
`intra_op_num_threads` is not bitwise-safe (1.6e-05 score drift from
float reduction order), so it would trade an output risk for nothing.
- **`str(content)` on block lists** counts a base64 image at 210,775
tokens instead of 1,604 (131x), pinning `context_pressure` to 1.0 and
forcing the most aggressive `min_ratio` on any conversation containing
an image. Real bug, but fixing it changes compression output — needs its
own reviewed behaviour-change PR.
- **`chunk_words=350` against the tokenizer's 512-token limit** silently
drops roughly a third of every full chunk (measured: 240/240 words kept
in the first 240, 15/110 in the tail). That is data loss rather than
latency, it changes every output, and correcting it costs ~1.3x latency.
Filing separately.
- **Telemetry off the request thread** — the TOIN auto-save is a 236ms
inline stall every 600s and the waste-signal re-parse is ~50ms/request
that is invisible in `pipeline_total` (computed before it). Both want
deferral rather than removal, which is a larger change than belongs
here.
2026-08-06 17:47:40 -07:00