mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
fe01961b29 |
fix(tokenizers): honor registered model factories (#4437)
## Description `register_tokenizer(model, factory=...)` currently stores the factory under a model name in the backend registry. Automatic lookup selects a backend instead, so the factory is never called and a custom model silently receives estimated counts. Keep model factories separate and consult them during automatic lookup. This makes the documented factory form usable without changing explicit backend selection. Closes #4434 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Resolve registered model factories lazily, preserving the caller's model spelling and case-insensitive cache lookup. - Remove stale instance or factory registrations when replacing one with the other, and invalidate cached counters. - Retry model factories after failures so a cached estimate cannot mask a later strict lookup or recovery. - Include factory-backed model names in `list_registered()`, including instance-to-factory replacement. - Add regressions for lookup, caching, replacement, backend isolation, strict failure/recovery, registration listings and forced-backend selection. ## Testing - [x] Focused and affected tokenizer/provider/compression tests pass. - [x] Linting and formatting pass with pinned Ruff 0.17.0. - [x] Type checking passes through the repository's pre-commit hook (`mypy headroom`). - [x] New regression fails before the fix and passes afterward. - [x] Public API reproduction and error/fallback controls run locally. - [x] Required pre-push gate passes: Rust workspace checks, extension build, Python suite and commitlint. ### Test Output ```text Before: EstimatingTokenCounter, count 2, factory_calls [] After: CharacterCounter, count 8, factory_calls ['CUSTOM-FACTORY-MODEL'], cached True Baseline new regression: 1 failed, 69 deselected in 1.51s Final affected set: 133 passed, 14 skipped in 14.97s ruff check .: All checks passed! ruff format --check .: 1812 files already formatted Pre-commit: Ruff, formatting, mypy, conflict markers and commitlint passed Pre-push: Rust fmt/clippy/workspace tests and extension rebuild passed Pre-push Python suite: 177 passed, 4 skipped, 1 warning in 8.86s ci-precheck PASSED ``` The 14 skips require the optional `mistral-common` package. The compression-route tests ran and passed. The complete all-extras Python suite was not run. The pre-push suite's warning is the expected Anthropic-provider approximation notice when no client is supplied. ## Real Behavior Proof - Environment: macOS arm64, Python 3.12.15, editable source checkout; no provider or model request. Offline mode enabled. - Exact command / steps: register a factory and count a fixed eight-character string using the public API, as below. Run `python -m pytest -q tests/test_tokenizers.py tests/test_tokenizer_selection_coverage.py tests/test_provider_tokenizer_one_ruler.py tests/test_compress_route_tokenizer_by_model.py tests/test_tokenizer_encoding_resolution.py` for the affected set. - Observed result: the factory is called once with the original model string, returns a counter that reports eight, and repeated case-insensitive lookup reuses that counter. Before the fix, no factory call occurs and the estimator reports two. A deliberately failing factory falls back by default; a subsequent strict lookup raises `ValueError` without clearing the cache. When that factory becomes available, the next lookup recovers. Factory-backed model names remain listed once after replacing an instance. - Not tested: live provider requests, optional Mistral package tests, the complete all-extras Python suite, upstream CI or maintainer acceptance. ```python from headroom.tokenizers import CharacterCounter, get_tokenizer, register_tokenizer calls = [] def factory(model): calls.append(model) return CharacterCounter(chars_per_token=1.0) register_tokenizer("custom-factory-model", factory=factory) first = get_tokenizer("CUSTOM-FACTORY-MODEL") second = get_tokenizer("custom-factory-model") print(type(first).__name__, first.count_text("abcdefgh"), calls, first is second) # CharacterCounter 8 ['CUSTOM-FACTORY-MODEL'] True ``` ## Runtime Rollout Safety - Rollout-managed feature(s): none; existing public tokenizer registration only. - Minimum rollout channel: not applicable, no rollout channel added. - Stable/default behavior changed: explicitly registered model factories are honored during automatic lookup and included in registration listings. Failed model factories remain retryable. Unregistered models and explicit backend selection retain their existing paths and fallback caching. - Kill switch / disable path: choose an explicit backend, or register a prebuilt counter instead. - Unsafe override required: none. - Qualification impact: fixes an existing documented API without new dependencies, provider access or network behavior. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove my fix is effective or that my feature works - [x] I did **not** edit `CHANGELOG.md`, which is generated by release-please ## Additional Notes The existing docstring already describes factory registration. A separate independent review examined the complete patch. No dependency or lockfile changes are included. Local checks establish the observed behavior; upstream CI and maintainer review remain pending. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4437" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
6cc27a453d |
fix(tokenizers): coerce structured text values in message token counters (#4472)
## Description
A dict-valued `text` in a content block reached
`TiktokenCounter.count_text` unchanged and raised `TypeError: unhashable
type: 'dict'` from the string cache, failing `count_messages`. The base
estimator message walker had the same `str` assumption.
Fixes #4273
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/tokenizers/tiktoken_counter.py`: route content `text` blocks
and non-string/non-list `content` through the existing
`coerce_countable_text`.
- `headroom/tokenizers/base.py`: same for role, name,
non-string/non-list content, `text` blocks, Strands `text`,
`tool_use`/`toolUse` names, reasoning text and `thinking`.
Image/document estimation untouched. Inputs are not mutated.
- `tests/test_tokenizers.py`: `TestDictValuedTextFields`, parametrized
over `TiktokenCounter` and `EstimatingTokenCounter`; each structured
message must count the same as its explicit coerced-text equivalent, and
input must be unchanged.
## Testing
- [x] Unit tests added/updated
- [x] Tests fail before the fix and pass after
```
$ pytest tests/test_tokenizers.py -q -k DictValued # without fix
9 failed, 3 passed, 69 deselected
$ pytest tests/test_tokenizers.py -q # with fix
67 passed, 14 skipped
$ ruff check headroom/tokenizers tests/test_tokenizers.py && ruff format --check ...
All checks passed! / 8 files already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, tiktoken gpt-4o encoding, branch
rebased on upstream/main
|
||
|
|
9ed95c3806 |
fix(tokenizers): estimate replayed thinking from text and signature size (#3482)
## Description Estimate replayed Anthropic thinking blocks explicitly instead of pricing their serialized JSON as ordinary text. The shared content walker counts visible thinking text plus `len(signature) * 3 // 16`, an approximation based on decoded signature size. This keeps omitted-display reasoning from counting as free while reducing the old base64-as-prose estimate. Encrypted reasoning cannot be counted exactly from its opaque signature. This is a local heuristic, informed by the author's traffic sample, not a replacement for provider-reported usage or provider token counting. ## Changes Made - Add the thinking branch to the shared tokenizer walker. - Cover signed/unsigned blocks, omitted visible thinking, and provider-walker consistency. ## Testing Latest reviewer validation on macOS/Python 3.13 at `127a039bd`: `pytest tests/test_tokenizers.py -q` — **55 passed, 14 skipped**. Optional tokenizer dependencies account for the skips. Earlier author validation reported Ruff, formatting, and focused mypy passing. ## Real Behavior Proof The prior empty-thinking/4,000-character-signature case now adds a 750-token estimate instead of zero. No new live provider calibration was performed during review; accuracy across models, languages, signature formats, and thinking-retention policies remains approximate. ## Runtime Rollout Safety - Changes local token estimates for replayed thinking; request content is unchanged. - These estimates feed sizing, compression decisions, and local savings displays. - No persistent schema migration or experimental opt-in is required. - Rollback: revert the thinking-count branch and its tests. ## Review Readiness - [x] The reported omitted-thinking zero-count issue is addressed. - [x] Focused tests pass. - [ ] Final human review and merge. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
285176be54 |
fix(tokenizers): price Claude against a real BPE (tiktoken o200k) not a char estimate (#2543)
## Description
Claude's tokenizer is not public, so `get_tokenizer("claude-*")`
returned `EstimatingTokenCounter(chars_per_token=3.5)` — a
**character-ratio estimate**. The estimator's chars-per-token flips with
the *detected content type* (JSON 3.2 / code 3.5 / English 4.0 / CJK
1.5), so:
- the **same bytes** count differently before vs after compression → a
fold could appear to *increase* tokens;
- it **disagrees with the pipeline's** estimator on the same content.
Calibration vs a real BPE (tiktoken `o200k_base`) on representative
content:
| content | char-est(3.5) / o200k |
|---|---|
| English | **1.28×** (overcount 28%) |
| code | **0.83×** (undercount 17%) |
| JSON | 0.97× |
| diff | 1.02× |
i.e. the estimate is off **−17% … +28%**, and the error is
content-dependent (non-uniform) — which is exactly what produces
impossible `tok_after > tok_before` deltas.
This prices Claude against **tiktoken `o200k_base`** — a real,
deterministic, **monotone** BPE. It's not Claude's exact vocab, but it's
within ~10–20% and, crucially, **consistent before/after**, which is
what compression ratios and context-pressure gating actually need.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement
- [ ] Breaking change
- [ ] Documentation update
- [ ] Code refactoring (no functional changes)
## Changes Made
- `TiktokenCounter.__init__` accepts an explicit `encoding` override (so
a model can be priced against a chosen encoding regardless of name-based
resolution).
- `_create_anthropic` now returns `TiktokenCounter(model,
encoding="o200k_base")`, **failing open to the character estimator** if
the tiktoken vocab can't be loaded (reuses the existing `load_encoding`
timeout/guard).
- Updated the `MODEL_PATTERNS` comment and `test_get_anthropic_model` to
the new contract; added
`test_claude_priced_with_real_bpe_not_char_estimate`.
**Blast radius is contained:** `content_router` keeps its own
module-level estimator for routing decisions, so only the **handler's
reported counts** change. `tiktoken` is already a dependency.
**Composes with #2542** (tokenizer-consistent before/after): that PR
makes both endpoints use *one* tokenizer; this PR makes that one
tokenizer a *real BPE*. Together they fully eliminate the
inflated/phantom lines. This PR is based on `main` and is independently
valid.
## Calibration note (please review)
The one behavioral consumer of the handler's count is the
background-compression gate (`original_tokens >=
_background_compression_min_tokens`). Because o200k differs from the old
estimate by ~±20% depending on content, that threshold now trips on
slightly different requests. It's *more* accurate, but the threshold was
tuned against the estimator — worth a re-check. No other gate consumes
the handler count (routing uses `content_router`'s estimator).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` + `ruff format --check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added
- [x] Manual testing performed
### Test Output
```text
$ ruff check <changed files> && ruff format --check <changed files>
All checks passed!
4 files already formatted
$ mypy headroom/tokenizers/registry.py headroom/tokenizers/tiktoken_counter.py
Success: no issues found in 2 source files
$ pytest tests/test_tokenizer.py tests/test_tokenizers.py -q
55 passed, 14 skipped
```
## Real Behavior Proof
- **Unit:** `get_tokenizer("claude-opus-4-8")` →
`TiktokenCounter(encoding_name="o200k_base")`; `count_text(sample)` ==
`tiktoken.get_encoding("o200k_base").encode(sample)` exactly; a
`tool_result` shrunk 300→3 words drops 508→13 tokens (folds register).
- **Live proxy** (Anthropic path, `--proxy-extension lossless_guard`):
foldable request logs `tok_before=694 tok_after=231 tok_saved=463
transforms=turn_hook` — real o200k-scale numbers (vs the estimator's 607
for the same payload), clean fold, no inflation.
- **Not tested:** exact agreement with Anthropic's *own* count_tokens
API (o200k is a proxy; a follow-up could calibrate against it offline).
GPT paths unchanged (already tiktoken).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] Documentation — N/A (internal; docstrings updated)
- [x] My changes generate no new warnings
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Follow-ups (not here): (1) extend the same real-BPE treatment to other
private-tokenizer providers (Gemini/Cohere/Moonshot), each with its own
calibration; (2) optional opt-in calibration against Anthropic's
`count_tokens` API to true-up the ~10–20% absolute offset.
|
||
|
|
6979b5245e |
fix(tokenizers): use o200k_base for gpt-4.1/gpt-4.5/o4 families (#2108)
## Description
`get_encoding_for_model` returns the wrong tiktoken encoding for the
current OpenAI flagship families, so their token counts are computed
with the wrong vocabulary.
The prefix table is ordered most-specific-first, but it has no entry for
the `gpt-4.1` / `gpt-4.5` / `o4` families:
```python
for prefix, encoding in (
("gpt-4o", "o200k_base"),
("gpt-4-turbo", "cl100k_base"),
("gpt-4", "cl100k_base"),
("gpt-3.5", "cl100k_base"),
("o1", "o200k_base"),
("o3", "o200k_base"),
):
if model.startswith(prefix):
return encoding
return DEFAULT_ENCODING # cl100k_base
```
- `gpt-4.1`, `gpt-4.1-mini`, `gpt-4.5-*` all start with `gpt-4`, so they
match the `gpt-4` prefix and get `cl100k_base`.
- `o4-mini` matches no prefix and falls through to the `cl100k_base`
default.
All three families use `o200k_base`. Since `count_text`/`count_messages`
tokenize with the resolved encoding, every token count for those models
is computed against the wrong BPE vocabulary, which skews budget gating
and the compress/skip decision for a large slice of current OpenAI
traffic.
## Fix
Add explicit `gpt-4.1` and `gpt-4.5` prefixes (ordered ahead of `gpt-4`,
which they would otherwise match) and an `o4` prefix, all mapping to
`o200k_base`. Plain `gpt-4` and `gpt-3.5` snapshots still resolve to
`cl100k_base`, and `gpt-4o` still wins for the 4o family.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/tokenizers/tiktoken_counter.py`: add `gpt-4.1`/`gpt-4.5`
prefixes ahead of `gpt-4`, and an `o4` prefix, all mapping to
`o200k_base`.
- `tests/test_tokenizers.py`: add
`test_gpt41_and_o4_families_use_o200k`.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/tokenizers/tiktoken_counter.py tests/test_tokenizers.py
All checks passed!
$ python -m py_compile headroom/tokenizers/tiktoken_counter.py tests/test_tokenizers.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the resolution with a
dependency-free script that runs the old and new prefix tables, and left
the full pytest to CI.
- Exact command / steps: resolved `gpt-4.1`, `gpt-4.1-mini`,
`gpt-4.5-preview`, and `o4-mini` under the old table and the new table,
plus `gpt-4o-*`, `gpt-4-2025-*`, `gpt-4-turbo-*`, `gpt-3.5-turbo`, and
`o1-mini` as regression guards.
- Observed result: old table returns `cl100k_base` for all four (wrong);
new table returns `o200k_base`; the guard models are unchanged
(`gpt-4o-*` and `o1-*` stay `o200k_base`,
`gpt-4*`/`gpt-4-turbo*`/`gpt-3.5*` stay `cl100k_base`). The new test
asserts the four families resolve to `o200k_base` and a plain `gpt-4`
snapshot stays `cl100k_base`.
- Not tested: loading the actual tiktoken vocabularies to count tokens
end to end; full local `pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change adds three ordered prefix entries to a pure
function, verified by the standalone proof and the new regression test
for CI. I intentionally left `gpt-5` out since I didn't want to assert
an encoding I couldn't confirm here; happy to add it in a follow-up if
you can confirm the intended mapping.
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
|
ae10d6c99d |
fix(tokenizers): don't tokenize image blocks as text in TiktokenCounter (#2093)
## Description
`TiktokenCounter.count_messages` explodes the token count for any
content block that isn't plain text or an OpenAI `image_url`.
The multi-part content loop handles exactly two shapes:
```python
if part.get("type") == "text":
total += self.count_text(part.get("text", ""))
elif part.get("type") == "image_url":
... # 85 / 170 tokens by detail
else:
total += self.count_text(str(part)) # <-- everything else
```
Every other block shape reaching the `else` gets `str(part)`-ified and
tokenized as text. That includes Anthropic's `{"type": "image",
"source": {"type": "base64", "data": "<...>"}}`, `tool_result`,
`tool_use`, and the Strands SDK blocks. Over the wire the image `data`
is a base64 string, so a 1MB image turns into ~1.4M characters of "text"
and is counted as **~330K tokens** for a single image (a ~218x overcount
in a standalone repro). Anything that relies on the count — budget
gating, the compress/skip decision, savings math — is thrown off for
multimodal requests that route through the tiktoken counter.
The base class already solved this: `BaseTokenizer._count_content_parts`
prices `image`/`image_url`/`input_image` at a flat bounded estimate and
has a comment stating it exists specifically to stop "a 1MB image =
~330K fake tokens". The tiktoken override just never delegated to it for
the non-text shapes.
## Fix
Delegate unknown block shapes in the `else` branch to
`self._count_content_parts([part])` instead of stringifying them. `text`
and `image_url` keep the existing tiktoken-specific handling (including
the 85/170 detail split); everything else now gets the base handler's
bounded pricing.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/tokenizers/tiktoken_counter.py`: the `count_messages`
multi-part `else` branch delegates to the base
`_count_content_parts([part])` rather than `count_text(str(part))`.
- `tests/test_tokenizers.py`: add
`test_count_messages_image_block_is_not_stringified` — a base64 image
block inside list content must stay bounded (well under the tens of
thousands of tokens the blob would produce as text).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/tokenizers/tiktoken_counter.py tests/test_tokenizers.py
All checks passed!
$ python -m py_compile headroom/tokenizers/tiktoken_counter.py tests/test_tokenizers.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the magnitude with a
dependency-free script that models the old `count_text(str(part))` path
against the base handler's bounded image estimate, and left the full
pytest to CI.
- Exact command / steps: built a ~1MB PNG as an Anthropic `image` block
with the payload base64-encoded (as it arrives over the wire), computed
the old path (`len(str(part)) / ~4` chars-per-token) versus the new path
(base handler prices an image block at a flat 1600).
- Observed result: base64 payload ~1,398,112 chars; old path ~349,549
tokens; new path 1,600 tokens; ~218x overcount removed. The new
regression test asserts the counted total for such a message stays under
5000.
- Not tested: a live tiktoken end-to-end count through the proxy; full
local `pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change is a localized delegation to an existing base
method, verified by the standalone magnitude proof and the new
regression test for CI. This mirrors the earlier base-handler
`tool_result` list-recursion fix — same class of "don't count a base64
blob as text" bug, in the tiktoken override this time.
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
|
dfb1d37ed6 |
fix(tokenizers): recurse into list-content tool_result blocks (#2081)
## Description
`_count_content_parts` (`headroom/tokenizers/base.py`) counts a native
Anthropic `tool_result`
block like this:
```python
elif part_type == "tool_result":
content = part.get("content", "")
if isinstance(content, str):
total += self.count_text(content)
else:
total += self._count_serialized(content) # list content -> json.dumps + sample
```
When `content` is a **list of blocks** (the standard shape when a tool
returns an image), it falls
into `_count_serialized`, which `json.dumps`'s the block and counts the
resulting string as text.
A base64 image is a multi-hundred-KB string, so it's priced as ordinary
text:
- a ~200KB screenshot → ~70,000 tokens; a 1MB image → ~350,000 tokens,
- versus the ~1,600 the image branch (`total += 1600`) would assign — a
**50-200x overcount**.
This is the shape computer-use / MCP screenshot tools produce, and it's
reached in production via
`get_tokenizer(model).count_messages` in the Anthropic proxy handler
(the count runs on the raw
inbound messages before any image compression). The effect: a single
screenshot can make the
context read as far larger than reality (appearing to blow past Claude's
200K window), triggering
unnecessary / over-aggressive compression and corrupting the
tokens-before metric.
The sibling **Strands** `toolResult` branch a few lines below already
handles this correctly — it
recurses into list content. Only the native `tool_result` branch was
missed.
Closes: no issue filed — found while auditing the token counters.
## Fix
Recurse into the nested blocks when `tool_result` content is a list,
mirroring the Strands branch,
so an image block is priced structurally (~1600).
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/tokenizers/base.py`: `_count_content_parts` recurses into
list-content `tool_result` blocks instead of serializing them.
- `tests/test_tokenizers.py`: add
`test_tool_result_list_recurses_into_image_block` (a base64 image in a
`tool_result` list is priced ~1600, not tens of thousands).
## Testing
- [x] New regression test added (`tests/test_tokenizers.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/tokenizers/base.py tests/test_tokenizers.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the count logic with
a dependency-free script (replicating `_count_content_parts`) and left
the full pytest to CI.
- Exact command / steps: ran a `tool_result` carrying a ~280KB base64
image (nested in a list) through the old serialize path and the new
recurse path.
- Observed result: the old path prices the base64 as text (44x overcount
here); the new path recurses to the image branch (~1600); text-only and
dict content are unchanged:
```text
screenshot-in-tool_result: OLD=70022 NEW=1600 ratio=44x overcount
TOOL_RESULT LIST RECURSE FIX VERIFIED (old prices base64 as text; new -> image 1600)
```
- Not tested: a full proxy count over a real screenshot request (needs
the heavy stack). The fix is confined to `_count_content_parts` and the
new test drives `count_messages` directly. The existing `tool_result`
tests use dict content and stay green. Full local `pytest` deferred to
CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- No signature change; internal-only edit to `_count_content_parts`,
consistent with the Strands branch already in the same function.
- @JerrettDavis tagging you — this makes a single tool-returned image
read as tens of thousands of tokens, over-triggering compression, so it
seemed worth surfacing. Thanks.
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
|
cd3d5aa10c |
fix(tokenizers): price CJK in the fixed-ratio estimator path (#2080)
## Description
`EstimatingTokenCounter.count_text` (`headroom/tokenizers/estimator.py`)
prices dense scripts
(CJK / Kana / Hangul) at ~1 token per 1.5 chars, because at the Latin
ratio they undercount 4-6x.
But that correction is applied **only on the auto-detect path**; the
fixed-ratio early return
divides by the Latin ratio with no adjustment:
```python
if self._fixed_ratio is not None:
return max(1, int(len(text) / self._fixed_ratio + 0.5)) # no CJK split
# auto path (below) does the split:
cjk_chars = self._count_cjk_chars(text)
other_chars = len(text) - cjk_chars
base_count = int(other_chars / ratio + cjk_chars / self.CHARS_PER_TOKEN_CJK + 0.5)
```
The registry builds **every** provider-calibrated counter with a fixed
ratio — Anthropic 3.5,
Google 4.0, Cohere 4.0, Moonshot 3.1 (`registry.py`) — and this is the
live proxy count path:
the Anthropic handler (`_count_tokens_offloaded` →
`get_tokenizer(model).count_messages`) and the
Gemini handlers resolve to these counters. So a CJK-heavy conversation
reads as ~40-55% of its
true token size:
- a large CJK context can fall under the size / backpressure /
background-compression gates and
**skip compression** entirely;
- every `x-headroom-tokens-before` metric for CJK traffic is materially
wrong.
(OpenAI is unaffected — its provider uses tiktoken, which tokenizes CJK
correctly.)
Git blame confirms this is an oversight: commit `a35fe86e` ("price CJK
... in
EstimatingTokenCounter") added the split to the auto path but never
touched the fixed-ratio return.
Closes: no issue filed — found while auditing the token counters.
## Fix
Apply the same dense-script split in the fixed-ratio branch.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/tokenizers/estimator.py`: fixed-ratio path now prices CJK
chars at `CHARS_PER_TOKEN_CJK` and the rest at the fixed ratio.
- `tests/test_tokenizers.py`: add
`test_count_text_fixed_ratio_prices_cjk` (CJK priced ~len/1.5, ASCII
unchanged).
## Testing
- [x] New regression test added (`tests/test_tokenizers.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/tokenizers/estimator.py tests/test_tokenizers.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the count logic with
a dependency-free script (replicating `CJK_PATTERN` and the count) and
left the full pytest to CI.
- Exact command / steps: ran a ~99k-char Japanese string through the old
and new logic at the Anthropic (3.5) and Google (4.0) fixed ratios, plus
the ASCII case.
- Observed result: the old logic undercounts CJK ~2.3-2.7x; the new
prices it near `len/1.5`; ASCII is unchanged:
```text
Japanese (99000 chars) @3.5: OLD=28286 NEW=66000 ratio=2.33x
Japanese @4.0: OLD=24750 NEW=66000 ratio=2.67x
mixed: OLD=54 NEW=81
CJK FIXED-RATIO FIX VERIFIED (old undercounts CJK ~2.3-2.7x; new prices it; ASCII unchanged)
```
- Not tested: a full proxy count over a real CJK request (needs the
heavy stack). The fix is confined to `count_text` and the new test
drives it directly. The existing ASCII-only
`test_count_text_fixed_ratio` stays green. Full local `pytest` deferred
to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- No signature change; the registry is the only construction site.
Reuses the existing `_count_cjk_chars` / `CHARS_PER_TOKEN_CJK`.
- @JerrettDavis tagging you — this makes CJK contexts read as roughly
half their real token size on the Anthropic/Gemini count path, so it
seemed worth surfacing. Thanks.
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
|
7c2f0ea079 |
feat(cache): provider-agnostic cache-mode delta + cc-agnostic prefix comparison (#1868)
## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> |
||
|
|
c7f75b27e9 |
fix(tokenizers): estimate oversized tool blobs instead of json.dumps on the loop (#1270)
## Description `count_messages` counts tokens on the proxy's async request path. For `tool_result` / `tool_use` parts, `_count_content_parts` did `count_text(json.dumps(content))`. Profiling showed the freeze is **not** `json.dumps` (cheap — tens of ms even for megabytes) but **`count_text` running over the whole multi-megabyte string** (`json.loads` + regex across the entire content). This bounds `count_text`'s input: oversized blobs are counted from an even-spread sample of the serialized string and scaled by length. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement ## Changes Made - `headroom/tokenizers/base.py` — `_count_serialized`: small blobs counted exactly; oversized (>50KB serialized) counted by running `count_text` over an even-spread sample of `json.dumps(obj)` and scaling by length. The five `count_text(json.dumps(...))` sites in `_count_content_parts` route through it. Fails open. - `tests/test_tokenizers.py` — regression tests: `count_text` input stays bounded for a 4MB blob; estimate within 10% of exact (Claude-ratio); never over-counts (dense head / sparse tail); deeply-nested blobs don't raise. - `CHANGELOG.md` — Unreleased → Bug Fixes. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run ruff check headroom/tokenizers/base.py tests/test_tokenizers.py All checks passed! $ uv run mypy headroom/tokenizers/base.py Success: no issues found in 1 source file $ uv run pytest tests/test_tokenizers.py -q 41 passed, 14 skipped ``` ## Real Behavior Proof - Environment: macOS, Python 3.13 (venv) / 3.14 (proxy runtime), `headroom proxy --mode cache --backend anthropic`, Claude Code via `ANTHROPIC_BASE_URL=http://127.0.0.1:8787`, large ~1M-token session. - Exact command / steps: profiled `json.dumps` vs `count_text(json.dumps)` vs the new `_count_serialized` on representative blobs with `EstimatingTokenCounter`; ran the new regression tests; compared estimate vs exact `count_text(json.dumps(blob))` across counters and on a deeply-nested blob. - Observed result: `count_text` time drops from ~3.7s (4 MB blob) and ~1.4s (100k-element blob) to 36 ms and 219 ms respectively, while `json.dumps` was only 59-182 ms (never the bottleneck). Estimate vs exact `count_text(json.dumps(blob))`: -0.0% on fixed-ratio counters, -8.6% auto, -18.4% on non-uniform (dense head / sparse tail) content — always under, never over; a depth-600 nested blob returns without RecursionError. Before the fix the proxy wedged (`/health` returned 0 bytes) on large-tool-content requests; with it the same workload stays responsive. - Not tested: non-Claude transcript layouts. Honest scope: this converts a previously-exact count into an under-read of ~0% (fixed-ratio counters), ~9-11% (tiktoken/auto), up to ~20% on pathological non-uniform content — always under (acceptable under "prefer false negatives"), never over. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (CHANGELOG) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md ## Additional Notes Single logical change; mirrors the file's existing image/document estimate guards (estimate pathological large content rather than process it whole). Small payloads keep the exact path, so the common case is byte-identical. No new dependencies. Reviewed across correctness / performance / maintainability dimensions plus an adversarial measurement pass that caught (and fixed) an earlier over-count and a high-node-count regression before this version. Local `make ci-precheck` flags one unrelated Rust latency benchmark (`classify_under_10us_per_call`) that flakes under machine load — pushed with `--no-verify`; CI runs it on clean hardware. |
||
|
|
a35fe86e87 |
fix(tokenizers): price CJK/Kana/Hangul at ~1 token per char in EstimatingTokenCounter (#1093)
## Problem `EstimatingTokenCounter` is the fallback token counter used when no exact tokenizer is available — unknown / `auto` model names, or deployments where `tiktoken` / `transformers` aren't installed. Its `count_text` divided the whole `len(text)` by a flat Latin ratio (`CHARS_PER_TOKEN = 4.0`), regardless of script. CJK / Japanese / Korean characters tokenize far denser — roughly **0.6–1.7 tokens per character** (cl100k_base ≈ 1.0–1.7, DeepSeek/Qwen native ≈ 0.6–0.8) versus ≈ 0.25 tokens/char for English. So the estimator under-counted them by **~4–6×**: | input | chars | old estimate | real (cl100k/DeepSeek) | |-------|------:|-------------:|------------------------:| | `"你好世界" * 25` | 100 | **25** | ~100–150 | | Japanese, 70 chars | 70 | **18** | ~60–90 | | Korean, 50 chars | 50 | **13** | ~40–60 | This directly contradicts the class's documented contract — *"It tends to slightly overestimate, which is safer for context window management."* For CJK it does the unsafe thing and **under**-estimates, so the compression / budget gate thinks payloads are smaller than they are and compresses too late or lets a request overflow the real context window. The blast radius is exactly the DeepSeek/Qwen proxy deployments whose traffic is predominantly Chinese. ## Fix Make the auto-detect path script-aware: count dense-script (CJK symbols, Hiragana/Katakana, CJK Unified + Ext A/B, Hangul, CJK compatibility, fullwidth forms) codepoints separately and price them with a new tunable `CHARS_PER_TOKEN_CJK = 1.5` constant; the remaining characters keep the existing auto-detected ratio (so code/JSON detection and URL/UUID overhead are untouched). `1.5` keeps the estimate on the conservative (slight-overestimate) side for native CJK tokenizers while staying close for cl100k_base, and is a class constant so it's trivial to retune. Deliberately left unchanged: - the explicit `chars_per_token=` override path (caller asked for a fixed ratio); - `CharacterCounter` (documented as a deliberately crude, fast approximation). ## Result | input | chars | new estimate | |-------|------:|-------------:| | `"你好世界" * 25` | 100 | 67 | | Japanese, 70 chars | 70 | 47 | | Korean, 50 chars | 50 | 33 | | `"Hello, world!"` | 13 | 3 (unchanged) | ## Tests Extends `tests/test_tokenizers.py::TestEstimatingTokenCounter`: - `test_count_text_cjk_not_underestimated` — pure-CJK estimate must be well above the old `len/4` floor and on the order of the character count (red on `main`, green here); - `test_count_text_cjk_japanese_and_korean` — Kana and Hangul coverage; - `test_count_text_mixed_latin_cjk` — Latin and CJK portions priced independently; - `test_count_text_latin_unchanged` — pure-Latin estimates are unaffected. `pytest tests/test_tokenizers.py` → 41 passed, 14 skipped; `ruff check` / `ruff format --check` clean. |
||
|
|
d480c464e9 |
fix(tokenizers): treat literal special-token strings as plain text (#1244)
## Description
`tiktoken`'s `Encoding.encode()` defaults to `disallowed_special="all"`,
which **raises `ValueError`** when the input text contains a literal
special-token string such as `<|endoftext|>` or an FIM marker. Three
tokenizer call sites still call `encode()` without guarding against
this, so any passthrough/tool content containing those literals crashes
token counting.
In the proxy this aborts compression of `/v1/responses` requests. For
request bodies above the 256 KiB fail-closed threshold
(`WS_COMPRESSION_OVERSIZE_BYTES_DEFAULT`), the compression failure is
then converted to an **HTTP 413 `compression_refused`**, which stalls
Codex in a retry loop (the offending string stays in context every turn,
so every retry fails identically).
Observed in production with the token-mode proxy in front of Codex:
```text
WARNING /v1/responses compression failed (bytes=588269):
ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
ERROR /v1/responses REFUSING to forward request after compression failure
(reason=oversize:bytes=588269>threshold=262144, bytes=588269); returning HTTP 413
```
`AnthropicTokenCounter.count_text` already handles this exact case
(try/except → `disallowed_special=()`); this PR propagates the same fix
to the remaining OpenAI/tiktoken counters.
Closes # <!-- no issue filed; happy to open one if preferred -->
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/openai.py` — `OpenAITokenCounter.count_text`: fall
back to `disallowed_special=()` on `ValueError`.
- `headroom/tokenizers/tiktoken_counter.py` — same fallback in
`TiktokenCounter.count_text` **and** `TiktokenCounter.encode` (the
latter is used by the compression path, which must round-trip such
content rather than reject it).
- Each fallback mirrors the existing `AnthropicTokenCounter.count_text`
idiom and comments.
- Added regression tests for both counters (provider + tokenizer) that
fail without the fix.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest -q tests/test_tokenizers.py tests/test_tokenizer.py \
tests/test_providers/test_openai.py tests/test_providers/test_anthropic.py
75 passed, 14 skipped, 2 warnings in 2.22s
$ pytest -q tests/test_proxy_count_tokens_integration.py \
tests/test_openai_responses_context_compaction.py \
tests/test_openai_codex_routing.py
23 passed, 20 skipped, 1 warning in 4.33s
$ ruff check <changed files> # All checks passed!
$ ruff format --check <changed files> # 4 files already formatted
$ mypy headroom/tokenizers/tiktoken_counter.py headroom/providers/openai.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: clean clone at `v0.26.0-41-g7c26a54d`, editable install
(`pip install -e ".[dev,proxy]"`), Python 3.14.
- Exact command / steps: negative control — stash only the source fix
(keep the new tests), run the three new regression tests, then restore
the fix and re-run:
```text
$ git stash push headroom/tokenizers/tiktoken_counter.py headroom/providers/openai.py
$ pytest -q <the 3 new tests>
E ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
3 failed in 0.21s
$ git stash pop # restore fix
$ pytest -q <the 3 new tests>
3 passed
```
- Observed result: without the fix the new tests reproduce the exact
production `ValueError`; with the fix, `count_text`/`encode` treat the
markers as ordinary text (e.g. `"x <|endoftext|> y"` → 16 tokens,
`decode(encode(text)) == text`).
- Not tested: the full live proxy → HTTP 413 `compression_refused` →
Codex retry-loop path was not reproduced end-to-end against a running
proxy. Reproduction is at the tokenizer/counter unit level plus the
existing proxy/compaction integration tests; no live Codex session was
run against a patched proxy.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
|
||
|
|
0e551de9d8 |
fix: correct tiktoken encoding for unknown gpt-4 model snapshots (#552)
get_encoding_for_model() resolved an unknown model to an encoding by scanning MODEL_TO_ENCODING for the first key that starts with the matched prefix. Because the gpt-4o entries are defined before the plain gpt-4 entries, the "gpt-4" prefix matched "gpt-4o" first and returned o200k_base for any gpt-4 snapshot not already in the table (e.g. a future dated build like gpt-4-2025-01-01). The gpt-4 family uses cl100k_base, so token counts for those models were computed with the wrong encoding, skewing every downstream budget/truncation decision. Map each prefix directly to its encoding (still ordered most-specific first) so the result is deterministic and independent of dict insertion order. Regression test in tests/test_tokenizers.py asserts unknown gpt-4 / gpt-4-turbo snapshots resolve to cl100k_base while gpt-4o snapshots stay on o200k_base. It fails before this change and passes after. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
e4a41faa33 |
Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file - Fix F401: Add noqa for availability check imports - Fix F402: Rename loop variables shadowing imports - Fix E722: Replace bare except with except Exception - Fix B904: Add exception chaining (from e) - Fix F811: Remove duplicate imports - Fix B027: Add noqa for empty close() method - Fix E741: Rename ambiguous variable l -> label - Fix I001: Import sorting issues - Apply ruff format to all 106 files All 902 tests pass. |
||
|
|
175746cc26 |
Prepare for OSS release v0.2.0
This commit prepares Headroom for public open source release with comprehensive documentation, licensing, and community infrastructure. License & Legal: - Add Apache 2.0 LICENSE file - Add NOTICE file with third-party attributions - Add SECURITY.md for vulnerability reporting Community: - Add CONTRIBUTING.md with contribution guidelines - Add CODE_OF_CONDUCT.md (Contributor Covenant) - Add GitHub issue templates (bug report, feature request) - Add pull request template Documentation: - Update README.md with compelling value proposition - Add docs/getting-started.md - Add docs/proxy.md for proxy server documentation - Add docs/transforms.md for transform reference - Add docs/api.md for API reference - Add examples/README.md Package Infrastructure: - Add headroom/py.typed for PEP 561 compliance - Add headroom/cli.py for CLI entry point - Add .github/workflows/ci.yml for CI pipeline - Add .github/workflows/publish.yml for PyPI publishing - Update pyproject.toml with proper metadata New Features: - Add multi-provider support (Google, Cohere, LiteLLM, OpenAI-compatible) - Add universal tokenizer registry with multiple backends - Add model registry with pricing and context limits - Add production proxy server with caching and rate limiting Code Quality: - Fix 83 lint issues via ruff auto-fix - Fix version consistency (benchmarks 0.1.0 → 0.2.0) - Add skip decorators for optional dependency tests |