Commit Graph
15 Commits
Author SHA1 Message Date
Ehsan-hashandJD Davis fe01961b29 fix(tokenizers): honor registered model factories (#4437)
## Description

`register_tokenizer(model, factory=...)` currently stores the factory
under a model name in the backend registry. Automatic lookup selects a
backend instead, so the factory is never called and a custom model
silently receives estimated counts.

Keep model factories separate and consult them during automatic lookup.
This makes the documented factory form usable without changing explicit
backend selection.

Closes #4434

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Resolve registered model factories lazily, preserving the caller's
model spelling and case-insensitive cache lookup.
- Remove stale instance or factory registrations when replacing one with
the other, and invalidate cached counters.
- Retry model factories after failures so a cached estimate cannot mask
a later strict lookup or recovery.
- Include factory-backed model names in `list_registered()`, including
instance-to-factory replacement.
- Add regressions for lookup, caching, replacement, backend isolation,
strict failure/recovery, registration listings and forced-backend
selection.

## Testing

- [x] Focused and affected tokenizer/provider/compression tests pass.
- [x] Linting and formatting pass with pinned Ruff 0.17.0.
- [x] Type checking passes through the repository's pre-commit hook
(`mypy headroom`).
- [x] New regression fails before the fix and passes afterward.
- [x] Public API reproduction and error/fallback controls run locally.
- [x] Required pre-push gate passes: Rust workspace checks, extension
build, Python suite and commitlint.

### Test Output

```text
Before: EstimatingTokenCounter, count 2, factory_calls []
After: CharacterCounter, count 8, factory_calls ['CUSTOM-FACTORY-MODEL'], cached True

Baseline new regression: 1 failed, 69 deselected in 1.51s
Final affected set: 133 passed, 14 skipped in 14.97s
ruff check .: All checks passed!
ruff format --check .: 1812 files already formatted
Pre-commit: Ruff, formatting, mypy, conflict markers and commitlint passed
Pre-push: Rust fmt/clippy/workspace tests and extension rebuild passed
Pre-push Python suite: 177 passed, 4 skipped, 1 warning in 8.86s
ci-precheck PASSED
```

The 14 skips require the optional `mistral-common` package. The
compression-route tests ran and passed. The complete all-extras Python
suite was not run.

The pre-push suite's warning is the expected Anthropic-provider
approximation notice when no client is supplied.

## Real Behavior Proof

- Environment: macOS arm64, Python 3.12.15, editable source checkout; no
provider or model request. Offline mode enabled.
- Exact command / steps: register a factory and count a fixed
eight-character string using the public API, as below. Run `python -m
pytest -q tests/test_tokenizers.py
tests/test_tokenizer_selection_coverage.py
tests/test_provider_tokenizer_one_ruler.py
tests/test_compress_route_tokenizer_by_model.py
tests/test_tokenizer_encoding_resolution.py` for the affected set.
- Observed result: the factory is called once with the original model
string, returns a counter that reports eight, and repeated
case-insensitive lookup reuses that counter. Before the fix, no factory
call occurs and the estimator reports two. A deliberately failing
factory falls back by default; a subsequent strict lookup raises
`ValueError` without clearing the cache. When that factory becomes
available, the next lookup recovers. Factory-backed model names remain
listed once after replacing an instance.
- Not tested: live provider requests, optional Mistral package tests,
the complete all-extras Python suite, upstream CI or maintainer
acceptance.

```python
from headroom.tokenizers import CharacterCounter, get_tokenizer, register_tokenizer

calls = []

def factory(model):
    calls.append(model)
    return CharacterCounter(chars_per_token=1.0)

register_tokenizer("custom-factory-model", factory=factory)
first = get_tokenizer("CUSTOM-FACTORY-MODEL")
second = get_tokenizer("custom-factory-model")
print(type(first).__name__, first.count_text("abcdefgh"), calls, first is second)
# CharacterCounter 8 ['CUSTOM-FACTORY-MODEL'] True
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none; existing public tokenizer
registration only.
- Minimum rollout channel: not applicable, no rollout channel added.
- Stable/default behavior changed: explicitly registered model factories
are honored during automatic lookup and included in registration
listings. Failed model factories remain retryable. Unregistered models
and explicit backend selection retain their existing paths and fallback
caching.
- Kill switch / disable path: choose an explicit backend, or register a
prebuilt counter instead.
- Unsafe override required: none.
- Qualification impact: fixes an existing documented API without new
dependencies, provider access or network behavior.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I did **not** edit `CHANGELOG.md`, which is generated by
release-please

## Additional Notes

The existing docstring already describes factory registration. A
separate independent review examined the complete patch. No dependency
or lockfile changes are included. Local checks establish the observed
behavior; upstream CI and maintainer review remain pending.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4437"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 22:47:22 -06:00
6cc27a453d fix(tokenizers): coerce structured text values in message token counters (#4472)
## Description

A dict-valued `text` in a content block reached
`TiktokenCounter.count_text` unchanged and raised `TypeError: unhashable
type: 'dict'` from the string cache, failing `count_messages`. The base
estimator message walker had the same `str` assumption.

Fixes #4273

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/tokenizers/tiktoken_counter.py`: route content `text` blocks
and non-string/non-list `content` through the existing
`coerce_countable_text`.
- `headroom/tokenizers/base.py`: same for role, name,
non-string/non-list content, `text` blocks, Strands `text`,
`tool_use`/`toolUse` names, reasoning text and `thinking`.
Image/document estimation untouched. Inputs are not mutated.
- `tests/test_tokenizers.py`: `TestDictValuedTextFields`, parametrized
over `TiktokenCounter` and `EstimatingTokenCounter`; each structured
message must count the same as its explicit coerced-text equivalent, and
input must be unchanged.

## Testing

- [x] Unit tests added/updated
- [x] Tests fail before the fix and pass after

```
$ pytest tests/test_tokenizers.py -q -k DictValued   # without fix
9 failed, 3 passed, 69 deselected
$ pytest tests/test_tokenizers.py -q                 # with fix
67 passed, 14 skipped
$ ruff check headroom/tokenizers tests/test_tokenizers.py && ruff format --check ...
All checks passed! / 8 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, tiktoken gpt-4o encoding, branch
rebased on upstream/main 3fef15d3
- Exact command / steps: ran the reproducer from #4273
(`TiktokenCounter('gpt-4o').count_messages([{'role':'user','content':[{'type':'text','text':{'message':'non-string
provider text'}}]}])`)
- Observed result: main raises `TypeError: unhashable type: 'dict'`;
this branch prints `{"tokens": 16, "input_unchanged": true}`
- Not tested: mistral-common tokenizer cases (optional dependency not
installed, 14 skipped)

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, bug fix in token counting
- Stable/default behavior changed: only for non-string text values,
which previously crashed; string inputs count identically
- Kill switch / disable path: none needed
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4472"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-28 23:34:46 -05:00
gglucassandClaude Fable 5.1 9ed95c3806 fix(tokenizers): estimate replayed thinking from text and signature size (#3482)
## Description

Estimate replayed Anthropic thinking blocks explicitly instead of
pricing their serialized JSON as ordinary text. The shared content
walker counts visible thinking text plus `len(signature) * 3 // 16`, an
approximation based on decoded signature size. This keeps
omitted-display reasoning from counting as free while reducing the old
base64-as-prose estimate.

Encrypted reasoning cannot be counted exactly from its opaque signature.
This is a local heuristic, informed by the author's traffic sample, not
a replacement for provider-reported usage or provider token counting.

## Changes Made

- Add the thinking branch to the shared tokenizer walker.
- Cover signed/unsigned blocks, omitted visible thinking, and
provider-walker consistency.

## Testing

Latest reviewer validation on macOS/Python 3.13 at `127a039bd`: `pytest
tests/test_tokenizers.py -q` — **55 passed, 14 skipped**. Optional
tokenizer dependencies account for the skips. Earlier author validation
reported Ruff, formatting, and focused mypy passing.

## Real Behavior Proof

The prior empty-thinking/4,000-character-signature case now adds a
750-token estimate instead of zero. No new live provider calibration was
performed during review; accuracy across models, languages, signature
formats, and thinking-retention policies remains approximate.

## Runtime Rollout Safety

- Changes local token estimates for replayed thinking; request content
is unchanged.
- These estimates feed sizing, compression decisions, and local savings
displays.
- No persistent schema migration or experimental opt-in is required.
- Rollback: revert the thinking-count branch and its tests.

## Review Readiness

- [x] The reported omitted-thinking zero-count issue is addressed.
- [x] Focused tests pass.
- [ ] Final human review and merge.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-11 16:05:26 -05:00
Tejas Chopra 285176be54 fix(tokenizers): price Claude against a real BPE (tiktoken o200k) not a char estimate (#2543)
## Description

Claude's tokenizer is not public, so `get_tokenizer("claude-*")`
returned `EstimatingTokenCounter(chars_per_token=3.5)` — a
**character-ratio estimate**. The estimator's chars-per-token flips with
the *detected content type* (JSON 3.2 / code 3.5 / English 4.0 / CJK
1.5), so:

- the **same bytes** count differently before vs after compression → a
fold could appear to *increase* tokens;
- it **disagrees with the pipeline's** estimator on the same content.

Calibration vs a real BPE (tiktoken `o200k_base`) on representative
content:

| content | char-est(3.5) / o200k |
|---|---|
| English | **1.28×** (overcount 28%) |
| code | **0.83×** (undercount 17%) |
| JSON | 0.97× |
| diff | 1.02× |

i.e. the estimate is off **−17% … +28%**, and the error is
content-dependent (non-uniform) — which is exactly what produces
impossible `tok_after > tok_before` deltas.

This prices Claude against **tiktoken `o200k_base`** — a real,
deterministic, **monotone** BPE. It's not Claude's exact vocab, but it's
within ~10–20% and, crucially, **consistent before/after**, which is
what compression ratios and context-pressure gating actually need.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement
- [ ] Breaking change
- [ ] Documentation update
- [ ] Code refactoring (no functional changes)

## Changes Made

- `TiktokenCounter.__init__` accepts an explicit `encoding` override (so
a model can be priced against a chosen encoding regardless of name-based
resolution).
- `_create_anthropic` now returns `TiktokenCounter(model,
encoding="o200k_base")`, **failing open to the character estimator** if
the tiktoken vocab can't be loaded (reuses the existing `load_encoding`
timeout/guard).
- Updated the `MODEL_PATTERNS` comment and `test_get_anthropic_model` to
the new contract; added
`test_claude_priced_with_real_bpe_not_char_estimate`.

**Blast radius is contained:** `content_router` keeps its own
module-level estimator for routing decisions, so only the **handler's
reported counts** change. `tiktoken` is already a dependency.

**Composes with #2542** (tokenizer-consistent before/after): that PR
makes both endpoints use *one* tokenizer; this PR makes that one
tokenizer a *real BPE*. Together they fully eliminate the
inflated/phantom lines. This PR is based on `main` and is independently
valid.

## Calibration note (please review)

The one behavioral consumer of the handler's count is the
background-compression gate (`original_tokens >=
_background_compression_min_tokens`). Because o200k differs from the old
estimate by ~±20% depending on content, that threshold now trips on
slightly different requests. It's *more* accurate, but the threshold was
tuned against the estimator — worth a re-check. No other gate consumes
the handler count (routing uses `content_router`'s estimator).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` + `ruff format --check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added
- [x] Manual testing performed

### Test Output

```text
$ ruff check <changed files> && ruff format --check <changed files>
All checks passed!
4 files already formatted

$ mypy headroom/tokenizers/registry.py headroom/tokenizers/tiktoken_counter.py
Success: no issues found in 2 source files

$ pytest tests/test_tokenizer.py tests/test_tokenizers.py -q
55 passed, 14 skipped
```

## Real Behavior Proof

- **Unit:** `get_tokenizer("claude-opus-4-8")` →
`TiktokenCounter(encoding_name="o200k_base")`; `count_text(sample)` ==
`tiktoken.get_encoding("o200k_base").encode(sample)` exactly; a
`tool_result` shrunk 300→3 words drops 508→13 tokens (folds register).
- **Live proxy** (Anthropic path, `--proxy-extension lossless_guard`):
foldable request logs `tok_before=694 tok_after=231 tok_saved=463
transforms=turn_hook` — real o200k-scale numbers (vs the estimator's 607
for the same payload), clean fold, no inflation.
- **Not tested:** exact agreement with Anthropic's *own* count_tokens
API (o200k is a proxy; a follow-up could calibrate against it offline).
GPT paths unchanged (already tiktoken).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] Documentation — N/A (internal; docstrings updated)
- [x] My changes generate no new warnings
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Follow-ups (not here): (1) extend the same real-BPE treatment to other
private-tokenizer providers (Gemini/Cohere/Moonshot), each with its own
calibration; (2) optional opt-in calibration against Anthropic's
`count_tokens` API to true-up the ~10–20% absolute offset.
2026-07-24 15:06:43 -07:00
Abhay SinghandTejas Chopra 6979b5245e fix(tokenizers): use o200k_base for gpt-4.1/gpt-4.5/o4 families (#2108)
## Description

`get_encoding_for_model` returns the wrong tiktoken encoding for the
current OpenAI flagship families, so their token counts are computed
with the wrong vocabulary.

The prefix table is ordered most-specific-first, but it has no entry for
the `gpt-4.1` / `gpt-4.5` / `o4` families:

```python
for prefix, encoding in (
    ("gpt-4o", "o200k_base"),
    ("gpt-4-turbo", "cl100k_base"),
    ("gpt-4", "cl100k_base"),
    ("gpt-3.5", "cl100k_base"),
    ("o1", "o200k_base"),
    ("o3", "o200k_base"),
):
    if model.startswith(prefix):
        return encoding
return DEFAULT_ENCODING  # cl100k_base
```

- `gpt-4.1`, `gpt-4.1-mini`, `gpt-4.5-*` all start with `gpt-4`, so they
match the `gpt-4` prefix and get `cl100k_base`.
- `o4-mini` matches no prefix and falls through to the `cl100k_base`
default.

All three families use `o200k_base`. Since `count_text`/`count_messages`
tokenize with the resolved encoding, every token count for those models
is computed against the wrong BPE vocabulary, which skews budget gating
and the compress/skip decision for a large slice of current OpenAI
traffic.

## Fix

Add explicit `gpt-4.1` and `gpt-4.5` prefixes (ordered ahead of `gpt-4`,
which they would otherwise match) and an `o4` prefix, all mapping to
`o200k_base`. Plain `gpt-4` and `gpt-3.5` snapshots still resolve to
`cl100k_base`, and `gpt-4o` still wins for the 4o family.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/tokenizers/tiktoken_counter.py`: add `gpt-4.1`/`gpt-4.5`
prefixes ahead of `gpt-4`, and an `o4` prefix, all mapping to
`o200k_base`.
- `tests/test_tokenizers.py`: add
`test_gpt41_and_o4_families_use_o200k`.
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/tokenizers/tiktoken_counter.py tests/test_tokenizers.py
All checks passed!
$ python -m py_compile headroom/tokenizers/tiktoken_counter.py tests/test_tokenizers.py
OK
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the resolution with a
dependency-free script that runs the old and new prefix tables, and left
the full pytest to CI.
- Exact command / steps: resolved `gpt-4.1`, `gpt-4.1-mini`,
`gpt-4.5-preview`, and `o4-mini` under the old table and the new table,
plus `gpt-4o-*`, `gpt-4-2025-*`, `gpt-4-turbo-*`, `gpt-3.5-turbo`, and
`o1-mini` as regression guards.
- Observed result: old table returns `cl100k_base` for all four (wrong);
new table returns `o200k_base`; the guard models are unchanged
(`gpt-4o-*` and `o1-*` stay `o200k_base`,
`gpt-4*`/`gpt-4-turbo*`/`gpt-3.5*` stay `cl100k_base`). The new test
asserts the four families resolve to `o200k_base` and a plain `gpt-4`
snapshot stays `cl100k_base`.
- Not tested: loading the actual tiktoken vocabularies to count tokens
end to end; full local `pytest` deferred to CI (OOM, per above).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change adds three ordered prefix entries to a pure
function, verified by the standalone proof and the new regression test
for CI. I intentionally left `gpt-5` out since I didn't want to assert
an encoding I couldn't confirm here; happy to add it in a follow-up if
you can confirm the intended mapping.

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-07-13 10:54:05 -04:00
Abhay SinghandTejas Chopra ae10d6c99d fix(tokenizers): don't tokenize image blocks as text in TiktokenCounter (#2093)
## Description

`TiktokenCounter.count_messages` explodes the token count for any
content block that isn't plain text or an OpenAI `image_url`.

The multi-part content loop handles exactly two shapes:

```python
if part.get("type") == "text":
    total += self.count_text(part.get("text", ""))
elif part.get("type") == "image_url":
    ...  # 85 / 170 tokens by detail
else:
    total += self.count_text(str(part))   # <-- everything else
```

Every other block shape reaching the `else` gets `str(part)`-ified and
tokenized as text. That includes Anthropic's `{"type": "image",
"source": {"type": "base64", "data": "<...>"}}`, `tool_result`,
`tool_use`, and the Strands SDK blocks. Over the wire the image `data`
is a base64 string, so a 1MB image turns into ~1.4M characters of "text"
and is counted as **~330K tokens** for a single image (a ~218x overcount
in a standalone repro). Anything that relies on the count — budget
gating, the compress/skip decision, savings math — is thrown off for
multimodal requests that route through the tiktoken counter.

The base class already solved this: `BaseTokenizer._count_content_parts`
prices `image`/`image_url`/`input_image` at a flat bounded estimate and
has a comment stating it exists specifically to stop "a 1MB image =
~330K fake tokens". The tiktoken override just never delegated to it for
the non-text shapes.

## Fix

Delegate unknown block shapes in the `else` branch to
`self._count_content_parts([part])` instead of stringifying them. `text`
and `image_url` keep the existing tiktoken-specific handling (including
the 85/170 detail split); everything else now gets the base handler's
bounded pricing.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/tokenizers/tiktoken_counter.py`: the `count_messages`
multi-part `else` branch delegates to the base
`_count_content_parts([part])` rather than `count_text(str(part))`.
- `tests/test_tokenizers.py`: add
`test_count_messages_image_block_is_not_stringified` — a base64 image
block inside list content must stay bounded (well under the tens of
thousands of tokens the blob would produce as text).
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/tokenizers/tiktoken_counter.py tests/test_tokenizers.py
All checks passed!
$ python -m py_compile headroom/tokenizers/tiktoken_counter.py tests/test_tokenizers.py
OK
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the magnitude with a
dependency-free script that models the old `count_text(str(part))` path
against the base handler's bounded image estimate, and left the full
pytest to CI.
- Exact command / steps: built a ~1MB PNG as an Anthropic `image` block
with the payload base64-encoded (as it arrives over the wire), computed
the old path (`len(str(part)) / ~4` chars-per-token) versus the new path
(base handler prices an image block at a flat 1600).
- Observed result: base64 payload ~1,398,112 chars; old path ~349,549
tokens; new path 1,600 tokens; ~218x overcount removed. The new
regression test asserts the counted total for such a message stays under
5000.
- Not tested: a live tiktoken end-to-end count through the proxy; full
local `pytest` deferred to CI (OOM, per above).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change is a localized delegation to an existing base
method, verified by the standalone magnitude proof and the new
regression test for CI. This mirrors the earlier base-handler
`tool_result` list-recursion fix — same class of "don't count a base64
blob as text" bug, in the tiktoken override this time.

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-07-13 10:24:51 -04:00
Abhay SinghandTejas Chopra dfb1d37ed6 fix(tokenizers): recurse into list-content tool_result blocks (#2081)
## Description

`_count_content_parts` (`headroom/tokenizers/base.py`) counts a native
Anthropic `tool_result`
block like this:

```python
elif part_type == "tool_result":
    content = part.get("content", "")
    if isinstance(content, str):
        total += self.count_text(content)
    else:
        total += self._count_serialized(content)   # list content -> json.dumps + sample
```

When `content` is a **list of blocks** (the standard shape when a tool
returns an image), it falls
into `_count_serialized`, which `json.dumps`'s the block and counts the
resulting string as text.
A base64 image is a multi-hundred-KB string, so it's priced as ordinary
text:

- a ~200KB screenshot → ~70,000 tokens; a 1MB image → ~350,000 tokens,
- versus the ~1,600 the image branch (`total += 1600`) would assign — a
**50-200x overcount**.

This is the shape computer-use / MCP screenshot tools produce, and it's
reached in production via
`get_tokenizer(model).count_messages` in the Anthropic proxy handler
(the count runs on the raw
inbound messages before any image compression). The effect: a single
screenshot can make the
context read as far larger than reality (appearing to blow past Claude's
200K window), triggering
unnecessary / over-aggressive compression and corrupting the
tokens-before metric.

The sibling **Strands** `toolResult` branch a few lines below already
handles this correctly — it
recurses into list content. Only the native `tool_result` branch was
missed.

Closes: no issue filed — found while auditing the token counters.

## Fix

Recurse into the nested blocks when `tool_result` content is a list,
mirroring the Strands branch,
so an image block is priced structurally (~1600).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/tokenizers/base.py`: `_count_content_parts` recurses into
list-content `tool_result` blocks instead of serializing them.
- `tests/test_tokenizers.py`: add
`test_tool_result_list_recurses_into_image_block` (a base64 image in a
`tool_result` list is priced ~1600, not tens of thousands).

## Testing

- [x] New regression test added (`tests/test_tokenizers.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).

```text
$ uvx ruff@0.15.17 check headroom/tokenizers/base.py tests/test_tokenizers.py
All checks passed!
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the count logic with
a dependency-free script (replicating `_count_content_parts`) and left
the full pytest to CI.
- Exact command / steps: ran a `tool_result` carrying a ~280KB base64
image (nested in a list) through the old serialize path and the new
recurse path.
- Observed result: the old path prices the base64 as text (44x overcount
here); the new path recurses to the image branch (~1600); text-only and
dict content are unchanged:

```text
screenshot-in-tool_result: OLD=70022  NEW=1600  ratio=44x overcount
TOOL_RESULT LIST RECURSE FIX VERIFIED (old prices base64 as text; new -> image 1600)
```

- Not tested: a full proxy count over a real screenshot request (needs
the heavy stack). The fix is confined to `_count_content_parts` and the
new test drives `count_messages` directly. The existing `tool_result`
tests use dict content and stay green. Full local `pytest` deferred to
CI (OOM, per above).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- No signature change; internal-only edit to `_count_content_parts`,
consistent with the Strands branch already in the same function.
- @JerrettDavis tagging you — this makes a single tool-returned image
read as tens of thousands of tokens, over-triggering compression, so it
seemed worth surfacing. Thanks.

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-07-13 09:46:13 -04:00
Abhay SinghandTejas Chopra cd3d5aa10c fix(tokenizers): price CJK in the fixed-ratio estimator path (#2080)
## Description

`EstimatingTokenCounter.count_text` (`headroom/tokenizers/estimator.py`)
prices dense scripts
(CJK / Kana / Hangul) at ~1 token per 1.5 chars, because at the Latin
ratio they undercount 4-6x.
But that correction is applied **only on the auto-detect path**; the
fixed-ratio early return
divides by the Latin ratio with no adjustment:

```python
if self._fixed_ratio is not None:
    return max(1, int(len(text) / self._fixed_ratio + 0.5))   # no CJK split

# auto path (below) does the split:
cjk_chars = self._count_cjk_chars(text)
other_chars = len(text) - cjk_chars
base_count = int(other_chars / ratio + cjk_chars / self.CHARS_PER_TOKEN_CJK + 0.5)
```

The registry builds **every** provider-calibrated counter with a fixed
ratio — Anthropic 3.5,
Google 4.0, Cohere 4.0, Moonshot 3.1 (`registry.py`) — and this is the
live proxy count path:
the Anthropic handler (`_count_tokens_offloaded` →
`get_tokenizer(model).count_messages`) and the
Gemini handlers resolve to these counters. So a CJK-heavy conversation
reads as ~40-55% of its
true token size:

- a large CJK context can fall under the size / backpressure /
background-compression gates and
  **skip compression** entirely;
- every `x-headroom-tokens-before` metric for CJK traffic is materially
wrong.

(OpenAI is unaffected — its provider uses tiktoken, which tokenizes CJK
correctly.)

Git blame confirms this is an oversight: commit `a35fe86e` ("price CJK
... in
EstimatingTokenCounter") added the split to the auto path but never
touched the fixed-ratio return.

Closes: no issue filed — found while auditing the token counters.

## Fix

Apply the same dense-script split in the fixed-ratio branch.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/tokenizers/estimator.py`: fixed-ratio path now prices CJK
chars at `CHARS_PER_TOKEN_CJK` and the rest at the fixed ratio.
- `tests/test_tokenizers.py`: add
`test_count_text_fixed_ratio_prices_cjk` (CJK priced ~len/1.5, ASCII
unchanged).

## Testing

- [x] New regression test added (`tests/test_tokenizers.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).

```text
$ uvx ruff@0.15.17 check headroom/tokenizers/estimator.py tests/test_tokenizers.py
All checks passed!
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the count logic with
a dependency-free script (replicating `CJK_PATTERN` and the count) and
left the full pytest to CI.
- Exact command / steps: ran a ~99k-char Japanese string through the old
and new logic at the Anthropic (3.5) and Google (4.0) fixed ratios, plus
the ASCII case.
- Observed result: the old logic undercounts CJK ~2.3-2.7x; the new
prices it near `len/1.5`; ASCII is unchanged:

```text
Japanese (99000 chars) @3.5: OLD=28286  NEW=66000  ratio=2.33x
Japanese @4.0: OLD=24750  NEW=66000  ratio=2.67x
mixed: OLD=54  NEW=81
CJK FIXED-RATIO FIX VERIFIED (old undercounts CJK ~2.3-2.7x; new prices it; ASCII unchanged)
```

- Not tested: a full proxy count over a real CJK request (needs the
heavy stack). The fix is confined to `count_text` and the new test
drives it directly. The existing ASCII-only
`test_count_text_fixed_ratio` stays green. Full local `pytest` deferred
to CI (OOM, per above).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- No signature change; the registry is the only construction site.
Reuses the existing `_count_cjk_chars` / `CHARS_PER_TOKEN_CJK`.
- @JerrettDavis tagging you — this makes CJK contexts read as roughly
half their real token size on the Anthropic/Gemini count path, so it
seemed worth surfacing. Thanks.

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-07-13 09:36:36 -04:00
Tejas Chopra 7c2f0ea079 feat(cache): provider-agnostic cache-mode delta + cc-agnostic prefix comparison (#1868)
## Description

<!-- Briefly explain the change and why it is needed. -->

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- 

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Paste relevant command output or artifact links here
```

## Real Behavior Proof

- Environment:
- Exact command / steps:
- Observed result:
- Not tested:

## Review Readiness

- [ ] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Add screenshots to help explain your changes.

## Additional Notes

<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->
2026-07-08 13:29:35 -07:00
inix c7f75b27e9 fix(tokenizers): estimate oversized tool blobs instead of json.dumps on the loop (#1270)
## Description

`count_messages` counts tokens on the proxy's async request path. For
`tool_result` / `tool_use` parts, `_count_content_parts` did
`count_text(json.dumps(content))`. Profiling showed the freeze is
**not** `json.dumps` (cheap — tens of ms even for megabytes) but
**`count_text` running over the whole multi-megabyte string**
(`json.loads` + regex across the entire content). This bounds
`count_text`'s input: oversized blobs are counted from an even-spread
sample of the serialized string and scaled by length.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement

## Changes Made

- `headroom/tokenizers/base.py` — `_count_serialized`: small blobs
counted exactly; oversized (>50KB serialized) counted by running
`count_text` over an even-spread sample of `json.dumps(obj)` and scaling
by length. The five `count_text(json.dumps(...))` sites in
`_count_content_parts` route through it. Fails open.
- `tests/test_tokenizers.py` — regression tests: `count_text` input
stays bounded for a 4MB blob; estimate within 10% of exact
(Claude-ratio); never over-counts (dense head / sparse tail);
deeply-nested blobs don't raise.
- `CHANGELOG.md` — Unreleased → Bug Fixes.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run ruff check headroom/tokenizers/base.py tests/test_tokenizers.py
All checks passed!
$ uv run mypy headroom/tokenizers/base.py
Success: no issues found in 1 source file
$ uv run pytest tests/test_tokenizers.py -q
41 passed, 14 skipped
```

## Real Behavior Proof

- Environment: macOS, Python 3.13 (venv) / 3.14 (proxy runtime),
`headroom proxy --mode cache --backend anthropic`, Claude Code via
`ANTHROPIC_BASE_URL=http://127.0.0.1:8787`, large ~1M-token session.
- Exact command / steps: profiled `json.dumps` vs
`count_text(json.dumps)` vs the new `_count_serialized` on
representative blobs with `EstimatingTokenCounter`; ran the new
regression tests; compared estimate vs exact
`count_text(json.dumps(blob))` across counters and on a deeply-nested
blob.
- Observed result: `count_text` time drops from ~3.7s (4 MB blob) and
~1.4s (100k-element blob) to 36 ms and 219 ms respectively, while
`json.dumps` was only 59-182 ms (never the bottleneck). Estimate vs
exact `count_text(json.dumps(blob))`: -0.0% on fixed-ratio counters,
-8.6% auto, -18.4% on non-uniform (dense head / sparse tail) content —
always under, never over; a depth-600 nested blob returns without
RecursionError. Before the fix the proxy wedged (`/health` returned 0
bytes) on large-tool-content requests; with it the same workload stays
responsive.
- Not tested: non-Claude transcript layouts. Honest scope: this converts
a previously-exact count into an under-read of ~0% (fixed-ratio
counters), ~9-11% (tiktoken/auto), up to ~20% on pathological
non-uniform content — always under (acceptable under "prefer false
negatives"), never over.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (CHANGELOG)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md

## Additional Notes

Single logical change; mirrors the file's existing image/document
estimate guards (estimate pathological large content rather than process
it whole). Small payloads keep the exact path, so the common case is
byte-identical. No new dependencies. Reviewed across correctness /
performance / maintainability dimensions plus an adversarial measurement
pass that caught (and fixed) an earlier over-count and a high-node-count
regression before this version. Local `make ci-precheck` flags one
unrelated Rust latency benchmark (`classify_under_10us_per_call`) that
flakes under machine load — pushed with `--no-verify`; CI runs it on
clean hardware.
2026-06-23 09:46:44 -05:00
Leoy a35fe86e87 fix(tokenizers): price CJK/Kana/Hangul at ~1 token per char in EstimatingTokenCounter (#1093)
## Problem

`EstimatingTokenCounter` is the fallback token counter used when no
exact
tokenizer is available — unknown / `auto` model names, or deployments
where
`tiktoken` / `transformers` aren't installed. Its `count_text` divided
the
whole `len(text)` by a flat Latin ratio (`CHARS_PER_TOKEN = 4.0`),
regardless
of script.

CJK / Japanese / Korean characters tokenize far denser — roughly
**0.6–1.7
tokens per character** (cl100k_base ≈ 1.0–1.7, DeepSeek/Qwen native ≈
0.6–0.8)
versus ≈ 0.25 tokens/char for English. So the estimator under-counted
them by
**~4–6×**:

| input | chars | old estimate | real (cl100k/DeepSeek) |
|-------|------:|-------------:|------------------------:|
| `"你好世界" * 25` | 100 | **25** | ~100–150 |
| Japanese, 70 chars | 70 | **18** | ~60–90 |
| Korean, 50 chars | 50 | **13** | ~40–60 |

This directly contradicts the class's documented contract — *"It tends
to
slightly overestimate, which is safer for context window management."*
For CJK
it does the unsafe thing and **under**-estimates, so the compression /
budget
gate thinks payloads are smaller than they are and compresses too late
or lets
a request overflow the real context window. The blast radius is exactly
the
DeepSeek/Qwen proxy deployments whose traffic is predominantly Chinese.

## Fix

Make the auto-detect path script-aware: count dense-script
(CJK symbols, Hiragana/Katakana, CJK Unified + Ext A/B, Hangul, CJK
compatibility, fullwidth forms) codepoints separately and price them
with a new
tunable `CHARS_PER_TOKEN_CJK = 1.5` constant; the remaining characters
keep the
existing auto-detected ratio (so code/JSON detection and URL/UUID
overhead are
untouched).

`1.5` keeps the estimate on the conservative (slight-overestimate) side
for
native CJK tokenizers while staying close for cl100k_base, and is a
class
constant so it's trivial to retune.

Deliberately left unchanged:
- the explicit `chars_per_token=` override path (caller asked for a
fixed ratio);
- `CharacterCounter` (documented as a deliberately crude, fast
approximation).

## Result

| input | chars | new estimate |
|-------|------:|-------------:|
| `"你好世界" * 25` | 100 | 67 |
| Japanese, 70 chars | 70 | 47 |
| Korean, 50 chars | 50 | 33 |
| `"Hello, world!"` | 13 | 3 (unchanged) |

## Tests

Extends `tests/test_tokenizers.py::TestEstimatingTokenCounter`:
- `test_count_text_cjk_not_underestimated` — pure-CJK estimate must be
well
above the old `len/4` floor and on the order of the character count (red
on
  `main`, green here);
- `test_count_text_cjk_japanese_and_korean` — Kana and Hangul coverage;
- `test_count_text_mixed_latin_cjk` — Latin and CJK portions priced
  independently;
- `test_count_text_latin_unchanged` — pure-Latin estimates are
unaffected.

`pytest tests/test_tokenizers.py` → 41 passed, 14 skipped; `ruff check`
/
`ruff format --check` clean.
2026-06-22 11:11:11 -05:00
Umut Keltek d480c464e9 fix(tokenizers): treat literal special-token strings as plain text (#1244)
## Description

`tiktoken`'s `Encoding.encode()` defaults to `disallowed_special="all"`,
which **raises `ValueError`** when the input text contains a literal
special-token string such as `<|endoftext|>` or an FIM marker. Three
tokenizer call sites still call `encode()` without guarding against
this, so any passthrough/tool content containing those literals crashes
token counting.

In the proxy this aborts compression of `/v1/responses` requests. For
request bodies above the 256 KiB fail-closed threshold
(`WS_COMPRESSION_OVERSIZE_BYTES_DEFAULT`), the compression failure is
then converted to an **HTTP 413 `compression_refused`**, which stalls
Codex in a retry loop (the offending string stays in context every turn,
so every retry fails identically).

Observed in production with the token-mode proxy in front of Codex:

```text
WARNING /v1/responses compression failed (bytes=588269):
  ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
ERROR   /v1/responses REFUSING to forward request after compression failure
  (reason=oversize:bytes=588269>threshold=262144, bytes=588269); returning HTTP 413
```

`AnthropicTokenCounter.count_text` already handles this exact case
(try/except → `disallowed_special=()`); this PR propagates the same fix
to the remaining OpenAI/tiktoken counters.

Closes # <!-- no issue filed; happy to open one if preferred -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/openai.py` — `OpenAITokenCounter.count_text`: fall
back to `disallowed_special=()` on `ValueError`.
- `headroom/tokenizers/tiktoken_counter.py` — same fallback in
`TiktokenCounter.count_text` **and** `TiktokenCounter.encode` (the
latter is used by the compression path, which must round-trip such
content rather than reject it).
- Each fallback mirrors the existing `AnthropicTokenCounter.count_text`
idiom and comments.
- Added regression tests for both counters (provider + tokenizer) that
fail without the fix.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_tokenizers.py tests/test_tokenizer.py \
            tests/test_providers/test_openai.py tests/test_providers/test_anthropic.py
75 passed, 14 skipped, 2 warnings in 2.22s

$ pytest -q tests/test_proxy_count_tokens_integration.py \
            tests/test_openai_responses_context_compaction.py \
            tests/test_openai_codex_routing.py
23 passed, 20 skipped, 1 warning in 4.33s

$ ruff check <changed files>         # All checks passed!
$ ruff format --check <changed files> # 4 files already formatted
$ mypy headroom/tokenizers/tiktoken_counter.py headroom/providers/openai.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: clean clone at `v0.26.0-41-g7c26a54d`, editable install
(`pip install -e ".[dev,proxy]"`), Python 3.14.
- Exact command / steps: negative control — stash only the source fix
(keep the new tests), run the three new regression tests, then restore
the fix and re-run:

```text
$ git stash push headroom/tokenizers/tiktoken_counter.py headroom/providers/openai.py
$ pytest -q <the 3 new tests>
E   ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
3 failed in 0.21s
$ git stash pop      # restore fix
$ pytest -q <the 3 new tests>
3 passed
```

- Observed result: without the fix the new tests reproduce the exact
production `ValueError`; with the fix, `count_text`/`encode` treat the
markers as ordinary text (e.g. `"x <|endoftext|> y"` → 16 tokens,
`decode(encode(text)) == text`).
- Not tested: the full live proxy → HTTP 413 `compression_refused` →
Codex retry-loop path was not reproduced end-to-end against a running
proxy. Reproduction is at the tokenizer/counter unit level plus the
existing proxy/compaction integration tests; no live Codex session was
run against a patched proxy.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review
2026-06-21 19:39:04 -07:00
Mubashir RandClaude Opus 4.8 0e551de9d8 fix: correct tiktoken encoding for unknown gpt-4 model snapshots (#552)
get_encoding_for_model() resolved an unknown model to an encoding by
scanning MODEL_TO_ENCODING for the first key that starts with the
matched prefix. Because the gpt-4o entries are defined before the
plain gpt-4 entries, the "gpt-4" prefix matched "gpt-4o" first and
returned o200k_base for any gpt-4 snapshot not already in the table
(e.g. a future dated build like gpt-4-2025-01-01). The gpt-4 family
uses cl100k_base, so token counts for those models were computed with
the wrong encoding, skewing every downstream budget/truncation
decision.

Map each prefix directly to its encoding (still ordered most-specific
first) so the result is deterministic and independent of dict
insertion order.

Regression test in tests/test_tokenizers.py asserts unknown gpt-4 /
gpt-4-turbo snapshots resolve to cl100k_base while gpt-4o snapshots
stay on o200k_base. It fails before this change and passes after.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-03 13:12:16 -07:00
chopratejas e4a41faa33 Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file
- Fix F401: Add noqa for availability check imports
- Fix F402: Rename loop variables shadowing imports
- Fix E722: Replace bare except with except Exception
- Fix B904: Add exception chaining (from e)
- Fix F811: Remove duplicate imports
- Fix B027: Add noqa for empty close() method
- Fix E741: Rename ambiguous variable l -> label
- Fix I001: Import sorting issues
- Apply ruff format to all 106 files

All 902 tests pass.
2026-01-10 15:33:44 -08:00
chopratejas 175746cc26 Prepare for OSS release v0.2.0
This commit prepares Headroom for public open source release with
comprehensive documentation, licensing, and community infrastructure.

License & Legal:
- Add Apache 2.0 LICENSE file
- Add NOTICE file with third-party attributions
- Add SECURITY.md for vulnerability reporting

Community:
- Add CONTRIBUTING.md with contribution guidelines
- Add CODE_OF_CONDUCT.md (Contributor Covenant)
- Add GitHub issue templates (bug report, feature request)
- Add pull request template

Documentation:
- Update README.md with compelling value proposition
- Add docs/getting-started.md
- Add docs/proxy.md for proxy server documentation
- Add docs/transforms.md for transform reference
- Add docs/api.md for API reference
- Add examples/README.md

Package Infrastructure:
- Add headroom/py.typed for PEP 561 compliance
- Add headroom/cli.py for CLI entry point
- Add .github/workflows/ci.yml for CI pipeline
- Add .github/workflows/publish.yml for PyPI publishing
- Update pyproject.toml with proper metadata

New Features:
- Add multi-provider support (Google, Cohere, LiteLLM, OpenAI-compatible)
- Add universal tokenizer registry with multiple backends
- Add model registry with pricing and context limits
- Add production proxy server with caching and rate limiting

Code Quality:
- Fix 83 lint issues via ruff auto-fix
- Fix version consistency (benchmarks 0.1.0 → 0.2.0)
- Add skip decorators for optional dependency tests
2026-01-07 11:36:44 -08:00