fix(tokenizers): unwrap BatchEncoding in HuggingFaceTokenizer.count_messages (#2447)

## Description

With a recent `transformers`, `apply_chat_template(..., tokenize=True)`
returns a `BatchEncoding` (dict-like with `input_ids`/`attention_mask`),
so `len(formatted)` in `HuggingFaceTokenizer.count_messages()` yields
the key count — **2** — for any conversation. The proxy then computes
`original_tokens = 2`, the inflation guard in the OpenAI-compatible
handler (`optimized_tokens > original_tokens`) always fires, and **every
compression is silently reverted** (0% savings, `transforms=none`, no
error surfaced) for HF-tokenized models such as DeepSeek and Qwen.

The fix unwraps `input_ids` when a dict-like is returned, keeping the
legacy flat-list path intact.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/tokenizers/huggingface.py`: in `count_messages`, return
`len(formatted["input_ids"])` when `apply_chat_template` returns a
`BatchEncoding`/dict-like; fall through to `len(formatted)` for older
transformers that return a flat token list.
- `tests/test_huggingface_batchencoding_count.py`: regression tests for
both return shapes (BatchEncoding-like dict and legacy flat list).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_huggingface_batchencoding_count.py tests/test_tokenizers.py tests/test_huggingface_tokenizer_timeout.py -q
62 passed, 14 skipped, 1 warning in 0.64s

$ ruff check headroom/tokenizers/huggingface.py tests/test_huggingface_batchencoding_count.py
All checks passed!
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.12, headroom-ai 0.32.1 from PyPI
(`pip install "headroom-ai[all]"`, July 2026), `headroom proxy --mode
token --openai-api-url https://api.deepseek.com`, model `deepseek-chat`.
- Exact command / steps: 13-message conversation (~25 KB system prompt +
three ~50 KB JSON tool results) sent to `/v1/chat/completions` through
the proxy; before/after compared via proxy.log.
- Observed result: before the fix the proxy logs `Optimization inflated
tokens (2 -> 28539), reverting to original messages` and `tok_saved=0
transforms=none`; after the fix the same request logs `tok_before=45944
tok_after=33299 tok_saved=12927 transforms=router:smart_crusher:0.53*3`
— DeepSeek-billed prompt tokens drop 46209 → 33299 (−28%). Direct repro:
`get_tokenizer("deepseek-chat").count_messages([{"role":"system","content":"word
"*5000}])` returns 2 before, correct count after.
- Not tested: `mypy` (fails in my env on an unrelated numpy stub error
before reaching this file); other HF-tokenized providers beyond
DeepSeek.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: N/A — internal token-counting fix, no user-facing
behavior change beyond restoring the documented compression.
- An alternative is `apply_chat_template(..., return_dict=False)`, but
the explicit unwrap is robust across transformers versions in both
directions.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2447"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `010881052c7afd41a30566a3cde3bf374a775564`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration against current upstream/main0ddaa56c. Contributor
HuggingFaceTokenizer.count_messages AST unchanged; main
return_dict=False preserved. Small maintainer test-only repair adds
return_dict to both fake apply_chat_template signatures, retaining
original1234/777expected counts. Before repair both regressions failed
because unexpected keyword triggered base fallback; after repair317tests
pass,11skip,1warning across HF counts/timeout/allowlist/tokenizer and
incoming-main regression coverage. Both changed Python files pass
Rufflint/format,candidate offline uv lock and whitespace pass. Complete
tracked native inputset/everyrecordedhash/artifactchecksum verified
before3982DLL reuse; no freshbuild.

Independent paired local-object probe uses installed transformers5.13.1
BatchEncoding(input_ids1234,attention_mask1234), returned by a synthetic
adapter. Branch counts1234; cleanmain counts2mappingkeys. Probe confirms
caller supplied
tokenize=True/add_generation_prompt=True/return_dict=False. This is
defensive handling when an adapter still returns a mapping; it does NOT
establish that a normal current transformers tokenizer honoring
return_dict=False remains broken. No model/tokenizer download, actual HF
model inference, live proxy optimization-reversion or provider-token
savings claim.

Read author discussion and outstanding Devin fixture-signature report;
the maintainer repair addresses that integration blocker without
changing contributor production logic or weakening assertions. Fresh
prepublication audit has no new external findings. Only two source/test
files differ from main; no workflow/setup/dependency/native/CLI delta,
central security publication gate passes. Exact published-head
CI/Actions/rollup, current-main ancestry,mergeability and fresh
discussion remain before approval. Maintainers handle CI; no PR merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: Никита Паевщиков <nbpaevshchikov@sberbank.ru>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
N
2026-11-06 21:31:15 -06:00
committed by GitHub
co-authored by Никита Паевщиков JD Davis
parent 3871b3664a
commit 9eed58f84c
2 changed files with 61 additions and 0 deletions
+5
View File
@@ -458,6 +458,11 @@ class HuggingFaceTokenizer(BaseTokenizer):
add_generation_prompt=True,
return_dict=False,
)
# Recent transformers return a BatchEncoding (dict-like with
# input_ids/attention_mask) here; len() of it is the key count
# (2), not the token count.
if hasattr(formatted, "keys") and "input_ids" in formatted:
return len(formatted["input_ids"])
return len(formatted)
except Exception:
# Fall back to base implementation
@@ -0,0 +1,56 @@
"""Regression test: count_messages must unwrap BatchEncoding from apply_chat_template.
Recent transformers versions return a BatchEncoding (dict-like with
``input_ids``/``attention_mask``) from ``apply_chat_template(tokenize=True)``.
``len()`` of it is the number of keys (2), not the token count, which made
every proxy request look like optimization "inflated" tokens (2 -> N) and
silently reverted all compression on HF-tokenized models (DeepSeek, Qwen, ...).
"""
from unittest.mock import patch
from headroom.tokenizers.huggingface import HuggingFaceTokenizer
class _BatchEncodingLike(dict):
"""Mimics transformers.BatchEncoding: dict of lists."""
class _FakeInnerTokenizer:
def apply_chat_template(
self, messages, tokenize=True, add_generation_prompt=True, return_dict=False
):
return _BatchEncodingLike(
input_ids=list(range(1234)),
attention_mask=[1] * 1234,
)
def encode(self, text, add_special_tokens=False):
return [0] * max(1, len(text) // 4)
def _make_tokenizer(inner):
with patch.object(HuggingFaceTokenizer, "__init__", lambda self, model: None):
tok = HuggingFaceTokenizer.__new__(HuggingFaceTokenizer)
tok.model = "deepseek-chat"
tok.tokenizer_name = "fake"
tok._tokenizer = inner
return tok
def test_count_messages_unwraps_batch_encoding():
tok = _make_tokenizer(_FakeInnerTokenizer())
messages = [{"role": "system", "content": "word " * 5000}]
assert tok.count_messages(messages) == 1234
def test_count_messages_accepts_flat_token_list():
class _LegacyInner(_FakeInnerTokenizer):
def apply_chat_template(
self, messages, tokenize=True, add_generation_prompt=True, return_dict=False
):
return list(range(777))
tok = _make_tokenizer(_LegacyInner())
messages = [{"role": "user", "content": "hello"}]
assert tok.count_messages(messages) == 777