mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(tokenizers): unwrap BatchEncoding in HuggingFaceTokenizer.count_messages (#2447)
## Description With a recent `transformers`, `apply_chat_template(..., tokenize=True)` returns a `BatchEncoding` (dict-like with `input_ids`/`attention_mask`), so `len(formatted)` in `HuggingFaceTokenizer.count_messages()` yields the key count — **2** — for any conversation. The proxy then computes `original_tokens = 2`, the inflation guard in the OpenAI-compatible handler (`optimized_tokens > original_tokens`) always fires, and **every compression is silently reverted** (0% savings, `transforms=none`, no error surfaced) for HF-tokenized models such as DeepSeek and Qwen. The fix unwraps `input_ids` when a dict-like is returned, keeping the legacy flat-list path intact. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/tokenizers/huggingface.py`: in `count_messages`, return `len(formatted["input_ids"])` when `apply_chat_template` returns a `BatchEncoding`/dict-like; fall through to `len(formatted)` for older transformers that return a flat token list. - `tests/test_huggingface_batchencoding_count.py`: regression tests for both return shapes (BatchEncoding-like dict and legacy flat list). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_huggingface_batchencoding_count.py tests/test_tokenizers.py tests/test_huggingface_tokenizer_timeout.py -q 62 passed, 14 skipped, 1 warning in 0.64s $ ruff check headroom/tokenizers/huggingface.py tests/test_huggingface_batchencoding_count.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.12, headroom-ai 0.32.1 from PyPI (`pip install "headroom-ai[all]"`, July 2026), `headroom proxy --mode token --openai-api-url https://api.deepseek.com`, model `deepseek-chat`. - Exact command / steps: 13-message conversation (~25 KB system prompt + three ~50 KB JSON tool results) sent to `/v1/chat/completions` through the proxy; before/after compared via proxy.log. - Observed result: before the fix the proxy logs `Optimization inflated tokens (2 -> 28539), reverting to original messages` and `tok_saved=0 transforms=none`; after the fix the same request logs `tok_before=45944 tok_after=33299 tok_saved=12927 transforms=router:smart_crusher:0.53*3` — DeepSeek-billed prompt tokens drop 46209 → 33299 (−28%). Direct repro: `get_tokenizer("deepseek-chat").count_messages([{"role":"system","content":"word "*5000}])` returns 2 before, correct count after. - Not tested: `mypy` (fails in my env on an unrelated numpy stub error before reaching this file); other HF-tokenized providers beyond DeepSeek. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Documentation: N/A — internal token-counting fix, no user-facing behavior change beyond restoring the documented compression. - An alternative is `apply_chat_template(..., return_dict=False)`, but the explicit unwrap is robust across transformers versions in both directions. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2447" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `010881052c7afd41a30566a3cde3bf374a775564`, upstream main `0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`. Clean integration against current upstream/main0ddaa56c. Contributor HuggingFaceTokenizer.count_messages AST unchanged; main return_dict=False preserved. Small maintainer test-only repair adds return_dict to both fake apply_chat_template signatures, retaining original1234/777expected counts. Before repair both regressions failed because unexpected keyword triggered base fallback; after repair317tests pass,11skip,1warning across HF counts/timeout/allowlist/tokenizer and incoming-main regression coverage. Both changed Python files pass Rufflint/format,candidate offline uv lock and whitespace pass. Complete tracked native inputset/everyrecordedhash/artifactchecksum verified before3982DLL reuse; no freshbuild. Independent paired local-object probe uses installed transformers5.13.1 BatchEncoding(input_ids1234,attention_mask1234), returned by a synthetic adapter. Branch counts1234; cleanmain counts2mappingkeys. Probe confirms caller supplied tokenize=True/add_generation_prompt=True/return_dict=False. This is defensive handling when an adapter still returns a mapping; it does NOT establish that a normal current transformers tokenizer honoring return_dict=False remains broken. No model/tokenizer download, actual HF model inference, live proxy optimization-reversion or provider-token savings claim. Read author discussion and outstanding Devin fixture-signature report; the maintainer repair addresses that integration blocker without changing contributor production logic or weakening assertions. Fresh prepublication audit has no new external findings. Only two source/test files differ from main; no workflow/setup/dependency/native/CLI delta, central security publication gate passes. Exact published-head CI/Actions/rollup, current-main ancestry,mergeability and fresh discussion remain before approval. Maintainers handle CI; no PR merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. --------- Co-authored-by: Никита Паевщиков <nbpaevshchikov@sberbank.ru> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Никита Паевщиков
JD Davis
parent
3871b3664a
commit
9eed58f84c
@@ -458,6 +458,11 @@ class HuggingFaceTokenizer(BaseTokenizer):
|
||||
add_generation_prompt=True,
|
||||
return_dict=False,
|
||||
)
|
||||
# Recent transformers return a BatchEncoding (dict-like with
|
||||
# input_ids/attention_mask) here; len() of it is the key count
|
||||
# (2), not the token count.
|
||||
if hasattr(formatted, "keys") and "input_ids" in formatted:
|
||||
return len(formatted["input_ids"])
|
||||
return len(formatted)
|
||||
except Exception:
|
||||
# Fall back to base implementation
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Regression test: count_messages must unwrap BatchEncoding from apply_chat_template.
|
||||
|
||||
Recent transformers versions return a BatchEncoding (dict-like with
|
||||
``input_ids``/``attention_mask``) from ``apply_chat_template(tokenize=True)``.
|
||||
``len()`` of it is the number of keys (2), not the token count, which made
|
||||
every proxy request look like optimization "inflated" tokens (2 -> N) and
|
||||
silently reverted all compression on HF-tokenized models (DeepSeek, Qwen, ...).
|
||||
"""
|
||||
|
||||
from unittest.mock import patch
|
||||
|
||||
from headroom.tokenizers.huggingface import HuggingFaceTokenizer
|
||||
|
||||
|
||||
class _BatchEncodingLike(dict):
|
||||
"""Mimics transformers.BatchEncoding: dict of lists."""
|
||||
|
||||
|
||||
class _FakeInnerTokenizer:
|
||||
def apply_chat_template(
|
||||
self, messages, tokenize=True, add_generation_prompt=True, return_dict=False
|
||||
):
|
||||
return _BatchEncodingLike(
|
||||
input_ids=list(range(1234)),
|
||||
attention_mask=[1] * 1234,
|
||||
)
|
||||
|
||||
def encode(self, text, add_special_tokens=False):
|
||||
return [0] * max(1, len(text) // 4)
|
||||
|
||||
|
||||
def _make_tokenizer(inner):
|
||||
with patch.object(HuggingFaceTokenizer, "__init__", lambda self, model: None):
|
||||
tok = HuggingFaceTokenizer.__new__(HuggingFaceTokenizer)
|
||||
tok.model = "deepseek-chat"
|
||||
tok.tokenizer_name = "fake"
|
||||
tok._tokenizer = inner
|
||||
return tok
|
||||
|
||||
|
||||
def test_count_messages_unwraps_batch_encoding():
|
||||
tok = _make_tokenizer(_FakeInnerTokenizer())
|
||||
messages = [{"role": "system", "content": "word " * 5000}]
|
||||
assert tok.count_messages(messages) == 1234
|
||||
|
||||
|
||||
def test_count_messages_accepts_flat_token_list():
|
||||
class _LegacyInner(_FakeInnerTokenizer):
|
||||
def apply_chat_template(
|
||||
self, messages, tokenize=True, add_generation_prompt=True, return_dict=False
|
||||
):
|
||||
return list(range(777))
|
||||
|
||||
tok = _make_tokenizer(_LegacyInner())
|
||||
messages = [{"role": "user", "content": "hello"}]
|
||||
assert tok.count_messages(messages) == 777
|
||||
Reference in New Issue
Block a user