fix(thinking): don't read the model's date suffix as its minor version (#3791)

## Description

`bills_prior_thinking(model)` decides whether a model re-bills
prior-turn thinking (so compaction pays off) or strips it server-side
(where compacting would turn *free* stripped thinking into *billed*
text). It parses the version from the model id by collecting **every**
contiguous digit group, then compares `(major, minor) >= (4, 6)`:

```python
nums = []
for part in model.lower().split("-"):
    if part.isdigit():
        nums.append(int(part))
    elif nums:
        break
major = nums[0]
minor = nums[1] if len(nums) > 1 else 0
return major >= 5 or (major, minor) >= (4, 6)
```

For a **major-only** id with a release-date suffix — e.g.
`claude-sonnet-4-20250514`, `claude-opus-4-20250514` — the 8-digit date
is collected as a version component: `nums = [4, 20250514]`, so `(4,
20250514) >= (4, 6)` → **True**. The gate is inverted for these ids.

Claude 4.0 strips prior-turn thinking server-side, so the gate must
return `False`. Returning `True` makes `compact_thinking_to_text` /
`compact_reasoning_openai_chat` rewrite those requests and convert the
free (stripped) thinking into billed text — the exact cost regression
the function's own docstring warns about ("compacting on a stripping
model would turn free (stripped) thinking into billed text") — on
current Claude 4.0 model ids. The in-module `_demo` self-check never
exercised a major-only id, so it missed this.

Confirmed by running the function:

```
bills_prior_thinking("claude-sonnet-4-20250514")  -> True   (should be False)
bills_prior_thinking("claude-opus-4-20250514")    -> True   (should be False)
# unaffected: 4-5-...->False, 4-1-...->False, opus-4-8->True, sonnet-5->True
```

Fix: only treat one- or two-digit groups as version components (major,
optional minor). A version is always 1–2 digits; the `YYYYMMDD` suffix
is 8, so it can never be mistaken for the minor.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/thinking_compactor.py`: `bills_prior_thinking`
now collects a digit group as a version component only when it is `<= 2`
digits, so an 8-digit `YYYYMMDD` release-date suffix is not read as the
minor version.
- `tests/test_thinking_compactor_version_gate.py` (new): parametrized
version-gate coverage — major-only-with-date ids
(`claude-sonnet-4-20250514` → False), the 4.5/4.6 boundary, the 5
family, legacy 3.x ids, and a plus a focused regression asserting the
date suffix is not read as the minor.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_thinking_compactor_version_gate.py  ->  16 passed
uvx ruff@0.16.8 check headroom/transforms/thinking_compactor.py tests/test_thinking_compactor_version_gate.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/transforms/thinking_compactor.py  ->  clean (no error in this file)
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.8 and mypy 1.20.2 via uvx.
- Exact command / steps: called `bills_prior_thinking` across
major-only-with-date, explicit-minor, 4.6+, 5-family, and legacy 3.x
ids. Before the fix, `claude-sonnet-4-20250514` and
`claude-opus-4-20250514` returned `True`; after, they return `False`
while every other case is unchanged. Red/green: with the fix reverted
the two date-suffix cases and the regression test fail; restored, all 16
pass.
- Observed result: the reasoning compactor no longer engages on Claude
4.0 ids (which strip prior thinking server-side), so it can't convert
free stripped thinking into billed text; 4.6+/5-family ids still gate
on.
- Not tested: a live billed request against the Anthropic API (the cost
impact follows directly from the gate; the gate logic is covered by unit
tests).

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: only for Claude 4.0 model ids
carrying a date suffix, which now correctly read as "strips prior
thinking" (gate False) instead of "bills" (gate True). Ids with an
explicit minor and the 4.6+/5 family are unchanged.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit; the date suffix is again read as
the minor version.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal gate; docstring already describes the intended behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

The `elif nums: break` still stops at the family/date boundary; the only
change is that an over-long (date) digit group no longer counts as a
version component before that break.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Abhay Singh
2026-09-30 11:09:56 -05:00
committed by GitHub
co-authored by JD Davis
parent 717527bc86
commit afaaaa8804
2 changed files with 58 additions and 1 deletions
+6 -1
View File
@@ -59,7 +59,12 @@ def bills_prior_thinking(model: str) -> bool:
"""
nums: list[int] = []
for part in model.lower().split("-"):
if part.isdigit():
# Version components are one or two digits (major, optional minor). Longer
# runs are the release-date suffix (``YYYYMMDD``) — treating it as the
# minor version made a major-only id like ``claude-sonnet-4-20250514``
# read as ``(4, 20250514) >= (4, 6)`` -> True, inverting the gate for
# Claude 4.0 (which strips prior thinking server-side).
if part.isdigit() and len(part) <= 2:
nums.append(int(part))
elif nums:
break # version digits are contiguous; stop at the family/date boundary
@@ -0,0 +1,52 @@
"""Version-gate tests for thinking_compactor.bills_prior_thinking.
The gate decides whether a model re-bills prior-turn thinking (so compaction
pays) or strips it server-side (so compacting would turn free thinking into
billed text). It parses the version out of the model id, and must not mistake
the ``YYYYMMDD`` release-date suffix for the minor version.
"""
from __future__ import annotations
import pytest
from headroom.transforms.thinking_compactor import bills_prior_thinking
@pytest.mark.parametrize(
("model", "expected"),
[
# Major-only ids with a date suffix: the date must NOT be read as the
# minor version. Claude 4.0 strips prior thinking server-side -> False.
("claude-sonnet-4-20250514", False),
("claude-opus-4-20250514", False),
("claude-opus-4", False),
# Explicit minor below the 4.6 threshold -> strips -> False.
("claude-sonnet-4-5-20250929", False),
("claude-haiku-4-5-20251001", False),
("claude-opus-4-1-20250805", False),
("claude-opus-4-5-20250101", False),
# 4.6+ and the 5 family re-bill prior thinking -> True.
("claude-opus-4-6", True),
("claude-sonnet-4-6", True),
("claude-opus-4-8", True),
("claude-sonnet-5", True),
("claude-sonnet-5-20260101", True),
# Legacy 3.x naming (with date) stays below threshold.
("claude-3-5-sonnet-20241022", False),
("claude-3-opus-20240229", False),
# No version at all -> conservative False.
("some-unversioned-model", False),
],
)
def test_bills_prior_thinking_version_gate(model: str, expected: bool) -> None:
assert bills_prior_thinking(model) is expected
def test_date_suffix_is_not_read_as_minor_version() -> None:
"""Regression for the specific inversion: a major-only id whose 8-digit date
suffix was parsed as the minor version, flipping a strip-model to 'bills'."""
# Same major (4), only the date suffix differs from an explicit-minor id.
assert bills_prior_thinking("claude-sonnet-4-20250514") is False
# Sanity: the real 4.6 id still bills.
assert bills_prior_thinking("claude-sonnet-4-6-20260101") is True