fix(gateway): keep input_includes_cache when re-drive usage is merged (#4423)

## Description

`NormalizedUsage.merged_with` and `summed_with` dropped
`input_includes_cache`, so after a re-drive round an OpenAI-shaped usage
was billed as if `input_tokens` excluded the cache, and the cached
tokens were counted twice in the recorded outcome. This carries the flag
through both merges. A usage block with no cache fields carries no
accounting dialect (`normalize_usage` defaults it to inclusive), so the
merge takes the flag from the latest round that reported cache fields
instead of OR-ing the rounds.

Fixes #4414

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Pass `input_includes_cache` through `merged_with` and `summed_with`,
chosen by a small helper `_merged_includes_cache`: the latest round that
reported cache fields decides (`headroom/proxy/gateway_turn.py`).
- Add `test_redrive_merges_keep_input_includes_cache` and
`test_redrive_merges_keep_anthropic_dialect_after_cacheless_round`
(cacheless then cached Anthropic rounds, both orders) to
`tests/test_gateway_turn_unit.py`.
- Second commit addresses the review comment on the first (Devin,
comment 4349159780): the first version OR-ed the flag, which marked a
cacheless-then-cached Anthropic turn as inclusive.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# new tests, source change reverted (state of the first commit, OR-ed flag)
E   assert True is False
E    +  where True = NormalizedUsage(input_tokens=110, output_tokens=10, cache_read=80, cache_write=None, input_includes_cache=True).input_includes_cache
1 failed, 47 passed

# with the fix: tests/test_gateway_turn_unit.py
48 passed

# with the fix: tests/test_gateway_turn_unit.py tests/gateway/test_redrive_loop.py tests/gateway/test_contract_response_half.py tests/gateway/test_usage_relay_openai_shape.py
110 passed, 1 warning

ruff check: All checks passed!
ruff format --check: 2 files already formatted
```

## Real Behavior Proof

- Environment: Ubuntu (WSL2), Python 3.12.15, upstream main e995eb9f.
`headroom/_core.abi3.so` from the official headroom-ai 0.41.0 wheel was
copied in (untracked, not committed). Real proxy on 127.0.0.1:18890; no
provider is contacted (the client plays the gateway and posts canned
OpenAI-shaped responses).
- Exact command / steps: start `python -P -m headroom.proxy.server
--host 127.0.0.1 --port 18890 --no-cache --no-rate-limit` with a
throwaway extension whose turn hook calls `call_model` once in
`on_response`; a client posts `/v1/compress` (gateway block with
`can_redrive`, `can_relay_response`, `session_affinity`) and then
`/v1/compress/response` for round 0 and round 1 with
`usage={"prompt_tokens":100000,"completion_tokens":10,"prompt_tokens_details":{"cached_tokens":80000}}`;
read `/stats`.
- Observed result: OpenAI-shaped turn (two rounds of `prompt_tokens`
100000, `cached_tokens` 80000): before the original fix
`cost.total_input_tokens 360000`,
`prefix_cache.totals.uncached_input_tokens 200000`; with the fix
`200000` and `40000`; `cache_read_tokens` 160000 in both.
Anthropic-shaped turn (round 0
`{"input_tokens":100000,"output_tokens":10}`, round 1
`{"input_tokens":10000,"output_tokens":10,"cache_read_input_tokens":80000}`):
with the first commit (OR-ed flag) `cost.total_input_tokens 110000`,
`uncached_input_tokens 30000`; with this commit `190000` and `110000`;
`cache_read_tokens` 80000 in both. The OpenAI-shaped run was repeated on
this commit and still gives `200000` / `40000`.
- Not tested: a real OpenAI-compatible or Anthropic gateway or provider;
streaming turns; a turn that mixes provider dialects between rounds; a
Rust core built from this checkout.

```text
== OpenAI shape, BEFORE the original fix
round 1 -> done billed_usage: {'input_tokens': 200000, 'output_tokens': 20, 'cache_read_input_tokens': 160000}
stats .prefix_cache.totals.cache_read_tokens 160000
stats .prefix_cache.totals.uncached_input_tokens 200000
stats .cost.total_input_tokens 360000
== OpenAI shape, AFTER (this commit)
stats .prefix_cache.totals.cache_read_tokens 160000
stats .prefix_cache.totals.uncached_input_tokens 40000
stats .cost.total_input_tokens 200000
== Anthropic shape, cacheless round then cached round, first commit (OR-ed flag)
stats .prefix_cache.totals.cache_read_tokens 80000
stats .prefix_cache.totals.uncached_input_tokens 30000
stats .cost.total_input_tokens 110000
== Anthropic shape, this commit
stats .prefix_cache.totals.cache_read_tokens 80000
stats .prefix_cache.totals.uncached_input_tokens 110000
stats .cost.total_input_tokens 190000
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: recorded input and uncached-input
counters for re-driven OpenAI-shaped gateway turns are no longer
inflated
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none known
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Only the four listed test files were run, not the whole suite. The
dialect is taken from the latest round that reported cache fields; a
turn that mixes provider dialects between rounds is not handled and is
not expected from a single gateway. Produced with AI assistance (Claude
Code) operated by me (HetCreep); the reproduction was run on my machine
on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
HetCreep
2026-10-24 08:44:12 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent bf34ccf588
commit 3f941446bf
2 changed files with 50 additions and 0 deletions
+11
View File
@@ -257,6 +257,15 @@ class NormalizedUsage:
return True
return self.cache_read is not None and self.cache_write is not None
def _merged_includes_cache(self, other: NormalizedUsage) -> bool:
"""Accounting dialect across rounds. A usage block without cache fields
carries no dialect (``normalize_usage`` defaults it to inclusive, which
is wrong for a cacheless Anthropic round), so the round that reported
cache fields decides; the latest such round wins."""
if other.has_cache_signal or not self.has_cache_signal:
return other.input_includes_cache
return self.input_includes_cache
def merged_with(self, other: NormalizedUsage) -> NormalizedUsage:
"""Sum the billed counters across re-drive rounds; cache fields come
from the latest round that carried them (the last provider call is the
@@ -272,6 +281,7 @@ class NormalizedUsage:
output_tokens=_sum(self.output_tokens, other.output_tokens),
cache_read=other.cache_read if other.has_cache_signal else self.cache_read,
cache_write=other.cache_write if other.has_cache_signal else self.cache_write,
input_includes_cache=self._merged_includes_cache(other),
)
def summed_with(self, other: NormalizedUsage) -> NormalizedUsage:
@@ -290,6 +300,7 @@ class NormalizedUsage:
output_tokens=_sum(self.output_tokens, other.output_tokens),
cache_read=_sum(self.cache_read, other.cache_read),
cache_write=_sum(self.cache_write, other.cache_write),
input_includes_cache=self._merged_includes_cache(other),
)
def as_anthropic(self) -> dict[str, int]:
+39
View File
@@ -530,6 +530,45 @@ def test_merged_with_sums_billed_and_keeps_latest_cache() -> None:
assert c.cache_read == 60 # a signal-free round does not erase the last signal
def test_redrive_merges_keep_input_includes_cache() -> None:
"""Re-drive rounds of an OpenAI-shaped turn must not be billed as Anthropic-shaped."""
first = normalize_usage(
{
"prompt_tokens": 100_000,
"completion_tokens": 10,
"prompt_tokens_details": {"cached_tokens": 80_000},
}
)
second = normalize_usage(
{
"prompt_tokens": 100_000,
"completion_tokens": 10,
"prompt_tokens_details": {"cached_tokens": 80_000},
}
)
for merged in (first.merged_with(second), first.summed_with(second)):
assert merged.input_includes_cache is True
billed = gt.complete_outcome(_outcome(), first.summed_with(second), status=200, latency_ms=1)
assert billed.provider_input_tokens == 200_000
assert billed.uncached_input_tokens == 40_000
def test_redrive_merges_keep_anthropic_dialect_after_cacheless_round() -> None:
"""A cacheless Anthropic round must not make a later cached round look inclusive."""
cacheless = normalize_usage({"input_tokens": 100, "output_tokens": 5})
cached = normalize_usage(
{"input_tokens": 10, "output_tokens": 5, "cache_read_input_tokens": 80}
)
for first, second in ((cacheless, cached), (cached, cacheless)):
for merged in (first.merged_with(second), first.summed_with(second)):
assert merged.input_includes_cache is False
billed = gt.complete_outcome(
_outcome(), first.summed_with(second), status=200, latency_ms=1
)
assert billed.provider_input_tokens == 190
assert billed.uncached_input_tokens == 110
# --------------------------------------------------------------------------- #
# Capabilities, obligations, outcome completion, compaction #
# --------------------------------------------------------------------------- #