mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(gateway): keep input_includes_cache when re-drive usage is merged (#4423)
## Description
`NormalizedUsage.merged_with` and `summed_with` dropped
`input_includes_cache`, so after a re-drive round an OpenAI-shaped usage
was billed as if `input_tokens` excluded the cache, and the cached
tokens were counted twice in the recorded outcome. This carries the flag
through both merges. A usage block with no cache fields carries no
accounting dialect (`normalize_usage` defaults it to inclusive), so the
merge takes the flag from the latest round that reported cache fields
instead of OR-ing the rounds.
Fixes #4414
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Pass `input_includes_cache` through `merged_with` and `summed_with`,
chosen by a small helper `_merged_includes_cache`: the latest round that
reported cache fields decides (`headroom/proxy/gateway_turn.py`).
- Add `test_redrive_merges_keep_input_includes_cache` and
`test_redrive_merges_keep_anthropic_dialect_after_cacheless_round`
(cacheless then cached Anthropic rounds, both orders) to
`tests/test_gateway_turn_unit.py`.
- Second commit addresses the review comment on the first (Devin,
comment 4349159780): the first version OR-ed the flag, which marked a
cacheless-then-cached Anthropic turn as inclusive.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# new tests, source change reverted (state of the first commit, OR-ed flag)
E assert True is False
E + where True = NormalizedUsage(input_tokens=110, output_tokens=10, cache_read=80, cache_write=None, input_includes_cache=True).input_includes_cache
1 failed, 47 passed
# with the fix: tests/test_gateway_turn_unit.py
48 passed
# with the fix: tests/test_gateway_turn_unit.py tests/gateway/test_redrive_loop.py tests/gateway/test_contract_response_half.py tests/gateway/test_usage_relay_openai_shape.py
110 passed, 1 warning
ruff check: All checks passed!
ruff format --check: 2 files already formatted
```
## Real Behavior Proof
- Environment: Ubuntu (WSL2), Python 3.12.15, upstream main e995eb9f.
`headroom/_core.abi3.so` from the official headroom-ai 0.41.0 wheel was
copied in (untracked, not committed). Real proxy on 127.0.0.1:18890; no
provider is contacted (the client plays the gateway and posts canned
OpenAI-shaped responses).
- Exact command / steps: start `python -P -m headroom.proxy.server
--host 127.0.0.1 --port 18890 --no-cache --no-rate-limit` with a
throwaway extension whose turn hook calls `call_model` once in
`on_response`; a client posts `/v1/compress` (gateway block with
`can_redrive`, `can_relay_response`, `session_affinity`) and then
`/v1/compress/response` for round 0 and round 1 with
`usage={"prompt_tokens":100000,"completion_tokens":10,"prompt_tokens_details":{"cached_tokens":80000}}`;
read `/stats`.
- Observed result: OpenAI-shaped turn (two rounds of `prompt_tokens`
100000, `cached_tokens` 80000): before the original fix
`cost.total_input_tokens 360000`,
`prefix_cache.totals.uncached_input_tokens 200000`; with the fix
`200000` and `40000`; `cache_read_tokens` 160000 in both.
Anthropic-shaped turn (round 0
`{"input_tokens":100000,"output_tokens":10}`, round 1
`{"input_tokens":10000,"output_tokens":10,"cache_read_input_tokens":80000}`):
with the first commit (OR-ed flag) `cost.total_input_tokens 110000`,
`uncached_input_tokens 30000`; with this commit `190000` and `110000`;
`cache_read_tokens` 80000 in both. The OpenAI-shaped run was repeated on
this commit and still gives `200000` / `40000`.
- Not tested: a real OpenAI-compatible or Anthropic gateway or provider;
streaming turns; a turn that mixes provider dialects between rounds; a
Rust core built from this checkout.
```text
== OpenAI shape, BEFORE the original fix
round 1 -> done billed_usage: {'input_tokens': 200000, 'output_tokens': 20, 'cache_read_input_tokens': 160000}
stats .prefix_cache.totals.cache_read_tokens 160000
stats .prefix_cache.totals.uncached_input_tokens 200000
stats .cost.total_input_tokens 360000
== OpenAI shape, AFTER (this commit)
stats .prefix_cache.totals.cache_read_tokens 160000
stats .prefix_cache.totals.uncached_input_tokens 40000
stats .cost.total_input_tokens 200000
== Anthropic shape, cacheless round then cached round, first commit (OR-ed flag)
stats .prefix_cache.totals.cache_read_tokens 80000
stats .prefix_cache.totals.uncached_input_tokens 30000
stats .cost.total_input_tokens 110000
== Anthropic shape, this commit
stats .prefix_cache.totals.cache_read_tokens 80000
stats .prefix_cache.totals.uncached_input_tokens 110000
stats .cost.total_input_tokens 190000
```
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: recorded input and uncached-input
counters for re-driven OpenAI-shaped gateway turns are no longer
inflated
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none known
- Rollback path: revert the commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Only the four listed test files were run, not the whole suite. The
dialect is taken from the latest round that reported cache fields; a
turn that mixes provider dialects between rounds is not handled and is
not expected from a single gateway. Produced with AI assistance (Claude
Code) operated by me (HetCreep); the reproduction was run on my machine
on a clone of the repository.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
bf34ccf588
commit
3f941446bf
@@ -257,6 +257,15 @@ class NormalizedUsage:
|
||||
return True
|
||||
return self.cache_read is not None and self.cache_write is not None
|
||||
|
||||
def _merged_includes_cache(self, other: NormalizedUsage) -> bool:
|
||||
"""Accounting dialect across rounds. A usage block without cache fields
|
||||
carries no dialect (``normalize_usage`` defaults it to inclusive, which
|
||||
is wrong for a cacheless Anthropic round), so the round that reported
|
||||
cache fields decides; the latest such round wins."""
|
||||
if other.has_cache_signal or not self.has_cache_signal:
|
||||
return other.input_includes_cache
|
||||
return self.input_includes_cache
|
||||
|
||||
def merged_with(self, other: NormalizedUsage) -> NormalizedUsage:
|
||||
"""Sum the billed counters across re-drive rounds; cache fields come
|
||||
from the latest round that carried them (the last provider call is the
|
||||
@@ -272,6 +281,7 @@ class NormalizedUsage:
|
||||
output_tokens=_sum(self.output_tokens, other.output_tokens),
|
||||
cache_read=other.cache_read if other.has_cache_signal else self.cache_read,
|
||||
cache_write=other.cache_write if other.has_cache_signal else self.cache_write,
|
||||
input_includes_cache=self._merged_includes_cache(other),
|
||||
)
|
||||
|
||||
def summed_with(self, other: NormalizedUsage) -> NormalizedUsage:
|
||||
@@ -290,6 +300,7 @@ class NormalizedUsage:
|
||||
output_tokens=_sum(self.output_tokens, other.output_tokens),
|
||||
cache_read=_sum(self.cache_read, other.cache_read),
|
||||
cache_write=_sum(self.cache_write, other.cache_write),
|
||||
input_includes_cache=self._merged_includes_cache(other),
|
||||
)
|
||||
|
||||
def as_anthropic(self) -> dict[str, int]:
|
||||
|
||||
@@ -530,6 +530,45 @@ def test_merged_with_sums_billed_and_keeps_latest_cache() -> None:
|
||||
assert c.cache_read == 60 # a signal-free round does not erase the last signal
|
||||
|
||||
|
||||
def test_redrive_merges_keep_input_includes_cache() -> None:
|
||||
"""Re-drive rounds of an OpenAI-shaped turn must not be billed as Anthropic-shaped."""
|
||||
first = normalize_usage(
|
||||
{
|
||||
"prompt_tokens": 100_000,
|
||||
"completion_tokens": 10,
|
||||
"prompt_tokens_details": {"cached_tokens": 80_000},
|
||||
}
|
||||
)
|
||||
second = normalize_usage(
|
||||
{
|
||||
"prompt_tokens": 100_000,
|
||||
"completion_tokens": 10,
|
||||
"prompt_tokens_details": {"cached_tokens": 80_000},
|
||||
}
|
||||
)
|
||||
for merged in (first.merged_with(second), first.summed_with(second)):
|
||||
assert merged.input_includes_cache is True
|
||||
billed = gt.complete_outcome(_outcome(), first.summed_with(second), status=200, latency_ms=1)
|
||||
assert billed.provider_input_tokens == 200_000
|
||||
assert billed.uncached_input_tokens == 40_000
|
||||
|
||||
|
||||
def test_redrive_merges_keep_anthropic_dialect_after_cacheless_round() -> None:
|
||||
"""A cacheless Anthropic round must not make a later cached round look inclusive."""
|
||||
cacheless = normalize_usage({"input_tokens": 100, "output_tokens": 5})
|
||||
cached = normalize_usage(
|
||||
{"input_tokens": 10, "output_tokens": 5, "cache_read_input_tokens": 80}
|
||||
)
|
||||
for first, second in ((cacheless, cached), (cached, cacheless)):
|
||||
for merged in (first.merged_with(second), first.summed_with(second)):
|
||||
assert merged.input_includes_cache is False
|
||||
billed = gt.complete_outcome(
|
||||
_outcome(), first.summed_with(second), status=200, latency_ms=1
|
||||
)
|
||||
assert billed.provider_input_tokens == 190
|
||||
assert billed.uncached_input_tokens == 110
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Capabilities, obligations, outcome completion, compaction #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
Reference in New Issue
Block a user