fix(ccr): lowercase the hash in the streaming feedback recorders (#2313)

## Description

The streaming CCR feedback recorders look the `headroom_retrieve` hash
up in the store with whatever casing the model produced, so an
uppercase-echoed hash misses the (lowercase-keyed) store and its
retrieval never reaches the TOIN learning loop.

`_record_ccr_feedback_from_response` (Anthropic) and
`_record_ccr_feedback_from_openai_sse` (OpenAI chat stream) both pull
the hash straight off the tool call and pass it to `store.retrieve`:

```python
hash_key = input_data.get("hash")
if not hash_key:
    continue
...
store.retrieve(hash_key)
```

The compression store keys every entry by a lowercase hash
(`sha256(...).hexdigest()` and `explicit_hash.lower()` on write). A
model that reproduces the marker hash in uppercase therefore misses the
lookup, so the retrieval side effect that feeds TOIN (`store.retrieve →
_log_retrieval → toin.record_retrieval`) never fires — the learning
signal for that retrieval is silently dropped.

The request path was already fixed for this (`parse_tool_call` now
returns the hash lowercased), but these two feedback recorders extract
the hash themselves and were missed.

## Fix

Normalise the hash to lowercase before the lookup in both recorders:

```python
hash_key = hash_key.lower()
...
store.retrieve(hash_key)
```

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/streaming.py`: lowercase the extracted hash
in `_record_ccr_feedback_from_response` and
`_record_ccr_feedback_from_openai_sse` before `store.retrieve`.
- `tests/test_ccr_feedback_hash_case.py`: new tests that both recorders
query the store with the lowercase hash when the model echoes it
uppercase.
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/proxy/handlers/streaming.py tests/test_ccr_feedback_hash_case.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/handlers/streaming.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOM-kills this box (ML stack import), so I
reproduced the recorder logic against a lowercase-keyed fake store with
a dependency-free script and left the full pytest to CI.
- Exact command / steps: fed a `headroom_retrieve` tool call whose hash
is the uppercase form of a stored (lowercase) key through the OLD
(verbatim) and NEW (`.lower()`) recorder logic.
- Observed result: OLD queries the store with the uppercase hash → miss
(no feedback); NEW queries with the lowercase hash → hit (feedback
recorded).
- Not tested: a live streaming retrieval with an uppercase echo
end-to-end; full local `pytest` deferred to CI (OOM).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" box is unchecked because the full suite
imports the ML stack, which I can't run here. The recorder methods use
no instance state, so the new tests exercise them on a bare
`object.__new__(StreamingMixin)` with the compression store
monkeypatched to a spy; they run under the normal CI pytest job, and the
standalone proof above corroborates them. This completes the
lowercase-hash fix that already landed for the request path
(`parse_tool_call`).

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2313"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `6075b100ab078d08769f58bf38769974064d3940`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration of upstream/main
0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c; scope remains streaming
handler plus focused tests. Contributor recorder method ASTs unchanged;
all other handler AST exactly matches main, retaining new CCR
continuation behavior. Fresh 96 tests pass (one warning), including CCR
feedback and current-main streaming/buffered continuations. Four focused
tests fail against byte-exact main handler; all pass on candidate.
Actual in-memory CompressionStore and actual TOIN with synthetic stored
JSON: uppercase hash passed through Anthropic response recorder and
OpenAI SSE recorder produces one TOIN retrieval each; same real-store
control against main produces zero for both. No store/TOIN spy. This
exercises actual recorder/store/feedback methods, not a live provider
stream or full HTTP gating; no cloud/provider/model/billing proof
claimed. Telemetry disabled, TOIN in-memory, no user store/config
accessed. Ruff lint/format, offline lock and whitespace pass. Full
recorded native tracked input-set/every hash and artifact checksum
verified before existing DLL reuse, no fresh build. Author fixes for
non-string hashes retained. Existing Devin mock-only proof finding
addressed within these limits; no duplicate feedback. Exact
published-head hosted checks remain required before formal approval;
maintainers handle CI. No PR merged.

Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


<!-- maintainer-refresh-12864fc8f01a1ca6c58440eccef0dabdd3e595b8 -->

Maintainer current-main refresh: exact head
`2bd5f40d6a91ffc308ce2ee41a34f49c2b951d3f`, authoritative main
`12864fc8f01a1ca6c58440eccef0dabdd3e595b8`. Clean integration patches
exactly match incoming main; contributor behavior preserved. Fresh
focused source tests plus tokenizer/response-header/HF/JSON-parser
incoming-main controls: 93 passed, 1 warning in 11.36s.
Ruff/format/whitespace pass. The two recorder method ASTs are unchanged
from the prior reviewed head, and all other streaming-handler AST
matches current main. Prior real in-memory store/TOIN evidence remains
historical within that method scope; fresh hash tests and incoming-main
controls do not claim a live provider stream or broaden the existing
direct-OpenAI tracking gate. Native artifact reuse verified against full
recorded input set/every hash and artifact checksum; no fresh native
build claimed. Static publication gate excludes inherited
workflow-validation/native Docker/Init/Wrap installer paths, with no
workflow/setup/dependency/native/CLI changes in PR diff. Actual
exact-head hosted workflows and final independent review remain pending.
No merge performed.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Abhay Singh
2026-11-07 20:45:40 -06:00
committed by GitHub
co-authored by JerrettDavis
parent 9ad16a77f8
commit 3912c04053
2 changed files with 146 additions and 2 deletions
+18 -2
View File
@@ -781,9 +781,19 @@ class StreamingMixin:
input_data = block.get("input", {})
hash_key = input_data.get("hash")
if not hash_key:
# The payload is parsed straight from model output, so `hash` may be
# missing or a non-string (e.g. {"hash": 123}). Skip anything that
# isn't a usable string rather than letting `.lower()` raise.
if not isinstance(hash_key, str) or not hash_key:
continue
# The store keys every entry by a lowercase hash (see
# compression_store: sha256 hexdigest / explicit_hash.lower()), so a
# model that echoes the marker hash uppercase would miss the lookup
# and the retrieval feedback would never reach TOIN. Normalise here,
# matching parse_tool_call.
hash_key = hash_key.lower()
logger.info(f"[{request_id}] CCR Feedback: Recording retrieval hash={hash_key[:8]}...")
# Call store.retrieve() for the side effect of triggering the
@@ -855,9 +865,15 @@ class StreamingMixin:
if not isinstance(input_data, dict):
continue
hash_key = input_data.get("hash")
if not hash_key:
# Skip a missing or non-string hash (malformed model output) instead
# of raising in `.lower()`.
if not isinstance(hash_key, str) or not hash_key:
continue
# Normalise to the store's lowercase key form (see
# _record_ccr_feedback_from_response).
hash_key = hash_key.lower()
logger.info(
f"[{request_id}] CCR Feedback (openai stream): Recording retrieval "
f"hash={hash_key[:8]}..."
+128
View File
@@ -0,0 +1,128 @@
"""The streaming CCR feedback recorders must look the hash up in lowercase.
The compression store keys every entry by a lowercase hash, so a model that
echoes the ``headroom_retrieve`` marker hash in uppercase would miss the store
and the retrieval would never reach the TOIN feedback loop. The recorders
extract the hash directly (they do not go through ``parse_tool_call``, which was
already fixed), so they must normalise it themselves.
"""
from __future__ import annotations
import headroom.cache.compression_store as cs
from headroom.proxy.handlers.streaming import StreamingMixin
HASH_LOWER = "abc123def456abc123def456"
class _SpyStore:
def __init__(self) -> None:
self.looked_up: list[str] = []
def retrieve(self, hash_key, query=None): # noqa: ANN001, ANN201
self.looked_up.append(hash_key)
return None
def test_response_feedback_lowercases_hash(monkeypatch):
spy = _SpyStore()
monkeypatch.setattr(cs, "get_compression_store", lambda: spy)
handler = object.__new__(StreamingMixin)
response = {
"content": [
{
"type": "tool_use",
"name": "headroom_retrieve",
"input": {"hash": HASH_LOWER.upper()},
}
]
}
handler._record_ccr_feedback_from_response(response, "anthropic", "req-1")
assert spy.looked_up == [HASH_LOWER]
def test_openai_stream_feedback_lowercases_hash(monkeypatch):
spy = _SpyStore()
monkeypatch.setattr(cs, "get_compression_store", lambda: spy)
import json
handler = object.__new__(StreamingMixin)
sse = (
"data: "
+ json.dumps(
{
"choices": [
{
"delta": {
"tool_calls": [
{
"index": 0,
"function": {
"name": "headroom_retrieve",
"arguments": json.dumps({"hash": HASH_LOWER.upper()}),
},
}
]
}
}
]
}
)
+ "\n"
)
handler._record_ccr_feedback_from_openai_sse(sse, "req-1")
assert spy.looked_up == [HASH_LOWER]
def test_response_feedback_ignores_non_string_hash(monkeypatch):
# A malformed non-string hash (e.g. {"hash": 123}) must be skipped, not
# raise in .lower().
spy = _SpyStore()
monkeypatch.setattr(cs, "get_compression_store", lambda: spy)
handler = object.__new__(StreamingMixin)
response = {
"content": [{"type": "tool_use", "name": "headroom_retrieve", "input": {"hash": 123}}]
}
handler._record_ccr_feedback_from_response(response, "anthropic", "req-1")
assert spy.looked_up == []
def test_openai_stream_feedback_ignores_non_string_hash(monkeypatch):
import json
spy = _SpyStore()
monkeypatch.setattr(cs, "get_compression_store", lambda: spy)
handler = object.__new__(StreamingMixin)
sse = (
"data: "
+ json.dumps(
{
"choices": [
{
"delta": {
"tool_calls": [
{
"index": 0,
"function": {
"name": "headroom_retrieve",
"arguments": json.dumps({"hash": 123}),
},
}
]
}
}
]
}
)
+ "\n"
)
handler._record_ccr_feedback_from_openai_sse(sse, "req-1")
assert spy.looked_up == []