mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:06:14 +08:00
## Description Fixes #4159. OpenCode exposes `headroom_headroom_retrieve`, but the proxy only checks the bare `headroom_retrieve` when deciding whether the client already owns retrieval. It therefore adds a second, unavailable client tool. Retrieval-only proxy turns can be intercepted internally; mixed retrieval + Read turns are forwarded to the client, which cannot execute the bare spelling. ## Type of Change - [x] Bug fix - [x] Documentation update ## Changes Made - Share one ownership selector between session-sticky injection and the per-request `CCRToolInjector` fallback. Accept the bare canonical name and the three supported default-Headroom-server client spellings. - Do not treat another MCP server's `headroom_retrieve` suffix as ownership of Headroom's cache. Existing broader compression-exclusion matching is unchanged. - Optional system instructions use the executable registered retrieval name in the legacy path and both Anthropic/OpenAI handlers. - Preserve client-owned aliases and bare-only proxy interception; do not alter mixed-tool response handling or add execution of client tools. - Add namespace/provider regressions across cold, compressed and later turns, plus real-store instruction and foreign-server boundaries. - Delete the incidental tool-description-size assertion and error-message wording pin, retaining real eager/lazy/cache-stability validation. - Document default-server ownership, alias-only forwarding and mixed-call behavior explicitly. ## Testing - [x] Relevant CCR/API tests pass: 161 cases across the seven suites below - [x] Changed Python source/test Ruff format and check pass - [x] Failing-before/passing-after behavioral regressions added - [x] Actual CLI/HTTP/native before-and-after smoke performed ### Test Output All **6** initial alias/provider regressions fail against exact upstream main `976aa714ef3e19174dc0cb586cee7b20d064e4ef` because unavailable bare retrieval is injected. All **10** first review regressions and **2** additional actual-API regressions fail against immutable first-PR commit `7601252be7e8b02297e342e2775bb153927e2fe9`: instructions name an unavailable tool, or a foreign MCP server displaces proxy retrieval. Current fixed result: **161 passed, 1 existing Starlette TestClient deprecation warning** across: ```text tests/test_ccr_tool_injection_cache_stability.py tests/test_ccr_tool_always_on.py tests/test_ccr_tool_injection.py tests/test_issue_728_empty_tools_injection.py tests/test_corrupt_golden_bytes_recovery.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_openai_cache_stability.py ``` Ruff format/check passed for the four changed Python sources and three regression files. A genuine-module-preloaded, coverage-instrumented CCR run passed all 110 cases (four warnings); an additional instrumented API run passed both new API regressions and measured both changed handler instruction calls as executed. Import-time coverage is incomplete in those local runs, so they are not whole-package or CI patch-coverage claims. Whole-repository pytest and whole-package mypy are not claimed by these scoped results. ## Real Behavior Proof - Environment: Linux x86_64, Python 3.13.14; genuine installed development/proxy extras and editable Rust extension, actual ONNX Runtime 1.26.0 via `ORT_DYLIB_PATH`. Independent baseline/fixed worktrees. Beacon off, isolated settings, stateless proxy, no live credentials. Provider is a controlled local Anthropic-shaped HTTP upstream; request model string is `claude-sonnet-4-20250514`, not a live model run. - Exact command / steps: start `headroom proxy --port 18858` for baseline and `--port 18859` for fixed source, each with `--anthropic-api-url http://127.0.0.1:45111` and: ```text --stateless --no-cache --no-rate-limit --no-subscription-tracking --disable-kompress --disable-kompress-fallback --no-read-lifecycle --no-telemetry ``` Optimization and CCR remain enabled. Both genuine CLI startups report `event=rust_core_loaded`. POST the same `/v1/messages` request to each with model above, `max_tokens: 128`, `system: "Use the available tools."`, user text `Retrieve the original compressed data and read the related file.`, and these client tool schemas: ```json [ {"name":"headroom_headroom_retrieve","description":"Retrieve original Headroom content through the client MCP server","input_schema":{"type":"object","properties":{"hash":{"type":"string"}},"required":["hash"]}}, {"name":"Read","description":"Read a local file","input_schema":{"type":"object","properties":{"file_path":{"type":"string"}},"required":["file_path"]}} ] ``` The controlled upstream observes the actual advertised tools, selects bare retrieval if advertised (otherwise the available alias), and returns a mixed retrieval + Read response. **Observed before:** upstream tools are `Read`, `headroom_headroom_retrieve`, **`headroom_retrieve`**. HTTP 200 forwards mixed `headroom_retrieve` + `Read`, leaving one returned tool unavailable in the client's registry. - Observed result: after the fix, upstream tools are only `Read`, `headroom_headroom_retrieve`. HTTP 200 forwards mixed **`headroom_headroom_retrieve`** + `Read`; zero unavailable returned names. No mixed-call suppression or response rewriting hides the problem. - Not tested: live Anthropic, installed OpenCode end-to-end, actual CCR hash/payload recovery, Windows or native Gemini HTTP. Per-request fallback is exercised directly by regressions. Controlled-upstream behavior alone is not offered as proof: this ran the real baseline/fixed proxy CLI, HTTP path and native extension. ### Additional review behavior proof The first PR SHA and corrected source each ran the actual proxy app through its full lifespan, with the genuine native extension loaded, actual `CompressionStore` and real HTTP to a controlled upstream at `127.0.0.1:45112`. Requests entered the genuine Anthropic and OpenAI ASGI routes, not patched handlers; this additional check was not an external CLI socket or live-provider run. Construct `ProxyConfig` with `ccr_inject_system_instructions=True`, `stateless=True`, no caching/rate limiting/subscription tracking/read lifecycle/Kompress, Anthropic URL `http://127.0.0.1:45112`, and OpenAI URL `http://127.0.0.1:45112/v1`. After startup, store original/compressed content in the actual in-memory store and use its returned hash in `<<ccr:HASH 1_rows_offloaded>>`. Send a valid assistant tool call and matching tool result to each provider endpoint, once with only `headroom_headroom_retrieve` registered and once with only `mcp__other__headroom_retrieve` registered. Anthropic uses tool-use/tool-result blocks and model `claude-sonnet-4-20250514`; OpenAI uses function tool-calls/tool messages and model `gpt-4o`. The upstream records advertised names and names invoked by the actual injected system instructions, then returns a normal final text response. All eight before/after requests returned HTTP 200. Before: both providers instructed bare `headroom_retrieve` while neither registry contained it. After: the Headroom alias is retained alone and instructions invoke it; a foreign server's tool is retained alongside proxy-owned bare retrieval and instructions invoke the bare tool. **Zero unavailable instruction names** in all four corrected requests. The upstream and both apps were stopped afterwards. The permanent API regressions reuse the existing full-lifespan provider `TestClient` fixtures and a real isolated in-memory store. Only the upstream response boundary supplies fixed provider-shaped JSON; the CCR injector, instruction generation and ownership selector are real. Both tests fail on the first PR SHA because the instructions name unavailable bare retrieval, then pass after the fix. They assert executable-name and real-hash invariants, not that a helper received forwarded arguments. Scoped coverage measured the previously untested instruction calls in both handlers as executed. ## Runtime Rollout Safety - Rollout-managed feature(s): existing CCR tool-injection paths and opt-in retrieval instructions; tool ownership and instruction naming only. - Minimum rollout channel: unchanged; no feature is enabled by this patch. - Stable/default behavior changed: stop injecting a duplicate bare schema when the client already owns a supported retrieval alias. - Kill switch / disable path: existing controls unchanged; no new switch. - Unsafe override required: none. - Qualification impact: alias/provider/cold-to-compressed regressions, foreign-server and actual-store boundaries, CLI/native proof and both-provider ASGI/native/HTTP proof above; no widened rollout. - Rollback path: revert this Python bug fix. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] One shared ownership selector, broader compression exclusions unchanged, no new dependency - [x] Documentation updated - [x] Consumer-visible failing-before/passing-after regressions included - [x] Relevant new and existing tests pass locally - [x] Did not edit generated `CHANGELOG.md` ## Mandatory Local Push Gates Normal `git push fork fix/issue-4159-client-retrieval-alias-20261011` executed the installed pre-push hook and **`make ci-precheck` passed**: - `cargo fmt --all -- --check` - `cargo clippy --workspace -- -D warnings` - `cargo test --workspace` - Genuine editable Rust extension build, install and import verification - Critical Python/native parity suite: **177 passed, 4 skipped, 1 warning** - Conventional commit validation through commitlint The warning is the existing no-client Anthropic tokenizer approximation reported by `test_relevance_with_query_context`; it was not suppressed. The earlier cold-build push timed out. Retrying with the real shared Cargo build cache and two build jobs completed all gates and pushed normally; no bypass flag or skipped hook was used. The review update `440fd96aaa11c0ea9263d9628ccc2b5f8b3865d4` also passed every normal pre-push gate above, including 177 critical Python tests. This run used `RUST_TEST_THREADS=2`: the preceding default-parallel attempt failed the unchanged native `expired_get_does_not_wipe_concurrent_refresh` test's final-live assertion with a 20 ms TTL. Baseline/current native source hashes are identical. No Rust fix is claimed; no tests or gates were filtered or bypassed. The API regression update `c5c36ba0e7074cbd815498d00ee8f394bd7287bc` again passed all normal local push gates above with the same bounded test/build concurrency, genuine native rebuild/import and 177 critical Python tests. The two additional API regressions and complete seven-suite 161-test run are separate measured verification, not mocked CCR plumbing. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
308 lines
12 KiB
Python
308 lines
12 KiB
Python
"""The CCR retrieval tool must not enter the tools array mid-session.
|
|
|
|
``tools`` is the head of Anthropic's cache key — the prefix is ordered
|
|
tools → system → messages — so any change to the array invalidates every
|
|
cached token behind it. Injecting ``headroom_retrieve`` at the first
|
|
compression means entering the array against a fully warm prefix.
|
|
|
|
Measured against the live API with the real definition: a warm prefix reading
|
|
4,335 tokens dropped to 0 read and 4,458 written the moment the 478-byte tool
|
|
was appended. On a customer session the same effect cost 113,888 tokens of
|
|
cache write to save 2,205 tokens of content — a payback of ~594 turns *within
|
|
one session*, which no session reaches.
|
|
|
|
The fix is timing, not mechanism: inject on the first request, while the
|
|
prefix is still cold, and the array never changes again.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import re
|
|
|
|
import pytest
|
|
|
|
from headroom.cache.compression_store import CompressionStore
|
|
from headroom.ccr.tool_injection import CCR_TOOL_NAME, CCRToolInjector, create_ccr_tool_definition
|
|
from headroom.proxy.helpers import (
|
|
_reset_session_ccr_tracker_for_test,
|
|
apply_session_sticky_ccr_tool,
|
|
)
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _reset_tracker():
|
|
_reset_session_ccr_tracker_for_test()
|
|
yield
|
|
_reset_session_ccr_tracker_for_test()
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _default_eager(monkeypatch):
|
|
"""Pin the default explicitly so a stray env cannot mask a regression."""
|
|
monkeypatch.delenv("HEADROOM_CCR_TOOL_INJECTION", raising=False)
|
|
|
|
|
|
CLIENT_TOOLS = [
|
|
{
|
|
"name": "Read",
|
|
"description": "Read a file.",
|
|
"input_schema": {"type": "object", "properties": {"path": {"type": "string"}}},
|
|
},
|
|
{
|
|
"name": "Bash",
|
|
"description": "Run a command.",
|
|
"input_schema": {"type": "object", "properties": {"cmd": {"type": "string"}}},
|
|
},
|
|
]
|
|
|
|
|
|
def _apply(session_id, *, compressed, provider="anthropic", allow_eager=True):
|
|
"""``allow_eager=True`` mirrors what both handlers pass for a request that
|
|
can actually compress (optimization on, no bypass header)."""
|
|
return apply_session_sticky_ccr_tool(
|
|
provider=provider,
|
|
session_id=session_id,
|
|
request_id="req-1",
|
|
existing_tools=CLIENT_TOOLS,
|
|
has_compressed_content_this_turn=compressed,
|
|
allow_eager=allow_eager,
|
|
)
|
|
|
|
|
|
def _names(tools):
|
|
return [t.get("name") or t.get("function", {}).get("name") for t in tools]
|
|
|
|
|
|
# ── the regression this file exists for ───────────────────────────────
|
|
|
|
|
|
@pytest.mark.parametrize("provider", ["anthropic", "openai"])
|
|
def test_tools_array_is_byte_stable_across_the_first_compression(provider):
|
|
"""The turn that first compresses must not change the tools array.
|
|
|
|
This is the whole bug. Before the fix, turn 1 returned two tools and turn 2
|
|
returned three — and because tools sit first in the cache prefix, that one
|
|
appended entry invalidated the entire warm prefix behind it.
|
|
"""
|
|
session = f"sess-{provider}-stability"
|
|
|
|
before, _ = _apply(session, compressed=False, provider=provider)
|
|
during, _ = _apply(session, compressed=True, provider=provider)
|
|
after, _ = _apply(session, compressed=False, provider=provider)
|
|
|
|
assert json.dumps(before) == json.dumps(during) == json.dumps(after), (
|
|
"tools array changed across the first compression — this invalidates "
|
|
"the whole provider cache prefix"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("provider", ["anthropic", "openai"])
|
|
def test_the_tool_is_present_from_the_very_first_request(provider):
|
|
"""Cold prefix is the only cheap moment to add to the tools array."""
|
|
tools, injected = _apply(f"sess-{provider}-first", compressed=False, provider=provider)
|
|
assert injected is True
|
|
assert CCR_TOOL_NAME in _names(tools)
|
|
|
|
|
|
def test_the_tool_definition_is_small_enough_that_eager_is_obviously_right():
|
|
"""Bound the cold-prefix overhead that justifies eager injection."""
|
|
size = len(json.dumps(create_ccr_tool_definition()))
|
|
assert size < 2000, (
|
|
f"headroom_retrieve is now {size} bytes; eager injection was justified "
|
|
"on it being ~478 bytes (~119 tokens) of the session's first cache write"
|
|
)
|
|
|
|
|
|
# ── the escape hatch still works ──────────────────────────────────────
|
|
|
|
|
|
def test_lazy_mode_restores_the_historical_gate(monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_CCR_TOOL_INJECTION", "lazy")
|
|
session = "sess-lazy"
|
|
|
|
tools, injected = _apply(session, compressed=False)
|
|
assert injected is False
|
|
assert CCR_TOOL_NAME not in _names(tools)
|
|
|
|
tools, injected = _apply(session, compressed=True)
|
|
assert injected is True
|
|
assert CCR_TOOL_NAME in _names(tools)
|
|
|
|
|
|
def test_an_invalid_mode_raises_rather_than_silently_defaulting(monkeypatch):
|
|
from headroom.proxy.helpers import get_ccr_tool_injection_mode
|
|
|
|
monkeypatch.setenv("HEADROOM_CCR_TOOL_INJECTION", "sometimes")
|
|
with pytest.raises(ValueError, match="HEADROOM_CCR_TOOL_INJECTION"):
|
|
get_ccr_tool_injection_mode()
|
|
|
|
|
|
# ── properties the fix must not break ─────────────────────────────────
|
|
|
|
|
|
@pytest.mark.parametrize("empty", [None, []])
|
|
def test_a_request_with_no_tools_is_not_armed_eagerly(empty):
|
|
"""Eager injection must not turn a no-tools request into a tools request.
|
|
|
|
Adding the first entry to an absent tools array lets the model emit
|
|
tool_use blocks the client never expected (#728). Such a client is also not
|
|
a harness: it has no tool results to compress, so there is no warm tools
|
|
segment for eager injection to protect. It keeps the historical gate.
|
|
"""
|
|
tools, injected = apply_session_sticky_ccr_tool(
|
|
provider="anthropic",
|
|
session_id="sess-no-tools",
|
|
request_id="req-1",
|
|
existing_tools=empty,
|
|
has_compressed_content_this_turn=False,
|
|
allow_eager=True,
|
|
)
|
|
assert injected is False
|
|
assert tools == []
|
|
|
|
|
|
def test_a_client_provided_tool_still_wins(monkeypatch):
|
|
"""An MCP-registered headroom_retrieve must not be doubled up."""
|
|
client_owned = [*CLIENT_TOOLS, {"name": CCR_TOOL_NAME, "description": "client's own"}]
|
|
tools, injected = apply_session_sticky_ccr_tool(
|
|
provider="anthropic",
|
|
session_id="sess-client-owned",
|
|
request_id="req-1",
|
|
existing_tools=client_owned,
|
|
has_compressed_content_this_turn=False,
|
|
allow_eager=True,
|
|
)
|
|
assert injected is False
|
|
assert _names(tools).count(CCR_TOOL_NAME) == 1
|
|
|
|
|
|
@pytest.mark.parametrize("provider", ["anthropic", "openai"])
|
|
@pytest.mark.parametrize(
|
|
"alias",
|
|
[
|
|
"headroom_headroom_retrieve",
|
|
"mcp__headroom__headroom_retrieve",
|
|
"mcp_Headroom_headroom_retrieve",
|
|
],
|
|
)
|
|
def test_client_retrieval_alias_never_advertises_an_unavailable_tool(provider, alias):
|
|
"""Mixed-tool turns must only offer retrieval names the client can execute."""
|
|
retrieval = create_ccr_tool_definition(provider)
|
|
if provider == "openai":
|
|
retrieval["function"]["name"] = alias
|
|
read_tool = {"type": "function", "function": {"name": "Read", "parameters": {}}}
|
|
else:
|
|
retrieval["name"] = alias
|
|
read_tool = CLIENT_TOOLS[0]
|
|
client_tools = [retrieval, read_tool]
|
|
session_id = f"sess-client-alias-{provider}-{alias}"
|
|
|
|
# A cold request, actual compression, and later replay all retain a usable
|
|
# retrieval tool without introducing the unavailable bare proxy spelling.
|
|
for compressed in (False, True, False):
|
|
tools, injected = apply_session_sticky_ccr_tool(
|
|
provider=provider,
|
|
session_id=session_id,
|
|
request_id="req-client-alias",
|
|
existing_tools=client_tools,
|
|
has_compressed_content_this_turn=compressed,
|
|
allow_eager=True,
|
|
)
|
|
assert not injected
|
|
assert alias in _names(tools)
|
|
assert CCR_TOOL_NAME not in _names(tools)
|
|
|
|
# The no-session library/Google fallback must obey the same client registry.
|
|
injector = CCRToolInjector(provider=provider, inject_tool=True)
|
|
tools, injected = injector.inject_tool_definition(client_tools, session_has_done_ccr=True)
|
|
assert not injected
|
|
assert alias in _names(tools)
|
|
assert CCR_TOOL_NAME not in _names(tools)
|
|
|
|
|
|
@pytest.mark.parametrize("provider", ["anthropic", "openai"])
|
|
@pytest.mark.parametrize("alias", ["mcp__other__headroom_retrieve", "mcp_Other_headroom_retrieve"])
|
|
def test_unrelated_mcp_server_cannot_displace_proxy_retrieval(provider, alias):
|
|
retrieval = create_ccr_tool_definition(provider)
|
|
if provider == "openai":
|
|
retrieval["function"]["name"] = alias
|
|
else:
|
|
retrieval["name"] = alias
|
|
client_tools = [retrieval]
|
|
tools, _ = apply_session_sticky_ccr_tool(
|
|
provider=provider,
|
|
session_id=None,
|
|
request_id="req-unrelated-mcp",
|
|
existing_tools=client_tools,
|
|
has_compressed_content_this_turn=True,
|
|
)
|
|
assert CCR_TOOL_NAME in _names(tools)
|
|
assert alias in _names(tools)
|
|
|
|
injector = CCRToolInjector(provider=provider)
|
|
fallback_tools, _ = injector.inject_tool_definition(client_tools, session_has_done_ccr=True)
|
|
assert CCR_TOOL_NAME in _names(fallback_tools)
|
|
assert alias in _names(fallback_tools)
|
|
|
|
|
|
@pytest.mark.parametrize("provider", ["anthropic", "openai"])
|
|
@pytest.mark.parametrize(
|
|
"alias",
|
|
[
|
|
"headroom_headroom_retrieve",
|
|
"mcp__headroom__headroom_retrieve",
|
|
"mcp_Headroom_headroom_retrieve",
|
|
],
|
|
)
|
|
def test_retrieval_instructions_name_an_executable_client_tool(provider, alias):
|
|
store = CompressionStore(enable_feedback=False)
|
|
hash_key = store.store(original='{"value":42}', compressed="compressed")
|
|
messages = [
|
|
{"role": "system", "content": "Use the available tools."},
|
|
{"role": "tool", "content": f"<<ccr:{hash_key} 1_rows_offloaded>>"},
|
|
]
|
|
retrieval = create_ccr_tool_definition(provider)
|
|
if provider == "openai":
|
|
retrieval["function"]["name"] = alias
|
|
else:
|
|
retrieval["name"] = alias
|
|
injector = CCRToolInjector(
|
|
provider=provider,
|
|
inject_system_instructions=True,
|
|
compression_store=store,
|
|
)
|
|
updated_messages, updated_tools, _ = injector.process_request(messages, [retrieval])
|
|
instructions = updated_messages[0]["content"]
|
|
call_names = re.findall(r"`([^`]+)\(hash=", instructions)
|
|
assert call_names == [alias]
|
|
assert set(call_names).issubset(_names(updated_tools))
|
|
assert hash_key in instructions
|
|
|
|
|
|
def test_sticky_replay_is_byte_identical_to_the_eager_injection():
|
|
"""Later turns replay the golden bytes recorded by the eager injection."""
|
|
session = "sess-golden"
|
|
first, _ = _apply(session, compressed=False)
|
|
later, _ = _apply(session, compressed=True)
|
|
assert json.dumps(first) == json.dumps(later)
|
|
|
|
|
|
def test_the_sessionless_path_is_also_stable():
|
|
"""No session id means no tracker, so the flag alone drove the toggle."""
|
|
a, _ = _apply(None, compressed=False)
|
|
b, _ = _apply(None, compressed=True)
|
|
assert json.dumps(a) == json.dumps(b)
|
|
assert CCR_TOOL_NAME in _names(a)
|
|
|
|
|
|
def test_a_request_that_cannot_compress_is_not_armed():
|
|
"""`--no-optimize` or a bypass header means nothing will ever be compressed.
|
|
|
|
Pre-arming there would leave a permanently unredeemable tool in the client's
|
|
array. The old gate got this for free: no compression meant no injection.
|
|
"""
|
|
tools, injected = _apply("sess-no-optimize", compressed=False, allow_eager=False)
|
|
assert injected is False
|
|
assert CCR_TOOL_NAME not in _names(tools)
|