mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat(proxy): report the same tool savings on gateway /v1/compress as the proxy path (#4467)
## Description Gateway-mode `/v1/compress` now reports the same savings as the proxy path. The why is in #4466. Closes #4466 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [x] Documentation update ## Changes Made Two commits, one per gap: 1. **`fix(proxy): count tool-schema compaction ...`** - The gateway turn counts the tool definitions before and after compaction. - The handler adds both counts to `tokens_before` / `tokens_after`, with the proxy path's guard (`0 < after < before`). - Tools the client already marked `defer_loading` are skipped. 2. **`feat(proxy): report tool_tokens_saved and tool_tokens_deferred ...`** - Gateway-mode answers add `tool_tokens_saved` (from the tags `/stats` sums) and `tool_tokens_deferred` (the estimated share). - Docs: `proxy.mdx`, `wiki/proxy.md`. Non-gateway and fail-open answers do not change. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`, on the changed files) - [x] Type checking passes (`mypy`, on the changed files) - [x] New tests added for new functionality ```text $ pytest -q tests/gateway/ tests/test_gateway_turn_unit.py tests/test_proxy_compress_endpoint.py 380 passed, 5 skipped ``` The tests also pass on the first commit alone (`tests/gateway/`: 280 passed). The new test for client-deferred tools fails without its fix. ## Real Behavior Proof - Environment: macOS, Python 3.12. Before: image `headroom:0.41.0-nonroot`. After: this branch, `HEADROOM_OFFLINE=1 headroom proxy`. - Exact command / steps: start each build, then send it the same gateway-mode `POST /v1/compress` — one message plus 5 tools whose schemas carry `title` / `examples` / `default` (the annotation keys schema compaction strips), with `"gateway": {"can_redrive": false, "can_relay_response": false}` so the request is a single call. Read `tokens_before`, `tokens_after`, `tokens_saved` and the `tool_*` fields out of the JSON answer. No provider is contacted. - Observed result: same request, same tools. On 0.41.0 the tool definitions are invisible to the counts, so `tokens_saved` is 0 and neither `tool_*` field is present. On this branch the compaction is counted, so `tokens_before` / `tokens_after` carry the tool bytes and `tokens_saved` is the 240 tokens compaction stripped (`tool_tokens_saved` / `tool_tokens_deferred` stay 0 here: no tool-search extension ran, and the extension is what books deferral). ```text 0.41.0: tokens_before 13, tokens_after 13, tokens_saved 0 (no tool_* fields) branch: tokens_before 830, tokens_after 590, tokens_saved 240, tool_tokens_saved 0, tool_tokens_deferred 0 ``` - Not tested: a live run with the headroom-tool-search extension (deferral is covered by unit tests), and Responses-API bodies. ## Runtime Rollout Safety - Rollout-managed feature(s): None gate this change — the gateway `/v1/compress` accounting is not behind a registry feature. One interaction: `proxy_output_shaper` (registry feature, available `stable`, deliberately not default-enabled) consumes the augmented count. `tokens_before` is folded at `headroom/proxy/handlers/openai.py:11064` and then handed to the turn at `:11083`, which passes it as `input_tokens` to `shape_gateway_body` (`headroom/proxy/gateway_extension.py:213`) → `stratum_key` → `input_bucket`. A turn whose tool definitions push it across one of the 2k / 8k / 32k / 128k boundaries therefore lands in a bigger stratum, which picks a different output-savings baseline. The chat path already has this coupling. - Minimum rollout channel: n/a — no channel gate, so the change takes effect wherever gateway mode is served, on every channel. - Stable/default behavior changed: Yes, gateway-mode answers only. `tokens_before` / `tokens_after` now include the tool definitions before and after schema compaction, so `tokens_saved` covers the compaction saving the same way the chat handlers already do, and two fields are added. Non-gateway and fail-open answers are byte-identical. - Kill switch / disable path: None. Without compaction there is nothing to fold, and the two fields are additive on answers; a gateway that does not want them ignores them. - Unsafe override required: No. Nothing crosses a channel boundary. - Qualification impact: None. No registry entry is added, so `registry_digest` and `snapshot_digest` are unchanged. The gateway `tokens_saved` figure itself changes, which a benchmark comparing arms across this commit will see. - Rollback path: Revert the two commits. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] I have made corresponding changes to the documentation - [x] I have added tests that my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - The fields are emitted for any claimed compress turn. Only the gateway extension claims turns today. - If a response-half hook rewrote the tool tags during a re-drive, `/stats` would no longer match the answer. Core has no such hook. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
3fef15d376
commit
a90421083a
@@ -824,6 +824,13 @@ The response keeps every field above and adds six:
|
||||
|
||||
Fail-open answers (bypass header, empty `messages`, stateless timeout) carry the same six keys with your original body, `obligations: []` and `headers: {}`.
|
||||
|
||||
**Savings on a gateway answer.** The token counts follow the proxy path, so a gateway can report the same figure `/stats` does. When compaction shortened `tools`, `tokens_before` and `tokens_after` also count the tool definitions before and after it, so `tokens_saved` includes the compaction saving. Two more fields carry the tool definitions kept out of the context, which no message count sees; both are absent on fail-open answers:
|
||||
|
||||
| Field | Description |
|
||||
| --- | --- |
|
||||
| `tool_tokens_saved` | Tool-search deferral plus tools a turn hook removed, net (`tool_search_deferred_tokens` after reconciliation + `turn_hook_tools_saved_tokens` after netting against message growth). `tokens_saved + tool_tokens_saved` is the `/stats` headline for the turn. |
|
||||
| `tool_tokens_deferred` | The deferral share of `tool_tokens_saved`. `/stats` books it as `estimated`: a deferred tool the model loads later is billed after all. Subtract it for a figure of measured savings only. |
|
||||
|
||||
When `relay_usage` is owed, the turn's `/stats` record is deferred until the response half arrives with billed input and output, cache counts, provider status and latency; a turn that never comes back is recorded as-is when it expires. Pending turns live `HEADROOM_GATEWAY_TURN_TTL_SECONDS` (default `120`, refreshed on every `redrive`), at most `HEADROOM_GATEWAY_MAX_PENDING_TURNS` at a time (default `10000`, oldest evicted first), and a turn may re-drive at most `HEADROOM_GATEWAY_MAX_REDRIVES` times (default `8`).
|
||||
|
||||
**`config.mode: "ccr"` on this contract.** When markers were inserted and re-driving is allowed, Headroom injects the `headroom_retrieve` tool into `body.tools` (`ccr_tool_injected` in `transforms_applied`), adds the `redrive` obligation, and the response half answers the model's retrieval calls itself — the gateway just keeps re-driving. Without `can_redrive`, `ccr` mode behaves as it does today: the markers are yours to resolve.
|
||||
|
||||
@@ -96,6 +96,9 @@ class CompressTurn(Protocol):
|
||||
# Likewise optional: ``message_growth_tokens`` (message tokens it added)
|
||||
# and ``hook_tool_saved_tokens`` (tool tokens it removed and booked), so
|
||||
# the handler can net the two before clamping the saving.
|
||||
# And ``compaction_tokens_before``/``compaction_tokens_after``, the tool
|
||||
# definitions before and after the schema compaction ``transform`` ran,
|
||||
# which the handler folds into both endpoints as the chat handlers do.
|
||||
|
||||
def finish(
|
||||
self,
|
||||
|
||||
@@ -153,6 +153,14 @@ class GatewayTurn:
|
||||
def hook_tool_saved_tokens(self) -> int:
|
||||
return self._transformer.hook_tool_saved_tokens if self._transformer is not None else 0
|
||||
|
||||
@property
|
||||
def compaction_tokens_before(self) -> int:
|
||||
return self._transformer.compaction_tokens_before if self._transformer is not None else 0
|
||||
|
||||
@property
|
||||
def compaction_tokens_after(self) -> int:
|
||||
return self._transformer.compaction_tokens_after if self._transformer is not None else 0
|
||||
|
||||
# -- event-loop side ---------------------------------------------------
|
||||
def finish(
|
||||
self,
|
||||
|
||||
@@ -866,6 +866,11 @@ class RequestTransformer:
|
||||
# (booked on turn_hook_tools_saved_tokens), for signed netting.
|
||||
self.message_growth_tokens = 0
|
||||
self.hook_tool_saved_tokens = 0
|
||||
# Tool-definition tokens before and after schema compaction, both 0 when
|
||||
# it changed nothing or could not be counted. The compress handler folds
|
||||
# them into ``tokens_before``/``tokens_after``, as the chat handlers do.
|
||||
self.compaction_tokens_before = 0
|
||||
self.compaction_tokens_after = 0
|
||||
self._tokenizer: Any = None
|
||||
|
||||
def _get_tokenizer(self) -> Any:
|
||||
@@ -906,6 +911,15 @@ class RequestTransformer:
|
||||
count_tools = _count_tools_with(tokenizer)
|
||||
except Exception:
|
||||
logger.debug("gateway turn: tokenizer unavailable; hook savings uncounted")
|
||||
if tool_transforms:
|
||||
# Counted before any hook runs: a hook may edit the compacted list.
|
||||
# Tools the client already deferred are not billed, so their
|
||||
# compaction saves nothing.
|
||||
compacted_before = _safe_count(count_tools, _billed_tools(self.tools))
|
||||
compacted_after = _safe_count(count_tools, _billed_tools(tools))
|
||||
if compacted_before is not None and compacted_after is not None:
|
||||
self.compaction_tokens_before = compacted_before
|
||||
self.compaction_tokens_after = compacted_after
|
||||
|
||||
# Built even with no hook registered: the response half drives CCR
|
||||
# retrieval through the same context (messages + live tools).
|
||||
@@ -981,6 +995,13 @@ class RequestTransformer:
|
||||
return messages
|
||||
|
||||
|
||||
def _billed_tools(tools: Any) -> Any:
|
||||
"""The tools that enter the billed context: all but those flagged ``defer_loading``."""
|
||||
if not isinstance(tools, list):
|
||||
return tools
|
||||
return [t for t in tools if not (isinstance(t, dict) and t.get("defer_loading"))]
|
||||
|
||||
|
||||
def _safe_count(counter: Callable[[Any], int] | None, value: Any) -> int | None:
|
||||
if counter is None:
|
||||
return None
|
||||
|
||||
@@ -54,6 +54,7 @@ from headroom.proxy.semantic_cache_key_policy import compute_request_cache_parti
|
||||
from headroom.proxy.stage_timer import StageTimer, emit_stage_timings_log
|
||||
from headroom.proxy.tool_schema_savings_policy import (
|
||||
net_hook_tool_saving,
|
||||
tool_schema_saved_from_tags,
|
||||
without_deferral_flags,
|
||||
)
|
||||
from headroom.proxy.turn_hooks import run_responses_create_hooks
|
||||
@@ -11055,6 +11056,13 @@ class OpenAIHandlerMixin:
|
||||
# the saving, and offset tools the hook removed (signed, clamped once).
|
||||
_hook_tool_saved = 0
|
||||
if _turn is not None:
|
||||
# Tool-schema compaction, folded into both endpoints as the chat
|
||||
# handlers do: the message counts never see tool bytes.
|
||||
_compacted_before = int(getattr(_turn, "compaction_tokens_before", 0) or 0)
|
||||
_compacted_after = int(getattr(_turn, "compaction_tokens_after", 0) or 0)
|
||||
if 0 < _compacted_after < _compacted_before:
|
||||
tokens_before += _compacted_before
|
||||
tokens_after += _compacted_after
|
||||
tokens_after += max(0, int(getattr(_turn, "tool_growth_tokens", 0) or 0))
|
||||
# Only a count of pre-hook messages lacks the text a hook added;
|
||||
# a recount of the final messages (stateless fold, session) has it.
|
||||
@@ -11134,6 +11142,19 @@ class OpenAIHandlerMixin:
|
||||
}
|
||||
if session_info is not None:
|
||||
_payload["session"] = session_info
|
||||
if _turn is not None:
|
||||
# Tool definitions kept out of the context (deferral, a hook's
|
||||
# tool shrink), which the message counts never see. The headline
|
||||
# /stats reports is tokens_saved + tool_tokens_saved. The
|
||||
# deferral share is an estimate (booked ``estimated``), so it is
|
||||
# also reported on its own for a caller that bills on measured
|
||||
# savings only.
|
||||
_payload["tool_tokens_saved"] = tool_schema_saved_from_tags(tags)
|
||||
try:
|
||||
_deferred = int(tags.get("tool_search_deferred_tokens", 0) or 0)
|
||||
except (TypeError, ValueError):
|
||||
_deferred = 0
|
||||
_payload["tool_tokens_deferred"] = max(0, _deferred)
|
||||
if _finished is not None and _finished.fields:
|
||||
_payload.update(_finished.fields)
|
||||
return JSONResponse(_payload)
|
||||
|
||||
@@ -111,14 +111,23 @@ def test_message_growth_offsets_tools_a_hook_removed(
|
||||
assert (net > 0) == (repeat == 5)
|
||||
outcome = outcomes[-1]
|
||||
headline = headline_tokens_saved(outcome.tokens_saved, outcome.tags)
|
||||
# The gateway compacts tool schemas before hooks run, so compare with the
|
||||
# hook's own measured deltas (the runner's ledger), not the raw schema.
|
||||
# The answer reports the same headline, none of it estimated.
|
||||
data = resp.json()
|
||||
assert data["tokens_saved"] + data["tool_tokens_saved"] == headline
|
||||
assert data["tool_tokens_deferred"] == 0
|
||||
# The gateway compacts tool schemas before hooks run: the hook's own ledger
|
||||
# covers the compacted schema, and the compaction delta is inside
|
||||
# tokens_saved, so the headline is the two together.
|
||||
from headroom.proxy.gateway_turn import compact_gateway_tools
|
||||
|
||||
compacted, _ = compact_gateway_tools([dict(_ADDED_TOOL)])
|
||||
compaction = removed - tok.count_text(json.dumps(compacted, default=str))
|
||||
entries = from_tags(outcome.tags)
|
||||
if net > 0:
|
||||
(entry,) = entries
|
||||
hook_net = entry["details"]["tool_tokens_saved"] + entry["details"]["message_tokens_saved"]
|
||||
assert entry["details"]["message_tokens_saved"] < 0
|
||||
assert headline == hook_net == entry["tokens"]
|
||||
assert headline == hook_net + compaction == entry["tokens"] + compaction
|
||||
else:
|
||||
assert entries == []
|
||||
assert headline == 0
|
||||
|
||||
@@ -233,6 +233,32 @@ def test_hook_deferral_reaches_body_headers_and_outcome(
|
||||
assert "turn_hook_tools_saved_tokens" not in tags
|
||||
|
||||
|
||||
def test_deferred_tokens_are_reported_beside_tokens_saved(
|
||||
headroom_client, outcome_spy, deferral_hook
|
||||
) -> None:
|
||||
"""The answer carries the tool tokens deferral kept out of the context, so a
|
||||
gateway can report the same headline as /stats."""
|
||||
from headroom.proxy.tool_schema_savings_policy import headline_tokens_saved
|
||||
|
||||
outcomes = outcome_spy(headroom_client)
|
||||
sent = _body(tools=_many_anthropic_tools(), gateway={})
|
||||
data = compress(headroom_client, sent).json()
|
||||
n_tok = _assert_deferred(data, sent["tools"])
|
||||
|
||||
assert data["tool_tokens_saved"] == n_tok
|
||||
assert data["tool_tokens_deferred"] == n_tok
|
||||
outcome = outcomes[-1]
|
||||
assert data["tokens_saved"] + data["tool_tokens_saved"] == headline_tokens_saved(
|
||||
outcome.tokens_saved, outcome.tags
|
||||
)
|
||||
|
||||
|
||||
def test_no_deferral_reports_no_tool_tokens_saved(headroom_client) -> None:
|
||||
data = compress(headroom_client, _body(tools=_many_anthropic_tools(), gateway={})).json()
|
||||
assert data["tool_tokens_saved"] == 0
|
||||
assert data["tool_tokens_deferred"] == 0
|
||||
|
||||
|
||||
def test_hook_header_without_client_value_is_just_the_hooks(headroom_client, deferral_hook) -> None:
|
||||
data = compress(headroom_client, _body(tools=_many_anthropic_tools(), gateway={})).json()
|
||||
assert data["headers"] == {"anthropic-beta": BETA_TOKEN}
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
"""/v1/compress counts the tool-schema compaction it ships.
|
||||
|
||||
The gateway compacts tool schemas (``compact_gateway_tools``) and returns the
|
||||
compacted array in ``body``, but the message counts never see tool bytes. The
|
||||
chat handlers fold the compaction delta into both endpoints, so
|
||||
``tokens_saved`` covers it (``tool_schema_savings_policy``: compaction "is
|
||||
ALREADY inside tokens_saved"); this path must do the same.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from headroom.tokenizers import get_tokenizer
|
||||
from tests.gateway.conftest import compress
|
||||
|
||||
MODEL = "claude-sonnet-4-5"
|
||||
|
||||
|
||||
def _annotated_tool(i: int) -> dict[str, Any]:
|
||||
# `title` and `examples` are annotation keys schema compaction strips.
|
||||
return {
|
||||
"name": f"tool_{i}",
|
||||
"description": "Look something up.",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"title": f"Tool {i} input, a long title compaction removes",
|
||||
"properties": {
|
||||
"q": {
|
||||
"type": "string",
|
||||
"title": "Query",
|
||||
"examples": ["first example query", "second example query"],
|
||||
}
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _compress(client: Any, tools: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
body = {
|
||||
"model": MODEL,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"tools": tools,
|
||||
"gateway": {"can_redrive": False, "can_relay_response": False},
|
||||
}
|
||||
resp = compress(client, body)
|
||||
assert resp.status_code == 200, resp.text
|
||||
return resp.json()
|
||||
|
||||
|
||||
def test_compaction_is_inside_tokens_saved(make_headroom_client) -> None:
|
||||
client = make_headroom_client(optimize=False)
|
||||
sent = [_annotated_tool(i) for i in range(3)]
|
||||
data = _compress(client, sent)
|
||||
|
||||
assert "tool_schema_compaction" in data["transforms_applied"]
|
||||
tok = get_tokenizer(MODEL)
|
||||
shipped = data["body"]["tools"]
|
||||
removed = tok.count_text(json.dumps(sent, default=str)) - tok.count_text(
|
||||
json.dumps(shipped, default=str)
|
||||
)
|
||||
assert removed > 0
|
||||
# optimize=False leaves the messages alone: the whole saving is the tools.
|
||||
assert data["tokens_saved"] == removed
|
||||
assert data["tokens_before"] - data["tokens_after"] == data["tokens_saved"]
|
||||
|
||||
|
||||
def test_tools_compaction_leaves_alone_count_nothing(make_headroom_client) -> None:
|
||||
client = make_headroom_client(optimize=False)
|
||||
plain = {"name": "plain", "description": "x", "input_schema": {"type": "object"}}
|
||||
data = _compress(client, [plain])
|
||||
|
||||
assert "tool_schema_compaction" not in data["transforms_applied"]
|
||||
assert data["tokens_saved"] == 0
|
||||
|
||||
|
||||
def test_compacting_client_deferred_tools_counts_nothing(make_headroom_client) -> None:
|
||||
# A tool the client marked defer_loading is not in the billed context.
|
||||
client = make_headroom_client(optimize=False)
|
||||
deferred = [{**_annotated_tool(i), "defer_loading": True} for i in range(3)]
|
||||
data = _compress(client, deferred)
|
||||
|
||||
assert "tool_schema_compaction" in data["transforms_applied"]
|
||||
assert data["tokens_saved"] == 0
|
||||
+1
-1
@@ -346,7 +346,7 @@ Note `protect_recent` is not a substitute — it guards the newest messages, whi
|
||||
|
||||
Optionally relay the provider's usage for attribution with `POST /v1/usage` `{"session_id": "...", "usage": {...}}`. Anthropic `cache_read_input_tokens` / `cache_creation_input_tokens` and OpenAI `prompt_tokens_details.cached_tokens` / flat `cached_tokens` are accepted; a block with none of them is a 400. Replies `{"session_id", "frozen_message_count", "applied"}` (`applied: false, reason: "no_cache_signal"` when the only present field is 0), 404 `unknown_session`, or 503 `session_busy` (retry). Telemetry only — it never raises the frozen count.
|
||||
|
||||
**Gateway mode (two-half turn contract).** Add a top-level `gateway` object (`{}` is enough) and one model turn becomes two calls, so a gateway that never lets Headroom see the provider response can still run transforms whose reload step needs it — the proxy's "no shrink without reload" rule at the API boundary. `gateway` fields: `can_redrive` (default `false`: the gateway can call the provider again with a request Headroom hands it; send `false` when streaming), `can_relay_response` (default `false`: the gateway will post status/usage after each turn), `session_affinity` (default `true`; `false` disables re-driving because pending turns are in-process), `plugin_version` (diagnostics). Every other top-level field (`system`, `tools`, `temperature`, …) passes through; `tools` is compacted deterministically (`tool_schema_compaction`; `tool_desc_compaction` with `HEADROOM_TOOL_DESC_MAX_CHARS`); extension turn hooks run, stream-safe-only unless `can_redrive` and `session_affinity` both hold. The response adds `body` (the complete provider request — forward it as-is; never contains `config`, `gateway`, `token_budget`), `turn_id`, `route` (`{model, provider, service_tier, reason}`, advisory; a routing extension's model is also written into `body.model`), `obligations` (`redrive` and/or `relay_usage`) and a `gateway` echo. Fail-open answers carry the same keys with originals and `obligations: []`. A turn is registered only when `obligations` is non-empty; with `relay_usage` the `/stats` record waits for the response half. Knobs: `HEADROOM_GATEWAY_TURN_TTL_SECONDS` (120), `HEADROOM_GATEWAY_MAX_PENDING_TURNS` (10000), `HEADROOM_GATEWAY_MAX_REDRIVES` (8). With `config.mode: "ccr"` and re-drive allowed, `headroom_retrieve` is injected into `body.tools` (`ccr_tool_injected`) and the response half answers the model's retrieval calls itself.
|
||||
**Gateway mode (two-half turn contract).** Add a top-level `gateway` object (`{}` is enough) and one model turn becomes two calls, so a gateway that never lets Headroom see the provider response can still run transforms whose reload step needs it — the proxy's "no shrink without reload" rule at the API boundary. `gateway` fields: `can_redrive` (default `false`: the gateway can call the provider again with a request Headroom hands it; send `false` when streaming), `can_relay_response` (default `false`: the gateway will post status/usage after each turn), `session_affinity` (default `true`; `false` disables re-driving because pending turns are in-process), `plugin_version` (diagnostics). Every other top-level field (`system`, `tools`, `temperature`, …) passes through; `tools` is compacted deterministically (`tool_schema_compaction`; `tool_desc_compaction` with `HEADROOM_TOOL_DESC_MAX_CHARS`); extension turn hooks run, stream-safe-only unless `can_redrive` and `session_affinity` both hold. The response adds `body` (the complete provider request — forward it as-is; never contains `config`, `gateway`, `token_budget`), `turn_id`, `route` (`{model, provider, service_tier, reason}`, advisory; a routing extension's model is also written into `body.model`), `obligations` (`redrive` and/or `relay_usage`) and a `gateway` echo. Fail-open answers carry the same keys with originals and `obligations: []`. Savings follow the proxy path: when compaction shortened `tools`, `tokens_before`/`tokens_after` also count the tool definitions, and the answer adds `tool_tokens_saved` (tool-search deferral plus tools a turn hook removed, both net; `tokens_saved + tool_tokens_saved` is the `/stats` headline) and `tool_tokens_deferred` (its deferral share, booked `estimated`); both are absent on fail-open answers. A turn is registered only when `obligations` is non-empty; with `relay_usage` the `/stats` record waits for the response half. Knobs: `HEADROOM_GATEWAY_TURN_TTL_SECONDS` (120), `HEADROOM_GATEWAY_MAX_PENDING_TURNS` (10000), `HEADROOM_GATEWAY_MAX_REDRIVES` (8). With `config.mode: "ccr"` and re-drive allowed, `headroom_retrieve` is injected into `body.tools` (`ccr_tool_injected`) and the response half answers the model's retrieval calls itself.
|
||||
|
||||
`POST /v1/compress/response` (same exposure rules as `/v1/compress`) is the response half: `{"turn_id", "status": 200, "latency_ms", "usage": {...}, "response": {...}}`. `usage` accepts Anthropic, OpenAI chat (`prompt_tokens_details.cached_tokens`), OpenAI Responses (`input_tokens_details.cached_tokens`) and Kong's flat `cached_tokens` shapes, or the whole provider body (a nested `usage` key is descended into once); billed counters are summed across re-drive rounds. `response` is required when the turn carries `redrive` and the provider call succeeded; a failed call closes the turn with its `status` alone. Answers: `{"action": "done", "turn_id", "response": <replacement, the latest provider response after a re-drive, or null — forward what you hold>, "frozen_message_count", "usage_applied", "rounds", "billed_usage"}` or `{"action": "redrive", "turn_id", "request": <full provider body to send>, "round"}` — post the provider's JSON back under the same `turn_id`; `billed_usage` sums every counter (cache reads and writes included) across rounds in Anthropic keys, and a re-driven turn's `response` is never null and reports that total as its `usage`; past `HEADROOM_GATEWAY_MAX_REDRIVES` the turn ends with `done` and the latest provider response. Errors: 400 `invalid_request` / `missing_response`, 404 `unknown_turn` (unregistered, finished, or expired), 409 `turn_busy`.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user