mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Several failures on the proxy request path could not be diagnosed from the logs, and some of the existing lines could write request content into proxy.log. This PR changes log output only; request handling is unchanged. - **Catch-all 502s.** The provider catch-alls logged `<ErrorType>: <exception text>` with no provider or model. This covers Anthropic messages, Anthropic batch, OpenAI chat, OpenAI responses, Gemini generateContent, Gemini countTokens and Google batch. The exception text can come from a compressor, a router or a hook, so it can quote request content. - **Compression hooks.** Hook failures (`pre_compress`, `compute_biases`, `protect_messages`, `post_compress`) were logged at debug, so a broken operator hook silently turned itself off on every request. On the Anthropic path, `compute_biases` and `protect_messages` failures surfaced only as a generic "Optimization failed". - **CCR continuation (Anthropic buffered path).** These logs had no request id. On failure they dumped every upstream response header at error. - **Prefix-mismatch skip line.** It said only `reason=cache_mode_prefix_mismatch`, with no model or sizes. - **Silent swallows.** These swallowed exceptions with no log (`except Exception: pass`): - the prefix-mismatch debug dump; - the OpenAI cache observation; - five savings token counts; - the OpenAI Responses tool-description compaction step. Refs #4136 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made No line this PR adds or changes, at any level, carries an exception message or a traceback. Hooks, compressors, routers and tool schemas can quote request content in their errors, and logs are kept and attached to bug reports. Failures are described with `describe_exception` from `headroom/log_safety.py` (#4138): the exception types and innermost code locations of the cause chain. The full traceback appears only when the operator sets `HEADROOM_DEBUG_DUMP=full`, the existing content opt-in. - **New `headroom/proxy/handlers/_failure_logging.py`** with two functions. - **`log_request_failure()`.** Used by the 7 catch-alls and the CCR continuation. - It logs one ERROR line: request id, `key=value` context (through `safe_id`) and `describe_exception`, e.g. `[<rid>] OpenAI request failed: provider='xai' model='grok-4' ValueError at headroom/...:12 in f`. - The provider is the resolved backend (`openai_chat_outcome_provider` / `provider_name`, the same value passed to `record_failed`). - Upstream transport errors follow the same rule. An httpx `ProxyError` quotes the reason phrase a proxy sent back (`502 <peer text>`), which a peer controls, so the raw httpx text appears only under `HEADROOM_DEBUG_DUMP=full`. - The `Request failed` wording is kept, because `tests/test_system_compaction_handler.py` asserts on it. - **`log_hook_failure()`.** Used for all hook stages. - The first failure per **hook class, stage and exception type** is a WARNING. Repeats log at DEBUG. - The guard is `WarnOnce(256, ...)`. A key is not used up while WARNING is disabled, so a suppressed warning is still emitted later. Past 256 keys it logs one overflow warning, then stays quiet. - The warning states what happens next. Anthropic `compute_biases`/`protect_messages` failures say `skipping compression for this turn`; every other stage says `continuing without it`. - **Stage tracking.** - OpenAI chat reports which of `pre_compress`, `compute_biases` or `protect_messages` raised. All three still run inside one try, as before. - Anthropic `compute_biases` and `protect_messages` failures are logged by stage and re-raised, so they still abort optimization for that turn as before. Compressing without the protect veto would be unsafe. - The Anthropic `Optimization failed` warning, which those re-raised failures reach, now logs `describe_exception` instead of `{type}: {message}`. - **CCR continuation logs** (`anthropic.py`) now have a `[request_id]` prefix. On failure they log `status=` and `upstream_request_id=` instead of the full response-header dict. - **Prefix-mismatch skip line** now logs `model=`, `current_len=` and `previous_len=`. - **Former silent swallows:** - The prefix-mismatch dump catches `OSError`, `TypeError`, `ValueError` and `RecursionError` and warns with the dump directory and `describe_exception`, which keeps an `OSError`'s errno text. - The cache-observation and savings token-count failures log at debug with `describe_exception`. - The tool-description compaction step warns with `describe_exception`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uvx ruff@0.16.8 check headroom/proxy/handlers tests/test_hook_failure_logging.py tests/test_proxy_anthropic_compression_diagnostics.py All checks passed! $ uvx ruff@0.16.8 format --check (same files) 10 files already formatted $ python -m mypy headroom/proxy/handlers/_failure_logging.py headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py headroom/proxy/handlers/gemini.py headroom/proxy/handlers/batch.py headroom/log_safety.py --ignore-missing-imports Success: no issues found $ pytest -q tests/test_hook_failure_logging.py tests/test_log_safety.py tests/test_proxy_anthropic_compression_diagnostics.py tests/test_transforms_log_compressor.py tests/test_system_compaction_handler.py tests/test_proxy_handler_helpers.py tests/test_anthropic_buffered_ccr_usage.py tests/test_ccr_inline_resolve_handlers.py tests/test_openai_chat_turn_hooks.py tests/test_gemini_nonjson_status.py tests/test_proxy_handlers_batch.py tests/test_hook_protect_messages.py tests/test_openai_response_cache_key.py tests/test_anthropic_compaction_transforms.py tests/test_prefix_tracker_client_rewrote_reply.py tests/test_cache_prefix_safety.py tests/test_cache_mode_cold_start.py tests/test_original_snapshot_in_place_hook.py tests/test_openai_chat_tool_desc_compaction.py tests/test_hooks.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_gemini_integration.py tests/test_proxy_openai_responses_integration.py tests/test_ccr_buffered_stream_signed_thinking.py tests/test_openai_responses_context_compaction.py tests/test_grok_shared_proxy_call_sites.py tests/test_litellm_openai_passthrough.py tests/test_proxy_batch_integration.py 374 passed, 37 skipped in 106.13s (also tests/test_openai_codex_ws_timings.py and tests/test_openai_responses_compression_units.py) ``` `tests/test_hook_failure_logging.py` renders every captured record with `logging.Formatter` (traceback included) at DEBUG and WARNING. It asserts that a content canary and a credential canary appear in none of them, with no opt-in set. Two more use a real local HTTP proxy that rejects CONNECT with a canary reason phrase, so httpx raises its real `ProxyError`, which goes through `log_request_failure`. The canary is absent from every formatted record without the opt-in, and present with `HEADROOM_DEBUG_DUMP=full`. Two tests drive the real handlers: OpenAI chat, where a `compute_biases` failure still warns after an earlier `pre_compress` failure of the same type, and Anthropic, where the warning says compression is skipped. Run against the code before1531ac59, 4 of the then 6 tests fail: the canaries leak through the DEBUG traceback, the `Optimization failed` warning and the transport URL. Run against the code before41c72cbf, the CONNECT test fails because the ERROR line ends `(502 CANARY-USER-CONTENT)`. `tests/test_proxy_anthropic_compression_diagnostics.py` now formats the `Optimization failed` call's arguments before checking for `TimeoutError`, because the message moved from an f-string to `%s` arguments. ## Real Behavior Proof - Environment: macOS, Python 3.12, repo venv, branch `cleanup/proxy-handlers-debuggable-logs` @be9a8e9c, temp `HOME`/`HEADROOM_WORKSPACE_DIR`/`CODEX_HOME`, `HEADROOM_OFFLINE=1`, `HEADROOM_DEBUG_DUMP` unset - Exact command / steps: (1) the shared smoke harness, which starts a real `headroom proxy` from this checkout against a local fake upstream and sends one Anthropic and one OpenAI request with a compressible tool result; (2) a direct run of the app (`create_app` + TestClient), with all `headroom.proxy` records at DEBUG printed. It uses a hook whose `compute_biases` raises `ValueError("cannot bias SECRET-USER-CONTENT")` and an OpenAI upstream on a port nothing listens on, and sends two chat requests whose message is `SECRET-USER-CONTENT`. - Observed result: (1) both requests return 200 with savings, and no new WARNING/ERROR lines appear. (2) The first request gives one hook WARNING; the second gives the same line at DEBUG. Each request also gives a 502 ERROR with provider, model and the exception chain's locations. The marker appears in no output line at any level. - Not tested: a real hosted upstream or a third-party hook package ```text $ bash smoke.sh <worktree> branch: cleanup/proxy-handlers-debuggable-logs @be9a8e9chealth: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines: (only the pre-existing offline-mode notice and Kompress offline-prefetch warnings) $ HOME=$T/home HEADROOM_WORKSPACE_DIR=$T/ws CODEX_HOME=$T/codex HEADROOM_OFFLINE=1 python proof_4127.py WARNING [hr_1791649130_000001] hook __main__.BrokenBiases compute_biases failed; continuing without it: ValueError at proof_4127.py:21 in compute_biases <- headroom/proxy/handlers/openai.py:3553 in handle_openai_chat (repeats log at debug) ERROR [hr_1791649130_000001] OpenAI request failed: provider='openai' model='gpt-4o' ConnectError at default.py:118 in map_httpcore_exceptions <- contextlib.py:158 in __exit__ <- default.py:393 in handle_async_request; caused by ConnectError at _exceptions.py:14 in map_exceptions <- contextlib.py:158 in __exit__ <- anyio.py:113 in connect_tcp attempt 1: HTTP 502 DEBUG [...000002] hook __main__.BrokenBiases compute_biases failed; continuing without it: ValueError at ... ERROR [...000002] OpenAI request failed: provider='openai' model='gpt-4o' ConnectError at ... attempt 2: HTTP 502 $ grep -c SECRET out.txt 0 ``` ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: stable - Stable/default behavior changed: log output only. The catch-all ERROR lines gain provider and model. At every level they show exception types, code locations and errno instead of exception text, transport errors included. Hook failures move from debug to a once-per-hook-stage-and-type warning. The Anthropic `Optimization failed` warning no longer prints exception text. No request or response changes. - Kill switch / disable path: standard log-level configuration for the `headroom.proxy` logger - Unsafe override required: no - Qualification impact: none - Rollback path: revert the PR's commits ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. Depends on #4138; its two files show in this diff until it merges. - Tradeoff: a failure line shows exception types and code locations, not the message. The message and the full traceback need `HEADROOM_DEBUG_DUMP=full`. This keeps request content out of proxy.log at every level. - Documentation: the new and changed log lines are recorded for the follow-up docs PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
328 lines
12 KiB
Python
328 lines
12 KiB
Python
"""Request-path failure logging: named at warning/error, no payload at any level."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import contextlib
|
|
import logging
|
|
import socket
|
|
import threading
|
|
|
|
import httpx
|
|
import pytest
|
|
|
|
from headroom.hooks import CompressionHooks
|
|
from headroom.log_safety import WarnOnce
|
|
from headroom.proxy.handlers import _failure_logging
|
|
from headroom.proxy.handlers._failure_logging import log_hook_failure, log_request_failure
|
|
|
|
CONTENT = "CANARY-USER-CONTENT"
|
|
CREDENTIAL = "sk-CANARY-CREDENTIAL"
|
|
|
|
|
|
class _Capture(logging.Handler):
|
|
def __init__(self) -> None:
|
|
super().__init__(level=logging.DEBUG)
|
|
self.records: list[logging.LogRecord] = []
|
|
|
|
def emit(self, record: logging.LogRecord) -> None:
|
|
self.records.append(record)
|
|
|
|
def at(self, level: int) -> list[logging.LogRecord]:
|
|
return [r for r in self.records if r.levelno == level]
|
|
|
|
def hook_warnings(self) -> list[str]:
|
|
return [r.getMessage() for r in self.at(logging.WARNING) if "] hook " in r.getMessage()]
|
|
|
|
def assert_no_canaries(self) -> None:
|
|
"""Every record, rendered the way a log file renders it (traceback included)."""
|
|
formatter = logging.Formatter()
|
|
assert self.records
|
|
for record in self.records:
|
|
text = formatter.format(record)
|
|
assert CONTENT not in text
|
|
assert CREDENTIAL not in text
|
|
|
|
|
|
@pytest.fixture
|
|
def capture(monkeypatch: pytest.MonkeyPatch):
|
|
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
|
monkeypatch.setattr(_failure_logging, "_HOOK_WARNINGS", WarnOnce(256, "failing hooks"))
|
|
log = logging.getLogger("headroom.proxy")
|
|
handler = _Capture()
|
|
old_level = log.level
|
|
log.addHandler(handler)
|
|
log.setLevel(logging.DEBUG)
|
|
try:
|
|
yield handler
|
|
finally:
|
|
log.removeHandler(handler)
|
|
log.setLevel(old_level)
|
|
|
|
|
|
class _HooksA(CompressionHooks):
|
|
pass
|
|
|
|
|
|
class _HooksB(CompressionHooks):
|
|
pass
|
|
|
|
|
|
def _raised(exc: BaseException) -> BaseException:
|
|
"""``exc`` with a traceback, as a caught exception has."""
|
|
try:
|
|
raise exc
|
|
except BaseException as caught: # noqa: BLE001 - test helper
|
|
return caught
|
|
|
|
|
|
def test_hook_failure_warns_once_and_no_record_carries_the_payload(capture) -> None:
|
|
for request_id in ("req-1", "req-2"):
|
|
err = _raised(ValueError(f"bad message {CONTENT} key={CREDENTIAL}"))
|
|
log_hook_failure(request_id, "compute_biases", _HooksA(), err)
|
|
|
|
(warning,) = capture.hook_warnings()
|
|
assert warning.startswith("[req-1] hook ")
|
|
assert "_HooksA compute_biases failed; continuing without it: ValueError" in warning
|
|
(repeat,) = capture.at(logging.DEBUG)
|
|
assert repeat.getMessage().startswith("[req-2] hook ")
|
|
capture.assert_no_canaries()
|
|
|
|
|
|
def test_hook_identity_stage_and_error_type_each_get_their_own_warning(capture) -> None:
|
|
log_hook_failure("r1", "pre_compress", _HooksA(), ValueError("x"))
|
|
log_hook_failure("r2", "pre_compress", _HooksB(), ValueError("x")) # other hook
|
|
log_hook_failure("r3", "post_compress", _HooksA(), ValueError("x")) # other stage
|
|
log_hook_failure("r4", "pre_compress", _HooksA(), KeyError("x")) # other type
|
|
log_hook_failure("r5", "pre_compress", _HooksA(), ValueError("y")) # repeat
|
|
|
|
assert [w.split()[0] for w in capture.hook_warnings()] == ["[r1]", "[r2]", "[r3]", "[r4]"]
|
|
|
|
|
|
def test_a_suppressed_warning_is_still_emitted_later(capture) -> None:
|
|
log = logging.getLogger("headroom.proxy")
|
|
log.setLevel(logging.ERROR)
|
|
log_hook_failure("r1", "post_compress", _HooksA(), ValueError("x"))
|
|
log.setLevel(logging.INFO)
|
|
log_hook_failure("r2", "post_compress", _HooksA(), ValueError("x"))
|
|
|
|
assert [w.split()[0] for w in capture.hook_warnings()] == ["[r2]"]
|
|
|
|
|
|
@pytest.mark.parametrize("level", [logging.INFO, logging.DEBUG])
|
|
@pytest.mark.parametrize("attribute", ["__module__", "__qualname__"])
|
|
def test_hook_metadata_cannot_disclose_content_or_forge_log_lines(
|
|
capture, level, attribute
|
|
) -> None:
|
|
hooks_type = type("UntrustedHook", (), {})
|
|
setattr(hooks_type, attribute, f"{CONTENT}\nFORGED_LOG_ENTRY")
|
|
logging.getLogger("headroom.proxy").setLevel(level)
|
|
for request_id in ("first", "repeat"):
|
|
log_hook_failure(request_id, "pre_compress", hooks_type(), ValueError("failure"))
|
|
assert len(capture.hook_warnings()) == 1
|
|
capture.assert_no_canaries()
|
|
for record in capture.records:
|
|
rendered = logging.Formatter().format(record)
|
|
assert "FORGED_LOG_ENTRY" not in rendered
|
|
assert "\n" not in rendered
|
|
|
|
|
|
def test_invalid_hook_names_keep_separate_warning_budgets(capture) -> None:
|
|
for name in (f"{CONTENT}\nfirst", f"{CONTENT}\nsecond"):
|
|
hooks_type = type(name, (), {})
|
|
log_hook_failure("r1", "pre_compress", hooks_type(), ValueError("failure"))
|
|
log_hook_failure("r2", "pre_compress", hooks_type(), ValueError("failure"))
|
|
assert len(capture.hook_warnings()) == 2
|
|
capture.assert_no_canaries()
|
|
|
|
|
|
def test_hook_warning_keys_do_not_run_metaclass_hash_or_equality(capture) -> None:
|
|
class PluginMeta(type):
|
|
def __hash__(cls):
|
|
raise AssertionError("plugin hash executed")
|
|
|
|
def __eq__(cls, other):
|
|
raise AssertionError("plugin equality executed")
|
|
|
|
class PluginHook(metaclass=PluginMeta):
|
|
pass
|
|
|
|
for request_id in ("first", "repeat"):
|
|
log_hook_failure(request_id, "pre_compress", PluginHook(), ValueError("failure"))
|
|
assert len(capture.hook_warnings()) == 1
|
|
capture.assert_no_canaries()
|
|
|
|
|
|
def test_request_failure_names_the_failure_without_payload(capture) -> None:
|
|
log_request_failure(
|
|
"r1", "Request", _raised(ValueError(CONTENT)), provider="anthropic", model="m"
|
|
)
|
|
|
|
(error,) = (r.getMessage() for r in capture.at(logging.ERROR))
|
|
assert error.startswith("[r1] Request failed: provider='anthropic' model='m' ValueError")
|
|
capture.assert_no_canaries()
|
|
|
|
|
|
@contextlib.contextmanager
|
|
def _proxy_rejecting_connect(reason: str):
|
|
"""A real local HTTP proxy that answers every CONNECT with ``502 <reason>``."""
|
|
server = socket.socket()
|
|
server.bind(("127.0.0.1", 0))
|
|
server.listen()
|
|
server.settimeout(5)
|
|
|
|
def serve() -> None:
|
|
try:
|
|
conn, _ = server.accept()
|
|
except OSError: # closed before a client came
|
|
return
|
|
with conn:
|
|
conn.recv(65536)
|
|
conn.sendall(f"HTTP/1.1 502 {reason}\r\nContent-Length: 0\r\n\r\n".encode())
|
|
|
|
thread = threading.Thread(target=serve, daemon=True)
|
|
thread.start()
|
|
try:
|
|
yield f"http://127.0.0.1:{server.getsockname()[1]}"
|
|
finally:
|
|
server.close()
|
|
thread.join(timeout=5)
|
|
|
|
|
|
def _proxy_error(reason: str) -> httpx.ProxyError:
|
|
with _proxy_rejecting_connect(reason) as proxy_url:
|
|
with httpx.Client(proxy=proxy_url, timeout=5) as client:
|
|
with pytest.raises(httpx.ProxyError) as caught:
|
|
client.get("https://upstream.example/v1/messages")
|
|
assert reason in str(caught.value) # the peer's text really is in the exception
|
|
return caught.value
|
|
|
|
|
|
def test_a_proxy_reason_phrase_stays_out_of_the_request_failure_line(capture) -> None:
|
|
log_request_failure("r1", "Request", _proxy_error(CONTENT), provider="anthropic", model="m")
|
|
|
|
(error,) = (r.getMessage() for r in capture.at(logging.ERROR))
|
|
assert error.startswith("[r1] Request failed: provider='anthropic' model='m' ProxyError")
|
|
capture.assert_no_canaries()
|
|
|
|
|
|
def test_the_content_opt_in_restores_the_proxy_text(
|
|
capture, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "full")
|
|
log_request_failure("r1", "Request", _proxy_error(CONTENT), model="m")
|
|
|
|
(error,) = capture.at(logging.ERROR)
|
|
assert CONTENT in logging.Formatter().format(error)
|
|
|
|
|
|
def _proxy_app(hooks: CompressionHooks, fake_retry, *, optimize: bool): # noqa: ANN001, ANN202
|
|
pytest.importorskip("fastapi")
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
app = create_app(
|
|
ProxyConfig(optimize=optimize, cache_enabled=False, rate_limit_enabled=False, hooks=hooks)
|
|
)
|
|
return app
|
|
|
|
|
|
def _chat_response() -> httpx.Response:
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "c",
|
|
"object": "chat.completion",
|
|
"model": "gpt-4o",
|
|
"choices": [
|
|
{
|
|
"index": 0,
|
|
"message": {"role": "assistant", "content": "ok"},
|
|
"finish_reason": "stop",
|
|
}
|
|
],
|
|
"usage": {"prompt_tokens": 5, "completion_tokens": 1, "total_tokens": 6},
|
|
},
|
|
)
|
|
|
|
|
|
def test_openai_chat_warns_for_each_failing_stage(capture) -> None:
|
|
"""Through the real handler: after a pre_compress failure, a later compute_biases
|
|
failure of the same type still warns, under its own stage."""
|
|
from fastapi.testclient import TestClient
|
|
|
|
class Flaky(CompressionHooks):
|
|
failing = "pre_compress"
|
|
|
|
def pre_compress(self, messages, ctx): # noqa: ANN001, ANN201
|
|
if self.failing == "pre_compress":
|
|
raise ValueError(CONTENT)
|
|
return messages
|
|
|
|
def compute_biases(self, messages, ctx): # noqa: ANN001, ANN201
|
|
raise ValueError(CONTENT)
|
|
|
|
async def fake_retry(method, url, headers, body, *args, **kwargs): # noqa: ANN001, ANN202
|
|
return _chat_response()
|
|
|
|
hooks = Flaky()
|
|
with TestClient(_proxy_app(hooks, fake_retry, optimize=False)) as client:
|
|
# Proxy startup may reset logger levels; re-assert capture afterwards.
|
|
logging.getLogger("headroom.proxy").setLevel(logging.DEBUG)
|
|
client.app.state.proxy._retry_request = fake_retry
|
|
for failing in ("pre_compress", "compute_biases"):
|
|
hooks.failing = failing
|
|
resp = client.post(
|
|
"/v1/chat/completions",
|
|
json={"model": "gpt-4o", "messages": [{"role": "user", "content": CONTENT}]},
|
|
headers={"Authorization": f"Bearer {CREDENTIAL}"},
|
|
)
|
|
assert resp.status_code == 200, resp.text
|
|
|
|
warnings = capture.hook_warnings()
|
|
assert len(warnings) == 2
|
|
assert "Flaky pre_compress failed" in warnings[0]
|
|
assert "Flaky compute_biases failed" in warnings[1]
|
|
capture.assert_no_canaries()
|
|
|
|
|
|
def test_anthropic_bias_failure_says_compression_is_skipped(capture) -> None:
|
|
"""Through the real handler: Anthropic aborts optimization when a bias hook fails,
|
|
so the warning must say compression is skipped, not that it continues."""
|
|
from fastapi.testclient import TestClient
|
|
|
|
class BrokenBiases(CompressionHooks):
|
|
def compute_biases(self, messages, ctx): # noqa: ANN001, ANN201
|
|
raise ValueError(CONTENT)
|
|
|
|
async def fake_retry(method, url, headers, body, *args, **kwargs): # noqa: ANN001, ANN202
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"model": "claude-sonnet-4-5",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"stop_reason": "end_turn",
|
|
"usage": {"input_tokens": 5, "output_tokens": 1},
|
|
},
|
|
)
|
|
|
|
with TestClient(_proxy_app(BrokenBiases(), fake_retry, optimize=True)) as client:
|
|
logging.getLogger("headroom.proxy").setLevel(logging.DEBUG)
|
|
client.app.state.proxy._retry_request = fake_retry
|
|
resp = client.post(
|
|
"/v1/messages",
|
|
json={
|
|
"model": "claude-sonnet-4-5",
|
|
"max_tokens": 16,
|
|
"messages": [{"role": "user", "content": CONTENT}],
|
|
},
|
|
headers={"x-api-key": CREDENTIAL, "anthropic-version": "2023-06-01"},
|
|
)
|
|
|
|
assert resp.status_code == 200, resp.text
|
|
(warning,) = capture.hook_warnings()
|
|
assert "BrokenBiases compute_biases failed; skipping compression for this turn" in warning
|
|
assert "continuing without it" not in warning
|
|
capture.assert_no_canaries()
|