mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
1
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
7dc20b6fe0 |
fix(proxy): make handler failures debuggable from the logs (#4127)
## Description Several failures on the proxy request path could not be diagnosed from the logs, and some of the existing lines could write request content into proxy.log. This PR changes log output only; request handling is unchanged. - **Catch-all 502s.** The provider catch-alls logged `<ErrorType>: <exception text>` with no provider or model. This covers Anthropic messages, Anthropic batch, OpenAI chat, OpenAI responses, Gemini generateContent, Gemini countTokens and Google batch. The exception text can come from a compressor, a router or a hook, so it can quote request content. - **Compression hooks.** Hook failures (`pre_compress`, `compute_biases`, `protect_messages`, `post_compress`) were logged at debug, so a broken operator hook silently turned itself off on every request. On the Anthropic path, `compute_biases` and `protect_messages` failures surfaced only as a generic "Optimization failed". - **CCR continuation (Anthropic buffered path).** These logs had no request id. On failure they dumped every upstream response header at error. - **Prefix-mismatch skip line.** It said only `reason=cache_mode_prefix_mismatch`, with no model or sizes. - **Silent swallows.** These swallowed exceptions with no log (`except Exception: pass`): - the prefix-mismatch debug dump; - the OpenAI cache observation; - five savings token counts; - the OpenAI Responses tool-description compaction step. Refs #4136 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made No line this PR adds or changes, at any level, carries an exception message or a traceback. Hooks, compressors, routers and tool schemas can quote request content in their errors, and logs are kept and attached to bug reports. Failures are described with `describe_exception` from `headroom/log_safety.py` (#4138): the exception types and innermost code locations of the cause chain. The full traceback appears only when the operator sets `HEADROOM_DEBUG_DUMP=full`, the existing content opt-in. - **New `headroom/proxy/handlers/_failure_logging.py`** with two functions. - **`log_request_failure()`.** Used by the 7 catch-alls and the CCR continuation. - It logs one ERROR line: request id, `key=value` context (through `safe_id`) and `describe_exception`, e.g. `[<rid>] OpenAI request failed: provider='xai' model='grok-4' ValueError at headroom/...:12 in f`. - The provider is the resolved backend (`openai_chat_outcome_provider` / `provider_name`, the same value passed to `record_failed`). - Upstream transport errors follow the same rule. An httpx `ProxyError` quotes the reason phrase a proxy sent back (`502 <peer text>`), which a peer controls, so the raw httpx text appears only under `HEADROOM_DEBUG_DUMP=full`. - The `Request failed` wording is kept, because `tests/test_system_compaction_handler.py` asserts on it. - **`log_hook_failure()`.** Used for all hook stages. - The first failure per **hook class, stage and exception type** is a WARNING. Repeats log at DEBUG. - The guard is `WarnOnce(256, ...)`. A key is not used up while WARNING is disabled, so a suppressed warning is still emitted later. Past 256 keys it logs one overflow warning, then stays quiet. - The warning states what happens next. Anthropic `compute_biases`/`protect_messages` failures say `skipping compression for this turn`; every other stage says `continuing without it`. - **Stage tracking.** - OpenAI chat reports which of `pre_compress`, `compute_biases` or `protect_messages` raised. All three still run inside one try, as before. - Anthropic `compute_biases` and `protect_messages` failures are logged by stage and re-raised, so they still abort optimization for that turn as before. Compressing without the protect veto would be unsafe. - The Anthropic `Optimization failed` warning, which those re-raised failures reach, now logs `describe_exception` instead of `{type}: {message}`. - **CCR continuation logs** (`anthropic.py`) now have a `[request_id]` prefix. On failure they log `status=` and `upstream_request_id=` instead of the full response-header dict. - **Prefix-mismatch skip line** now logs `model=`, `current_len=` and `previous_len=`. - **Former silent swallows:** - The prefix-mismatch dump catches `OSError`, `TypeError`, `ValueError` and `RecursionError` and warns with the dump directory and `describe_exception`, which keeps an `OSError`'s errno text. - The cache-observation and savings token-count failures log at debug with `describe_exception`. - The tool-description compaction step warns with `describe_exception`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uvx ruff@0.16.8 check headroom/proxy/handlers tests/test_hook_failure_logging.py tests/test_proxy_anthropic_compression_diagnostics.py All checks passed! $ uvx ruff@0.16.8 format --check (same files) 10 files already formatted $ python -m mypy headroom/proxy/handlers/_failure_logging.py headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py headroom/proxy/handlers/gemini.py headroom/proxy/handlers/batch.py headroom/log_safety.py --ignore-missing-imports Success: no issues found $ pytest -q tests/test_hook_failure_logging.py tests/test_log_safety.py tests/test_proxy_anthropic_compression_diagnostics.py tests/test_transforms_log_compressor.py tests/test_system_compaction_handler.py tests/test_proxy_handler_helpers.py tests/test_anthropic_buffered_ccr_usage.py tests/test_ccr_inline_resolve_handlers.py tests/test_openai_chat_turn_hooks.py tests/test_gemini_nonjson_status.py tests/test_proxy_handlers_batch.py tests/test_hook_protect_messages.py tests/test_openai_response_cache_key.py tests/test_anthropic_compaction_transforms.py tests/test_prefix_tracker_client_rewrote_reply.py tests/test_cache_prefix_safety.py tests/test_cache_mode_cold_start.py tests/test_original_snapshot_in_place_hook.py tests/test_openai_chat_tool_desc_compaction.py tests/test_hooks.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_gemini_integration.py tests/test_proxy_openai_responses_integration.py tests/test_ccr_buffered_stream_signed_thinking.py tests/test_openai_responses_context_compaction.py tests/test_grok_shared_proxy_call_sites.py tests/test_litellm_openai_passthrough.py tests/test_proxy_batch_integration.py 374 passed, 37 skipped in 106.13s (also tests/test_openai_codex_ws_timings.py and tests/test_openai_responses_compression_units.py) ``` `tests/test_hook_failure_logging.py` renders every captured record with `logging.Formatter` (traceback included) at DEBUG and WARNING. It asserts that a content canary and a credential canary appear in none of them, with no opt-in set. Two more use a real local HTTP proxy that rejects CONNECT with a canary reason phrase, so httpx raises its real `ProxyError`, which goes through `log_request_failure`. The canary is absent from every formatted record without the opt-in, and present with `HEADROOM_DEBUG_DUMP=full`. Two tests drive the real handlers: OpenAI chat, where a `compute_biases` failure still warns after an earlier `pre_compress` failure of the same type, and Anthropic, where the warning says compression is skipped. Run against the code before |