mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 12:05:34 +08:00
## Description Several savings and metrics paths swallow exceptions and log nothing, so when one fails, a customer-facing number can come out wrong with no record of why. This PR adds a log line on each of those paths. The numbers and the control flow are unchanged. Refs #4136 | Where | Before | Now | |---|---|---| | `tool_schema_savings_policy.reconcile_deferred_tokens` | A failure left the tool-search deferral credit as booked, so `/stats` over-reported. Nothing was logged. | Warns once per exception type, giving the booked token count and the fact that `/stats` is over-reported. Every occurrence is also logged at debug. | | `outcome.py` output-savings record/estimate | Booked 0 silently. | Debug, with the request id. | | `server.py` `/stats` `output_reduction` | The section went missing silently. | Debug. | | `server.py` `create_app` saved-settings apply | `settings_store.load()` already warns about an unreadable or corrupt file. Any other error while applying it was swallowed. | Warns with the settings file path and says to fix or remove that file and restart. Startup continues. | | `server.py` security gate, audit record | Debug only. | Warns with the method and path, and says the request is missing from the `headroom.audit` trail. | | `savings_tracker` pricing for an unpriced model | Raised `RuntimeError` only to reach the fallback, and logged nothing. | Returns the fallback directly and logs once per model at debug. The `except` blocks now see only unexpected errors, and log them at debug. | | `savings_tracker._resolve_litellm_model` probes | Two `except Exception: pass` blocks. | Debug. litellm raises a bare `Exception` for a model it does not know, so the catch cannot be narrower. | **No content at any level.** A tokenizer error can quote the client's tool schema, and a settings or pricing error can quote a credential. So every line this PR adds describes a failure with `describe_exception(exc)` from `headroom/log_safety.py`: the exception types and code locations of the cause chain, never the message and never a traceback. Model ids go through `safe_id`, and audit paths are capped the same way. The full text appears only when the operator sets `HEADROOM_DEBUG_DUMP=full`. **Bounded dedupe.** The reconcile guard (keyed by exception type) and the unpriced-model guard (keyed by model) use the shared `WarnOnce`. Each key is logged once, at most a fixed number of keys per hour, followed by one overflow notice. The unpriced-model guard is consulted only when DEBUG is enabled, so a model is not used up while its line is suppressed. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/tool_schema_savings_policy.py`: `_log_reconcile_failure` uses `WarnOnce(32, ...)` and `describe_exception`. - `headroom/proxy/outcome.py`, `headroom/proxy/server.py`, `headroom/proxy/savings_tracker.py`: add the lines in the table above, all through `describe_exception` and `safe_id`. `_log_unpriced_model_once` uses `WarnOnce`. - Tests: - `tests/test_tool_schema_savings_policy.py`: - a tokenizer error quoting a private-schema canary; - every captured DEBUG and WARNING record is rendered with `logging.Formatter()`, which includes `exc_info`, and none contains the canary; - the booked credit stays unchanged; - one warning per exception type, a debug line per failure. - `tests/test_savings_pricing_fallback_logging.py`: - a pricing error and a litellm probe error that quote an API-key canary, with the canary absent from every formatted record; - an unpriced model logs once; - nothing is used up while DEBUG is off. - `tests/test_saved_settings_apply_logging.py`: a settings-apply error that quotes an API-key canary through a real `create_app`. One warning names the file and the exception type, and no formatted record contains the canary. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # New regressions against the previous commit's product files (c5850f02), in a scratch worktree: FAILED tests/test_tool_schema_savings_policy.py::test_reconcile_failure_keeps_private_schema_out_of_every_log_record E assert 'PRIVATE-SCHEMA-CANARY-7f3a' not in "tool-schema...NARY-7f3a'}]" FAILED tests/test_tool_schema_savings_policy.py::test_reconcile_warns_again_for_a_new_exception_type FAILED tests/test_savings_pricing_fallback_logging.py::test_unpriced_model_logs_once FAILED tests/test_savings_pricing_fallback_logging.py::test_credential_bearing_pricing_error_is_described_not_quoted FAILED tests/test_savings_pricing_fallback_logging.py::test_unpriced_model_not_marked_when_debug_disabled FAILED tests/test_saved_settings_apply_logging.py::test_settings_apply_failure_warns_without_credential =================== 6 failed, 10 passed, 1 warning in 3.54s ==================== # This branch: $ pytest -q -p no:cacheprovider tests/test_tool_schema_savings_policy.py tests/test_savings_pricing_fallback_logging.py tests/test_saved_settings_apply_logging.py ======================== 16 passed, 1 warning in 2.77s ========================= $ pytest -q -p no:cacheprovider <every test file referencing the touched estimators, reconcile_deferred_tokens, settings_store, record_admin_action or log_safety> + the savings/stats/output-shaper files ======================= 401 passed, 1 warning in 44.33s ======================== $ uvx ruff@0.16.8 check <touched files> && uvx ruff@0.16.8 format --check headroom/proxy All checks passed! $ mypy headroom/proxy/tool_schema_savings_policy.py headroom/proxy/savings_tracker.py headroom/proxy/server.py headroom/proxy/outcome.py --ignore-missing-imports Success: no issues found in 4 source files ``` `tests/test_cost_card_attribution.py` and `tests/test_savings_provider_matrix.py` also fail at collection on a clean `origin/main`, because the local litellm build prices none of their candidate models. I left them out of the runs above. ## Real Behavior Proof - Environment: macOS, Python 3.12.13, this branch atd3834fea, the repo .venv, a temp `HOME`, `HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`, and `HEADROOM_OFFLINE=1`. - Exact command / steps: - (1) The shared smoke harness, `smoke.sh <worktree>`. It runs a real `headroom proxy` from this checkout against a local fake upstream, sends one Anthropic and one OpenAI request that each carry a compressible tool result, reads `/stats`, and greps `proxy.log`. - (2) A script that forces each new path: - `settings_store.apply_to_environ` raises `RuntimeError("value sk-secret-should-not-appear")` under `create_app()`; - `reconcile_deferred_tokens` is called twice with a counter that raises `RuntimeError("... secret-schema-text")`; - the unknown model `local-llama-7b` is priced three times with the savings-tracker logger at DEBUG. - Observed result: ```text (1) branch: cleanup/proxy-core-b-savings-logging @d3834feahealth: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 (2) WARNING headroom.proxy: Could not apply saved settings from <tmp>/ws/settings.json (RuntimeError at pcb_4084_proof.py:16 in _boom <- headroom/proxy/server.py:3141 in create_app); continuing with env and defaults. Fix or remove that file and restart the proxy. WARNING headroom.proxy.tool_schema_savings_policy: tool-schema deferral reconcile failed (RuntimeError at pcb_4084_proof.py:31 in _broken <- headroom/proxy/tool_schema_savings_policy.py:146 in reconcile_deferred_tokens); credit left at 100 booked tokens, so /stats over-reports tool-search savings for this request. Please report this as a bug. DEBUG headroom.proxy.savings_tracker: litellm cannot price model='local-llama-7b' as given: BadRequestError at get_llm_provider_logic.py:515 in get_llm_provider <- get_llm_provider_logic.py:528 in get_llm_provider <- cost_calculator.py:472 in cost_per_token DEBUG headroom.proxy.savings_tracker: No litellm input_cost_per_token for model='local-llama-7b'; savings and cost use the fallback rate ``` - Neither secret string appears in any line. - The second reconcile failure does not warn again. - The unpriced model is logged once across three calls. - Not tested: - Forcing the audit-record failure inside a running proxy. It uses the same `describe_exception` pattern. - Seeing the debug lines in a running proxy's `proxy.log`. That depends on #4137, which makes `HEADROOM_LOG_LEVEL=debug` reach `headroom.*`. Until it lands, these debug lines show only when the `headroom` logger is set to DEBUG programmatically, as in the script. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A; logging only. - Stable/default behavior changed: No. Every dollar value is the same as before, and an unpriced model now gets its fallback returned directly instead of through a raise and catch, with an identical result. Only the log output changes. - Kill switch / disable path: None needed. Warnings fire only on a real failure, and they are deduplicated with a bound. - Unsafe override required: No. `HEADROOM_DEBUG_DUMP=full` is the existing explicit opt-in for full exception text. - Qualification impact: None. - Rollback path: Revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. Docs for the new warnings are collected for the single follow-up docs PR. Depends on #4138; its files (`headroom/log_safety.py`, `handlers/_debug_dump.py` and their tests) show in this diff until it merges. This PR's own diff, excluding #4138's files, is +377/−20, and 256 of those added lines are tests. `cost._warn_pricing_once` is not reused for unpriced models because it logs at WARNING, and it shares its set with the live cost path, which already warns once per unpriceable model. For an unknown model, cache savings book 0 while compression savings use the $3/M fallback. This PR leaves that unchanged for a maintainer decision. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
254 lines
9.4 KiB
Python
254 lines
9.4 KiB
Python
"""Tests for the upstream-error diagnostic-dump gating.
|
|
|
|
The dump can contain cleartext prompt/tool/system content, so it must be OFF by
|
|
default, never written in stateless mode, and content-redacted unless the
|
|
operator explicitly opts in to full content.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import inspect
|
|
import json
|
|
import os
|
|
from types import SimpleNamespace
|
|
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
from headroom.proxy.handlers._debug_dump import _debug_dump_mode, _redact_debug_value
|
|
|
|
|
|
def _config(stateless: bool = False) -> SimpleNamespace:
|
|
return SimpleNamespace(stateless=stateless)
|
|
|
|
|
|
def test_debug_dump_off_by_default(monkeypatch):
|
|
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
|
assert _debug_dump_mode(_config()) == "off"
|
|
|
|
|
|
@pytest.mark.parametrize("value", ["1", "true", "yes", "on", "redacted", "REDACTED"])
|
|
def test_debug_dump_opt_in_redacted(monkeypatch, value):
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", value)
|
|
assert _debug_dump_mode(_config()) == "redacted"
|
|
|
|
|
|
@pytest.mark.parametrize("value", ["full", "all", "content"])
|
|
def test_debug_dump_opt_in_full(monkeypatch, value):
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", value)
|
|
assert _debug_dump_mode(_config()) == "full"
|
|
|
|
|
|
def test_debug_dump_unknown_value_is_off(monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "maybe")
|
|
assert _debug_dump_mode(_config()) == "off"
|
|
|
|
|
|
def test_stateless_forces_dump_off_even_when_opted_in(monkeypatch):
|
|
# Stateless mode must win over any opt-in: no filesystem writes, period.
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "full")
|
|
assert _debug_dump_mode(_config(stateless=True)) == "off"
|
|
|
|
|
|
def test_redact_elides_long_strings_keeps_structure():
|
|
payload = {
|
|
"role": "user",
|
|
"type": "text",
|
|
"id": "msg_123",
|
|
"text": "secret prompt content " * 20, # long → redacted
|
|
"blocks": [
|
|
{"type": "tool_use", "name": "search", "input": "x" * 500},
|
|
{"type": "text", "text": "short"},
|
|
],
|
|
}
|
|
out = _redact_debug_value(payload)
|
|
# Short structural fields preserved:
|
|
assert out["role"] == "user"
|
|
assert out["type"] == "text"
|
|
assert out["id"] == "msg_123"
|
|
assert out["blocks"][0]["name"] == "search"
|
|
assert out["blocks"][1]["text"] == "short"
|
|
# Long content elided to a length placeholder (no original content leaks):
|
|
assert out["text"].startswith("<redacted:") and "secret prompt" not in out["text"]
|
|
assert out["blocks"][0]["input"].startswith("<redacted:")
|
|
|
|
|
|
def test_redact_passes_through_non_strings():
|
|
assert _redact_debug_value(42) == 42
|
|
assert _redact_debug_value(None) is None
|
|
assert _redact_debug_value(True) is True
|
|
|
|
|
|
@pytest.mark.parametrize("module_name", ["anthropic", "openai"])
|
|
def test_both_handlers_gate_the_dump(module_name):
|
|
"""Regression guard: every handler that writes a debug dump must gate it on
|
|
_debug_dump_mode (off by default). Prevents reintroducing an unguarded dump
|
|
that writes cleartext prompts to disk."""
|
|
import importlib
|
|
|
|
module = importlib.import_module(f"headroom.proxy.handlers.{module_name}")
|
|
src = inspect.getsource(module)
|
|
if "debug_400_dir(" in src:
|
|
assert "_debug_dump_mode(self.config)" in src, (
|
|
f"{module_name} writes a debug dump but does not gate it on _debug_dump_mode"
|
|
)
|
|
assert 'if dump_mode != "off":' in src, (
|
|
f"{module_name} debug dump is not guarded by an off-by-default check"
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# write_upstream_error_dump: the shared writer used by the streaming handler
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@pytest.fixture
|
|
def dump_dir(tmp_path, monkeypatch):
|
|
"""Point ``paths.debug_400_dir()`` at a temp dir for the writer tests."""
|
|
from headroom import paths
|
|
|
|
target = tmp_path / "debug_400"
|
|
monkeypatch.setattr(paths, "debug_400_dir", lambda: target)
|
|
return target
|
|
|
|
|
|
def _write(**kwargs):
|
|
from headroom.proxy.handlers._debug_dump import write_upstream_error_dump
|
|
|
|
params = {
|
|
"request_id": "req_1",
|
|
"url": "https://api.anthropic.com/v1/messages",
|
|
"status": 400,
|
|
"provider": "anthropic",
|
|
"model": "claude-opus-5",
|
|
"body": {"messages": [{"role": "user", "content": "secret prompt content " * 20}]},
|
|
"transforms": ["tool_search_history_repair"],
|
|
"stream": True,
|
|
}
|
|
params.update(kwargs)
|
|
config = params.pop("config", _config())
|
|
return write_upstream_error_dump(config, **params)
|
|
|
|
|
|
def test_upstream_dump_off_by_default_writes_nothing(dump_dir, monkeypatch):
|
|
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
|
assert _write() is None
|
|
assert not dump_dir.exists()
|
|
|
|
|
|
def test_upstream_dump_stateless_writes_nothing(dump_dir, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "full")
|
|
assert _write(config=_config(stateless=True)) is None
|
|
assert not dump_dir.exists()
|
|
|
|
|
|
def test_upstream_dump_full_records_request(dump_dir, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "full")
|
|
path = _write()
|
|
assert path is not None and path.parent == dump_dir
|
|
payload = json.loads(path.read_text())
|
|
assert payload["request_id"] == "req_1"
|
|
assert payload["status"] == 400
|
|
assert payload["provider"] == "anthropic"
|
|
assert payload["model"] == "claude-opus-5"
|
|
assert payload["stream"] is True
|
|
assert payload["transforms"] == ["tool_search_history_repair"]
|
|
assert "secret prompt content" in payload["body"]["messages"][0]["content"]
|
|
|
|
|
|
def test_upstream_dump_redacted_elides_content(dump_dir, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "1")
|
|
path = _write()
|
|
assert path is not None
|
|
payload = json.loads(path.read_text())
|
|
content = payload["body"]["messages"][0]["content"]
|
|
assert content.startswith("<redacted:") and "secret prompt" not in content
|
|
# Structure is still there to debug with:
|
|
assert payload["body"]["messages"][0]["role"] == "user"
|
|
|
|
|
|
def test_upstream_dump_serializes_non_json_values(dump_dir, monkeypatch):
|
|
# ``default=str`` must keep an exotic body from raising mid-dump.
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "full")
|
|
path = _write(body={"when": object()})
|
|
assert path is not None
|
|
assert "object object" in json.loads(path.read_text())["body"]["when"]
|
|
|
|
|
|
@pytest.mark.parametrize("mode", ["1", "full"])
|
|
def test_upstream_dump_never_writes_url_credentials(dump_dir, monkeypatch, mode):
|
|
# Gemini streaming URLs carry the API key as ``?key=``; ``full`` opts in to
|
|
# prompt content, not to credentials, so the query is dropped in every mode.
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", mode)
|
|
url = (
|
|
"https://generativelanguage.googleapis.com/v1beta/models/gemini:streamGenerateContent"
|
|
"?alt=sse&key=AIzaSECRET"
|
|
)
|
|
path = _write(url=url, provider="gemini")
|
|
assert path is not None
|
|
text = path.read_text()
|
|
assert "AIzaSECRET" not in text
|
|
if mode == "full": # the content opt-in keeps the route, still never the key
|
|
expected = (
|
|
"https://generativelanguage.googleapis.com/v1beta/models/gemini:streamGenerateContent"
|
|
)
|
|
else: # a path segment can carry a credential, so it is dropped without the opt-in
|
|
expected = "https://generativelanguage.googleapis.com/<path>"
|
|
assert json.loads(text)["url"] == expected + "?<redacted>"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"url",
|
|
["https://upstream.example/botCANARY/v1/messages", "https://upstream.example/bot%43ANARY/v1"],
|
|
)
|
|
def test_redacted_dump_never_writes_path_credentials(dump_dir, monkeypatch, url):
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "redacted")
|
|
path = _write(url=url)
|
|
assert path is not None
|
|
text = path.read_text()
|
|
assert "CANARY" not in text and "%43ANARY" not in text
|
|
|
|
|
|
def test_upstream_dump_keeps_query_free_url(dump_dir, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "full")
|
|
path = _write()
|
|
assert json.loads(path.read_text())["url"] == "https://api.anthropic.com/v1/messages"
|
|
|
|
|
|
def test_upstream_dump_records_the_bytes_actually_sent(dump_dir, monkeypatch):
|
|
# When the proxy's edits are dropped (passthrough), the dump must show the
|
|
# body that went on the wire, not the edited one that never left.
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "full")
|
|
sent = json.dumps({"messages": [{"role": "user", "content": "as sent"}]}).encode()
|
|
path = _write(body=sent, body_source="passthrough")
|
|
payload = json.loads(path.read_text())
|
|
assert payload["body"]["messages"][0]["content"] == "as sent"
|
|
assert payload["body_source"] == "passthrough"
|
|
|
|
|
|
def test_upstream_dump_tolerates_non_json_bytes(dump_dir, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "full")
|
|
path = _write(body=b"\x00\x01not json")
|
|
assert json.loads(path.read_text())["body"] == "<10 bytes, not JSON>"
|
|
|
|
|
|
@pytest.mark.skipif(os.name == "nt", reason="POSIX permission bits")
|
|
def test_upstream_dump_is_owner_only(dump_dir, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "full")
|
|
path = _write()
|
|
assert path.stat().st_mode & 0o777 == 0o600
|
|
|
|
|
|
def test_upstream_dump_never_raises_when_write_fails(monkeypatch):
|
|
# A diagnostic must not turn an upstream error into a proxy error.
|
|
from headroom import paths
|
|
|
|
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "full")
|
|
|
|
def _boom():
|
|
raise OSError("read-only filesystem")
|
|
|
|
monkeypatch.setattr(paths, "debug_400_dir", _boom)
|
|
assert _write() is None
|