mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
## Description Several savings and metrics paths swallow exceptions and log nothing, so when one fails, a customer-facing number can come out wrong with no record of why. This PR adds a log line on each of those paths. The numbers and the control flow are unchanged. Refs #4136 | Where | Before | Now | |---|---|---| | `tool_schema_savings_policy.reconcile_deferred_tokens` | A failure left the tool-search deferral credit as booked, so `/stats` over-reported. Nothing was logged. | Warns once per exception type, giving the booked token count and the fact that `/stats` is over-reported. Every occurrence is also logged at debug. | | `outcome.py` output-savings record/estimate | Booked 0 silently. | Debug, with the request id. | | `server.py` `/stats` `output_reduction` | The section went missing silently. | Debug. | | `server.py` `create_app` saved-settings apply | `settings_store.load()` already warns about an unreadable or corrupt file. Any other error while applying it was swallowed. | Warns with the settings file path and says to fix or remove that file and restart. Startup continues. | | `server.py` security gate, audit record | Debug only. | Warns with the method and path, and says the request is missing from the `headroom.audit` trail. | | `savings_tracker` pricing for an unpriced model | Raised `RuntimeError` only to reach the fallback, and logged nothing. | Returns the fallback directly and logs once per model at debug. The `except` blocks now see only unexpected errors, and log them at debug. | | `savings_tracker._resolve_litellm_model` probes | Two `except Exception: pass` blocks. | Debug. litellm raises a bare `Exception` for a model it does not know, so the catch cannot be narrower. | **No content at any level.** A tokenizer error can quote the client's tool schema, and a settings or pricing error can quote a credential. So every line this PR adds describes a failure with `describe_exception(exc)` from `headroom/log_safety.py`: the exception types and code locations of the cause chain, never the message and never a traceback. Model ids go through `safe_id`, and audit paths are capped the same way. The full text appears only when the operator sets `HEADROOM_DEBUG_DUMP=full`. **Bounded dedupe.** The reconcile guard (keyed by exception type) and the unpriced-model guard (keyed by model) use the shared `WarnOnce`. Each key is logged once, at most a fixed number of keys per hour, followed by one overflow notice. The unpriced-model guard is consulted only when DEBUG is enabled, so a model is not used up while its line is suppressed. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/tool_schema_savings_policy.py`: `_log_reconcile_failure` uses `WarnOnce(32, ...)` and `describe_exception`. - `headroom/proxy/outcome.py`, `headroom/proxy/server.py`, `headroom/proxy/savings_tracker.py`: add the lines in the table above, all through `describe_exception` and `safe_id`. `_log_unpriced_model_once` uses `WarnOnce`. - Tests: - `tests/test_tool_schema_savings_policy.py`: - a tokenizer error quoting a private-schema canary; - every captured DEBUG and WARNING record is rendered with `logging.Formatter()`, which includes `exc_info`, and none contains the canary; - the booked credit stays unchanged; - one warning per exception type, a debug line per failure. - `tests/test_savings_pricing_fallback_logging.py`: - a pricing error and a litellm probe error that quote an API-key canary, with the canary absent from every formatted record; - an unpriced model logs once; - nothing is used up while DEBUG is off. - `tests/test_saved_settings_apply_logging.py`: a settings-apply error that quotes an API-key canary through a real `create_app`. One warning names the file and the exception type, and no formatted record contains the canary. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # New regressions against the previous commit's product files (c5850f02), in a scratch worktree: FAILED tests/test_tool_schema_savings_policy.py::test_reconcile_failure_keeps_private_schema_out_of_every_log_record E assert 'PRIVATE-SCHEMA-CANARY-7f3a' not in "tool-schema...NARY-7f3a'}]" FAILED tests/test_tool_schema_savings_policy.py::test_reconcile_warns_again_for_a_new_exception_type FAILED tests/test_savings_pricing_fallback_logging.py::test_unpriced_model_logs_once FAILED tests/test_savings_pricing_fallback_logging.py::test_credential_bearing_pricing_error_is_described_not_quoted FAILED tests/test_savings_pricing_fallback_logging.py::test_unpriced_model_not_marked_when_debug_disabled FAILED tests/test_saved_settings_apply_logging.py::test_settings_apply_failure_warns_without_credential =================== 6 failed, 10 passed, 1 warning in 3.54s ==================== # This branch: $ pytest -q -p no:cacheprovider tests/test_tool_schema_savings_policy.py tests/test_savings_pricing_fallback_logging.py tests/test_saved_settings_apply_logging.py ======================== 16 passed, 1 warning in 2.77s ========================= $ pytest -q -p no:cacheprovider <every test file referencing the touched estimators, reconcile_deferred_tokens, settings_store, record_admin_action or log_safety> + the savings/stats/output-shaper files ======================= 401 passed, 1 warning in 44.33s ======================== $ uvx ruff@0.16.8 check <touched files> && uvx ruff@0.16.8 format --check headroom/proxy All checks passed! $ mypy headroom/proxy/tool_schema_savings_policy.py headroom/proxy/savings_tracker.py headroom/proxy/server.py headroom/proxy/outcome.py --ignore-missing-imports Success: no issues found in 4 source files ``` `tests/test_cost_card_attribution.py` and `tests/test_savings_provider_matrix.py` also fail at collection on a clean `origin/main`, because the local litellm build prices none of their candidate models. I left them out of the runs above. ## Real Behavior Proof - Environment: macOS, Python 3.12.13, this branch atd3834fea, the repo .venv, a temp `HOME`, `HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`, and `HEADROOM_OFFLINE=1`. - Exact command / steps: - (1) The shared smoke harness, `smoke.sh <worktree>`. It runs a real `headroom proxy` from this checkout against a local fake upstream, sends one Anthropic and one OpenAI request that each carry a compressible tool result, reads `/stats`, and greps `proxy.log`. - (2) A script that forces each new path: - `settings_store.apply_to_environ` raises `RuntimeError("value sk-secret-should-not-appear")` under `create_app()`; - `reconcile_deferred_tokens` is called twice with a counter that raises `RuntimeError("... secret-schema-text")`; - the unknown model `local-llama-7b` is priced three times with the savings-tracker logger at DEBUG. - Observed result: ```text (1) branch: cleanup/proxy-core-b-savings-logging @d3834feahealth: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 (2) WARNING headroom.proxy: Could not apply saved settings from <tmp>/ws/settings.json (RuntimeError at pcb_4084_proof.py:16 in _boom <- headroom/proxy/server.py:3141 in create_app); continuing with env and defaults. Fix or remove that file and restart the proxy. WARNING headroom.proxy.tool_schema_savings_policy: tool-schema deferral reconcile failed (RuntimeError at pcb_4084_proof.py:31 in _broken <- headroom/proxy/tool_schema_savings_policy.py:146 in reconcile_deferred_tokens); credit left at 100 booked tokens, so /stats over-reports tool-search savings for this request. Please report this as a bug. DEBUG headroom.proxy.savings_tracker: litellm cannot price model='local-llama-7b' as given: BadRequestError at get_llm_provider_logic.py:515 in get_llm_provider <- get_llm_provider_logic.py:528 in get_llm_provider <- cost_calculator.py:472 in cost_per_token DEBUG headroom.proxy.savings_tracker: No litellm input_cost_per_token for model='local-llama-7b'; savings and cost use the fallback rate ``` - Neither secret string appears in any line. - The second reconcile failure does not warn again. - The unpriced model is logged once across three calls. - Not tested: - Forcing the audit-record failure inside a running proxy. It uses the same `describe_exception` pattern. - Seeing the debug lines in a running proxy's `proxy.log`. That depends on #4137, which makes `HEADROOM_LOG_LEVEL=debug` reach `headroom.*`. Until it lands, these debug lines show only when the `headroom` logger is set to DEBUG programmatically, as in the script. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A; logging only. - Stable/default behavior changed: No. Every dollar value is the same as before, and an unpriced model now gets its fallback returned directly instead of through a raise and catch, with an identical result. Only the log output changes. - Kill switch / disable path: None needed. Warnings fire only on a real failure, and they are deduplicated with a bound. - Unsafe override required: No. `HEADROOM_DEBUG_DUMP=full` is the existing explicit opt-in for full exception text. - Qualification impact: None. - Rollback path: Revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. Docs for the new warnings are collected for the single follow-up docs PR. Depends on #4138; its files (`headroom/log_safety.py`, `handlers/_debug_dump.py` and their tests) show in this diff until it merges. This PR's own diff, excluding #4138's files, is +377/−20, and 256 of those added lines are tests. `cost._warn_pricing_once` is not reused for unpriced models because it logs at WARNING, and it shares its set with the live cost path, which already warns once per unpriceable model. For an unknown model, cache savings book 0 while compression savings use the $3/M fallback. This PR leaves that unchanged for a maintainer decision. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
106 lines
3.7 KiB
Python
106 lines
3.7 KiB
Python
"""Savings pricing fallbacks are debuggable without flooding or leaking into the log.
|
|
|
|
An unpriced model is expected (local models, gateway aliases) and the estimators
|
|
run on every request, so it is logged once per model. Unexpected pricing errors
|
|
are logged by exception type and code location only: their text can carry a
|
|
credential (a provider error quoting an API key) and must not reach any level.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
|
|
import pytest
|
|
|
|
import headroom.proxy.savings_tracker as st
|
|
from headroom.log_safety import WarnOnce
|
|
|
|
CREDENTIAL_CANARY = "sk-ant-CREDENTIAL-CANARY-91c2"
|
|
|
|
|
|
class _Capture(logging.Handler):
|
|
def __init__(self) -> None:
|
|
super().__init__(level=logging.DEBUG)
|
|
self.records: list[logging.LogRecord] = []
|
|
|
|
def emit(self, record: logging.LogRecord) -> None:
|
|
self.records.append(record)
|
|
|
|
|
|
class _CredentialEchoingPrice:
|
|
"""A price value whose conversion fails with a message quoting a key."""
|
|
|
|
def __float__(self) -> float:
|
|
raise ValueError(f"bad price from provider config api_key={CREDENTIAL_CANARY}")
|
|
|
|
|
|
@pytest.fixture
|
|
def captured(monkeypatch: pytest.MonkeyPatch):
|
|
class _FakeLiteLLM:
|
|
model_cost = {"weird-priced-model": {"input_cost_per_token": _CredentialEchoingPrice()}}
|
|
|
|
@staticmethod
|
|
def cost_per_token(**_kwargs):
|
|
raise Exception(f"This model isn't mapped yet. key={CREDENTIAL_CANARY}")
|
|
|
|
monkeypatch.setattr(st, "_get_litellm_module", lambda: _FakeLiteLLM())
|
|
monkeypatch.setattr(st, "_unpriced_models_logged", WarnOnce(256, "test unpriced models"))
|
|
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
|
st._resolve_litellm_model.cache_clear()
|
|
handler = _Capture()
|
|
old_level = st.logger.level
|
|
st.logger.addHandler(handler)
|
|
st.logger.setLevel(logging.DEBUG)
|
|
try:
|
|
yield handler.records
|
|
finally:
|
|
st.logger.removeHandler(handler)
|
|
st.logger.setLevel(old_level)
|
|
st._resolve_litellm_model.cache_clear()
|
|
|
|
|
|
def _assert_no_canary(records: list[logging.LogRecord]) -> None:
|
|
formatter = logging.Formatter()
|
|
for record in records:
|
|
assert CREDENTIAL_CANARY not in formatter.format(record)
|
|
|
|
|
|
def test_unpriced_model_logs_once(captured) -> None:
|
|
for _ in range(3):
|
|
assert st._estimate_compression_savings_usd("local-llama-7b", 1000) == pytest.approx(
|
|
1000 * st.DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN
|
|
)
|
|
st._estimate_output_savings_usd("local-llama-7b", 1000)
|
|
st._estimate_input_cost_usd("local-llama-7b", 1000)
|
|
|
|
unpriced = [r for r in captured if "No litellm" in r.getMessage()]
|
|
assert len(unpriced) == 1
|
|
assert "model='local-llama-7b'" in unpriced[0].getMessage()
|
|
assert all(r.exc_info is None for r in captured)
|
|
# The litellm probe error quoted a key; it never reaches the log.
|
|
_assert_no_canary(captured)
|
|
|
|
|
|
def test_credential_bearing_pricing_error_is_described_not_quoted(captured) -> None:
|
|
assert st._estimate_compression_savings_usd("weird-priced-model", 1000) == pytest.approx(
|
|
1000 * st.DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN
|
|
)
|
|
|
|
failures = [
|
|
r for r in captured if "fallback rate for model='weird-priced-model'" in r.getMessage()
|
|
]
|
|
assert len(failures) == 1
|
|
assert "ValueError" in failures[0].getMessage()
|
|
assert not any("No litellm" in r.getMessage() for r in captured)
|
|
_assert_no_canary(captured)
|
|
|
|
|
|
def test_unpriced_model_not_marked_when_debug_disabled(captured) -> None:
|
|
st.logger.setLevel(logging.INFO)
|
|
st._estimate_compression_savings_usd("quiet-model", 10)
|
|
st.logger.setLevel(logging.DEBUG)
|
|
st._estimate_compression_savings_usd("quiet-model", 10)
|
|
assert [
|
|
r for r in captured if "quiet-model" in r.getMessage() and "No litellm" in r.getMessage()
|
|
]
|