mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): say why proxy.log is missing and why a knob or extension was ignored (#4088)
## Description Several failures an operator needs to see are swallowed silently or logged only at DEBUG. This PR makes each of them visible with one warning. It does not change defaults or control flow. 1. **Missing runtime log.** `_setup_file_logging` caught `OSError` with `pass`. On a read-only workspace, a wrong owner, or a file where the logs directory belongs, `proxy.log` simply never appeared and nothing said why. It now logs `Headroom runtime log disabled: cannot write the log file: <OSError>. Point HEADROOM_WORKSPACE_DIR at a writable directory to restore proxy.log.` The OSError text names the path. The symlink case just above it already warned this way. 2. **Six import-time numeric knobs.** Each one fell back to its default on an unparseable value, without a word: - `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` - `HEADROOM_COLD_START_FAST_PASS_TIMEOUT_SECONDS` - `HEADROOM_EAGER_PRELOAD_TIMEOUT_SECONDS` - `HEADROOM_COMPRESSION_CACHE_MAX_SESSIONS` - `HEADROOM_COMPRESSION_CACHE_TTL_SECONDS` - `HEADROOM_COMPRESSION_CACHE_MAX_ENTRIES` Each now logs `Ignoring invalid NAME='value'; using DEFAULT` once, at import. That matches the idiom in `hard_watchdog.py`. 3. **Extension discovery failure.** When entry-point enumeration fails, every proxy extension is dropped, Enterprise plugins included. That was logged at DEBUG. It is now a warning: `proxy extensions: entry-point enumeration failed (<describe_exception>), so no proxy extensions will be installed`. `describe_exception` gives the exception types and code locations, never the message, and no traceback is logged at any level. It runs at startup only: once in `install_all`, and once more for the CLI banner. 4. **Pricing lookups in `cost.py`.** `_get_list_price`, `_get_output_price`, `_get_cache_prices` and the cache-rate lookup in the savings summary returned `None` on any exception. A pricing bug therefore showed up as a $0 cost with no trace. All four, plus the existing `estimate_cost` path, go through one helper, `_warn_pricing_once(lookup, what, model, exc)`. Example line: `Failed to get pricing for model 'gpt-4o': RuntimeError at headroom/proxy/cost.py:... in estimate_cost`. It logs the lookup name, `safe_id(model)` and `describe_exception(exc)` (types and code locations, never the message), at WARNING only. No level carries the raw exception or a traceback, because a pricing backend error can quote provider responses or credentials. Dedupe is by `(lookup, model)` through the shared `WarnOnce(1024, "unpriceable models")`. Each pair warns at most once an hour (the `WarnOnce` window). Past 1024 models, one overflow notice is logged, and a key is not used up while WARNING is disabled. 5. **Malformed `HEADROOM_OPENAI_TOOL_SEARCH_MODELS` regex.** The override was ignored on every request with no log. It now warns once per bad pattern, and still falls back to the version gate as before. A valid pattern re-arms the warning, so bad A, then valid B, then bad A again warns twice. None of the new lines logs header values, request content or credentials. They log only knob names, their raw values (operator-set env), model names, regex patterns, exception type names, the `re.error` text, and `OSError` text naming a local path. No exception message or traceback from pricing or extension discovery reaches any log level. Refs #4136 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: - add `_warn_invalid_env_number(name, default)` and call it from the six import-time `except ValueError` branches; - log the `OSError` in `_setup_file_logging`; - add a once-per-pattern warning for a malformed `HEADROOM_OPENAI_TOOL_SEARCH_MODELS`. - `headroom/proxy/extensions.py`: entry-point enumeration failure logs at warning with `describe_exception(exc)` instead of at debug. - `headroom/proxy/cost.py`: - the four silent pricing `except Exception` branches call `_warn_pricing_once`; - `_warn_pricing_once(lookup, what, model, exc)` logs `safe_id(model)` plus `describe_exception(exc)` at WARNING, with nothing at DEBUG. It dedupes through `log_safety.WarnOnce`. - `tests/test_cost_pricing_warning_dedup.py`: the tests reset with a fresh `WarnOnce`, and match the quoted model id (`for model 'glm-5.2':`). - Regression tests. Each one attaches a handler to the module logger directly, because caplog misses `headroom.*` records after proxy startup. Each positive test fails on main and is paired with a negative case that passes on both. - `tests/test_cache_breakpoint_diagnostics.py`: - `test_runtime_log_says_why_it_is_missing` makes the logs directory a regular file. It asserts one warning that names the path and `HEADROOM_WORKSPACE_DIR`. - Negative: `test_runtime_log_stays_quiet_when_writable`. - `tests/test_openai_tool_search_deferral.py`: - `test_malformed_override_warns_once_not_per_request` calls the gate 5 times with a bad pattern. It asserts exactly one warning and that the version-gate result is unchanged. - Negative: a valid override is silent (same test). - `test_valid_override_rearms_the_malformed_warning`: bad, valid, bad gives two warnings. - `tests/test_proxy/test_compression_timeout_config.py`: - `test_bad_env_says_it_was_ignored` reloads `helpers` with `HEADROOM_COMPRESSION_TIMEOUT_SECONDS=not-a-number` and asserts the exact warning. - Negative: `test_valid_env_is_silent`. The existing fixture reloads `helpers` afterwards. - `tests/test_proxy_extensions.py`: - `test_discover_warns_when_entry_point_enumeration_fails` makes `importlib.metadata.entry_points` raise. It asserts one WARNING that says no extensions will be installed and names `RuntimeError` but not the exception text. - Negative: `test_discover_is_quiet_when_nothing_is_installed`. - `tests/test_cost_cache_prices.py`: - `test_cache_rate_failure_warns_once_per_model` makes `resolve_rates` raise. It asserts two `_get_cache_prices` calls give one warning naming the model and `RuntimeError`, without the exception text. - Negative: a resolvable model is silent (same test). - `test_pricing_error_never_logs_its_text_at_any_level` runs the real `CostTracker.estimate_cost('gpt-4o', 100, 10)` with a pricing error whose text holds an Authorization credential canary, with `headroom.proxy` at DEBUG. It formats every captured record with `logging.Formatter` (which renders `exc_info`), asserts the canary is absent and the result is `None`, and asserts the warning is present. On the previous commit it fails: the formatted DEBUG traceback contains the canary. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q tests/test_cost_cache_prices.py -k never_logs # previous commit3202058c(test adapted to its old dedupe set) E RuntimeError: upstream rejected request (Authorization: Bearer sk-canary-7f3a9c)'] ======================= 1 failed, 25 deselected in 3.48s ======================= $ pytest -q -p no:cacheprovider tests/test_pr208_changes.py tests/test_cost_budget_basis.py tests/test_cost_tracker_counterfactual.py tests/test_cost_tracker_totals.py tests/test_tool_search_client_deferral.py tests/test_cache_breakpoint_diagnostics.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_byte_faithful_forwarding.py tests/test_memory_tool_mode.py tests/test_codex_ws_compression_scheduler.py tests/test_proxy_savings_history.py tests/test_compression_cache_registry.py tests/test_proxy_extension_ordering.py tests/test_proxy_eager_preload_bind.py tests/test_perf_per_port_logs.py tests/test_cost_cache_prices.py tests/test_openai_tool_search_deferral.py tests/test_proxy_extensions.py tests/test_direct_chat_ccr_resolution.py tests/test_org_scale_limits.py tests/test_savings_ledger.py tests/test_cost_pricing_warning_dedup.py tests/test_proxy/test_compression_timeout_config.py tests/test_log_safety.py #64ff3f03================== 517 passed, 1 skipped in 85.70s (0:01:25) =================== $ pytest -q (same pricing/extension/regex/log/knob/log_safety files) after merging log-safety-helpers @0ce74a63#4ff57af4======================== 139 passed, 1 skipped in 4.23s ======================== $ uvx ruff@0.16.8 check / format --check headroom/proxy/cost.py headroom/proxy/extensions.py tests/test_cost_cache_prices.py tests/test_cost_pricing_warning_dedup.py tests/test_proxy_extensions.py All checks passed! / 5 files already formatted $ mypy headroom/proxy/cost.py headroom/proxy/extensions.py --ignore-missing-imports Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS, Python 3.12.13, branch checkout at4ff57af4(log-safety-helpers merged at0ce74a63), using the repo .venv. A temp HOME, workspace and CODEX_HOME were used each time; the real HOME was not touched. - Exact command / steps: 1. Ran the shared smoke harness, which starts a real proxy from this checkout against a local fake upstream and sends one Anthropic and one OpenAI request. 2. Ran the real `CostTracker.estimate_cost('gpt-4o', 100, 10)` twice, with every logger at DEBUG and a LiteLLM stand-in whose `cost_per_token` raises an error quoting `Authorization: Bearer sk-canary-7f3a9c`. 3. The earlier forced-failure run for the knob, log-directory and regex warnings (d83e7d5c, below) is unchanged by this commit. - Observed result: ```text branch: cleanup/proxy-core-a-surface-ignored-config @4ff57af4health: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines (<tmp>): (smoke dir: <tmp>) ``` ```text WARNING headroom.proxy: Failed to get pricing for model 'gpt-4o': RuntimeError at <string>:8 in cost_per_token <- headroom/proxy/cost.py:999 in estimate_cost result: None None ``` - One payload-safe WARNING appears, with the exception type and code location, and both calls return `None`. - The canary appears in no output line at any level: `grep -c canary` over the whole run, LiteLLM's own DEBUG chatter included, gives 0. - Earlier forced-failure run: ```text branch: cleanup/proxy-core-a-surface-ignored-config @d83e7d5chealth: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines (<tmp>): (smoke dir: <tmp>) ``` The harness also printed two pre-existing WARNING lines, which I left out above: offline-mode air-gap active, and the Kompress model prefetch refused offline. Neither comes from this PR. Output of the direct command: ```text WARNING headroom.proxy: Ignoring invalid HEADROOM_COMPRESSION_TIMEOUT_SECONDS='abc'; using 30.0 WARNING headroom.proxy: Headroom runtime log disabled: cannot write the log file: [Errno 17] File exists: '<tmp>/ws/logs'. Point HEADROOM_WORKSPACE_DIR at a writable directory to restore proxy.log. WARNING headroom.proxy: Ignoring invalid HEADROOM_OPENAI_TOOL_SEARCH_MODELS regex '[bad' (unterminated character set at position 0); using the gpt-5.4+ version gate gate: True True ``` - The proxy serves both providers, and the normal path adds no new warnings. - Each forced failure gives exactly one warning, even though the regex gate ran twice. The fallbacks are unchanged: 30.0, and the version gate is True for gpt-5.4. - Not tested: A real read-only filesystem, a real broken `importlib.metadata` and a live pricing-catalog failure (the stand-in above raises from the same call). Each goes through the same `except` branch that the unit tests force. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A; log output only. - Stable/default behavior changed: No. Every fallback value and code path is the same; only new WARNING lines appear, when something was already failing. - Kill switch / disable path: N/A. A pricing warning fires at most once an hour per (lookup, model), through `WarnOnce`; the regex warning fires once per bad pattern. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Part of a maintainer-requested codebase cleanup (dead code, duplication, debuggability), split into small PRs for review. It is part of maintainer-requested cleanup #4136. Depends on #4138; its two files show in this diff until it merges. #4137 also edits `_setup_file_logging`, so whichever of the two merges second rebases. Docs are unchanged because no knob's meaning changed. The doc impacts are recorded in the cleanup docs ledger for the follow-up docs PR. The six import-time knob warnings have no test: testing them would need `importlib.reload(helpers)`, which would reset the module globals other tests depend on. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
db72e6717f
commit
72302a3998
+30
-15
@@ -16,6 +16,7 @@ from datetime import datetime, timedelta
|
||||
from typing import TYPE_CHECKING, Any, NamedTuple
|
||||
|
||||
from headroom.cache_economics import CACHE_ECONOMICS as _CACHE_ECONOMICS
|
||||
from headroom.log_safety import WarnOnce, describe_exception, safe_id
|
||||
from headroom.proxy.budget_basis_policy import (
|
||||
BUDGET_BASIS_BLOCK,
|
||||
BUDGET_BASIS_IGNORE,
|
||||
@@ -57,17 +58,26 @@ logger = logging.getLogger("headroom.proxy")
|
||||
# Pricing-lookup warnings are emitted on the per-request cost path, so an
|
||||
# unresolvable model (a custom / OpenAI-compatible name LiteLLM can't price,
|
||||
# e.g. glm-5.2) floods proxy.log with an identical WARNING every single request
|
||||
# (#2504). Track which models have already been warned so each fires once per
|
||||
# process; the set is tiny and bounded by the number of distinct models seen.
|
||||
_warned_pricing_models: set[str] = set()
|
||||
# (#2504). Each (lookup, model) pair warns once; model names come from clients,
|
||||
# so WarnOnce also caps how many distinct models can warn.
|
||||
_pricing_warnings = WarnOnce(1024, "unpriceable models")
|
||||
|
||||
|
||||
def _warn_pricing_once(model: str, message: str) -> None:
|
||||
"""Emit ``message`` at WARNING only the first time ``model`` fails pricing."""
|
||||
if model in _warned_pricing_models:
|
||||
def _warn_pricing_once(
|
||||
lookup: str, what: str, model: str, exc: BaseException | None = None
|
||||
) -> None:
|
||||
"""Warn once per ``(lookup, model)`` that pricing failed, payload-safe.
|
||||
|
||||
A pricing backend's exception can quote provider responses or credentials,
|
||||
so only :func:`describe_exception` (types and code locations) is logged, at
|
||||
every level.
|
||||
"""
|
||||
if not _pricing_warnings.first((lookup, model), logger):
|
||||
return
|
||||
_warned_pricing_models.add(model)
|
||||
logger.warning(message)
|
||||
if exc is None:
|
||||
logger.warning("%s for model %s", what, safe_id(model))
|
||||
else:
|
||||
logger.warning("%s for model %s: %s", what, safe_id(model), describe_exception(exc))
|
||||
|
||||
|
||||
# A route whose responses never carry a usage breakdown hits the estimated-basis
|
||||
@@ -275,7 +285,10 @@ def build_prefix_cache_stats(
|
||||
from headroom.pricing.counterfactual import resolve_rates
|
||||
|
||||
cache_prices = resolve_rates(model_name)
|
||||
except Exception:
|
||||
except Exception as e:
|
||||
_warn_pricing_once(
|
||||
"cache_rates", "Failed to resolve cache rates", model_name, e
|
||||
)
|
||||
cache_prices = None
|
||||
best_tokens = tokens_sent
|
||||
|
||||
@@ -972,8 +985,7 @@ class CostTracker:
|
||||
litellm = _get_litellm_module()
|
||||
if litellm is None:
|
||||
_warn_pricing_once(
|
||||
f"__litellm_unavailable__:{model}",
|
||||
f"LiteLLM not available - cannot calculate costs for model {model}",
|
||||
"litellm_unavailable", "LiteLLM not available - cannot calculate costs", model
|
||||
)
|
||||
return None
|
||||
|
||||
@@ -996,7 +1008,7 @@ class CostTracker:
|
||||
return float(total_cost) if total_cost > 0 else None
|
||||
|
||||
except Exception as e:
|
||||
_warn_pricing_once(model, f"Failed to get pricing for model {model}: {e}")
|
||||
_warn_pricing_once("estimate", "Failed to get pricing", model, e)
|
||||
return None
|
||||
|
||||
def _prune_old_costs(self):
|
||||
@@ -1369,7 +1381,8 @@ class CostTracker:
|
||||
info = litellm.model_cost.get(resolved, {})
|
||||
cost_per_token = info.get("input_cost_per_token")
|
||||
return cost_per_token * 1_000_000 if cost_per_token else None
|
||||
except Exception:
|
||||
except Exception as e:
|
||||
_warn_pricing_once("list_price", "Failed to get list price", model, e)
|
||||
return None
|
||||
|
||||
def _get_output_price(self, model: str, *, long_context: bool = False) -> float | None:
|
||||
@@ -1390,7 +1403,8 @@ class CostTracker:
|
||||
if long_context:
|
||||
return info.get("output_cost_per_token_above_200k_tokens") or base or None
|
||||
return base or None
|
||||
except Exception:
|
||||
except Exception as e:
|
||||
_warn_pricing_once("output_price", "Failed to get output price", model, e)
|
||||
return None
|
||||
|
||||
def _write_cost_usd(self, model: str, w5m_price: float, w1h_price: float) -> float:
|
||||
@@ -1444,7 +1458,8 @@ class CostTracker:
|
||||
from headroom.pricing.counterfactual import resolve_rates
|
||||
|
||||
rates = resolve_rates(model, long_context=long_context, for_billing=for_billing)
|
||||
except Exception:
|
||||
except Exception as e:
|
||||
_warn_pricing_once("cache_rates", "Failed to resolve cache rates", model, e)
|
||||
return None
|
||||
if rates is None or not rates.uncached:
|
||||
return None
|
||||
|
||||
@@ -103,6 +103,8 @@ import sys
|
||||
from collections.abc import Callable, Iterable, Iterator
|
||||
from typing import Any
|
||||
|
||||
from headroom.log_safety import describe_exception
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
ENTRY_POINT_GROUP = "headroom.proxy_extension"
|
||||
@@ -123,7 +125,11 @@ def discover() -> Iterator[tuple[str, ProxyExtension]]:
|
||||
try:
|
||||
entries = importlib.metadata.entry_points(group=ENTRY_POINT_GROUP)
|
||||
except Exception as exc: # noqa: BLE001 — importlib.metadata can raise varied types
|
||||
log.debug("proxy extensions: entry-point enumeration failed: %s", exc)
|
||||
log.warning(
|
||||
"proxy extensions: entry-point enumeration failed (%s), so no proxy "
|
||||
"extensions will be installed",
|
||||
describe_exception(exc),
|
||||
)
|
||||
return
|
||||
for entry in entries:
|
||||
try:
|
||||
|
||||
@@ -1416,6 +1416,12 @@ def parse_sse_events_from_byte_buffer(
|
||||
# Maximum message array length (prevents DoS from deeply nested payloads)
|
||||
MAX_MESSAGE_ARRAY_LENGTH = 10000
|
||||
|
||||
|
||||
def _warn_invalid_env_number(name: str, default: float) -> None:
|
||||
"""Say which numeric knob was ignored, so a typo does not silently revert it."""
|
||||
logger.warning("Ignoring invalid %s=%r; using %s", name, os.environ.get(name), default)
|
||||
|
||||
|
||||
# Compression pipeline timeout in seconds. Override via the
|
||||
# HEADROOM_COMPRESSION_TIMEOUT_SECONDS env var for slow CPUs or long Claude Code
|
||||
# conversations (GH #946). Falls back to 30 on an unparseable value.
|
||||
@@ -1425,6 +1431,7 @@ try:
|
||||
)
|
||||
except ValueError:
|
||||
COMPRESSION_TIMEOUT_SECONDS = 30.0
|
||||
_warn_invalid_env_number("HEADROOM_COMPRESSION_TIMEOUT_SECONDS", COMPRESSION_TIMEOUT_SECONDS)
|
||||
|
||||
# Cold-start fast-pass timeout in seconds. When background compression defers
|
||||
# a cold-start-large request, the handler still runs the pipeline synchronously
|
||||
@@ -1439,6 +1446,9 @@ try:
|
||||
)
|
||||
except ValueError:
|
||||
COLD_START_FAST_PASS_TIMEOUT_SECONDS = 10.0
|
||||
_warn_invalid_env_number(
|
||||
"HEADROOM_COLD_START_FAST_PASS_TIMEOUT_SECONDS", COLD_START_FAST_PASS_TIMEOUT_SECONDS
|
||||
)
|
||||
|
||||
# Eager startup preload timeout in seconds. The preload (compressor/parser models,
|
||||
# cache-only, allow_download=False) runs off the event loop during startup; this
|
||||
@@ -1451,6 +1461,9 @@ try:
|
||||
)
|
||||
except ValueError:
|
||||
EAGER_PRELOAD_TIMEOUT_SECONDS = 120.0
|
||||
_warn_invalid_env_number(
|
||||
"HEADROOM_EAGER_PRELOAD_TIMEOUT_SECONDS", EAGER_PRELOAD_TIMEOUT_SECONDS
|
||||
)
|
||||
|
||||
# Maximum compression cache sessions (prevents unbounded memory growth).
|
||||
# Overridable via HEADROOM_COMPRESSION_CACHE_MAX_SESSIONS for gateway
|
||||
@@ -1462,6 +1475,9 @@ try:
|
||||
)
|
||||
except ValueError:
|
||||
MAX_COMPRESSION_CACHE_SESSIONS = 500
|
||||
_warn_invalid_env_number(
|
||||
"HEADROOM_COMPRESSION_CACHE_MAX_SESSIONS", MAX_COMPRESSION_CACHE_SESSIONS
|
||||
)
|
||||
|
||||
# Idle TTL for per-session compression caches. Eviction is bust-free only
|
||||
# once the provider's own prompt cache has lapsed, so this must exceed the
|
||||
@@ -1486,6 +1502,9 @@ try:
|
||||
COMPRESSION_CACHE_TTL_SECONDS = max(600.0, _ttl_env)
|
||||
except ValueError:
|
||||
COMPRESSION_CACHE_TTL_SECONDS = 3900.0
|
||||
_warn_invalid_env_number(
|
||||
"HEADROOM_COMPRESSION_CACHE_TTL_SECONDS", COMPRESSION_CACHE_TTL_SECONDS
|
||||
)
|
||||
|
||||
# Entries per session compression cache. 10k covers a single conversation with
|
||||
# ~2x headroom even at a 1M-token context (a compressible tool_result is at
|
||||
@@ -1500,6 +1519,9 @@ try:
|
||||
)
|
||||
except ValueError:
|
||||
COMPRESSION_CACHE_MAX_ENTRIES = 10000
|
||||
_warn_invalid_env_number(
|
||||
"HEADROOM_COMPRESSION_CACHE_MAX_ENTRIES", COMPRESSION_CACHE_MAX_ENTRIES
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -1933,9 +1955,14 @@ def _setup_file_logging(
|
||||
headroom_logger.removeHandler(stale)
|
||||
stale.close()
|
||||
headroom_logger.addHandler(handler)
|
||||
except OSError:
|
||||
# Non-fatal: can't write logs (read-only fs, permissions, etc.)
|
||||
pass
|
||||
except OSError as exc:
|
||||
# Non-fatal: can't write logs (read-only fs, permissions, etc.). Say so,
|
||||
# or the missing proxy.log is the only clue. The OSError names the path.
|
||||
logger.warning(
|
||||
"Headroom runtime log disabled: cannot write the log file: %s. "
|
||||
"Point HEADROOM_WORKSPACE_DIR at a writable directory to restore proxy.log.",
|
||||
exc,
|
||||
)
|
||||
|
||||
|
||||
def is_anthropic_auth(headers: dict[str, str]) -> bool:
|
||||
@@ -4561,6 +4588,9 @@ _OPENAI_TOOL_SEARCH_UNSUPPORTED_CLIENTS = frozenset({"codex", "opencode"})
|
||||
# HEADROOM_OPENAI_TOOL_SEARCH_MODELS (matched against the model name) so new
|
||||
# model families can be enabled without a code edit + release.
|
||||
_OPENAI_TOOL_SEARCH_MIN_VERSION = (5, 4)
|
||||
# Last malformed override pattern warned about, so a bad regex is reported once
|
||||
# instead of on every request.
|
||||
_openai_tool_search_bad_pattern: str | None = None
|
||||
|
||||
|
||||
def _model_supports_openai_tool_search(model: str | None) -> bool:
|
||||
@@ -4571,14 +4601,27 @@ def _model_supports_openai_tool_search(model: str | None) -> bool:
|
||||
when set; a malformed pattern falls back to the version gate rather than
|
||||
crashing.
|
||||
"""
|
||||
global _openai_tool_search_bad_pattern
|
||||
if not model:
|
||||
return False
|
||||
override = os.environ.get("HEADROOM_OPENAI_TOOL_SEARCH_MODELS", "").strip()
|
||||
if override:
|
||||
try:
|
||||
return re.search(override, model) is not None
|
||||
except re.error:
|
||||
pass # malformed override → fall back to the version gate
|
||||
matched = re.search(override, model) is not None
|
||||
except re.error as exc:
|
||||
# Malformed override → fall back to the version gate.
|
||||
if override != _openai_tool_search_bad_pattern and logger.isEnabledFor(logging.WARNING):
|
||||
_openai_tool_search_bad_pattern = override
|
||||
logger.warning(
|
||||
"Ignoring invalid HEADROOM_OPENAI_TOOL_SEARCH_MODELS regex %r (%s); "
|
||||
"using the gpt-5.4+ version gate",
|
||||
override,
|
||||
exc,
|
||||
)
|
||||
else:
|
||||
# A valid pattern re-arms the warning for the next bad one.
|
||||
_openai_tool_search_bad_pattern = None
|
||||
return matched
|
||||
match = re.match(r"gpt-(\d+)(?:\.(\d+))?", model.strip().lower())
|
||||
if not match:
|
||||
return False
|
||||
|
||||
@@ -403,6 +403,63 @@ def test_runtime_log_refuses_a_symlinked_path(tmp_path, monkeypatch) -> None:
|
||||
assert stat.S_IMODE(elsewhere.stat().st_mode) == 0o666, "the symlink target was chmodded"
|
||||
|
||||
|
||||
class _ListHandler(logging.Handler):
|
||||
def __init__(self) -> None:
|
||||
super().__init__(logging.DEBUG)
|
||||
self.records: list[logging.LogRecord] = []
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
self.records.append(record)
|
||||
|
||||
|
||||
def _runtime_log_warnings(port: int) -> list[str]:
|
||||
"""Run ``_setup_file_logging`` and return the 'runtime log disabled' warnings.
|
||||
|
||||
The handler sits on ``headroom.proxy`` directly: caplog does not see
|
||||
``headroom.*`` records once a proxy app has configured logging.
|
||||
"""
|
||||
from headroom.proxy.helpers import _setup_file_logging
|
||||
|
||||
proxy_logger = logging.getLogger("headroom.proxy")
|
||||
headroom_logger = logging.getLogger("headroom")
|
||||
before = list(headroom_logger.handlers)
|
||||
capture = _ListHandler()
|
||||
proxy_logger.addHandler(capture)
|
||||
try:
|
||||
_setup_file_logging(port)
|
||||
finally:
|
||||
proxy_logger.removeHandler(capture)
|
||||
for handler in list(headroom_logger.handlers):
|
||||
if handler not in before:
|
||||
headroom_logger.removeHandler(handler)
|
||||
handler.close()
|
||||
return [
|
||||
r.getMessage()
|
||||
for r in capture.records
|
||||
if r.levelno == logging.WARNING and "runtime log disabled" in r.getMessage()
|
||||
]
|
||||
|
||||
|
||||
def test_runtime_log_says_why_it_is_missing(tmp_path, monkeypatch) -> None:
|
||||
"""An unwritable log directory used to disable proxy.log without a word."""
|
||||
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
|
||||
# A regular file where the logs directory belongs: mkdir raises OSError.
|
||||
_paths.log_dir().write_text("", encoding="utf-8")
|
||||
|
||||
warnings = _runtime_log_warnings(18808)
|
||||
|
||||
assert len(warnings) == 1
|
||||
assert str(_paths.log_dir()) in warnings[0]
|
||||
assert "HEADROOM_WORKSPACE_DIR" in warnings[0]
|
||||
|
||||
|
||||
def test_runtime_log_stays_quiet_when_writable(tmp_path, monkeypatch) -> None:
|
||||
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
|
||||
|
||||
assert _runtime_log_warnings(18809) == []
|
||||
assert _paths.proxy_log_path(18809).exists()
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
os.name != "posix",
|
||||
reason="creating a symlink needs elevation on Windows, and O_NOFOLLOW does not exist there",
|
||||
|
||||
@@ -9,10 +9,12 @@ and inferred writes cannot double-charge the uncached input bucket.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.log_safety import WarnOnce
|
||||
from headroom.pricing.counterfactual import resolve_rates
|
||||
from headroom.proxy.cost import CostTracker
|
||||
|
||||
@@ -358,3 +360,79 @@ class TestGetCachePrices:
|
||||
rates = resolve_rates("m", provider="anthropic")
|
||||
assert rates.read == rates.write_5m == 1e-6
|
||||
assert rates.read_is_catalog and rates.write_is_catalog
|
||||
|
||||
|
||||
class _Capture(logging.Handler):
|
||||
def __init__(self) -> None:
|
||||
super().__init__(logging.DEBUG)
|
||||
self.records: list[logging.LogRecord] = []
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
self.records.append(record)
|
||||
|
||||
|
||||
def test_cache_rate_failure_warns_once_per_model(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A raising rate resolver used to read as 'unpriced' ($0) with no trace."""
|
||||
from headroom.proxy import cost
|
||||
|
||||
monkeypatch.setattr(cost, "_pricing_warnings", WarnOnce(1024, "unpriceable models"))
|
||||
_patch_litellm(
|
||||
monkeypatch,
|
||||
{"m": {"input_cost_per_token": 1e-6, "litellm_provider": "anthropic"}},
|
||||
)
|
||||
capture = _Capture()
|
||||
cost.logger.addHandler(capture)
|
||||
try:
|
||||
assert CostTracker()._get_cache_prices("m") is not None
|
||||
assert not capture.records # a resolvable model is silent
|
||||
|
||||
def broken(*_args, **_kwargs):
|
||||
raise RuntimeError("pricing table corrupt")
|
||||
|
||||
monkeypatch.setattr("headroom.pricing.counterfactual.resolve_rates", broken)
|
||||
assert CostTracker()._get_cache_prices("m") is None
|
||||
assert CostTracker()._get_cache_prices("m") is None
|
||||
finally:
|
||||
cost.logger.removeHandler(capture)
|
||||
|
||||
warnings = [r.getMessage() for r in capture.records if r.levelno == logging.WARNING]
|
||||
assert len(warnings) == 1
|
||||
assert "for model 'm'" in warnings[0] and "RuntimeError" in warnings[0]
|
||||
assert "pricing table corrupt" not in warnings[0]
|
||||
|
||||
|
||||
_CREDENTIAL_CANARY = "Authorization: Bearer sk-canary-7f3a9c"
|
||||
|
||||
|
||||
def test_pricing_error_never_logs_its_text_at_any_level(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A pricing backend error quoting a credential must not reach any log record.
|
||||
|
||||
Formats every WARNING and DEBUG record with a real Formatter, which renders
|
||||
exc_info too, so a traceback carrying the text would be caught.
|
||||
"""
|
||||
from headroom.proxy import cost
|
||||
|
||||
class _LeakyLiteLLM:
|
||||
model_cost: dict = {}
|
||||
|
||||
@staticmethod
|
||||
def cost_per_token(**_kwargs):
|
||||
raise RuntimeError(f"upstream rejected request ({_CREDENTIAL_CANARY})")
|
||||
|
||||
monkeypatch.setattr(cost, "_pricing_warnings", WarnOnce(1024, "unpriceable models"))
|
||||
monkeypatch.setattr(cost, "_get_litellm_module", lambda: _LeakyLiteLLM())
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
capture = _Capture()
|
||||
previous_level = cost.logger.level
|
||||
cost.logger.setLevel(logging.DEBUG)
|
||||
cost.logger.addHandler(capture)
|
||||
try:
|
||||
assert CostTracker().estimate_cost("gpt-4o", 100, 10) is None
|
||||
finally:
|
||||
cost.logger.removeHandler(capture)
|
||||
cost.logger.setLevel(previous_level)
|
||||
|
||||
formatter = logging.Formatter()
|
||||
rendered = [formatter.format(r) for r in capture.records if r.levelno >= logging.DEBUG]
|
||||
assert any("Failed to get pricing for model 'gpt-4o'" in line for line in rendered)
|
||||
assert all("sk-canary" not in line for line in rendered), rendered
|
||||
|
||||
@@ -10,13 +10,15 @@ import logging
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.log_safety import WarnOnce
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def cost_tracker(monkeypatch: pytest.MonkeyPatch):
|
||||
import headroom.proxy.cost as cost_mod
|
||||
|
||||
# Reset the per-process dedup set so tests are order-independent.
|
||||
cost_mod._warned_pricing_models.clear()
|
||||
# A fresh dedup guard so tests are order-independent.
|
||||
monkeypatch.setattr(cost_mod, "_pricing_warnings", WarnOnce(1024, "unpriceable models"))
|
||||
|
||||
class _FakeLiteLLM:
|
||||
@staticmethod
|
||||
@@ -33,7 +35,7 @@ def test_pricing_failure_warns_once_per_model(cost_tracker, caplog):
|
||||
assert cost_tracker.estimate_cost("glm-5.2", 100, 50) is None
|
||||
|
||||
warnings = [
|
||||
r for r in caplog.records if "Failed to get pricing for model glm-5.2" in r.getMessage()
|
||||
r for r in caplog.records if "Failed to get pricing for model 'glm-5.2'" in r.getMessage()
|
||||
]
|
||||
assert len(warnings) == 1
|
||||
|
||||
@@ -46,14 +48,14 @@ def test_distinct_models_each_warn_once(cost_tracker, caplog):
|
||||
cost_tracker.estimate_cost("mystery-model", 10, 5)
|
||||
|
||||
msgs = [r.getMessage() for r in caplog.records if "Failed to get pricing" in r.getMessage()]
|
||||
assert sum("for model glm-5.2:" in m for m in msgs) == 1
|
||||
assert sum("for model mystery-model:" in m for m in msgs) == 1
|
||||
assert sum("for model 'glm-5.2':" in m for m in msgs) == 1
|
||||
assert sum("for model 'mystery-model':" in m for m in msgs) == 1
|
||||
|
||||
|
||||
def test_litellm_unavailable_warns_once_per_model(monkeypatch, caplog):
|
||||
import headroom.proxy.cost as cost_mod
|
||||
|
||||
cost_mod._warned_pricing_models.clear()
|
||||
monkeypatch.setattr(cost_mod, "_pricing_warnings", WarnOnce(1024, "unpriceable models"))
|
||||
monkeypatch.setattr(cost_mod, "_get_litellm_module", lambda: None)
|
||||
tracker = cost_mod.CostTracker()
|
||||
|
||||
|
||||
@@ -9,6 +9,7 @@ searched. Gated to gpt-5.4+ (older models 400 on the fields).
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import logging
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -55,6 +56,57 @@ def test_env_override_wins_then_falls_back(monkeypatch):
|
||||
assert _model_supports_openai_tool_search("gpt-5.4") is True
|
||||
|
||||
|
||||
class _Capture(logging.Handler):
|
||||
def __init__(self) -> None:
|
||||
super().__init__(logging.DEBUG)
|
||||
self.records: list[logging.LogRecord] = []
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
self.records.append(record)
|
||||
|
||||
|
||||
def test_malformed_override_warns_once_not_per_request(monkeypatch):
|
||||
"""A bad regex used to be ignored on every request without a word."""
|
||||
from headroom.proxy import helpers
|
||||
|
||||
monkeypatch.setattr(helpers, "_openai_tool_search_bad_pattern", None)
|
||||
capture = _Capture()
|
||||
proxy_logger = logging.getLogger("headroom.proxy")
|
||||
proxy_logger.addHandler(capture)
|
||||
try:
|
||||
monkeypatch.setenv("HEADROOM_OPENAI_TOOL_SEARCH_MODELS", r"^gpt-5\.4")
|
||||
assert _model_supports_openai_tool_search("gpt-5.4") is True
|
||||
assert not capture.records # a valid override is silent
|
||||
|
||||
monkeypatch.setenv("HEADROOM_OPENAI_TOOL_SEARCH_MODELS", "[unclosed")
|
||||
for _ in range(5):
|
||||
assert _model_supports_openai_tool_search("gpt-5.4") is True
|
||||
finally:
|
||||
proxy_logger.removeHandler(capture)
|
||||
|
||||
warnings = [r for r in capture.records if r.levelno == logging.WARNING]
|
||||
assert len(warnings) == 1
|
||||
assert "[unclosed" in warnings[0].getMessage()
|
||||
|
||||
|
||||
def test_valid_override_rearms_the_malformed_warning(monkeypatch):
|
||||
"""Bad A, valid B, bad A again: the second A is reported too."""
|
||||
from headroom.proxy import helpers
|
||||
|
||||
monkeypatch.setattr(helpers, "_openai_tool_search_bad_pattern", None)
|
||||
capture = _Capture()
|
||||
proxy_logger = logging.getLogger("headroom.proxy")
|
||||
proxy_logger.addHandler(capture)
|
||||
try:
|
||||
for pattern in ("[unclosed", r"^gpt-5\.4", "[unclosed"):
|
||||
monkeypatch.setenv("HEADROOM_OPENAI_TOOL_SEARCH_MODELS", pattern)
|
||||
assert _model_supports_openai_tool_search("gpt-5.4") is True
|
||||
finally:
|
||||
proxy_logger.removeHandler(capture)
|
||||
|
||||
assert [r.levelno for r in capture.records] == [logging.WARNING, logging.WARNING]
|
||||
|
||||
|
||||
# --- deferral behavior -------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import logging
|
||||
import os
|
||||
|
||||
import pytest
|
||||
@@ -39,3 +40,37 @@ def test_bad_env_falls_back_to_default(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("HEADROOM_COMPRESSION_TIMEOUT_SECONDS", "not-a-number")
|
||||
importlib.reload(helpers)
|
||||
assert helpers.COMPRESSION_TIMEOUT_SECONDS == 30.0
|
||||
|
||||
|
||||
class _Capture(logging.Handler):
|
||||
def __init__(self) -> None:
|
||||
super().__init__(logging.DEBUG)
|
||||
self.messages: list[str] = []
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
if record.levelno == logging.WARNING:
|
||||
self.messages.append(record.getMessage())
|
||||
|
||||
|
||||
def _reload_warnings() -> list[str]:
|
||||
capture = _Capture()
|
||||
proxy_logger = logging.getLogger("headroom.proxy")
|
||||
proxy_logger.addHandler(capture)
|
||||
try:
|
||||
importlib.reload(helpers)
|
||||
finally:
|
||||
proxy_logger.removeHandler(capture)
|
||||
return [m for m in capture.messages if "HEADROOM_COMPRESSION_TIMEOUT_SECONDS" in m]
|
||||
|
||||
|
||||
def test_bad_env_says_it_was_ignored(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A typo in the knob used to revert to 30s without a word."""
|
||||
monkeypatch.setenv("HEADROOM_COMPRESSION_TIMEOUT_SECONDS", "not-a-number")
|
||||
assert _reload_warnings() == [
|
||||
"Ignoring invalid HEADROOM_COMPRESSION_TIMEOUT_SECONDS='not-a-number'; using 30.0"
|
||||
]
|
||||
|
||||
|
||||
def test_valid_env_is_silent(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("HEADROOM_COMPRESSION_TIMEOUT_SECONDS", "75")
|
||||
assert _reload_warnings() == []
|
||||
|
||||
@@ -44,3 +44,41 @@ def test_install_all_warns_for_missing_requested_extension(caplog, monkeypatch)
|
||||
|
||||
assert installed == []
|
||||
assert "proxy extensions requested but not found: missing_ext" in caplog.text
|
||||
|
||||
|
||||
class _Capture(logging.Handler):
|
||||
def __init__(self) -> None:
|
||||
super().__init__(logging.DEBUG)
|
||||
self.records: list[logging.LogRecord] = []
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
self.records.append(record)
|
||||
|
||||
|
||||
def _discover_warnings(monkeypatch, entry_points: Any) -> list[str]:
|
||||
monkeypatch.setattr(extensions.importlib.metadata, "entry_points", entry_points)
|
||||
capture = _Capture()
|
||||
extensions.log.addHandler(capture)
|
||||
try:
|
||||
assert list(extensions.discover()) == []
|
||||
finally:
|
||||
extensions.log.removeHandler(capture)
|
||||
return [r.getMessage() for r in capture.records if r.levelno == logging.WARNING]
|
||||
|
||||
|
||||
def test_discover_warns_when_entry_point_enumeration_fails(monkeypatch) -> None:
|
||||
"""This drops every extension, so it must not hide at DEBUG."""
|
||||
|
||||
def broken(**_: Any) -> Any:
|
||||
raise RuntimeError("corrupt dist-info")
|
||||
|
||||
warnings = _discover_warnings(monkeypatch, broken)
|
||||
|
||||
assert len(warnings) == 1
|
||||
assert "no proxy extensions will be installed" in warnings[0]
|
||||
assert "RuntimeError" in warnings[0]
|
||||
assert "corrupt dist-info" not in warnings[0] # detail stays at DEBUG
|
||||
|
||||
|
||||
def test_discover_is_quiet_when_nothing_is_installed(monkeypatch) -> None:
|
||||
assert _discover_warnings(monkeypatch, lambda **_: []) == []
|
||||
|
||||
Reference in New Issue
Block a user