fix(proxy): say why proxy.log is missing and why a knob or extension was ignored (#4088)

## Description

Several failures an operator needs to see are swallowed silently or
logged only at DEBUG. This PR makes each of them visible with one
warning. It does not change defaults or control flow.

1. **Missing runtime log.** `_setup_file_logging` caught `OSError` with
`pass`. On a read-only workspace, a wrong owner, or a file where the
logs directory belongs, `proxy.log` simply never appeared and nothing
said why. It now logs `Headroom runtime log disabled: cannot write the
log file: <OSError>. Point HEADROOM_WORKSPACE_DIR at a writable
directory to restore proxy.log.` The OSError text names the path. The
symlink case just above it already warned this way.
2. **Six import-time numeric knobs.** Each one fell back to its default
on an unparseable value, without a word:
   - `HEADROOM_COMPRESSION_TIMEOUT_SECONDS`
   - `HEADROOM_COLD_START_FAST_PASS_TIMEOUT_SECONDS`
   - `HEADROOM_EAGER_PRELOAD_TIMEOUT_SECONDS`
   - `HEADROOM_COMPRESSION_CACHE_MAX_SESSIONS`
   - `HEADROOM_COMPRESSION_CACHE_TTL_SECONDS`
   - `HEADROOM_COMPRESSION_CACHE_MAX_ENTRIES`

Each now logs `Ignoring invalid NAME='value'; using DEFAULT` once, at
import. That matches the idiom in `hard_watchdog.py`.
3. **Extension discovery failure.** When entry-point enumeration fails,
every proxy extension is dropped, Enterprise plugins included. That was
logged at DEBUG. It is now a warning: `proxy extensions: entry-point
enumeration failed (<describe_exception>), so no proxy extensions will
be installed`. `describe_exception` gives the exception types and code
locations, never the message, and no traceback is logged at any level.
It runs at startup only: once in `install_all`, and once more for the
CLI banner.
4. **Pricing lookups in `cost.py`.** `_get_list_price`,
`_get_output_price`, `_get_cache_prices` and the cache-rate lookup in
the savings summary returned `None` on any exception. A pricing bug
therefore showed up as a $0 cost with no trace. All four, plus the
existing `estimate_cost` path, go through one helper,
`_warn_pricing_once(lookup, what, model, exc)`. Example line: `Failed to
get pricing for model 'gpt-4o': RuntimeError at
headroom/proxy/cost.py:... in estimate_cost`. It logs the lookup name,
`safe_id(model)` and `describe_exception(exc)` (types and code
locations, never the message), at WARNING only. No level carries the raw
exception or a traceback, because a pricing backend error can quote
provider responses or credentials. Dedupe is by `(lookup, model)`
through the shared `WarnOnce(1024, "unpriceable models")`. Each pair
warns at most once an hour (the `WarnOnce` window). Past 1024 models,
one overflow notice is logged, and a key is not used up while WARNING is
disabled.
5. **Malformed `HEADROOM_OPENAI_TOOL_SEARCH_MODELS` regex.** The
override was ignored on every request with no log. It now warns once per
bad pattern, and still falls back to the version gate as before. A valid
pattern re-arms the warning, so bad A, then valid B, then bad A again
warns twice.

None of the new lines logs header values, request content or
credentials. They log only knob names, their raw values (operator-set
env), model names, regex patterns, exception type names, the `re.error`
text, and `OSError` text naming a local path. No exception message or
traceback from pricing or extension discovery reaches any log level.

Refs #4136

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/helpers.py`:
- add `_warn_invalid_env_number(name, default)` and call it from the six
import-time `except ValueError` branches;
  - log the `OSError` in `_setup_file_logging`;
- add a once-per-pattern warning for a malformed
`HEADROOM_OPENAI_TOOL_SEARCH_MODELS`.
- `headroom/proxy/extensions.py`: entry-point enumeration failure logs
at warning with `describe_exception(exc)` instead of at debug.
- `headroom/proxy/cost.py`:
- the four silent pricing `except Exception` branches call
`_warn_pricing_once`;
- `_warn_pricing_once(lookup, what, model, exc)` logs `safe_id(model)`
plus `describe_exception(exc)` at WARNING, with nothing at DEBUG. It
dedupes through `log_safety.WarnOnce`.
- `tests/test_cost_pricing_warning_dedup.py`: the tests reset with a
fresh `WarnOnce`, and match the quoted model id (`for model
'glm-5.2':`).
- Regression tests. Each one attaches a handler to the module logger
directly, because caplog misses `headroom.*` records after proxy
startup. Each positive test fails on main and is paired with a negative
case that passes on both.
  - `tests/test_cache_breakpoint_diagnostics.py`:
- `test_runtime_log_says_why_it_is_missing` makes the logs directory a
regular file. It asserts one warning that names the path and
`HEADROOM_WORKSPACE_DIR`.
    - Negative: `test_runtime_log_stays_quiet_when_writable`.
  - `tests/test_openai_tool_search_deferral.py`:
- `test_malformed_override_warns_once_not_per_request` calls the gate 5
times with a bad pattern. It asserts exactly one warning and that the
version-gate result is unchanged.
    - Negative: a valid override is silent (same test).
- `test_valid_override_rearms_the_malformed_warning`: bad, valid, bad
gives two warnings.
  - `tests/test_proxy/test_compression_timeout_config.py`:
- `test_bad_env_says_it_was_ignored` reloads `helpers` with
`HEADROOM_COMPRESSION_TIMEOUT_SECONDS=not-a-number` and asserts the
exact warning.
- Negative: `test_valid_env_is_silent`. The existing fixture reloads
`helpers` afterwards.
  - `tests/test_proxy_extensions.py`:
- `test_discover_warns_when_entry_point_enumeration_fails` makes
`importlib.metadata.entry_points` raise. It asserts one WARNING that
says no extensions will be installed and names `RuntimeError` but not
the exception text.
    - Negative: `test_discover_is_quiet_when_nothing_is_installed`.
  - `tests/test_cost_cache_prices.py`:
- `test_cache_rate_failure_warns_once_per_model` makes `resolve_rates`
raise. It asserts two `_get_cache_prices` calls give one warning naming
the model and `RuntimeError`, without the exception text.
    - Negative: a resolvable model is silent (same test).
- `test_pricing_error_never_logs_its_text_at_any_level` runs the real
`CostTracker.estimate_cost('gpt-4o', 100, 10)` with a pricing error
whose text holds an Authorization credential canary, with
`headroom.proxy` at DEBUG. It formats every captured record with
`logging.Formatter` (which renders `exc_info`), asserts the canary is
absent and the result is `None`, and asserts the warning is present. On
the previous commit it fails: the formatted DEBUG traceback contains the
canary.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_cost_cache_prices.py -k never_logs   # previous commit 3202058c (test adapted to its old dedupe set)
E     RuntimeError: upstream rejected request (Authorization: Bearer sk-canary-7f3a9c)']
======================= 1 failed, 25 deselected in 3.48s =======================
$ pytest -q -p no:cacheprovider tests/test_pr208_changes.py tests/test_cost_budget_basis.py tests/test_cost_tracker_counterfactual.py tests/test_cost_tracker_totals.py tests/test_tool_search_client_deferral.py tests/test_cache_breakpoint_diagnostics.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_byte_faithful_forwarding.py tests/test_memory_tool_mode.py tests/test_codex_ws_compression_scheduler.py tests/test_proxy_savings_history.py tests/test_compression_cache_registry.py tests/test_proxy_extension_ordering.py tests/test_proxy_eager_preload_bind.py tests/test_perf_per_port_logs.py tests/test_cost_cache_prices.py tests/test_openai_tool_search_deferral.py tests/test_proxy_extensions.py tests/test_direct_chat_ccr_resolution.py tests/test_org_scale_limits.py tests/test_savings_ledger.py tests/test_cost_pricing_warning_dedup.py tests/test_proxy/test_compression_timeout_config.py tests/test_log_safety.py   # 64ff3f03
================== 517 passed, 1 skipped in 85.70s (0:01:25) ===================
$ pytest -q (same pricing/extension/regex/log/knob/log_safety files) after merging log-safety-helpers @ 0ce74a63   # 4ff57af4
======================== 139 passed, 1 skipped in 4.23s ========================
$ uvx ruff@0.16.8 check / format --check headroom/proxy/cost.py headroom/proxy/extensions.py tests/test_cost_cache_prices.py tests/test_cost_pricing_warning_dedup.py tests/test_proxy_extensions.py
All checks passed! / 5 files already formatted
$ mypy headroom/proxy/cost.py headroom/proxy/extensions.py --ignore-missing-imports
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, branch checkout at 4ff57af4
(log-safety-helpers merged at 0ce74a63), using the repo .venv. A temp
HOME, workspace and CODEX_HOME were used each time; the real HOME was
not touched.
- Exact command / steps:
1. Ran the shared smoke harness, which starts a real proxy from this
checkout against a local fake upstream and sends one Anthropic and one
OpenAI request.
2. Ran the real `CostTracker.estimate_cost('gpt-4o', 100, 10)` twice,
with every logger at DEBUG and a LiteLLM stand-in whose `cost_per_token`
raises an error quoting `Authorization: Bearer sk-canary-7f3a9c`.
3. The earlier forced-failure run for the knob, log-directory and regex
warnings (d83e7d5c, below) is unchanged by this commit.
- Observed result:
```text
branch: cleanup/proxy-core-a-surface-ignored-config @ 4ff57af4
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines (<tmp>):
(smoke dir: <tmp>)
```
```text
WARNING headroom.proxy: Failed to get pricing for model 'gpt-4o': RuntimeError at <string>:8 in cost_per_token <- headroom/proxy/cost.py:999 in estimate_cost
result: None None
```
- One payload-safe WARNING appears, with the exception type and code
location, and both calls return `None`.
- The canary appears in no output line at any level: `grep -c canary`
over the whole run, LiteLLM's own DEBUG chatter included, gives 0.
  - Earlier forced-failure run:
```text
branch: cleanup/proxy-core-a-surface-ignored-config @ d83e7d5c
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines (<tmp>):
(smoke dir: <tmp>)
```
The harness also printed two pre-existing WARNING lines, which I left
out above: offline-mode air-gap active, and the Kompress model prefetch
refused offline. Neither comes from this PR. Output of the direct
command:
```text
WARNING headroom.proxy: Ignoring invalid HEADROOM_COMPRESSION_TIMEOUT_SECONDS='abc'; using 30.0
WARNING headroom.proxy: Headroom runtime log disabled: cannot write the log file: [Errno 17] File exists: '<tmp>/ws/logs'. Point HEADROOM_WORKSPACE_DIR at a writable directory to restore proxy.log.
WARNING headroom.proxy: Ignoring invalid HEADROOM_OPENAI_TOOL_SEARCH_MODELS regex '[bad' (unterminated character set at position 0); using the gpt-5.4+ version gate
gate: True True
```
- The proxy serves both providers, and the normal path adds no new
warnings.
- Each forced failure gives exactly one warning, even though the regex
gate ran twice. The fallbacks are unchanged: 30.0, and the version gate
is True for gpt-5.4.
- Not tested: A real read-only filesystem, a real broken
`importlib.metadata` and a live pricing-catalog failure (the stand-in
above raises from the same call). Each goes through the same `except`
branch that the unit tests force.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; log output only.
- Stable/default behavior changed: No. Every fallback value and code
path is the same; only new WARNING lines appear, when something was
already failing.
- Kill switch / disable path: N/A. A pricing warning fires at most once
an hour per (lookup, model), through `WarnOnce`; the regex warning fires
once per bad pattern.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of a maintainer-requested codebase cleanup (dead code, duplication,
debuggability), split into small PRs for review. It is part of
maintainer-requested cleanup #4136.

Depends on #4138; its two files show in this diff until it merges. #4137
also edits `_setup_file_logging`, so whichever of the two merges second
rebases.

Docs are unchanged because no knob's meaning changed. The doc impacts
are recorded in the cleanup docs ledger for the follow-up docs PR. The
six import-time knob warnings have no test: testing them would need
`importlib.reload(helpers)`, which would reset the module globals other
tests depend on.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
2026-10-12 11:51:06 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent db72e6717f
commit 72302a3998
9 changed files with 354 additions and 28 deletions
+30 -15
View File
@@ -16,6 +16,7 @@ from datetime import datetime, timedelta
from typing import TYPE_CHECKING, Any, NamedTuple
from headroom.cache_economics import CACHE_ECONOMICS as _CACHE_ECONOMICS
from headroom.log_safety import WarnOnce, describe_exception, safe_id
from headroom.proxy.budget_basis_policy import (
BUDGET_BASIS_BLOCK,
BUDGET_BASIS_IGNORE,
@@ -57,17 +58,26 @@ logger = logging.getLogger("headroom.proxy")
# Pricing-lookup warnings are emitted on the per-request cost path, so an
# unresolvable model (a custom / OpenAI-compatible name LiteLLM can't price,
# e.g. glm-5.2) floods proxy.log with an identical WARNING every single request
# (#2504). Track which models have already been warned so each fires once per
# process; the set is tiny and bounded by the number of distinct models seen.
_warned_pricing_models: set[str] = set()
# (#2504). Each (lookup, model) pair warns once; model names come from clients,
# so WarnOnce also caps how many distinct models can warn.
_pricing_warnings = WarnOnce(1024, "unpriceable models")
def _warn_pricing_once(model: str, message: str) -> None:
"""Emit ``message`` at WARNING only the first time ``model`` fails pricing."""
if model in _warned_pricing_models:
def _warn_pricing_once(
lookup: str, what: str, model: str, exc: BaseException | None = None
) -> None:
"""Warn once per ``(lookup, model)`` that pricing failed, payload-safe.
A pricing backend's exception can quote provider responses or credentials,
so only :func:`describe_exception` (types and code locations) is logged, at
every level.
"""
if not _pricing_warnings.first((lookup, model), logger):
return
_warned_pricing_models.add(model)
logger.warning(message)
if exc is None:
logger.warning("%s for model %s", what, safe_id(model))
else:
logger.warning("%s for model %s: %s", what, safe_id(model), describe_exception(exc))
# A route whose responses never carry a usage breakdown hits the estimated-basis
@@ -275,7 +285,10 @@ def build_prefix_cache_stats(
from headroom.pricing.counterfactual import resolve_rates
cache_prices = resolve_rates(model_name)
except Exception:
except Exception as e:
_warn_pricing_once(
"cache_rates", "Failed to resolve cache rates", model_name, e
)
cache_prices = None
best_tokens = tokens_sent
@@ -972,8 +985,7 @@ class CostTracker:
litellm = _get_litellm_module()
if litellm is None:
_warn_pricing_once(
f"__litellm_unavailable__:{model}",
f"LiteLLM not available - cannot calculate costs for model {model}",
"litellm_unavailable", "LiteLLM not available - cannot calculate costs", model
)
return None
@@ -996,7 +1008,7 @@ class CostTracker:
return float(total_cost) if total_cost > 0 else None
except Exception as e:
_warn_pricing_once(model, f"Failed to get pricing for model {model}: {e}")
_warn_pricing_once("estimate", "Failed to get pricing", model, e)
return None
def _prune_old_costs(self):
@@ -1369,7 +1381,8 @@ class CostTracker:
info = litellm.model_cost.get(resolved, {})
cost_per_token = info.get("input_cost_per_token")
return cost_per_token * 1_000_000 if cost_per_token else None
except Exception:
except Exception as e:
_warn_pricing_once("list_price", "Failed to get list price", model, e)
return None
def _get_output_price(self, model: str, *, long_context: bool = False) -> float | None:
@@ -1390,7 +1403,8 @@ class CostTracker:
if long_context:
return info.get("output_cost_per_token_above_200k_tokens") or base or None
return base or None
except Exception:
except Exception as e:
_warn_pricing_once("output_price", "Failed to get output price", model, e)
return None
def _write_cost_usd(self, model: str, w5m_price: float, w1h_price: float) -> float:
@@ -1444,7 +1458,8 @@ class CostTracker:
from headroom.pricing.counterfactual import resolve_rates
rates = resolve_rates(model, long_context=long_context, for_billing=for_billing)
except Exception:
except Exception as e:
_warn_pricing_once("cache_rates", "Failed to resolve cache rates", model, e)
return None
if rates is None or not rates.uncached:
return None
+7 -1
View File
@@ -103,6 +103,8 @@ import sys
from collections.abc import Callable, Iterable, Iterator
from typing import Any
from headroom.log_safety import describe_exception
log = logging.getLogger(__name__)
ENTRY_POINT_GROUP = "headroom.proxy_extension"
@@ -123,7 +125,11 @@ def discover() -> Iterator[tuple[str, ProxyExtension]]:
try:
entries = importlib.metadata.entry_points(group=ENTRY_POINT_GROUP)
except Exception as exc: # noqa: BLE001 — importlib.metadata can raise varied types
log.debug("proxy extensions: entry-point enumeration failed: %s", exc)
log.warning(
"proxy extensions: entry-point enumeration failed (%s), so no proxy "
"extensions will be installed",
describe_exception(exc),
)
return
for entry in entries:
try:
+49 -6
View File
@@ -1416,6 +1416,12 @@ def parse_sse_events_from_byte_buffer(
# Maximum message array length (prevents DoS from deeply nested payloads)
MAX_MESSAGE_ARRAY_LENGTH = 10000
def _warn_invalid_env_number(name: str, default: float) -> None:
"""Say which numeric knob was ignored, so a typo does not silently revert it."""
logger.warning("Ignoring invalid %s=%r; using %s", name, os.environ.get(name), default)
# Compression pipeline timeout in seconds. Override via the
# HEADROOM_COMPRESSION_TIMEOUT_SECONDS env var for slow CPUs or long Claude Code
# conversations (GH #946). Falls back to 30 on an unparseable value.
@@ -1425,6 +1431,7 @@ try:
)
except ValueError:
COMPRESSION_TIMEOUT_SECONDS = 30.0
_warn_invalid_env_number("HEADROOM_COMPRESSION_TIMEOUT_SECONDS", COMPRESSION_TIMEOUT_SECONDS)
# Cold-start fast-pass timeout in seconds. When background compression defers
# a cold-start-large request, the handler still runs the pipeline synchronously
@@ -1439,6 +1446,9 @@ try:
)
except ValueError:
COLD_START_FAST_PASS_TIMEOUT_SECONDS = 10.0
_warn_invalid_env_number(
"HEADROOM_COLD_START_FAST_PASS_TIMEOUT_SECONDS", COLD_START_FAST_PASS_TIMEOUT_SECONDS
)
# Eager startup preload timeout in seconds. The preload (compressor/parser models,
# cache-only, allow_download=False) runs off the event loop during startup; this
@@ -1451,6 +1461,9 @@ try:
)
except ValueError:
EAGER_PRELOAD_TIMEOUT_SECONDS = 120.0
_warn_invalid_env_number(
"HEADROOM_EAGER_PRELOAD_TIMEOUT_SECONDS", EAGER_PRELOAD_TIMEOUT_SECONDS
)
# Maximum compression cache sessions (prevents unbounded memory growth).
# Overridable via HEADROOM_COMPRESSION_CACHE_MAX_SESSIONS for gateway
@@ -1462,6 +1475,9 @@ try:
)
except ValueError:
MAX_COMPRESSION_CACHE_SESSIONS = 500
_warn_invalid_env_number(
"HEADROOM_COMPRESSION_CACHE_MAX_SESSIONS", MAX_COMPRESSION_CACHE_SESSIONS
)
# Idle TTL for per-session compression caches. Eviction is bust-free only
# once the provider's own prompt cache has lapsed, so this must exceed the
@@ -1486,6 +1502,9 @@ try:
COMPRESSION_CACHE_TTL_SECONDS = max(600.0, _ttl_env)
except ValueError:
COMPRESSION_CACHE_TTL_SECONDS = 3900.0
_warn_invalid_env_number(
"HEADROOM_COMPRESSION_CACHE_TTL_SECONDS", COMPRESSION_CACHE_TTL_SECONDS
)
# Entries per session compression cache. 10k covers a single conversation with
# ~2x headroom even at a 1M-token context (a compressible tool_result is at
@@ -1500,6 +1519,9 @@ try:
)
except ValueError:
COMPRESSION_CACHE_MAX_ENTRIES = 10000
_warn_invalid_env_number(
"HEADROOM_COMPRESSION_CACHE_MAX_ENTRIES", COMPRESSION_CACHE_MAX_ENTRIES
)
# ---------------------------------------------------------------------------
@@ -1933,9 +1955,14 @@ def _setup_file_logging(
headroom_logger.removeHandler(stale)
stale.close()
headroom_logger.addHandler(handler)
except OSError:
# Non-fatal: can't write logs (read-only fs, permissions, etc.)
pass
except OSError as exc:
# Non-fatal: can't write logs (read-only fs, permissions, etc.). Say so,
# or the missing proxy.log is the only clue. The OSError names the path.
logger.warning(
"Headroom runtime log disabled: cannot write the log file: %s. "
"Point HEADROOM_WORKSPACE_DIR at a writable directory to restore proxy.log.",
exc,
)
def is_anthropic_auth(headers: dict[str, str]) -> bool:
@@ -4561,6 +4588,9 @@ _OPENAI_TOOL_SEARCH_UNSUPPORTED_CLIENTS = frozenset({"codex", "opencode"})
# HEADROOM_OPENAI_TOOL_SEARCH_MODELS (matched against the model name) so new
# model families can be enabled without a code edit + release.
_OPENAI_TOOL_SEARCH_MIN_VERSION = (5, 4)
# Last malformed override pattern warned about, so a bad regex is reported once
# instead of on every request.
_openai_tool_search_bad_pattern: str | None = None
def _model_supports_openai_tool_search(model: str | None) -> bool:
@@ -4571,14 +4601,27 @@ def _model_supports_openai_tool_search(model: str | None) -> bool:
when set; a malformed pattern falls back to the version gate rather than
crashing.
"""
global _openai_tool_search_bad_pattern
if not model:
return False
override = os.environ.get("HEADROOM_OPENAI_TOOL_SEARCH_MODELS", "").strip()
if override:
try:
return re.search(override, model) is not None
except re.error:
pass # malformed override → fall back to the version gate
matched = re.search(override, model) is not None
except re.error as exc:
# Malformed override → fall back to the version gate.
if override != _openai_tool_search_bad_pattern and logger.isEnabledFor(logging.WARNING):
_openai_tool_search_bad_pattern = override
logger.warning(
"Ignoring invalid HEADROOM_OPENAI_TOOL_SEARCH_MODELS regex %r (%s); "
"using the gpt-5.4+ version gate",
override,
exc,
)
else:
# A valid pattern re-arms the warning for the next bad one.
_openai_tool_search_bad_pattern = None
return matched
match = re.match(r"gpt-(\d+)(?:\.(\d+))?", model.strip().lower())
if not match:
return False
@@ -403,6 +403,63 @@ def test_runtime_log_refuses_a_symlinked_path(tmp_path, monkeypatch) -> None:
assert stat.S_IMODE(elsewhere.stat().st_mode) == 0o666, "the symlink target was chmodded"
class _ListHandler(logging.Handler):
def __init__(self) -> None:
super().__init__(logging.DEBUG)
self.records: list[logging.LogRecord] = []
def emit(self, record: logging.LogRecord) -> None:
self.records.append(record)
def _runtime_log_warnings(port: int) -> list[str]:
"""Run ``_setup_file_logging`` and return the 'runtime log disabled' warnings.
The handler sits on ``headroom.proxy`` directly: caplog does not see
``headroom.*`` records once a proxy app has configured logging.
"""
from headroom.proxy.helpers import _setup_file_logging
proxy_logger = logging.getLogger("headroom.proxy")
headroom_logger = logging.getLogger("headroom")
before = list(headroom_logger.handlers)
capture = _ListHandler()
proxy_logger.addHandler(capture)
try:
_setup_file_logging(port)
finally:
proxy_logger.removeHandler(capture)
for handler in list(headroom_logger.handlers):
if handler not in before:
headroom_logger.removeHandler(handler)
handler.close()
return [
r.getMessage()
for r in capture.records
if r.levelno == logging.WARNING and "runtime log disabled" in r.getMessage()
]
def test_runtime_log_says_why_it_is_missing(tmp_path, monkeypatch) -> None:
"""An unwritable log directory used to disable proxy.log without a word."""
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
# A regular file where the logs directory belongs: mkdir raises OSError.
_paths.log_dir().write_text("", encoding="utf-8")
warnings = _runtime_log_warnings(18808)
assert len(warnings) == 1
assert str(_paths.log_dir()) in warnings[0]
assert "HEADROOM_WORKSPACE_DIR" in warnings[0]
def test_runtime_log_stays_quiet_when_writable(tmp_path, monkeypatch) -> None:
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
assert _runtime_log_warnings(18809) == []
assert _paths.proxy_log_path(18809).exists()
@pytest.mark.skipif(
os.name != "posix",
reason="creating a symlink needs elevation on Windows, and O_NOFOLLOW does not exist there",
+78
View File
@@ -9,10 +9,12 @@ and inferred writes cannot double-charge the uncached input bucket.
from __future__ import annotations
import logging
from types import SimpleNamespace
import pytest
from headroom.log_safety import WarnOnce
from headroom.pricing.counterfactual import resolve_rates
from headroom.proxy.cost import CostTracker
@@ -358,3 +360,79 @@ class TestGetCachePrices:
rates = resolve_rates("m", provider="anthropic")
assert rates.read == rates.write_5m == 1e-6
assert rates.read_is_catalog and rates.write_is_catalog
class _Capture(logging.Handler):
def __init__(self) -> None:
super().__init__(logging.DEBUG)
self.records: list[logging.LogRecord] = []
def emit(self, record: logging.LogRecord) -> None:
self.records.append(record)
def test_cache_rate_failure_warns_once_per_model(monkeypatch: pytest.MonkeyPatch) -> None:
"""A raising rate resolver used to read as 'unpriced' ($0) with no trace."""
from headroom.proxy import cost
monkeypatch.setattr(cost, "_pricing_warnings", WarnOnce(1024, "unpriceable models"))
_patch_litellm(
monkeypatch,
{"m": {"input_cost_per_token": 1e-6, "litellm_provider": "anthropic"}},
)
capture = _Capture()
cost.logger.addHandler(capture)
try:
assert CostTracker()._get_cache_prices("m") is not None
assert not capture.records # a resolvable model is silent
def broken(*_args, **_kwargs):
raise RuntimeError("pricing table corrupt")
monkeypatch.setattr("headroom.pricing.counterfactual.resolve_rates", broken)
assert CostTracker()._get_cache_prices("m") is None
assert CostTracker()._get_cache_prices("m") is None
finally:
cost.logger.removeHandler(capture)
warnings = [r.getMessage() for r in capture.records if r.levelno == logging.WARNING]
assert len(warnings) == 1
assert "for model 'm'" in warnings[0] and "RuntimeError" in warnings[0]
assert "pricing table corrupt" not in warnings[0]
_CREDENTIAL_CANARY = "Authorization: Bearer sk-canary-7f3a9c"
def test_pricing_error_never_logs_its_text_at_any_level(monkeypatch: pytest.MonkeyPatch) -> None:
"""A pricing backend error quoting a credential must not reach any log record.
Formats every WARNING and DEBUG record with a real Formatter, which renders
exc_info too, so a traceback carrying the text would be caught.
"""
from headroom.proxy import cost
class _LeakyLiteLLM:
model_cost: dict = {}
@staticmethod
def cost_per_token(**_kwargs):
raise RuntimeError(f"upstream rejected request ({_CREDENTIAL_CANARY})")
monkeypatch.setattr(cost, "_pricing_warnings", WarnOnce(1024, "unpriceable models"))
monkeypatch.setattr(cost, "_get_litellm_module", lambda: _LeakyLiteLLM())
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
capture = _Capture()
previous_level = cost.logger.level
cost.logger.setLevel(logging.DEBUG)
cost.logger.addHandler(capture)
try:
assert CostTracker().estimate_cost("gpt-4o", 100, 10) is None
finally:
cost.logger.removeHandler(capture)
cost.logger.setLevel(previous_level)
formatter = logging.Formatter()
rendered = [formatter.format(r) for r in capture.records if r.levelno >= logging.DEBUG]
assert any("Failed to get pricing for model 'gpt-4o'" in line for line in rendered)
assert all("sk-canary" not in line for line in rendered), rendered
+8 -6
View File
@@ -10,13 +10,15 @@ import logging
import pytest
from headroom.log_safety import WarnOnce
@pytest.fixture
def cost_tracker(monkeypatch: pytest.MonkeyPatch):
import headroom.proxy.cost as cost_mod
# Reset the per-process dedup set so tests are order-independent.
cost_mod._warned_pricing_models.clear()
# A fresh dedup guard so tests are order-independent.
monkeypatch.setattr(cost_mod, "_pricing_warnings", WarnOnce(1024, "unpriceable models"))
class _FakeLiteLLM:
@staticmethod
@@ -33,7 +35,7 @@ def test_pricing_failure_warns_once_per_model(cost_tracker, caplog):
assert cost_tracker.estimate_cost("glm-5.2", 100, 50) is None
warnings = [
r for r in caplog.records if "Failed to get pricing for model glm-5.2" in r.getMessage()
r for r in caplog.records if "Failed to get pricing for model 'glm-5.2'" in r.getMessage()
]
assert len(warnings) == 1
@@ -46,14 +48,14 @@ def test_distinct_models_each_warn_once(cost_tracker, caplog):
cost_tracker.estimate_cost("mystery-model", 10, 5)
msgs = [r.getMessage() for r in caplog.records if "Failed to get pricing" in r.getMessage()]
assert sum("for model glm-5.2:" in m for m in msgs) == 1
assert sum("for model mystery-model:" in m for m in msgs) == 1
assert sum("for model 'glm-5.2':" in m for m in msgs) == 1
assert sum("for model 'mystery-model':" in m for m in msgs) == 1
def test_litellm_unavailable_warns_once_per_model(monkeypatch, caplog):
import headroom.proxy.cost as cost_mod
cost_mod._warned_pricing_models.clear()
monkeypatch.setattr(cost_mod, "_pricing_warnings", WarnOnce(1024, "unpriceable models"))
monkeypatch.setattr(cost_mod, "_get_litellm_module", lambda: None)
tracker = cost_mod.CostTracker()
+52
View File
@@ -9,6 +9,7 @@ searched. Gated to gpt-5.4+ (older models 400 on the fields).
from __future__ import annotations
import copy
import logging
import pytest
@@ -55,6 +56,57 @@ def test_env_override_wins_then_falls_back(monkeypatch):
assert _model_supports_openai_tool_search("gpt-5.4") is True
class _Capture(logging.Handler):
def __init__(self) -> None:
super().__init__(logging.DEBUG)
self.records: list[logging.LogRecord] = []
def emit(self, record: logging.LogRecord) -> None:
self.records.append(record)
def test_malformed_override_warns_once_not_per_request(monkeypatch):
"""A bad regex used to be ignored on every request without a word."""
from headroom.proxy import helpers
monkeypatch.setattr(helpers, "_openai_tool_search_bad_pattern", None)
capture = _Capture()
proxy_logger = logging.getLogger("headroom.proxy")
proxy_logger.addHandler(capture)
try:
monkeypatch.setenv("HEADROOM_OPENAI_TOOL_SEARCH_MODELS", r"^gpt-5\.4")
assert _model_supports_openai_tool_search("gpt-5.4") is True
assert not capture.records # a valid override is silent
monkeypatch.setenv("HEADROOM_OPENAI_TOOL_SEARCH_MODELS", "[unclosed")
for _ in range(5):
assert _model_supports_openai_tool_search("gpt-5.4") is True
finally:
proxy_logger.removeHandler(capture)
warnings = [r for r in capture.records if r.levelno == logging.WARNING]
assert len(warnings) == 1
assert "[unclosed" in warnings[0].getMessage()
def test_valid_override_rearms_the_malformed_warning(monkeypatch):
"""Bad A, valid B, bad A again: the second A is reported too."""
from headroom.proxy import helpers
monkeypatch.setattr(helpers, "_openai_tool_search_bad_pattern", None)
capture = _Capture()
proxy_logger = logging.getLogger("headroom.proxy")
proxy_logger.addHandler(capture)
try:
for pattern in ("[unclosed", r"^gpt-5\.4", "[unclosed"):
monkeypatch.setenv("HEADROOM_OPENAI_TOOL_SEARCH_MODELS", pattern)
assert _model_supports_openai_tool_search("gpt-5.4") is True
finally:
proxy_logger.removeHandler(capture)
assert [r.levelno for r in capture.records] == [logging.WARNING, logging.WARNING]
# --- deferral behavior -------------------------------------------------------
@@ -1,6 +1,7 @@
from __future__ import annotations
import importlib
import logging
import os
import pytest
@@ -39,3 +40,37 @@ def test_bad_env_falls_back_to_default(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("HEADROOM_COMPRESSION_TIMEOUT_SECONDS", "not-a-number")
importlib.reload(helpers)
assert helpers.COMPRESSION_TIMEOUT_SECONDS == 30.0
class _Capture(logging.Handler):
def __init__(self) -> None:
super().__init__(logging.DEBUG)
self.messages: list[str] = []
def emit(self, record: logging.LogRecord) -> None:
if record.levelno == logging.WARNING:
self.messages.append(record.getMessage())
def _reload_warnings() -> list[str]:
capture = _Capture()
proxy_logger = logging.getLogger("headroom.proxy")
proxy_logger.addHandler(capture)
try:
importlib.reload(helpers)
finally:
proxy_logger.removeHandler(capture)
return [m for m in capture.messages if "HEADROOM_COMPRESSION_TIMEOUT_SECONDS" in m]
def test_bad_env_says_it_was_ignored(monkeypatch: pytest.MonkeyPatch) -> None:
"""A typo in the knob used to revert to 30s without a word."""
monkeypatch.setenv("HEADROOM_COMPRESSION_TIMEOUT_SECONDS", "not-a-number")
assert _reload_warnings() == [
"Ignoring invalid HEADROOM_COMPRESSION_TIMEOUT_SECONDS='not-a-number'; using 30.0"
]
def test_valid_env_is_silent(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("HEADROOM_COMPRESSION_TIMEOUT_SECONDS", "75")
assert _reload_warnings() == []
+38
View File
@@ -44,3 +44,41 @@ def test_install_all_warns_for_missing_requested_extension(caplog, monkeypatch)
assert installed == []
assert "proxy extensions requested but not found: missing_ext" in caplog.text
class _Capture(logging.Handler):
def __init__(self) -> None:
super().__init__(logging.DEBUG)
self.records: list[logging.LogRecord] = []
def emit(self, record: logging.LogRecord) -> None:
self.records.append(record)
def _discover_warnings(monkeypatch, entry_points: Any) -> list[str]:
monkeypatch.setattr(extensions.importlib.metadata, "entry_points", entry_points)
capture = _Capture()
extensions.log.addHandler(capture)
try:
assert list(extensions.discover()) == []
finally:
extensions.log.removeHandler(capture)
return [r.getMessage() for r in capture.records if r.levelno == logging.WARNING]
def test_discover_warns_when_entry_point_enumeration_fails(monkeypatch) -> None:
"""This drops every extension, so it must not hide at DEBUG."""
def broken(**_: Any) -> Any:
raise RuntimeError("corrupt dist-info")
warnings = _discover_warnings(monkeypatch, broken)
assert len(warnings) == 1
assert "no proxy extensions will be installed" in warnings[0]
assert "RuntimeError" in warnings[0]
assert "corrupt dist-info" not in warnings[0] # detail stays at DEBUG
def test_discover_is_quiet_when_nothing_is_installed(monkeypatch) -> None:
assert _discover_warnings(monkeypatch, lambda **_: []) == []