mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): log the savings and metrics failures that currently vanish (#4084)
## Description Several savings and metrics paths swallow exceptions and log nothing, so when one fails, a customer-facing number can come out wrong with no record of why. This PR adds a log line on each of those paths. The numbers and the control flow are unchanged. Refs #4136 | Where | Before | Now | |---|---|---| | `tool_schema_savings_policy.reconcile_deferred_tokens` | A failure left the tool-search deferral credit as booked, so `/stats` over-reported. Nothing was logged. | Warns once per exception type, giving the booked token count and the fact that `/stats` is over-reported. Every occurrence is also logged at debug. | | `outcome.py` output-savings record/estimate | Booked 0 silently. | Debug, with the request id. | | `server.py` `/stats` `output_reduction` | The section went missing silently. | Debug. | | `server.py` `create_app` saved-settings apply | `settings_store.load()` already warns about an unreadable or corrupt file. Any other error while applying it was swallowed. | Warns with the settings file path and says to fix or remove that file and restart. Startup continues. | | `server.py` security gate, audit record | Debug only. | Warns with the method and path, and says the request is missing from the `headroom.audit` trail. | | `savings_tracker` pricing for an unpriced model | Raised `RuntimeError` only to reach the fallback, and logged nothing. | Returns the fallback directly and logs once per model at debug. The `except` blocks now see only unexpected errors, and log them at debug. | | `savings_tracker._resolve_litellm_model` probes | Two `except Exception: pass` blocks. | Debug. litellm raises a bare `Exception` for a model it does not know, so the catch cannot be narrower. | **No content at any level.** A tokenizer error can quote the client's tool schema, and a settings or pricing error can quote a credential. So every line this PR adds describes a failure with `describe_exception(exc)` from `headroom/log_safety.py`: the exception types and code locations of the cause chain, never the message and never a traceback. Model ids go through `safe_id`, and audit paths are capped the same way. The full text appears only when the operator sets `HEADROOM_DEBUG_DUMP=full`. **Bounded dedupe.** The reconcile guard (keyed by exception type) and the unpriced-model guard (keyed by model) use the shared `WarnOnce`. Each key is logged once, at most a fixed number of keys per hour, followed by one overflow notice. The unpriced-model guard is consulted only when DEBUG is enabled, so a model is not used up while its line is suppressed. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/tool_schema_savings_policy.py`: `_log_reconcile_failure` uses `WarnOnce(32, ...)` and `describe_exception`. - `headroom/proxy/outcome.py`, `headroom/proxy/server.py`, `headroom/proxy/savings_tracker.py`: add the lines in the table above, all through `describe_exception` and `safe_id`. `_log_unpriced_model_once` uses `WarnOnce`. - Tests: - `tests/test_tool_schema_savings_policy.py`: - a tokenizer error quoting a private-schema canary; - every captured DEBUG and WARNING record is rendered with `logging.Formatter()`, which includes `exc_info`, and none contains the canary; - the booked credit stays unchanged; - one warning per exception type, a debug line per failure. - `tests/test_savings_pricing_fallback_logging.py`: - a pricing error and a litellm probe error that quote an API-key canary, with the canary absent from every formatted record; - an unpriced model logs once; - nothing is used up while DEBUG is off. - `tests/test_saved_settings_apply_logging.py`: a settings-apply error that quotes an API-key canary through a real `create_app`. One warning names the file and the exception type, and no formatted record contains the canary. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # New regressions against the previous commit's product files (c5850f02), in a scratch worktree: FAILED tests/test_tool_schema_savings_policy.py::test_reconcile_failure_keeps_private_schema_out_of_every_log_record E assert 'PRIVATE-SCHEMA-CANARY-7f3a' not in "tool-schema...NARY-7f3a'}]" FAILED tests/test_tool_schema_savings_policy.py::test_reconcile_warns_again_for_a_new_exception_type FAILED tests/test_savings_pricing_fallback_logging.py::test_unpriced_model_logs_once FAILED tests/test_savings_pricing_fallback_logging.py::test_credential_bearing_pricing_error_is_described_not_quoted FAILED tests/test_savings_pricing_fallback_logging.py::test_unpriced_model_not_marked_when_debug_disabled FAILED tests/test_saved_settings_apply_logging.py::test_settings_apply_failure_warns_without_credential =================== 6 failed, 10 passed, 1 warning in 3.54s ==================== # This branch: $ pytest -q -p no:cacheprovider tests/test_tool_schema_savings_policy.py tests/test_savings_pricing_fallback_logging.py tests/test_saved_settings_apply_logging.py ======================== 16 passed, 1 warning in 2.77s ========================= $ pytest -q -p no:cacheprovider <every test file referencing the touched estimators, reconcile_deferred_tokens, settings_store, record_admin_action or log_safety> + the savings/stats/output-shaper files ======================= 401 passed, 1 warning in 44.33s ======================== $ uvx ruff@0.16.8 check <touched files> && uvx ruff@0.16.8 format --check headroom/proxy All checks passed! $ mypy headroom/proxy/tool_schema_savings_policy.py headroom/proxy/savings_tracker.py headroom/proxy/server.py headroom/proxy/outcome.py --ignore-missing-imports Success: no issues found in 4 source files ``` `tests/test_cost_card_attribution.py` and `tests/test_savings_provider_matrix.py` also fail at collection on a clean `origin/main`, because the local litellm build prices none of their candidate models. I left them out of the runs above. ## Real Behavior Proof - Environment: macOS, Python 3.12.13, this branch atd3834fea, the repo .venv, a temp `HOME`, `HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`, and `HEADROOM_OFFLINE=1`. - Exact command / steps: - (1) The shared smoke harness, `smoke.sh <worktree>`. It runs a real `headroom proxy` from this checkout against a local fake upstream, sends one Anthropic and one OpenAI request that each carry a compressible tool result, reads `/stats`, and greps `proxy.log`. - (2) A script that forces each new path: - `settings_store.apply_to_environ` raises `RuntimeError("value sk-secret-should-not-appear")` under `create_app()`; - `reconcile_deferred_tokens` is called twice with a counter that raises `RuntimeError("... secret-schema-text")`; - the unknown model `local-llama-7b` is priced three times with the savings-tracker logger at DEBUG. - Observed result: ```text (1) branch: cleanup/proxy-core-b-savings-logging @d3834feahealth: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 (2) WARNING headroom.proxy: Could not apply saved settings from <tmp>/ws/settings.json (RuntimeError at pcb_4084_proof.py:16 in _boom <- headroom/proxy/server.py:3141 in create_app); continuing with env and defaults. Fix or remove that file and restart the proxy. WARNING headroom.proxy.tool_schema_savings_policy: tool-schema deferral reconcile failed (RuntimeError at pcb_4084_proof.py:31 in _broken <- headroom/proxy/tool_schema_savings_policy.py:146 in reconcile_deferred_tokens); credit left at 100 booked tokens, so /stats over-reports tool-search savings for this request. Please report this as a bug. DEBUG headroom.proxy.savings_tracker: litellm cannot price model='local-llama-7b' as given: BadRequestError at get_llm_provider_logic.py:515 in get_llm_provider <- get_llm_provider_logic.py:528 in get_llm_provider <- cost_calculator.py:472 in cost_per_token DEBUG headroom.proxy.savings_tracker: No litellm input_cost_per_token for model='local-llama-7b'; savings and cost use the fallback rate ``` - Neither secret string appears in any line. - The second reconcile failure does not warn again. - The unpriced model is logged once across three calls. - Not tested: - Forcing the audit-record failure inside a running proxy. It uses the same `describe_exception` pattern. - Seeing the debug lines in a running proxy's `proxy.log`. That depends on #4137, which makes `HEADROOM_LOG_LEVEL=debug` reach `headroom.*`. Until it lands, these debug lines show only when the `headroom` logger is set to DEBUG programmatically, as in the script. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A; logging only. - Stable/default behavior changed: No. Every dollar value is the same as before, and an unpriced model now gets its fallback returned directly instead of through a raise and catch, with an identical result. Only the log output changes. - Kill switch / disable path: None needed. Warnings fire only on a real failure, and they are deduplicated with a bound. - Unsafe override required: No. `HEADROOM_DEBUG_DUMP=full` is the existing explicit opt-in for full exception text. - Qualification impact: None. - Rollback path: Revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. Docs for the new warnings are collected for the single follow-up docs PR. Depends on #4138; its files (`headroom/log_safety.py`, `handlers/_debug_dump.py` and their tests) show in this diff until it merges. This PR's own diff, excluding #4138's files, is +377/−20, and 256 of those added lines are tests. `cost._warn_pricing_once` is not reused for unpriced models because it logs at WARNING, and it shares its set with the live cost path, which already warns once per unpriceable model. For an unknown model, cache savings book 0 while compression savings use the $3/M fallback. This PR leaves that unchanged for a maintainer decision. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
5698e91b2a
commit
2800932e18
@@ -0,0 +1,383 @@
|
||||
"""Payload-safe building blocks for log lines.
|
||||
|
||||
Log records are kept, rotated, attached to bug reports and shipped to log
|
||||
collectors, so no level may carry request content, tool output or
|
||||
credentials. Exception messages are the usual leak: a compressor, hook or
|
||||
storage backend can quote the payload or a connection URL in its error. These
|
||||
helpers describe a failure by what is safe to keep (exception types, code
|
||||
locations, errno, ids, counts) and leave the message out.
|
||||
|
||||
Content is logged only when the operator opts in to it explicitly with
|
||||
``HEADROOM_DEBUG_DUMP=full``, the same switch that gates full request dumps.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import traceback
|
||||
from collections import deque
|
||||
from collections.abc import Hashable
|
||||
from types import FrameType, ModuleType, TracebackType
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
CONTENT_OPT_IN_ENV = "HEADROOM_DEBUG_DUMP"
|
||||
_CONTENT_OPT_IN_VALUES = frozenset({"full", "all", "content"})
|
||||
_MAX_FRAMES = 3
|
||||
_MAX_CHAIN = 4
|
||||
_ID_MAX_CHARS = 80
|
||||
_MAX_ERRNO = 2**31 - 1
|
||||
# Schemes whose URLs have no host ("file:///path").
|
||||
_LOCAL_SCHEMES = frozenset({"file", "sqlite", "unix"})
|
||||
# Strict DNS labels. "_" is left out on purpose: it is not valid in a public
|
||||
# hostname but is common in keys (sk_live_...), so a host with it is masked.
|
||||
_HOSTNAME = re.compile(
|
||||
r"[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?(?:\.[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?)*\.?"
|
||||
)
|
||||
_HOSTLESS_URL = re.compile(r"([A-Za-z][A-Za-z0-9+.-]*):///")
|
||||
# What urlsplit removes before parsing: leading C0 controls and spaces, and
|
||||
# every tab and newline. The hostless check must look at the same text.
|
||||
_URL_LEADING_JUNK = "".join(chr(code) for code in range(33))
|
||||
_URL_REMOVED = str.maketrans("", "", "\t\r\n")
|
||||
_HEADROOM_MODULE = re.compile(r"headroom(?:\.[A-Za-z_][A-Za-z0-9_]*)*")
|
||||
_IDENTIFIER = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")
|
||||
# A class's __qualname__: identifiers joined by dots, with <locals> for nested ones.
|
||||
_QUALIFIED_NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*(?:\.(?:<locals>|[A-Za-z_][A-Za-z0-9_]*))*")
|
||||
_FIXED_CODE_NAMES = frozenset(
|
||||
{"<module>", "<lambda>", "<genexpr>", "<listcomp>", "<dictcomp>", "<setcomp>"}
|
||||
)
|
||||
# The base classes' own attribute descriptors. Reading through them means no
|
||||
# subclass property (a __cause__ or errno that raises, a __qualname__ chosen by
|
||||
# a metaclass) can run while a failure is being described.
|
||||
_CAUSE = vars(BaseException)["__cause__"]
|
||||
_CONTEXT = vars(BaseException)["__context__"]
|
||||
_SUPPRESS_CONTEXT = vars(BaseException)["__suppress_context__"]
|
||||
_TRACEBACK = vars(BaseException)["__traceback__"]
|
||||
_ERRNO = vars(OSError)["errno"]
|
||||
_QUALNAME = vars(type)["__qualname__"]
|
||||
# Ints above this many bits have more than 4300 digits, which repr refuses.
|
||||
_MAX_ID_INT_BITS = 14_000
|
||||
# The module __dict__ descriptor itself, so a ModuleType subclass cannot override it.
|
||||
_MODULE_DICT = vars(ModuleType)["__dict__"]
|
||||
# Schemes a log may name; any other text before "://" could be a key someone
|
||||
# pasted where a URL belongs, so it is replaced.
|
||||
_KNOWN_SCHEMES = frozenset(
|
||||
{
|
||||
"http",
|
||||
"https",
|
||||
"ws",
|
||||
"wss",
|
||||
"grpc",
|
||||
"grpcs",
|
||||
"redis",
|
||||
"rediss",
|
||||
"postgres",
|
||||
"postgresql",
|
||||
"mysql",
|
||||
"mongodb",
|
||||
"mongodb+srv",
|
||||
"sqlite",
|
||||
"file",
|
||||
"unix",
|
||||
"s3",
|
||||
"gs",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def content_logging_enabled() -> bool:
|
||||
"""True when the operator opted in to content in diagnostics (``HEADROOM_DEBUG_DUMP=full``)."""
|
||||
return os.environ.get(CONTENT_OPT_IN_ENV, "").strip().lower() in _CONTENT_OPT_IN_VALUES
|
||||
|
||||
|
||||
def describe_exception(exc: BaseException) -> str:
|
||||
"""Describe ``exc`` for a log line without its message.
|
||||
|
||||
Gives each exception in the cause chain as its type plus the innermost code
|
||||
locations, e.g. ``ValueError at headroom/x.py:12 in parse; caused by
|
||||
KeyError at ...``. An ``OSError`` keeps its errno and the system's text for
|
||||
it, never the exception's own strerror. With ``HEADROOM_DEBUG_DUMP=full`` it returns the full
|
||||
traceback, messages included.
|
||||
"""
|
||||
if content_logging_enabled():
|
||||
return "".join(traceback.format_exception(exc)).rstrip()
|
||||
parts: list[str] = []
|
||||
seen: set[int] = set()
|
||||
current: BaseException | None = exc
|
||||
while current is not None and id(current) not in seen and len(parts) < _MAX_CHAIN:
|
||||
seen.add(id(current))
|
||||
parts.append(_describe_one(current))
|
||||
current = _CAUSE.__get__(current) or (
|
||||
None if _SUPPRESS_CONTEXT.__get__(current) else _CONTEXT.__get__(current)
|
||||
)
|
||||
return "; caused by ".join(parts)
|
||||
|
||||
|
||||
def _describe_one(exc: BaseException) -> str:
|
||||
# A class name is chosen by whoever defined it (type("Err\nforged", ...)).
|
||||
text = _class_name(type(exc)) or "<exception>"
|
||||
# An int subclass errno could format itself as anything, so only a plain int.
|
||||
errno = _ERRNO.__get__(exc) if issubclass(type(exc), OSError) else None
|
||||
if type(errno) is int:
|
||||
# The system's own text for the errno; exc.strerror is free text in some
|
||||
# subclasses (ssl.SSLError) and can quote a path or a peer's message.
|
||||
# An out-of-range errno is not formatted at all: converting a huge int
|
||||
# to text can itself raise, and a log helper must never raise.
|
||||
# Negative codes (socket.gaierror's EAI_*) have no os.strerror text.
|
||||
if 0 <= errno <= _MAX_ERRNO:
|
||||
text += f" [Errno {errno}] {_errno_text(errno)}"
|
||||
elif -_MAX_ERRNO - 1 <= errno < 0:
|
||||
text += f" [Errno {errno}]"
|
||||
else:
|
||||
text += " [Errno out of range]"
|
||||
frames = _innermost_headroom_frames(_TRACEBACK.__get__(exc))
|
||||
if frames:
|
||||
text += " at " + " <- ".join(reversed(frames))
|
||||
return text
|
||||
|
||||
|
||||
def _errno_text(errno: int) -> str:
|
||||
try:
|
||||
return os.strerror(errno)
|
||||
except (ValueError, OverflowError):
|
||||
return "unknown error"
|
||||
|
||||
|
||||
def _innermost_headroom_frames(tb: TracebackType | None) -> list[str]:
|
||||
"""Locations of the innermost Headroom frames, at most ``_MAX_FRAMES``, in memory bounded by that."""
|
||||
frames: deque[str] = deque(maxlen=_MAX_FRAMES)
|
||||
while tb is not None:
|
||||
location = _headroom_frame_location(tb.tb_frame, tb.tb_lineno)
|
||||
if location is not None:
|
||||
frames.append(location)
|
||||
tb = tb.tb_next
|
||||
return list(frames)
|
||||
|
||||
|
||||
def _headroom_frame_location(frame: FrameType, lineno: int) -> str | None:
|
||||
"""``headroom/<module>.py:line in function`` for a frame of Headroom's own code, else None.
|
||||
|
||||
Only Headroom's own frames are named; library and stdlib frames are skipped,
|
||||
since any text a frame carries (file, module or function name) can be set by
|
||||
code compiled at runtime and quote request data. A frame counts as Headroom's
|
||||
only when all of these hold, so none of its text is chosen at runtime:
|
||||
its globals are the namespace of the module ``sys.modules`` holds under that
|
||||
name; that name is a plain dotted ``headroom.*`` identifier; the code's file
|
||||
is the module's own file (compared by stem, so ``.pyc``-only and zip installs
|
||||
match); and the function name is a plain identifier. Headroom never execs code.
|
||||
"""
|
||||
namespace = frame.f_globals
|
||||
# Read through dict's own methods: globals can be a dict subclass.
|
||||
name = dict.get(namespace, "__name__")
|
||||
if type(name) is not str or not _HEADROOM_MODULE.fullmatch(name):
|
||||
return None
|
||||
module = sys.modules.get(name)
|
||||
if not isinstance(module, ModuleType) or _MODULE_DICT.__get__(module) is not namespace:
|
||||
return None
|
||||
code = frame.f_code
|
||||
module_file = dict.get(namespace, "__file__")
|
||||
if type(module_file) is not str or _stem(module_file) != _stem(code.co_filename):
|
||||
return None
|
||||
function = code.co_name
|
||||
if not (_IDENTIFIER.fullmatch(function) or function in _FIXED_CODE_NAMES):
|
||||
return None
|
||||
package = dict.__contains__(namespace, "__path__")
|
||||
path = name.replace(".", "/") + ("/__init__.py" if package else ".py")
|
||||
return f"{path}:{lineno} in {function}"
|
||||
|
||||
|
||||
def _stem(path: str) -> str:
|
||||
return os.path.splitext(path.replace("\\", "/").rsplit("/", 1)[-1])[0]
|
||||
|
||||
|
||||
def redact_url(url: str) -> str:
|
||||
"""Return ``url`` as scheme, host and port only, safe for any log line.
|
||||
|
||||
Userinfo, query and fragment are always dropped, and so is the path, since
|
||||
a path segment can carry a credential (``/bot<token>/``, webhook URLs) and
|
||||
no rule tells such a segment from a route name. The path is kept only with
|
||||
the explicit content opt-in, ``HEADROOM_DEBUG_DUMP=full``. The host comes
|
||||
back lowercased, and only when it is a DNS name or IP address; a URL with
|
||||
no host is accepted only for ``file``, ``sqlite`` and ``unix``.
|
||||
"""
|
||||
try:
|
||||
parts = urlsplit(url)
|
||||
host = parts.hostname or ""
|
||||
port = parts.port
|
||||
except ValueError:
|
||||
return "<unparseable url>"
|
||||
# Without "//" urlsplit reads "sk-key:secret" as scheme "sk-key", so a string
|
||||
# with no host is logged only for the hostless schemes ("file:///path").
|
||||
if not parts.netloc and not _is_hostless_local_url(url, parts.scheme):
|
||||
return "<unparseable url>"
|
||||
if parts.netloc:
|
||||
# A host that is not an IP or a strict DNS name may be a pasted secret;
|
||||
# it is masked, and scheme and port still say which endpoint was meant.
|
||||
host = _checked_host(host) or "<host>"
|
||||
scheme = parts.scheme if parts.scheme in _KNOWN_SCHEMES else "<scheme>"
|
||||
if ":" in host and host != "<host>":
|
||||
host = f"[{host}]"
|
||||
netloc = f"{host}:{port}" if port is not None else host
|
||||
if content_logging_enabled():
|
||||
path = parts.path
|
||||
else:
|
||||
path = "/<path>" if parts.path not in ("", "/") else parts.path
|
||||
redacted = f"{scheme}://{netloc}{path}"
|
||||
return f"{redacted}?<redacted>" if parts.query else redacted
|
||||
|
||||
|
||||
def _is_hostless_local_url(url: str, scheme: str) -> bool:
|
||||
"""True for ``file:///…``, ``sqlite:///…`` or ``unix:///…`` as urlsplit read it."""
|
||||
cleaned = url.lstrip(_URL_LEADING_JUNK).translate(_URL_REMOVED)
|
||||
match = _HOSTLESS_URL.match(cleaned)
|
||||
return scheme in _LOCAL_SCHEMES and match is not None and match.group(1).lower() == scheme
|
||||
|
||||
|
||||
def _checked_host(host: str) -> str | None:
|
||||
"""``host`` as a loggable name or address, or None when it may not be one.
|
||||
|
||||
An IPv6 zone id (``fe80::1%eth0``) is free text, so it is dropped. A
|
||||
non-ASCII name is given in its ASCII (IDNA) form.
|
||||
"""
|
||||
address = host.split("%", 1)[0]
|
||||
try:
|
||||
ipaddress.ip_address(address)
|
||||
except ValueError:
|
||||
pass
|
||||
else:
|
||||
return address
|
||||
try:
|
||||
ascii_host = host.encode("idna").decode("ascii").lower()
|
||||
except UnicodeError:
|
||||
return None
|
||||
return ascii_host if _HOSTNAME.fullmatch(ascii_host) else None
|
||||
|
||||
|
||||
def safe_id(value: object) -> str:
|
||||
"""Render a client- or model-supplied id as one bounded log token.
|
||||
|
||||
Uses the built-in ``repr`` of ``str``, ``bytes`` and numbers, so a newline
|
||||
in the id cannot forge a second log line and no ``__repr__`` of the
|
||||
caller's object runs; any other type is shown by its name. Cuts the result
|
||||
to a fixed length so an oversized id cannot flood the log.
|
||||
"""
|
||||
text = _builtin_repr(value)
|
||||
if len(text) <= _ID_MAX_CHARS:
|
||||
return text
|
||||
return f"{text[:_ID_MAX_CHARS]}…(+{len(text) - _ID_MAX_CHARS} chars)"
|
||||
|
||||
|
||||
def _builtin_repr(value: object) -> str:
|
||||
# type() rather than isinstance: isinstance consults a __class__ the object
|
||||
# can fake, and then a built-in repr would run on the wrong type.
|
||||
kind = type(value)
|
||||
if value is None or kind is bool:
|
||||
return "None" if value is None else ("True" if value else "False")
|
||||
if issubclass(kind, str):
|
||||
return str.__repr__(value) # type: ignore[arg-type]
|
||||
if issubclass(kind, bytes):
|
||||
return bytes.__repr__(value) # type: ignore[arg-type]
|
||||
if issubclass(kind, int):
|
||||
return _int_repr(value) # type: ignore[arg-type]
|
||||
if issubclass(kind, float):
|
||||
return float.__repr__(value) # type: ignore[arg-type]
|
||||
return f"<{_class_name(kind) or 'object'}>"
|
||||
|
||||
|
||||
def _int_repr(value: int) -> str:
|
||||
bits = int.bit_length(value)
|
||||
# Digits an int of this many bits can have, against the process's own limit
|
||||
# (sys.set_int_max_str_digits); past it, repr raises ValueError.
|
||||
# The limit and its getter arrived in 3.10.7; earlier releases have no limit.
|
||||
get_limit = getattr(sys, "get_int_max_str_digits", None)
|
||||
limit = get_limit() if get_limit is not None else 0
|
||||
if bits > _MAX_ID_INT_BITS or (limit and bits * 0.30103 + 1 >= limit):
|
||||
return f"<int of {bits} bits>"
|
||||
return int.__repr__(value)
|
||||
|
||||
|
||||
def _class_name(cls: type) -> str | None:
|
||||
"""The class's qualified name when it is plain identifiers, else None.
|
||||
|
||||
Whoever defines a class chooses its name (``type("sk-...\\r", ...)``), so only
|
||||
names shaped like source identifiers are logged.
|
||||
"""
|
||||
name = _QUALNAME.__get__(cls)
|
||||
return name if type(name) is str and _QUALIFIED_NAME.fullmatch(name) else None
|
||||
|
||||
|
||||
class WarnOnce:
|
||||
"""Warn once per key per window, for at most ``limit`` keys per window.
|
||||
|
||||
The first warning opens a window of ``window_seconds`` (an hour by
|
||||
default). Inside it each key warns once; past ``limit`` keys it logs one
|
||||
overflow WARNING and stays quiet, so a failure that hits many keys
|
||||
(paths, rows, models) cannot flood the log: at most ``limit + 1`` warnings
|
||||
per window. When the window ends the guard starts over, so a failure that
|
||||
keeps recurring, or a newly broken key, warns again once an hour. A key is
|
||||
used up only when WARNING is enabled, so a warning suppressed by the log
|
||||
level is still emitted later. ``forget`` re-arms a key once its failure has
|
||||
cleared; a re-armed key that fails again counts against the same window's
|
||||
budget, so forgetting can never push a window past ``limit + 1`` warnings.
|
||||
"""
|
||||
|
||||
def __init__(self, limit: int, what: str, *, window_seconds: float = 3600.0) -> None:
|
||||
if limit < 1:
|
||||
raise ValueError(f"WarnOnce limit must be at least 1, got {limit}")
|
||||
if not 0 < window_seconds < math.inf: # also rejects nan
|
||||
raise ValueError(
|
||||
f"WarnOnce window_seconds must be positive and finite, got {window_seconds}"
|
||||
)
|
||||
self._limit = limit
|
||||
self._what = what
|
||||
self._window = window_seconds
|
||||
self._keys: set[Hashable] = set()
|
||||
self._issued = 0 # warnings issued this window; forget() does not refund them
|
||||
self._window_start: float | None = None
|
||||
self._overflowed = False
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def first(self, key: Hashable, log: logging.Logger) -> bool:
|
||||
"""Return True if the caller should log this key's WARNING now."""
|
||||
if not log.isEnabledFor(logging.WARNING):
|
||||
return False
|
||||
with self._lock:
|
||||
now = time.monotonic()
|
||||
if self._window_start is not None and now - self._window_start >= self._window:
|
||||
self._keys.clear()
|
||||
self._issued = 0
|
||||
self._overflowed = False
|
||||
self._window_start = None
|
||||
if key in self._keys:
|
||||
return False
|
||||
if not self._overflowed and self._issued < self._limit:
|
||||
self._keys.add(key)
|
||||
self._issued += 1
|
||||
if self._window_start is None:
|
||||
self._window_start = now
|
||||
return True
|
||||
report_overflow = not self._overflowed
|
||||
self._overflowed = True
|
||||
started = now if self._window_start is None else self._window_start
|
||||
remaining = self._window - (now - started)
|
||||
if report_overflow:
|
||||
log.warning(
|
||||
"More than %d warnings about %s this window; further ones are not logged "
|
||||
"at WARNING "
|
||||
"for the next %d minutes",
|
||||
self._limit,
|
||||
self._what,
|
||||
max(1, math.ceil(remaining / 60)),
|
||||
)
|
||||
return False
|
||||
|
||||
def forget(self, key: Hashable) -> None:
|
||||
"""Re-arm ``key`` so its next failure warns again."""
|
||||
with self._lock:
|
||||
self._keys.discard(key)
|
||||
@@ -14,7 +14,8 @@ import os
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
|
||||
from headroom.log_safety import content_logging_enabled, redact_url
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -29,9 +30,9 @@ def _debug_dump_mode(config: Any) -> str:
|
||||
"""
|
||||
if getattr(config, "stateless", False):
|
||||
return "off"
|
||||
raw = os.environ.get("HEADROOM_DEBUG_DUMP", "").strip().lower()
|
||||
if raw in ("full", "all", "content"):
|
||||
if content_logging_enabled():
|
||||
return "full"
|
||||
raw = os.environ.get("HEADROOM_DEBUG_DUMP", "").strip().lower()
|
||||
if raw in ("1", "true", "yes", "on", "redacted"):
|
||||
return "redacted"
|
||||
return "off"
|
||||
@@ -54,23 +55,6 @@ def _redact_debug_value(value: Any, _max_len: int = 80) -> Any:
|
||||
return value
|
||||
|
||||
|
||||
def _redact_url(url: str) -> str:
|
||||
"""Drop the query string and fragment from ``url``.
|
||||
|
||||
Some upstreams carry credentials in the query (Gemini's ``?key=``), and the
|
||||
query is never needed to debug a rejected request, so it is elided in every
|
||||
mode — ``full`` opts in to prompt content, not to credentials.
|
||||
"""
|
||||
try:
|
||||
parts = urlsplit(url)
|
||||
except ValueError:
|
||||
return "<unparseable url>"
|
||||
if not parts.query and not parts.fragment:
|
||||
return url
|
||||
redacted = urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))
|
||||
return f"{redacted}?<redacted>" if parts.query else redacted
|
||||
|
||||
|
||||
def _decode_sent_body(body: Any) -> Any:
|
||||
"""Decode the exact bytes sent upstream back to JSON for the dump."""
|
||||
if not isinstance(body, (bytes, bytearray)):
|
||||
@@ -119,7 +103,7 @@ def write_upstream_error_dump(
|
||||
payload = json.dumps(
|
||||
{
|
||||
"request_id": request_id,
|
||||
"url": _redact_url(url),
|
||||
"url": redact_url(url),
|
||||
"status": status,
|
||||
"provider": provider,
|
||||
"model": model,
|
||||
|
||||
@@ -31,6 +31,7 @@ from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from headroom.log_safety import describe_exception
|
||||
from headroom.proxy.conversation_savings import get_conversation_savings
|
||||
from headroom.proxy.tool_schema_savings_policy import (
|
||||
headline_tokens_saved,
|
||||
@@ -629,8 +630,12 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
|
||||
# does a full read-modify-write of the ledger file — run them
|
||||
# together off the event loop (#18) so a slow flush can't stall it.
|
||||
output_tokens_saved_est = await asyncio.to_thread(_record_and_estimate)
|
||||
except Exception: # pragma: no cover - defensive
|
||||
pass
|
||||
except Exception as exc: # pragma: no cover - defensive
|
||||
logger.debug(
|
||||
"Output savings record/estimate failed for request %s; booking 0: %s",
|
||||
outcome.request_id,
|
||||
describe_exception(exc),
|
||||
)
|
||||
|
||||
# Project attribution: explicit outcome field wins, else the value the
|
||||
# HTTP middleware / WS accept captured from ``X-Headroom-Project``.
|
||||
|
||||
@@ -24,6 +24,7 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from headroom import paths as _paths
|
||||
from headroom.log_safety import WarnOnce, describe_exception, safe_id
|
||||
from headroom.proxy import project_name_policy
|
||||
from headroom.proxy.persistent_metrics import PersistentMetricsState
|
||||
|
||||
@@ -355,14 +356,23 @@ def _resolve_litellm_model(model: str) -> str:
|
||||
info = litellm.model_cost.get(resolved)
|
||||
if info and info.get("input_cost_per_token") is not None:
|
||||
return resolved
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as exc:
|
||||
logger.debug(
|
||||
"Shared litellm model resolution failed for model=%s: %s",
|
||||
safe_id(model),
|
||||
describe_exception(exc),
|
||||
)
|
||||
|
||||
# litellm raises a bare Exception for a model it has not mapped, so this is
|
||||
# the narrowest catch that works. A miss means "not this spelling, try the
|
||||
# provider prefixes next". Runs once per model (lru_cache above).
|
||||
try:
|
||||
litellm.cost_per_token(model=model, prompt_tokens=1, completion_tokens=0)
|
||||
return model
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as exc:
|
||||
logger.debug(
|
||||
"litellm cannot price model=%s as given: %s", safe_id(model), describe_exception(exc)
|
||||
)
|
||||
|
||||
prefixes = {
|
||||
"claude-": "anthropic/",
|
||||
@@ -388,6 +398,25 @@ def _resolve_litellm_model(model: str) -> str:
|
||||
return model
|
||||
|
||||
|
||||
# An unpriced model (a local model, a gateway alias, a model newer than the
|
||||
# bundled litellm) is an expected case, and the estimators below run on every
|
||||
# request, so each model is logged once. Debug, not warning: proxy/cost.py
|
||||
# already warns once per unpriceable model on the live path. Model names come
|
||||
# from clients, so the guard is bounded like _resolve_litellm_model's cache.
|
||||
_unpriced_models_logged = WarnOnce(_MODEL_RESOLUTION_CACHE_MAXSIZE, "unpriced models")
|
||||
|
||||
|
||||
def _log_unpriced_model_once(model: str, field: str) -> None:
|
||||
if not logger.isEnabledFor(logging.DEBUG):
|
||||
return
|
||||
if _unpriced_models_logged.first(model, logger):
|
||||
logger.debug(
|
||||
"No litellm %s for model=%s; savings and cost use the fallback rate",
|
||||
field,
|
||||
safe_id(model),
|
||||
)
|
||||
|
||||
|
||||
def _estimate_compression_savings_usd(model: str, tokens_saved: int) -> float:
|
||||
"""Estimate compression savings in USD from saved input tokens."""
|
||||
litellm = _get_litellm_module()
|
||||
@@ -405,9 +434,15 @@ def _estimate_compression_savings_usd(model: str, tokens_saved: int) -> float:
|
||||
# a real 0.0 as unavailable and billed the $3/M fallback — phantom savings
|
||||
# for a model that costs nothing.
|
||||
if input_cost_per_token is None:
|
||||
raise RuntimeError("input cost unavailable")
|
||||
_log_unpriced_model_once(model, "input_cost_per_token")
|
||||
return float(tokens_saved) * float(DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN)
|
||||
return float(tokens_saved) * float(input_cost_per_token)
|
||||
except Exception:
|
||||
except Exception as exc:
|
||||
logger.debug(
|
||||
"Compression savings priced at the fallback rate for model=%s: %s",
|
||||
safe_id(model),
|
||||
describe_exception(exc),
|
||||
)
|
||||
return float(tokens_saved) * float(DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN)
|
||||
|
||||
|
||||
@@ -432,9 +467,15 @@ def _estimate_output_savings_usd(model: str, tokens_saved: int) -> float:
|
||||
# rate -> phantom output savings for a model that costs nothing. Mirrors
|
||||
# the fix already applied to `_estimate_compression_savings_usd`.
|
||||
if output_cost_per_token is None:
|
||||
raise RuntimeError("output cost unavailable")
|
||||
_log_unpriced_model_once(model, "output_cost_per_token")
|
||||
return float(tokens_saved) * float(DEFAULT_FALLBACK_OUTPUT_COST_PER_TOKEN)
|
||||
return float(tokens_saved) * float(output_cost_per_token)
|
||||
except Exception:
|
||||
except Exception as exc:
|
||||
logger.debug(
|
||||
"Output savings priced at the fallback rate for model=%s: %s",
|
||||
safe_id(model),
|
||||
describe_exception(exc),
|
||||
)
|
||||
return float(tokens_saved) * float(DEFAULT_FALLBACK_OUTPUT_COST_PER_TOKEN)
|
||||
|
||||
|
||||
@@ -481,7 +522,12 @@ def _estimate_cache_savings_usd(model: str, cache_read_tokens: int) -> float:
|
||||
if discount <= 0:
|
||||
return 0.0
|
||||
return float(cache_read_tokens) * discount
|
||||
except Exception:
|
||||
except Exception as exc:
|
||||
logger.debug(
|
||||
"Cache savings pricing failed for model=%s; booking 0: %s",
|
||||
safe_id(model),
|
||||
describe_exception(exc),
|
||||
)
|
||||
return 0.0
|
||||
|
||||
|
||||
@@ -628,7 +674,8 @@ def _estimate_input_cost_usd(
|
||||
# A missing key means the model is unknown → fall back to a blended rate.
|
||||
# A present 0.0 means the model is free and must cost $0, not the fallback.
|
||||
if input_cost_per_token is None:
|
||||
raise RuntimeError("input cost unavailable")
|
||||
_log_unpriced_model_once(model, "input_cost_per_token")
|
||||
return float(chargeable_tokens) * float(DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN)
|
||||
|
||||
if use_breakdown:
|
||||
cache_read_cost = info.get(
|
||||
@@ -646,7 +693,12 @@ def _estimate_input_cost_usd(
|
||||
)
|
||||
|
||||
return float(total_input_tokens) * float(input_cost_per_token)
|
||||
except Exception:
|
||||
except Exception as exc:
|
||||
logger.debug(
|
||||
"Input cost priced at the fallback rate for model=%s: %s",
|
||||
safe_id(model),
|
||||
describe_exception(exc),
|
||||
)
|
||||
return float(chargeable_tokens) * float(DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN)
|
||||
|
||||
|
||||
|
||||
@@ -86,6 +86,7 @@ from headroom.config import (
|
||||
ReadLifecycleConfig,
|
||||
)
|
||||
from headroom.dashboard import get_dashboard_html
|
||||
from headroom.log_safety import describe_exception, safe_id
|
||||
from headroom.observability import (
|
||||
LangfuseTracingConfig,
|
||||
OTelMetricsConfig,
|
||||
@@ -3129,12 +3130,24 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
# that construct the app without going through the `headroom` CLI entrypoint
|
||||
# (which already applies them before Click parsing). setdefault keeps
|
||||
# explicit env exports authoritative; fail-open so it never blocks startup.
|
||||
# settings_store.load() already reports an unreadable or corrupt file; this
|
||||
# catches anything else that goes wrong while applying it.
|
||||
settings_file: object = "the saved settings file"
|
||||
try:
|
||||
from headroom import paths as _hr_paths
|
||||
from headroom import settings_store
|
||||
|
||||
settings_file = _hr_paths.settings_path()
|
||||
settings_store.apply_to_environ(settings_store.load())
|
||||
except Exception: # noqa: BLE001 — settings load must never break startup
|
||||
pass
|
||||
except Exception as exc: # noqa: BLE001 — settings load must never break startup
|
||||
# Settings values can hold credentials, so the exception is described by
|
||||
# type and code location only, never its message.
|
||||
logger.warning(
|
||||
"Could not apply saved settings from %s (%s); continuing with env and "
|
||||
"defaults. Fix or remove that file and restart the proxy.",
|
||||
settings_file,
|
||||
describe_exception(exc),
|
||||
)
|
||||
|
||||
# Air-gap master switch. Propagate config.offline to the env so the
|
||||
# env-based egress predicates (telemetry, update check, license) all honor
|
||||
@@ -4148,8 +4161,14 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
action="admin_request",
|
||||
status_code=response.status_code,
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("admin audit emission failed", exc_info=True)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"Admin audit record not written for %s %s (%s); this request is missing "
|
||||
"from the headroom.audit trail. Please report this as a bug.",
|
||||
request.method,
|
||||
safe_id(request.url.path),
|
||||
describe_exception(exc),
|
||||
)
|
||||
return response
|
||||
|
||||
# The gate above is http-only (BaseHTTPMiddleware ignores every other
|
||||
@@ -4994,8 +5013,8 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
|
||||
_oest = get_recorder().estimate(_olevel)
|
||||
output_reduction = _output_reduction_payload(_shaper_active, _oest)
|
||||
except Exception: # pragma: no cover - defensive
|
||||
pass
|
||||
except Exception as exc: # pragma: no cover - defensive
|
||||
logger.debug("/stats output_reduction section failed: %s", describe_exception(exc))
|
||||
|
||||
# Model-routing section: populated only when an extension registered a
|
||||
# routing-stats provider (headroom.proxy.routing_stats); None otherwise.
|
||||
|
||||
@@ -26,8 +26,32 @@ name to :data:`TOOL_SCHEMA_SAVINGS_TAGS` and every surface picks it up.
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from headroom.log_safety import WarnOnce, describe_exception
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# A failure that repeats on every request warns once per exception type; every
|
||||
# occurrence also goes to debug.
|
||||
_reconcile_warned = WarnOnce(32, "tool-schema reconcile failure types")
|
||||
|
||||
|
||||
def _log_reconcile_failure(exc: Exception, booked: int) -> None:
|
||||
# The exception comes from tokenizing client tool schemas and can quote
|
||||
# them, so it is described by type and code location only, at every level.
|
||||
where = describe_exception(exc)
|
||||
if _reconcile_warned.first(type(exc).__qualname__, logger):
|
||||
logger.warning(
|
||||
"tool-schema deferral reconcile failed (%s); credit left at %d booked tokens, "
|
||||
"so /stats over-reports tool-search savings for this request. Please report "
|
||||
"this as a bug.",
|
||||
where,
|
||||
booked,
|
||||
)
|
||||
logger.debug("tool-schema deferral reconcile failed; booked=%d: %s", booked, where)
|
||||
|
||||
|
||||
def without_deferral_flags(tools: object) -> object:
|
||||
"""The tool array as it is MEASURED for a hook's tool delta.
|
||||
@@ -128,7 +152,8 @@ def reconcile_deferred_tokens(
|
||||
tags["tool_search_deferred_tools"] = len(kept)
|
||||
if isinstance(entry, dict):
|
||||
entry["tokens"] = max(0, int(entry.get("tokens") or 0) - released)
|
||||
except Exception: # accounting must never break a request
|
||||
except Exception as exc: # accounting must never break a request
|
||||
_log_reconcile_failure(exc, booked)
|
||||
return
|
||||
|
||||
|
||||
|
||||
@@ -189,10 +189,25 @@ def test_upstream_dump_never_writes_url_credentials(dump_dir, monkeypatch, mode)
|
||||
assert path is not None
|
||||
text = path.read_text()
|
||||
assert "AIzaSECRET" not in text
|
||||
assert json.loads(text)["url"] == (
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/gemini:streamGenerateContent"
|
||||
"?<redacted>"
|
||||
)
|
||||
if mode == "full": # the content opt-in keeps the route, still never the key
|
||||
expected = (
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/gemini:streamGenerateContent"
|
||||
)
|
||||
else: # a path segment can carry a credential, so it is dropped without the opt-in
|
||||
expected = "https://generativelanguage.googleapis.com/<path>"
|
||||
assert json.loads(text)["url"] == expected + "?<redacted>"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"url",
|
||||
["https://upstream.example/botCANARY/v1/messages", "https://upstream.example/bot%43ANARY/v1"],
|
||||
)
|
||||
def test_redacted_dump_never_writes_path_credentials(dump_dir, monkeypatch, url):
|
||||
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "redacted")
|
||||
path = _write(url=url)
|
||||
assert path is not None
|
||||
text = path.read_text()
|
||||
assert "CANARY" not in text and "%43ANARY" not in text
|
||||
|
||||
|
||||
def test_upstream_dump_keeps_query_free_url(dump_dir, monkeypatch):
|
||||
|
||||
@@ -0,0 +1,555 @@
|
||||
"""Tests for the payload-safe log helpers in ``headroom.log_safety``."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import py_compile
|
||||
import sys
|
||||
import types
|
||||
import zipfile
|
||||
import zipimport
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom import log_safety
|
||||
from headroom.log_safety import WarnOnce, describe_exception, redact_url, safe_id
|
||||
|
||||
CANARY = "sk-live-CANARY tool output: patient record 42"
|
||||
|
||||
|
||||
def _raise_chain() -> BaseException:
|
||||
try:
|
||||
try:
|
||||
raise KeyError(CANARY)
|
||||
except KeyError as inner:
|
||||
raise ValueError(f"could not parse {CANARY}") from inner
|
||||
except ValueError as exc:
|
||||
return exc
|
||||
|
||||
|
||||
def test_exception_description_names_types_and_places_but_not_messages(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
text = describe_exception(_raise_chain())
|
||||
|
||||
assert "CANARY" not in text
|
||||
# The chain is raised from this test file, which is not Headroom's code,
|
||||
# so only the types are named.
|
||||
assert text == "ValueError; caused by KeyError"
|
||||
|
||||
|
||||
def test_oserror_keeps_errno_but_not_its_filename(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
text = describe_exception(PermissionError(13, "Permission denied", f"/tmp/{CANARY}"))
|
||||
|
||||
assert text == "PermissionError [Errno 13] Permission denied"
|
||||
|
||||
|
||||
def test_oserror_free_text_strerror_is_replaced_by_the_system_text() -> None:
|
||||
text = describe_exception(OSError(2, f"token={CANARY}"))
|
||||
|
||||
assert "CANARY" not in text
|
||||
assert text == f"FileNotFoundError [Errno 2] {os.strerror(2)}"
|
||||
|
||||
|
||||
def test_content_opt_in_gives_the_full_traceback(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "full")
|
||||
|
||||
text = describe_exception(_raise_chain())
|
||||
|
||||
assert "Traceback" in text
|
||||
assert CANARY in text
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("url", "expected"),
|
||||
[
|
||||
(
|
||||
"https://user:hunter2@proxy.example:8443/v1?key=abc#frag",
|
||||
"https://proxy.example:8443/<path>?<redacted>",
|
||||
),
|
||||
("https://upstream.example/botSECRET/v1/messages", "https://upstream.example/<path>"),
|
||||
("https://upstream.example/bot%53ECRET/v1", "https://upstream.example/<path>"),
|
||||
("http://127.0.0.1:8787", "http://127.0.0.1:8787"),
|
||||
("http://[::1]:8787/", "http://[::1]:8787/"),
|
||||
("http://[::1", "<unparseable url>"),
|
||||
],
|
||||
)
|
||||
def test_redact_url_keeps_only_scheme_host_and_port(
|
||||
url: str, expected: str, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
assert redact_url(url) == expected
|
||||
|
||||
|
||||
def test_content_opt_in_keeps_the_path_but_never_credentials(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "full")
|
||||
|
||||
assert redact_url("https://u:pw@upstream.example/v1/messages?key=abc") == (
|
||||
"https://upstream.example/v1/messages?<redacted>"
|
||||
)
|
||||
|
||||
|
||||
def test_safe_id_escapes_newlines_and_bounds_length() -> None:
|
||||
assert safe_id("m1\nWARNING forged") == "'m1\\nWARNING forged'"
|
||||
long = safe_id("x" * 10_000)
|
||||
assert len(long) < 120
|
||||
assert long.endswith("chars)")
|
||||
|
||||
|
||||
def _warnings(caplog: pytest.LogCaptureFixture) -> list[str]:
|
||||
return [r.getMessage() for r in caplog.records if r.levelno == logging.WARNING]
|
||||
|
||||
|
||||
def test_warn_once_stays_bounded_across_repeated_passes(caplog: pytest.LogCaptureFixture) -> None:
|
||||
log = logging.getLogger("headroom.test_log_safety.bounded")
|
||||
guard = WarnOnce(limit=64, what="unwritable ledgers")
|
||||
|
||||
with caplog.at_level(logging.INFO, logger=log.name):
|
||||
for _ in range(3):
|
||||
for path in range(65):
|
||||
if guard.first(path, log):
|
||||
log.warning("ledger %d failed", path)
|
||||
|
||||
messages = _warnings(caplog)
|
||||
assert len(messages) == 65 # 64 keys plus one overflow notice, not 65 per pass
|
||||
assert messages[-1] == (
|
||||
"More than 64 warnings about unwritable ledgers this window; further ones are not logged "
|
||||
"at WARNING "
|
||||
"for the next 60 minutes"
|
||||
)
|
||||
|
||||
|
||||
def test_warn_once_keeps_the_key_while_warnings_are_disabled() -> None:
|
||||
log = logging.getLogger("headroom.test_log_safety.level")
|
||||
guard = WarnOnce(limit=4, what="failures")
|
||||
log.setLevel(logging.ERROR)
|
||||
try:
|
||||
assert guard.first("k", log) is False
|
||||
log.setLevel(logging.INFO)
|
||||
assert guard.first("k", log) is True
|
||||
assert guard.first("k", log) is False
|
||||
finally:
|
||||
log.setLevel(logging.NOTSET)
|
||||
|
||||
|
||||
def test_forget_rearms_a_key() -> None:
|
||||
log = logging.getLogger("headroom.test_log_safety.forget")
|
||||
guard = WarnOnce(limit=4, what="failures")
|
||||
|
||||
assert guard.first("path", log) is True
|
||||
guard.forget("path")
|
||||
assert guard.first("path", log) is True
|
||||
|
||||
|
||||
def test_raise_from_none_hides_the_suppressed_context(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
try:
|
||||
try:
|
||||
raise KeyError(CANARY)
|
||||
except KeyError:
|
||||
raise ValueError("bad") from None
|
||||
except ValueError as exc:
|
||||
text = describe_exception(exc)
|
||||
|
||||
assert "KeyError" not in text
|
||||
|
||||
|
||||
def test_a_cause_cycle_is_described_once() -> None:
|
||||
first = ValueError(CANARY)
|
||||
second = KeyError(CANARY)
|
||||
first.__cause__ = second
|
||||
second.__cause__ = first
|
||||
|
||||
assert describe_exception(first) == "ValueError; caused by KeyError"
|
||||
|
||||
|
||||
def test_forgetting_after_overflow_does_not_reopen_the_cap(
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
log = logging.getLogger("headroom.test_log_safety.reopen")
|
||||
guard = WarnOnce(limit=2, what="failures")
|
||||
|
||||
with caplog.at_level(logging.INFO, logger=log.name):
|
||||
assert guard.first("a", log) and guard.first("b", log)
|
||||
assert guard.first("c", log) is False # overflow notice
|
||||
for cycle in range(5):
|
||||
guard.forget("a")
|
||||
assert guard.first(f"new-{cycle}", log) is False
|
||||
|
||||
assert len(_warnings(caplog)) == 1
|
||||
|
||||
|
||||
def test_warn_once_starts_over_after_the_window(
|
||||
caplog: pytest.LogCaptureFixture, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
log = logging.getLogger("headroom.test_log_safety.window")
|
||||
guard = WarnOnce(limit=2, what="hook failures", window_seconds=600)
|
||||
clock = [1000.0]
|
||||
monkeypatch.setattr(log_safety.time, "monotonic", lambda: clock[0])
|
||||
|
||||
results = []
|
||||
with caplog.at_level(logging.INFO, logger=log.name):
|
||||
steps = (("a", 0), ("b", 0), ("c", 0), ("a", 300), ("d", 299), ("a", 1), ("new-hook", 0))
|
||||
for key, advance in steps:
|
||||
clock[0] += advance
|
||||
results.append(guard.first(key, log))
|
||||
|
||||
# a and b warn and c is the overflow notice; inside the window a stays quiet
|
||||
# and d is suppressed; once the window ends, the still-recurring a and a
|
||||
# newly broken key both warn again.
|
||||
assert results == [True, True, False, False, False, True, True]
|
||||
assert len(_warnings(caplog)) == 1
|
||||
|
||||
|
||||
def test_warn_once_needs_a_positive_window() -> None:
|
||||
with pytest.raises(ValueError, match="window_seconds must be positive"):
|
||||
WarnOnce(limit=1, what="failures", window_seconds=0)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("digits", [19, 10_001])
|
||||
def test_out_of_range_errno_never_breaks_the_description(digits: int) -> None:
|
||||
"""An errno too large to print (Python caps int-to-text at 4300 digits) is not printed."""
|
||||
exc = OSError()
|
||||
exc.errno = 10 ** (digits - 1) # computed, not a literal, so collection never formats it
|
||||
|
||||
assert describe_exception(exc) == "OSError [Errno out of range]"
|
||||
|
||||
|
||||
def test_negative_dns_error_codes_are_kept() -> None:
|
||||
import socket
|
||||
|
||||
assert describe_exception(socket.gaierror(-2, "Name or service not known")) == (
|
||||
"gaierror [Errno -2]"
|
||||
)
|
||||
|
||||
|
||||
def test_overflow_notice_states_the_time_left_in_the_window(
|
||||
caplog: pytest.LogCaptureFixture, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
log = logging.getLogger("headroom.test_log_safety.remaining")
|
||||
guard = WarnOnce(limit=1, what="failures", window_seconds=3600)
|
||||
clock = [0.0]
|
||||
monkeypatch.setattr(log_safety.time, "monotonic", lambda: clock[0])
|
||||
|
||||
with caplog.at_level(logging.INFO, logger=log.name):
|
||||
guard.first("a", log)
|
||||
clock[0] = 3000.0 # ten minutes left in the window
|
||||
guard.first("b", log)
|
||||
|
||||
assert _warnings(caplog)[-1].endswith("for the next 10 minutes")
|
||||
|
||||
|
||||
def test_forget_cycles_cannot_exceed_the_window_budget(caplog: pytest.LogCaptureFixture) -> None:
|
||||
log = logging.getLogger("headroom.test_log_safety.forget_budget")
|
||||
guard = WarnOnce(limit=3, what="failures")
|
||||
|
||||
with caplog.at_level(logging.INFO, logger=log.name):
|
||||
issued = 0
|
||||
for cycle in range(50):
|
||||
if guard.first(f"path-{cycle}", log):
|
||||
issued += 1
|
||||
log.warning("path-%d failed", cycle)
|
||||
guard.forget(f"path-{cycle}")
|
||||
|
||||
assert issued == 3
|
||||
assert len(_warnings(caplog)) == 4 # three warnings plus one overflow notice
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"value",
|
||||
["sk-ant-abc123:x", "AKIAABCDEFGH:secret", "myuser:hunter2@db:5432", "AKIASECRET://host/x"],
|
||||
)
|
||||
def test_text_that_is_not_a_url_never_reaches_the_log(value: str) -> None:
|
||||
text = redact_url(value)
|
||||
for secret in ("sk-ant", "abc123", "AKIA", "akia", "hunter2", "myuser", "SECRET", "secret"):
|
||||
assert secret not in text
|
||||
|
||||
|
||||
@pytest.mark.parametrize("host", ["a
b", "a\x0bb\x0cc", "a\x85b"])
|
||||
def test_hosts_with_line_breaks_are_not_logged(host: str) -> None:
|
||||
assert redact_url(f"http://{host}/") == "http://<host>/"
|
||||
|
||||
|
||||
def test_class_names_cannot_forge_a_log_line(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
forged = type("Err\nWARNING forged sk-QUAL", (Exception,), {})
|
||||
|
||||
text = describe_exception(forged("x"))
|
||||
|
||||
assert "\n" not in text and "sk-QUAL" not in text
|
||||
assert text.startswith("<exception>")
|
||||
|
||||
|
||||
def test_odd_errno_and_module_objects_never_break_the_description(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
|
||||
class FormatsItself(int):
|
||||
def __format__(self, spec: str) -> str:
|
||||
return "sk-ERRNO-CANARY"
|
||||
|
||||
exc = OSError()
|
||||
exc.errno = FormatsItself(2)
|
||||
assert "CANARY" not in describe_exception(exc)
|
||||
|
||||
class RaisingGet(dict): # type: ignore[type-arg]
|
||||
def get(self, *args: object) -> object:
|
||||
raise RuntimeError("get")
|
||||
|
||||
namespace = RaisingGet()
|
||||
exec(compile("def boom():\n raise ValueError('x')\n", "<x>", "exec"), namespace)
|
||||
with pytest.raises(ValueError) as caught:
|
||||
namespace["boom"]()
|
||||
assert " at " not in describe_exception(caught.value)
|
||||
|
||||
lazy = types.ModuleType("lazy_sk_mod")
|
||||
|
||||
def lazy_getattr(attr: str) -> str:
|
||||
return "sk-FILE-CANARY.py"
|
||||
|
||||
lazy.__getattr__ = lazy_getattr # type: ignore[method-assign]
|
||||
monkeypatch.setitem(sys.modules, "lazy_sk_mod", lazy)
|
||||
exec(compile("def boom():\n raise ValueError('x')\n", "<x>", "exec"), vars(lazy))
|
||||
with pytest.raises(ValueError) as caught:
|
||||
lazy.boom()
|
||||
assert "CANARY" not in describe_exception(caught.value)
|
||||
|
||||
|
||||
def test_safe_id_never_raises() -> None:
|
||||
class BadRepr:
|
||||
def __repr__(self) -> str:
|
||||
raise RuntimeError("repr")
|
||||
|
||||
huge = 10 ** int("5000")
|
||||
assert safe_id(huge) == "<int of 16610 bits>"
|
||||
assert safe_id(BadRepr()) == f"<{BadRepr.__qualname__}>"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("window", [math.inf, math.nan, 0, -1])
|
||||
def test_warn_once_needs_a_finite_positive_window(window: float) -> None:
|
||||
with pytest.raises(ValueError, match="window_seconds must be positive and finite"):
|
||||
WarnOnce(limit=1, what="failures", window_seconds=window)
|
||||
|
||||
|
||||
def test_headroom_frames_are_named_by_module_path() -> None:
|
||||
with pytest.raises(ValueError) as caught:
|
||||
WarnOnce(limit=0, what="failures")
|
||||
|
||||
assert describe_exception(caught.value) == (
|
||||
f"ValueError at headroom/log_safety.py:{caught.traceback[-1].lineno + 1} in __init__"
|
||||
)
|
||||
|
||||
|
||||
def test_library_frames_are_skipped_but_the_headroom_caller_is_kept() -> None:
|
||||
with pytest.raises(json.JSONDecodeError) as caught:
|
||||
json.loads("{")
|
||||
|
||||
assert describe_exception(caught.value) == "JSONDecodeError"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("filename", "module_name"),
|
||||
[
|
||||
("sk-FILE-CANARY\nforged", None),
|
||||
(os.__file__, "os"),
|
||||
(log_safety.__file__, "headroom.log_safety"),
|
||||
],
|
||||
ids=["forged-name", "claims-stdlib-file", "claims-headroom-module"],
|
||||
)
|
||||
def test_runtime_compiled_frames_are_not_logged(
|
||||
monkeypatch: pytest.MonkeyPatch, filename: str, module_name: str | None
|
||||
) -> None:
|
||||
"""Runtime code may claim a real file and module name; it is still not named."""
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
code = compile("def sk_FUNC_CANARY():\n raise ValueError('x')\n", filename, "exec")
|
||||
namespace: dict[str, object] = {} if module_name is None else {"__name__": module_name}
|
||||
exec(code, namespace)
|
||||
with pytest.raises(ValueError) as caught:
|
||||
namespace["sk_FUNC_CANARY"]() # type: ignore[operator]
|
||||
|
||||
assert describe_exception(caught.value) == "ValueError"
|
||||
|
||||
|
||||
def test_code_execd_into_a_headroom_module_is_not_named(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
namespace = vars(log_safety)
|
||||
code = compile("def sk_GEN_CANARY():\n raise ValueError('x')\n", "<generated>", "exec")
|
||||
exec(code, namespace)
|
||||
monkeypatch.delitem(namespace, "sk_GEN_CANARY")
|
||||
function = code.co_consts[0]
|
||||
generated = types.FunctionType(function, namespace)
|
||||
|
||||
with pytest.raises(ValueError) as caught:
|
||||
generated()
|
||||
|
||||
assert describe_exception(caught.value) == "ValueError"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"module_name", ["headroom.x\nWARNING forged", "headroom.sk-live-CANARY", "headroomx"]
|
||||
)
|
||||
def test_odd_module_names_are_not_named(monkeypatch: pytest.MonkeyPatch, module_name: str) -> None:
|
||||
module = types.ModuleType(module_name)
|
||||
module.__file__ = "probe.py"
|
||||
monkeypatch.setitem(sys.modules, module_name, module)
|
||||
exec(compile("def boom():\n raise ValueError('x')\n", "probe.py", "exec"), vars(module))
|
||||
|
||||
with pytest.raises(ValueError) as caught:
|
||||
module.boom()
|
||||
|
||||
assert describe_exception(caught.value) == "ValueError"
|
||||
|
||||
|
||||
def test_zip_and_sourceless_headroom_modules_keep_their_locations(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
source = "def boom():\n raise ValueError('x')\n"
|
||||
with zipfile.ZipFile(tmp_path / "bundle.zip", "w") as bundle:
|
||||
bundle.writestr("_zip_probe.py", source)
|
||||
(tmp_path / "_pyc_probe.py").write_text(source)
|
||||
py_compile.compile(str(tmp_path / "_pyc_probe.py"), cfile=str(tmp_path / "_pyc_probe.pyc"))
|
||||
(tmp_path / "_pyc_probe.py").unlink()
|
||||
|
||||
zip_spec = zipimport.zipimporter(str(tmp_path / "bundle.zip")).find_spec("headroom._zip_probe")
|
||||
pyc_loader = importlib.machinery.SourcelessFileLoader(
|
||||
"headroom._pyc_probe", str(tmp_path / "_pyc_probe.pyc")
|
||||
)
|
||||
pyc_spec = importlib.util.spec_from_loader("headroom._pyc_probe", pyc_loader)
|
||||
texts = []
|
||||
for spec in (zip_spec, pyc_spec):
|
||||
assert spec is not None and spec.loader is not None
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
monkeypatch.setitem(sys.modules, spec.name, module)
|
||||
spec.loader.exec_module(module)
|
||||
with pytest.raises(ValueError) as caught:
|
||||
module.boom()
|
||||
texts.append(describe_exception(caught.value))
|
||||
|
||||
assert texts == [
|
||||
"ValueError at headroom/_zip_probe.py:2 in boom",
|
||||
"ValueError at headroom/_pyc_probe.py:2 in boom",
|
||||
]
|
||||
|
||||
|
||||
def test_deep_tracebacks_keep_only_the_innermost_headroom_frames() -> None:
|
||||
def recurse(depth: int) -> None:
|
||||
if depth == 0:
|
||||
WarnOnce(limit=0, what="failures")
|
||||
recurse(depth - 1)
|
||||
|
||||
with pytest.raises(ValueError) as caught:
|
||||
recurse(500)
|
||||
|
||||
assert describe_exception(caught.value).count(" in ") == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("url", "expected"),
|
||||
[
|
||||
("file:///var/lib/headroom/ledger.db", "file:///<path>"),
|
||||
("sqlite:///tmp/x.db", "sqlite:///<path>"),
|
||||
("unix:///run/headroom.sock", "unix:///<path>"),
|
||||
("https://sk_live_CANARY@/v1", "https://<host>/<path>"),
|
||||
("https://sk_live_CANARY/v1", "https://<host>/<path>"),
|
||||
("https://abcDEF+ghi=/v1", "https://<host>/<path>"),
|
||||
("http://model_gateway:8000/v1", "http://<host>:8000/<path>"),
|
||||
("https://bücher.example/v1", "https://xn--bcher-kva.example/<path>"),
|
||||
("http://[fe80::1%25sk-ZONE-CANARY]:8787/", "http://[fe80::1]:8787/"),
|
||||
("https://upstream.example/file:///etc", "https://upstream.example/<path>"),
|
||||
("sk-key-CANARY:file:///x", "<unparseable url>"),
|
||||
("https://xn--bcher-kva.example:8443/v1", "https://xn--bcher-kva.example:8443/<path>"),
|
||||
("http://10.0.0.7:8787", "http://10.0.0.7:8787"),
|
||||
],
|
||||
)
|
||||
def test_redact_url_hosts_and_hostless_schemes(
|
||||
url: str, expected: str, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
assert redact_url(url) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize("error", [RuntimeError, LookupError, OSError, OverflowError, KeyError])
|
||||
def test_safe_id_never_runs_the_callers_repr(error: type[Exception]) -> None:
|
||||
class RaisingStr(str):
|
||||
def __repr__(self) -> str:
|
||||
raise error("repr")
|
||||
|
||||
assert safe_id(RaisingStr("model-x")) == "'model-x'"
|
||||
|
||||
|
||||
def test_broken_exception_attributes_never_break_the_description(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
|
||||
class Broken(OSError):
|
||||
@property # type: ignore[override]
|
||||
def __cause__(self) -> BaseException:
|
||||
raise LookupError("cause")
|
||||
|
||||
@property # type: ignore[override]
|
||||
def __traceback__(self) -> None:
|
||||
raise RuntimeError("traceback")
|
||||
|
||||
@property # type: ignore[override]
|
||||
def errno(self) -> int:
|
||||
raise OverflowError("errno")
|
||||
|
||||
error = Broken(2, "x")
|
||||
error.__context__ = KeyError("k")
|
||||
|
||||
assert describe_exception(error) == (
|
||||
f"{Broken.__qualname__} [Errno 2] {os.strerror(2)}; caused by KeyError"
|
||||
)
|
||||
|
||||
|
||||
def test_hostless_urls_need_the_triple_slash_right_after_the_scheme() -> None:
|
||||
assert redact_url("file:sk-CANARY-x://y") == "<unparseable url>"
|
||||
assert redact_url("file:///var/x") == "file:///<path>"
|
||||
|
||||
|
||||
def test_int_ids_respect_the_process_digit_limit(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setattr(sys, "get_int_max_str_digits", lambda: 640)
|
||||
big = 10 ** int("700")
|
||||
|
||||
assert safe_id(big) == f"<int of {big.bit_length()} bits>"
|
||||
assert safe_id(12345) == "12345"
|
||||
|
||||
|
||||
def test_safe_id_ignores_a_faked_class() -> None:
|
||||
class Pretender:
|
||||
@property # type: ignore[misc]
|
||||
def __class__(self) -> type:
|
||||
return str
|
||||
|
||||
assert safe_id(Pretender()) == f"<{Pretender.__qualname__}>"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name", ["Err\rforged", "sk-live-CANARY", "Err\u2028x"])
|
||||
def test_class_names_must_be_plain_identifiers(name: str) -> None:
|
||||
cls = type(name, (Exception,), {})
|
||||
|
||||
assert describe_exception(cls("x")) == "<exception>"
|
||||
assert safe_id(cls("x")) == "<object>"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("url", [" \x01file:///var/x", "fi\tle:///var/x", "file:/\n//var/x"])
|
||||
def test_hostless_check_reads_the_url_the_way_urlsplit_does(url: str) -> None:
|
||||
assert redact_url(url) == "file:///<path>"
|
||||
|
||||
|
||||
def test_int_ids_work_without_a_digit_limit_getter(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.delattr(sys, "get_int_max_str_digits")
|
||||
|
||||
assert safe_id(12345) == "12345"
|
||||
@@ -0,0 +1,59 @@
|
||||
"""A failure applying saved settings is reported without the settings' values.
|
||||
|
||||
Saved settings can hold credentials, and an error raised while applying them can
|
||||
quote a value, so the warning names the file and the failure's type and code
|
||||
location, never its message, at any level.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
import pytest
|
||||
|
||||
CREDENTIAL_CANARY = "sk-ant-SETTINGS-CANARY-4d18"
|
||||
|
||||
|
||||
def test_settings_apply_failure_warns_without_credential(monkeypatch, tmp_path) -> None:
|
||||
from headroom import settings_store
|
||||
from headroom.proxy.server import ProxyConfig, create_app
|
||||
|
||||
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path / "ws"))
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
|
||||
def _apply_fails(_values: object) -> None:
|
||||
raise ValueError(f"cannot apply ANTHROPIC_API_KEY={CREDENTIAL_CANARY}")
|
||||
|
||||
monkeypatch.setattr(settings_store, "apply_to_environ", _apply_fails)
|
||||
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Capture(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
proxy_logger = logging.getLogger("headroom.proxy")
|
||||
handler = _Capture(level=logging.DEBUG)
|
||||
old_level = proxy_logger.level
|
||||
proxy_logger.addHandler(handler)
|
||||
proxy_logger.setLevel(logging.DEBUG)
|
||||
try:
|
||||
create_app(ProxyConfig(port=0))
|
||||
finally:
|
||||
proxy_logger.removeHandler(handler)
|
||||
proxy_logger.setLevel(old_level)
|
||||
|
||||
warnings = [r for r in records if "Could not apply saved settings" in r.getMessage()]
|
||||
assert len(warnings) == 1
|
||||
assert warnings[0].levelno == logging.WARNING
|
||||
assert "ValueError" in warnings[0].getMessage()
|
||||
assert "settings.json" in warnings[0].getMessage()
|
||||
formatter = logging.Formatter()
|
||||
for record in records:
|
||||
assert CREDENTIAL_CANARY not in formatter.format(record)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolated_home(monkeypatch, tmp_path):
|
||||
monkeypatch.setenv("HOME", str(tmp_path / "home"))
|
||||
monkeypatch.setenv("CODEX_HOME", str(tmp_path / "codex"))
|
||||
@@ -0,0 +1,105 @@
|
||||
"""Savings pricing fallbacks are debuggable without flooding or leaking into the log.
|
||||
|
||||
An unpriced model is expected (local models, gateway aliases) and the estimators
|
||||
run on every request, so it is logged once per model. Unexpected pricing errors
|
||||
are logged by exception type and code location only: their text can carry a
|
||||
credential (a provider error quoting an API key) and must not reach any level.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
import pytest
|
||||
|
||||
import headroom.proxy.savings_tracker as st
|
||||
from headroom.log_safety import WarnOnce
|
||||
|
||||
CREDENTIAL_CANARY = "sk-ant-CREDENTIAL-CANARY-91c2"
|
||||
|
||||
|
||||
class _Capture(logging.Handler):
|
||||
def __init__(self) -> None:
|
||||
super().__init__(level=logging.DEBUG)
|
||||
self.records: list[logging.LogRecord] = []
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
self.records.append(record)
|
||||
|
||||
|
||||
class _CredentialEchoingPrice:
|
||||
"""A price value whose conversion fails with a message quoting a key."""
|
||||
|
||||
def __float__(self) -> float:
|
||||
raise ValueError(f"bad price from provider config api_key={CREDENTIAL_CANARY}")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def captured(monkeypatch: pytest.MonkeyPatch):
|
||||
class _FakeLiteLLM:
|
||||
model_cost = {"weird-priced-model": {"input_cost_per_token": _CredentialEchoingPrice()}}
|
||||
|
||||
@staticmethod
|
||||
def cost_per_token(**_kwargs):
|
||||
raise Exception(f"This model isn't mapped yet. key={CREDENTIAL_CANARY}")
|
||||
|
||||
monkeypatch.setattr(st, "_get_litellm_module", lambda: _FakeLiteLLM())
|
||||
monkeypatch.setattr(st, "_unpriced_models_logged", WarnOnce(256, "test unpriced models"))
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
st._resolve_litellm_model.cache_clear()
|
||||
handler = _Capture()
|
||||
old_level = st.logger.level
|
||||
st.logger.addHandler(handler)
|
||||
st.logger.setLevel(logging.DEBUG)
|
||||
try:
|
||||
yield handler.records
|
||||
finally:
|
||||
st.logger.removeHandler(handler)
|
||||
st.logger.setLevel(old_level)
|
||||
st._resolve_litellm_model.cache_clear()
|
||||
|
||||
|
||||
def _assert_no_canary(records: list[logging.LogRecord]) -> None:
|
||||
formatter = logging.Formatter()
|
||||
for record in records:
|
||||
assert CREDENTIAL_CANARY not in formatter.format(record)
|
||||
|
||||
|
||||
def test_unpriced_model_logs_once(captured) -> None:
|
||||
for _ in range(3):
|
||||
assert st._estimate_compression_savings_usd("local-llama-7b", 1000) == pytest.approx(
|
||||
1000 * st.DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN
|
||||
)
|
||||
st._estimate_output_savings_usd("local-llama-7b", 1000)
|
||||
st._estimate_input_cost_usd("local-llama-7b", 1000)
|
||||
|
||||
unpriced = [r for r in captured if "No litellm" in r.getMessage()]
|
||||
assert len(unpriced) == 1
|
||||
assert "model='local-llama-7b'" in unpriced[0].getMessage()
|
||||
assert all(r.exc_info is None for r in captured)
|
||||
# The litellm probe error quoted a key; it never reaches the log.
|
||||
_assert_no_canary(captured)
|
||||
|
||||
|
||||
def test_credential_bearing_pricing_error_is_described_not_quoted(captured) -> None:
|
||||
assert st._estimate_compression_savings_usd("weird-priced-model", 1000) == pytest.approx(
|
||||
1000 * st.DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN
|
||||
)
|
||||
|
||||
failures = [
|
||||
r for r in captured if "fallback rate for model='weird-priced-model'" in r.getMessage()
|
||||
]
|
||||
assert len(failures) == 1
|
||||
assert "ValueError" in failures[0].getMessage()
|
||||
assert not any("No litellm" in r.getMessage() for r in captured)
|
||||
_assert_no_canary(captured)
|
||||
|
||||
|
||||
def test_unpriced_model_not_marked_when_debug_disabled(captured) -> None:
|
||||
st.logger.setLevel(logging.INFO)
|
||||
st._estimate_compression_savings_usd("quiet-model", 10)
|
||||
st.logger.setLevel(logging.DEBUG)
|
||||
st._estimate_compression_savings_usd("quiet-model", 10)
|
||||
assert [
|
||||
r for r in captured if "quiet-model" in r.getMessage() and "No litellm" in r.getMessage()
|
||||
]
|
||||
@@ -140,3 +140,95 @@ def test_tool_schema_compaction_saves_real_tokens_not_just_bytes() -> None:
|
||||
assert tokens_saved == tool_before - tool_after
|
||||
assert original_tokens - optimized_tokens == tokens_saved
|
||||
assert headline_tokens_saved(tokens_saved, {}) == tokens_saved
|
||||
|
||||
|
||||
SCHEMA_CANARY = "PRIVATE-SCHEMA-CANARY-7f3a"
|
||||
|
||||
|
||||
def _capture_policy_logs(monkeypatch):
|
||||
import logging
|
||||
|
||||
from headroom.log_safety import WarnOnce
|
||||
from headroom.proxy import tool_schema_savings_policy as policy
|
||||
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Capture(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
handler = _Capture(level=logging.DEBUG)
|
||||
policy.logger.addHandler(handler)
|
||||
old_level = policy.logger.level
|
||||
policy.logger.setLevel(logging.DEBUG)
|
||||
monkeypatch.setattr(
|
||||
policy, "_reconcile_warned", WarnOnce(32, "test reconcile types"), raising=False
|
||||
)
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
return policy, records, handler, old_level
|
||||
|
||||
|
||||
def test_reconcile_failure_keeps_private_schema_out_of_every_log_record(monkeypatch) -> None:
|
||||
import logging
|
||||
|
||||
policy, records, handler, old_level = _capture_policy_logs(monkeypatch)
|
||||
|
||||
tool = {"name": "Read", "defer_loading": True, "description": SCHEMA_CANARY}
|
||||
|
||||
def _tokenizer_echoes_schema(tools: object) -> int:
|
||||
# A tokenizer error that quotes the schema it choked on.
|
||||
raise ValueError(f"cannot tokenize {tools!r}")
|
||||
|
||||
tags = {"tool_search_deferred_tokens": 100}
|
||||
booking = (100, {"Read": dict(tool)}, None)
|
||||
try:
|
||||
policy.reconcile_deferred_tokens(tags, booking, [dict(tool)], _tokenizer_echoes_schema)
|
||||
policy.reconcile_deferred_tokens(tags, booking, [dict(tool)], _tokenizer_echoes_schema)
|
||||
finally:
|
||||
policy.logger.removeHandler(handler)
|
||||
policy.logger.setLevel(old_level)
|
||||
|
||||
# The booked credit is left as it was.
|
||||
assert tags["tool_search_deferred_tokens"] == 100
|
||||
|
||||
levels = [r.levelno for r in records]
|
||||
assert levels.count(logging.WARNING) == 1, "warns once per exception type"
|
||||
assert levels.count(logging.DEBUG) == 2, "every failure is recorded at debug"
|
||||
formatter = logging.Formatter()
|
||||
for record in records:
|
||||
rendered = formatter.format(record)
|
||||
assert SCHEMA_CANARY not in rendered
|
||||
assert "ValueError" in rendered
|
||||
assert "100 booked tokens" in records[0].getMessage()
|
||||
|
||||
|
||||
def test_reconcile_warns_again_for_a_new_exception_type(monkeypatch) -> None:
|
||||
import logging
|
||||
|
||||
policy, records, handler, old_level = _capture_policy_logs(monkeypatch)
|
||||
tool = {"name": "Read", "defer_loading": True, "description": "x"}
|
||||
booking = (100, {"Read": dict(tool)}, None)
|
||||
|
||||
def _runtime(tools: object) -> int:
|
||||
raise RuntimeError("boom")
|
||||
|
||||
def _value(tools: object) -> int:
|
||||
raise ValueError("boom")
|
||||
|
||||
try:
|
||||
# A healthy reconcile logs nothing.
|
||||
policy.reconcile_deferred_tokens(
|
||||
{"tool_search_deferred_tokens": 100}, booking, [dict(tool)], lambda t: 10
|
||||
)
|
||||
assert records == []
|
||||
for counter in (_runtime, _runtime, _value):
|
||||
policy.reconcile_deferred_tokens(
|
||||
{"tool_search_deferred_tokens": 100}, booking, [dict(tool)], counter
|
||||
)
|
||||
finally:
|
||||
policy.logger.removeHandler(handler)
|
||||
policy.logger.setLevel(old_level)
|
||||
|
||||
warnings = [r.getMessage() for r in records if r.levelno == logging.WARNING]
|
||||
assert len(warnings) == 2
|
||||
assert "RuntimeError" in warnings[0] and "ValueError" in warnings[1]
|
||||
|
||||
Reference in New Issue
Block a user