fix(savings,ccr): log the ledger and MCP failures that are swallowed silently (#4105)

## Description

`headroom/savings_ledger.py` had no logger. Its broad `except Exception`
blocks turned every failure into `False`, `[]` or `None`, with no trace.
So a ledger that could not be written (permissions, read-only home, disk
full) or read made `headroom savings` report zero, and nothing said why.

The write case was the worse one. If every append fails, the file is
never created, and `headroom savings` shows zero as if no compression
had happened.

The MCP server had the same pattern in four places:
- the proxy-retrieve fallback (`except Exception: pass`);
- the shared session-stats file I/O;
- the proxy `/stats` fetch;
- the client-name lookup (`except Exception: pass`).

A bug in `_retrieve_via_proxy` showed up to the user only as "not
found".

This PR makes those failures visible and changes nothing else:
- **Ledger append failure:** the first failure on each ledger path logs
a WARNING with the path, the error and what to do. Repeats on the same
path drop to DEBUG. A successful append re-arms its path, so a new
failure episode warns again. The budget is `WarnOnce(64)` from #4138:
after 64 distinct paths in an hour it logs one overflow WARNING and
stays at DEBUG for the rest of that hour, so many unwritable ledgers
cannot warn again on each cycle. Each hour the budget starts over for
all paths, so a failure that keeps recurring warns again once an hour.
- **Unreadable ledger:** logs a WARNING. Only the `headroom savings` CLI
reads it.
- **Everything else:** DEBUG, because these paths must never raise or
run per request.
- **Client-name lookup:** the catch is narrowed to `(LookupError,
AttributeError)`, the two exceptions it expects outside a request, and
the failure is logged.

No line at any level carries content, tokens or credentials:
- exception detail comes from `describe_exception()` (types, code
locations, and errno/strerror for an `OSError`), never `exc_info=True`
or the exception text;
- proxy URLs go through `redact_url()`, which keeps only scheme, host
and port (the path, userinfo, query and fragment are dropped unless
`HEADROOM_DEBUG_DUMP=full`);
- the model and hash go through `safe_id()`.

Refs #4136

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/savings_ledger.py`: add a module logger, plus:
- `record_savings_event` append failure: the first failure per ledger
path is `WARNING savings ledger: append to <path> failed: <error>;
savings will not be recorded. Make the directory writable or set
HEADROOM_SAVINGS_EVENTS_PATH.`; repeats log the same detail at DEBUG.
- The guard is `WarnOnce(64, "unwritable savings ledgers")`, which never
clears.
- A successful append calls `forget(path)`, so that path warns again on
its next failure.
    - `<error>` is `describe_exception(exc)`.
- `_read_events`: `WARNING savings ledger: could not read <path>:
<error>. Check that it is a readable file, or set
HEADROOM_SAVINGS_EVENTS_PATH.`
- DEBUG with `describe_exception` on compaction failure, on the pricing
import and pricing call (with `safe_id(model)`), and on the
pricing-basis comparison in `_weakest`.
  - Return values are unchanged.
- `headroom/ccr/mcp_server.py`, using the existing `headroom.ccr.mcp`
logger, all at DEBUG with `describe_exception`:
- the proxy-retrieve fallback, with `safe_id(hash)` and
`redact_url(proxy_url)`;
  - the shared-stats append, read and prune, with the file path;
  - the proxy `/stats` fetch, with `redact_url(proxy_url)`;
  - the durable-savings recording failure in `_handle_compress`;
- `_current_client`: `except Exception: pass` becomes `except
(LookupError, AttributeError):` with a log line;
  - the startup INFO line now logs `redact_url(proxy_url)`.
- `tests/test_savings_ledger.py`: five regression tests, which attach a
handler to `headroom.savings_ledger` directly.
- An unreadable ledger logs exactly one WARNING naming the path. A
readable ledger logs none.
- Failed appends warn once per ledger path: two failures on one path and
one on another give two WARNINGs.
  - A failure after a successful append warns again.
- A non-`OSError` failure's text is absent from every formatted record
at WARNING and at DEBUG.
- 65 unwritable ledger paths over 3 cycles give 65 WARNINGs in total: 64
per path plus one overflow.
- `tests/test_ccr_mcp_server.py`: `_fetch_full_proxy_stats` against a
local endpoint returning malformed JSON, with userinfo and a path
carrying a plain and a %-encoded token in the proxy URL, at DEBUG and at
WARNING. It still returns None, and neither the password nor the path
canaries appear in any formatted record. At DEBUG the record still names
the host and `JSONDecodeError`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# the three new or tightened regressions against the previous head (3e0054b9 + #4138): all fail
FAILED tests/test_savings_ledger.py::test_append_warning_names_only_the_type_of_a_non_os_error   (payload in the DEBUG traceback)
FAILED tests/test_savings_ledger.py::test_many_unwritable_ledgers_do_not_warn_on_every_cycle     (assert 195 == 65)
FAILED tests/test_ccr_mcp_server.py::test_proxy_stats_failure_keeps_url_credentials_out_of_logs[10]  (canary in the DEBUG record)
# on this branch:
$ pytest -q -p no:cacheprovider tests/test_savings_ledger.py tests/test_ccr_mcp_server.py tests/test_log_safety.py
73 passed in 18.91s
$ uvx ruff@0.16.8 check / format --check (the four touched files)
All checks passed! / 4 files already formatted
$ python -m mypy headroom/savings_ledger.py headroom/ccr/mcp_server.py headroom/log_safety.py --ignore-missing-imports
Success: no issues found in 3 source files
```

`tests/test_savings_provider_matrix.py` errors at collection on
origin/main as well: the installed litellm price table no longer lists
its candidate models. That is pre-existing and unrelated to this PR.

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, this branch checked out in a
worktree, repo .venv; temp HOME and HEADROOM_WORKSPACE_DIR, offline.
- Exact command / steps: (1) The shared smoke harness `smoke.sh
<worktree>`: it starts a real proxy from this checkout against a local
fake upstream and sends one Anthropic and one OpenAI request with a
compressible tool result. (2) A direct script on this PR's own paths,
with root logging at DEBUG: it calls the real
`HeadroomMCPServer._fetch_full_proxy_stats` with proxy URL
`http://user:pw-canary@127.0.0.1:<port>?token=t` against a local
endpoint that returns `{not json`, then calls the real
`record_savings_event` on 65 directory paths (each append fails) for 3
cycles.
- Observed result: Both smoke requests return 200 and `/stats` shows
savings, with only the expected offline-mode warnings in proxy.log. The
script's fetch returns None, its DEBUG line has no userinfo or query,
and the 195 failed appends give 64 per-path WARNINGs plus one overflow
WARNING.
- Not tested: The MCP debug lines under a live MCP client.

```text
$ bash smoke.sh <worktree>
branch: cleanup/core-misc-swallowed-failures @ 9711990f
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE_TMP/ws/logs/proxy-65299.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   2 headroom.transforms.kompress_compressor - WARNING - Kompress: 'onnx/kompress-int8-wo.onnx' for chopratejas/kompress-v2-base is not cached and HEADROOM_OFFLINE forbids fetching it (HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.). Compression falls back to the non-ML path.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE_TMP)

$ HOME=$TMP/home HEADROOM_WORKSPACE_DIR=$TMP/ws python - $TMP/ledgers < rt4105.py
DEBUG headroom.ccr.mcp MCP stats: proxy /stats fetch failed via http://127.0.0.1:49173?<redacted>: JSONDecodeError at decoder.py:354 in raw_decode <- decoder.py:338 in decode <- __init__.py:346 in loads
stats fetch returned: None
WARNING headroom.savings_ledger savings ledger: append to $TMP/ledgers/l0 failed: IsADirectoryError [Errno 21] Is a directory at headroom/savings_ledger.py:435 in record_savings_event; savings will not be recorded. Make the directory writable or set HEADROOM_SAVINGS_EVENTS_PATH.
... (64 per-path 'append to' WARNINGs in total over 3 cycles of 65 paths)
WARNING headroom.savings_ledger More than 64 distinct unwritable savings ledgers; further ones are logged at debug only
occurrences of the password canary in the output: 0
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; logging only.
- Stable/default behavior changed: No. Return values and control flow
are unchanged. `_current_client` now lets exceptions other than
`LookupError` and `AttributeError` propagate; they would be bugs.
- Kill switch / disable path: Standard logging levels. The append
WARNING fires once per ledger path per failure episode, for at most 64
paths per hour, then one overflow WARNING; after that it stays at DEBUG
until the hour is up, when the budget starts over for all paths. A
long-lived process with persistent failures can therefore log up to 65
append WARNINGs per hour. A successful append re-arms that path. The
read WARNING fires once per `headroom savings` run.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Depends on #4138; its files (`headroom/log_safety.py`, its tests, and
the `_debug_dump.py` change) show in this diff until it merges.

The lazy-export guard test, the Copilot error-message change and the
package docstring URL from the first draft are out of scope here, so
they were removed from this PR. In touched files, two handlers are left
alone. `call_tool`'s `logger.error(f"Tool {name} failed: {e}",
exc_info=True)` is pre-existing and out of scope for this PR. And
`HeadroomMCPServer._probe_proxy_unreachable` falls back from
`response.json()` to `response.text`, which is a deliberate parse
fallback, not a hidden failure.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
2026-10-12 18:59:54 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent f3d240bcae
commit 97680e7782
4 changed files with 292 additions and 20 deletions
+42 -14
View File
@@ -36,6 +36,7 @@ from typing import TYPE_CHECKING, Any
from headroom import paths as _paths
from headroom import savings_ledger
from headroom.cache.compression_store import format_retrieval_miss_detail
from headroom.log_safety import describe_exception, redact_url, safe_id
from headroom.telemetry import session as telemetry_session
if TYPE_CHECKING:
@@ -226,8 +227,13 @@ def _append_shared_event(event: dict[str, Any]) -> None:
f.write(line)
if _HAS_FCNTL:
fcntl.flock(f, fcntl.LOCK_UN)
except Exception:
pass # Never break compression because of stats
except Exception as exc:
# Never break compression because of stats
logger.debug(
"MCP session stats: append to %s failed: %s",
SHARED_STATS_FILE,
describe_exception(exc),
)
def _read_shared_events(window_seconds: int = SESSION_WINDOW_SECONDS) -> list[dict[str, Any]]:
@@ -265,10 +271,18 @@ def _read_shared_events(window_seconds: int = SESSION_WINDOW_SECONDS) -> list[di
f.writelines(keep_lines)
if _HAS_FCNTL:
fcntl.flock(f, fcntl.LOCK_UN)
except Exception:
pass
except Exception:
pass
except Exception as exc:
logger.debug(
"MCP session stats: pruning %s failed: %s",
SHARED_STATS_FILE,
describe_exception(exc),
)
except Exception as exc:
logger.debug(
"MCP session stats: reading %s failed: %s",
SHARED_STATS_FILE,
describe_exception(exc),
)
return events
@@ -529,8 +543,14 @@ class HeadroomMCPServer:
result["source"] = "proxy"
self._stats.record_retrieval(hash_key)
return result
except Exception:
pass # Proxy unavailable, that's fine
except Exception as exc:
# Proxy unavailable is expected; log so a bug here is not silent.
logger.debug(
"MCP retrieve: proxy fallback failed for hash=%s via %s: %s",
safe_id(hash_key),
redact_url(self.proxy_url),
describe_exception(exc),
)
if expired_entry_status:
ttl_seconds = expired_entry_status.get(
@@ -795,8 +815,8 @@ class HeadroomMCPServer:
# restarts. Best-effort: never let savings bookkeeping break the tool.
try:
self._record_savings(result)
except Exception:
logger.debug("durable savings recording failed", exc_info=True)
except Exception as exc:
logger.debug("durable savings recording failed: %s", describe_exception(exc))
proxy_status = await self._probe_proxy_unreachable()
if proxy_status:
@@ -845,8 +865,11 @@ class HeadroomMCPServer:
name = getattr(info, "name", None)
if name:
return str(name)
except Exception:
pass
except (LookupError, AttributeError) as exc:
# Outside a request there is no request context; fall back to "unknown".
logger.debug(
"MCP client name unavailable; reporting 'unknown': %s", describe_exception(exc)
)
return "unknown"
async def _handle_retrieve(self, arguments: dict[str, Any]) -> list[TextContent]:
@@ -945,7 +968,12 @@ class HeadroomMCPServer:
return None
result: dict[str, Any] = response.json()
return result
except Exception:
except Exception as exc:
logger.debug(
"MCP stats: proxy /stats fetch failed via %s: %s",
redact_url(self.proxy_url),
describe_exception(exc),
)
return None
@staticmethod
@@ -1113,7 +1141,7 @@ class HeadroomMCPServer:
watchdog forces shutdown once we are reparented.
"""
async with stdio_server() as (read_stream, write_stream):
logger.info(f"Headroom MCP Server starting (proxy: {self.proxy_url})")
logger.info("Headroom MCP Server starting (proxy: %s)", redact_url(self.proxy_url))
serve_task = asyncio.create_task(
self.server.run(
read_stream,
+44 -6
View File
@@ -41,6 +41,7 @@ blending two meanings.
from __future__ import annotations
import json
import logging
import os
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
@@ -48,6 +49,7 @@ from pathlib import Path
from typing import Any
from headroom import paths as _paths
from headroom.log_safety import WarnOnce, describe_exception, safe_id
# Reuse the proxy tracker's pricing + normalization so MCP and proxy events
# bucket models identically and price them through one implementation.
@@ -68,6 +70,26 @@ try:
except ImportError:
_HAS_FCNTL = False
logger = logging.getLogger(__name__)
# The first failed append to each ledger path is a WARNING so an unwritable
# ledger is visible; repeats drop to DEBUG so a persistent failure cannot flood
# logs. A successful append re-arms the path.
_append_warnings = WarnOnce(64, "unwritable savings ledgers")
def _warn_append_failure(target: Path, exc: BaseException) -> None:
if _append_warnings.first(str(target), logger):
logger.warning(
"savings ledger: append to %s failed: %s; savings will not be recorded. "
"Make the directory writable or set HEADROOM_SAVINGS_EVENTS_PATH.",
target,
describe_exception(exc),
)
else:
logger.debug("savings ledger: append to %s failed: %s", target, describe_exception(exc))
SCHEMA_VERSION = 2
UNKNOWN = "unknown"
@@ -201,7 +223,8 @@ def _price_event(
"""
try:
from headroom.pricing.counterfactual import CacheMix, Region, price_savings
except Exception: # pragma: no cover - defensive
except Exception as exc: # pragma: no cover - defensive
logger.debug("savings ledger: pricing module failed to import: %s", describe_exception(exc))
return None
cache = event.get("cache") or {}
@@ -244,7 +267,12 @@ def _price_event(
total += priced.usd
total_list += priced.usd_list
bases.append(priced.basis)
except Exception: # pragma: no cover - defensive
except Exception as exc: # pragma: no cover - defensive
logger.debug(
"savings ledger: pricing failed for model=%s: %s",
safe_id(model),
describe_exception(exc),
)
return None
if not bases:
@@ -412,9 +440,11 @@ def record_savings_event(
finally:
if _HAS_FCNTL and fcntl is not None:
fcntl.flock(handle, fcntl.LOCK_UN)
except Exception:
except Exception as exc:
_warn_append_failure(target, exc)
return False
_append_warnings.forget(str(target))
_maybe_compact(target)
return True
@@ -454,7 +484,13 @@ def _read_events(
finally:
if _HAS_FCNTL and fcntl is not None:
fcntl.flock(handle, fcntl.LOCK_UN)
except Exception:
except Exception as exc:
logger.warning(
"savings ledger: could not read %s: %s. Check that it is a readable file, "
"or set HEADROOM_SAVINGS_EVENTS_PATH.",
target,
describe_exception(exc),
)
return []
return events
@@ -553,7 +589,8 @@ def _weakest(existing: str | None, incoming: str | None) -> str | None:
from headroom.pricing.counterfactual import weakest_basis
return weakest_basis(existing, incoming)
except Exception: # pragma: no cover - defensive
except Exception as exc: # pragma: no cover - defensive
logger.debug("savings ledger: could not compare pricing bases: %s", describe_exception(exc))
return existing
@@ -758,7 +795,8 @@ def _maybe_compact(target: Path) -> None:
finally:
if _HAS_FCNTL and fcntl is not None:
fcntl.flock(handle, fcntl.LOCK_UN)
except Exception:
except Exception as exc:
logger.debug("savings ledger: compaction of %s failed: %s", target, describe_exception(exc))
return
+68
View File
@@ -2,7 +2,10 @@ from __future__ import annotations
import asyncio
import contextlib
import http.server
import json
import logging
import threading
import pytest
@@ -537,3 +540,68 @@ def test_run_stdio_reaps_process_on_parent_death(monkeypatch) -> None:
assert excinfo.value.args[0] == 0
assert cleaned["done"] is True
class _RecordList(logging.Handler):
def __init__(self) -> None:
super().__init__(level=logging.DEBUG)
self.records: list[logging.LogRecord] = []
def emit(self, record: logging.LogRecord) -> None:
self.records.append(record)
@pytest.mark.parametrize("level", [logging.DEBUG, logging.WARNING])
def test_proxy_stats_failure_keeps_url_credentials_out_of_logs(level) -> None:
canary = "pw-canary-7c1e"
# A path segment can carry a token too, plain or %-encoded (%50 is "P").
path_canaries = ("PLAIN_PATH_CANARY", "%50RIVATE_PATH_CANARY", "PRIVATE_PATH_CANARY")
class _MalformedJson(http.server.BaseHTTPRequestHandler):
def do_GET(self) -> None: # noqa: N802 - stdlib handler name
body = b"{not json"
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *_args: object) -> None:
pass
httpd = http.server.ThreadingHTTPServer(("127.0.0.1", 0), _MalformedJson)
thread = threading.Thread(target=httpd.serve_forever, daemon=True)
thread.start()
# Attach directly: caplog misses `headroom.*` loggers once logging is configured.
logger = logging.getLogger("headroom.ccr.mcp")
handler = _RecordList()
previous = logger.level
logger.addHandler(handler)
logger.setLevel(level)
try:
port = httpd.server_address[1]
server = mcp_server.HeadroomMCPServer(
proxy_url=f"http://user:{canary}@127.0.0.1:{port}/bot{path_canaries[0]}/{path_canaries[1]}",
check_proxy=True,
)
async def fetch() -> object:
try:
return await server._fetch_full_proxy_stats()
finally:
if server._http_client is not None:
await server._http_client.aclose()
assert asyncio.run(fetch()) is None
finally:
logger.removeHandler(handler)
logger.setLevel(previous)
httpd.shutdown()
httpd.server_close()
formatted = [logging.Formatter().format(record) for record in handler.records]
for secret in (canary, *path_canaries):
assert all(secret not in line for line in formatted)
if level == logging.DEBUG:
# The failure is still diagnosable: the redacted URL and the exception type.
assert any(f"127.0.0.1:{port}" in line and "JSONDecodeError" in line for line in formatted)
+138
View File
@@ -3,11 +3,13 @@
from __future__ import annotations
import json
import logging
from datetime import datetime, timedelta, timezone
import pytest
from headroom import savings_ledger as L
from headroom.log_safety import WarnOnce
from tests._pricing_models import anthropic_pricing_model
MODEL = anthropic_pricing_model()
@@ -528,3 +530,139 @@ def test_zero_saving_request_keeps_its_place_in_the_new_input_denominator(monkey
assert window["new_input_tokens"] == 10_100
assert window["new_input_savings_percent"] == pytest.approx(1.0, abs=0.05)
assert window["savings_percent"] == pytest.approx(50.0)
# --------------------------------------------------------------------------- #
# failures are logged, not swallowed
# --------------------------------------------------------------------------- #
class _ListHandler(logging.Handler):
def __init__(self) -> None:
super().__init__(level=logging.DEBUG)
self.records: list[logging.LogRecord] = []
def emit(self, record: logging.LogRecord) -> None:
self.records.append(record)
@pytest.fixture
def ledger_records():
# Attach directly: caplog misses `headroom.*` loggers once the proxy has
# configured logging earlier in the session.
logger = logging.getLogger("headroom.savings_ledger")
handler = _ListHandler()
previous = logger.level
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
yield handler.records
logger.removeHandler(handler)
logger.setLevel(previous)
@pytest.fixture
def fresh_append_warnings(monkeypatch):
monkeypatch.setattr(L, "_append_warnings", WarnOnce(64, "unwritable savings ledgers"))
def _warnings(records: list[logging.LogRecord]) -> list[str]:
return [r.getMessage() for r in records if r.levelno == logging.WARNING]
def test_unreadable_ledger_warns_instead_of_reporting_zero_silently(tmp_path, ledger_records):
readable = tmp_path / "savings_events.jsonl"
assert L.record_savings_event(
tokens_before=1000, tokens_after=400, model=None, client="c", path=readable
)
assert L.aggregate_savings(path=readable).lifetime["tokens_saved"] == 600
assert _warnings(ledger_records) == []
# A directory where the ledger file should be: unreadable.
report = L.aggregate_savings(path=tmp_path)
assert report.lifetime["tokens_saved"] == 0
warnings = _warnings(ledger_records)
assert len(warnings) == 1
assert "could not read" in warnings[0]
assert str(tmp_path) in warnings[0]
def test_failed_append_warns_once_per_ledger_path(tmp_path, ledger_records, fresh_append_warnings):
first = tmp_path / "a"
second = tmp_path / "b"
first.mkdir()
second.mkdir()
# A directory where the ledger file should be: every append fails.
assert not L.record_savings_event(tokens_before=1000, tokens_after=400, path=first)
assert not L.record_savings_event(tokens_before=1000, tokens_after=400, path=first)
assert not L.record_savings_event(tokens_before=1000, tokens_after=400, path=second)
warnings = _warnings(ledger_records)
# One WARNING per path; the repeat on `first` goes to DEBUG only.
assert len(warnings) == 2
assert str(first) in warnings[0] and "append to" in warnings[0]
assert str(second) in warnings[1]
assert any(r.levelno == logging.DEBUG and "append to" in r.getMessage() for r in ledger_records)
def test_successful_append_rearms_the_warning(tmp_path, ledger_records, fresh_append_warnings):
ledger = tmp_path / "savings_events.jsonl"
ledger.mkdir()
assert not L.record_savings_event(tokens_before=1000, tokens_after=400, path=ledger)
ledger.rmdir()
assert L.record_savings_event(tokens_before=1000, tokens_after=400, path=ledger)
ledger.unlink()
ledger.mkdir()
assert not L.record_savings_event(tokens_before=1000, tokens_after=400, path=ledger)
# The failure after a recovery is a new episode and warns again.
assert len(_warnings(ledger_records)) == 2
def test_append_warning_names_only_the_type_of_a_non_os_error(
monkeypatch, tmp_path, ledger_records, fresh_append_warnings
):
def fail(*_args, **_kwargs):
raise TypeError("secret-looking event payload")
# Shadow open() inside the module only, so nothing else is affected.
monkeypatch.setattr(L, "open", fail, raising=False)
assert not L.record_savings_event(
tokens_before=1000, tokens_after=400, path=tmp_path / "savings_events.jsonl"
)
warnings = _warnings(ledger_records)
assert len(warnings) == 1
assert "TypeError" in warnings[0]
formatted = [logging.Formatter().format(record) for record in ledger_records]
assert all("secret-looking" not in line for line in formatted)
# The repeat goes to DEBUG, and DEBUG carries no payload either.
assert not L.record_savings_event(
tokens_before=1000, tokens_after=400, path=tmp_path / "savings_events.jsonl"
)
formatted = [logging.Formatter().format(record) for record in ledger_records]
assert any(r.levelno == logging.DEBUG and "TypeError" in r.getMessage() for r in ledger_records)
assert all("secret-looking" not in line for line in formatted)
def test_many_unwritable_ledgers_do_not_warn_on_every_cycle(
tmp_path, ledger_records, fresh_append_warnings
):
# A directory where each ledger file should be: every append fails.
ledgers = [tmp_path / f"ledger-{i}" for i in range(65)]
for ledger in ledgers:
ledger.mkdir()
for _cycle in range(3):
for ledger in ledgers:
assert not L.record_savings_event(tokens_before=1000, tokens_after=400, path=ledger)
warnings = _warnings(ledger_records)
# 64 per-path warnings plus one overflow notice, over all three cycles.
assert len(warnings) == 65
assert sum("append to" in w for w in warnings) == 64
assert "More than 64 warnings about unwritable savings ledgers" in warnings[-1]