mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(output-savings): rewrite a pre-rule ledger once instead of warning on every /stats poll (#4196)
## Description `SavingsLedger.from_dict` logs "output-savings ledger predates shaped-only recording; dropping N treatment and M control strata …" whenever it loads a ledger that has no `shaped_only` marker. `/stats` calls `get_recorder().estimate()`, which reloads the ledger from disk through `_reload_baseline_locked()` on every poll. The dropped arms are only written back by a periodic flush, and a flush needs 25 shaped recordings. With `proxy_output_shaper` disabled, or very little shaped traffic, that flush never happens. The legacy file then stays on disk for good and the warning repeats on every `/stats` poll. In stateless mode the file is never written at all, so the same thing happens. On a proxy whose `/stats` endpoint is polled frequently this came to 1,436–8,869 warnings per day. This PR persists the drop once and reports it once: - `SavingsRecorder.__init__` saves the cleaned ledger as soon as it loads a pre-rule one. The cleaned ledger has `shaped_only: true`, empty arms and the **baseline kept**. That is exactly what the warning already says happens. It does not write in stateless mode. - `_reload_baseline_locked()` only reads the baseline, so it loads with `warn_legacy=False`. The constructor has already reported the drop, and this stops the repeat in stateless mode, where the file can't be rewritten. The warning text is unchanged, and the drop and baseline-retention logic is unchanged. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/output_savings.py`: `SavingsLedger` gains `dropped_legacy_arms`, set by `from_dict` when it drops pre-rule arms. The field is not compared or repr'd and is not serialized. - `SavingsLedger.from_dict` / `SavingsLedger.load` take a keyword-only `warn_legacy=True`. The default keeps current behaviour for every other caller. - `SavingsRecorder.__init__` persists a dropped-legacy ledger immediately through a new `_save_unless_stateless()`, which uses the same atomic `fsutil.write_text` path, honours `process_is_stateless()` and fails open on `OSError` like `_flush_locked`. - `SavingsRecorder._reload_baseline_locked` loads with `warn_legacy=False`. - `tests/test_output_savings.py`: two tests. (1) A pre-rule ledger gives exactly one warning across construction plus three `estimate()` calls, and is rewritten with `shaped_only`, empty arms and the original baseline. (2) In stateless mode there is exactly one warning and the file is byte-for-byte untouched. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_output_savings.py -k pre_rule -v tests/test_output_savings.py::TestLedgerPersistence::test_pre_rule_ledger_warns_once_and_is_rewritten PASSED [ 50%] tests/test_output_savings.py::TestLedgerPersistence::test_stateless_pre_rule_ledger_warns_once_and_is_left_alone PASSED [100%] ================= 2 passed, 75 deselected, 1 warning in 0.67s ================== # both new tests fail on main (warning count is 4, file not rewritten) $ pytest tests/test_output_savings.py tests/test_output_savings_cli.py tests/test_stateless_writers.py FAILED tests/test_stateless_writers.py::test_memory_disabled_under_stateless =================== 1 failed, 92 passed, 1 warning in 2.94s ==================== # that failure is pre-existing and order-dependent: identical on unpatched main # (1 failed, 90 passed) and it passes when run alone $ ruff check headroom/proxy/output_savings.py tests/test_output_savings.py All checks passed! $ ruff format --check headroom/proxy/output_savings.py tests/test_output_savings.py 2 files already formatted $ mypy headroom/proxy/output_savings.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS, Python 3.13. A persistent `headroom proxy` (0.40.0) with `HEADROOM_DISABLE_FEATURES=proxy_output_shaper`, whose `/stats` endpoint is polled frequently by monitoring clients. The ledger was written before shaped-only recording (51 treatment and 20 control strata, 8 baseline strata). - Exact command / steps: (1) Counted the warning in the proxy's archived error log per day. (2) Built a legacy ledger with that live ledger's real baseline and the same strata counts, then ran a `SavingsRecorder` on it and called `estimate()` 100 times, simulating 100 `/stats` polls. This was run once on `main` and once on this branch, each on its own copy of the file. - Observed result: (1) 8,698 / 8,869 / 5,323 / 1,436 / 2,760 / 3,140 warnings on six consecutive days. (2) `main`: 101 warnings in 100 polls, and the file still had no `shaped_only` with all 51/20 arms. This branch: **1 warning**, and the file now has `shaped_only=true`, 0/0 arms and all 8 baseline strata. - Not tested: a full proxy restart on this branch in production. The fix was exercised through `SavingsRecorder`, the same object `/stats` uses. Multi-process proxies (several workers sharing one ledger) were not tested; the first worker to start rewrites the file, and the others then load a current ledger. ## Runtime Rollout Safety - Rollout-managed feature(s): none. This is a persistence and logging fix inside output-savings accounting, and `proxy_output_shaper` gating is untouched. - Minimum rollout channel: n/a (no rollout-gated behaviour added). - Stable/default behavior changed: yes, narrowly. A pre-rule `output_savings.json` is rewritten once at startup, rather than at the first periodic flush (which may never come). The rewritten content equals what that flush would have written: baseline kept, legacy arms dropped. Every ledger already carrying `shaped_only` loads exactly as before. - Kill switch / disable path: `HEADROOM_STATELESS=1` (no write). Otherwise none is needed, since the rewrite only happens once, for a file the current code already discards on load. - Unsafe override required: no. - Qualification impact: none. The estimates are unchanged because the dropped arms were already ignored in memory; only the on-disk file and the log volume change. - Rollback path: revert the commit. The rewritten file is a valid current-format ledger that older and newer builds both load. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - No documentation change: the warning and the drop semantics are unchanged, and only their repetition is fixed. - Related, but different: #3395 is about the validity of this ledger's data. This PR only stops the repeated reload warning. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4196" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -755,9 +755,7 @@ def _run_verbosity(
|
||||
# Seed the savings baseline: replace baseline, preserve any live
|
||||
# treatment/control already accumulated.
|
||||
ledger_path = ws / "output_savings.json"
|
||||
ledger = SavingsLedger.load(ledger_path)
|
||||
ledger.baseline = aggregated
|
||||
ledger.save(ledger_path)
|
||||
SavingsLedger.store_baseline(ledger_path, aggregated)
|
||||
click.echo(f"\n [WROTE] {ws / 'verbosity.json'} (level {best_profile.level})")
|
||||
click.echo(
|
||||
f" [WROTE] {ledger_path} (baseline: {aggregated.total_samples} samples, "
|
||||
|
||||
@@ -41,6 +41,8 @@ from __future__ import annotations
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from typing import Any
|
||||
|
||||
@@ -392,6 +394,52 @@ def _measured_supersedes(measured: SavingsEstimate, labelled_requests: int) -> b
|
||||
return _measured_covers(measured, labelled_requests)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def ledger_lock(path: Any) -> Iterator[None]:
|
||||
"""Serialize read-modify-write of the ledger file across processes.
|
||||
|
||||
The lock lives in a sidecar file because ``save`` replaces the ledger's
|
||||
inode. Same platform split as ``cli.wrap._locked_file``.
|
||||
"""
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
lock_path = Path(str(path) + ".lock")
|
||||
lock_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(lock_path, "a+b") as handle:
|
||||
if sys.platform == "win32":
|
||||
import msvcrt
|
||||
import os
|
||||
import time
|
||||
|
||||
# msvcrt.locking locks bytes from the current position, so lock a
|
||||
# real byte at offset 0. LK_LOCK raises after ~10 one-second
|
||||
# retries, so loop on the non-blocking form instead.
|
||||
if os.fstat(handle.fileno()).st_size == 0:
|
||||
handle.write(b"0")
|
||||
handle.flush()
|
||||
handle.seek(0)
|
||||
while True:
|
||||
try:
|
||||
msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1)
|
||||
break
|
||||
except OSError:
|
||||
time.sleep(0.05)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
handle.seek(0)
|
||||
msvcrt.locking(handle.fileno(), msvcrt.LK_UNLCK, 1)
|
||||
else:
|
||||
import fcntl
|
||||
|
||||
fcntl.flock(handle, fcntl.LOCK_EX)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
fcntl.flock(handle, fcntl.LOCK_UN)
|
||||
|
||||
|
||||
@dataclass
|
||||
class SavingsLedger:
|
||||
"""Accumulates shaped (treatment) and unshaped (control) observations and
|
||||
@@ -406,6 +454,9 @@ class SavingsLedger:
|
||||
baseline: BaselineModel = field(default_factory=BaselineModel)
|
||||
treatment: dict[str, _Accum] = field(default_factory=dict)
|
||||
control: dict[str, _Accum] = field(default_factory=dict)
|
||||
# Set by ``from_dict`` when it dropped pre-rule arms, so the owner can
|
||||
# persist the drop instead of re-dropping (and re-warning) on every load.
|
||||
dropped_legacy_arms: bool = field(default=False, compare=False, repr=False)
|
||||
|
||||
# ---- recording -------------------------------------------------------
|
||||
|
||||
@@ -616,7 +667,7 @@ class SavingsLedger:
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, d: dict[str, Any]) -> SavingsLedger:
|
||||
def from_dict(cls, d: dict[str, Any], *, warn_legacy: bool = True) -> SavingsLedger:
|
||||
ledger = cls(baseline=BaselineModel.from_dict(d.get("baseline") or {}))
|
||||
if not d.get("shaped_only"):
|
||||
# Pre-rule arms cannot be told apart from shaped ones entry by
|
||||
@@ -625,6 +676,8 @@ class SavingsLedger:
|
||||
# baseline is kept: it is learned from pre-shaper history, costs a
|
||||
# `learn --verbosity` run to rebuild, and was never the poisoned part.
|
||||
if d.get("treatment") or d.get("control"):
|
||||
ledger.dropped_legacy_arms = True
|
||||
if ledger.dropped_legacy_arms and warn_legacy:
|
||||
logger.warning(
|
||||
"output-savings ledger predates shaped-only recording; "
|
||||
"dropping %d treatment and %d control strata and "
|
||||
@@ -651,14 +704,14 @@ class SavingsLedger:
|
||||
fsutil.write_text(p, json.dumps(self.to_dict(), separators=(",", ":")))
|
||||
|
||||
@classmethod
|
||||
def load(cls, path: Any) -> SavingsLedger:
|
||||
def load(cls, path: Any, *, warn_legacy: bool = True) -> SavingsLedger:
|
||||
from pathlib import Path
|
||||
|
||||
p = Path(path)
|
||||
if not p.exists():
|
||||
return cls()
|
||||
try:
|
||||
return cls.from_dict(json.loads(p.read_text()))
|
||||
return cls.from_dict(json.loads(p.read_text()), warn_legacy=warn_legacy)
|
||||
except (json.JSONDecodeError, ValueError, OSError) as exc:
|
||||
# Fail open (empty ledger), but surface the loss — silently
|
||||
# swallowing a corrupt file made lost history indistinguishable
|
||||
@@ -666,6 +719,14 @@ class SavingsLedger:
|
||||
logger.warning("output-savings ledger %s unreadable, starting empty: %s", p, exc)
|
||||
return cls()
|
||||
|
||||
@classmethod
|
||||
def store_baseline(cls, path: Any, baseline: BaselineModel) -> None:
|
||||
"""Replace the on-disk baseline, keeping the live arms already there."""
|
||||
with ledger_lock(path):
|
||||
ledger = cls.load(path)
|
||||
ledger.baseline = baseline
|
||||
ledger.save(path)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Live recording — rides the existing ``transforms_applied`` label channel so
|
||||
@@ -692,6 +753,11 @@ class SavingsRecorder:
|
||||
self._ledger = SavingsLedger.load(self._path)
|
||||
self._flush_every = flush_every
|
||||
self._since_flush = 0
|
||||
if self._ledger.dropped_legacy_arms:
|
||||
# Persist the drop now. With the shaper off nothing ever records,
|
||||
# so waiting for a periodic flush would leave the legacy file on
|
||||
# disk indefinitely.
|
||||
self._migrate_legacy()
|
||||
|
||||
def record_from_labels(self, labels: Any, output_tokens: int) -> bool:
|
||||
"""Record one outcome given its transforms_applied labels. Returns True
|
||||
@@ -779,7 +845,10 @@ class SavingsRecorder:
|
||||
same number of samples still takes effect, and the empty-disk guard keeps
|
||||
a truncated file from wiping a baseline we already hold."""
|
||||
try:
|
||||
disk = SavingsLedger.load(self._path)
|
||||
# Only the baseline is read here, and the constructor already
|
||||
# reported any legacy arms. Warning again would repeat on every
|
||||
# /stats poll whenever the file can't be rewritten (stateless).
|
||||
disk = SavingsLedger.load(self._path, warn_legacy=False)
|
||||
except OSError:
|
||||
return
|
||||
if disk.baseline.total_samples == 0:
|
||||
@@ -787,6 +856,21 @@ class SavingsRecorder:
|
||||
if disk.baseline.to_dict() != self._ledger.baseline.to_dict():
|
||||
self._ledger.baseline = disk.baseline
|
||||
|
||||
def _migrate_legacy(self) -> None:
|
||||
from ..paths import process_is_stateless
|
||||
|
||||
if process_is_stateless():
|
||||
return
|
||||
try:
|
||||
with ledger_lock(self._path):
|
||||
# Re-read under the lock: another writer may have replaced the
|
||||
# file since the first load, and its copy is the newer one.
|
||||
self._ledger = SavingsLedger.load(self._path, warn_legacy=False)
|
||||
if self._ledger.dropped_legacy_arms:
|
||||
self._ledger.save(self._path)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def _flush_locked(self) -> None:
|
||||
from ..paths import process_is_stateless
|
||||
|
||||
@@ -795,8 +879,9 @@ class SavingsRecorder:
|
||||
self._since_flush = 0
|
||||
return
|
||||
try:
|
||||
self._reload_baseline_locked()
|
||||
self._ledger.save(self._path)
|
||||
with ledger_lock(self._path):
|
||||
self._reload_baseline_locked()
|
||||
self._ledger.save(self._path)
|
||||
self._since_flush = 0
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
@@ -17,6 +17,7 @@ from headroom.proxy.output_savings import (
|
||||
conversation_label,
|
||||
echo_ratio,
|
||||
input_bucket,
|
||||
ledger_lock,
|
||||
model_family,
|
||||
stratum_key,
|
||||
stratum_label,
|
||||
@@ -685,6 +686,181 @@ class TestLedgerPersistence:
|
||||
ledger = SavingsLedger.load(p)
|
||||
assert ledger.baseline.total_samples == 0
|
||||
|
||||
@staticmethod
|
||||
def _pre_rule_ledger(path):
|
||||
"""A ledger written before shaped-only recording: arms, no marker."""
|
||||
baseline = BaselineModel()
|
||||
baseline.observe("opus|a|s|tools", 1000)
|
||||
legacy = {
|
||||
"baseline": baseline.to_dict(),
|
||||
"treatment": {"opus|a|s|tools": {"n": 3, "sum": 2400.0, "sumsq": 1920000.0}},
|
||||
"control": {},
|
||||
}
|
||||
path.write_text(json.dumps(legacy))
|
||||
return legacy
|
||||
|
||||
@staticmethod
|
||||
def _legacy_warnings(caplog):
|
||||
return [r for r in caplog.records if "predates shaped-only" in r.getMessage()]
|
||||
|
||||
def test_pre_rule_ledger_warns_once_and_is_rewritten(self, tmp_path, caplog):
|
||||
"""/stats reloads the ledger on every poll. With the shaper disabled
|
||||
nothing ever flushes, so the legacy file used to survive and warn on
|
||||
every poll forever."""
|
||||
import logging
|
||||
|
||||
path = tmp_path / "output_savings.json"
|
||||
legacy = self._pre_rule_ledger(path)
|
||||
|
||||
with caplog.at_level(logging.WARNING):
|
||||
recorder = SavingsRecorder(str(path))
|
||||
for _ in range(3):
|
||||
recorder.estimate()
|
||||
|
||||
assert len(self._legacy_warnings(caplog)) == 1
|
||||
on_disk = json.loads(path.read_text())
|
||||
assert on_disk["shaped_only"] is True
|
||||
assert on_disk["treatment"] == {}
|
||||
assert on_disk["baseline"] == legacy["baseline"]
|
||||
|
||||
def test_stateless_pre_rule_ledger_warns_once_and_is_left_alone(self, tmp_path, caplog):
|
||||
import logging
|
||||
|
||||
from headroom import paths
|
||||
|
||||
path = tmp_path / "output_savings.json"
|
||||
legacy = self._pre_rule_ledger(path)
|
||||
|
||||
paths.set_process_stateless(True)
|
||||
try:
|
||||
with caplog.at_level(logging.WARNING):
|
||||
recorder = SavingsRecorder(str(path))
|
||||
for _ in range(3):
|
||||
recorder.estimate()
|
||||
finally:
|
||||
paths.set_process_stateless(False)
|
||||
|
||||
assert len(self._legacy_warnings(caplog)) == 1
|
||||
assert json.loads(path.read_text()) == legacy
|
||||
assert not (tmp_path / "output_savings.json.lock").exists()
|
||||
|
||||
def test_migration_keeps_a_baseline_learned_after_the_first_load(self, tmp_path, monkeypatch):
|
||||
"""learn replaced the file between the recorder's first load and its
|
||||
migration write; the stale legacy baseline must not overwrite it."""
|
||||
path = tmp_path / "output_savings.json"
|
||||
self._pre_rule_ledger(path)
|
||||
learned = BaselineModel()
|
||||
for output_tokens in (400, 420, 380):
|
||||
learned.observe("opus|b|s|tools", output_tokens)
|
||||
|
||||
real_load = SavingsLedger.load.__func__
|
||||
loads = []
|
||||
|
||||
def load_then_learn(cls, p, **kwargs):
|
||||
ledger = real_load(cls, p, **kwargs)
|
||||
loads.append(p)
|
||||
if len(loads) == 1:
|
||||
SavingsLedger.store_baseline(path, learned)
|
||||
return ledger
|
||||
|
||||
monkeypatch.setattr(SavingsLedger, "load", classmethod(load_then_learn))
|
||||
recorder = SavingsRecorder(str(path))
|
||||
|
||||
on_disk = json.loads(path.read_text())
|
||||
assert on_disk["baseline"] == learned.to_dict()
|
||||
assert recorder._ledger.baseline.to_dict() == learned.to_dict()
|
||||
|
||||
def test_migration_holds_the_ledger_lock_while_writing(self, tmp_path, monkeypatch):
|
||||
import threading
|
||||
|
||||
path = tmp_path / "output_savings.json"
|
||||
self._pre_rule_ledger(path)
|
||||
real_save = SavingsLedger.save
|
||||
contended = []
|
||||
|
||||
def probing_save(self, p):
|
||||
acquired = threading.Event()
|
||||
|
||||
def other_writer():
|
||||
with ledger_lock(p):
|
||||
acquired.set()
|
||||
|
||||
probe = threading.Thread(target=other_writer)
|
||||
probe.start()
|
||||
contended.append(not acquired.wait(timeout=0.3))
|
||||
real_save(self, p)
|
||||
probe_threads.append(probe)
|
||||
|
||||
probe_threads: list[threading.Thread] = []
|
||||
monkeypatch.setattr(SavingsLedger, "save", probing_save)
|
||||
SavingsRecorder(str(path))
|
||||
for probe in probe_threads:
|
||||
probe.join(timeout=5)
|
||||
|
||||
assert contended == [True]
|
||||
assert json.loads(path.read_text())["shaped_only"] is True
|
||||
|
||||
def test_migration_keeps_a_baseline_learned_during_the_locked_reload(
|
||||
self, tmp_path, monkeypatch
|
||||
):
|
||||
"""learn writes while the migration holds its re-read snapshot; it
|
||||
must wait for the lock rather than be overwritten by that snapshot."""
|
||||
import threading
|
||||
|
||||
path = tmp_path / "output_savings.json"
|
||||
legacy = self._pre_rule_ledger(path)
|
||||
learned = BaselineModel()
|
||||
for output_tokens in (400, 420, 380):
|
||||
learned.observe("opus|b|s|tools", output_tokens)
|
||||
|
||||
real_load = SavingsLedger.load.__func__
|
||||
loads = []
|
||||
writers: list[threading.Thread] = []
|
||||
|
||||
def learn_during_second_load(cls, p, **kwargs):
|
||||
ledger = real_load(cls, p, **kwargs)
|
||||
loads.append(p)
|
||||
if len(loads) == 2:
|
||||
writer = threading.Thread(target=SavingsLedger.store_baseline, args=(path, learned))
|
||||
writer.start()
|
||||
writer.join(timeout=0.3)
|
||||
writers.append(writer)
|
||||
return ledger
|
||||
|
||||
monkeypatch.setattr(SavingsLedger, "load", classmethod(learn_during_second_load))
|
||||
recorder = SavingsRecorder(str(path))
|
||||
assert len(writers) == 1
|
||||
writers[0].join(timeout=5)
|
||||
assert not writers[0].is_alive()
|
||||
|
||||
on_disk = json.loads(path.read_text())
|
||||
assert on_disk["shaped_only"] is True
|
||||
assert on_disk["baseline"] == learned.to_dict()
|
||||
assert on_disk["baseline"] != legacy["baseline"]
|
||||
recorder.estimate()
|
||||
assert recorder._ledger.baseline.to_dict() == learned.to_dict()
|
||||
|
||||
def test_store_baseline_waits_for_the_ledger_lock(self, tmp_path):
|
||||
"""learn must not replace the file while another writer holds the lock
|
||||
between its re-read and its save."""
|
||||
import threading
|
||||
|
||||
path = tmp_path / "output_savings.json"
|
||||
legacy = self._pre_rule_ledger(path)
|
||||
learned = BaselineModel()
|
||||
learned.observe("opus|b|s|tools", 400)
|
||||
|
||||
writer = threading.Thread(target=SavingsLedger.store_baseline, args=(path, learned))
|
||||
with ledger_lock(path):
|
||||
writer.start()
|
||||
writer.join(timeout=0.3)
|
||||
assert writer.is_alive()
|
||||
assert json.loads(path.read_text()) == legacy
|
||||
writer.join(timeout=5)
|
||||
|
||||
assert not writer.is_alive()
|
||||
assert json.loads(path.read_text())["baseline"] == learned.to_dict()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# echo ratio (direct waste signal)
|
||||
@@ -769,6 +945,21 @@ class TestRecorderBaselineReload:
|
||||
# in-memory one over it.
|
||||
assert SavingsLedger.load(path).baseline.total_samples == 4
|
||||
|
||||
def test_store_baseline_keeps_live_arms(self, tmp_path):
|
||||
path = str(tmp_path / "output_savings.json")
|
||||
key = self._key()
|
||||
|
||||
recorder = SavingsRecorder(path, flush_every=1)
|
||||
recorder.record_from_labels([stratum_label("treatment", key), SHAPED], 200)
|
||||
|
||||
learned = BaselineModel()
|
||||
learned.observe(key, 400)
|
||||
SavingsLedger.store_baseline(path, learned)
|
||||
|
||||
on_disk = SavingsLedger.load(path)
|
||||
assert on_disk.baseline.to_dict() == learned.to_dict()
|
||||
assert on_disk.treatment[key].n == 1
|
||||
|
||||
def test_does_not_downgrade_to_empty_disk_baseline(self, tmp_path):
|
||||
path = str(tmp_path / "output_savings.json")
|
||||
key = self._key()
|
||||
|
||||
Reference in New Issue
Block a user