mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
A user running an overnight Claude Code session through the proxy on
Linux reported it consuming 100 GB of a 128 GB host (about 3,500
requests at ~92k input tokens each, `--log-messages` on). Two stacked
causes, both fixed here:
1. `RequestLogger` keeps message payloads on all 10,000 entries. With
`--log-messages`, every `RequestLog` in the `MAX_LOG_ENTRIES` deque
holds `request_messages`, `compressed_messages` and `response_content`:
two parsed copies of the whole conversation, ~1.9x their JSON size as
Python objects. The only reader of those payloads is
`/transformations/feed`, which is hard-capped at 100 entries, so the
other 9,900 are retained for nothing and grow linearly for the life of
the process.
2. The periodic allocator trim (#2820, #2879) defaults on only for
macOS. glibc retains freed pages the same way, and per-thread arenas on
a many-core host multiply the high-water mark.
Not changed: `MAX_LOG_ENTRIES` itself (`_build_stats_payload` reads the
light fields of all 10,000), and `get_recent_with_messages(n)` for `n >
100`, which now returns bodies only for the newest 100, matching what
the feed could ever serve.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/request_logger.py`: `MESSAGE_WINDOW = 100`; on every
append, null the three payload fields on the entry that just left the
window. Light fields stay on every entry, so `get_recent` and `/stats`
are unchanged.
- `headroom/proxy/models.py`, `headroom/proxy/server.py`:
`periodic_malloc_trim_enabled` defaults on for `darwin` and `linux` (was
`darwin` only). Platforms without a trim call already disable the task
themselves; `HEADROOM_MALLOC_TRIM=0` still opts out.
- `tests/test_proxy/test_request_logger.py`: window tests (payloads
nulled beyond the newest 100, light fields intact and ordered, feed path
fully populated, short logs untouched).
- `tests/test_malloc_tuning.py`: default-scope test asserts darwin and
linux on, win32 off.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py -q
22 passed in 1.10s
$ uvx ruff check headroom/proxy/request_logger.py headroom/proxy/models.py headroom/proxy/server.py tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py
All checks passed!
$ uvx ruff format --check <same files>
5 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/request_logger.py headroom/proxy/models.py
Success: no issues found
```
## Real Behavior Proof
- Environment: macOS 15.6 (Darwin 24.6.0), Python 3.12, this branch at
ba5f99d7, `uv run --frozen`.
- Exact command / steps: Logged 2,000 `RequestLog` entries through a
real `RequestLogger(log_full_messages=True)`, each carrying a
300-message, ~128 KB conversation in both `request_messages` and
`compressed_messages`, under `tracemalloc`; once with the fix
(`MESSAGE_WINDOW = 100`) and once with the window set to 10,000 to
reproduce the pre-fix retention.
- Observed result: fix: 41 MB live, 100 entries with bodies; pre-fix:
802 MB live, 2,000 entries with bodies. Same 2,000 entries remain in the
deque either way. Extrapolated to the reporter's 10,000-entry cap and
~1.5 MB per entry, the pre-fix retention alone is ~15 GB before
allocator bloat.
- Not tested: end-to-end RSS on a Linux host with the trim default
flipped (no Linux machine available; the trim path itself is covered by
the existing `test_malloc_tuning.py` suite and was measured on macOS in
#2820).
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, two defaults. (1) Request-log
entries older than the newest 100 no longer carry message payloads; only
`/transformations/feed` reads them and it is capped at 100, so no served
response changes. (2) The periodic allocator trim is now on by default
on glibc Linux, as it already was on macOS.
- Kill switch / disable path: `HEADROOM_MALLOC_TRIM=0` disables the
trim. The payload window has no switch; it changes no output.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert the PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from the Conventional Commit PR title (a CI guard
enforces this)
## Screenshots (if applicable)
N/A
## Additional Notes
Documentation: no user-facing docs describe `MAX_LOG_ENTRIES` or the
trim default, so nothing to update. Two commits, one per cause, in case
the maintainers want the Linux trim default separately.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
347 lines
13 KiB
Python
347 lines
13 KiB
Python
"""macOS libmalloc tuning: pre-main re-exec gating + periodic allocator trim (#2820)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
|
|
import pytest
|
|
|
|
import headroom.cli.proxy as proxy_cli
|
|
from headroom.proxy import malloc_trim
|
|
|
|
|
|
class _ExecCalled(Exception):
|
|
"""Sentinel so a fake execv can stop execution the way real execv would."""
|
|
|
|
|
|
def _fake_execv(recorder: dict):
|
|
def _execv(path, argv): # noqa: ANN001
|
|
recorder["path"] = path
|
|
recorder["argv"] = list(argv)
|
|
raise _ExecCalled
|
|
|
|
return _execv
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _clean_malloc_env(monkeypatch):
|
|
for var in (
|
|
"HEADROOM_MALLOC_TUNING",
|
|
"_HEADROOM_MALLOC_TUNED",
|
|
"MallocAggressiveMadvise",
|
|
"MallocLargeCache",
|
|
):
|
|
monkeypatch.delenv(var, raising=False)
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
# _reexec_with_malloc_tuning
|
|
# --------------------------------------------------------------------------- #
|
|
def test_reexec_noop_off_darwin(monkeypatch):
|
|
monkeypatch.setattr(proxy_cli.sys, "platform", "linux")
|
|
rec: dict = {}
|
|
monkeypatch.setattr(proxy_cli.os, "execv", _fake_execv(rec))
|
|
proxy_cli._reexec_with_malloc_tuning() # must not raise / exec
|
|
assert rec == {}
|
|
|
|
|
|
def test_reexec_respects_opt_out(monkeypatch):
|
|
monkeypatch.setattr(proxy_cli.sys, "platform", "darwin")
|
|
monkeypatch.setenv("HEADROOM_MALLOC_TUNING", "0")
|
|
rec: dict = {}
|
|
monkeypatch.setattr(proxy_cli.os, "execv", _fake_execv(rec))
|
|
proxy_cli._reexec_with_malloc_tuning()
|
|
assert rec == {}
|
|
|
|
|
|
def test_reexec_guard_prevents_loop(monkeypatch):
|
|
monkeypatch.setattr(proxy_cli.sys, "platform", "darwin")
|
|
monkeypatch.setenv("_HEADROOM_MALLOC_TUNED", "1")
|
|
rec: dict = {}
|
|
monkeypatch.setattr(proxy_cli.os, "execv", _fake_execv(rec))
|
|
proxy_cli._reexec_with_malloc_tuning()
|
|
assert rec == {}
|
|
|
|
|
|
def test_reexec_skips_when_operator_already_set_vars(monkeypatch):
|
|
monkeypatch.setattr(proxy_cli.sys, "platform", "darwin")
|
|
# A real CLI launch, like the sibling exec test below: the tuning path is
|
|
# only reachable when this process is the Headroom CLI entrypoint, and
|
|
# under pytest argv[0] is pytest's own.
|
|
monkeypatch.setattr(proxy_cli.sys, "argv", ["headroom", "proxy"])
|
|
monkeypatch.setenv("MallocAggressiveMadvise", "1")
|
|
monkeypatch.setenv("MallocLargeCache", "0")
|
|
rec: dict = {}
|
|
monkeypatch.setattr(proxy_cli.os, "execv", _fake_execv(rec))
|
|
proxy_cli._reexec_with_malloc_tuning()
|
|
# No re-exec (vars present), but the guard is still stamped.
|
|
assert rec == {}
|
|
assert proxy_cli.os.environ.get("_HEADROOM_MALLOC_TUNED") == "1"
|
|
|
|
|
|
def test_reexec_sets_vars_and_execs_once(monkeypatch):
|
|
monkeypatch.setattr(proxy_cli.sys, "platform", "darwin")
|
|
monkeypatch.setattr(proxy_cli.sys, "executable", "/usr/bin/python3")
|
|
monkeypatch.setattr(proxy_cli.sys, "argv", ["headroom", "proxy", "--port", "8787"])
|
|
rec: dict = {}
|
|
monkeypatch.setattr(proxy_cli.os, "execv", _fake_execv(rec))
|
|
|
|
with pytest.raises(_ExecCalled):
|
|
proxy_cli._reexec_with_malloc_tuning()
|
|
|
|
# The tuning knobs and the loop guard are exported to the replacement process.
|
|
assert proxy_cli.os.environ["MallocAggressiveMadvise"] == "1"
|
|
assert proxy_cli.os.environ["MallocLargeCache"] == "0"
|
|
assert proxy_cli.os.environ["_HEADROOM_MALLOC_TUNED"] == "1"
|
|
# Re-exec normalizes to `python -m headroom.cli <args>`, preserving the PID.
|
|
assert rec["path"] == "/usr/bin/python3"
|
|
assert rec["argv"] == ["/usr/bin/python3", "-m", "headroom.cli", "proxy", "--port", "8787"]
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
# malloc_trim.trim / trim_periodically
|
|
# --------------------------------------------------------------------------- #
|
|
def test_trim_calls_platform_fn(monkeypatch):
|
|
def fake_fn(ptr, size): # noqa: ANN001 (mac signature)
|
|
return 4096
|
|
|
|
monkeypatch.setattr(malloc_trim, "_resolve", lambda: ("darwin", fake_fn))
|
|
assert malloc_trim.trim() == 4096
|
|
|
|
|
|
def test_trim_never_runs_python_gc(monkeypatch):
|
|
# The periodic trim must NOT trigger a full cyclic collection: gc.collect()
|
|
# holds the GIL for a whole-heap traversal, which would stall the event loop
|
|
# even though the C purge itself is dispatched off-thread. Only the
|
|
# GIL-releasing allocator C call may run.
|
|
import gc
|
|
|
|
ran: list[str] = []
|
|
monkeypatch.setattr(gc, "collect", lambda *a, **k: ran.append("gc") or 0)
|
|
monkeypatch.setattr(malloc_trim, "_resolve", lambda: ("glibc", lambda _size: 0))
|
|
malloc_trim.trim()
|
|
assert ran == []
|
|
|
|
|
|
def test_trim_is_noop_on_unsupported_platform(monkeypatch):
|
|
monkeypatch.setattr(malloc_trim, "_resolve", lambda: ("unsupported", None))
|
|
assert malloc_trim.trim() == 0
|
|
|
|
|
|
def test_trim_periodically_trims_each_interval(monkeypatch):
|
|
monkeypatch.setattr(malloc_trim, "_resolve", lambda: ("glibc", object()))
|
|
trims: list[int] = []
|
|
monkeypatch.setattr(malloc_trim, "trim", lambda: trims.append(1) or 0)
|
|
|
|
async def fake_sleep(_seconds):
|
|
if len(trims) >= 2: # let two ticks run, then break the loop
|
|
raise asyncio.CancelledError
|
|
|
|
monkeypatch.setattr(malloc_trim.asyncio, "sleep", fake_sleep)
|
|
|
|
with pytest.raises(asyncio.CancelledError):
|
|
asyncio.run(malloc_trim.trim_periodically(interval_seconds=1))
|
|
assert len(trims) == 2
|
|
|
|
|
|
def test_trim_periodically_is_disabled_on_unsupported_platform(monkeypatch):
|
|
# No supported trim call: the task must return at once, never scheduling a
|
|
# wakeup (so it is a true no-op on Windows/musl, not a 60s spinner).
|
|
monkeypatch.setattr(malloc_trim, "_resolve", lambda: ("unsupported", None))
|
|
trims: list[int] = []
|
|
monkeypatch.setattr(malloc_trim, "trim", lambda: trims.append(1) or 0)
|
|
|
|
async def _no_sleep(_seconds):
|
|
raise AssertionError("unsupported platform must not schedule a trim wakeup")
|
|
|
|
monkeypatch.setattr(malloc_trim.asyncio, "sleep", _no_sleep)
|
|
asyncio.run(malloc_trim.trim_periodically(interval_seconds=60)) # returns, no raise
|
|
assert trims == []
|
|
|
|
|
|
@pytest.mark.parametrize("bad_interval", [0, -5])
|
|
def test_trim_periodically_rejects_non_positive_interval(monkeypatch, bad_interval):
|
|
# A non-positive interval would make asyncio.sleep return immediately and
|
|
# spin a continuous collect/trim loop; it must fall back to the default.
|
|
monkeypatch.setattr(malloc_trim, "_resolve", lambda: ("glibc", object()))
|
|
monkeypatch.setattr(malloc_trim, "trim", lambda: 0)
|
|
slept: list[float] = []
|
|
|
|
async def capture_sleep(seconds):
|
|
slept.append(seconds)
|
|
raise asyncio.CancelledError # stop after the first sleep
|
|
|
|
monkeypatch.setattr(malloc_trim.asyncio, "sleep", capture_sleep)
|
|
with pytest.raises(asyncio.CancelledError):
|
|
asyncio.run(malloc_trim.trim_periodically(interval_seconds=bad_interval))
|
|
assert slept == [malloc_trim._DEFAULT_TRIM_INTERVAL_SECONDS]
|
|
|
|
|
|
def test_trim_runs_off_the_event_loop_thread(monkeypatch):
|
|
# The blocking trim must run in a worker thread (via asyncio.to_thread), not
|
|
# on the event loop, so a slow trim cannot stall other async work.
|
|
import threading
|
|
|
|
monkeypatch.setattr(malloc_trim, "_resolve", lambda: ("glibc", object()))
|
|
seen: dict[str, int] = {}
|
|
|
|
def record():
|
|
seen["thread"] = threading.get_ident()
|
|
return 0
|
|
|
|
monkeypatch.setattr(malloc_trim, "trim", record)
|
|
calls = {"n": 0}
|
|
|
|
async def sleeper(_seconds):
|
|
calls["n"] += 1
|
|
if calls["n"] >= 2: # first sleep returns; after the trim, stop
|
|
raise asyncio.CancelledError
|
|
|
|
monkeypatch.setattr(malloc_trim.asyncio, "sleep", sleeper)
|
|
|
|
async def _run() -> int:
|
|
loop_thread = threading.get_ident()
|
|
with pytest.raises(asyncio.CancelledError):
|
|
await malloc_trim.trim_periodically(interval_seconds=60)
|
|
return loop_thread
|
|
|
|
loop_thread = asyncio.run(_run())
|
|
assert "thread" in seen # trim actually ran
|
|
assert seen["thread"] != loop_thread # ran off the event-loop thread
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_slow_trim_does_not_stop_unrelated_async_work(monkeypatch):
|
|
# The periodic trim is dispatched off the event-loop thread via
|
|
# asyncio.to_thread and runs no Python gc.collect(), so even a slow purge
|
|
# must not freeze the loop. It is modeled here with a worker-thread park
|
|
# which, like the real GIL-releasing allocator C call, does not hold the
|
|
# GIL while it waits: unrelated coroutines keep making progress meanwhile.
|
|
import threading
|
|
|
|
monkeypatch.setattr(malloc_trim, "_resolve", lambda: ("glibc", object()))
|
|
started = threading.Event()
|
|
release = threading.Event()
|
|
|
|
def slow_trim() -> int:
|
|
started.set()
|
|
release.wait(5.0) # hold the worker thread until the test lets go
|
|
return 0
|
|
|
|
monkeypatch.setattr(malloc_trim, "trim", slow_trim)
|
|
|
|
# Fire the trim's interval immediately (the interval is >= 1s) while leaving
|
|
# the counter's sub-second sleeps to behave normally.
|
|
real_sleep = asyncio.sleep
|
|
|
|
async def smart_sleep(seconds):
|
|
if seconds >= 1:
|
|
return
|
|
await real_sleep(seconds)
|
|
|
|
monkeypatch.setattr(malloc_trim.asyncio, "sleep", smart_sleep)
|
|
|
|
ticks = 0
|
|
|
|
async def counter() -> None:
|
|
nonlocal ticks
|
|
while True:
|
|
await real_sleep(0.005)
|
|
ticks += 1
|
|
|
|
counter_task = asyncio.create_task(counter())
|
|
trim_task = asyncio.create_task(malloc_trim.trim_periodically(interval_seconds=60))
|
|
try:
|
|
# Wait for the trim to actually start blocking a worker thread.
|
|
for _ in range(400):
|
|
if started.is_set():
|
|
break
|
|
await real_sleep(0.005)
|
|
assert started.is_set(), "trim never started"
|
|
|
|
# The trim is now parked off-loop. The event loop must keep ticking.
|
|
ticks_before = ticks
|
|
await real_sleep(0.2)
|
|
ticks_during_trim = ticks - ticks_before
|
|
finally:
|
|
release.set()
|
|
counter_task.cancel()
|
|
trim_task.cancel()
|
|
|
|
# On-loop blocking would freeze the counter (~0 ticks); off-thread it keeps
|
|
# ticking (~40 in 0.2s). Generous floor for scheduler jitter.
|
|
assert ticks_during_trim >= 10
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
# ProxyConfig wiring
|
|
# --------------------------------------------------------------------------- #
|
|
def test_proxy_config_malloc_trim_default_is_scoped_to_platforms_with_a_trim_call(monkeypatch):
|
|
# Default-on on macOS and glibc Linux, the two platforms with a trim call
|
|
# and a documented RSS ratchet; elsewhere the periodic task is a no-op, so
|
|
# the default stays off rather than scheduling wakeups for nothing.
|
|
from headroom.proxy import models
|
|
|
|
monkeypatch.setattr(models.sys, "platform", "darwin")
|
|
assert models.ProxyConfig().periodic_malloc_trim_enabled is True
|
|
|
|
monkeypatch.setattr(models.sys, "platform", "linux")
|
|
assert models.ProxyConfig().periodic_malloc_trim_enabled is True
|
|
|
|
monkeypatch.setattr(models.sys, "platform", "win32")
|
|
assert models.ProxyConfig().periodic_malloc_trim_enabled is False
|
|
|
|
# The interval knob is platform-independent.
|
|
assert models.ProxyConfig().malloc_trim_interval_seconds == 60
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("platform", "expected"),
|
|
[("darwin", True), ("linux", True), ("win32", False)],
|
|
)
|
|
def test_cli_proxy_uses_the_same_trim_default_as_the_dataclass(monkeypatch, platform, expected):
|
|
# `headroom proxy` builds its ProxyConfig field by field, so it can only
|
|
# inherit the platform scope by calling the shared default: a literal here
|
|
# (as `sys.platform == "darwin"` was) silently overrides the dataclass for
|
|
# every CLI-launched proxy. Covers the third entry point; the server's
|
|
# env factory and ProxyConfig() itself are covered above.
|
|
pytest.importorskip("click")
|
|
pytest.importorskip("fastapi")
|
|
from click.testing import CliRunner
|
|
|
|
from headroom.cli.main import main
|
|
|
|
monkeypatch.setattr(proxy_cli.sys, "platform", platform)
|
|
monkeypatch.delenv("HEADROOM_MALLOC_TRIM", raising=False)
|
|
captured: dict = {}
|
|
|
|
def fake_run_server(config, **kwargs): # noqa: ANN001
|
|
captured["config"] = config
|
|
|
|
monkeypatch.setattr("headroom.proxy.server.run_server", fake_run_server)
|
|
result = CliRunner().invoke(main, ["proxy"], catch_exceptions=False)
|
|
|
|
assert result.exit_code == 0, result.output
|
|
assert captured["config"].periodic_malloc_trim_enabled is expected
|
|
|
|
|
|
def test_cli_proxy_trim_env_opt_out_still_wins(monkeypatch):
|
|
pytest.importorskip("click")
|
|
pytest.importorskip("fastapi")
|
|
from click.testing import CliRunner
|
|
|
|
from headroom.cli.main import main
|
|
|
|
monkeypatch.setattr(proxy_cli.sys, "platform", "linux")
|
|
monkeypatch.setenv("HEADROOM_MALLOC_TRIM", "0")
|
|
captured: dict = {}
|
|
|
|
def fake_run_server(config, **kwargs): # noqa: ANN001
|
|
captured["config"] = config
|
|
|
|
monkeypatch.setattr("headroom.proxy.server.run_server", fake_run_server)
|
|
result = CliRunner().invoke(main, ["proxy"], catch_exceptions=False)
|
|
|
|
assert result.exit_code == 0, result.output
|
|
assert captured["config"].periodic_malloc_trim_enabled is False
|