fix(proxy): bound request-log message payloads to the feed window; trim glibc too (#3717)

## Description

A user running an overnight Claude Code session through the proxy on
Linux reported it consuming 100 GB of a 128 GB host (about 3,500
requests at ~92k input tokens each, `--log-messages` on). Two stacked
causes, both fixed here:

1. `RequestLogger` keeps message payloads on all 10,000 entries. With
`--log-messages`, every `RequestLog` in the `MAX_LOG_ENTRIES` deque
holds `request_messages`, `compressed_messages` and `response_content`:
two parsed copies of the whole conversation, ~1.9x their JSON size as
Python objects. The only reader of those payloads is
`/transformations/feed`, which is hard-capped at 100 entries, so the
other 9,900 are retained for nothing and grow linearly for the life of
the process.
2. The periodic allocator trim (#2820, #2879) defaults on only for
macOS. glibc retains freed pages the same way, and per-thread arenas on
a many-core host multiply the high-water mark.

Not changed: `MAX_LOG_ENTRIES` itself (`_build_stats_payload` reads the
light fields of all 10,000), and `get_recent_with_messages(n)` for `n >
100`, which now returns bodies only for the newest 100, matching what
the feed could ever serve.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/proxy/request_logger.py`: `MESSAGE_WINDOW = 100`; on every
append, null the three payload fields on the entry that just left the
window. Light fields stay on every entry, so `get_recent` and `/stats`
are unchanged.
- `headroom/proxy/models.py`, `headroom/proxy/server.py`:
`periodic_malloc_trim_enabled` defaults on for `darwin` and `linux` (was
`darwin` only). Platforms without a trim call already disable the task
themselves; `HEADROOM_MALLOC_TRIM=0` still opts out.
- `tests/test_proxy/test_request_logger.py`: window tests (payloads
nulled beyond the newest 100, light fields intact and ordered, feed path
fully populated, short logs untouched).
- `tests/test_malloc_tuning.py`: default-scope test asserts darwin and
linux on, win32 off.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py -q
22 passed in 1.10s
$ uvx ruff check headroom/proxy/request_logger.py headroom/proxy/models.py headroom/proxy/server.py tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py
All checks passed!
$ uvx ruff format --check <same files>
5 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/request_logger.py headroom/proxy/models.py
Success: no issues found
```

## Real Behavior Proof

- Environment: macOS 15.6 (Darwin 24.6.0), Python 3.12, this branch at
ba5f99d7, `uv run --frozen`.
- Exact command / steps: Logged 2,000 `RequestLog` entries through a
real `RequestLogger(log_full_messages=True)`, each carrying a
300-message, ~128 KB conversation in both `request_messages` and
`compressed_messages`, under `tracemalloc`; once with the fix
(`MESSAGE_WINDOW = 100`) and once with the window set to 10,000 to
reproduce the pre-fix retention.
- Observed result: fix: 41 MB live, 100 entries with bodies; pre-fix:
802 MB live, 2,000 entries with bodies. Same 2,000 entries remain in the
deque either way. Extrapolated to the reporter's 10,000-entry cap and
~1.5 MB per entry, the pre-fix retention alone is ~15 GB before
allocator bloat.
- Not tested: end-to-end RSS on a Linux host with the trim default
flipped (no Linux machine available; the trim path itself is covered by
the existing `test_malloc_tuning.py` suite and was measured on macOS in
#2820).

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, two defaults. (1) Request-log
entries older than the newest 100 no longer carry message payloads; only
`/transformations/feed` reads them and it is capped at 100, so no served
response changes. (2) The periodic allocator trim is now on by default
on glibc Linux, as it already was on macOS.
- Kill switch / disable path: `HEADROOM_MALLOC_TRIM=0` disables the
trim. The payload window has no switch; it changes no output.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert the PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from the Conventional Commit PR title (a CI guard
enforces this)

## Screenshots (if applicable)

N/A

## Additional Notes

Documentation: no user-facing docs describe `MAX_LOG_ENTRIES` or the
trim default, so nothing to update. Two commits, one per cause, in case
the maintainers want the Linux trim default separately.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
gglucass
2026-09-24 10:51:02 -05:00
committed by GitHub
co-authored by Claude Fable 5.1 JerrettDavis
parent ecb5e5afea
commit 994a89f169
6 changed files with 140 additions and 13 deletions
+2 -1
View File
@@ -1137,6 +1137,7 @@ def proxy(
_parse_csv_tools,
_parse_exclude_tools,
_parse_tool_profiles,
default_periodic_malloc_trim,
run_server,
)
@@ -1326,7 +1327,7 @@ def proxy(
rate_limit_tokens_per_minute=tpm if tpm is not None else 100_000,
compress_user_messages=_get_env_bool("HEADROOM_COMPRESS_USER_MESSAGES", False),
periodic_malloc_trim_enabled=_get_env_bool(
"HEADROOM_MALLOC_TRIM", sys.platform == "darwin"
"HEADROOM_MALLOC_TRIM", default_periodic_malloc_trim()
),
malloc_trim_interval_seconds=_get_env_int("HEADROOM_MALLOC_TRIM_INTERVAL_SECONDS", 60),
min_tokens_to_crush=_get_env_int("HEADROOM_MIN_TOKENS", 500),
+17 -6
View File
@@ -42,6 +42,16 @@ def _qdrant_env_port_or_default() -> int:
return qdrant_env.DEFAULT_QDRANT_PORT
def default_periodic_malloc_trim() -> bool:
"""Platforms where the periodic allocator trim is on unless opted out.
Shared by ``ProxyConfig``'s default and by the two call sites that build a
config explicitly (``headroom proxy`` and ``_proxy_config_from_env``), so the
platform scope cannot drift between them.
"""
return sys.platform in ("darwin", "linux")
# =============================================================================
# Data Models
# =============================================================================
@@ -470,12 +480,13 @@ class ProxyConfig:
# Periodic allocator trim. Long-lived proxies processing large concurrent
# request bodies ratchet RSS through freed-but-retained allocator pages;
# this returns them to the OS (malloc_zone_pressure_relief on macOS,
# malloc_trim on glibc). Default-on only on macOS, where the retained-page
# ratchet is the documented failure (#2820); an opt-in elsewhere via
# HEADROOM_MALLOC_TRIM=1 so glibc deployments do not silently take on a
# once-a-minute allocator purge they did not ask for. Envs:
# HEADROOM_MALLOC_TRIM=0/1, HEADROOM_MALLOC_TRIM_INTERVAL_SECONDS.
periodic_malloc_trim_enabled: bool = field(default_factory=lambda: sys.platform == "darwin")
# malloc_trim on glibc). Default-on where a trim call exists: macOS, where
# the retained-page ratchet was first documented (#2820), and glibc Linux,
# which ratchets the same way (a coding-agent session took one proxy to
# 100 GB RSS on a 128 GB host with the trim off). Platforms without a trim
# call disable the task themselves. Envs: HEADROOM_MALLOC_TRIM=0/1,
# HEADROOM_MALLOC_TRIM_INTERVAL_SECONDS.
periodic_malloc_trim_enabled: bool = field(default_factory=default_periodic_malloc_trim)
malloc_trim_interval_seconds: int = 60
# Stateless mode — disable all filesystem writes for read-only / container deployments
+16
View File
@@ -97,6 +97,14 @@ class RequestLogger:
"""
MAX_LOG_ENTRIES = 10_000
# How many of the newest entries keep their message payloads
# (``request_messages`` / ``compressed_messages`` / ``response_content``).
# ``/transformations/feed`` serves at most this many, and nothing else
# reads the payloads back, so keeping them on all MAX_LOG_ENTRIES retained
# two parsed copies of every conversation for the life of the process:
# a long agentic session grew one proxy to 100 GB RSS. Light fields stay
# on every entry for ``get_recent`` / ``/stats``.
MESSAGE_WINDOW = 100
def __init__(self, log_file: str | None = None, log_full_messages: bool = False):
self.log_file = Path(log_file) if log_file else None
@@ -136,6 +144,14 @@ class RequestLogger:
entry.response_content = redact_image_base64(entry.response_content)
self._logs.append(entry)
# Drop the payloads of the entry that just left the message window.
# One entry per append keeps the newest MESSAGE_WINDOW populated and
# every older one light without a scan.
if len(self._logs) > self.MESSAGE_WINDOW:
aged = self._logs[-self.MESSAGE_WINDOW - 1]
aged.request_messages = None
aged.compressed_messages = None
aged.response_content = None
if self.log_file:
try:
+8 -2
View File
@@ -153,7 +153,13 @@ from headroom.proxy.memory_handler import MemoryConfig, MemoryHandler
# Data models (extracted to headroom/proxy/models.py for maintainability)
from headroom.proxy.model_router import ModelRouter, ModelRouterConfig
from headroom.proxy.models import CacheEntry, ProxyConfig, RateLimitState, RequestLog # noqa: F401
from headroom.proxy.models import ( # noqa: F401
CacheEntry,
ProxyConfig,
RateLimitState,
RequestLog,
default_periodic_malloc_trim,
)
from headroom.proxy.modes import (
PROXY_MODE_CACHE,
PROXY_MODE_TOKEN,
@@ -5723,7 +5729,7 @@ def _proxy_config_from_env() -> ProxyConfig:
http_proxy=os.environ.get("HEADROOM_HTTP_PROXY") or None,
periodic_toin_stats_enabled=_get_env_bool("HEADROOM_PERIODIC_TOIN_STATS", True),
periodic_malloc_trim_enabled=_get_env_bool(
"HEADROOM_MALLOC_TRIM", sys.platform == "darwin"
"HEADROOM_MALLOC_TRIM", default_periodic_malloc_trim()
),
malloc_trim_interval_seconds=_get_env_int("HEADROOM_MALLOC_TRIM_INTERVAL_SECONDS", 60),
proxy_token=os.environ.get("HEADROOM_PROXY_TOKEN") or None,
+58 -4
View File
@@ -276,17 +276,71 @@ async def test_slow_trim_does_not_stop_unrelated_async_work(monkeypatch):
# --------------------------------------------------------------------------- #
# ProxyConfig wiring
# --------------------------------------------------------------------------- #
def test_proxy_config_malloc_trim_default_is_darwin_scoped(monkeypatch):
# Default-on only on macOS (the platform with the documented RSS ratchet);
# elsewhere it is opt-in, so glibc deployments do not silently take on a
# once-a-minute allocator purge.
def test_proxy_config_malloc_trim_default_is_scoped_to_platforms_with_a_trim_call(monkeypatch):
# Default-on on macOS and glibc Linux, the two platforms with a trim call
# and a documented RSS ratchet; elsewhere the periodic task is a no-op, so
# the default stays off rather than scheduling wakeups for nothing.
from headroom.proxy import models
monkeypatch.setattr(models.sys, "platform", "darwin")
assert models.ProxyConfig().periodic_malloc_trim_enabled is True
monkeypatch.setattr(models.sys, "platform", "linux")
assert models.ProxyConfig().periodic_malloc_trim_enabled is True
monkeypatch.setattr(models.sys, "platform", "win32")
assert models.ProxyConfig().periodic_malloc_trim_enabled is False
# The interval knob is platform-independent.
assert models.ProxyConfig().malloc_trim_interval_seconds == 60
@pytest.mark.parametrize(
("platform", "expected"),
[("darwin", True), ("linux", True), ("win32", False)],
)
def test_cli_proxy_uses_the_same_trim_default_as_the_dataclass(monkeypatch, platform, expected):
# `headroom proxy` builds its ProxyConfig field by field, so it can only
# inherit the platform scope by calling the shared default: a literal here
# (as `sys.platform == "darwin"` was) silently overrides the dataclass for
# every CLI-launched proxy. Covers the third entry point; the server's
# env factory and ProxyConfig() itself are covered above.
pytest.importorskip("click")
pytest.importorskip("fastapi")
from click.testing import CliRunner
from headroom.cli.main import main
monkeypatch.setattr(proxy_cli.sys, "platform", platform)
monkeypatch.delenv("HEADROOM_MALLOC_TRIM", raising=False)
captured: dict = {}
def fake_run_server(config, **kwargs): # noqa: ANN001
captured["config"] = config
monkeypatch.setattr("headroom.proxy.server.run_server", fake_run_server)
result = CliRunner().invoke(main, ["proxy"], catch_exceptions=False)
assert result.exit_code == 0, result.output
assert captured["config"].periodic_malloc_trim_enabled is expected
def test_cli_proxy_trim_env_opt_out_still_wins(monkeypatch):
pytest.importorskip("click")
pytest.importorskip("fastapi")
from click.testing import CliRunner
from headroom.cli.main import main
monkeypatch.setattr(proxy_cli.sys, "platform", "linux")
monkeypatch.setenv("HEADROOM_MALLOC_TRIM", "0")
captured: dict = {}
def fake_run_server(config, **kwargs): # noqa: ANN001
captured["config"] = config
monkeypatch.setattr("headroom.proxy.server.run_server", fake_run_server)
result = CliRunner().invoke(main, ["proxy"], catch_exceptions=False)
assert result.exit_code == 0, result.output
assert captured["config"].periodic_malloc_trim_enabled is False
+39
View File
@@ -134,6 +134,45 @@ def test_get_recent_never_walks_message_payloads():
assert again["savings_breakdown"] == [{"tokens": 60}]
def test_payloads_survive_only_on_the_newest_message_window():
logger = RequestLogger(log_file=None, log_full_messages=True)
window = RequestLogger.MESSAGE_WINDOW
total = window + 50
for i in range(total):
logger.log(
_entry(
request_id=f"r{i}",
request_messages=[{"role": "user", "content": f"pre{i}"}],
compressed_messages=[{"role": "user", "content": f"post{i}"}],
response_content=f"resp{i}",
)
)
logs = list(logger._logs)
assert len(logs) == total
# Every entry is still there for /stats, in order, light fields intact.
assert [e.request_id for e in logs] == [f"r{i}" for i in range(total)]
assert all(e.tokens_saved == 60 for e in logs)
# Only the newest MESSAGE_WINDOW keep their payloads.
aged, kept = logs[:-window], logs[-window:]
assert all(
e.request_messages is None and e.compressed_messages is None and e.response_content is None
for e in aged
)
assert all(e.request_messages and e.compressed_messages and e.response_content for e in kept)
# The feed path is fully populated for its whole cap.
feed = logger.get_recent_with_messages(window)
assert len(feed) == window
assert all(item["request_messages"] for item in feed)
def test_message_window_leaves_a_short_log_untouched():
logger = RequestLogger(log_file=None, log_full_messages=True)
for i in range(RequestLogger.MESSAGE_WINDOW):
logger.log(_entry(request_id=f"r{i}", request_messages=[{"role": "user", "content": "x"}]))
assert all(e.request_messages for e in logger._logs)
def test_get_recent_with_messages_can_skip_payloads_without_walking_them():
"""A feed poller that only reads the numbers passes include_messages=False;
the three body fields are absent and never traversed (asdict deep-copied