mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description
A user running an overnight Claude Code session through the proxy on
Linux reported it consuming 100 GB of a 128 GB host (about 3,500
requests at ~92k input tokens each, `--log-messages` on). Two stacked
causes, both fixed here:
1. `RequestLogger` keeps message payloads on all 10,000 entries. With
`--log-messages`, every `RequestLog` in the `MAX_LOG_ENTRIES` deque
holds `request_messages`, `compressed_messages` and `response_content`:
two parsed copies of the whole conversation, ~1.9x their JSON size as
Python objects. The only reader of those payloads is
`/transformations/feed`, which is hard-capped at 100 entries, so the
other 9,900 are retained for nothing and grow linearly for the life of
the process.
2. The periodic allocator trim (#2820, #2879) defaults on only for
macOS. glibc retains freed pages the same way, and per-thread arenas on
a many-core host multiply the high-water mark.
Not changed: `MAX_LOG_ENTRIES` itself (`_build_stats_payload` reads the
light fields of all 10,000), and `get_recent_with_messages(n)` for `n >
100`, which now returns bodies only for the newest 100, matching what
the feed could ever serve.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/request_logger.py`: `MESSAGE_WINDOW = 100`; on every
append, null the three payload fields on the entry that just left the
window. Light fields stay on every entry, so `get_recent` and `/stats`
are unchanged.
- `headroom/proxy/models.py`, `headroom/proxy/server.py`:
`periodic_malloc_trim_enabled` defaults on for `darwin` and `linux` (was
`darwin` only). Platforms without a trim call already disable the task
themselves; `HEADROOM_MALLOC_TRIM=0` still opts out.
- `tests/test_proxy/test_request_logger.py`: window tests (payloads
nulled beyond the newest 100, light fields intact and ordered, feed path
fully populated, short logs untouched).
- `tests/test_malloc_tuning.py`: default-scope test asserts darwin and
linux on, win32 off.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py -q
22 passed in 1.10s
$ uvx ruff check headroom/proxy/request_logger.py headroom/proxy/models.py headroom/proxy/server.py tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py
All checks passed!
$ uvx ruff format --check <same files>
5 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/request_logger.py headroom/proxy/models.py
Success: no issues found
```
## Real Behavior Proof
- Environment: macOS 15.6 (Darwin 24.6.0), Python 3.12, this branch at
ba5f99d7, `uv run --frozen`.
- Exact command / steps: Logged 2,000 `RequestLog` entries through a
real `RequestLogger(log_full_messages=True)`, each carrying a
300-message, ~128 KB conversation in both `request_messages` and
`compressed_messages`, under `tracemalloc`; once with the fix
(`MESSAGE_WINDOW = 100`) and once with the window set to 10,000 to
reproduce the pre-fix retention.
- Observed result: fix: 41 MB live, 100 entries with bodies; pre-fix:
802 MB live, 2,000 entries with bodies. Same 2,000 entries remain in the
deque either way. Extrapolated to the reporter's 10,000-entry cap and
~1.5 MB per entry, the pre-fix retention alone is ~15 GB before
allocator bloat.
- Not tested: end-to-end RSS on a Linux host with the trim default
flipped (no Linux machine available; the trim path itself is covered by
the existing `test_malloc_tuning.py` suite and was measured on macOS in
#2820).
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, two defaults. (1) Request-log
entries older than the newest 100 no longer carry message payloads; only
`/transformations/feed` reads them and it is capped at 100, so no served
response changes. (2) The periodic allocator trim is now on by default
on glibc Linux, as it already was on macOS.
- Kill switch / disable path: `HEADROOM_MALLOC_TRIM=0` disables the
trim. The payload window has no switch; it changes no output.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert the PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from the Conventional Commit PR title (a CI guard
enforces this)
## Screenshots (if applicable)
N/A
## Additional Notes
Documentation: no user-facing docs describe `MAX_LOG_ENTRIES` or the
trim default, so nothing to update. Two commits, one per cause, in case
the maintainers want the Linux trim default separately.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
198 lines
7.3 KiB
Python
198 lines
7.3 KiB
Python
"""Tests for the in-memory request logger.
|
|
|
|
Covers the `log_full_messages` gate, which controls whether the
|
|
pre-compression (`request_messages`) and post-compression
|
|
(`compressed_messages`) payloads persist past the in-memory entry onto disk.
|
|
Both sides are governed by the same flag so the two sides of the compression
|
|
stay in sync - it's pointless to store one without the other.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from headroom.proxy.models import RequestLog
|
|
from headroom.proxy.request_logger import RequestLogger
|
|
|
|
|
|
def _entry(**overrides) -> RequestLog:
|
|
base: dict = {
|
|
"request_id": "r1",
|
|
"timestamp": "2026-04-24T10:00:00Z",
|
|
"provider": "anthropic",
|
|
"model": "claude-sonnet-4-6",
|
|
"input_tokens_original": 100,
|
|
"input_tokens_optimized": 40,
|
|
"output_tokens": 10,
|
|
"tokens_saved": 60,
|
|
"savings_percent": 60.0,
|
|
"optimization_latency_ms": 1.0,
|
|
"total_latency_ms": 20.0,
|
|
"tags": {},
|
|
"cache_hit": False,
|
|
"transforms_applied": ["kompress:user:0.4"],
|
|
}
|
|
base.update(overrides)
|
|
return RequestLog(**base)
|
|
|
|
|
|
def test_get_recent_strips_compressed_messages_alongside_request_and_response():
|
|
logger = RequestLogger(log_file=None, log_full_messages=True)
|
|
logger.log(
|
|
_entry(
|
|
request_messages=[{"role": "user", "content": "pre"}],
|
|
compressed_messages=[{"role": "user", "content": "post"}],
|
|
response_content="ok",
|
|
)
|
|
)
|
|
|
|
recent = logger.get_recent(10)
|
|
assert len(recent) == 1
|
|
assert "request_messages" not in recent[0]
|
|
assert "compressed_messages" not in recent[0]
|
|
assert "response_content" not in recent[0]
|
|
|
|
|
|
def test_get_recent_with_messages_returns_compressed_messages():
|
|
logger = RequestLogger(log_file=None, log_full_messages=True)
|
|
logger.log(
|
|
_entry(
|
|
request_messages=[{"role": "user", "content": "pre"}],
|
|
compressed_messages=[{"role": "user", "content": "post"}],
|
|
)
|
|
)
|
|
|
|
recent = logger.get_recent_with_messages(10)
|
|
assert len(recent) == 1
|
|
assert recent[0]["request_messages"] == [{"role": "user", "content": "pre"}]
|
|
assert recent[0]["compressed_messages"] == [{"role": "user", "content": "post"}]
|
|
|
|
|
|
def test_jsonl_file_strips_both_sides_when_log_full_messages_disabled(tmp_path):
|
|
log_file = tmp_path / "requests.jsonl"
|
|
logger = RequestLogger(log_file=str(log_file), log_full_messages=False)
|
|
logger.log(
|
|
_entry(
|
|
request_messages=[{"role": "user", "content": "pre"}],
|
|
compressed_messages=[{"role": "user", "content": "post"}],
|
|
response_content="ok",
|
|
)
|
|
)
|
|
|
|
import json
|
|
|
|
lines = log_file.read_text().strip().splitlines()
|
|
assert len(lines) == 1
|
|
obj = json.loads(lines[0])
|
|
assert "request_messages" not in obj
|
|
assert "compressed_messages" not in obj
|
|
assert "response_content" not in obj
|
|
|
|
|
|
def test_get_memory_stats_accounts_for_compressed_messages():
|
|
logger = RequestLogger(log_file=None)
|
|
logger.log(
|
|
_entry(
|
|
compressed_messages=[{"role": "user", "content": "post"}],
|
|
)
|
|
)
|
|
|
|
stats = logger.get_memory_stats()
|
|
assert stats.entry_count == 1
|
|
assert stats.size_bytes > 0
|
|
|
|
|
|
def test_get_recent_never_walks_message_payloads():
|
|
"""`/stats` calls get_recent(10_000) on the event loop; the payloads it
|
|
drops must not be traversed first (asdict deep-copied them, ~30 s per call
|
|
on a full deque). A leaf that refuses to be deep-copied proves the walk
|
|
is gone."""
|
|
|
|
class _NoCopy:
|
|
def __deepcopy__(self, memo):
|
|
raise AssertionError("get_recent walked a message payload")
|
|
|
|
logger = RequestLogger(log_file=None, log_full_messages=True)
|
|
logger.log(
|
|
_entry(
|
|
request_messages=[{"role": "user", "content": _NoCopy()}],
|
|
compressed_messages=[{"role": "user", "content": _NoCopy()}],
|
|
tags={"agent": "codex", "meta": {"depth": 1}},
|
|
transforms_applied=["smart_crusher"],
|
|
savings_breakdown=[{"tokens": 60}],
|
|
)
|
|
)
|
|
|
|
recent = logger.get_recent(10)
|
|
assert recent[0]["tags"] == {"agent": "codex", "meta": {"depth": 1}}
|
|
assert recent[0]["transforms_applied"] == ["smart_crusher"]
|
|
# Still copies, not aliases, of the entry's own containers, all the way
|
|
# down: mutating a nested value must not change the next /stats result.
|
|
recent[0]["tags"]["agent"] = "x"
|
|
recent[0]["tags"]["meta"]["depth"] = 99
|
|
recent[0]["savings_breakdown"][0]["tokens"] = 0
|
|
again = logger.get_recent(10)[0]
|
|
assert again["tags"] == {"agent": "codex", "meta": {"depth": 1}}
|
|
assert again["savings_breakdown"] == [{"tokens": 60}]
|
|
|
|
|
|
def test_payloads_survive_only_on_the_newest_message_window():
|
|
logger = RequestLogger(log_file=None, log_full_messages=True)
|
|
window = RequestLogger.MESSAGE_WINDOW
|
|
total = window + 50
|
|
for i in range(total):
|
|
logger.log(
|
|
_entry(
|
|
request_id=f"r{i}",
|
|
request_messages=[{"role": "user", "content": f"pre{i}"}],
|
|
compressed_messages=[{"role": "user", "content": f"post{i}"}],
|
|
response_content=f"resp{i}",
|
|
)
|
|
)
|
|
|
|
logs = list(logger._logs)
|
|
assert len(logs) == total
|
|
# Every entry is still there for /stats, in order, light fields intact.
|
|
assert [e.request_id for e in logs] == [f"r{i}" for i in range(total)]
|
|
assert all(e.tokens_saved == 60 for e in logs)
|
|
# Only the newest MESSAGE_WINDOW keep their payloads.
|
|
aged, kept = logs[:-window], logs[-window:]
|
|
assert all(
|
|
e.request_messages is None and e.compressed_messages is None and e.response_content is None
|
|
for e in aged
|
|
)
|
|
assert all(e.request_messages and e.compressed_messages and e.response_content for e in kept)
|
|
# The feed path is fully populated for its whole cap.
|
|
feed = logger.get_recent_with_messages(window)
|
|
assert len(feed) == window
|
|
assert all(item["request_messages"] for item in feed)
|
|
|
|
|
|
def test_message_window_leaves_a_short_log_untouched():
|
|
logger = RequestLogger(log_file=None, log_full_messages=True)
|
|
for i in range(RequestLogger.MESSAGE_WINDOW):
|
|
logger.log(_entry(request_id=f"r{i}", request_messages=[{"role": "user", "content": "x"}]))
|
|
assert all(e.request_messages for e in logger._logs)
|
|
|
|
|
|
def test_get_recent_with_messages_can_skip_payloads_without_walking_them():
|
|
"""A feed poller that only reads the numbers passes include_messages=False;
|
|
the three body fields are absent and never traversed (asdict deep-copied
|
|
them). A leaf that refuses to be deep-copied proves the walk is gone."""
|
|
|
|
class _NoCopy:
|
|
def __deepcopy__(self, memo):
|
|
raise AssertionError("get_recent_with_messages walked a message payload")
|
|
|
|
logger = RequestLogger(log_file=None, log_full_messages=True)
|
|
logger.log(
|
|
_entry(
|
|
request_messages=[{"role": "user", "content": _NoCopy()}],
|
|
compressed_messages=[{"role": "user", "content": _NoCopy()}],
|
|
response_content="ok",
|
|
)
|
|
)
|
|
|
|
slim = logger.get_recent_with_messages(10, include_messages=False)
|
|
assert slim[0]["tokens_saved"] == 60
|
|
assert slim[0]["transforms_applied"] == ["kompress:user:0.4"]
|
|
assert not {"request_messages", "compressed_messages", "response_content"} & slim[0].keys()
|