Commit Graph
4 Commits
Author SHA1 Message Date
994a89f169 fix(proxy): bound request-log message payloads to the feed window; trim glibc too (#3717)
## Description

A user running an overnight Claude Code session through the proxy on
Linux reported it consuming 100 GB of a 128 GB host (about 3,500
requests at ~92k input tokens each, `--log-messages` on). Two stacked
causes, both fixed here:

1. `RequestLogger` keeps message payloads on all 10,000 entries. With
`--log-messages`, every `RequestLog` in the `MAX_LOG_ENTRIES` deque
holds `request_messages`, `compressed_messages` and `response_content`:
two parsed copies of the whole conversation, ~1.9x their JSON size as
Python objects. The only reader of those payloads is
`/transformations/feed`, which is hard-capped at 100 entries, so the
other 9,900 are retained for nothing and grow linearly for the life of
the process.
2. The periodic allocator trim (#2820, #2879) defaults on only for
macOS. glibc retains freed pages the same way, and per-thread arenas on
a many-core host multiply the high-water mark.

Not changed: `MAX_LOG_ENTRIES` itself (`_build_stats_payload` reads the
light fields of all 10,000), and `get_recent_with_messages(n)` for `n >
100`, which now returns bodies only for the newest 100, matching what
the feed could ever serve.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/proxy/request_logger.py`: `MESSAGE_WINDOW = 100`; on every
append, null the three payload fields on the entry that just left the
window. Light fields stay on every entry, so `get_recent` and `/stats`
are unchanged.
- `headroom/proxy/models.py`, `headroom/proxy/server.py`:
`periodic_malloc_trim_enabled` defaults on for `darwin` and `linux` (was
`darwin` only). Platforms without a trim call already disable the task
themselves; `HEADROOM_MALLOC_TRIM=0` still opts out.
- `tests/test_proxy/test_request_logger.py`: window tests (payloads
nulled beyond the newest 100, light fields intact and ordered, feed path
fully populated, short logs untouched).
- `tests/test_malloc_tuning.py`: default-scope test asserts darwin and
linux on, win32 off.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py -q
22 passed in 1.10s
$ uvx ruff check headroom/proxy/request_logger.py headroom/proxy/models.py headroom/proxy/server.py tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py
All checks passed!
$ uvx ruff format --check <same files>
5 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/request_logger.py headroom/proxy/models.py
Success: no issues found
```

## Real Behavior Proof

- Environment: macOS 15.6 (Darwin 24.6.0), Python 3.12, this branch at
ba5f99d7, `uv run --frozen`.
- Exact command / steps: Logged 2,000 `RequestLog` entries through a
real `RequestLogger(log_full_messages=True)`, each carrying a
300-message, ~128 KB conversation in both `request_messages` and
`compressed_messages`, under `tracemalloc`; once with the fix
(`MESSAGE_WINDOW = 100`) and once with the window set to 10,000 to
reproduce the pre-fix retention.
- Observed result: fix: 41 MB live, 100 entries with bodies; pre-fix:
802 MB live, 2,000 entries with bodies. Same 2,000 entries remain in the
deque either way. Extrapolated to the reporter's 10,000-entry cap and
~1.5 MB per entry, the pre-fix retention alone is ~15 GB before
allocator bloat.
- Not tested: end-to-end RSS on a Linux host with the trim default
flipped (no Linux machine available; the trim path itself is covered by
the existing `test_malloc_tuning.py` suite and was measured on macOS in
#2820).

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, two defaults. (1) Request-log
entries older than the newest 100 no longer carry message payloads; only
`/transformations/feed` reads them and it is capped at 100, so no served
response changes. (2) The periodic allocator trim is now on by default
on glibc Linux, as it already was on macOS.
- Kill switch / disable path: `HEADROOM_MALLOC_TRIM=0` disables the
trim. The payload window has no switch; it changes no output.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert the PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from the Conventional Commit PR title (a CI guard
enforces this)

## Screenshots (if applicable)

N/A

## Additional Notes

Documentation: no user-facing docs describe `MAX_LOG_ENTRIES` or the
trim default, so nothing to update. Two commits, one per cause, in case
the maintainers want the Linux trim default separately.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-24 10:51:02 -05:00
gglucass 46ac52d39d feat(proxy): let /transformations/feed omit message bodies for number-only pollers (#3672)
## Description

`/transformations/feed` always returns `request_messages`,
`compressed_messages` and `response_content` for every entry, and
`RequestLogger.get_recent_with_messages` builds them with `asdict()`,
which deep-copies each transcript first. A poller that only reads the
per-request numbers (tokens, model, transforms) still pays for a full
copy and a multi-MB JSON serialization on the event loop for each pull.
Measured on a desktop client polling every 20 s: a `limit=100` pull is
~44 MB, of which ~400 bytes per entry are read, and three concurrent
pulls take `/stats?cached=1` from 45 ms to 1.3 s.

This adds `?include_messages=0` (default `1`, byte-identical output to
today). When off, the three body fields are omitted from the response
and never walked server-side.

Closes #

## Type of Change

- [x] Performance improvement

## Changes Made

- `RequestLogger.get_recent_with_messages(n, include_messages=True)`:
when `False`, builds each entry from the dataclass fields minus the
three payload fields, deep-copying only what it returns (same idiom as
#3613 for `get_recent`).
- `/transformations/feed` accepts `include_messages: bool = True` and
omits the three body keys when it is off. Everything else in the item,
including `log_full_messages`, is unchanged.
- Each item also carries `uncached_input_tokens`, `cache_write_tokens`
and `cache_read_tokens` (already stored on `RequestLog`, never exposed
by the feed). A number-only poller can then rate `tokens_saved` against
new input (uncached + cache_write) the way `/stats`
`new_input_savings_percent` does; the item's full-transcript
`savings_percent` is unchanged. Present in both the full and the slim
response.
- Tests: logger test proves the payloads are not traversed (a leaf whose
`__deepcopy__` raises); endpoint test proves the slim item equals the
full item minus the three keys and carries the cache split.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_transformations_feed.py tests/test_proxy/test_request_logger.py -q
9 passed in 4.53s
$ uvx ruff check headroom/proxy/request_logger.py headroom/proxy/server.py tests/test_proxy/test_transformations_feed.py tests/test_proxy/test_request_logger.py
All checks passed!
$ uv run --frozen --extra dev mypy headroom/proxy/request_logger.py headroom/proxy/server.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS 15, Python 3.12, this branch, in-process
`RequestLogger` seeded with 100 entries carrying ~160 KB of messages
each (the size of a mid-session Claude Code turn).
- Exact command / steps: `get_recent_with_messages(100)` and
`get_recent_with_messages(100, include_messages=False)`, timed, then
`json.dumps` of each result, timed and sized.
- Observed result: include_messages=True: copy 29.4 ms, json 56.5 ms,
24.45 MB. include_messages=False: copy 1.3 ms, json 1.9 ms, 0.05 MB.
Same 100 entries, same numeric fields.
- Not tested: the dashboard UI (it does not pass the parameter, so it
gets the default, unchanged response).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a (opt-in query parameter)
- Stable/default behavior changed: additive only; without the parameter
every existing key is byte-identical, and three integer keys are added
to each item
- Kill switch / disable path: callers stop passing `include_messages=0`
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit; no persisted state or schema
involved

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: the endpoint has no reference page; the parameter is
documented in the handler docstring.
- Overlaps #3613 on the `_HEAVY_FIELDS` constant and the import lines;
both hunks are identical so either merge order applies cleanly.
2026-09-23 23:12:15 -05:00
44b8e7ba87 fix(proxy): stop get_recent deep-copying dropped message payloads (#3613)
## Description

`RequestLogger.get_recent()` built each returned row with
`dataclasses.asdict(entry)` and only then dropped `request_messages`,
`compressed_messages` and `response_content`. `asdict` deep-copies every
field first, so every call walked the full request and compressed
transcripts of every entry it returned, for fields it was about to
discard.

`/stats` calls `get_recent(10_000)` synchronously on the event loop on
every build (both the uncached path and the `?cached=1` snapshot
rebuild). Measured against 0.37.0 with a 300-message Claude Code
transcript shape (~400 KB per entry): 3.4 ms per entry, so a full 10k
deque costs ~34 s per `/stats` call, during which the loop serves
nothing else. A desktop dashboard polling `/stats` reproduced this in
production: 4,769 `/stats` calls in 55 min with a mean latency of 517 s,
`/v1/messages` mean 135 s (max 1,839 s), 61k `Exceeded concurrency
limit` warnings and a watchdog restart storm (Headroom Desktop user
report, 2026-09-16).

The fix builds the row from `fields()` while skipping the heavy fields,
so the payloads are never touched. Kept fields are scalars plus flat
`dict`/`list` containers; those get the same shallow copy `asdict`
produced for them, so callers still get copies rather than aliases into
the log entry.

Fixes the backend half of the desktop report; the desktop side stops
polling uncached `/stats` for connector verification separately.

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- `headroom/proxy/request_logger.py`: `get_recent` iterates `fields()`
and skips `_HEAVY_FIELDS` instead of calling `asdict()` and filtering
afterwards. Shallow-copies the kept `dict`/`list` fields.
- `tests/test_proxy/test_request_logger.py`: new test plants a leaf
whose `__deepcopy__` raises inside both message payloads and asserts
`get_recent` still returns the row with copied (not aliased) containers.
Fails on the previous implementation, passes on this one.

## Testing

- [x] Unit tests pass locally (`uv run --frozen --extra dev pytest
tests/test_proxy/test_request_logger.py`, 5 passed; `pytest
tests/test_proxy -k "stats and not history"`, 6 passed)
- [x] `uvx ruff check` and `uvx ruff format --check` clean on both files
- [x] New test verified to fail on the old code (`git stash` the source
change: 1 failed, 4 passed)

### Test Output

```
$ uv run --frozen --extra dev pytest tests/test_proxy/test_request_logger.py -q
tests/test_proxy/test_request_logger.py .....                            [100%]

============================== 5 passed in 0.23s ===============================

$ uvx ruff check headroom/proxy/request_logger.py tests/test_proxy/test_request_logger.py
All checks passed!
```

## Real Behavior Proof

- Environment: macOS 15, Python 3.12, headroom-ai 0.37.0 venv (the
desktop app's pinned runtime), plus this branch.
- Exact command / steps: Built a `RequestLog` with
`request_messages`/`compressed_messages` set to a 300-message transcript
(3 content blocks each, ~416 KB serialized) and timed
`dataclasses.asdict(entry)` in a loop; then ran the new unit test
against the old and new `get_recent`.
- Observed result: 3.4 ms per `asdict` call on 0.37.0, i.e. ~34 s for
the 10,000-entry `get_recent` that `/stats` performs on the event loop.
With this change the per-row cost is a `fields()` walk plus shallow
copies of two small containers (microseconds), and the new test passes
only on the new code.
- Not tested: an end-to-end `/stats` latency comparison on a long-lived
proxy with a full 10k deque; the production numbers above come from a
user's `proxy.log`, not from a controlled run.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no; `/stats` returns the same rows
and keys, they are just built without walking the dropped payloads
- Kill switch / disable path: none needed; revert the commit
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes

## Screenshots

n/a

## Additional Notes

`get_recent_with_messages` keeps `asdict` on purpose: it returns the
payloads, so the walk is the point there.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-21 09:37:34 -07:00
gglucassandClaude Opus 4.8 2269e40bde feat(proxy): log compressed messages alongside original request (#261)
## Description

Expose the post-compression message list that was actually sent upstream
as a new `compressed_messages` field on `RequestLog`, paired with the
existing (now consistently pre-compression) `request_messages`.
Consumers of `/transformations/feed` — dashboards and any downstream
observability — can now diff the two sides of a compression to see
exactly what the pipeline stripped, replaced, or kept. Turns an abstract
"saved N tokens" into a legible before/after.

Gated by the same `log_full_messages` flag as `request_messages` so the
two sides stay in sync; it's pointless to store one without the other.

Also fixes a latent correctness bug: today's `request_messages` field is
inconsistent across the four `RequestLog` construction sites — sometimes
it's the pre-compression snapshot, sometimes it's the mutated
`body["messages"]` (which is the compressed list, because the proxy
mutates `body` in place before the log call). After this change,
`request_messages` always means pre-compression and
`compressed_messages` always means what went upstream.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

Note on "breaking": strictly speaking this is a semantic correction of
an inconsistently-populated field, not a schema break. The field name
`request_messages` is unchanged and the JSON shape is unchanged; what
changes is that the field now consistently holds the pre-compression
list. Consumers that treated it as "whatever messages we have" continue
to work. Consumers that depended on the accidental post-compression
value (if any existed) would shift to `compressed_messages`.

## Changes Made

- **`headroom/proxy/models.py`**: `RequestLog` gains
`compressed_messages: list[dict] | None = None`. Doc comment explains
it's paired with `request_messages` and gated by the same
`log_full_messages` flag.
- **`headroom/proxy/handlers/anthropic.py`** (2 sites — Bedrock
non-streaming and main non-streaming): `request_messages` now
consistently sources from `original_messages` (the pre-compression
snapshot at line 724), `compressed_messages` sources from
`body["messages"]` (the compressed list after in-place mutation at line
1189). Both gated symmetrically.
- **`headroom/proxy/handlers/streaming.py`** (2 sites — main streaming
in `_finalize_stream_response`, Bedrock streaming in
`_stream_response_bedrock`): same treatment. `_stream_response_bedrock`
gains a new `original_messages: list[dict] | None = None` parameter so
it has access to the pre-compression snapshot; the sole caller in
`anthropic.py` now threads it through.
- **`headroom/proxy/server.py`**: `/transformations/feed` adds
`compressed_messages` to the JSON payload alongside the existing
`request_messages` / `response_content`. *Split into a separate
preceding commit is a one-time EOL normalization to LF — the file blob
in history carries CRLF but `.gitattributes` declares `*.py text
eol=lf`, so any contributor editing `server.py` triggers the same
whole-file renormalization. Separating the two commits keeps this
feature commit's diff at a single line.*
- **`headroom/proxy/request_logger.py`**: `compressed_messages` is
stripped from the JSONL file log and from `get_recent()` alongside the
existing `request_messages` / `response_content` stripping.
`get_memory_stats()` also counts it toward the deque's byte budget.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .` and `ruff format --check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed (via the Headroom Desktop client that
consumes `/transformations/feed` — confirmed both fields arrive and
render)

Test coverage added/extended:

- `tests/test_proxy/test_request_logger.py` (new file): round-trip unit
tests for `RequestLogger`. Confirms `get_recent` strips both sides (pre
+ post), `get_recent_with_messages` exposes both, and the JSONL file log
drops both when `log_full_messages=False`.
- `tests/test_proxy/test_transformations_feed.py`: extended to assert
`compressed_messages` appears in the endpoint payload alongside
`request_messages` / `response_content`.
- `tests/test_proxy_streaming_request_logger.py`: existing include/omit
tests updated to assert both sides populate when the flag is on and both
are `None` when it's off.

## Test Output

```
$ uv run ruff check headroom tests
All checks passed!
$ uv run ruff format --check headroom tests
614 files already formatted
$ uv run pytest tests/test_proxy/test_request_logger.py tests/test_proxy_streaming_request_logger.py tests/test_proxy/test_transformations_feed.py -v
...
tests/test_proxy/test_request_logger.py::test_get_recent_strips_compressed_messages_alongside_request_and_response PASSED
tests/test_proxy/test_request_logger.py::test_get_recent_with_messages_returns_compressed_messages PASSED
tests/test_proxy/test_request_logger.py::test_jsonl_file_strips_both_sides_when_log_full_messages_disabled PASSED
tests/test_proxy_streaming_request_logger.py::test_finalize_stream_response_logs_original_and_compressed_messages PASSED
tests/test_proxy_streaming_request_logger.py::test_finalize_stream_response_omits_messages_when_log_full_messages_disabled PASSED
tests/test_proxy/test_transformations_feed.py::test_transformations_feed_returns_messages PASSED
...
============================== 11 passed in 5.36s ==============================
```

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
(the two-sided gating at each log site, the `_stream_response_bedrock`
parameter addition, and the `get_memory_stats` accounting)
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

### Non-Anthropic backends

`handlers/openai.py` and `handlers/gemini.py` do not currently emit
`RequestLog` entries at all — only Anthropic and the shared streaming
paths do. This PR therefore only populates `compressed_messages` on
Anthropic traffic (which is what `/transformations/feed` shows today).
Wiring OpenAI and Gemini into `RequestLogger` end-to-end is a separate,
larger gap worth its own PR.

### `server.py` EOL normalization

The feature change in `server.py` is a single line. To keep the diff
readable, the preceding commit is a whitespace-only `chore(proxy):
normalize server.py to LF per .gitattributes` — the file blob was stored
with CRLF terminators but `.gitattributes` declares `*.py text eol=lf`.
Any contributor touching `server.py` triggers this renormalization;
isolating it here keeps the feature commit reviewable. Happy to rebase /
drop / reshape as preferred.

### Downstream desktop compatibility

The Headroom Desktop client I work on now consumes `compressed_messages`
and renders the pre/post pair side-by-side on the "Recent large
compression" card. The desktop was updated to handle both shapes:
proxies without the field render the legacy single "Request" block;
proxies with the field render "Request (original, N tokens)" + "Request
(compressed, M tokens)" where N/M come from `input_tokens_original` /
`input_tokens_optimized`. No changes needed downstream if this PR lands.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 19:02:54 -05:00