mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
994a89f169 |
fix(proxy): bound request-log message payloads to the feed window; trim glibc too (#3717)
## Description
A user running an overnight Claude Code session through the proxy on
Linux reported it consuming 100 GB of a 128 GB host (about 3,500
requests at ~92k input tokens each, `--log-messages` on). Two stacked
causes, both fixed here:
1. `RequestLogger` keeps message payloads on all 10,000 entries. With
`--log-messages`, every `RequestLog` in the `MAX_LOG_ENTRIES` deque
holds `request_messages`, `compressed_messages` and `response_content`:
two parsed copies of the whole conversation, ~1.9x their JSON size as
Python objects. The only reader of those payloads is
`/transformations/feed`, which is hard-capped at 100 entries, so the
other 9,900 are retained for nothing and grow linearly for the life of
the process.
2. The periodic allocator trim (#2820, #2879) defaults on only for
macOS. glibc retains freed pages the same way, and per-thread arenas on
a many-core host multiply the high-water mark.
Not changed: `MAX_LOG_ENTRIES` itself (`_build_stats_payload` reads the
light fields of all 10,000), and `get_recent_with_messages(n)` for `n >
100`, which now returns bodies only for the newest 100, matching what
the feed could ever serve.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/request_logger.py`: `MESSAGE_WINDOW = 100`; on every
append, null the three payload fields on the entry that just left the
window. Light fields stay on every entry, so `get_recent` and `/stats`
are unchanged.
- `headroom/proxy/models.py`, `headroom/proxy/server.py`:
`periodic_malloc_trim_enabled` defaults on for `darwin` and `linux` (was
`darwin` only). Platforms without a trim call already disable the task
themselves; `HEADROOM_MALLOC_TRIM=0` still opts out.
- `tests/test_proxy/test_request_logger.py`: window tests (payloads
nulled beyond the newest 100, light fields intact and ordered, feed path
fully populated, short logs untouched).
- `tests/test_malloc_tuning.py`: default-scope test asserts darwin and
linux on, win32 off.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py -q
22 passed in 1.10s
$ uvx ruff check headroom/proxy/request_logger.py headroom/proxy/models.py headroom/proxy/server.py tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py
All checks passed!
$ uvx ruff format --check <same files>
5 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/request_logger.py headroom/proxy/models.py
Success: no issues found
```
## Real Behavior Proof
- Environment: macOS 15.6 (Darwin 24.6.0), Python 3.12, this branch at
|
||
|
|
46ac52d39d |
feat(proxy): let /transformations/feed omit message bodies for number-only pollers (#3672)
## Description `/transformations/feed` always returns `request_messages`, `compressed_messages` and `response_content` for every entry, and `RequestLogger.get_recent_with_messages` builds them with `asdict()`, which deep-copies each transcript first. A poller that only reads the per-request numbers (tokens, model, transforms) still pays for a full copy and a multi-MB JSON serialization on the event loop for each pull. Measured on a desktop client polling every 20 s: a `limit=100` pull is ~44 MB, of which ~400 bytes per entry are read, and three concurrent pulls take `/stats?cached=1` from 45 ms to 1.3 s. This adds `?include_messages=0` (default `1`, byte-identical output to today). When off, the three body fields are omitted from the response and never walked server-side. Closes # ## Type of Change - [x] Performance improvement ## Changes Made - `RequestLogger.get_recent_with_messages(n, include_messages=True)`: when `False`, builds each entry from the dataclass fields minus the three payload fields, deep-copying only what it returns (same idiom as #3613 for `get_recent`). - `/transformations/feed` accepts `include_messages: bool = True` and omits the three body keys when it is off. Everything else in the item, including `log_full_messages`, is unchanged. - Each item also carries `uncached_input_tokens`, `cache_write_tokens` and `cache_read_tokens` (already stored on `RequestLog`, never exposed by the feed). A number-only poller can then rate `tokens_saved` against new input (uncached + cache_write) the way `/stats` `new_input_savings_percent` does; the item's full-transcript `savings_percent` is unchanged. Present in both the full and the slim response. - Tests: logger test proves the payloads are not traversed (a leaf whose `__deepcopy__` raises); endpoint test proves the slim item equals the full item minus the three keys and carries the cache split. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_proxy/test_transformations_feed.py tests/test_proxy/test_request_logger.py -q 9 passed in 4.53s $ uvx ruff check headroom/proxy/request_logger.py headroom/proxy/server.py tests/test_proxy/test_transformations_feed.py tests/test_proxy/test_request_logger.py All checks passed! $ uv run --frozen --extra dev mypy headroom/proxy/request_logger.py headroom/proxy/server.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS 15, Python 3.12, this branch, in-process `RequestLogger` seeded with 100 entries carrying ~160 KB of messages each (the size of a mid-session Claude Code turn). - Exact command / steps: `get_recent_with_messages(100)` and `get_recent_with_messages(100, include_messages=False)`, timed, then `json.dumps` of each result, timed and sized. - Observed result: include_messages=True: copy 29.4 ms, json 56.5 ms, 24.45 MB. include_messages=False: copy 1.3 ms, json 1.9 ms, 0.05 MB. Same 100 entries, same numeric fields. - Not tested: the dashboard UI (it does not pass the parameter, so it gets the default, unchanged response). ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a (opt-in query parameter) - Stable/default behavior changed: additive only; without the parameter every existing key is byte-identical, and three integer keys are added to each item - Kill switch / disable path: callers stop passing `include_messages=0` - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit; no persisted state or schema involved ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Documentation: the endpoint has no reference page; the parameter is documented in the handler docstring. - Overlaps #3613 on the `_HEAVY_FIELDS` constant and the import lines; both hunks are identical so either merge order applies cleanly. |
||
|
|
44b8e7ba87 |
fix(proxy): stop get_recent deep-copying dropped message payloads (#3613)
## Description `RequestLogger.get_recent()` built each returned row with `dataclasses.asdict(entry)` and only then dropped `request_messages`, `compressed_messages` and `response_content`. `asdict` deep-copies every field first, so every call walked the full request and compressed transcripts of every entry it returned, for fields it was about to discard. `/stats` calls `get_recent(10_000)` synchronously on the event loop on every build (both the uncached path and the `?cached=1` snapshot rebuild). Measured against 0.37.0 with a 300-message Claude Code transcript shape (~400 KB per entry): 3.4 ms per entry, so a full 10k deque costs ~34 s per `/stats` call, during which the loop serves nothing else. A desktop dashboard polling `/stats` reproduced this in production: 4,769 `/stats` calls in 55 min with a mean latency of 517 s, `/v1/messages` mean 135 s (max 1,839 s), 61k `Exceeded concurrency limit` warnings and a watchdog restart storm (Headroom Desktop user report, 2026-09-16). The fix builds the row from `fields()` while skipping the heavy fields, so the payloads are never touched. Kept fields are scalars plus flat `dict`/`list` containers; those get the same shallow copy `asdict` produced for them, so callers still get copies rather than aliases into the log entry. Fixes the backend half of the desktop report; the desktop side stops polling uncached `/stats` for connector verification separately. ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Changes Made - `headroom/proxy/request_logger.py`: `get_recent` iterates `fields()` and skips `_HEAVY_FIELDS` instead of calling `asdict()` and filtering afterwards. Shallow-copies the kept `dict`/`list` fields. - `tests/test_proxy/test_request_logger.py`: new test plants a leaf whose `__deepcopy__` raises inside both message payloads and asserts `get_recent` still returns the row with copied (not aliased) containers. Fails on the previous implementation, passes on this one. ## Testing - [x] Unit tests pass locally (`uv run --frozen --extra dev pytest tests/test_proxy/test_request_logger.py`, 5 passed; `pytest tests/test_proxy -k "stats and not history"`, 6 passed) - [x] `uvx ruff check` and `uvx ruff format --check` clean on both files - [x] New test verified to fail on the old code (`git stash` the source change: 1 failed, 4 passed) ### Test Output ``` $ uv run --frozen --extra dev pytest tests/test_proxy/test_request_logger.py -q tests/test_proxy/test_request_logger.py ..... [100%] ============================== 5 passed in 0.23s =============================== $ uvx ruff check headroom/proxy/request_logger.py tests/test_proxy/test_request_logger.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS 15, Python 3.12, headroom-ai 0.37.0 venv (the desktop app's pinned runtime), plus this branch. - Exact command / steps: Built a `RequestLog` with `request_messages`/`compressed_messages` set to a 300-message transcript (3 content blocks each, ~416 KB serialized) and timed `dataclasses.asdict(entry)` in a loop; then ran the new unit test against the old and new `get_recent`. - Observed result: 3.4 ms per `asdict` call on 0.37.0, i.e. ~34 s for the 10,000-entry `get_recent` that `/stats` performs on the event loop. With this change the per-row cost is a `fields()` walk plus shallow copies of two small containers (microseconds), and the new test passes only on the new code. - Not tested: an end-to-end `/stats` latency comparison on a long-lived proxy with a full 10k deque; the production numbers above come from a user's `proxy.log`, not from a controlled run. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: no; `/stats` returns the same rows and keys, they are just built without walking the dropped payloads - Kill switch / disable path: none needed; revert the commit - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes ## Screenshots n/a ## Additional Notes `get_recent_with_messages` keeps `asdict` on purpose: it returns the payloads, so the walk is the point there. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
2269e40bde |
feat(proxy): log compressed messages alongside original request (#261)
## Description Expose the post-compression message list that was actually sent upstream as a new `compressed_messages` field on `RequestLog`, paired with the existing (now consistently pre-compression) `request_messages`. Consumers of `/transformations/feed` — dashboards and any downstream observability — can now diff the two sides of a compression to see exactly what the pipeline stripped, replaced, or kept. Turns an abstract "saved N tokens" into a legible before/after. Gated by the same `log_full_messages` flag as `request_messages` so the two sides stay in sync; it's pointless to store one without the other. Also fixes a latent correctness bug: today's `request_messages` field is inconsistent across the four `RequestLog` construction sites — sometimes it's the pre-compression snapshot, sometimes it's the mutated `body["messages"]` (which is the compressed list, because the proxy mutates `body` in place before the log call). After this change, `request_messages` always means pre-compression and `compressed_messages` always means what went upstream. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) Note on "breaking": strictly speaking this is a semantic correction of an inconsistently-populated field, not a schema break. The field name `request_messages` is unchanged and the JSON shape is unchanged; what changes is that the field now consistently holds the pre-compression list. Consumers that treated it as "whatever messages we have" continue to work. Consumers that depended on the accidental post-compression value (if any existed) would shift to `compressed_messages`. ## Changes Made - **`headroom/proxy/models.py`**: `RequestLog` gains `compressed_messages: list[dict] | None = None`. Doc comment explains it's paired with `request_messages` and gated by the same `log_full_messages` flag. - **`headroom/proxy/handlers/anthropic.py`** (2 sites — Bedrock non-streaming and main non-streaming): `request_messages` now consistently sources from `original_messages` (the pre-compression snapshot at line 724), `compressed_messages` sources from `body["messages"]` (the compressed list after in-place mutation at line 1189). Both gated symmetrically. - **`headroom/proxy/handlers/streaming.py`** (2 sites — main streaming in `_finalize_stream_response`, Bedrock streaming in `_stream_response_bedrock`): same treatment. `_stream_response_bedrock` gains a new `original_messages: list[dict] | None = None` parameter so it has access to the pre-compression snapshot; the sole caller in `anthropic.py` now threads it through. - **`headroom/proxy/server.py`**: `/transformations/feed` adds `compressed_messages` to the JSON payload alongside the existing `request_messages` / `response_content`. *Split into a separate preceding commit is a one-time EOL normalization to LF — the file blob in history carries CRLF but `.gitattributes` declares `*.py text eol=lf`, so any contributor editing `server.py` triggers the same whole-file renormalization. Separating the two commits keeps this feature commit's diff at a single line.* - **`headroom/proxy/request_logger.py`**: `compressed_messages` is stripped from the JSONL file log and from `get_recent()` alongside the existing `request_messages` / `response_content` stripping. `get_memory_stats()` also counts it toward the deque's byte budget. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .` and `ruff format --check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed (via the Headroom Desktop client that consumes `/transformations/feed` — confirmed both fields arrive and render) Test coverage added/extended: - `tests/test_proxy/test_request_logger.py` (new file): round-trip unit tests for `RequestLogger`. Confirms `get_recent` strips both sides (pre + post), `get_recent_with_messages` exposes both, and the JSONL file log drops both when `log_full_messages=False`. - `tests/test_proxy/test_transformations_feed.py`: extended to assert `compressed_messages` appears in the endpoint payload alongside `request_messages` / `response_content`. - `tests/test_proxy_streaming_request_logger.py`: existing include/omit tests updated to assert both sides populate when the flag is on and both are `None` when it's off. ## Test Output ``` $ uv run ruff check headroom tests All checks passed! $ uv run ruff format --check headroom tests 614 files already formatted $ uv run pytest tests/test_proxy/test_request_logger.py tests/test_proxy_streaming_request_logger.py tests/test_proxy/test_transformations_feed.py -v ... tests/test_proxy/test_request_logger.py::test_get_recent_strips_compressed_messages_alongside_request_and_response PASSED tests/test_proxy/test_request_logger.py::test_get_recent_with_messages_returns_compressed_messages PASSED tests/test_proxy/test_request_logger.py::test_jsonl_file_strips_both_sides_when_log_full_messages_disabled PASSED tests/test_proxy_streaming_request_logger.py::test_finalize_stream_response_logs_original_and_compressed_messages PASSED tests/test_proxy_streaming_request_logger.py::test_finalize_stream_response_omits_messages_when_log_full_messages_disabled PASSED tests/test_proxy/test_transformations_feed.py::test_transformations_feed_returns_messages PASSED ... ============================== 11 passed in 5.36s ============================== ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas (the two-sided gating at each log site, the `_stream_response_bedrock` parameter addition, and the `get_memory_stats` accounting) - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes ### Non-Anthropic backends `handlers/openai.py` and `handlers/gemini.py` do not currently emit `RequestLog` entries at all — only Anthropic and the shared streaming paths do. This PR therefore only populates `compressed_messages` on Anthropic traffic (which is what `/transformations/feed` shows today). Wiring OpenAI and Gemini into `RequestLogger` end-to-end is a separate, larger gap worth its own PR. ### `server.py` EOL normalization The feature change in `server.py` is a single line. To keep the diff readable, the preceding commit is a whitespace-only `chore(proxy): normalize server.py to LF per .gitattributes` — the file blob was stored with CRLF terminators but `.gitattributes` declares `*.py text eol=lf`. Any contributor touching `server.py` triggers this renormalization; isolating it here keeps the feature commit reviewable. Happy to rebase / drop / reshape as preferred. ### Downstream desktop compatibility The Headroom Desktop client I work on now consumes `compressed_messages` and renders the pre/post pair side-by-side on the "Recent large compression" card. The desktop was updated to handle both shapes: proxies without the field render the legacy single "Request" block; proxies with the field render "Request (original, N tokens)" + "Request (compressed, M tokens)" where N/M come from `input_tokens_original` / `input_tokens_optimized`. No changes needed downstream if this PR lands. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |