mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat/https-compose-caddy
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
994a89f169 |
fix(proxy): bound request-log message payloads to the feed window; trim glibc too (#3717)
## Description
A user running an overnight Claude Code session through the proxy on
Linux reported it consuming 100 GB of a 128 GB host (about 3,500
requests at ~92k input tokens each, `--log-messages` on). Two stacked
causes, both fixed here:
1. `RequestLogger` keeps message payloads on all 10,000 entries. With
`--log-messages`, every `RequestLog` in the `MAX_LOG_ENTRIES` deque
holds `request_messages`, `compressed_messages` and `response_content`:
two parsed copies of the whole conversation, ~1.9x their JSON size as
Python objects. The only reader of those payloads is
`/transformations/feed`, which is hard-capped at 100 entries, so the
other 9,900 are retained for nothing and grow linearly for the life of
the process.
2. The periodic allocator trim (#2820, #2879) defaults on only for
macOS. glibc retains freed pages the same way, and per-thread arenas on
a many-core host multiply the high-water mark.
Not changed: `MAX_LOG_ENTRIES` itself (`_build_stats_payload` reads the
light fields of all 10,000), and `get_recent_with_messages(n)` for `n >
100`, which now returns bodies only for the newest 100, matching what
the feed could ever serve.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/request_logger.py`: `MESSAGE_WINDOW = 100`; on every
append, null the three payload fields on the entry that just left the
window. Light fields stay on every entry, so `get_recent` and `/stats`
are unchanged.
- `headroom/proxy/models.py`, `headroom/proxy/server.py`:
`periodic_malloc_trim_enabled` defaults on for `darwin` and `linux` (was
`darwin` only). Platforms without a trim call already disable the task
themselves; `HEADROOM_MALLOC_TRIM=0` still opts out.
- `tests/test_proxy/test_request_logger.py`: window tests (payloads
nulled beyond the newest 100, light fields intact and ordered, feed path
fully populated, short logs untouched).
- `tests/test_malloc_tuning.py`: default-scope test asserts darwin and
linux on, win32 off.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py -q
22 passed in 1.10s
$ uvx ruff check headroom/proxy/request_logger.py headroom/proxy/models.py headroom/proxy/server.py tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py
All checks passed!
$ uvx ruff format --check <same files>
5 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/request_logger.py headroom/proxy/models.py
Success: no issues found
```
## Real Behavior Proof
- Environment: macOS 15.6 (Darwin 24.6.0), Python 3.12, this branch at
|
||
|
|
96c25f5181 |
fix(cli): stop the macOS malloc re-exec replacing an embedder's process (#3064)
## Description **`main` cannot currently run its own test suite on macOS.** `pytest tests/` dies at roughly 2% with exit code 2 — no traceback, no summary, no failing test named. The pytest process is simply gone. Two independent defects, both landed today, both invisible to CI. ### 1. The macOS malloc re-exec replaces the calling process `headroom proxy` re-execs itself once on Darwin to apply two libmalloc knobs that libmalloc only reads before `main()` (#2820, PR #2879): ```python os.execv(sys.executable, [sys.executable, "-m", "headroom.cli", *sys.argv[1:]]) ``` That reconstruction is only faithful when the process really *is* the Headroom CLI. Ten-plus test files invoke the `proxy` command in-process through Click's `CliRunner`. There, `os.execv` replaces **pytest** with a Headroom process holding pytest's argv. Run with `-s`, the mechanism is visible: ``` tests/test_agent_savings.py Usage: python -m headroom.cli [OPTIONS] COMMAND [ARGS]... Error: No such command 'tests/test_agent_savings.py::test_proxy_cli_reads_agent_90_profile_env'. ``` Everything after the first such test — roughly 98% of the suite — never runs. The same hazard applies to any application embedding the CLI. **The documented kill switch does not help.** `tests/conftest.py:41` scrubs every `HEADROOM_*` variable for hermeticity, so `HEADROOM_MALLOC_TUNING` is deleted before the guard reads it. Only the private `_HEADROOM_MALLOC_TUNED` survives, because it starts with an underscore. **CI could not have caught this.** The tuning is Darwin-only, and while the repo *does* have macOS jobs (`macos-native-wrapper`, `wrap-native (macos-latest)`), neither runs the Python test suite — the `test` shards are `ubuntu-latest` only. So `sys.platform != "darwin"` returns first everywhere pytest actually runs. #2879 merged with 37 green checks. ### 2. A semantic merge conflict between two green PRs #3051 added `bind_scope(tags, request.scope)` at `gemini.py:325` and updated the three Gemini fakes it knew about. #3035 branched earlier and added a fourth `_FakeRequest` without `.scope`. Each was green against its own base; together they fail: ``` AttributeError: '_FakeRequest' object has no attribute 'scope' ``` Git merged both cleanly. Only running the suite on merged `main` surfaces it. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update ## Changes Made - Added `_process_is_headroom_cli_entrypoint()`: the re-exec now verifies its own precondition — `argv[0]` must be the `headroom` console script or `headroom/cli/__main__.py`. - The embedded path returns **before** stamping `_HEADROOM_MALLOC_TUNED`, so a genuine CLI child inheriting the environment can still apply the tuning. - Gave the Gemini `_FakeRequest` the `.scope` every real Starlette `Request` carries. - `test_reexec_skips_when_operator_already_set_vars` now sets a realistic `argv[0]`, matching its sibling exec test. - New `tests/test_cli_proxy_malloc_reexec_guard.py` asserting the guard's logic on **every** platform, since no CI runner is macOS. ## Testing - [x] Unit tests pass - [x] Linting passes (ruff check + format) - [ ] Type checking passes (`uv run mypy headroom`) — not run - [x] New tests added for new functionality ### Test Output Before, on `main`: ```text $ .venv/bin/python -m pytest tests/ -q collected 11622 items / 8 skipped ... tests/test_agent_savings.py ............................ $ echo $? 2 ``` No summary line — the run does not end, it is replaced. After, on this branch: ```text $ .venv/bin/python -m pytest tests/ -q 3 failed, 11055 passed, 581 skipped, 6034 warnings in 303.69s (0:05:03) ``` All three remaining failures reproduce at `f9807fd6`, before today's merges, and are unrelated: | test | cause | |---|---| | `test_graceful_shutdown::test_run_server_installs_cancelled_error_filter` | full-suite ordering; passes in isolation (11 passed) | | `test_learn/test_integration::TestCodexIntegration::test_full_pipeline` | pre-existing | | `test_release_workflows::test_no_native_tls_in_wheel_build_tree` | requires `cargo`, absent on this host | ## Real Behavior Proof - Environment: macOS 15 (darwin 25.4.0), Python 3.12.13, arm64, real checkout of `main` at `ef7e07e0`. - Exact command / steps: bisected the crash to a single test, then to a single commit — `be5b26d8` (parent) exits 0, `6d87825f` (#2879) exits 2. Confirmed causation by temporarily replacing the `os.execv` line with `return`, which makes the test pass. Recovered the mechanism by running the crashing test with `-s`, which prints the Headroom CLI rejecting pytest's own argv. - Observed result: on `main` the suite cannot reach a summary; on this branch it completes with 11,055 passing. The two-file reproduction (`test_agent_savings.py` + `test_anthropic_beta_session_sticky.py`) goes from exit 2 to 62 passed. - Not tested: a real `headroom proxy` launch on macOS confirming libmalloc still receives the knobs after re-exec. The guard is covered by unit tests asserting `execv` is still called with `["-m", "headroom.cli", "proxy", "--port", "8787"]` for a console-script `argv[0]`, but I have not watched `vmmap` on a live proxy. **A macOS maintainer should confirm #2820's RSS fix still works end to end before this ships.** ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: no for a real CLI launch; the re-exec no longer fires when the CLI is invoked in-process, which was never intended to work. - Kill switch / disable path: `HEADROOM_MALLOC_TUNING=0` still disables the tuning outright. - Unsafe override required: none. - Qualification impact: none. - Rollback path: revert this commit — but that restores a `main` whose test suite cannot run on macOS. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes ## Additional Notes **This is my fault and worth recording.** I merged both #2879 and #3035 earlier today on the rule "approved + green CI". Both were genuinely approved and genuinely green. Neither was rebased onto current `main` first, and CI has no macOS runner, so green meant less than it appeared to. Two process gaps this exposes, neither of which this PR fixes: 1. **The Python test suite never runs on macOS.** The repo has macOS jobs (`macos-native-wrapper`, `wrap-native (macos-latest)`), but the `test` shards are `ubuntu-latest` only, so Darwin-only code paths — the allocator tuning is one, `wrap` has others — are unreachable by pytest in CI. Even a reduced macOS shard would have caught this. 2. **Nothing requires a PR to be current with `main` before merging.** Both defects here are cross-PR interactions that no per-PR check can see. Enabling "require branches to be up to date before merging" on `main` would have forced a rebase and surfaced the Gemini fake. I would suggest an issue for each rather than folding them in here. Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|
|
6d87825f62 |
fix(proxy): tune macOS libmalloc and trim allocator pages so long-lived RSS stays bounded (#2879)
## Summary Fixes #2820. Prevents long-lived macOS proxies from retaining every largest transient request-body allocation in libmalloc. The reporter’s production A/B isolated the allocator behavior and verified the two pre-main libmalloc knobs; this PR applies them through a one-time Darwin-only re-exec and adds periodic per-worker pressure relief. - `MallocAggressiveMadvise=1` returns freed pages eagerly. - `MallocLargeCache=0` disables the large-allocation death-row cache. - Operator-set allocator variables are preserved; `HEADROOM_MALLOC_TUNING=0` is the kill switch. - Periodic trim defaults on only for macOS, runs off the event loop, performs no forced Python GC, validates its interval, and is retained/cancelled through the app lifecycle. - Non-Darwin behavior remains unchanged unless explicitly enabled. - Semantically rebased onto current `main`, retaining startup dependency validation, MCP SDK v1 compatibility, and all newer proxy behavior. ## Verification - 147 proxy CLI/config/malloc/MCP-contract tests pass; 1 platform skip. - Ruff check and formatting clean; `git diff --check` clean. - The reporter’s macOS A/B reduced dirty empty malloc regions to zero and lowered steady/startup RSS; the control flow and shutdown lifecycle are covered locally. ## Safety The re-exec is Darwin-only, PID-preserving, loop-guarded, and opt-out. The trim task is per worker because allocator state is per process, and shutdown cancels it explicitly. |