mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-23 02:49:17 +08:00
feat/https-compose-caddy
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
5503383bef |
fix(proxy): bound brotli output per step and lossless inverse expansion (#4861)
## Description
Two places could allocate far more memory than the input size justifies:
1. **Brotli request bodies** (`_brotli_bounded` in
`headroom/proxy/helpers.py`): the body was fed to
`Decompressor.process()` in 64 KiB input slices, with the 100 MB cap
checked after each slice. One slice of a highly compressed body (a
1,681-byte body holding 1 GiB of zeros) expands in full inside a single
`process()` call before the check runs.
2. **Lossless compaction self-checks** (`compact_lossless` in
`headroom/transforms/lossless_compaction.py`): to confirm a fold
round-trips, it runs `expand_runs` / `unfold_repeated_blocks` on the
candidate. Those inverses read their repeat counts from the text, so a
marker line already present in tool output (`... (repeated 2000000
times)`, or chained `... (repeats N lines from N lines back)` lines that
double each time) made the check allocate millions of lines.
Fixes:
1. `process()` is called with `output_buffer_limit=64 KiB`. Pending
output is drained with `process(b"")` before the next slice (brotli
accepts only empty input while it holds unprocessed input) and again
after the last slice until `is_finished()`, with the cap checked after
every step. brotli older than 1.2.0 has no output limit, so it is now
refused for `br` bodies with a clear upgrade message, the same way a
brotli without `Decompressor` already was.
2. `expand_runs` and `unfold_repeated_blocks` take an optional
`max_chars`. Every line is charged its length plus its newline before it
is appended: a run costs `count * (len(line) + 1)`, and a block copy is
costed by summing the copied lines. Going over the limit raises a
private `CompressionError` subclass before anything is allocated, and
the content is returned unchanged. The extra character for the dropped
last newline is allowed only when the input has no trailing newline.
`compact_lossless` bounds each inverse by the text it must reproduce:
the content itself, or for config, the run-collapsed text that the block
inverse must rebuild (it can be longer than the content). Called without
`max_chars`, both public functions behave exactly as before.
Fix adapted from a proposal by HetCreep. Change from that proposal: with
real brotli 1.2.1, `can_accept_more_data()` is already True once all
input is consumed even while output is still pending, so stopping there
cut off normal bodies (600 KB of text failed as "truncated stream").
This version drains until `is_finished()`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/helpers.py`: `_brotli_bounded` pulls output in limited
steps, drains pending output, and requires `can_accept_more_data`
(brotli>=1.2.0).
- `headroom/transforms/lossless_compaction.py`: `_OutputBudget` and
`max_chars` on `expand_runs` / `unfold_repeated_blocks`;
`compact_lossless` passes per-stage limits and returns the content
unchanged when one is exceeded.
- `tests/test_lossless_expansion_limit.py` (new), using tracemalloc so
it works on any OS:
- a run marker (log/text/config) and chained block markers stay under an
8 MB peak;
- a 1 MiB line expanded 1000x with 1000 shrinking markers, and 1000
block copies of it, stay under 16 MB;
- the exact `max_chars` boundary holds for both inverses, with and
without a trailing newline;
- a config fold whose run-collapsed intermediate is longer than the
content is still kept;
- the block inverse on 400 KB of short lines stays under 6 MB;
- real repeats still fold.
- `tests/test_request_body_decompression_limits.py`: fake decompressors
for the per-step bound, the fail-closed path on old brotli, and draining
output that is still pending after the last slice.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
New tests on `origin/main` code:
```text
E AssertionError: peak 133.5MB: the marker count was expanded
E AssertionError: peak 217.4MB: the block markers were unfolded
E AssertionError: peak 128.0MB suggests one slice was fully expanded
FAILED tests/test_lossless_expansion_limit.py::test_run_marker_in_input_is_not_expanded[log]
FAILED tests/test_lossless_expansion_limit.py::test_run_marker_in_input_is_not_expanded[text]
FAILED tests/test_lossless_expansion_limit.py::test_run_marker_in_input_is_not_expanded[config]
FAILED tests/test_lossless_expansion_limit.py::test_chained_block_markers_in_config_are_not_expanded
FAILED tests/test_request_body_decompression_limits.py::test_brotli_step_output_is_bounded
FAILED tests/test_request_body_decompression_limits.py::test_brotli_without_output_limit_fails_closed
```
With the fix:
```text
$ pytest -q tests/test_request_body_decompression_limits.py tests/test_lossless_expansion_limit.py
======================== 32 passed, 2 skipped in 1.68s =========================
# same file with brotli 1.2.1 importable (the 2 skips are real-brotli tests):
$ pytest -q tests/test_request_body_decompression_limits.py
============================== 29 passed in 4.32s ==============================
$ pytest -q tests/test_lossless_expansion_limit.py tests/test_lossless_excluded_compaction.py tests/test_transforms_config_compressor.py tests/test_request_body_decompression_limits.py tests/test_lossless_diff_fold_guard.py tests/test_lossless_timestamp_guard.py tests/test_bash_search_lossless_fold.py
======================== 177 passed, 2 skipped in 6.90s ========================
$ ruff check / ruff format --check (0.17.3): All checks passed
$ mypy headroom/proxy/helpers.py headroom/transforms/lossless_compaction.py --ignore-missing-imports
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: macOS (Apple silicon), Python 3.12, brotli 1.2.1.
- Exact command / steps: built a 1 GiB-of-zeros brotli stream (1,681
bytes compressed) and called `_brotli_bounded`. Then round-tripped
`b""`, `b"x"`, 600 KB of text, 300 KB of random bytes, 50 MB of zeros
and 5 MB of repeated random data, plus a stream with its last 3 bytes
cut off.
- Observed result: `refused: Decompressed brotli request body exceeds
100MB`, max RSS growth 100 MB, 0.09 s. All round trips were exact. The
cut-off stream gave `Failed to decompress brotli request body: truncated
stream`.
- Lossless: `compact_lossless(("X" * 1 MiB) + "\n... (repeated 1000
times)\n" + "a\n... (repeated 1 times)\n" * 1000, "text")` returns the
input unchanged with a 3.1 MiB tracemalloc peak. With a line limit only,
it peaked at 1002 MiB.
- Not tested: a full proxy request carrying `Content-Encoding: br` (the
reader dispatches to the same function).
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: `br` request bodies are now refused
(400 with an upgrade message) when the installed `brotli` is older than
1.2.0. brotli is an optional install, and gzip/deflate/zstd are
unaffected. Lossless compaction leaves content unchanged only when it
carries markers asking for more lines than it has.
- Kill switch / disable path: upgrade `brotli` to >=1.2.0
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — n/a
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
The memory tests use tracemalloc peaks, not RLIMIT_AS, so they run on
macOS as well as Linux.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reported by @JoaoMarcos44. Thank you for your contribution! 🩵
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
50bf775e8c |
fix(proxy): bypass compression for large admitted histories (#4142)
## Description Closes #4139. A history that is too large to compress should still reach the provider. This keeps the 100 MiB hard admission limit and bypasses request-side transforms above 16 MiB decoded size. The hard limit still applies before forwarding. Current head: `62ffe3defef1684f70d96f1640818d99445a43bb`, merging current main `a2d4fe72` into `1902d8ef`. The 42-test/native/manual/pre-push results below qualify `1902d8ef`, not this new merge. New-head shard 4 has a cached-prefix fixture failure under investigation; coverage uploads are still incomplete. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement - [x] Documentation update ## Changes Made - Keep the startup override and both wire/decompressed hard bounds. Large admitted histories skip normalization, image processing, memory injection, tokenization and compression on Messages, Chat/Responses, Gemini and Bedrock HTTP routes. - Keep provider routing and backend translation, including Copilot Chat, Codex `store:false` and Vertex route-forced streaming. Native forwarding retains decoded bytes unless the provider's wire contract requires a rewrite; translation retains the full message text. - Report actual provider usage, with zero compression savings or attempted tokens for the bypass. Keep `/v1/compress` and WebSocket contracts unchanged. - Add real-provider coverage for large Gemini JSON/SSE requests with query-key auth, and OpenAI JSON/SSE through the installed LiteLLM adapter. No admission-policy or CI changes in this follow-up. ## Testing - [x] Unit tests pass (focused suite and normal critical pre-push suite) - [x] Linting passes (normal hooks) - [x] New tests added for admission, recovery and passthrough accounting - [x] Manual testing performed against the real native build ### Test Output ```sh python -m pytest -q tests/test_proxy_compression_admission.py tests/test_proxy/test_gemini_plugin_routed_streaming.py --cov=headroom --cov-branch # 42 passed, 41 warnings ``` Normal pre-push `make ci-precheck` passed: 1668 Rust tests (3 ignored), 181 Python tests, lint/type checks and commitlint. No hook bypass. The initial deep-module coverage command hit a Pydantic import-isolation error; using the repo's `--cov=headroom` convention passed both complete modules without disabling plugins or tests. The earlier `96c5fe7e` qualification covered the other native protocols, configured/per-request backends, hard-ceiling recovery and complete local shards 2/3. Those runs are historical, not a new whole-suite claim. New-head hosted success is not claimed. ## Real Behavior Proof - Environment: Linux x86_64, Python 3.13.14, freshly built real `headroom._core`, public Headroom ASGI routes, a real localhost HTTP provider and installed LiteLLM. Synthetic credentials, offline mode and `DO_NOT_TRACK=1`; no live provider calls. - Exact command / steps: Start `create_app(ProxyConfig(backend="litellm-openai", ...))` with all upstream URLs set to the localhost peer and `litellm.api_base=peer+"/v1"`. Leave the production 16 MiB soft/100 MiB hard limits unchanged. POST `"history " * (16777216 // 8 + 1024) + "LAST TURN"` as a user message to Chat Completions, then as a Gemini content part to generateContent/streamGenerateContent, with streaming off/on. The peer rejects a changed history, requires Gemini query-key auth and one `alt=sse`, and returns protocol-valid JSON/SSE with 101 input/1 output tokens. The maintained four small-ceiling cases are in `tests/test_proxy_compression_admission.py` and run with the command above. - Observed result: All four >16 MiB requests returned 200 with the full history intact. JSON/SSE and Gemini auth survived; streamGenerateContent forced SSE even with incoming `alt=json`. Each request recorded exactly 101 provider input/1 output tokens, zero compression attempts/savings and no failures; `/livez` stayed 200. Earlier recorded qualification also covered all five native HTTP protocols, real translating backends and 413 recovery. - Not tested: Live cloud accounts, the reporter's deployed 4 GB VPS, screenshot/image quality or a zero-JSON-parse path. One envelope/model JSON parse remains; no proxy-only RSS guarantee is claimed. ## Runtime Rollout Safety - Rollout-managed feature(s): None; this is the admission/compatibility fix. - Minimum rollout channel: Stable/default. - Stable/default behavior changed: Keep the 100 MiB hard ceiling; bypass request-side transforms above 16 MiB decoded size on the named HTTP routes. - Kill switch / disable path: Set `HEADROOM_REQUEST_BODY_MAX_BYTES=16777216` (or lower) and restart, accepting local 413s instead of the large-body path. - Unsafe override required: No. - Qualification impact: Local native protocols, backend translation, hard bounds and recovery exercised; live-provider/VPS qualification remains external. - Rollback path: Revert the admission revision and restart; no storage migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] I have made corresponding changes to the documentation - [x] I did not edit `CHANGELOG.md` --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
c946b6b0b0 |
fix(proxy): gate the raw request body, not just its Content-Length header (#3338)
## Description Closes #3326. The four handlers named in the issue (`openai.py`, `batch.py`, `gemini.py`, `anthropic.py`) all gate on the `Content-Length` header before calling `_read_request_json`, but that header is only present when the client sends one. A chunked-transfer-encoding request has no `Content-Length`, so it walks straight past every one of those checks, and `_read_request_body_bytes` then reads the whole body through the unbounded `request.body()`. `_read_request_json` and `_read_request_body_bytes` are the single chokepoint all six call sites route through (confirmed by reading each of the six named lines, plus a repo-wide grep for other `_read_request_json` callers), so one change at that chokepoint covers all of them. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `_read_request_body_bounded` in `headroom/proxy/helpers.py`: reads the body via `Request.stream()` instead of `Request.body()`, refusing the request with `RequestBodyTooLarge` once the running total crosses `MAX_REQUEST_BODY_SIZE`, before the rest of the body arrives. - `_read_request_body_bytes` now reads through it instead of calling `request.body()` directly. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_request_body_decompression_limits.py -q 25 passed, 2 skipped in 4.25s # 2 skipped = brotli not installed in this environment $ pytest tests/ -k "helpers or request_body or decompress or proxy_body or size_limit or content_length" -q 246 passed, 12 skipped, 3 failed in 62.57s # The 3 failures are in tests/test_cli/test_wrap_helpers.py::TestProxyClientRefCounting # (process ref-counting / PID checks) and reproduce identically against # unmodified main in this sandboxed container, unrelated to this change. $ ruff check headroom/proxy/helpers.py tests/test_request_body_decompression_limits.py tests/test_proxy_handler_helpers.py All checks passed! $ ruff format --check headroom/proxy/helpers.py tests/test_request_body_decompression_limits.py tests/test_proxy_handler_helpers.py 3 files already formatted ``` `mypy headroom --ignore-missing-imports` did not run cleanly in this environment: it fails at `numpy/__init__.pyi` with "Type statement is only supported in Python 3.12 and greater" before reaching this file, a pre-existing mismatch between the installed numpy stubs and this repo's pinned `python_version = "3.10"` mypy target, not something this diff introduces. One existing test needed a fixture update: `test_proxy_handler_helpers.py`'s `_VertexGeminiImageRequest` stand-in only implemented `.body()`, and the new code path calls `.stream()` (a real Starlette `Request` always has both). Added a `.stream()` method that yields the same body in one chunk. ## Real Behavior Proof - Environment: Docker `python:3.12-slim`, headroom-ai 0.37.0's published wheel (for the compiled `_core` extension) with this branch's `headroom/proxy/helpers.py` overlaid on top. - Exact command / steps: built a genuine Starlette `Request` over a hand-rolled chunked ASGI `receive()` channel (no `content-length` header, matching what a real chunked-transfer-encoding request looks like at the ASGI layer), and called the actual `_read_request_body_bytes` with `MAX_REQUEST_BODY_SIZE` turned down to 10KB so the demonstration does not need to stream tens of megabytes. - Observed result: against unmodified `main`, a 50,000-byte chunked body with no `Content-Length` header is read in full with no error. Against this branch, the same request raises `RequestBodyTooLarge` before the stream finishes, and a 5,000-byte chunked body under the cap still round-trips byte for byte. - Not tested: brotli-encoded bodies (package not installed in this sandbox, same as the two skipped unit tests above), and the repo's own CI matrix (Rust extension build, HuggingFace model prefetch) end to end. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a. - Stable/default behavior changed: no for any request under `MAX_REQUEST_BODY_SIZE` (100MB by default); a chunked request that would previously have been read unbounded is now refused past that size, which is the fix. - Kill switch / disable path: none by design; raising `MAX_REQUEST_BODY_SIZE` widens the ceiling if an operator needs to. - Unsafe override required: none. - Qualification impact: none. - Rollback path: revert this commit, one function plus a two-line call-site change in a single file. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did not edit `CHANGELOG.md` ## Additional Notes Not in scope: a handful of other call sites in these same handler files (`batch.py`'s and `bedrock.py`'s passthrough forwarders, and two streaming passthrough paths in `openai.py`/`anthropic.py`) also call `request.body()` directly rather than through this chokepoint. None of them are among the six lines the issue names, they exist to forward a request byte-for-byte rather than to parse it as JSON, and fixing them is a larger, separate change than this one. Flagging it here rather than leaving it implicit. --------- Signed-off-by: Amir Fathi <amirfathi.me@gmail.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
690fb251dc |
harden: fix security issue in server.py (#3479)
## Summary
Harden input handling in `headroom/proxy/server.py` (flagged by
multi_agent_ai).
## Vulnerability
| Field | Value |
|-------|-------|
| **ID** | V-001 |
| **Severity** | HIGH |
| **Scanner** | multi_agent_ai |
| **Rule** | `V-001` |
| **File** | `headroom/proxy/server.py:3803` |
| **Assessment** | Defensive hardening |
**Description**: The proxy server parses JSON request bodies using
Starlette's request.json() at four locations without visible enforcement
of the MAX_REQUEST_BODY_SIZE constant defined in helpers.py. While the
configuration defines size limits, the actual enforcement in the request
handling path requires verification. Without guaranteed size limits,
attackers can send oversized JSON payloads causing memory exhaustion.
## Threat Model Context
This server appears to be publicly accessible. This is a web service -
vulnerabilities in request handlers are directly exploitable by remote
attackers.
## Changes
- `headroom/proxy/server.py`
## Behavior Preservation
The change is scoped to 1 file on the vulnerable path.
## Security Invariant
> **Property**: The security boundary is maintained under adversarial
input
<details>
<summary>Regression test</summary>
```python
import pytest
import asyncio
from unittest.mock import AsyncMock, MagicMock
import sys
import os
# Ensure the module can be imported
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from headroom.proxy.server import MAX_REQUEST_BODY_SIZE
@pytest.mark.parametrize("payload_size", [
MAX_REQUEST_BODY_SIZE - 1, # valid: just under limit
MAX_REQUEST_BODY_SIZE, # boundary: exactly at limit
MAX_REQUEST_BODY_SIZE + 1, # exploit: exceeds limit
])
def test_request_body_size_enforced(payload_size):
"""Invariant: Request body size MUST NOT exceed MAX_REQUEST_BODY_SIZE."""
# Verify the constant exists and is reasonable
assert isinstance(MAX_REQUEST_BODY_SIZE, int)
assert MAX_REQUEST_BODY_SIZE > 0
# The security property: any payload at or below limit is acceptable,
# anything above MUST be rejected. This test documents the boundary.
is_acceptable = payload_size <= MAX_REQUEST_BODY_SIZE
# For the exploit case, this MUST be False (would be rejected by proper enforcement)
if payload_size > MAX_REQUEST_BODY_SIZE:
assert not is_acceptable, f"Payload size {payload_size} exceeds limit {MAX_REQUEST_BODY_SIZE}"
else:
assert is_acceptable, f"Payload size {payload_size} should be within limit {MAX_REQUEST_BODY_SIZE}"
```
</details>
This test guards against regressions — it's useful independent of the
code change above.
---
*This patch removes an exploit primitive — a code pattern that, while
not independently exploitable today, could be chained with other
weaknesses by automated exploit-development tooling. Proactive removal
of such primitives raises the bar against increasingly capable automated
attack tools.*
---
*Automated security fix by [OrbisAI Security](https://orbisappsec.com)*
<!-- orbis-meta: rule=V-001 scanner=multi_agent_ai -->
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
|
25a4e71b29 |
fix(proxy): bound decompressed request bodies to close a zip-bomb DoS (#3325)
## Description Closes #3284. The handlers gate on `Content-Length` — the **compressed** wire size — against `MAX_REQUEST_BODY_SIZE`, but every codec then decompressed with a one-shot call (`gzip.decompress`, `zlib.decompress`, `brotli.decompress`, `stream_reader.read()`) that materializes the whole expansion before anything can inspect it. A client could buy unlimited capacity with a compression ratio. I reproduced it end to end. `zstd` is considerably worse than the reported gzip case: | codec | wire size | expands to | |---|---|---| | gzip | 1.99 MB | 2 GB | | zstd | **16 KB** | 512 MB | At 16 KB the attack costs essentially no bandwidth, and one request is enough. All four codecs converge on a single chokepoint, `_read_request_body_bytes`, which `_read_request_json` also funnels through — so one change covers every caller. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `MAX_DECOMPRESSED_BODY_SIZE`, set to `MAX_REQUEST_BODY_SIZE` deliberately: nobody should get more room by arriving compressed. - Decompress incrementally, re-checking the cap after every 64 KiB chunk, so peak allocation is the cap plus one chunk rather than the full expansion: - **gzip/deflate** — `zlib.decompressobj` with a `max_length`, which leaves unproduced output in `unconsumed_tail` instead of memory. - **zstd** — `stream_reader` read in chunks. - **brotli** — `Decompressor` fed in slices. - `RequestBodyTooLarge` subclasses `ValueError`, so every existing `except ValueError` call site keeps answering 400 unchanged while a caller that would rather answer 413 has a type to branch on. Re-raised ahead of the generic handlers so the size refusal isn't reworded into a vague "failed to decompress". Behaviour deliberately preserved: - **Multi-member gzip** still concatenates, matching `gzip.decompress`. A plain `decompressobj` stops after the first member, which would have silently truncated such bodies. - **Truncated streams still raise** rather than yielding a short body — losing the one-shot call must not turn a corrupt request into a silently empty one. - An **old brotli** without `Decompressor` now fails closed with a clear message instead of falling back to the unbounded call. An outdated library is not a reason to reopen the hole. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (CI-pinned `ruff` 0.16.3) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_request_body_decompression_limits.py 16 passed, 2 skipped in 1.20s # 2 skipped = brotli not installed locally $ pytest tests/test_proxy/ tests/test_proxy_hardening.py tests/test_request_body_decompression_limits.py 311 passed, 2 skipped in 48.45s $ pytest tests/ -k "body or compression or forwarding or passthrough or helpers or batch or gemini" 1538 passed, 201 skipped, 10589 deselected in 85.42s $ uvx ruff@0.16.3 check headroom/proxy/helpers.py tests/test_request_body_decompression_limits.py All checks passed! $ mypy headroom/proxy/helpers.py Success: no issues found in 1 source file ``` The tests assert the property that actually matters — **refused without being materialized** — not merely "an oversized body is rejected". A guard placed *after* a one-shot `decompress()` would satisfy the latter while still OOM-ing the process, so `test_bomb_never_materializes` asserts on peak allocation via `tracemalloc`. Verified they catch the bug by swapping the old one-shot implementation back in behind the same signature: ```text FAILED test_gzip_bomb_is_refused FAILED test_deflate_bomb_is_refused FAILED test_bomb_never_materializes[_gzip_bomb-<lambda>] FAILED test_bomb_never_materializes[_deflate_bomb-<lambda>] FAILED test_truncated_gzip_still_errors FAILED test_reader_refuses_a_bomb 6 failed, 10 passed, 2 skipped ``` ## Real Behavior Proof - **Environment:** macOS arm64, Python 3.12.13, `main` @ 0.37.0. - **Exact command / steps:** build real bombs by streaming a compressor (so the fixture itself never holds the expansion), then call `_read_request_body_bytes` through a stand-in Request, measuring peak allocation with `tracemalloc`. - **Observed result**, 400 MB bomb from 398 KB of wire: ```text OLD gzip.decompress() peak = 829.8 MB (materialized the whole bomb) NEW bounded peak = 105.0 MB (refused at the cap) ``` A 2 GB bomb is refused in 185 ms with peak 108.2 MB. End to end through the real entry point, both gzip and zstd bombs raise `RequestBodyTooLarge`, while ordinary compressed bodies, identity bodies, and unknown-encoding rejection all behave exactly as before. - **Not tested:** brotli is not installed in this environment, so its two tests skip locally and are exercised in CI where the extra is present. The implementation is the same shape as the others, but I have not run it. ## Known gap, deliberately not in scope The `Content-Length` gate is skipped entirely for **chunked** requests (no `Content-Length` header), so `await request.body()` will still read an arbitrarily large *compressed* body into memory. That is a real but much weaker vector — it needs bandwidth proportional to the damage, with no amplification, which is what made the bomb dangerous. Bounding the raw read affects every request path including legitimate large uploads, so it deserves its own change rather than riding along here. Filed as #3326. ## Runtime Rollout Safety - **Rollout-managed feature(s):** none. - **Minimum rollout channel:** n/a. - **Stable/default behavior changed:** no for any body under the cap — bodies round-trip byte-identically, including multi-member gzip. The only behavior change is that a body decompressing past 100 MB is now refused instead of being allocated, which is the bug. - **Kill switch / disable path:** none by design; raising `MAX_DECOMPRESSED_BODY_SIZE` widens the ceiling if an operator genuinely needs it. - **Unsafe override required:** none. - **Qualification impact:** none. - **Rollback path:** revert this commit — one function plus three helpers in a single file. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |