mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description
Two places could allocate far more memory than the input size justifies:
1. **Brotli request bodies** (`_brotli_bounded` in
`headroom/proxy/helpers.py`): the body was fed to
`Decompressor.process()` in 64 KiB input slices, with the 100 MB cap
checked after each slice. One slice of a highly compressed body (a
1,681-byte body holding 1 GiB of zeros) expands in full inside a single
`process()` call before the check runs.
2. **Lossless compaction self-checks** (`compact_lossless` in
`headroom/transforms/lossless_compaction.py`): to confirm a fold
round-trips, it runs `expand_runs` / `unfold_repeated_blocks` on the
candidate. Those inverses read their repeat counts from the text, so a
marker line already present in tool output (`... (repeated 2000000
times)`, or chained `... (repeats N lines from N lines back)` lines that
double each time) made the check allocate millions of lines.
Fixes:
1. `process()` is called with `output_buffer_limit=64 KiB`. Pending
output is drained with `process(b"")` before the next slice (brotli
accepts only empty input while it holds unprocessed input) and again
after the last slice until `is_finished()`, with the cap checked after
every step. brotli older than 1.2.0 has no output limit, so it is now
refused for `br` bodies with a clear upgrade message, the same way a
brotli without `Decompressor` already was.
2. `expand_runs` and `unfold_repeated_blocks` take an optional
`max_chars`. Every line is charged its length plus its newline before it
is appended: a run costs `count * (len(line) + 1)`, and a block copy is
costed by summing the copied lines. Going over the limit raises a
private `CompressionError` subclass before anything is allocated, and
the content is returned unchanged. The extra character for the dropped
last newline is allowed only when the input has no trailing newline.
`compact_lossless` bounds each inverse by the text it must reproduce:
the content itself, or for config, the run-collapsed text that the block
inverse must rebuild (it can be longer than the content). Called without
`max_chars`, both public functions behave exactly as before.
Fix adapted from a proposal by HetCreep. Change from that proposal: with
real brotli 1.2.1, `can_accept_more_data()` is already True once all
input is consumed even while output is still pending, so stopping there
cut off normal bodies (600 KB of text failed as "truncated stream").
This version drains until `is_finished()`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/helpers.py`: `_brotli_bounded` pulls output in limited
steps, drains pending output, and requires `can_accept_more_data`
(brotli>=1.2.0).
- `headroom/transforms/lossless_compaction.py`: `_OutputBudget` and
`max_chars` on `expand_runs` / `unfold_repeated_blocks`;
`compact_lossless` passes per-stage limits and returns the content
unchanged when one is exceeded.
- `tests/test_lossless_expansion_limit.py` (new), using tracemalloc so
it works on any OS:
- a run marker (log/text/config) and chained block markers stay under an
8 MB peak;
- a 1 MiB line expanded 1000x with 1000 shrinking markers, and 1000
block copies of it, stay under 16 MB;
- the exact `max_chars` boundary holds for both inverses, with and
without a trailing newline;
- a config fold whose run-collapsed intermediate is longer than the
content is still kept;
- the block inverse on 400 KB of short lines stays under 6 MB;
- real repeats still fold.
- `tests/test_request_body_decompression_limits.py`: fake decompressors
for the per-step bound, the fail-closed path on old brotli, and draining
output that is still pending after the last slice.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
New tests on `origin/main` code:
```text
E AssertionError: peak 133.5MB: the marker count was expanded
E AssertionError: peak 217.4MB: the block markers were unfolded
E AssertionError: peak 128.0MB suggests one slice was fully expanded
FAILED tests/test_lossless_expansion_limit.py::test_run_marker_in_input_is_not_expanded[log]
FAILED tests/test_lossless_expansion_limit.py::test_run_marker_in_input_is_not_expanded[text]
FAILED tests/test_lossless_expansion_limit.py::test_run_marker_in_input_is_not_expanded[config]
FAILED tests/test_lossless_expansion_limit.py::test_chained_block_markers_in_config_are_not_expanded
FAILED tests/test_request_body_decompression_limits.py::test_brotli_step_output_is_bounded
FAILED tests/test_request_body_decompression_limits.py::test_brotli_without_output_limit_fails_closed
```
With the fix:
```text
$ pytest -q tests/test_request_body_decompression_limits.py tests/test_lossless_expansion_limit.py
======================== 32 passed, 2 skipped in 1.68s =========================
# same file with brotli 1.2.1 importable (the 2 skips are real-brotli tests):
$ pytest -q tests/test_request_body_decompression_limits.py
============================== 29 passed in 4.32s ==============================
$ pytest -q tests/test_lossless_expansion_limit.py tests/test_lossless_excluded_compaction.py tests/test_transforms_config_compressor.py tests/test_request_body_decompression_limits.py tests/test_lossless_diff_fold_guard.py tests/test_lossless_timestamp_guard.py tests/test_bash_search_lossless_fold.py
======================== 177 passed, 2 skipped in 6.90s ========================
$ ruff check / ruff format --check (0.17.3): All checks passed
$ mypy headroom/proxy/helpers.py headroom/transforms/lossless_compaction.py --ignore-missing-imports
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: macOS (Apple silicon), Python 3.12, brotli 1.2.1.
- Exact command / steps: built a 1 GiB-of-zeros brotli stream (1,681
bytes compressed) and called `_brotli_bounded`. Then round-tripped
`b""`, `b"x"`, 600 KB of text, 300 KB of random bytes, 50 MB of zeros
and 5 MB of repeated random data, plus a stream with its last 3 bytes
cut off.
- Observed result: `refused: Decompressed brotli request body exceeds
100MB`, max RSS growth 100 MB, 0.09 s. All round trips were exact. The
cut-off stream gave `Failed to decompress brotli request body: truncated
stream`.
- Lossless: `compact_lossless(("X" * 1 MiB) + "\n... (repeated 1000
times)\n" + "a\n... (repeated 1 times)\n" * 1000, "text")` returns the
input unchanged with a 3.1 MiB tracemalloc peak. With a line limit only,
it peaked at 1002 MiB.
- Not tested: a full proxy request carrying `Content-Encoding: br` (the
reader dispatches to the same function).
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: `br` request bodies are now refused
(400 with an upgrade message) when the installed `brotli` is older than
1.2.0. brotli is an optional install, and gzip/deflate/zstd are
unaffected. Lossless compaction leaves content unchanged only when it
carries markers asking for more lines than it has.
- Kill switch / disable path: upgrade `brotli` to >=1.2.0
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — n/a
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
The memory tests use tracemalloc peaks, not RLIMIT_AS, so they run on
macOS as well as Linux.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reported by @JoaoMarcos44. Thank you for your contribution! 🩵
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
496 lines
18 KiB
Python
496 lines
18 KiB
Python
"""Decompressed request bodies must obey the same size ceiling as plain ones.
|
|
|
|
The handlers gate on ``Content-Length`` — the *compressed* wire size — so a
|
|
high-ratio body bought unlimited capacity: ~2MB of zeros expands to 2GB and the
|
|
process dies allocating it, from a single unauthenticated request (#3284).
|
|
|
|
The property under test is not merely "an oversized body is rejected" but
|
|
"it is rejected *without being materialized first*" — a guard that checks the
|
|
size after a one-shot ``decompress()`` has already OOM'd. So the memory tests
|
|
below assert on peak allocation, not just on the exception.
|
|
|
|
Bombs are built by streaming a compressor, so the fixtures themselves stay
|
|
small; the cap is monkeypatched down so the tests neither allocate nor scan
|
|
100MB to prove the point.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import gzip
|
|
import io
|
|
import sys
|
|
import tracemalloc
|
|
import zlib
|
|
|
|
import pytest
|
|
|
|
|
|
def _helpers():
|
|
"""Resolve the helpers module at call time, not at import time.
|
|
|
|
Other suites swap ``headroom.proxy`` in and out of ``sys.modules`` for their
|
|
own isolation, which can leave two live copies of this module — and two
|
|
distinct ``RequestBodyTooLarge`` classes, so a class captured at import time
|
|
stops matching the one actually raised. Production never reimports this way,
|
|
and callers catch the builtin ``ValueError`` regardless, so this is a test
|
|
concern only. Resolving through ``sys.modules`` keeps the helper and the
|
|
exception it raises on the same module object.
|
|
"""
|
|
import importlib
|
|
|
|
return importlib.import_module("headroom.proxy.helpers")
|
|
|
|
|
|
MB = 1024 * 1024
|
|
PAYLOAD = b'{"model":"claude-sonnet-5","messages":[{"role":"user","content":"hi"}]}'
|
|
|
|
# Expands to 64MB. Every test that uses it caps well below that, so the refusal
|
|
# has to happen mid-stream rather than at the end.
|
|
BOMB_PLAIN_SIZE = 64 * MB
|
|
SMALL_CAP = 1 * MB
|
|
|
|
|
|
def _gzip_bomb(total: int = BOMB_PLAIN_SIZE) -> bytes:
|
|
"""A gzip stream expanding to ``total`` bytes, built without holding them."""
|
|
compressor = zlib.compressobj(9, zlib.DEFLATED, 16 + zlib.MAX_WBITS)
|
|
buf = io.BytesIO()
|
|
chunk = b"\x00" * MB
|
|
for _ in range(total // MB):
|
|
buf.write(compressor.compress(chunk))
|
|
buf.write(compressor.flush())
|
|
return buf.getvalue()
|
|
|
|
|
|
def _deflate_bomb(total: int = BOMB_PLAIN_SIZE) -> bytes:
|
|
compressor = zlib.compressobj(9)
|
|
buf = io.BytesIO()
|
|
chunk = b"\x00" * MB
|
|
for _ in range(total // MB):
|
|
buf.write(compressor.compress(chunk))
|
|
buf.write(compressor.flush())
|
|
return buf.getvalue()
|
|
|
|
|
|
class _Request:
|
|
"""Minimal stand-in for the Starlette Request the reader actually takes."""
|
|
|
|
def __init__(
|
|
self,
|
|
body: bytes,
|
|
content_encoding: str = "",
|
|
*,
|
|
headers: dict | None = None,
|
|
stream_chunk_size: int = 64 * 1024,
|
|
) -> None:
|
|
self._body = body
|
|
self.headers = {"content-encoding": content_encoding, **(headers or {})}
|
|
self._stream_chunk_size = stream_chunk_size
|
|
self.chunks_yielded = 0
|
|
|
|
async def body(self) -> bytes:
|
|
return self._body
|
|
|
|
async def stream(self):
|
|
# Real ASGI servers hand the body over in chunks, not one blob, which
|
|
# is exactly what lets the streaming bound-check in
|
|
# `_read_request_body_bytes` refuse a body before all of it has
|
|
# arrived.
|
|
for start in range(0, len(self._body), self._stream_chunk_size):
|
|
self.chunks_yielded += 1
|
|
yield self._body[start : start + self._stream_chunk_size]
|
|
|
|
|
|
class _NeverStreamedRequest(_Request):
|
|
"""A request whose ``.stream()`` fails if ever iterated.
|
|
|
|
Used to prove a fast-path (e.g. a Content-Length pre-check) rejects
|
|
before touching the body, rather than merely rejecting eventually.
|
|
"""
|
|
|
|
async def stream(self):
|
|
raise AssertionError("stream() should not have been read")
|
|
yield b"" # pragma: no cover - unreachable, keeps this an async generator
|
|
|
|
|
|
@pytest.fixture
|
|
def small_cap(monkeypatch: pytest.MonkeyPatch) -> int:
|
|
monkeypatch.setattr(_helpers(), "MAX_DECOMPRESSED_BODY_SIZE", SMALL_CAP)
|
|
return SMALL_CAP
|
|
|
|
|
|
@pytest.fixture
|
|
def small_raw_cap(monkeypatch: pytest.MonkeyPatch) -> int:
|
|
monkeypatch.setattr(_helpers(), "MAX_REQUEST_BODY_SIZE", SMALL_CAP)
|
|
return SMALL_CAP
|
|
|
|
|
|
# ───────────────────────────── the ceiling itself ──────────────────────────
|
|
|
|
|
|
# ───────────────────────────── refusal, per codec ──────────────────────────
|
|
|
|
|
|
def test_gzip_bomb_is_refused(small_cap: int) -> None:
|
|
with pytest.raises(_helpers().RequestBodyTooLarge):
|
|
_helpers()._inflate_bounded(
|
|
_gzip_bomb(), wbits=16 + zlib.MAX_WBITS, label="gzip", multi_member=True
|
|
)
|
|
|
|
|
|
def test_deflate_bomb_is_refused(small_cap: int) -> None:
|
|
with pytest.raises(_helpers().RequestBodyTooLarge):
|
|
_helpers()._inflate_bounded(_deflate_bomb(), wbits=zlib.MAX_WBITS, label="deflate")
|
|
|
|
|
|
def test_zstd_bomb_is_refused(small_cap: int) -> None:
|
|
zstandard = pytest.importorskip("zstandard")
|
|
bomb = zstandard.ZstdCompressor(level=19).compress(b"\x00" * BOMB_PLAIN_SIZE)
|
|
with pytest.raises(_helpers().RequestBodyTooLarge):
|
|
_helpers()._zstd_bounded(bomb)
|
|
|
|
|
|
def test_brotli_bomb_is_refused(small_cap: int) -> None:
|
|
brotli = pytest.importorskip("brotli")
|
|
bomb = brotli.compress(b"\x00" * BOMB_PLAIN_SIZE)
|
|
with pytest.raises(_helpers().RequestBodyTooLarge):
|
|
_helpers()._brotli_bounded(bomb)
|
|
|
|
|
|
# ─────────────────────── refused *before* materializing ────────────────────
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("make_bomb", "call"),
|
|
[
|
|
(
|
|
_gzip_bomb,
|
|
lambda b: _helpers()._inflate_bounded(
|
|
b, wbits=16 + zlib.MAX_WBITS, label="gzip", multi_member=True
|
|
),
|
|
),
|
|
(
|
|
_deflate_bomb,
|
|
lambda b: _helpers()._inflate_bounded(b, wbits=zlib.MAX_WBITS, label="deflate"),
|
|
),
|
|
],
|
|
)
|
|
def test_bomb_never_materializes(make_bomb, call, small_cap: int) -> None:
|
|
"""Peak allocation stays near the cap, not near the expansion.
|
|
|
|
This is the whole fix: a check placed after a one-shot decompress() would
|
|
pass the refusal assertions above while still OOM-ing the process.
|
|
"""
|
|
bomb = make_bomb()
|
|
|
|
already_tracing = tracemalloc.is_tracing()
|
|
if not already_tracing:
|
|
tracemalloc.start()
|
|
try:
|
|
# reset_peak so a peak set by whatever ran before this test cannot be
|
|
# mistaken for the bomb materializing here.
|
|
tracemalloc.reset_peak()
|
|
with pytest.raises(_helpers().RequestBodyTooLarge):
|
|
call(bomb)
|
|
_, peak = tracemalloc.get_traced_memory()
|
|
finally:
|
|
if not already_tracing:
|
|
tracemalloc.stop()
|
|
|
|
# Generous headroom over the 1MB cap; the unbounded path would peak near
|
|
# the 64MB expansion (and higher still, with the copy decompress() makes).
|
|
assert peak < 8 * MB, f"peak {peak / MB:.1f}MB suggests the bomb was materialized"
|
|
|
|
|
|
# ─────────────────────────── ordinary bodies still work ────────────────────
|
|
|
|
|
|
def test_gzip_round_trips() -> None:
|
|
assert (
|
|
_helpers()._inflate_bounded(
|
|
gzip.compress(PAYLOAD), wbits=16 + zlib.MAX_WBITS, label="gzip", multi_member=True
|
|
)
|
|
== PAYLOAD
|
|
)
|
|
|
|
|
|
def test_deflate_round_trips() -> None:
|
|
assert (
|
|
_helpers()._inflate_bounded(zlib.compress(PAYLOAD), wbits=zlib.MAX_WBITS, label="deflate")
|
|
== PAYLOAD
|
|
)
|
|
|
|
|
|
def test_zstd_round_trips() -> None:
|
|
zstandard = pytest.importorskip("zstandard")
|
|
assert _helpers()._zstd_bounded(zstandard.ZstdCompressor().compress(PAYLOAD)) == PAYLOAD
|
|
|
|
|
|
def test_brotli_round_trips() -> None:
|
|
brotli = pytest.importorskip("brotli")
|
|
assert _helpers()._brotli_bounded(brotli.compress(PAYLOAD)) == PAYLOAD
|
|
|
|
|
|
def test_multi_member_gzip_is_still_concatenated() -> None:
|
|
"""`gzip.decompress` joins members; a single-member decompressobj would not."""
|
|
stream = gzip.compress(b'{"a":1}') + gzip.compress(b'{"b":2}')
|
|
|
|
assert (
|
|
_helpers()._inflate_bounded(
|
|
stream, wbits=16 + zlib.MAX_WBITS, label="gzip", multi_member=True
|
|
)
|
|
== b'{"a":1}{"b":2}'
|
|
)
|
|
|
|
|
|
def test_gzip_trailing_nul_padding_is_tolerated() -> None:
|
|
"""Real clients pad gzip bodies with NULs and `gzip.decompress` accepts it.
|
|
|
|
A member-restart loop that treats the padding as a fresh member rejects
|
|
bodies the one-shot call happily decoded — a silent compatibility break
|
|
that no bomb test would have caught.
|
|
"""
|
|
padded = gzip.compress(PAYLOAD) + b"\x00" * 16
|
|
|
|
assert gzip.decompress(padded) == PAYLOAD # the behavior being matched
|
|
assert (
|
|
_helpers()._inflate_bounded(
|
|
padded, wbits=16 + zlib.MAX_WBITS, label="gzip", multi_member=True
|
|
)
|
|
== PAYLOAD
|
|
)
|
|
|
|
|
|
def test_gzip_padding_between_members_is_skipped() -> None:
|
|
stream = gzip.compress(b'{"a":1}') + b"\x00" * 8 + gzip.compress(b'{"b":2}')
|
|
|
|
assert (
|
|
_helpers()._inflate_bounded(
|
|
stream, wbits=16 + zlib.MAX_WBITS, label="gzip", multi_member=True
|
|
)
|
|
== b'{"a":1}{"b":2}'
|
|
)
|
|
|
|
|
|
def test_empty_gzip_body_matches_the_one_shot_call() -> None:
|
|
"""`gzip.decompress(b"")` returns b""; refusing it would be a new rejection."""
|
|
assert gzip.decompress(b"") == b""
|
|
assert (
|
|
_helpers()._inflate_bounded(b"", wbits=16 + zlib.MAX_WBITS, label="gzip", multi_member=True)
|
|
== b""
|
|
)
|
|
|
|
|
|
def test_gzip_body_of_only_padding_is_still_rejected() -> None:
|
|
"""Tolerating padding must not turn a bodyless run of NULs into success."""
|
|
with pytest.raises(gzip.BadGzipFile):
|
|
gzip.decompress(b"\x00" * 8)
|
|
# zlib.error, not ValueError: only `_read_request_body_bytes` wraps the
|
|
# codec's own error, and it turns this into the same 400 as before.
|
|
with pytest.raises(zlib.error):
|
|
_helpers()._inflate_bounded(
|
|
b"\x00" * 8, wbits=16 + zlib.MAX_WBITS, label="gzip", multi_member=True
|
|
)
|
|
|
|
|
|
def test_truncated_gzip_still_errors() -> None:
|
|
"""Losing the one-shot call must not turn a corrupt body into a silent empty one."""
|
|
truncated = gzip.compress(PAYLOAD)[:-5]
|
|
|
|
with pytest.raises(ValueError):
|
|
_helpers()._inflate_bounded(
|
|
truncated, wbits=16 + zlib.MAX_WBITS, label="gzip", multi_member=True
|
|
)
|
|
|
|
|
|
# ───────────────── raw body ceiling (the chunked-transfer bypass) ──────────
|
|
#
|
|
# The handlers only ever see the *compressed*, wire-level size via
|
|
# `Content-Length`, and that header is absent on a chunked-transfer-encoding
|
|
# request. The streaming loop in `_read_request_body_bytes` is the backstop
|
|
# the handlers fall through to regardless of whether that header was there
|
|
# (#3326).
|
|
|
|
|
|
async def test_raw_body_refused_without_consuming_the_whole_stream(small_raw_cap: int) -> None:
|
|
"""The point is refusing early, not just refusing eventually."""
|
|
request = _Request(b"a" * (small_raw_cap * 10), stream_chunk_size=small_raw_cap // 4)
|
|
total_chunks = -(-len(request._body) // request._stream_chunk_size) # ceil div
|
|
|
|
with pytest.raises(_helpers().RequestBodyTooLarge):
|
|
await _helpers()._read_request_body_bytes(request)
|
|
|
|
assert request.chunks_yielded < total_chunks
|
|
|
|
|
|
# ──────────────────────────── through the entry point ──────────────────────
|
|
|
|
|
|
async def test_reader_refuses_a_bomb(small_cap: int) -> None:
|
|
with pytest.raises(_helpers().RequestBodyTooLarge):
|
|
await _helpers()._read_request_body_bytes(_Request(_gzip_bomb(), "gzip"))
|
|
|
|
|
|
async def test_reader_passes_an_ordinary_compressed_body(small_cap: int) -> None:
|
|
assert (
|
|
await _helpers()._read_request_body_bytes(_Request(gzip.compress(PAYLOAD), "gzip"))
|
|
== PAYLOAD
|
|
)
|
|
|
|
|
|
async def test_reader_leaves_uncompressed_bodies_alone(small_cap: int) -> None:
|
|
assert await _helpers()._read_request_body_bytes(_Request(PAYLOAD, "")) == PAYLOAD
|
|
assert await _helpers()._read_request_body_bytes(_Request(PAYLOAD, "identity")) == PAYLOAD
|
|
|
|
|
|
async def test_reader_refuses_an_oversized_plain_body_with_no_content_length(
|
|
small_raw_cap: int,
|
|
) -> None:
|
|
"""The regression this PR fixes: no `Content-Length` header, no compression.
|
|
|
|
Before this fix, `_read_request_body_bytes` called `request.body()`
|
|
directly, so a handler's `Content-Length` precheck was the only gate and a
|
|
chunked request (no `Content-Length` to check) reached this function
|
|
regardless of size.
|
|
"""
|
|
oversized = b"a" * (small_raw_cap + 1)
|
|
with pytest.raises(_helpers().RequestBodyTooLarge):
|
|
await _helpers()._read_request_body_bytes(_Request(oversized, ""))
|
|
|
|
|
|
async def test_reader_still_rejects_an_unknown_encoding() -> None:
|
|
with pytest.raises(ValueError, match="Unsupported Content-Encoding"):
|
|
await _helpers()._read_request_body_bytes(_Request(PAYLOAD, "snappy"))
|
|
|
|
|
|
# ────────────────── the raw (pre-decompression) wire size is bounded too ────
|
|
#
|
|
# A plain (identity-encoded) body was never bounded at all: the old reader did
|
|
# ``raw = await request.body()`` and only checked size *after* decompression,
|
|
# which is a no-op for identity bodies. #3479.
|
|
|
|
|
|
async def test_reader_refuses_an_oversized_plain_body(small_raw_cap: int) -> None:
|
|
oversized = b"x" * (small_raw_cap + 1)
|
|
with pytest.raises(_helpers().RequestBodyTooLarge):
|
|
await _helpers()._read_request_body_bytes(_Request(oversized, ""))
|
|
|
|
|
|
async def test_reader_accepts_a_plain_body_at_the_cap(small_raw_cap: int) -> None:
|
|
at_cap = b"x" * small_raw_cap
|
|
assert await _helpers()._read_request_body_bytes(_Request(at_cap, "")) == at_cap
|
|
|
|
|
|
async def test_reader_honors_a_truthful_content_length_before_reading(
|
|
small_raw_cap: int,
|
|
) -> None:
|
|
"""A Content-Length that already overstates the cap must short-circuit.
|
|
|
|
``_NeverStreamedRequest.stream()`` raises if ever iterated, so this only
|
|
passes if the Content-Length check runs (and rejects) before the body is
|
|
ever read.
|
|
"""
|
|
request = _NeverStreamedRequest(b"", "", headers={"content-length": str(small_raw_cap + 1)})
|
|
with pytest.raises(_helpers().RequestBodyTooLarge):
|
|
await _helpers()._read_request_body_bytes(request)
|
|
|
|
|
|
async def test_reader_ignores_an_untrustworthy_content_length(small_raw_cap: int) -> None:
|
|
"""Content-Length is an optimization, not the enforcement boundary.
|
|
|
|
A header that understates (or lies about) the body's real size must not
|
|
let an oversized body through -- the streaming check still catches it.
|
|
"""
|
|
oversized = b"x" * (small_raw_cap + 1)
|
|
request = _Request(oversized, "", headers={"content-length": "1"})
|
|
with pytest.raises(_helpers().RequestBodyTooLarge):
|
|
await _helpers()._read_request_body_bytes(request)
|
|
|
|
# A non-numeric header is likewise not trusted -- it neither
|
|
# short-circuits acceptance nor raises on its own; the stream still runs.
|
|
request = _Request(oversized, "", headers={"content-length": "not-a-number"})
|
|
with pytest.raises(_helpers().RequestBodyTooLarge):
|
|
await _helpers()._read_request_body_bytes(request)
|
|
|
|
|
|
# ───────────────── brotli: output bounded per step, not per input slice ─────
|
|
|
|
|
|
class _FakeBrotliDecompressor:
|
|
"""Mimics brotli>=1.2.0 ``Decompressor`` output limiting without the package.
|
|
|
|
Each non-empty input slice "expands" to ``EXPANSION`` zero bytes. With
|
|
``output_buffer_limit`` the surplus stays pending inside the decompressor
|
|
(``can_accept_more_data()`` is False until drained); without it the whole
|
|
expansion is returned at once, which is the bomb.
|
|
"""
|
|
|
|
EXPANSION = 64 * MB
|
|
|
|
def __init__(self) -> None:
|
|
self.pending = 0
|
|
|
|
def process(self, data: bytes, output_buffer_limit: int | None = None) -> bytes:
|
|
self.pending += self.EXPANSION if data else 0
|
|
n = self.pending if output_buffer_limit is None else min(self.pending, output_buffer_limit)
|
|
self.pending -= n
|
|
return b"\x00" * n
|
|
|
|
def can_accept_more_data(self) -> bool:
|
|
return self.pending == 0
|
|
|
|
def is_finished(self) -> bool:
|
|
return self.pending == 0
|
|
|
|
|
|
def test_brotli_step_output_is_bounded(monkeypatch, small_cap: int) -> None:
|
|
"""One input slice that expands far past the cap must not be materialized."""
|
|
import types
|
|
|
|
monkeypatch.setitem(
|
|
sys.modules, "brotli", types.SimpleNamespace(Decompressor=_FakeBrotliDecompressor)
|
|
)
|
|
tracemalloc.start()
|
|
try:
|
|
tracemalloc.reset_peak()
|
|
with pytest.raises(_helpers().RequestBodyTooLarge):
|
|
_helpers()._brotli_bounded(b"x" * 100)
|
|
_, peak = tracemalloc.get_traced_memory()
|
|
finally:
|
|
tracemalloc.stop()
|
|
assert peak < 8 * MB, f"peak {peak / MB:.1f}MB suggests one slice was fully expanded"
|
|
|
|
|
|
def test_brotli_without_output_limit_fails_closed(monkeypatch) -> None:
|
|
import types
|
|
|
|
class _Old:
|
|
def process(self, data: bytes) -> bytes:
|
|
return data
|
|
|
|
monkeypatch.setitem(sys.modules, "brotli", types.SimpleNamespace(Decompressor=_Old))
|
|
with pytest.raises(ValueError, match="brotli>=1.2.0"):
|
|
_helpers()._brotli_bounded(b"x")
|
|
|
|
|
|
class _FakeBrotliAcceptingWhilePending(_FakeBrotliDecompressor):
|
|
"""Real brotli 1.2 reports it can accept input once all input is consumed,
|
|
even while limited output is still pending; the rest comes from empty
|
|
``process(b"")`` calls."""
|
|
|
|
EXPANSION = 3 * 1024 * 1024 + 123
|
|
|
|
def can_accept_more_data(self) -> bool:
|
|
return True
|
|
|
|
|
|
def test_brotli_drains_output_pending_after_the_last_slice(monkeypatch) -> None:
|
|
import types
|
|
|
|
monkeypatch.setitem(
|
|
sys.modules,
|
|
"brotli",
|
|
types.SimpleNamespace(Decompressor=_FakeBrotliAcceptingWhilePending),
|
|
)
|
|
out = _helpers()._brotli_bounded(b"x")
|
|
assert len(out) == _FakeBrotliAcceptingWhilePending.EXPANSION
|