mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(router): bound ML compression per request, not just per block (#3721)
## Description > Revised 2026-10-05 after review: the first ML call of a request is now capped by the remaining ML budget through Kompress's existing chunk-boundary time budget (no seed estimate), and budget-refused blocks pass through unchanged so their bytes stay stable on later turns. Third review: the same remainder now caps `RemoteKompressCompressor` too, over the whole HTTP request, and the remote call is declined when 50 ms or less is left. Closes #3711. Thanks to @gaaaork for the report — and for the follow-up correction, which is what made this findable. The corrected finding (it *is* the Kompress stage; `kompress` alone reproduces it) was right, and this PR answers the open question at the end of that comment. ### The question was: why did the size gate never fire? > `headroom_kompress_size_gate_total{outcome="within"} 2584` — only `within`. Either the gate is evaluated per chunk (each chunk under the ceiling, while their sum takes 70s), or it is not applied on this path. **It is the first.** The gate lives inside `_try_ml_compressor` (`content_router.py:4152`), and `_compress_mixed` calls that function **once per section** in a serial loop (`content_router.py:2815-2841`). So the gate only ever sees one section. It bounds **per-call size**; nothing bounded the **request**. Reproduced with the reported shape — a `tool_result` of prose blocks separated by small JSON objects — instrumented through the real router: ``` strategy : router:tool_result:mixed _try_ml_compressor calls : 8 size-gate outcomes : {'within': 8} <- no 'exceeded', exactly as reported ``` Eight separate model entries, eight independent gate decisions, no aggregate accounting. The 30s `compression_first_stage` budget is per-request while the only guard was per-section — a granularity mismatch. And there was no cumulative deadline either: the one existing helper (`_compression_deadline_seconds`) is used at a single site and only when `len(pending_tasks) == 1`, so the section loop ran unbounded. One note on the payload shape, since it cost me a false start: plain user text never reaches the ML stage at all (`router:protected:user_message`). The content has to arrive as a `tool_result` to reproduce — which is exactly what was reported. ### Why not just gate on the whole request That was the suggestion in the issue, and on its own it is not enough. It would fix the 1.4MB case while leaving a request of 100 sections at 40k tokens each equally able to hang — every section still passes any size ceiling. The guard needs to match the budget's *granularity*, not just its scope. So this adds a **request-scoped wall-clock ceiling**. Time is also the honest unit here: the estimator is already known to misjudge dense payloads (see the `#1171` comment above the gate), and the symptom being protected against is wall-clock, not tokens. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `_ml_stage_deadline_seconds()` — `HEADROOM_ML_STAGE_DEADLINE_SECONDS`, default **15s**, deliberately under the 30s `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` so we degrade to a *partial* compression **before** the executor abandons a non-preemptible worker. Losing some savings beats losing the request and then quarantining the stage for everything behind it. `0` restores the previous unbounded behaviour. - Enabled by `apply()` — the one place that owns a whole request — and carried on `_PerRequestRuntimeState`, so the existing `copy_context()` fan-out (documented under #3486/#3556) propagates it to worker threads for free. The budget counts only time spent **inside** the ML compressor (cumulative, charged in a `finally`), and is clamped to half of `HEADROOM_COMPRESSION_TIMEOUT_SECONDS`. - Admission: a block is refused when its projected cost (this request's worst measured seconds-per-token) exceeds what is left. Every admitted call — **including the first, which has no measurement** — is also **capped**: the remainder goes to Kompress as `_time_budget_cap_seconds`, lowering its existing per-call time budget, which Kompress already checks at every chunk boundary. So no call can run more than one chunk past the budget, and parallel workers each end by their admission time plus the remainder. - Remote Kompress (`HEADROOM_KOMPRESS_ENDPOINT`) gets the same cap. httpx timeouts are per phase, so the cap is applied to the request as a whole: the POST runs on a small worker pool (16) and is awaited with one overall deadline. Admission is bounded: a permit per worker is taken before submit, waiting at most until the caller's deadline, so a saturated call fails open unchanged and its payload is never enqueued. The worker holds its permit until it actually finishes, and it finishes by an absolute deadline: per-phase timeouts are capped to the time left and the body is streamed and cut off at the deadline, so a slow-drip response cannot hold a worker. At or below 50 ms left, no request is sent and the block passes through unchanged. With no request budget (direct callers) the call and its 20s timeout are exactly as before. - Blocks refused by the budget (or cut off by the cap) **pass through unchanged**. Not the size gate's lossy fallback: the refusal depends on the request, not the content, and the router caches that verdict, so raw bytes are the only ones that stay stable on later turns (prompt-cache safety). - Recorded as `headroom_kompress_size_gate_total{outcome="deadline"}`, so this is visible in the metric you were already reading, next to `within`/`exceeded`. - Direct `compress()` callers (tests, `/v1/compress`) stay unarmed and keep the old behaviour. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed The original four tests, using a stub in place of Kompress so they assert the *budget* rather than ONNX latency (the real model is not available in CI, and the bug is about how many times a slow stage is entered): - `test_size_gate_alone_cannot_bound_a_request` — pins the precondition: many `within`, zero `exceeded`, repeated model entry. Passes on `main`; it *is* the bug. - `test_ml_deadline_stops_further_ml_work_once_the_budget_is_spent` — fails on `main`. Measures the saving by running the same payload with the ceiling disabled and asserting strictly more model calls. - `test_deadline_zero_restores_previous_unbounded_behaviour` - `test_direct_compress_callers_stay_unarmed` — fails on `main`. Verified fail-before/pass-after by reverting only `content_router.py` and keeping the tests. Regression sweep `-k "router or kompress or mixed or compress"`: **2555 passed**, 10 failed — and those same 10 fail identically on `origin/main` with this file reverted, so **zero new failures**. Ruff 0.16.7 (the CI pin) clean. Added after review (each fails with the source reverted and the tests kept, passes with it): - `test_non_ml_prework_does_not_consume_the_ml_budget`, `test_a_slow_call_cannot_be_admitted_against_a_budget_it_will_overrun`, `test_budget_is_clamped_below_the_timeout_that_quarantines_the_stage`, `test_a_failing_ml_call_still_draws_down_the_budget` (first review). - `test_first_ml_call_is_capped_by_the_budget_it_was_admitted_against` — first block's worst case is 3s under a 1s executor timeout (0.5s ML budget); asserts the first call is handed a cap ≤ 0.5s and total ML time stays within budget + one chunk. - `test_blocks_refused_by_the_budget_pass_through_unchanged` — refused block is byte-identical even with TextCrusher available. - `test_time_budget_cap_stops_a_call_and_returns_it_unchanged`, `test_time_budget_cap_bounds_the_batched_path_too` (real `KompressCompressor`, fake model/clock) — the cap stops at the first chunk boundary past it and returns the block unchanged, on both the single and batched paths. - `test_cold_remote_call_cannot_overrun_a_tiny_ml_budget` (third review, mirrors the reviewer's probe): real `RemoteKompressCompressor` over `httpx.MockTransport` with a 100 ms endpoint, 10 ms ML budget, 40 ms outer timeout, via public `apply()`. Asserts ML time < 40 ms, no HTTP request sent, prose blocks byte-identical. - `test_cold_remote_call_is_bounded_as_a_whole_by_the_ml_budget`: 200 ms budget, endpoint that hangs for 5 s. Asserts the first request's per-phase timeouts are lowered to the budget, total ML time ≤ budget + 0.1 s, blocks unchanged. - `test_remote_kompress_keeps_its_default_timeout_without_a_budget`: uncapped call keeps the 20s timeouts and does not hop threads. - `test_saturated_remote_calls_fail_open_without_queueing_their_payloads` (fourth review, the reviewer's repro): 16 workers held by requests whose 200 ms callers timed out, then 5 calls with 60 ms caps. Each fails open unchanged within its cap, the executor queue stays empty, none reaches the endpoint, and capacity returns once the held requests end. - `test_slow_drip_response_is_cut_off_at_an_absolute_deadline`: a body dripping a chunk every 50 ms for 2 s under a 200 ms cap; the worker stops at the deadline and returns its permit well before the drip ends. - `test_permit_is_held_until_the_worker_finishes_not_the_caller`. ### Test Output ```text $ LITELLM_LOCAL_MODEL_COST_MAP=True uv run --frozen --extra dev --extra proxy pytest -q \ tests/test_content_router*.py tests/test_transforms/test_content_router*.py \ tests/test_transforms/test_kompress*.py tests/test_kompress*.py tests/test_force_kompress_all.py \ tests/test_remote_kompress_dropin.py tests/test_transforms_content_router.py tests/test_router_*.py \ tests/test_onnx_router_threads.py tests/test_proxy_disable_kompress.py tests/test_proxy_per_provider_kompress.py 486 passed, 2 skipped, 1 warning in 19.42s # with only the source hunks reverted: FAILED tests/test_content_router_ml_stage_deadline.py::test_first_ml_call_is_capped_by_the_budget_it_was_admitted_against FAILED tests/test_content_router_ml_stage_deadline.py::test_blocks_refused_by_the_budget_pass_through_unchanged FAILED tests/test_transforms/test_kompress_deadline.py::test_time_budget_cap_stops_a_call_and_returns_it_unchanged FAILED tests/test_transforms/test_kompress_deadline.py::test_time_budget_cap_bounds_the_batched_path_too ``` ## Real Behavior Proof - Environment: local macOS, Python 3.14, branch merged with current `main`. - Exact command / steps: the pytest invocation above; the reporter's payload shape (prose blocks separated by small JSON objects inside a `tool_result`) driven through the real `ContentRouter.apply()` with a stub in place of the ONNX model. - Observed result: ML time stays within the budget (plus at most one chunk) even when the first block's worst case exceeds the executor timeout; refused blocks are byte-identical. - Not tested: real Kompress ONNX latency on the reporter's hardware (model not downloaded locally); a real remote Kompress endpoint (covered network-free via `httpx.MockTransport`). ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: stable. - Stable/default behavior changed: yes — a request now spends at most 15s (or half of `HEADROOM_COMPRESSION_TIMEOUT_SECONDS`, if lower) inside ML; blocks past that are sent uncompressed. - Kill switch / disable path: `HEADROOM_ML_STAGE_DEADLINE_SECONDS=0` restores the previous unbounded behaviour. - Unsafe override required: no. - Qualification impact: none expected; only requests whose ML stage exceeds the budget see less compression. - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes ### What this does not claim to fix The ~70s figure is still the reporter's measurement, not mine — I could not reproduce it locally because the Kompress model is not downloaded on this machine, so I discarded my own latency numbers as meaningless. What this PR fixes is the *consequence*: a request that spends too long in ML now degrades to partial compression instead of blowing the budget and quarantining the stage. @gaaaork — you offered to test a patch. If you can run this with `HEADROOM_ML_STAGE_DEADLINE_SECONDS` at the default 15, the stage should return in ~15s with partial compression and no `CompressionQuarantinedError`, and `headroom_kompress_size_gate_total` should start showing `deadline`. The separate question of why one 40k-token section takes seconds on your hardware is still worth chasing — this bounds the blast radius, it does not make the model faster. Documentation: no docs page covers `HEADROOM_ML_STAGE_DEADLINE_SECONDS` yet; mypy not run locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3721" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5
JerrettDavis
parent
f8851878a4
commit
f37980f7f7
@@ -166,6 +166,68 @@ def _estimate_tokens(text: str) -> int:
|
||||
return max(1, _TOKEN_ESTIMATOR.count_text(text))
|
||||
|
||||
|
||||
#: Share of the outer compression timeout the ML stage may consume. The
|
||||
#: remainder has to cover this request's non-ML work AND the overrun of the one
|
||||
#: ML call that can still be running when the ceiling is reached -- the worker
|
||||
#: is not preemptible, so the ceiling can only stop the NEXT call.
|
||||
_ML_STAGE_BUDGET_FRACTION = 0.5
|
||||
|
||||
|
||||
def _ml_stage_deadline_seconds() -> float:
|
||||
"""Wall-clock ceiling for ALL ML (Kompress) work inside one request.
|
||||
|
||||
``_kompress_max_tokens`` bounds a single block. It cannot bound a request:
|
||||
the mixed-content path splits a payload into sections and calls
|
||||
``_try_ml_compressor`` once per section (see ``_compress_mixed``), so every
|
||||
section can sit under the per-block ceiling while their sum runs for
|
||||
minutes. #3711 reported ~70s on a 1.4MB tool_result of prose blocks
|
||||
separated by small JSON objects -- with the size gate recording only
|
||||
``within`` -- which blew the 30s compression budget and then quarantined
|
||||
compression for every following request.
|
||||
|
||||
So the guard has to match the budget's granularity: once this much time
|
||||
has gone into ML for this request, remaining blocks pass through unchanged.
|
||||
Raw rather than a cheaper lossy fallback: the router caches that verdict,
|
||||
so the block keeps the same bytes on later turns and the prompt cache
|
||||
holds.
|
||||
|
||||
The point of the ceiling is to degrade to a partial compression *before*
|
||||
the executor abandons a non-preemptible worker, so it is worthless unless
|
||||
it sits below the deadline that does the abandoning. A bare default could
|
||||
not promise that: an operator who lowers
|
||||
``HEADROOM_COMPRESSION_TIMEOUT_SECONDS`` below 15s got a guard that could
|
||||
never fire first -- exactly the failure this exists to prevent. The budget
|
||||
is therefore clamped to :data:`_ML_STAGE_BUDGET_FRACTION` of the effective
|
||||
outer timeout, leaving the remainder for the non-ML work in the same
|
||||
request (lossless transforms, assembly) plus the overrun of the one ML call
|
||||
already in flight when the ceiling is reached.
|
||||
|
||||
The outer budget read here is ``HEADROOM_COMPRESSION_TIMEOUT_SECONDS``, the
|
||||
executor timeout that quarantines the stage -- NOT
|
||||
``HEADROOM_COMPRESSION_DEADLINE_MS``, which bounds a different, softer
|
||||
in-router deadline (see ``_compression_deadline_seconds``).
|
||||
|
||||
``0`` disables the ceiling (previous behavior), and an explicit request
|
||||
below the clamp is honoured -- the clamp only ever lowers.
|
||||
"""
|
||||
try:
|
||||
configured = max(
|
||||
0.0,
|
||||
float(os.environ.get("HEADROOM_ML_STAGE_DEADLINE_SECONDS", "15")),
|
||||
)
|
||||
except ValueError:
|
||||
configured = 15.0
|
||||
if configured <= 0.0:
|
||||
return 0.0
|
||||
try:
|
||||
outer = float(os.environ.get("HEADROOM_COMPRESSION_TIMEOUT_SECONDS", "30"))
|
||||
except ValueError:
|
||||
outer = 30.0
|
||||
if outer <= 0.0:
|
||||
return configured
|
||||
return min(configured, outer * _ML_STAGE_BUDGET_FRACTION)
|
||||
|
||||
|
||||
def _compression_deadline_seconds() -> float:
|
||||
try:
|
||||
return max(
|
||||
@@ -2047,6 +2109,22 @@ class _PerRequestRuntimeState:
|
||||
|
||||
compression_policy: Any = None
|
||||
target_ratio: float | None = None
|
||||
# Whether an `apply()` call owns this state and the ML budget therefore
|
||||
# applies. False on the INSTANCE-SCOPED DEFAULT that direct `compress()`
|
||||
# callers read (see `ContentRouter.__init__`): that object is shared by
|
||||
# every such call for the life of the router, so charging it would
|
||||
# accumulate across unrelated calls until ML switched itself off forever.
|
||||
# Direct callers keep the old unbounded behaviour.
|
||||
ml_budget_active: bool = False
|
||||
# Seconds this request has spent INSIDE the ML compressor (#3711).
|
||||
# Cumulative rather than a wall-clock deadline stamped at `apply()` entry:
|
||||
# the budget is a statement about ML work, so time spent on lifecycle,
|
||||
# classification and lossless transforms must not draw it down.
|
||||
ml_elapsed: float = 0.0
|
||||
# Longest single ML call observed in THIS request, seconds per estimated
|
||||
# token. Used to project whether the next block can finish inside what is
|
||||
# left; see `_try_ml_compressor`.
|
||||
ml_seconds_per_token: float = 0.0
|
||||
force_kompress: bool = False
|
||||
skip_kompress: bool = False
|
||||
kompress_model: str | None = None
|
||||
@@ -4378,6 +4456,28 @@ class ContentRouter(Transform):
|
||||
return None
|
||||
return elided, _estimate_tokens(elided)
|
||||
|
||||
def _charge_ml_time(self, elapsed: float, text: str) -> None:
|
||||
"""Bill *elapsed* seconds of ML work to this request's budget.
|
||||
|
||||
Also records the worst seconds-per-token seen so far, which is what the
|
||||
admission check in :meth:`_try_ml_compressor` projects the NEXT block
|
||||
with. Worst case rather than an average: the ceiling exists to stop a
|
||||
single long call from overrunning the outer timeout, and an average
|
||||
would let one slow block hide behind several fast ones.
|
||||
|
||||
Best-effort -- accounting must never break a compression that worked.
|
||||
"""
|
||||
try:
|
||||
state = self._runtime_state_var.get()
|
||||
if not state.ml_budget_active:
|
||||
return
|
||||
state.ml_elapsed += max(0.0, elapsed)
|
||||
tokens = _estimate_tokens(text)
|
||||
if tokens > 0 and elapsed > 0.0:
|
||||
state.ml_seconds_per_token = max(state.ml_seconds_per_token, elapsed / tokens)
|
||||
except Exception: # pragma: no cover - defensive
|
||||
logger.debug("ML budget accounting failed", exc_info=True)
|
||||
|
||||
def _embedded_json_route(
|
||||
self, content: str, context: str, question: str | None, bias: float
|
||||
) -> str | None:
|
||||
@@ -4468,6 +4568,60 @@ class ContentRouter(Transform):
|
||||
# (under the 50,000 cap, gate silent) but 55,557 estimator tokens, 11%
|
||||
# over. That is exactly the >30s non-preemptible ONNX inference this gate
|
||||
# exists to prevent (#1171).
|
||||
# Request-scoped ML ceiling (#3711). The size check below bounds ONE
|
||||
# block; this bounds the request. Without it a payload split into many
|
||||
# individually-small sections calls this function once per section, each
|
||||
# passing the size gate, until the sum blows the compression budget and
|
||||
# quarantines the stage for every later request. Checked before the size
|
||||
# gate so an over-budget request stops paying the estimator too.
|
||||
# An admission check that only asks "is the budget gone yet?" lets a
|
||||
# call start at 14.9s of a 15s budget and run for another minute: the
|
||||
# ONNX worker is not preemptible, so nothing stops it once it begins,
|
||||
# and it can overrun the outer executor timeout the ceiling exists to
|
||||
# protect. So project this block's cost and refuse it unless it can
|
||||
# plausibly FINISH inside what is left.
|
||||
#
|
||||
# The projection uses the worst seconds-per-token this request has
|
||||
# actually measured. The first block of a request has no measurement,
|
||||
# so it cannot be refused on one -- instead the call itself is capped:
|
||||
# what is left of the budget goes to Kompress as its per-call time
|
||||
# budget (below), and Kompress stops at the next chunk boundary when
|
||||
# it runs out, returning the block unchanged. That holds for every
|
||||
# call, so a projection that guessed low cannot overrun either.
|
||||
_ml_state = self._runtime_state_var.get()
|
||||
_ml_budget = _ml_stage_deadline_seconds() if _ml_state.ml_budget_active else 0.0
|
||||
_remaining = (_ml_budget - _ml_state.ml_elapsed) if _ml_budget > 0 else None
|
||||
# Exhausted is decided before any estimator work, so an over-budget
|
||||
# request stops paying for tokenization too.
|
||||
_over_budget = _remaining is not None and _remaining <= 0.0
|
||||
_block_tokens = -1
|
||||
_projected = 0.0
|
||||
if _remaining is not None and not _over_budget:
|
||||
_rate = _ml_state.ml_seconds_per_token
|
||||
if _rate > 0.0:
|
||||
_block_tokens = _estimate_tokens(text_to_compress)
|
||||
_projected = _block_tokens * _rate
|
||||
_over_budget = _projected > _remaining
|
||||
if _over_budget:
|
||||
self._observe_kompress_size_gate("deadline")
|
||||
if _block_tokens < 0:
|
||||
_block_tokens = _estimate_tokens(text_to_compress)
|
||||
logger.info(
|
||||
"ML stage budget spent (%.1fs of %.1fs used, ~%.1fs projected for a "
|
||||
"~%d tok block, %.1fs left); passing remaining blocks through "
|
||||
"unchanged. Partial compression beats a blown budget.",
|
||||
_ml_state.ml_elapsed,
|
||||
_ml_budget,
|
||||
_projected,
|
||||
_block_tokens,
|
||||
max(0.0, _remaining or 0.0),
|
||||
)
|
||||
# Unchanged, not the size gate's lossy fallback below: this
|
||||
# verdict depends on the request, not the content, and the router
|
||||
# caches it, so the bytes must be ones it can keep sending on
|
||||
# later turns without busting the prompt cache.
|
||||
return content, _estimate_tokens(content)
|
||||
|
||||
if (
|
||||
self._kompress_max_tokens > 0
|
||||
and _estimate_tokens(text_to_compress) > self._kompress_max_tokens
|
||||
@@ -4564,7 +4718,28 @@ class ContentRouter(Transform):
|
||||
compressor, "shares_request_deadline", False
|
||||
):
|
||||
compress_kwargs["_deadline_started_at"] = deadline_origin
|
||||
result = compressor.compress(text_to_compress, **compress_kwargs)
|
||||
# The ML budget's remainder caps THIS call, so the
|
||||
# first one of a request -- admitted without a measured
|
||||
# cost -- cannot run past the budget either. Remote
|
||||
# Kompress takes the cap too (it bounds the whole HTTP
|
||||
# request, and declines the call when too little is
|
||||
# left) but not the shared deadline origin, hence the
|
||||
# separate flag.
|
||||
if _remaining is not None and (
|
||||
getattr(compressor, "shares_request_deadline", False)
|
||||
or getattr(compressor, "accepts_time_budget_cap", False)
|
||||
):
|
||||
compress_kwargs["_time_budget_cap_seconds"] = _remaining
|
||||
_ml_started = time.monotonic()
|
||||
try:
|
||||
result = compressor.compress(text_to_compress, **compress_kwargs)
|
||||
finally:
|
||||
# In a `finally` so a call that RAISES still draws
|
||||
# down the budget: a failing compressor burns the
|
||||
# same non-preemptible wall clock as a passing one,
|
||||
# and not charging for it let a request retry its
|
||||
# way past the ceiling.
|
||||
self._charge_ml_time(time.monotonic() - _ml_started, text_to_compress)
|
||||
compressed = result.compressed
|
||||
compressed_tokens = result.compressed_tokens
|
||||
except Exception as e:
|
||||
@@ -5485,8 +5660,18 @@ class ContentRouter(Transform):
|
||||
# The kompress deadline origin is stamped HERE, with the rest of the
|
||||
# per-request state, so every block this call compresses draws down one
|
||||
# shared budget instead of restarting it.
|
||||
#
|
||||
# The ML ceiling (#3711) is deliberately NOT armed here. It used to be,
|
||||
# which meant lifecycle work, message classification, lossless
|
||||
# transforms and task preparation all ran down a budget whose log line
|
||||
# claimed to measure time "in ML". `_try_ml_compressor` accumulates the
|
||||
# time it actually spends inside the compressor instead, so the number
|
||||
# means what it says.
|
||||
self._runtime_state_var.set(
|
||||
_PerRequestRuntimeState(kompress_deadline_started_at=time.perf_counter())
|
||||
_PerRequestRuntimeState(
|
||||
kompress_deadline_started_at=time.perf_counter(),
|
||||
ml_budget_active=True,
|
||||
)
|
||||
)
|
||||
|
||||
# Pre-process: Read lifecycle management (stale/superseded detection)
|
||||
|
||||
@@ -441,6 +441,21 @@ def _time_budget_seconds() -> float | None:
|
||||
return raw if raw > 0 else None
|
||||
|
||||
|
||||
def _capped_time_budget(cap: float | None) -> float | None:
|
||||
"""The per-call budget, lowered to *cap* when the caller has less to give.
|
||||
|
||||
The router passes what is left of its request-scoped ML budget, so a call
|
||||
-- including the first one of a request, which has no measured cost to be
|
||||
refused on -- stops at a chunk boundary when that budget runs out instead
|
||||
of running to its own, larger, per-call budget.
|
||||
"""
|
||||
budget = _time_budget_seconds()
|
||||
if cap is None:
|
||||
return budget
|
||||
cap = max(0.0, cap)
|
||||
return cap if budget is None else min(budget, cap)
|
||||
|
||||
|
||||
def _canary_threshold_seconds() -> float | None:
|
||||
"""Startup canary threshold; inference slower than this disables Kompress.
|
||||
|
||||
@@ -1616,11 +1631,15 @@ class KompressCompressor(Transform):
|
||||
|
||||
# ``compress()`` accepts ``_deadline_started_at``, so a caller that
|
||||
# compresses many blocks for ONE request can hand every call the same
|
||||
# origin and have the deadline bound the request. Duck-typed rather than
|
||||
# origin and have the deadline bound the request. It also accepts
|
||||
# ``_time_budget_cap_seconds``, which lowers this call's time budget to
|
||||
# what the caller's request-scoped ML budget has left. Duck-typed rather than
|
||||
# isinstance-checked at the call site because ``RemoteKompressCompressor``
|
||||
# is the other compressor the router may get back and its ``compress()``
|
||||
# does not take the argument.
|
||||
# is the other compressor the router may get back: its ``compress()`` takes
|
||||
# the cap (flagged by ``accepts_time_budget_cap``) but not the deadline
|
||||
# origin.
|
||||
shares_request_deadline: bool = True
|
||||
accepts_time_budget_cap: bool = True
|
||||
|
||||
def __init__(self, config: KompressConfig | None = None):
|
||||
self.config = config or KompressConfig()
|
||||
@@ -1762,6 +1781,7 @@ class KompressCompressor(Transform):
|
||||
allow_download: bool = True,
|
||||
ccr_original: str | None = None,
|
||||
_deadline_started_at: float | None = None,
|
||||
_time_budget_cap_seconds: float | None = None,
|
||||
) -> KompressResult:
|
||||
"""Compress content using Kompress model.
|
||||
|
||||
@@ -1824,6 +1844,7 @@ class KompressCompressor(Transform):
|
||||
batch_size=_batch_size(),
|
||||
ccr_originals=[ccr_original],
|
||||
_deadline_started_at=t_deadline,
|
||||
_time_budget_cap_seconds=_time_budget_cap_seconds,
|
||||
)
|
||||
if batch_result:
|
||||
return batch_result[0]
|
||||
@@ -1834,7 +1855,7 @@ class KompressCompressor(Transform):
|
||||
chunk_count = 0
|
||||
|
||||
acquire_timeout = _acquire_timeout_seconds()
|
||||
budget = _time_budget_seconds()
|
||||
budget = _capped_time_budget(_time_budget_cap_seconds)
|
||||
deadline = time.monotonic() + budget if budget is not None else None
|
||||
|
||||
for chunk_start in range(0, n_words, max_chunk_words):
|
||||
@@ -2104,6 +2125,7 @@ class KompressCompressor(Transform):
|
||||
*,
|
||||
ccr_originals: list[str | None] | None = None,
|
||||
_deadline_started_at: float | None = None,
|
||||
_time_budget_cap_seconds: float | None = None,
|
||||
) -> list[KompressResult]:
|
||||
"""Compress multiple texts. Uses batched inference on GPU, sequential on CPU.
|
||||
|
||||
@@ -2198,6 +2220,12 @@ class KompressCompressor(Transform):
|
||||
# per-item slowdown measured on ONNX CPU (~0.7-0.9x vs sequential).
|
||||
# GPU users still benefit from the batched forward pass below.
|
||||
if self._should_use_sequential_fallback():
|
||||
# The cap is for the whole batch, so each item gets what is left.
|
||||
cap_ends_at = (
|
||||
None
|
||||
if _time_budget_cap_seconds is None
|
||||
else time.monotonic() + _time_budget_cap_seconds
|
||||
)
|
||||
return [
|
||||
self.compress(
|
||||
content,
|
||||
@@ -2207,6 +2235,9 @@ class KompressCompressor(Transform):
|
||||
target_ratio=r,
|
||||
ccr_original=ccr_source,
|
||||
_deadline_started_at=t_deadline,
|
||||
_time_budget_cap_seconds=(
|
||||
None if cap_ends_at is None else cap_ends_at - time.monotonic()
|
||||
),
|
||||
)
|
||||
for content, r, ccr_source in zip(contents, ratios, ccr_sources, strict=True)
|
||||
]
|
||||
@@ -2257,7 +2288,7 @@ class KompressCompressor(Transform):
|
||||
self._deadline_s = deadline_s
|
||||
|
||||
acquire_timeout = _acquire_timeout_seconds()
|
||||
budget = _time_budget_seconds()
|
||||
budget = _capped_time_budget(_time_budget_cap_seconds)
|
||||
deadline = time.monotonic() + budget if budget is not None else None
|
||||
|
||||
def _bail_remaining(reason: str, batch_start: int) -> None:
|
||||
|
||||
@@ -53,7 +53,12 @@ never correctness.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import concurrent.futures
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
@@ -102,6 +107,21 @@ _MIN_WORDS = 10
|
||||
# Accept-any-shrink CCR gate, identical to KompressCompressor.compress: only
|
||||
# store + mark when the shrink is worth the retrieval marker's own cost.
|
||||
|
||||
# A remote call is declined outright when the caller's request-scoped ML budget
|
||||
# has this little left: no HTTP round trip (connect + write + server inference +
|
||||
# read) fits in it, so starting one only burns the budget and leaves a worker
|
||||
# behind. The block passes through unchanged instead.
|
||||
_MIN_REMOTE_BUDGET_SECONDS = 0.05
|
||||
|
||||
# Workers for budget-capped calls, and the number of admission permits. A call
|
||||
# whose caller times out is abandoned, not killed (a sync httpx request cannot be
|
||||
# interrupted from another thread), so the worker keeps its permit until the
|
||||
# request itself ends -- and every request ends at an absolute deadline (see
|
||||
# ``_capped_request``). With permits == workers nothing ever waits in the
|
||||
# executor's (unbounded) queue: a saturated caller waits for a permit, not with
|
||||
# its payload enqueued, and fails open when its own deadline passes.
|
||||
_CAPPED_CALL_WORKERS = 16
|
||||
|
||||
|
||||
class RemoteKompressCompressor:
|
||||
"""Drop-in for KompressCompressor that POSTs to a hosted ``/compress`` endpoint.
|
||||
@@ -112,6 +132,12 @@ class RemoteKompressCompressor:
|
||||
|
||||
name = "kompress_compressor"
|
||||
|
||||
# ``compress()`` accepts ``_time_budget_cap_seconds``: what the router's
|
||||
# request-scoped ML budget has left. Unlike local Kompress this class does
|
||||
# not take ``_deadline_started_at``, so it does not set
|
||||
# ``shares_request_deadline``; the router checks this flag separately.
|
||||
accepts_time_budget_cap: bool = True
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
endpoint: str,
|
||||
@@ -120,6 +146,7 @@ class RemoteKompressCompressor:
|
||||
timeout: float = 20.0,
|
||||
path: str | None = DEFAULT_ENDPOINT_PATH,
|
||||
headers: dict[str, str] | None = None,
|
||||
transport: httpx.BaseTransport | None = None,
|
||||
) -> None:
|
||||
self.config = config or KompressConfig()
|
||||
# Default keeps the pre-existing behaviour exactly: <endpoint>/compress.
|
||||
@@ -148,7 +175,13 @@ class RemoteKompressCompressor:
|
||||
# Kompress, rather than discovering it in a packet capture.
|
||||
guard_egress("remote Kompress inference", self._url)
|
||||
# httpx.Client is safe to share across the proxy's worker threads.
|
||||
self._client = httpx.Client(timeout=timeout)
|
||||
# ``transport`` exists for tests (httpx.MockTransport); None is the
|
||||
# normal network transport.
|
||||
self._timeout = timeout
|
||||
self._client = httpx.Client(timeout=timeout, transport=transport)
|
||||
self._capped_pool: concurrent.futures.ThreadPoolExecutor | None = None
|
||||
self._capped_pool_lock = threading.Lock()
|
||||
self._capped_slots = threading.BoundedSemaphore(_CAPPED_CALL_WORKERS)
|
||||
|
||||
@property
|
||||
def url(self) -> str:
|
||||
@@ -193,9 +226,18 @@ class RemoteKompressCompressor:
|
||||
*,
|
||||
allow_download: bool = True,
|
||||
ccr_original: str | None = None,
|
||||
_time_budget_cap_seconds: float | None = None,
|
||||
) -> KompressResult:
|
||||
"""Compress via the remote endpoint.
|
||||
|
||||
``_time_budget_cap_seconds`` is what is left of the caller's
|
||||
request-scoped ML budget (ContentRouter passes it). It bounds the HTTP
|
||||
request AS A WHOLE -- connect, write, server time and read together,
|
||||
not each phase -- and a call that runs out passes the block through
|
||||
unchanged. When the remainder is at or below
|
||||
:data:`_MIN_REMOTE_BUDGET_SECONDS` no request is made at all. ``None``
|
||||
(direct callers) keeps the configured per-phase timeout, 20s by default.
|
||||
|
||||
``ccr_original`` mirrors :meth:`KompressCompressor.compress`: text to
|
||||
store in CCR instead of ``content``, used when ``content`` is a
|
||||
tag-protected placeholder intermediate ({{HEADROOM_TAG_N}}). It is
|
||||
@@ -230,14 +272,23 @@ class RemoteKompressCompressor:
|
||||
# the hole this guard closes.
|
||||
guard_egress("remote Kompress inference", self._url)
|
||||
|
||||
try:
|
||||
resp = self._client.post(
|
||||
self._url,
|
||||
headers=self._headers,
|
||||
json={"content": content, "target_ratio": target_ratio},
|
||||
payload = {"content": content, "target_ratio": target_ratio}
|
||||
if _time_budget_cap_seconds is not None and (
|
||||
_time_budget_cap_seconds <= _MIN_REMOTE_BUDGET_SECONDS
|
||||
):
|
||||
logger.info(
|
||||
"Remote Kompress declined: %.3fs of request ML budget left (floor %.3fs); "
|
||||
"passing through unchanged",
|
||||
max(0.0, _time_budget_cap_seconds),
|
||||
_MIN_REMOTE_BUDGET_SECONDS,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return self._passthrough(content, n_words)
|
||||
|
||||
try:
|
||||
if _time_budget_cap_seconds is None:
|
||||
data = self._post(payload)
|
||||
else:
|
||||
data = self._post_within(payload, _time_budget_cap_seconds)
|
||||
compressed = data["compressed"]
|
||||
if not isinstance(compressed, str):
|
||||
raise TypeError("remote Kompress response field 'compressed' must be a string")
|
||||
@@ -299,5 +350,95 @@ class RemoteKompressCompressor:
|
||||
|
||||
return result
|
||||
|
||||
def _post(self, payload: dict[str, Any], timeout: float | None = None) -> Any:
|
||||
# No ``timeout`` kwarg on the default path: the client's own (20s)
|
||||
# applies, and the call is exactly what it was before the cap existed.
|
||||
extra: dict[str, Any] = {} if timeout is None else {"timeout": timeout}
|
||||
resp = self._client.post(self._url, headers=self._headers, json=payload, **extra)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
def _post_within(self, payload: dict[str, Any], cap: float) -> Any:
|
||||
"""POST and read the response within *cap* seconds in total.
|
||||
|
||||
httpx timeouts are per phase (connect, write, pool, and each read
|
||||
between chunks), so ``timeout=cap`` alone neither bounds the call nor
|
||||
stops a slow-drip body. Three things together do:
|
||||
|
||||
* **Admission.** A permit (one per worker) is taken before submitting,
|
||||
waiting at most until the caller's deadline. Saturated means fail
|
||||
open, with the payload never enqueued -- ``future.cancel()`` does not
|
||||
free a queued payload, so an unbounded queue would retain every
|
||||
timed-out prompt.
|
||||
* **Caller deadline.** The caller waits for the worker with one overall
|
||||
deadline and passes the block through unchanged when it runs out.
|
||||
* **Worker deadline.** The worker keeps its permit until it actually
|
||||
finishes (released in its own ``finally``, never on caller timeout),
|
||||
and it finishes by an absolute deadline: per-phase timeouts are capped
|
||||
to what is left, and the body is streamed and cut off at the
|
||||
deadline, so a dripping response cannot hold a permit indefinitely.
|
||||
"""
|
||||
deadline = time.monotonic() + cap
|
||||
if not self._capped_slots.acquire(timeout=max(0.0, cap)):
|
||||
raise TimeoutError(
|
||||
f"all {_CAPPED_CALL_WORKERS} remote Kompress workers busy for the "
|
||||
f"request ML budget ({cap:.3f}s)"
|
||||
)
|
||||
try:
|
||||
future = self._get_capped_pool().submit(self._capped_request, payload, deadline)
|
||||
except BaseException:
|
||||
self._capped_slots.release() # never reached the worker that would
|
||||
raise
|
||||
try:
|
||||
return future.result(timeout=max(0.0, deadline - time.monotonic()))
|
||||
except concurrent.futures.TimeoutError:
|
||||
raise TimeoutError(
|
||||
f"exceeded the request ML budget ({cap:.3f}s) before the endpoint answered"
|
||||
) from None
|
||||
|
||||
def _capped_request(self, payload: dict[str, Any], deadline: float) -> Any:
|
||||
"""The worker side of :meth:`_post_within`. Owns one admission permit.
|
||||
|
||||
Lifetime is bounded by *deadline* plus at most one per-phase timeout
|
||||
(itself no longer than what was left at start): every httpx phase is
|
||||
capped to the time remaining, and the body is read chunk by chunk with
|
||||
the absolute deadline checked after each one.
|
||||
"""
|
||||
try:
|
||||
left = deadline - time.monotonic()
|
||||
if left <= 0.0:
|
||||
raise TimeoutError("request ML budget spent before the request started")
|
||||
with self._client.stream(
|
||||
"POST",
|
||||
self._url,
|
||||
headers=self._headers,
|
||||
json=payload,
|
||||
timeout=min(left, self._timeout),
|
||||
) as resp:
|
||||
resp.raise_for_status()
|
||||
body = bytearray()
|
||||
for chunk in resp.iter_bytes():
|
||||
body += chunk
|
||||
if time.monotonic() >= deadline:
|
||||
raise TimeoutError(
|
||||
"remote Kompress response outlived the request ML budget"
|
||||
)
|
||||
return json.loads(bytes(body))
|
||||
finally:
|
||||
self._capped_slots.release()
|
||||
|
||||
def _get_capped_pool(self) -> concurrent.futures.ThreadPoolExecutor:
|
||||
with self._capped_pool_lock:
|
||||
if self._capped_pool is None:
|
||||
self._capped_pool = concurrent.futures.ThreadPoolExecutor(
|
||||
max_workers=_CAPPED_CALL_WORKERS,
|
||||
thread_name_prefix="remote-kompress",
|
||||
)
|
||||
return self._capped_pool
|
||||
|
||||
def close(self) -> None:
|
||||
with self._capped_pool_lock:
|
||||
pool, self._capped_pool = self._capped_pool, None
|
||||
if pool is not None:
|
||||
pool.shutdown(wait=False, cancel_futures=True)
|
||||
self._client.close()
|
||||
|
||||
@@ -0,0 +1,678 @@
|
||||
"""Request-scoped ML ceiling for the mixed-content path (#3711).
|
||||
|
||||
``HEADROOM_KOMPRESS_MAX_TOKENS`` bounds a single block. It cannot bound a
|
||||
request: ``_compress_mixed`` splits a payload into sections and calls
|
||||
``_try_ml_compressor`` once per section, so every section can sit under the
|
||||
per-block ceiling while their sum runs for minutes. The reporter measured ~70s
|
||||
on a 1.4MB ``tool_result`` of prose blocks separated by small JSON objects,
|
||||
which blew the 30s compression budget and then quarantined compression for
|
||||
every following request -- while
|
||||
``headroom_kompress_size_gate_total`` recorded only ``within``.
|
||||
|
||||
A stub stands in for Kompress so these assert the *budget*, not ONNX latency:
|
||||
the real model is not available in CI, and the bug is about how many times a
|
||||
slow stage is entered, not how slow it is.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import concurrent.futures
|
||||
import json
|
||||
import random
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from headroom.transforms import content_router as cr
|
||||
from headroom.transforms.kompress_remote import RemoteKompressCompressor
|
||||
|
||||
|
||||
class _Tokenizer:
|
||||
def count_text(self, content: str) -> int:
|
||||
return len(content.split())
|
||||
|
||||
|
||||
@dataclass
|
||||
class _StubResult:
|
||||
compressed: str
|
||||
compressed_tokens: int
|
||||
|
||||
|
||||
class _SlowKompress:
|
||||
"""Stands in for Kompress: ready, and costs `per_call_s` every call."""
|
||||
|
||||
def __init__(self, per_call_s: float) -> None:
|
||||
self.per_call_s = per_call_s
|
||||
self.calls = 0
|
||||
|
||||
def is_ready(self) -> bool:
|
||||
return True
|
||||
|
||||
def ensure_background_load(self) -> None: # pragma: no cover - never reached
|
||||
raise AssertionError("stub is always ready")
|
||||
|
||||
def compress(self, text: str, **_kwargs: object) -> _StubResult:
|
||||
self.calls += 1
|
||||
time.sleep(self.per_call_s)
|
||||
out = text[: max(1, len(text) // 2)]
|
||||
return _StubResult(compressed=out, compressed_tokens=cr._estimate_tokens(out))
|
||||
|
||||
|
||||
def _prose(n_chars: int, seed: int = 7) -> str:
|
||||
rng = random.Random(seed)
|
||||
words = (
|
||||
"analysis deployment configuration throughput latency pipeline compression "
|
||||
"artifact identifier resolution boundary telemetry inference workspace"
|
||||
).split()
|
||||
out: list[str] = []
|
||||
total = 0
|
||||
while total < n_chars:
|
||||
line = " ".join(rng.choice(words) for _ in range(14))
|
||||
out.append(line)
|
||||
total += len(line) + 1
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def _reporter_payload(block_chars: int = 40_000, blocks: int = 8) -> str:
|
||||
"""Prose blocks separated by small JSON objects -- the #3711 shape.
|
||||
|
||||
Each block stays under the 50k-token per-block gate, so the size gate
|
||||
cannot fire; only a request-scoped ceiling can stop this.
|
||||
"""
|
||||
parts: list[str] = []
|
||||
for i in range(blocks):
|
||||
parts.append(_prose(block_chars, seed=i))
|
||||
parts.append(json.dumps({"id": i, "status": "ok", "note": f"marker {i}"}))
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
def _tool_result_messages() -> list[dict]:
|
||||
"""The reported shape: the payload arrives as a tool_result, not user text.
|
||||
|
||||
Plain user text is protected from compression
|
||||
(``router:protected:user_message``) and never reaches the ML stage at all.
|
||||
"""
|
||||
return [
|
||||
{"role": "user", "content": "go"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [{"type": "tool_use", "id": "t1", "name": "Bash", "input": {}}],
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "tool_result", "tool_use_id": "t1", "content": _reporter_payload()}
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def gate_outcomes(monkeypatch) -> dict[str, int]:
|
||||
seen: dict[str, int] = {}
|
||||
monkeypatch.setattr(
|
||||
cr.ContentRouter,
|
||||
"_observe_kompress_size_gate",
|
||||
lambda self, outcome: seen.__setitem__(outcome, seen.get(outcome, 0) + 1),
|
||||
)
|
||||
return seen
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def slow_kompress(monkeypatch) -> _SlowKompress:
|
||||
stub = _SlowKompress(per_call_s=0.05)
|
||||
monkeypatch.setattr(cr.ContentRouter, "_get_kompress", lambda self: stub)
|
||||
return stub
|
||||
|
||||
|
||||
def _apply(router: cr.ContentRouter) -> None:
|
||||
router.apply(
|
||||
_tool_result_messages(),
|
||||
_Tokenizer(),
|
||||
frozen_message_count=1,
|
||||
min_tokens_to_compress=1,
|
||||
)
|
||||
|
||||
|
||||
def test_size_gate_alone_cannot_bound_a_request(gate_outcomes, slow_kompress, monkeypatch) -> None:
|
||||
"""Every section passes the per-block gate -- this is the #3711 precondition.
|
||||
|
||||
Reproduces the reporter's metric exactly: many ``within`` decisions and no
|
||||
``exceeded``. If this ever records ``exceeded`` the payload has stopped
|
||||
reproducing the report, and the deadline test below would pass for the
|
||||
wrong reason.
|
||||
"""
|
||||
monkeypatch.setenv("HEADROOM_ML_STAGE_DEADLINE_SECONDS", "0") # isolate the size gate
|
||||
|
||||
_apply(cr.ContentRouter(cr.ContentRouterConfig()))
|
||||
|
||||
assert gate_outcomes.get("exceeded", 0) == 0, (
|
||||
f"payload no longer reproduces #3711 (a section exceeded the gate): {gate_outcomes}"
|
||||
)
|
||||
assert gate_outcomes.get("within", 0) > 1, (
|
||||
"expected repeated per-section ML entry, the shape that blows the budget; "
|
||||
f"got {gate_outcomes}"
|
||||
)
|
||||
assert slow_kompress.calls > 1, "the ML stage was entered once or not at all"
|
||||
|
||||
|
||||
def test_ml_deadline_stops_further_ml_work_once_the_budget_is_spent(
|
||||
gate_outcomes, slow_kompress, monkeypatch
|
||||
) -> None:
|
||||
"""Past the ceiling, remaining sections route off ML instead of compounding."""
|
||||
monkeypatch.setenv("HEADROOM_ML_STAGE_DEADLINE_SECONDS", "0.06")
|
||||
|
||||
_apply(cr.ContentRouter(cr.ContentRouterConfig()))
|
||||
|
||||
assert gate_outcomes.get("deadline", 0) > 0, (
|
||||
f"request-scoped ceiling never fired: {gate_outcomes}"
|
||||
)
|
||||
|
||||
# The saving has to be real, so measure it: the same payload with the
|
||||
# ceiling disabled must enter the model strictly more often.
|
||||
bounded_calls = slow_kompress.calls
|
||||
monkeypatch.setenv("HEADROOM_ML_STAGE_DEADLINE_SECONDS", "0")
|
||||
unbounded = _SlowKompress(per_call_s=0.05)
|
||||
monkeypatch.setattr(cr.ContentRouter, "_get_kompress", lambda self: unbounded)
|
||||
|
||||
_apply(cr.ContentRouter(cr.ContentRouterConfig()))
|
||||
|
||||
assert bounded_calls < unbounded.calls, (
|
||||
f"ceiling skipped no model calls: {bounded_calls} with it, {unbounded.calls} without"
|
||||
)
|
||||
|
||||
|
||||
def test_deadline_zero_restores_previous_unbounded_behaviour(
|
||||
gate_outcomes, slow_kompress, monkeypatch
|
||||
) -> None:
|
||||
monkeypatch.setenv("HEADROOM_ML_STAGE_DEADLINE_SECONDS", "0")
|
||||
|
||||
_apply(cr.ContentRouter(cr.ContentRouterConfig()))
|
||||
|
||||
assert gate_outcomes.get("deadline", 0) == 0, (
|
||||
f"deadline fired although disabled: {gate_outcomes}"
|
||||
)
|
||||
|
||||
|
||||
def test_direct_compress_callers_stay_unarmed() -> None:
|
||||
"""``compress()`` without ``apply()`` keeps the old unbounded behaviour.
|
||||
|
||||
Tests and the ``/v1/compress`` path call ``compress()`` directly; they must
|
||||
not inherit a request budget nobody set.
|
||||
"""
|
||||
router = cr.ContentRouter(cr.ContentRouterConfig())
|
||||
state = router._runtime_state_var.get()
|
||||
assert state.ml_budget_active is False
|
||||
|
||||
# That default is instance-scoped and shared by every direct call for the
|
||||
# life of the router, so charging it would accumulate across unrelated
|
||||
# calls until ML switched itself off permanently.
|
||||
router._charge_ml_time(5.0, "some text to compress")
|
||||
assert state.ml_elapsed == 0.0
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Review follow-ups: the budget must mean ML time, must refuse a call it cannot
|
||||
# afford, and must sit under the timeout that quarantines the stage.
|
||||
#
|
||||
# The original tests asserted that LATER model calls are skipped. That is not
|
||||
# the same claim as the request-wide time bound the module documents, and all
|
||||
# three gaps below passed those tests.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
class _SlowTokenizer(_Tokenizer):
|
||||
"""Expensive NON-ML prework: token counting during classification."""
|
||||
|
||||
def __init__(self, per_call_s: float) -> None:
|
||||
self.per_call_s = per_call_s
|
||||
self.calls = 0
|
||||
|
||||
def count_text(self, content: str) -> int:
|
||||
self.calls += 1
|
||||
time.sleep(self.per_call_s)
|
||||
return super().count_text(content)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def charged(monkeypatch) -> list[float]:
|
||||
"""Every increment billed to the ML budget, in order."""
|
||||
seen: list[float] = []
|
||||
real = cr.ContentRouter._charge_ml_time
|
||||
|
||||
def _spy(self, elapsed: float, text: str) -> None:
|
||||
seen.append(elapsed)
|
||||
real(self, elapsed, text)
|
||||
|
||||
monkeypatch.setattr(cr.ContentRouter, "_charge_ml_time", _spy)
|
||||
return seen
|
||||
|
||||
|
||||
def test_non_ml_prework_does_not_consume_the_ml_budget(
|
||||
gate_outcomes, slow_kompress, charged, monkeypatch
|
||||
) -> None:
|
||||
"""The budget was armed at ``apply()`` entry, so lifecycle work, message
|
||||
classification and lossless transforms all ran it down before ML began --
|
||||
while the log line claimed to measure time "in ML"."""
|
||||
monkeypatch.setenv("HEADROOM_ML_STAGE_DEADLINE_SECONDS", "0.4")
|
||||
router = cr.ContentRouter(cr.ContentRouterConfig(enable_kompress=True))
|
||||
|
||||
tokenizer = _SlowTokenizer(per_call_s=0.01)
|
||||
started = time.monotonic()
|
||||
router.apply(
|
||||
_tool_result_messages(),
|
||||
tokenizer,
|
||||
frozen_message_count=1,
|
||||
min_tokens_to_compress=1,
|
||||
)
|
||||
wall = time.monotonic() - started
|
||||
|
||||
assert tokenizer.calls > 0, "prework must actually have run"
|
||||
assert slow_kompress.calls > 0, "prework time must not have spent the ML budget"
|
||||
# The bound is on ML time, not on the call.
|
||||
assert sum(charged) <= wall
|
||||
assert sum(charged) == pytest.approx(slow_kompress.calls * slow_kompress.per_call_s, rel=0.5)
|
||||
|
||||
|
||||
def test_a_slow_call_cannot_be_admitted_against_a_budget_it_will_overrun(
|
||||
gate_outcomes, charged, monkeypatch
|
||||
) -> None:
|
||||
"""Admission used to be ``now >= deadline`` only, so a call starting at
|
||||
14.9s of a 15s budget ran to completion well past it -- the ONNX worker is
|
||||
not preemptible, so nothing stops it once it has begun."""
|
||||
monkeypatch.setenv("HEADROOM_ML_STAGE_DEADLINE_SECONDS", "0.5")
|
||||
stub = _SlowKompress(per_call_s=0.35)
|
||||
monkeypatch.setattr(cr.ContentRouter, "_get_kompress", lambda self: stub)
|
||||
router = cr.ContentRouter(cr.ContentRouterConfig(enable_kompress=True))
|
||||
|
||||
_apply(router)
|
||||
|
||||
assert stub.calls >= 1, "the first block has no measurement yet and must run"
|
||||
# The point of the change: total ML time stays inside the budget, rather
|
||||
# than the budget merely being the moment after which no call STARTS.
|
||||
assert sum(charged) <= 0.5, (
|
||||
f"ML overran its 0.5s budget: {sum(charged):.2f}s across {stub.calls} calls"
|
||||
)
|
||||
assert gate_outcomes.get("deadline", 0) >= 1, "remaining blocks must route off ML"
|
||||
|
||||
|
||||
def test_budget_is_clamped_below_the_timeout_that_quarantines_the_stage(
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
"""A 15s guard under a 10s executor timeout can never fire first, which is
|
||||
exactly the failure the ceiling exists to prevent."""
|
||||
monkeypatch.delenv("HEADROOM_ML_STAGE_DEADLINE_SECONDS", raising=False)
|
||||
|
||||
monkeypatch.setenv("HEADROOM_COMPRESSION_TIMEOUT_SECONDS", "30")
|
||||
assert cr._ml_stage_deadline_seconds() == 15.0, "the documented default is unchanged"
|
||||
|
||||
monkeypatch.setenv("HEADROOM_COMPRESSION_TIMEOUT_SECONDS", "10")
|
||||
clamped = cr._ml_stage_deadline_seconds()
|
||||
assert clamped == 5.0
|
||||
assert clamped < 10.0, "must leave room to degrade before the executor gives up"
|
||||
|
||||
# The clamp only ever lowers: an explicit request below it is honoured.
|
||||
monkeypatch.setenv("HEADROOM_ML_STAGE_DEADLINE_SECONDS", "2")
|
||||
assert cr._ml_stage_deadline_seconds() == 2.0
|
||||
|
||||
# And 0 still disables the ceiling entirely.
|
||||
monkeypatch.setenv("HEADROOM_ML_STAGE_DEADLINE_SECONDS", "0")
|
||||
assert cr._ml_stage_deadline_seconds() == 0.0
|
||||
|
||||
|
||||
def test_a_failing_ml_call_still_draws_down_the_budget(charged, monkeypatch) -> None:
|
||||
"""Otherwise a request could retry its way past the ceiling: a compressor
|
||||
that raises burns the same non-preemptible wall clock as one that returns."""
|
||||
monkeypatch.setenv("HEADROOM_ML_STAGE_DEADLINE_SECONDS", "5")
|
||||
|
||||
class _Exploding(_SlowKompress):
|
||||
def compress(self, text: str, **_kwargs: object):
|
||||
self.calls += 1
|
||||
time.sleep(self.per_call_s)
|
||||
raise RuntimeError("onnx said no")
|
||||
|
||||
stub = _Exploding(per_call_s=0.05)
|
||||
monkeypatch.setattr(cr.ContentRouter, "_get_kompress", lambda self: stub)
|
||||
router = cr.ContentRouter(cr.ContentRouterConfig(enable_kompress=True))
|
||||
|
||||
_apply(router)
|
||||
|
||||
assert stub.calls > 0
|
||||
assert charged, "a raising call must still be billed"
|
||||
assert sum(charged) > 0.0
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Second review: the FIRST ML call of a request was always admitted (the seed
|
||||
# rate was 0, and there is no measurement yet), and it is non-preemptible, so
|
||||
# one huge first block could still run past the ML budget and the executor
|
||||
# timeout. The call is now capped rather than estimated.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
class _CooperativeKompress(_SlowKompress):
|
||||
"""Behaves like Kompress under a time cap: stops at a chunk boundary once
|
||||
``_time_budget_cap_seconds`` is spent and returns the block unchanged."""
|
||||
|
||||
shares_request_deadline = True
|
||||
|
||||
def __init__(self, worst_case_s: float, chunk_s: float = 0.02) -> None:
|
||||
super().__init__(per_call_s=worst_case_s)
|
||||
self.chunk_s = chunk_s
|
||||
self.caps: list[float | None] = []
|
||||
|
||||
def compress(self, text: str, **kwargs: object) -> _StubResult:
|
||||
self.calls += 1
|
||||
cap = kwargs.get("_time_budget_cap_seconds")
|
||||
self.caps.append(cap) # type: ignore[arg-type]
|
||||
ends_at = None if cap is None else time.monotonic() + float(cap) # type: ignore[arg-type]
|
||||
spent = 0.0
|
||||
while spent < self.per_call_s:
|
||||
if ends_at is not None and time.monotonic() >= ends_at:
|
||||
return _StubResult(compressed=text, compressed_tokens=cr._estimate_tokens(text))
|
||||
time.sleep(self.chunk_s)
|
||||
spent += self.chunk_s
|
||||
out = text[: max(1, len(text) // 2)]
|
||||
return _StubResult(compressed=out, compressed_tokens=cr._estimate_tokens(out))
|
||||
|
||||
|
||||
def test_first_ml_call_is_capped_by_the_budget_it_was_admitted_against(
|
||||
charged, monkeypatch
|
||||
) -> None:
|
||||
"""A first block whose worst case (3s) exceeds both the outer timeout (1s)
|
||||
and the ML budget derived from it (0.5s) must still stop inside the budget."""
|
||||
monkeypatch.delenv("HEADROOM_ML_STAGE_DEADLINE_SECONDS", raising=False)
|
||||
monkeypatch.setenv("HEADROOM_COMPRESSION_TIMEOUT_SECONDS", "1")
|
||||
budget = cr._ml_stage_deadline_seconds()
|
||||
assert budget == 0.5
|
||||
|
||||
stub = _CooperativeKompress(worst_case_s=3.0)
|
||||
monkeypatch.setattr(cr.ContentRouter, "_get_kompress", lambda self: stub)
|
||||
|
||||
_apply(cr.ContentRouter(cr.ContentRouterConfig(enable_kompress=True)))
|
||||
|
||||
assert stub.caps, "the first block must still reach the model"
|
||||
first_cap = stub.caps[0]
|
||||
assert first_cap is not None and 0.0 < first_cap <= budget, (
|
||||
f"first call was not capped by the ML budget: {stub.caps}"
|
||||
)
|
||||
# One chunk of slack: the cap is checked at chunk boundaries, as in Kompress.
|
||||
assert sum(charged) <= budget + 0.2, (
|
||||
f"ML ran {sum(charged):.2f}s against a {budget}s budget under a 1s executor timeout"
|
||||
)
|
||||
|
||||
|
||||
def test_blocks_refused_by_the_budget_pass_through_unchanged(monkeypatch) -> None:
|
||||
"""Not the size gate's lossy fallback: the refusal depends on the request,
|
||||
the router caches the result, and only raw bytes stay the same on later
|
||||
turns, so the prompt cache holds."""
|
||||
router = cr.ContentRouter(cr.ContentRouterConfig(enable_kompress=True))
|
||||
router._runtime_state_var.set(
|
||||
cr._PerRequestRuntimeState(ml_budget_active=True, ml_elapsed=999.0)
|
||||
)
|
||||
|
||||
class _Crusher:
|
||||
def compress(self, text: str, context: str = "") -> _StubResult:
|
||||
return _StubResult(compressed="crushed", compressed_tokens=1)
|
||||
|
||||
monkeypatch.setattr(router, "_get_text_crusher", lambda: _Crusher())
|
||||
stub = _SlowKompress(per_call_s=0.0)
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: stub)
|
||||
|
||||
text = _prose(2_000)
|
||||
out, _ = router._try_ml_compressor(text, "")
|
||||
|
||||
assert out == text
|
||||
assert stub.calls == 0
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Third review: the cap reached only LOCAL Kompress. RemoteKompressCompressor
|
||||
# got no cap and kept its independent 20s per-phase HTTP timeouts, so a cold
|
||||
# remote call -- no measured rate, so nothing to refuse it on -- could run past
|
||||
# both the ML budget and the outer timeout. These drive the real remote class
|
||||
# through the public apply() with an httpx.MockTransport (no network).
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def _tool_result_text(messages: list[dict]) -> str:
|
||||
return messages[-1]["content"][0]["content"]
|
||||
|
||||
|
||||
def _assert_prose_blocks_unchanged(messages: list[dict]) -> None:
|
||||
"""Every ML-eligible prose block reaches the provider byte-identical.
|
||||
|
||||
(The small JSON separators go through the lossless JSON path, which
|
||||
minifies them; that is not ML and not what is under test.)
|
||||
"""
|
||||
out = _tool_result_text(messages)
|
||||
for i in range(8):
|
||||
assert _prose(40_000, seed=i) in out, f"prose block {i} was altered"
|
||||
|
||||
|
||||
def _remote_kompress(handler) -> RemoteKompressCompressor:
|
||||
return RemoteKompressCompressor(
|
||||
endpoint="http://kompress.invalid",
|
||||
transport=httpx.MockTransport(handler),
|
||||
)
|
||||
|
||||
|
||||
def _compressed_reply(request: httpx.Request) -> httpx.Response:
|
||||
content = json.loads(request.content)["content"]
|
||||
return httpx.Response(200, json={"compressed": " ".join(content.split()[::2])})
|
||||
|
||||
|
||||
def test_cold_remote_call_cannot_overrun_a_tiny_ml_budget(charged, monkeypatch) -> None:
|
||||
"""The reviewer's probe: 10 ms ML budget, 40 ms outer timeout, an endpoint
|
||||
that takes 100 ms. No HTTP round trip fits in what is left, so the remote
|
||||
call is declined and every block passes through unchanged."""
|
||||
monkeypatch.setenv("HEADROOM_ML_STAGE_DEADLINE_SECONDS", "0.01")
|
||||
monkeypatch.setenv("HEADROOM_COMPRESSION_TIMEOUT_SECONDS", "0.04")
|
||||
assert cr._ml_stage_deadline_seconds() == 0.01
|
||||
|
||||
requests: list[httpx.Request] = []
|
||||
|
||||
def _slow_endpoint(request: httpx.Request) -> httpx.Response:
|
||||
requests.append(request)
|
||||
time.sleep(0.1)
|
||||
return _compressed_reply(request)
|
||||
|
||||
remote = _remote_kompress(_slow_endpoint)
|
||||
monkeypatch.setattr(cr.ContentRouter, "_get_kompress", lambda self: remote)
|
||||
router = cr.ContentRouter(cr.ContentRouterConfig(enable_kompress=True))
|
||||
|
||||
messages = _tool_result_messages()
|
||||
result = router.apply(messages, _Tokenizer(), frozen_message_count=1, min_tokens_to_compress=1)
|
||||
|
||||
assert charged, "the remote compressor must still be reached and billed"
|
||||
assert sum(charged) < 0.04, f"remote ML ran {sum(charged):.3f}s under a 40 ms outer timeout"
|
||||
assert requests == [], "a call that cannot fit the budget must not be sent"
|
||||
_assert_prose_blocks_unchanged(result.messages)
|
||||
|
||||
|
||||
def test_cold_remote_call_is_bounded_as_a_whole_by_the_ml_budget(charged, monkeypatch) -> None:
|
||||
"""With enough budget to try, the first (unmeasured) remote call is capped
|
||||
by the remainder -- over the whole request, not per httpx phase -- and an
|
||||
endpoint slower than that leaves the block unchanged."""
|
||||
budget = 0.2
|
||||
monkeypatch.setenv("HEADROOM_ML_STAGE_DEADLINE_SECONDS", str(budget))
|
||||
monkeypatch.setenv("HEADROOM_COMPRESSION_TIMEOUT_SECONDS", "30")
|
||||
|
||||
release = threading.Event()
|
||||
timeouts: list[dict] = []
|
||||
|
||||
def _hung_endpoint(request: httpx.Request) -> httpx.Response:
|
||||
timeouts.append(dict(request.extensions["timeout"]))
|
||||
release.wait(5.0) # far past the budget; MockTransport ignores httpx timeouts
|
||||
return _compressed_reply(request)
|
||||
|
||||
remote = _remote_kompress(_hung_endpoint)
|
||||
monkeypatch.setattr(cr.ContentRouter, "_get_kompress", lambda self: remote)
|
||||
router = cr.ContentRouter(cr.ContentRouterConfig(enable_kompress=True))
|
||||
|
||||
messages = _tool_result_messages()
|
||||
try:
|
||||
result = router.apply(
|
||||
messages, _Tokenizer(), frozen_message_count=1, min_tokens_to_compress=1
|
||||
)
|
||||
finally:
|
||||
release.set()
|
||||
remote.close()
|
||||
|
||||
assert timeouts, "the first block must reach the endpoint"
|
||||
assert all(0.0 < t <= budget for t in timeouts[0].values()), (
|
||||
f"per-phase timeouts were not lowered to the budget: {timeouts[0]}"
|
||||
)
|
||||
assert sum(charged) <= budget + 0.1, (
|
||||
f"remote ML ran {sum(charged):.3f}s against a {budget}s budget"
|
||||
)
|
||||
_assert_prose_blocks_unchanged(result.messages)
|
||||
|
||||
|
||||
def test_remote_kompress_keeps_its_default_timeout_without_a_budget() -> None:
|
||||
"""Direct callers pass no cap: same single request, 20s per-phase timeouts."""
|
||||
timeouts: list[dict] = []
|
||||
|
||||
def _endpoint(request: httpx.Request) -> httpx.Response:
|
||||
timeouts.append(dict(request.extensions["timeout"]))
|
||||
return _compressed_reply(request)
|
||||
|
||||
remote = _remote_kompress(_endpoint)
|
||||
text = _prose(2_000)
|
||||
out = remote.compress(text)
|
||||
|
||||
assert out.compressed != text
|
||||
assert timeouts == [{"connect": 20.0, "read": 20.0, "write": 20.0, "pool": 20.0}]
|
||||
assert remote._capped_pool is None, "the default path must not hop threads"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Fourth review: the capped worker pool had no admission bound. Its queue is
|
||||
# unbounded and ``future.cancel()`` does not drop a queued payload, so once
|
||||
# every worker was held by an abandoned request, each further capped call
|
||||
# queued its full prompt, timed out, and left it there. A slow-drip body could
|
||||
# hold a worker forever, since httpx read timeouts reset on every chunk.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def _free_permits(remote: RemoteKompressCompressor) -> int:
|
||||
return remote._capped_slots._value # type: ignore[attr-defined]
|
||||
|
||||
|
||||
def _wait_until(predicate, timeout_s: float = 3.0) -> bool:
|
||||
ends = time.monotonic() + timeout_s
|
||||
while time.monotonic() < ends:
|
||||
if predicate():
|
||||
return True
|
||||
time.sleep(0.01)
|
||||
return predicate()
|
||||
|
||||
|
||||
def test_saturated_remote_calls_fail_open_without_queueing_their_payloads() -> None:
|
||||
"""The reviewer's repro: hold all 16 workers with requests whose 200 ms
|
||||
callers have already timed out, then make five more calls with 60 ms caps.
|
||||
Each must fail open within its cap with its payload never enqueued, and
|
||||
capacity must come back once the held requests end."""
|
||||
from headroom.transforms.kompress_remote import _CAPPED_CALL_WORKERS
|
||||
|
||||
release = threading.Event()
|
||||
reached: list[str] = []
|
||||
|
||||
def _stalled_endpoint(request: httpx.Request) -> httpx.Response:
|
||||
reached.append(json.loads(request.content)["content"])
|
||||
release.wait(5.0)
|
||||
return _compressed_reply(request)
|
||||
|
||||
remote = _remote_kompress(_stalled_endpoint)
|
||||
text = _prose(2_000)
|
||||
try:
|
||||
with concurrent.futures.ThreadPoolExecutor(_CAPPED_CALL_WORKERS) as callers:
|
||||
held = list(
|
||||
callers.map(
|
||||
lambda i: remote.compress(f"held{i} {text}", _time_budget_cap_seconds=0.2),
|
||||
range(_CAPPED_CALL_WORKERS),
|
||||
)
|
||||
)
|
||||
assert all(out.compressed == f"held{i} {text}" for i, out in enumerate(held))
|
||||
assert len(reached) == _CAPPED_CALL_WORKERS
|
||||
assert _free_permits(remote) == 0, "abandoned requests must keep their permits"
|
||||
|
||||
for i in range(5):
|
||||
late = f"late{i} {text}"
|
||||
started = time.monotonic()
|
||||
out = remote.compress(late, _time_budget_cap_seconds=0.06)
|
||||
took = time.monotonic() - started
|
||||
assert out.compressed == late, "a saturated call must fail open unchanged"
|
||||
assert took < 0.06 + 0.1, f"saturated call took {took:.3f}s against a 60 ms cap"
|
||||
assert remote._capped_pool._work_queue.qsize() == 0, "a payload was enqueued"
|
||||
|
||||
assert len(reached) == _CAPPED_CALL_WORKERS, "a saturated call reached the endpoint"
|
||||
finally:
|
||||
release.set()
|
||||
|
||||
assert _wait_until(lambda: _free_permits(remote) == _CAPPED_CALL_WORKERS)
|
||||
recovered = remote.compress(text, _time_budget_cap_seconds=2.0)
|
||||
assert recovered.compressed != text, "capacity must return once the held requests end"
|
||||
remote.close()
|
||||
|
||||
|
||||
def test_slow_drip_response_is_cut_off_at_an_absolute_deadline() -> None:
|
||||
"""A body that keeps dripping (a chunk every 50 ms for 2 s) resets httpx's
|
||||
read timeout on every chunk. The worker must stop at the cap's absolute
|
||||
deadline and give its permit back, not ride the response to the end."""
|
||||
from headroom.transforms.kompress_remote import _CAPPED_CALL_WORKERS
|
||||
|
||||
chunks_sent: list[int] = []
|
||||
|
||||
def _dripping_endpoint(request: httpx.Request) -> httpx.Response:
|
||||
reply = json.dumps({"compressed": "short"}).encode()
|
||||
|
||||
def _drip():
|
||||
for i in range(40):
|
||||
chunks_sent.append(i)
|
||||
time.sleep(0.05)
|
||||
yield b" " # JSON-legal leading whitespace
|
||||
yield reply
|
||||
|
||||
return httpx.Response(200, content=_drip())
|
||||
|
||||
remote = _remote_kompress(_dripping_endpoint)
|
||||
text = _prose(2_000)
|
||||
cap = 0.2
|
||||
started = time.monotonic()
|
||||
out = remote.compress(text, _time_budget_cap_seconds=cap)
|
||||
assert out.compressed == text
|
||||
assert time.monotonic() - started < cap + 0.1
|
||||
|
||||
assert _wait_until(lambda: _free_permits(remote) == _CAPPED_CALL_WORKERS, timeout_s=1.0), (
|
||||
"the dripping request still holds its permit"
|
||||
)
|
||||
assert time.monotonic() - started < 1.0, "the worker rode the drip instead of the deadline"
|
||||
assert len(chunks_sent) < 10, f"read {len(chunks_sent)} chunks past a {cap}s deadline"
|
||||
remote.close()
|
||||
|
||||
|
||||
def test_permit_is_held_until_the_worker_finishes_not_the_caller() -> None:
|
||||
"""Releasing on caller timeout would let abandoned requests pile up again;
|
||||
the permit belongs to the running request."""
|
||||
from headroom.transforms.kompress_remote import _CAPPED_CALL_WORKERS
|
||||
|
||||
release = threading.Event()
|
||||
|
||||
def _stalled_endpoint(request: httpx.Request) -> httpx.Response:
|
||||
release.wait(5.0)
|
||||
return _compressed_reply(request)
|
||||
|
||||
remote = _remote_kompress(_stalled_endpoint)
|
||||
text = _prose(2_000)
|
||||
try:
|
||||
out = remote.compress(text, _time_budget_cap_seconds=0.1)
|
||||
assert out.compressed == text
|
||||
assert _free_permits(remote) == _CAPPED_CALL_WORKERS - 1
|
||||
finally:
|
||||
release.set()
|
||||
assert _wait_until(lambda: _free_permits(remote) == _CAPPED_CALL_WORKERS)
|
||||
remote.close()
|
||||
@@ -218,3 +218,66 @@ def test_a_compressor_that_cannot_take_the_deadline_is_not_handed_one(monkeypatc
|
||||
|
||||
assert fake.deadlines, "the stub should still have been called"
|
||||
assert all(d is None for d in fake.deadlines), fake.deadlines
|
||||
|
||||
|
||||
def _chunked_kompress(monkeypatch, chunk_s: float):
|
||||
"""Kompress over a fake model; each chunk advances a fake monotonic clock."""
|
||||
clock = {"now": 0.0, "chunks": 0}
|
||||
|
||||
class _Enc(dict):
|
||||
def word_ids(self, batch_index=0):
|
||||
return self["_word_ids"]
|
||||
|
||||
class _Tok:
|
||||
def __call__(self, chunk_words, **kw):
|
||||
n = len(chunk_words)
|
||||
return _Enc(input_ids=[[0] * n], attention_mask=[[1] * n], _word_ids=list(range(n)))
|
||||
|
||||
class _Model:
|
||||
def get_keep_mask(self, input_ids, attention_mask):
|
||||
clock["now"] += chunk_s
|
||||
clock["chunks"] += 1
|
||||
n = len(input_ids[0])
|
||||
return [[i < n // 2 for i in range(n)]]
|
||||
|
||||
def get_scores(self, input_ids, attention_mask): # the batched path
|
||||
return [
|
||||
[0.9 if keep else 0.1 for keep in row]
|
||||
for row in self.get_keep_mask(input_ids, attention_mask)
|
||||
]
|
||||
|
||||
monkeypatch.setattr(kc.time, "monotonic", lambda: clock["now"])
|
||||
monkeypatch.setattr(kc, "_load_kompress", lambda *a, **k: (_Model(), _Tok(), "onnx"))
|
||||
monkeypatch.setattr(kc, "_model_device_type", lambda *a, **k: "cpu")
|
||||
monkeypatch.setenv("HEADROOM_COMPRESSION_DEADLINE_MS", "0")
|
||||
monkeypatch.delenv(kc.KOMPRESS_TIME_BUDGET_ENV, raising=False)
|
||||
|
||||
comp = kc.KompressCompressor(kc.KompressConfig(min_input_words=10))
|
||||
comp.config.chunk_words = 100
|
||||
monkeypatch.setattr(comp, "_should_batch_single_content", lambda *a, **k: False)
|
||||
monkeypatch.setattr(comp, "_should_use_sequential_fallback", lambda: False)
|
||||
return comp, clock
|
||||
|
||||
|
||||
def test_time_budget_cap_stops_a_call_and_returns_it_unchanged(monkeypatch):
|
||||
"""The router hands Kompress what is left of its ML budget, so the first
|
||||
call of a request -- which has no measured cost to be refused on -- stops
|
||||
at a chunk boundary instead of running to the 20s per-call budget. The
|
||||
block comes back byte-identical, so it is stable on later turns."""
|
||||
comp, clock = _chunked_kompress(monkeypatch, chunk_s=0.6)
|
||||
content = " ".join(f"w{i}" for i in range(500)) # five chunks, 3.0s uncapped
|
||||
|
||||
result = comp.compress(content, _time_budget_cap_seconds=1.0)
|
||||
|
||||
assert clock["chunks"] == 2, "must stop at the first chunk boundary past the cap"
|
||||
assert result.compressed == content
|
||||
|
||||
|
||||
def test_time_budget_cap_bounds_the_batched_path_too(monkeypatch):
|
||||
comp, clock = _chunked_kompress(monkeypatch, chunk_s=0.6)
|
||||
content = " ".join(f"w{i}" for i in range(500))
|
||||
|
||||
[result] = comp.compress_batch([content], batch_size=1, _time_budget_cap_seconds=1.0)
|
||||
|
||||
assert clock["chunks"] == 2
|
||||
assert result.compressed == content
|
||||
|
||||
Reference in New Issue
Block a user