Commit Graph
7 Commits
Author SHA1 Message Date
e2350bc8fd fix(proxy): warm deferred kompress model so /health stops reporting backend=null (#2799)
## Description

On an idle proxy with the ML extras installed and the Kompress model
cached, `/health` and `/readyz` reported `kompress: {"ready": false,
"status": "unhealthy", "backend": null}` forever (#2730).

The cause was not missing wiring: `enable_kompress` defaults to `True`
and `ContentRouter` sits in both pipelines. The chain was:

1. `ContentRouter.eager_load_compressors()` deliberately skips
`preload()`, so a native load never runs before the port binds (#790
hang on Windows, #1908 segfault on RHEL/CentOS 7). It returns `kompress
= "deferred"`.
2. `WarmupRegistry.merge_transform_status` leaves a deferred slot at
`status="null"`.
3. `_reconcile_kompress_health()` promotes the slot only once a
compressor reports `is_ready()`, and nothing loaded the model without a
kompress-routed request.

#3619 (merged after this PR was opened) added
`_start_kompress_background_warmup()`, which loads the model on a daemon
thread shortly after startup. That fixes the load itself. Rebased onto
it, this PR covers what is left:

1. **`/health` says why `ready` is false.** The kompress check gains a
`detail` field. `ready: false` alone cannot tell "still warming" from
"extras not installed" from "warm failed", which is the confusion in
#2730.
2. **The warm thread records its outcome on `warmup.kompress`.** Success
promotes the slot immediately with `source_status="background"`, instead
of leaving `source_status="deferred"` on a loaded slot until the next
reconcile. Failure leaves a bounded reason.
3. **The detail is a closed vocabulary.** `/health` and `/readyz` are
auth-exempt, and loader exceptions carry paths, model ids and URLs, so
anything outside the allowlist collapses to `unavailable`.

Closes #2730

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/kompress_health_detail_policy.py` (new, pure, matching
the existing `*_policy.py` modules): `KOMPRESS_HEALTH_DETAILS` lists the
values the field may report (`loaded`, `warming`, `model not cached`,
`warm failed`, `not installed`, plus the `eager_load_compressors()`
statuses `enabled` / `deferred` / `unavailable`). `public_detail()`
passes those through and maps anything else to `unavailable`.
- `headroom/proxy/server.py`:
- `_start_kompress_background_warmup()` (from #3619, still
`preload(allow_download=True)` on thread `kompress-warmup`) now writes
its result to the slot. On success it calls `mark_loaded(...,
source_status="background")`. On any exception it sets detail `warm
failed` and logs the real cause with `exc_info=True`; the exception text
never reaches the slot. A compressor that raises
`KompressModelNotCached` gets `model not cached`. The local loader only
raises that with downloads off, so on a cold cache it downloads and
`/health` reports `warming` in the meantime.
- `_kompress_detail()` feeds the new field. Every candidate, including
raw `slot.error` and `source_status`, goes through `public_detail()`, so
the allowlist has a single choke point.
- Router/compressor lookup moves to module-level `_kompress_routers()` /
`_kompress_compressors()`, shared by startup and `/health`.
- `enabled`, `ready` and `backend` keep their meaning, so the `/readyz`
aggregate exclusion of optional Kompress still holds.
- Tests:
- `tests/test_kompress_preload_deferral.py`: the warm thread promotes a
cached model with no request (`source_status == "background"`), a
compressor-reported cache miss stays deferred with `model not cached`,
and a failure carrying a path and a token leaves exactly `warm failed`.
- `tests/test_proxy_health.py`: sentinel tests over `/health` and
`/readyz` for both leak paths (`slot.info["detail"]` and `slot.error`).
- `tests/test_kompress_health_detail_policy.py`: covers the policy
directly.
- `tests/test_proxy/test_kompress_background_warmup.py` and
`tests/test_proxy_eager_preload_bind.py`: expectations updated for the
slot promotion.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Type checking: `mypy headroom --ignore-missing-imports` (mypy 1.20.2)
reports one error, in `headroom/fileperms.py` (`os.fchmod` does not
exist on Windows). That file is not touched by this PR, and there are no
errors in `server.py` or the new module.

### Test Output

```text
$ pytest tests/test_kompress_preload_deferral.py tests/test_proxy_health.py tests/test_proxy/test_kompress_background_warmup.py tests/test_kompress_health_detail_policy.py tests/test_proxy_eager_preload_bind.py tests/test_proxy_warmup.py -q
73 passed in 22.36s

$ ruff check headroom tests
All checks passed!

$ ruff format --check headroom tests
1519 files already formatted

$ mypy headroom --ignore-missing-imports --no-site-packages
headroom\fileperms.py:71: error: Module has no attribute "fchmod"; maybe "chmod"?  [attr-defined]
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.11, headroom installed from this
branch, Kompress model `chopratejas/kompress-v2-base` in the local HF
cache, `HF_HUB_OFFLINE=1`, branch rebased on `upstream/main` 540f4da1
- Exact command / steps: built the real app with
`create_app(ProxyConfig(optimize=True, cache_enabled=False,
rate_limit_enabled=False, cost_tracking_enabled=False))`, started it
through the FastAPI lifespan with
`HEADROOM_KOMPRESS_WARMUP_DELAY_SECONDS=0`, sent no compression request,
and polled `GET /health` every 250 ms until Kompress reported ready,
then read `/readyz` and the warmup slot
- Observed result: at t=0.25 s `/health` returned `{"ready": false,
"status": "degraded", "backend": null, "detail": "warming"}`. The log
showed `Kompress: warmed in the background in 7618 ms (backend onnx)`,
and at t=6.44 s `/health` returned `{"ready": true, "status": "healthy",
"backend": "onnx", "detail": "loaded"}`. `/readyz` matched, and the slot
was `{"status": "loaded", "info": {"source_status": "background",
"backend": "onnx"}}`. No request was needed.
- Not tested: macOS arm64 (the reporter's platform), a real cold-cache
download through the warm thread, and glibc < 2.28 hosts, where #3619
skips the warm-up and `/health` keeps reporting `deferred`.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. Health reporting for the existing
background warm-up from #3619.
- Minimum rollout channel: stable
- Stable/default behavior changed: Yes, additively. The kompress health
check gains a `detail` field, and a successful background warm marks the
slot `loaded` straight away instead of at the next `/health` call.
`enabled`, `ready`, `status` and `backend` are unchanged.
- Kill switch / disable path: `HEADROOM_KOMPRESS_WARMUP=0` disables the
background warm-up (from #3619). The `detail` field has no separate
switch; it only reads state.
- Unsafe override required: No
- Qualification impact: None expected. Health payloads gain one field,
and every value comes from a fixed allowlist.
- Rollback path: Revert the PR. There is no stored state or migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`. It is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A. No user-facing UI change.

## Additional Notes

- No documentation change: the `detail` field is diagnostic.
`HEADROOM_KOMPRESS_WARMUP` (from #3619) is not in the env-var reference
yet either; happy to add it here if maintainers prefer.
- An earlier revision of this PR added its own
`HEADROOM_KOMPRESS_BACKGROUND_WARM` switch. It was removed after the
rebase because it did the same job as `HEADROOM_KOMPRESS_WARMUP` and was
never released.
- Follow-up, out of scope: the #3619 warm thread is not told about
`shutdown()`. It is a daemon thread and production shutdown ends the
process, but an in-process restart could still run one load after stop.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 22:30:37 -06:00
lorenzozaneandJD Davis 430ea4f4eb fix(proxy): report disabled Kompress at startup (#4217)
## Description

When Kompress is intentionally disabled, startup currently falls through
to the missing-dependency message, even though `/health` reports it as
disabled. The startup logger now checks the resolved provider pipelines
and reports `Kompress: DISABLED` when neither enables Kompress, while
retaining install guidance when dependencies are missing.

Fixes #4208

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/server.py`: derive whether Kompress is enabled from
the constructed Anthropic and OpenAI pipelines and log the disabled
state before the missing-dependency fallback.
- `tests/test_proxy_eager_preload_bind.py`: cover global and provider
disablement, provider re-enablement, omitted compressor selection, and
missing-dependency guidance.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
Focused proxy tests: 29 passed, 1 existing StarletteDeprecationWarning
Repository-wide pytest: not completed; public-network tests stalled and the external tool installer did not finish
Ruff check/format and git diff --check: passed
mypy: unavailable in the job environment
```

## Real Behavior Proof

- Environment: Linux, Python 3.14.7, local source checkout with `uv sync
--extra proxy`; no ML extra, model download, or provider request. Beacon
and telemetry were disabled.
- Exact command / steps: with `HEADROOM_DISABLE_KOMPRESS=1`, ran
`.venv/bin/headroom proxy --host 127.0.0.1 --port 18791 --mode cache
--no-http2`, then polled `/health` and checked `.checks.kompress`.
- Observed result: HTTP 200; health reported `enabled=false`,
`ready=true`, `status=disabled`; startup logged `Kompress: DISABLED`.
- Not tested: provider requests, model downloads, macOS/Python 3.13,
full pytest, and upstream CI. The command and compact output are
recorded in
`artifacts/20261015T041001Z_asset-01-1/live-behavior-command.md` and
adjacent live-behavior artifacts.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: startup reports disabled Kompress
accurately instead of suggesting installation when it is disabled.
- Kill switch / disable path: existing Kompress disable settings remain
available.
- Unsafe override required: no.
- Qualification impact: none known.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

The focused suite emitted one Starlette deprecation warning from
`starlette.testclient`; the full suite and upstream CI have not been
verified. The two failure markers from the interrupted broad pytest run
did not reproduce when their test files were run alone. No documentation
or changelog changes were needed for this startup-log correction.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4217"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-15 16:27:19 -05:00
f78e66f7cc fix(proxy): stop headroom logger from suppressing propagation to stdout/stderr (#3096)
## Description

`_setup_file_logging()` in `headroom/proxy/helpers.py` unconditionally
set `headroom_logger.propagate = False`, on the theory that `wrap.py`
redirects stderr into the same `proxy.log` file the rotating handler
writes to, so propagation would double-write records. That premise is
wrong: `wrap.py` redirects the proxy subprocess's stdout/stderr to a
separate sibling file, `proxy-stdio.log` (see
`_get_proxy_stdio_log_path()`), never `proxy.log`. Severing propagation
bought no deduplication — it only stopped every `headroom.*` log record
from reaching root's `logging.basicConfig()` handler
(`proxy/server.py`), which is what writes to stderr/stdout. In
containerized deployments (Docker/Kubernetes), stdout/stderr is the only
channel the platform collects, so this silently dropped all application
logs.

Closes #3087

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Removed `headroom_logger.propagate = False` from
`_setup_file_logging()` in `headroom/proxy/helpers.py`.
- Updated the surrounding comment to explain why propagation is safe to
leave on: `proxy.log` (file handler) and stderr/stdout (root's
`basicConfig` handler) are two distinct destinations, so nothing is
duplicated in either file.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check .
All checks passed!

$ pytest tests/test_corrupt_golden_bytes_recovery.py -q
collected 9 items
tests/test_corrupt_golden_bytes_recovery.py .........                    [100%]
9 passed in 0.47s
```

`mypy headroom` was not run to a clean result: it fails before reaching
this file with a pre-existing, unrelated error in the installed `numpy`
stub (`Type statement is only supported in Python 3.12 and greater`) in
this sandbox's dependency set. Not caused by this change (a 4-line
pure-Python logging tweak).

The broader `pytest tests/` collection also hits ~170 pre-existing,
unrelated errors in this sandbox: `headroom._core`, the compiled Rust
extension, isn't built here (no `cargo`/`maturin` available), so every
module importing it fails at collection. This change doesn't touch Rust
code or `_core` and isn't implicated.

No new tests added: this restores an existing, already-tested code path
(see `tests/conftest.py`'s autouse fixture and
`tests/test_corrupt_golden_bytes_recovery.py`, both of which already
assert/require `propagate=True` for `caplog` to work) to its intended
default rather than adding new behavior.

## Real Behavior Proof

- Environment: local sandbox, Python 3.12.3, unable to build/import
`headroom.proxy.server` directly (missing compiled `headroom._core`).
Verified with an isolated script that copies `_setup_file_logging()`'s
exact logic byte-for-byte plus the same `logging.basicConfig()` call
`proxy/server.py` makes at import time.
- Exact command / steps: ran the fixed logic, then emitted
`logging.getLogger("headroom.proxy").info("compression timeout: request
abc123 exceeded 30s budget")`.
- Observed result: the `headroom.proxy` record was written to stderr AND
to `proxy.log` simultaneously (full output below); re-running the same
script with the old `headroom_logger.propagate = False` line restored
reproduces the bug — the record disappears from stderr entirely and only
lands in `proxy.log`.
  ```text
  propagate flag on 'headroom' logger: True
--- emitting a headroom.proxy log record, watch for it below on stderr
---
2026-08-17 21:17:39,545 - headroom.proxy - INFO - compression timeout:
request abc123 exceeded 30s budget
  --- also present in proxy.log (file handler): OK ---
  ```
- Not tested: end-to-end through the actual compiled proxy binary/server
process (blocked by missing `headroom._core` in this sandbox) and real
container log collection (Docker/Kubernetes) — the isolated repro
exercises the exact same logging configuration code the real process
runs, so this is considered sufficient proxy proof for a pure
logging-config change.

## Runtime Rollout Safety

- Rollout-managed feature(s): None — this is unconditional startup
logging configuration, not behind a rollout flag.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes — `headroom.*` log records now
also reach stderr/stdout (previously file-only). No change to
request/response handling, compression, or any user-facing proxy
behavior.
- Kill switch / disable path: None needed; this restores default Python
logging propagation. If a deployment truly wants file-only logging,
`HEADROOM_LOG_TO_STDOUT`-style suppression could be added later, but no
such request exists today.
- Unsafe override required: No
- Qualification impact: None
- Rollback path: Revert this commit to restore `propagate = False`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — logging configuration change, no UI impact.

## Additional Notes

- "New and existing unit tests pass locally" is left unchecked because
the full `pytest tests/` collection can't complete in this sandbox
(missing compiled `headroom._core` Rust extension, unrelated to this
change) — see Test Output above for what was actually run and passed.
- "I have added tests" is left unchecked: existing tests
(`tests/conftest.py` autouse fixture,
`tests/test_corrupt_golden_bytes_recovery.py`) already assert the code
path this restores; happy to add a focused regression test asserting
`logging.getLogger("headroom").propagate is True` after
`_setup_file_logging()` if maintainers want one.

---------

Co-authored-by: = <=>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 00:03:53 -05:00
Tejas Chopra 53af90d68c perf: cut hot-path latency 27% (token-count memo, startup preloads, JSON scan memo) (#2838)
## Description

Four independent latency fixes on the request hot path, found by
profiling and each measured in isolation. No behaviour changes: every
commit is either a memo of a pure function, work moved to startup, or
work that was computed and discarded.

**End to end: 287ms → 210ms (−27%) on a 68k-token mixed payload, with
byte-identical output** (68,514 → 48,725 tokens both before and after).

Plus one-off costs removed that don't show in steady-state numbers:
~4.9s of lazy imports that were firing *inside* user requests, and
~750ms of HuggingFace round-trips per process start.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

**1. Memoise `count_text` (`ac369277`)** — tiktoken's `CoreBPE.encode`
was 0.243s of a 0.30s profiled request. It dominates because the same
string is counted repeatedly: a 103KB payload drove 600KB of encoding,
~6x the content, across six call sites (`tokenizers/base.py:196`,
`content_router.py:4704` and `:5474`, `parser.py:185/192/298`). 35% of
encode calls and 22% of encoded characters were an exact repeat *within
one request*.

`count_text` is a pure function of its text, so replaying a stored count
returns the same integer. That is the whole safety argument, and it is
what makes this safe at the sites whose count feeds a routing decision
(`context_pressure` → `min_ratio`) rather than a log line — an
*estimate* there would change which blocks compress; a memo cannot.

Keyed on the text itself, not a hash: a collision would hand back a
wrong count for real content and silently change compression. The cost
is holding the strings, so entries and total characters are both capped.
Clear-on-full rather than LRU eviction — the pipeline runs on a thread
pool, `dict` get/set/clear are atomic under the GIL but
`OrderedDict.move_to_end` is not.

**2. Preload what was importing mid-request (`2921a15b`)** — `litellm`
(2.9–3.8s) was imported lazily *on the event loop* during the first
request: `emit_request_outcome` → `record_request` →
`_estimate_compression_savings_usd` calls the loader before its own
`tokens_saved <= 0` early return, so even a request that saved nothing
paid it. `trafilatura` (978ms, pulling `htmldate` → `dateparser` and its
timezone tables) is the most expensive lazy import in the transform tree
— every other compressor module is 1–20ms — and fires on the first
request carrying an HTML-ish or mixed-content block. The TOIN singleton
reads ~5MB of learned patterns on construction (~150ms); a stale comment
claimed the SmartCrusher preload covered it, and it does not.

All three now load in `_eager_preload_transforms`, which already runs
under `asyncio.to_thread` and so cannot delay the port bind.

Same commit, two Kompress cold-path fixes: `_load_modernbert_tokenizer`
always used `local_files_only=False`, which makes transformers
re-validate against the Hub on every load — a tree listing plus a HEAD
per file — even when fully cached (~900ms warm-cache vs ~150ms
local-only). And `ensure_background_download` re-spawned a
finished-or-failed thread on the next call, so an unreachable Hub meant
one fresh download thread *per request* for the life of the process,
each importing transformers and holding the GIL against the event loop.
Consecutive failures now back off; success clears it, so the happy path
and the transient-failure path are unchanged.

**3. Memoise the JSON-block scan (`039c9735`)** —
`_has_valid_json_block_with_text` tries every `{`/`[`-leading line as a
possible block start. When a candidate never balances,
`_extract_json_block` scans character-by-character to the end of the
content and returns nothing — then the next candidate does it again.
Quadratic, on the request path, growing exactly 4x per doubling.

**4. `CostTracker.totals()` (`286b97e4`)** —
`_current_savings_tracker_totals` called `stats()` once per request and
read two of its fields. Building the rest includes
`period_cost_breakdown()`, which walks up to 100k cost records over 31
days, on the event loop, holding the metrics lock. It degrades with
proxy **uptime**, not load, which is why no short benchmark would
surface it.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/ tests/ --exclude headroom/dashboard/templates
All checks passed!

$ mypy --python-version 3.12 headroom/
Found 1 error in 1 file (checked 515 source files)
  headroom/release_version.py:235: error: Name "tomllib" already defined (by an import)
  # pre-existing on main, in a file this PR does not touch — verified by
  # running the same command on a clean main checkout.

$ python -m pytest tests/test_token_count_cache.py tests/test_mixed_content_scan_cache.py \
    tests/test_kompress_download_backoff.py tests/test_cost_tracker_totals.py -q
306 passed

$ python -m pytest tests/ -q -k "token or tokenizer or count or estimator or provider"
1303 passed, 105 skipped in 423.42s

$ python -m pytest tests/ -q -k "cost or budget or metrics or savings or stats"
683 passed, 127 skipped, 1 failed
  # tests/test_proxy_memory_integration.py::TestMemoryStats::test_health_endpoint_works_with_memory
  # Order-dependent and pre-existing: it SKIPS in isolation, and fails identically
  # on a clean main checkout under the same -k selection (681 passed, 1 failed).
```

## Real Behavior Proof

- **Environment:** macOS, Python 3.12.6, local CPU, remote Kompress
disabled. Profiled with `cProfile` on `anthropic_pipeline.apply`.
- **Exact command / steps:** a 68k-token payload of four `tool_result`
blocks (900-item pretty JSON, 60KB of Python source, 500 lines of
JS-style object logs, 500 plain log lines), six reps, **content unique
per rep so every run is router-cache-cold**, run on this branch and on
main in alternation.
- **Observed result:**

  | | median | min | tokens |
  |---|---|---|---|
  | main | 287ms | 286ms | 68,514 → 48,725 |
  | this branch | 210ms | 208ms | 68,514 → 48,725 |

  Per-change, measured in isolation:

  | change | before | after |
  |---|---|---|
| `count_text` memo | — | −25% pipeline wall; 44% of counted chars from
cache on new content, 100% when history repeats |
| litellm / trafilatura / TOIN | 3829 / 978 / 150ms mid-request | at
startup, off the event loop |
  | Kompress tokenizer | ~900ms | ~150ms |
  | JS-style object logs (1200 lines) | 4643ms | 183ms |
  | truncated JSONL (1200 lines) | 3737ms | 116ms |
| `cost_tracker` per request | 2.8ms @20k records, 13.6ms @100k | loop
over models, not records |

Output equality: 18/18 payloads byte-identical on `tokens_before`,
`tokens_after` and a sha256 of the resulting messages, with the memo
forced on vs off.

- **Not tested:** Windows and Linux (the ORT dylib and CPU-arena paths
differ); multi-worker deployments; a proxy with a genuinely large live
cost ledger (the 100k figure is from a synthetic ledger); real
HTML-heavy traffic through the preloaded trafilatura path.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

**Docs:** N/A — no user-facing surface changes. The reasoning lives in
the code, at the sites where someone debugging would look.

**A regression I introduced and caught.** The scan memo initially made
pretty-printed JSON ~2x **slower**: content that balances on the first
scan has nothing to reuse and just pays the per-line dict traffic. The
cache is now built only *after* a scan has run to the end without
balancing, which is the actual signal that later candidates will re-walk
the same tail. Every shape now improves and none regress:

```
                      before     after
js object logs       4642.9ms    182.7ms   25x
JSONL truncated      3736.8ms    115.9ms   32x
pretty JSON             5.6ms      3.5ms
JSONL valid             5.6ms      3.2ms
plain logs              1.0ms      0.6ms
python source           1.0ms      0.5ms
markdown prose          0.9ms      0.5ms
```

Worth stating plainly: had I only benchmarked the shape I was fixing,
I'd have shipped a win on rare content and a loss on the common case.

**The scan fix is constant-factor, not asymptotic.** The walk over
remaining lines is still O(candidates × lines), so 3200 lines of the
pathological shape is still ~1.4s. The tests assert scan-call counts
rather than implying linearity. True linearity needs a prefix-sum
rewrite with a string-state fallback; that seemed like the wrong risk
for this PR.

**How the parser change is proven safe.** `_extract_json_block` is a
parser, so golden values would only encode whatever the new code does.
Instead the pre-memo implementation is kept verbatim in the test file as
an oracle, and every candidate index of a 139-document corpus — escapes,
unterminated strings, delimiters inside strings, code fences, truncated
JSON, randomised mixtures — is asserted equal, with a cold cache, with
the shared cache the real callers use, and replayed.

**Measurement trap, for anyone re-running these numbers.** Give each arm
its own content. Reusing one payload across arms lets the second arm hit
the router's result cache, which reads as a speedup having nothing to do
with the change under test. I hit this twice while working on it: it
manufactured a fake "INFO logging costs 21.8%" finding (real answer:
0.3%) and it *understated* the memo win.

**Deliberately not in this PR:**
- **ONNX thread tuning** — measured zero gain, and
`intra_op_num_threads` is not bitwise-safe (1.6e-05 score drift from
float reduction order), so it would trade an output risk for nothing.
- **`str(content)` on block lists** counts a base64 image at 210,775
tokens instead of 1,604 (131x), pinning `context_pressure` to 1.0 and
forcing the most aggressive `min_ratio` on any conversation containing
an image. Real bug, but fixing it changes compression output — needs its
own reviewed behaviour-change PR.
- **`chunk_words=350` against the tokenizer's 512-token limit** silently
drops roughly a third of every full chunk (measured: 240/240 words kept
in the first 240, 15/110 in the tail). That is data loss rather than
latency, it changes every output, and correcting it costs ~1.3x latency.
Filing separately.
- **Telemetry off the request thread** — the TOIN auto-save is a 236ms
inline stall every 600s and the waste-signal re-parse is ~50ms/request
that is invisible in `pipeline_total` (computed before it). Both want
deferral rather than removal, which is a larger change than belongs
here.
2026-08-06 17:47:40 -07:00
Tejas Chopra f2c48e26c6 feat(compress): reach the lossless provider seam on the general path and default /v1/compress to marker-free output (#2691)
## Description

Two related changes to the compression seams, plus the review fixes for
both. Supersedes #2661 and #2662, which are closed in favour of this
branch — the fixes are inseparable from the code they fix, so reviewing
them together is cheaper than landing two PRs and patching them
afterwards.

**1. A registered lossless provider now competes on the general path.**
The `headroom.transforms.lossless_provider` seam was only ever consulted
from `_lossless_compact_excluded`, gated on `DEFAULT_EXCLUDE_TOOLS`
(`config.py:216` — `Read/Grep/Glob/Write/Edit/WebSearch/WebFetch`).
Gateway traffic carries the caller's own tool names — LiteLLM's
`headroom` guardrail (https://docs.litellm.ai/docs/proxy/headroom) posts
requests containing tools like `search_docs` / `run_ci` / `fetch_rows` —
so a registered provider was structurally unreachable for every
gateway/sidecar deployment. The seam existed; nothing could get to it.

**2. `POST /v1/compress` is marker-free by default.** A CCR marker is
only useful to a caller that also injects the `headroom_retrieve` tool
AND can reach `/v1/retrieve`. Neither holds here: tool injection lives
in the provider request handlers (`handlers/anthropic.py:1894`), never
in `handle_compress`; and every `/v1/retrieve*` route is
`Depends(_require_loopback)` (`server.py:4422, 4470, 4749, 4781`) with
no remote opt-in — `HEADROOM_COMPRESS_ALLOW_REMOTE` drops the loopback
dependency on `/v1/compress` only. So a gateway forwards a `Retrieve
more: hash=…` pointer the model cannot follow, and the proxy pays a CCR
store write nobody reads. `config.mode="ccr"` opts back in.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### Seam — `content_router.py`, `lossless_provider.py`

- `_lossless_first` (STAGE 0, every block on every path) consults
`get_lossless_provider()` and keeps whichever output is smaller.
**Strict no-op when no provider is registered**, which is the default;
and because it is best-of rather than authoritative, a provider can
never do worse than the built-in folds.
- Malformed provider output can no longer escape. Every shape check runs
inside the `try`: result must be `None`, or a 2-element tuple/list of
two `str`. Anything else is ignored at debug level. (Previously the
unpack sat outside the `try`, so a 3-tuple raised `ValueError` up
through `TransformPipeline.apply`, which re-raises.)
- Empty / whitespace-only candidates are rejected rather than silently
replacing block content.
- **Providers are never offered diff content.** Diff folding is
subtractive with no inverse check and a reflowed hunk breaks `git apply`
— the same reason the built-in `diff` fold is restricted at
`content_router.py:2481`.
- The third-party `kind` label is sanitised against `^[a-z0-9_]{1,32}$`
before reaching `transforms_applied` and the per-strategy metric dicts,
so a caller-controlled string cannot explode Prometheus label
cardinality. `fullmatch`, not `match`: `$` also matches before a
trailing newline, which would put a newline in a label.
- `set_lossless_provider(provider, *, verifier=None)` — in lossless-only
mode, where STAGE 0's output is final and there is no marker to recover
from, a registered verifier must confirm the fold or the candidate is
dropped. No verifier registered = today's behaviour. `provider=None`
clears both.
- The provider is invoked once per block, not twice
(`_has_lossless_fold` probes `_lossless_first` and discards the result,
then STAGE 0 recomputes). Bounded memo, wholesale clear on overflow, no
lock — a race costs one redundant fold. The memo keys on the provider
registration generation, so registering or clearing a provider after a
block was already folded takes effect.
- The seam docstring now records that the provider runs on the general
path and inside the parallel compression pool, so it must be thread-safe
as well as deterministic.

### Route — `handlers/openai.py`, `server.py`

- `_derived_compress_pipeline(key, **overrides)` replaces the
copy-pasted pipeline-derivation block; `_no_ccr_pipeline` (the new
default) and `_lossy_inline_pipeline` both use it.
- The default pipeline is **built at startup** and included in
`_eager_preload_transforms`, so a fresh pod does not pay ContentRouter
construction and compressor load on its first request, inside the
compression-executor budget.
- An unrecognised `config.mode` returns 400 naming the valid values
instead of silently falling back to the default.
- Claude-family model names resolve their context limit from the
Anthropic provider. Real divergence:
`bedrock/anthropic.claude-3-5-sonnet` is 200000 there and 128000 on the
OpenAI provider. The tokenizer still comes from the OpenAI pipeline's
provider — a separate, larger change, noted in a comment.
- Documents why the derived router deliberately does **not** share the
base router's compression cache: keys do not encode CCR-marker mode, so
sharing would leak marker-laden entries into the marker-free path.

### Behavior change

A `/v1/compress` caller that relied on default markers now gets none.
The only in-tree caller that can resolve them is the TypeScript SDK
(`sdk/typescript/src/client.ts:398 retrieve`, `:422 handleToolCall`); it
needs `config: {"mode": "ccr"}` to keep today's behaviour, and landing
that SDK default in the same release would leave only gateway callers —
for whom markers were never resolvable — seeing a difference.

No `HEADROOM_COMPRESS_DEFAULT_MODE` compat env deliberately: a flag
nobody sets becomes permanent debt, and the wire-level `mode` already
covers the one caller that needs it.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_lossless_first_dispatch.py tests/test_lossless_excluded_compaction.py \
    tests/test_lossless_mode.py tests/test_lossless_then_lossy.py tests/test_lossless_diff_fold_guard.py \
    tests/test_bash_search_lossless_fold.py tests/test_proxy_compress_endpoint.py \
    tests/test_ccr_row_drop_store_bridge.py tests/test_gateway_sidecar_ports.py tests/test_compress_api.py \
    tests/test_platform_stabilization_functional.py tests/test_proxy_eager_preload_bind.py \
    tests/test_proxy_warmup.py tests/test_router_registry_smartcrusher.py -q
189 passed in 32.04s

$ ruff check <all 9 changed files>
All checks passed!

$ ruff format --check <all 9 changed files>
9 files already formatted

$ mypy headroom/transforms/content_router.py headroom/transforms/lossless_provider.py \
      headroom/proxy/handlers/openai.py headroom/proxy/server.py
Success: no issues found in 4 source files
```

New tests cover, one concern each: every malformed provider shape; empty
and whitespace-only results; diff content never reaching a provider
(call-recording); `kind` sanitisation including the trailing-newline
case; the verifier accepting / rejecting / raising; clearing a provider
clearing its verifier; single provider invocation per block; memo
invalidation on re-registration; unknown and valid `mode` values; the
default pipeline existing before any request; and Claude vs OpenAI
context-limit resolution with `token_budget` precedence preserved.

## Real Behavior Proof

- **Environment:** macOS arm64, Python 3.12.6, proxy built from
`_proxy_config_from_env()` with the default `coding` savings profile;
Kompress both disabled and offloaded to a remote `kompress-v2-base`
endpoint; `HEADROOM_COMPRESSION_TIMEOUT_SECONDS=300`.
- **Steps:** `POST /v1/compress` over `TestClient` with OpenAI-shaped
payloads under non-excluded tool names (`run_ci`, `list_files`,
`code_search`, `fetch_rows`) — a CI log with ANSI escapes and repeated
lines, a 160-path listing, a 150-line grep dump, a 150-row JSON array;
plus a second payload with a RAG user blob, a 200-row JSON tool result
and a 300-line log. Ran with and without a provider registered via
`set_lossless_provider`.
- **Observed — seam reachability:**

  | Kompress | no provider registered | provider registered |
  |---|---|---|
  | off | 19,284 → 10,265 tokens (46.8%) | 19,284 → **7,879 (59.1%)** |
| on (remote) | 19,284 → 9,366 tokens (51.4%) | 19,284 → **7,146
(62.9%)** |

Before this change the right-hand column was identical to the left — the
registered provider was never called on this payload.

- **Observed — marker-free default costs nothing:**

  | Config | tokens | saved |
  |---|---|---|
  | markers on (previous default) | 37,791 → 24,415 | 35.4% |
| markers off (new default) | 37,791 → 24,415 | **35.4% — identical** |
  | `mode="lossy_inline"` | 37,791 → 25,129 | 33.5% |
  | `--lossless` | 37,791 → 35,100 | 7.1% |

- **Observed — memo staleness, before the fix:** registering a provider
that folds a grep block to 5 bytes left the block at its 1496-byte
built-in fold, and clearing a provider kept serving the provider's
output. Both correct after keying on the registration generation.
- **Observed — context limit:** `bedrock/anthropic.claude-3-5-sonnet`
resolves 200000 via the Anthropic provider, 128000 via the OpenAI
provider.
- **Not tested:** `tests/test_transforms_content_router.py` was not run
— it does not complete on this machine, wedging on its 5th test while
that test passes in 5.7s alone. Verified pre-existing before this work:
with the diff stashed, the clean tree stalled at the identical test, and
it stalled the same way under `HF_HUB_OFFLINE=1 HEADROOM_OFFLINE=1`. The
machine was also out of disk at the time, which may be the real cause
rather than the suspected native-detector deadlock (#575) — worth a
separate issue either way. CI should be the arbiter here.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Two pre-existing test files needed adjusting, both direct consequences
rather than scope creep:

-
`test_platform_stabilization_functional.py::test_v1_compress_success_reports_actual_metrics`
patches `openai_pipeline.apply`, which the default mode no longer routes
through. **This test was already failing on the marker-free-default
commit** before any of the fixes — my original test selection missed it.
It now patches the pipeline the route actually uses.
- `test_proxy_eager_preload_bind.py` substitutes fake pipelines to
control exactly what the preload walks; the eager build injected the
real derived router's statuses into an exact-equality assertion. Its
shared helper now clears the derived cache, preserving each test's
intent without weakening an assertion.

Docs unchecked — follow-ups worth doing in the same release: document
`config.mode` values in `docs/content/docs/litellm.mdx` and
`wiki/proxy.md`; the TS SDK `mode:"ccr"` default; and an operational
note that `COMPRESSION_TIMEOUT_SECONDS` defaults to 30
(`helpers.py:687`) while a remote ML endpoint makes one sequential call
per unit — on a large payload it trips the executor timeout and the
handler fails open, returning `compression_skipped: true` with
`tokens_before: 0`, which reads as "nothing to save" rather than "we
gave up". Those zeroed counters are misleading and worth a separate fix.
2026-07-31 12:31:38 -07:00
AxelRayandaxelray-dev d50cfabedc fix(proxy): report deferred Kompress status and promote health from cache (#2564)
## Description

When Kompress preload is deferred until first request, startup still
logs "not installed" even if ML deps are present. After the model later
loads into the module cache, /readyz and /health can keep reporting
kompress as unhealthy because reconcile only inspected attached
compressor instances. This PR reports deferred startup accurately and
promotes health from the live module cache once the model is ready,
without starting loads from health checks.

Closes #2560

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Treat eager-status `deferred` as installed-but-deferred at proxy
startup and log that state instead of "not installed".
- Promote `/readyz` and `/health` Kompress readiness from the
module-level model cache when attached compressors are missing or not
ready.
- Keep health inspection free of lazy getters and download side effects.
- Add regressions for deferred startup logging and cache-based health
promotion.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ PYTHONPATH=/tmp/headroom-2561 python -m pytest tests/test_proxy_health.py tests/test_proxy_eager_preload_bind.py -q -o addopts=
21 passed, 1 warning in 2.73s

$ ruff format --check headroom/proxy/server.py tests/test_proxy_health.py tests/test_proxy_eager_preload_bind.py
3 files already formatted

$ ruff check headroom/proxy/server.py tests/test_proxy_health.py tests/test_proxy_eager_preload_bind.py
All checks passed!
```

## Real Behavior Proof

- Environment: Linux VPS, Python 3.11 venv with headroom-ai 0.32.1 wheel
for `_core`, checked out main + this branch overlayed for source under
test
- Exact command / steps: `PYTHONPATH=/tmp/headroom-2561 python -m pytest
tests/test_proxy_health.py tests/test_proxy_eager_preload_bind.py -q -o
addopts=`; `ruff format --check` and `ruff check` on the three changed
files
- Observed result: 21 focused tests passed, including deferred startup
log regression and module-cache health promotion; ruff format/check
clean
- Not tested: live multi-request proxy with real ONNX model download on
this host; install-status follow-up mentioned in the issue comment

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Scoped to Kompress status reporting only. The separate `headroom
install status` ownership probe in the issue comment is left for a
follow-up.

Co-authored-by: axelray-dev <axelray-dev@users.noreply.github.com>
2026-07-25 19:24:08 -07:00
ParideboyandClaude Opus 4.8 d5ac07fc45 fix(proxy): bind before eager preload so a hung compressor load can't block startup (#1500)
## Description

On Windows, `headroom proxy` with optimization enabled sometimes never
opens its
listening port. `HeadroomProxy.startup()` runs inside the ASGI lifespan,
which
completes **before** uvicorn binds the socket, and the eager
compressor/parser/detector
preload ran synchronously there. The per-transform loop already swallows
exceptions,
so the only thing that can still block the bind is a **hang or an
uncatchable native
stall** during a model load. That matches the report exactly, including
that
`--no-optimize` (which skips the preload) binds fine.

This decouples the preload from the bind by running it off the event
loop under a
timeout, so startup always returns and the port binds.

Closes #790

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `proxy/server.py`:
  - Extracted the eager-preload loop into a pure sync helper
`_eager_preload_transforms()` that returns `(eager_status,
transform_statuses)`
and does **not** mutate `self.warmup` (so it is safe to run off-thread).
  - `startup()` now runs it via
    `asyncio.wait_for(asyncio.to_thread(self._eager_preload_transforms),
timeout=EAGER_PRELOAD_TIMEOUT_SECONDS)`. On timeout/exception it logs a
warning
and continues with empty status, so startup returns and uvicorn binds;
transforms
fall back to lazy loading on first use. Warmup status is merged on the
main thread
    after the await.
- `proxy/helpers.py`: added `EAGER_PRELOAD_TIMEOUT_SECONDS` (default
120s, override via
  `HEADROOM_EAGER_PRELOAD_TIMEOUT_SECONDS`). The preload is cache-only
(`allow_download=False`), so the cap only ever fires on a true hang,
never on normal
  load.
- Tests: `tests/test_proxy_eager_preload_bind.py` — helper
dedup/exception-swallow, and
(via a real `startup()`) that a hung preload no longer blocks startup
from returning
  while a normal transform still merges its warmup status.

The happy path is unchanged: a fast preload still completes before
`startup()` returns
and still populates `self.warmup`.

## Testing

- [x] Unit tests pass (`pytest tests/test_proxy_eager_preload_bind.py`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed (live Windows proxy smoke — see proof)

### Test Output

```text
$ pytest tests/test_proxy_eager_preload_bind.py -q
tests\test_proxy_eager_preload_bind.py ...                               [100%]
3 passed in 7.66s

$ ruff check headroom/proxy/server.py headroom/proxy/helpers.py tests/test_proxy_eager_preload_bind.py
All checks passed!

$ mypy headroom --ignore-missing-imports        # changed files: no new errors
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.11, `headroom` 0.28.0, Rust
`_core` loaded.
- Exact command / steps: start the proxy with optimization enabled
(which runs the preload), then curl `/health`.
  ```text
headroom proxy --port 8799 --no-telemetry # optimization ENABLED (runs
the preload)
  curl http://127.0.0.1:8799/health
  ```
- Observed result: the port binds and `/health` returns HTTP 200 with
the preload-bearing
  startup reported healthy:
  ```text
  HTTP_STATUS=200
  {"service":"headroom-proxy","status":"healthy","ready":true,

"checks":{"startup":{"enabled":true,"ready":true,"status":"healthy","error":null},
...},
   "config":{"optimize":true, ...}, "rust_core":"loaded"}
  ```
Startup completed and the socket bound with `optimize:true` on a Windows
host — the
  path that previously could hang before binding.
- Not tested: a real native model-load hang on Windows (no reliable way
to induce the
uncatchable native stall on demand). The regression test proves the
timeout/bind
decoupling deterministically by injecting a transform that blocks past
the timeout
  and asserting `startup()` still returns promptly.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Additional Notes

- Linux CI cannot reproduce the native Windows hang; the regression test
proves the
decoupling (startup returns despite a blocking preload), not the native
root cause.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 13:15:50 -07:00