mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
50bf775e8c |
fix(proxy): bypass compression for large admitted histories (#4142)
## Description Closes #4139. A history that is too large to compress should still reach the provider. This keeps the 100 MiB hard admission limit and bypasses request-side transforms above 16 MiB decoded size. The hard limit still applies before forwarding. Current head: `62ffe3defef1684f70d96f1640818d99445a43bb`, merging current main `a2d4fe72` into `1902d8ef`. The 42-test/native/manual/pre-push results below qualify `1902d8ef`, not this new merge. New-head shard 4 has a cached-prefix fixture failure under investigation; coverage uploads are still incomplete. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement - [x] Documentation update ## Changes Made - Keep the startup override and both wire/decompressed hard bounds. Large admitted histories skip normalization, image processing, memory injection, tokenization and compression on Messages, Chat/Responses, Gemini and Bedrock HTTP routes. - Keep provider routing and backend translation, including Copilot Chat, Codex `store:false` and Vertex route-forced streaming. Native forwarding retains decoded bytes unless the provider's wire contract requires a rewrite; translation retains the full message text. - Report actual provider usage, with zero compression savings or attempted tokens for the bypass. Keep `/v1/compress` and WebSocket contracts unchanged. - Add real-provider coverage for large Gemini JSON/SSE requests with query-key auth, and OpenAI JSON/SSE through the installed LiteLLM adapter. No admission-policy or CI changes in this follow-up. ## Testing - [x] Unit tests pass (focused suite and normal critical pre-push suite) - [x] Linting passes (normal hooks) - [x] New tests added for admission, recovery and passthrough accounting - [x] Manual testing performed against the real native build ### Test Output ```sh python -m pytest -q tests/test_proxy_compression_admission.py tests/test_proxy/test_gemini_plugin_routed_streaming.py --cov=headroom --cov-branch # 42 passed, 41 warnings ``` Normal pre-push `make ci-precheck` passed: 1668 Rust tests (3 ignored), 181 Python tests, lint/type checks and commitlint. No hook bypass. The initial deep-module coverage command hit a Pydantic import-isolation error; using the repo's `--cov=headroom` convention passed both complete modules without disabling plugins or tests. The earlier `96c5fe7e` qualification covered the other native protocols, configured/per-request backends, hard-ceiling recovery and complete local shards 2/3. Those runs are historical, not a new whole-suite claim. New-head hosted success is not claimed. ## Real Behavior Proof - Environment: Linux x86_64, Python 3.13.14, freshly built real `headroom._core`, public Headroom ASGI routes, a real localhost HTTP provider and installed LiteLLM. Synthetic credentials, offline mode and `DO_NOT_TRACK=1`; no live provider calls. - Exact command / steps: Start `create_app(ProxyConfig(backend="litellm-openai", ...))` with all upstream URLs set to the localhost peer and `litellm.api_base=peer+"/v1"`. Leave the production 16 MiB soft/100 MiB hard limits unchanged. POST `"history " * (16777216 // 8 + 1024) + "LAST TURN"` as a user message to Chat Completions, then as a Gemini content part to generateContent/streamGenerateContent, with streaming off/on. The peer rejects a changed history, requires Gemini query-key auth and one `alt=sse`, and returns protocol-valid JSON/SSE with 101 input/1 output tokens. The maintained four small-ceiling cases are in `tests/test_proxy_compression_admission.py` and run with the command above. - Observed result: All four >16 MiB requests returned 200 with the full history intact. JSON/SSE and Gemini auth survived; streamGenerateContent forced SSE even with incoming `alt=json`. Each request recorded exactly 101 provider input/1 output tokens, zero compression attempts/savings and no failures; `/livez` stayed 200. Earlier recorded qualification also covered all five native HTTP protocols, real translating backends and 413 recovery. - Not tested: Live cloud accounts, the reporter's deployed 4 GB VPS, screenshot/image quality or a zero-JSON-parse path. One envelope/model JSON parse remains; no proxy-only RSS guarantee is claimed. ## Runtime Rollout Safety - Rollout-managed feature(s): None; this is the admission/compatibility fix. - Minimum rollout channel: Stable/default. - Stable/default behavior changed: Keep the 100 MiB hard ceiling; bypass request-side transforms above 16 MiB decoded size on the named HTTP routes. - Kill switch / disable path: Set `HEADROOM_REQUEST_BODY_MAX_BYTES=16777216` (or lower) and restart, accepting local 413s instead of the large-body path. - Unsafe override required: No. - Qualification impact: Local native protocols, backend translation, hard bounds and recovery exercised; live-provider/VPS qualification remains external. - Rollback path: Revert the admission revision and restart; no storage migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] I have made corresponding changes to the documentation - [x] I did not edit `CHANGELOG.md` --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
72302a3998 |
fix(proxy): say why proxy.log is missing and why a knob or extension was ignored (#4088)
## Description Several failures an operator needs to see are swallowed silently or logged only at DEBUG. This PR makes each of them visible with one warning. It does not change defaults or control flow. 1. **Missing runtime log.** `_setup_file_logging` caught `OSError` with `pass`. On a read-only workspace, a wrong owner, or a file where the logs directory belongs, `proxy.log` simply never appeared and nothing said why. It now logs `Headroom runtime log disabled: cannot write the log file: <OSError>. Point HEADROOM_WORKSPACE_DIR at a writable directory to restore proxy.log.` The OSError text names the path. The symlink case just above it already warned this way. 2. **Six import-time numeric knobs.** Each one fell back to its default on an unparseable value, without a word: - `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` - `HEADROOM_COLD_START_FAST_PASS_TIMEOUT_SECONDS` - `HEADROOM_EAGER_PRELOAD_TIMEOUT_SECONDS` - `HEADROOM_COMPRESSION_CACHE_MAX_SESSIONS` - `HEADROOM_COMPRESSION_CACHE_TTL_SECONDS` - `HEADROOM_COMPRESSION_CACHE_MAX_ENTRIES` Each now logs `Ignoring invalid NAME='value'; using DEFAULT` once, at import. That matches the idiom in `hard_watchdog.py`. 3. **Extension discovery failure.** When entry-point enumeration fails, every proxy extension is dropped, Enterprise plugins included. That was logged at DEBUG. It is now a warning: `proxy extensions: entry-point enumeration failed (<describe_exception>), so no proxy extensions will be installed`. `describe_exception` gives the exception types and code locations, never the message, and no traceback is logged at any level. It runs at startup only: once in `install_all`, and once more for the CLI banner. 4. **Pricing lookups in `cost.py`.** `_get_list_price`, `_get_output_price`, `_get_cache_prices` and the cache-rate lookup in the savings summary returned `None` on any exception. A pricing bug therefore showed up as a $0 cost with no trace. All four, plus the existing `estimate_cost` path, go through one helper, `_warn_pricing_once(lookup, what, model, exc)`. Example line: `Failed to get pricing for model 'gpt-4o': RuntimeError at headroom/proxy/cost.py:... in estimate_cost`. It logs the lookup name, `safe_id(model)` and `describe_exception(exc)` (types and code locations, never the message), at WARNING only. No level carries the raw exception or a traceback, because a pricing backend error can quote provider responses or credentials. Dedupe is by `(lookup, model)` through the shared `WarnOnce(1024, "unpriceable models")`. Each pair warns at most once an hour (the `WarnOnce` window). Past 1024 models, one overflow notice is logged, and a key is not used up while WARNING is disabled. 5. **Malformed `HEADROOM_OPENAI_TOOL_SEARCH_MODELS` regex.** The override was ignored on every request with no log. It now warns once per bad pattern, and still falls back to the version gate as before. A valid pattern re-arms the warning, so bad A, then valid B, then bad A again warns twice. None of the new lines logs header values, request content or credentials. They log only knob names, their raw values (operator-set env), model names, regex patterns, exception type names, the `re.error` text, and `OSError` text naming a local path. No exception message or traceback from pricing or extension discovery reaches any log level. Refs #4136 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: - add `_warn_invalid_env_number(name, default)` and call it from the six import-time `except ValueError` branches; - log the `OSError` in `_setup_file_logging`; - add a once-per-pattern warning for a malformed `HEADROOM_OPENAI_TOOL_SEARCH_MODELS`. - `headroom/proxy/extensions.py`: entry-point enumeration failure logs at warning with `describe_exception(exc)` instead of at debug. - `headroom/proxy/cost.py`: - the four silent pricing `except Exception` branches call `_warn_pricing_once`; - `_warn_pricing_once(lookup, what, model, exc)` logs `safe_id(model)` plus `describe_exception(exc)` at WARNING, with nothing at DEBUG. It dedupes through `log_safety.WarnOnce`. - `tests/test_cost_pricing_warning_dedup.py`: the tests reset with a fresh `WarnOnce`, and match the quoted model id (`for model 'glm-5.2':`). - Regression tests. Each one attaches a handler to the module logger directly, because caplog misses `headroom.*` records after proxy startup. Each positive test fails on main and is paired with a negative case that passes on both. - `tests/test_cache_breakpoint_diagnostics.py`: - `test_runtime_log_says_why_it_is_missing` makes the logs directory a regular file. It asserts one warning that names the path and `HEADROOM_WORKSPACE_DIR`. - Negative: `test_runtime_log_stays_quiet_when_writable`. - `tests/test_openai_tool_search_deferral.py`: - `test_malformed_override_warns_once_not_per_request` calls the gate 5 times with a bad pattern. It asserts exactly one warning and that the version-gate result is unchanged. - Negative: a valid override is silent (same test). - `test_valid_override_rearms_the_malformed_warning`: bad, valid, bad gives two warnings. - `tests/test_proxy/test_compression_timeout_config.py`: - `test_bad_env_says_it_was_ignored` reloads `helpers` with `HEADROOM_COMPRESSION_TIMEOUT_SECONDS=not-a-number` and asserts the exact warning. - Negative: `test_valid_env_is_silent`. The existing fixture reloads `helpers` afterwards. - `tests/test_proxy_extensions.py`: - `test_discover_warns_when_entry_point_enumeration_fails` makes `importlib.metadata.entry_points` raise. It asserts one WARNING that says no extensions will be installed and names `RuntimeError` but not the exception text. - Negative: `test_discover_is_quiet_when_nothing_is_installed`. - `tests/test_cost_cache_prices.py`: - `test_cache_rate_failure_warns_once_per_model` makes `resolve_rates` raise. It asserts two `_get_cache_prices` calls give one warning naming the model and `RuntimeError`, without the exception text. - Negative: a resolvable model is silent (same test). - `test_pricing_error_never_logs_its_text_at_any_level` runs the real `CostTracker.estimate_cost('gpt-4o', 100, 10)` with a pricing error whose text holds an Authorization credential canary, with `headroom.proxy` at DEBUG. It formats every captured record with `logging.Formatter` (which renders `exc_info`), asserts the canary is absent and the result is `None`, and asserts the warning is present. On the previous commit it fails: the formatted DEBUG traceback contains the canary. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q tests/test_cost_cache_prices.py -k never_logs # previous commit |
||
|
|
addebdb29c |
feat(proxy): make COMPRESSION_TIMEOUT_SECONDS configurable via env (#946) (#991)
## Description The compression-pipeline timeout was hard-coded at 30s, so slow CPUs and long Claude Code conversations had no recourse. #946 asks to wire `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` through. Refs #946. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - Read `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` from the environment (float), falling back to 30 on an unparseable value. ## Testing - [x] Unit tests pass (`pytest`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_proxy/test_compression_timeout_config.py -q 4 passed in 0.09s ``` ## Real Behavior Proof - Environment: Linux, Python 3.13, editable build of this branch - Exact command / steps: `HEADROOM_COMPRESSION_TIMEOUT_SECONDS=88 python -c "import headroom.proxy.helpers as h; print(h.COMPRESSION_TIMEOUT_SECONDS)"` - Observed result: prints `88.0` (default `30.0`; an unparseable value falls back to `30.0`) - Not tested: a live compression actually exceeding the configured timeout under real load ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |