mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
feat/https-compose-caddy
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
9ad16a77f8 |
feat(proxy): report DeepSeek as its own provider in dashboard (#2565)
## Description Fixes #1832. DeepSeek traffic through the OpenAI-compatible chat path (chat/completions, GitHub Copilot, custom bases) is labelled provider `openai` unless the upstream host is OpenCode Zen. So DeepSeek requests collapse into the generic `openai` bucket in the dashboard `by_provider` / Prometheus `requests_by_provider` breakdown — the reporter sees no separate DeepSeek line. Closes #1832 ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - Added `provider_label_from_model()` in `passthrough.py`: `deepseek*` model ids now report provider `deepseek`. - Wired it into the OpenAI chat handler's provider fallback: `custom_chat_provider or provider_label_from_model(model) or "openai"`. - Vendor routing prefixes (`deepseek/deepseek-chat`, `github/deepseek-v4`) resolve to the underlying model; the prefix table is ordered so adding more OpenAI-compatible providers later is a one-line change. - Display-only: pricing stays keyed on the model, not this label, so cost math is unchanged. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest tests/test_proxy_passthrough.py -q tests/test_proxy_passthrough.py .... [100%] 4 passed in 0.30s $ ruff check headroom/proxy/handlers/openai.py headroom/proxy/passthrough.py tests/test_proxy_passthrough.py All checks passed! $ mypy headroom Success: no issues found in 509 source files ``` ## Real Behavior Proof - Environment: local checkout, Python venv, `pytest`/`ruff`/`mypy` as above. - Exact command / steps: classification is unit-tested in `tests/test_proxy_passthrough.py` — DeepSeek ids (incl. vendor prefixes and casing) map to provider `deepseek`; OpenAI/o-series/zen ids stay unlabelled. - Observed result: `deepseek*` requests now surface as a distinct `deepseek` provider in the `by_provider` / `requests_by_provider` breakdown instead of folding into `openai`. - Not tested: no live dashboard round against a real DeepSeek endpoint; verified at the classifier/handler boundary. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Documentation checklist item is N/A — this is a display-only provider-label classification with no user-facing config or docs surface. Manual testing box left unchecked: validated at the unit boundary rather than a live DeepSeek dashboard round. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2565" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `ecc8dc149036ca65dd9828ab17e31e0a9f35c06c`, upstream main `0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`. Current-main integration preserves the contributor's model-based DeepSeek reporting and known-upstream/xAI precedence. One handler conflict was mechanically reconciled with main's _resolve_openai_chat_forward_path; the contributor attribution assignment AST is identical to its published head, and the chat-path calls match main's AST. All other413incoming patches (414paths total) match old-main to current-main after diff-location normalization. Main-relative scope remains only the intended three Python files. A local UTF-8 reconstruction corrected an initial encoding mistake before tests/final commit; no unintended text/comment changes remain. Fresh validation:384 passed,27 skipped,2 warnings across provider/passthrough/integration/routing/xAI and incoming-current-main regressions. Three contributor Python files pass Ruff lint/format; candidate offline uv lock check and main-relative whitespace pass. Complete tracked native inputset/everyrecordedinputhash/artifactchecksum verified before reuse of existing3982DLL; not a fresh build. Static assignment/patch proof is not live provider, dashboard or HTTP execution. Previous106attribution/cardinality probe results remain historical; no new106caseclaim. Fresh tests use synthetic/mocked upstreams, not a live DeepSeek credential/account. Author discussion and all current inline thread states reviewed; no outstanding inline findings or author replies. Recognized upstream labels and fixed unknown-custom bucket remain ahead of model fallback; no hostname-derived telemetry label introduced. Pricing unchanged. Source/test delta contains no workflow/setup/dependency/CLI/native path changes, and current-main publication security gate passes. Contributor correctness/intent unchanged. Exact published-head hosted checks and final fresh discussion/main/mergeability guard remain before approval. Maintainers handle CI; no PR merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
b1fb162b62 |
fix(grok): attribute api.x.ai chat outcomes as provider xai, recognize grok-shell UA (#2626)
## Description Grok Build chat requests were recorded as `custom` or `openai`, preventing xAI provider rollups. This change records custom-base and directly routed Grok chat traffic as `xai`, recognizes both `grok/` and `grok-shell/` clients, and keeps API-key requests on credential-driven PAYG policy. Closes #3996 ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Changes Made - Attribute supported xAI chat routes, including trailing-dot hosts, to `xai`. - Preserve per-model catalog cache prices and share the xAI cache fallback between dashboard costs and persisted savings. - Exclude inferred cache writes from billed aggregate totals; those tokens already belong to uncached input. Explicit provider-reported writes remain billable. - Cover custom-base outcome integration, catalog pricing precedence, legacy and current client credentials, and inferred writes with and without TTL splits. ## Testing At `29fd4f9d7a24a76f3a3f53edacc5832e357cae58`: 130 focused pricing, accounting, auth, cache, and routing tests pass (one existing Starlette warning). Ruff lint and format, whitespace checks, and Linux-target mypy over 567 source files pass. Main refresh retains `3c418f6f` ancestry. Latest documentation dependency audit found zero vulnerabilities. ## Real Behavior Proof - Environment: Windows, Python 3.12, a real uvicorn proxy and a separate loopback HTTP upstream. Subscription polling disabled. Only upstream network delivery is redirected; this is not live xAI billing traffic. - Steps: send four real TCP chat POSTs with custom-base, direct, legacy Grok, and trailing-dot routing; query the running proxy's `/stats` endpoint. The upstream returns 10,000 input tokens, 5,000 cached input tokens, and 1,000 output tokens per request. A priced xAI catalog fixture has $1/M input and $2/M output, with no explicit cache-read rate. - Observed: all four requests return HTTP 200, outcomes record `provider=xai` and `client=grok_build`, and `/stats` records `{"xai":4}`. Input spend is $0.0232 using the xAI 0.16 cache fallback. Before the aggregate-accounting correction this same replay reported $0.0432. - Evidence: `pr2626-resume15-pricing-runtime.log` and `pr2626-resume15-final-tests.log`; regressions reproduce the pricing/auth defects before correction. ## Runtime Rollout Safety - Rollout-managed features: none; attribution, credential classification, and accounting use the existing request paths. - Default behavior: supported xAI routes receive the xAI label; API keys use PAYG policy, JWT credentials retain OAuth policy. Unknown gateways retain the bounded custom provider bucket. - Pricing behavior: when a priced catalog row has no cache rates and the caller omits `provider`, `resolve_rates` now identifies the provider from `litellm_provider` and uses the shared fallback. For xAI this means cached input costs 0.16 times list input; explicit model cache rates still win. Dashboard aggregates also exclude inferred writes already counted as uncached input. These corrections change reported input costs and persisted savings, not upstream requests or billing. - Rollback: revert the pricing/auth/attribution commits; no new rollout switch or unsafe override is introduced. - Coverage: real HTTP routing and `/stats` replay plus focused default-policy and accounting tests. ## Checklist - [x] Self-review completed - [x] Added regression coverage - [x] Relevant local tests and static checks pass - [x] Did not edit CHANGELOG.md ## Additional Notes Hosted checks and automated review must qualify this newly published head before approval. No PR merge performed. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
f0ec2bb376 |
fix(telemetry): label custom-base chat upstreams from a fixed provider set (#3759)
## Description
Unrecognized custom-base chat upstreams report
`outcome_provider="openai"`.
Dashboards and per-provider stores merge every OpenAI-compatible vendor
except Zen into `openai`.
### Fix
Labels come from a fixed set in `custom_base_passthrough_telemetry()`:
| Custom base host | Label |
| --- | --- |
| `api.z.ai` | `zai` |
| `api.meta.ai` | `meta` |
| `api.openai.com` | `openai` |
| anything else | `custom` |
- **Match:** exact host, `chat/completions` path
- **Zen:** unchanged
- **No base header:** `openai`, as before
- **Bounded:** the hostname is never a label
Formatting and the SSRF guard are unchanged.
Pricing tables are unchanged; provider-dependent savings estimates can
change.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **Fix:** `CUSTOM_BASE_CHAT_PROVIDERS`, exact host to label
- **Fix:** `CUSTOM_BASE_PROVIDER = "custom"` fallback
- **Fix:** chat handler reports the fixed-set label
- **Removed:** `_custom_base_provider_label()`
- **Test:** known hosts, subpath bases, host case
- **Test:** lookalike hosts, non-chat paths, bad URLs
- **Test:** `zai`, `meta`, `custom`, no-header control
- **Test:** 1,000-request flood across every sink
### Flood regression
Through the real chat handler:
1. 980 distinct hosts: ok, upstream 502, local 429, upstream 429
2. 20 `api.z.ai` requests: ok and local 429
3. Exact counts: in-memory, lifetime, OTel
4. `/metrics` labels: inside `{openai, zen, zai, meta, custom}`
The pre-seeded zero `unknown` failure bucket is the one extra label, on
`main` too.
## Testing
- [x] Unit tests pass (`pytest`): no new failures; see baseline below
- [x] Linting passes (`ruff check .`), plus `ruff format --check .`
- [x] Type checking passes (`mypy headroom`):
`--ignore-missing-imports`, mypy 1.20.2 as in CI
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
new tests against main
|
||
|
|
6f404de54e |
fix(proxy): keep OpenCode Zen /responses requests streaming (#3659)
## Description Fixes #3656. When the OpenCode transport plugin is loaded, OpenCode Zen free-tier models on `/zen/v1/responses` (e.g. `muse-spark-1.3-contributor-free`) fail with `403 FreeTierError` ("OpenCode's free tier can only be used from within OpenCode"), while the same request succeeds with the plugin removed. Root cause: the Responses path's buffered-CCR mode rewrites a streaming request to `stream:false` and `accept: application/json` before it goes upstream. Zen validates OpenCode-client attribution on the wire and rejects the reshaped request. The chat-completions path is unaffected because `_should_inject_openai_chat_ccr_tool` already leaves streaming chat requests alone. The general "opt a provider/route out of plugin routing" escape hatch is tracked separately in #3657 and is not needed for this fix. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/passthrough.py`: add `is_opencode_zen_base()`, reusing the existing `OPENCODE_ZEN_HOSTS`. - `headroom/proxy/handlers/openai.py`: `_should_buffer_openai_responses_stream_ccr()` takes `upstream_base_url` and returns `False` for OpenCode Zen, so the client's streaming request is forwarded unchanged. `x-headroom-base-url` is now resolved once and reused for the secret-header gate, the routing decision, and this decision. - Tests for both helpers, including that other custom bases still buffer. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) — could not run cleanly in this local env (numpy stub/Python-version mismatch aborts mypy); `mypy --ignore-missing-imports headroom/proxy/passthrough.py` passes. CI will run the full check. - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/proxy/handlers/openai.py headroom/proxy/passthrough.py tests/test_proxy_openai_responses_stream_ccr.py tests/test_proxy_passthrough.py All checks passed! $ ruff format --check <same files> 4 files already formatted $ pytest tests/test_proxy_openai_responses_stream_ccr.py tests/test_proxy_passthrough.py -q 9 passed $ pytest tests/test_proxy_openai_responses_stream_ccr.py tests/test_proxy_passthrough.py \ tests/test_ccr_response_handler_extra.py tests/test_proxy_byte_faithful_forwarding.py \ tests/test_proxy_openai_responses_integration.py tests/test_openai_codex_routing.py -q 140 passed, 14 skipped ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.13.7, OpenCode CLI 1.18.31, provider `opencode` (OpenCode Zen pay-as-you-go), model `muse-spark-1.3-contributor-free`, Headroom OpenCode transport plugin loaded, OpenCode pointed at the proxy under test. - Exact command / steps: `opencode run -m opencode/muse-spark-1.3-contributor-free "Reply with exactly: OK"`. - Observed result: before the fix the command fails with `Error from provider (Console): OpenCode's free tier can only be used from within OpenCode` (installed 0.37.0 proxy, unpatched); after the fix the same command returns `OK` (this branch, `python -m headroom.cli proxy`, 0.38.0-dev). Control: the unpatched 0.37.0 proxy with `--no-ccr` also returns `OK`, which isolates the buffered-CCR rewrite as the trigger rather than the proxy hop or headers. - Not tested: Codex/ChatGPT auth flow, other custom gateways, and any other Zen model. The "before" run used released 0.37.0 while the "after" run used `main` (0.38.0-dev); the unit tests pin the predicate behaviour on this branch. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a. - Stable/default behavior changed: buffered CCR on `/v1/responses` is skipped **only** when the client-supplied upstream is `opencode.ai` / `www.opencode.ai`. Every other upstream keeps the existing buffered path. - Kill switch / disable path: `--no-ccr` still disables the path entirely; omitting `x-headroom-base-url` falls back to the configured upstream. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Happy to adjust the carve-out (host-level vs path-level) if you'd prefer a different surface. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Screenshots (if applicable) N/A (CLI proof in Real Behavior Proof above). ## Additional Notes - Kept the carve-out host-scoped to match the existing OpenCode Zen special-casing in `headroom/proxy/passthrough.py` (#1602). A broader "never reshape custom-base `/responses` requests" rule is possible but would affect Codex-style custom gateways, so it is left out intentionally. - Trade-off: on Zen `/responses`, CCR retrieval is no longer resolved server-side. The precondition for this path is that the client advertises the `headroom_retrieve` tool, so the model can still call it and have the client resolve it. |
||
|
|
e6243f65c9 |
refactor(providers): split proxy route adapters (#1934)
## Description Refactors provider-specific proxy routing into provider-owned helper modules so `headroom/providers/proxy_routes.py` primarily registers routes and delegates behavior. This keeps Codex, OpenAI Responses/images, model metadata, Vertex, Cloud Code, passthrough target selection, and request path normalization logic testable outside the route table. Closes # ## Type of Change - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Extracted Codex routing helpers for headers, endpoint URLs, image forwarding, response subpaths, and model metadata. - Moved provider target selection, route specs, OpenAI Responses/images helpers, Vertex runtime helpers, Cloud Code path normalization, passthrough telemetry, and request scope normalization into focused modules. - Kept `proxy_routes.py` as route registration/delegation and preserved current-main `/v1/messages` custom-base behavior. - Added focused provider/proxy tests for the extracted modules and route delegation behavior. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_package_init_lazy.py::test_codex_package_import_stays_runtime_only tests/test_provider_cloudcode_runtime.py tests/test_provider_codex_endpoints.py tests/test_provider_codex_headers.py tests/test_provider_codex_images.py tests/test_provider_codex_model_metadata.py tests/test_provider_codex_responses.py tests/test_provider_model_metadata.py tests/test_provider_openai_images.py tests/test_provider_openai_responses.py tests/test_provider_proxy_targets.py tests/test_provider_route_specs.py tests/test_provider_vertex_runtime.py tests/test_proxy_request_scope.py tests/test_provider_proxy_routes.py::test_provider_passthrough_routes_forward_expected_targets tests/test_provider_proxy_routes.py::test_proxy_route_helpers_prefer_legacy_targets_and_gemini_passthrough tests/test_provider_proxy_routes.py::test_provider_specific_routes_delegate_to_expected_proxy_handlers tests/test_provider_proxy_routes.py::test_openai_response_websocket_aliases_delegate_to_openai_ws_handler tests/test_provider_proxy_routes.py::test_openai_response_subpath_passthrough_returns_502_on_http_failure tests/test_provider_proxy_routes.py::test_openai_response_subpath_passthrough_uses_openai_target tests/test_provider_proxy_routes.py::test_openai_response_subpath_aliases_and_chatgpt_auth_use_expected_targets tests/test_provider_proxy_routes.py::test_openai_image_routes_use_codex_backend_under_chatgpt_auth tests/test_provider_proxy_routes.py::test_openai_image_codex_response_strips_stale_compression_headers tests/test_provider_proxy_routes.py::test_openai_image_edits_api_key_auth_falls_through_to_openai_passthrough tests/test_provider_proxy_routes.py::test_openai_image_edits_preserves_multipart_body_under_chatgpt_auth tests/test_provider_proxy_routes.py::test_gemini_batch_embed_contents_passthrough_uses_gemini_target tests/test_provider_proxy_routes.py::test_v1_models_fetches_codex_registry_under_chatgpt_auth tests/test_provider_proxy_routes.py::test_v1_models_falls_back_to_synthetic_list_under_chatgpt_auth tests/test_provider_proxy_routes.py::test_v1_models_get_single_dynamic_under_chatgpt_auth tests/test_provider_proxy_routes.py::test_v1_models_still_forwards_under_non_chatgpt_auth tests/test_provider_proxy_routes.py::test_v1_models_routes_claude_code_gateway_discovery_to_anthropic tests/test_provider_proxy_routes.py::test_anthropic_model_metadata_strips_ansi_model_ids tests/test_custom_base_passthrough_telemetry.py tests/test_proxy_passthrough.py tests/test_proxy_google_cloudcode_route_aliases.py tests/test_proxy_project_savings.py::test_with_project_prefix_round_trips_through_split tests/test_vertex_claude_compression.py ============================ 102 passed in 34.83s ============================= python -m ruff check headroom/providers/cloudcode headroom/providers/codex headroom/providers/vertex headroom/providers/model_metadata.py headroom/providers/openai_images.py headroom/providers/openai_responses.py headroom/providers/proxy_targets.py headroom/providers/route_specs.py headroom/providers/proxy_routes.py headroom/proxy/handlers/openai.py headroom/proxy/passthrough.py headroom/proxy/request_scope.py headroom/proxy/project_context.py tests/test_package_init_lazy.py tests/test_provider_cloudcode_runtime.py tests/test_provider_codex_endpoints.py tests/test_provider_codex_headers.py tests/test_provider_codex_images.py tests/test_provider_codex_model_metadata.py tests/test_provider_codex_responses.py tests/test_provider_model_metadata.py tests/test_provider_openai_images.py tests/test_provider_openai_responses.py tests/test_provider_proxy_targets.py tests/test_provider_route_specs.py tests/test_provider_vertex_runtime.py tests/test_proxy_request_scope.py tests/test_provider_proxy_routes.py tests/test_custom_base_passthrough_telemetry.py tests/test_proxy_passthrough.py tests/test_proxy_google_cloudcode_route_aliases.py tests/test_proxy_project_savings.py tests/test_vertex_claude_compression.py All checks passed! python -m compileall -q headroom\providers\cloudcode headroom\providers\codex headroom\providers\vertex headroom\providers\model_metadata.py headroom\providers\openai_images.py headroom\providers\openai_responses.py headroom\providers\proxy_targets.py headroom\providers\route_specs.py headroom\providers\proxy_routes.py headroom\proxy\handlers\openai.py headroom\proxy\passthrough.py headroom\proxy\request_scope.py headroom\proxy\project_context.py # no output; exited 0 git commit -m "refactor(providers): split proxy route adapters" Sync plugin versions.....................................................Passed check for merge conflicts................................................Passed ruff.....................................................................Passed ruff-format..............................................................Passed mypy.....................................................................Passed ``` ## Real Behavior Proof - Environment: Windows PowerShell, Python 3.13.13, branch `jd/provider-route-slices` based on `headroomlabs/main`. - Exact command / steps: Ran the focused provider/proxy pytest suite, focused ruff command, compileall over changed Python modules, and commit hooks. - Observed result: Provider/proxy route behavior tests passed; lint, formatting, and mypy passed. - Not tested: Full pytest suite, live upstream provider calls, and manual end-to-end proxy traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Documentation and CHANGELOG updates are N/A for this internal refactor. The full pytest suite was not run; coverage here is focused on provider/proxy routing behavior touched by this slice. --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> |