mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Fixes #1832. DeepSeek traffic through the OpenAI-compatible chat path (chat/completions, GitHub Copilot, custom bases) is labelled provider `openai` unless the upstream host is OpenCode Zen. So DeepSeek requests collapse into the generic `openai` bucket in the dashboard `by_provider` / Prometheus `requests_by_provider` breakdown — the reporter sees no separate DeepSeek line. Closes #1832 ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - Added `provider_label_from_model()` in `passthrough.py`: `deepseek*` model ids now report provider `deepseek`. - Wired it into the OpenAI chat handler's provider fallback: `custom_chat_provider or provider_label_from_model(model) or "openai"`. - Vendor routing prefixes (`deepseek/deepseek-chat`, `github/deepseek-v4`) resolve to the underlying model; the prefix table is ordered so adding more OpenAI-compatible providers later is a one-line change. - Display-only: pricing stays keyed on the model, not this label, so cost math is unchanged. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest tests/test_proxy_passthrough.py -q tests/test_proxy_passthrough.py .... [100%] 4 passed in 0.30s $ ruff check headroom/proxy/handlers/openai.py headroom/proxy/passthrough.py tests/test_proxy_passthrough.py All checks passed! $ mypy headroom Success: no issues found in 509 source files ``` ## Real Behavior Proof - Environment: local checkout, Python venv, `pytest`/`ruff`/`mypy` as above. - Exact command / steps: classification is unit-tested in `tests/test_proxy_passthrough.py` — DeepSeek ids (incl. vendor prefixes and casing) map to provider `deepseek`; OpenAI/o-series/zen ids stay unlabelled. - Observed result: `deepseek*` requests now surface as a distinct `deepseek` provider in the `by_provider` / `requests_by_provider` breakdown instead of folding into `openai`. - Not tested: no live dashboard round against a real DeepSeek endpoint; verified at the classifier/handler boundary. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Documentation checklist item is N/A — this is a display-only provider-label classification with no user-facing config or docs surface. Manual testing box left unchecked: validated at the unit boundary rather than a live DeepSeek dashboard round. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2565" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `ecc8dc149036ca65dd9828ab17e31e0a9f35c06c`, upstream main `0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`. Current-main integration preserves the contributor's model-based DeepSeek reporting and known-upstream/xAI precedence. One handler conflict was mechanically reconciled with main's _resolve_openai_chat_forward_path; the contributor attribution assignment AST is identical to its published head, and the chat-path calls match main's AST. All other413incoming patches (414paths total) match old-main to current-main after diff-location normalization. Main-relative scope remains only the intended three Python files. A local UTF-8 reconstruction corrected an initial encoding mistake before tests/final commit; no unintended text/comment changes remain. Fresh validation:384 passed,27 skipped,2 warnings across provider/passthrough/integration/routing/xAI and incoming-current-main regressions. Three contributor Python files pass Ruff lint/format; candidate offline uv lock check and main-relative whitespace pass. Complete tracked native inputset/everyrecordedinputhash/artifactchecksum verified before reuse of existing3982DLL; not a fresh build. Static assignment/patch proof is not live provider, dashboard or HTTP execution. Previous106attribution/cardinality probe results remain historical; no new106caseclaim. Fresh tests use synthetic/mocked upstreams, not a live DeepSeek credential/account. Author discussion and all current inline thread states reviewed; no outstanding inline findings or author replies. Recognized upstream labels and fixed unknown-custom bucket remain ahead of model fallback; no hostname-derived telemetry label introduced. Pricing unchanged. Source/test delta contains no workflow/setup/dependency/CLI/native path changes, and current-main publication security gate passes. Contributor correctness/intent unchanged. Exact published-head hosted checks and final fresh discussion/main/mergeability guard remain before approval. Maintainers handle CI; no PR merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
166 lines
5.9 KiB
Python
166 lines
5.9 KiB
Python
from headroom.proxy.passthrough import (
|
|
custom_base_passthrough_telemetry,
|
|
is_opencode_zen_base,
|
|
provider_label_from_model,
|
|
)
|
|
|
|
|
|
def test_custom_base_passthrough_telemetry_recognizes_opencode_zen_chat() -> None:
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"/zen/v1/chat/completions",
|
|
"https://opencode.ai/",
|
|
) == ("chat/completions", "zen")
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"zen/v1/chat/completions",
|
|
"https://www.opencode.ai",
|
|
) == ("chat/completions", "zen")
|
|
|
|
|
|
def test_custom_base_passthrough_telemetry_ignores_non_matching_traffic() -> None:
|
|
assert custom_base_passthrough_telemetry(
|
|
"GET",
|
|
"/zen/v1/chat/completions",
|
|
"https://opencode.ai/",
|
|
) == ("", "")
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"/v1/chat/completions",
|
|
"https://opencode.ai/",
|
|
) == ("", "")
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"/zen/v1/chat/completions",
|
|
"https://custom.example/",
|
|
) == ("", "")
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"/zen/v1/chat/completions",
|
|
"://bad-url",
|
|
) == ("", "")
|
|
|
|
|
|
def test_provider_label_from_model_classifies_deepseek() -> None:
|
|
assert provider_label_from_model("deepseek-chat") == "deepseek"
|
|
assert provider_label_from_model("deepseek-reasoner") == "deepseek"
|
|
assert provider_label_from_model("DeepSeek-V4-Flash") == "deepseek"
|
|
# Vendor-prefixed routing ids resolve to the underlying model.
|
|
assert provider_label_from_model("deepseek/deepseek-chat") == "deepseek"
|
|
assert provider_label_from_model("github/deepseek-v4") == "deepseek"
|
|
|
|
|
|
def test_provider_label_from_model_leaves_openai_models_unlabeled() -> None:
|
|
for model in ("gpt-5.4-mini", "openai/gpt-4o", "o3", "zen", "", None):
|
|
assert provider_label_from_model(model) == ""
|
|
|
|
|
|
def test_is_opencode_zen_base_recognizes_zen_origins() -> None:
|
|
assert is_opencode_zen_base("https://opencode.ai")
|
|
assert is_opencode_zen_base("https://www.opencode.ai")
|
|
assert is_opencode_zen_base("https://opencode.ai/zen")
|
|
|
|
|
|
def test_is_opencode_zen_base_rejects_other_or_missing_bases() -> None:
|
|
assert not is_opencode_zen_base(None)
|
|
assert not is_opencode_zen_base("")
|
|
assert not is_opencode_zen_base("https://custom.example")
|
|
assert not is_opencode_zen_base("https://opencode.ai.evil.example")
|
|
assert not is_opencode_zen_base("://bad-url")
|
|
|
|
|
|
def test_custom_base_passthrough_telemetry_names_known_chat_hosts() -> None:
|
|
# Exact hosts only, fixed labels: the taxonomy, not the request, decides.
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"/v1/chat/completions",
|
|
"https://api.z.ai/api/coding/paas/v4",
|
|
) == ("chat/completions", "zai")
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"v1/chat/completions",
|
|
"https://api.meta.ai/v1",
|
|
) == ("chat/completions", "meta")
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"/v1/chat/completions",
|
|
"https://api.openai.com/v1",
|
|
) == ("chat/completions", "openai")
|
|
# Host parsing is case-insensitive; the label stays fixed.
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"/v1/chat/completions",
|
|
"https://API.Z.AI/api/coding/paas/v4",
|
|
) == ("chat/completions", "zai")
|
|
|
|
|
|
def test_custom_base_passthrough_telemetry_keeps_everything_else_unnamed() -> None:
|
|
# Lookalike hosts never match: an exact-host set cannot be talked into
|
|
# naming an attacker-controlled upstream.
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"/v1/chat/completions",
|
|
"https://api.z.ai.evil.test/v1",
|
|
) == ("", "")
|
|
# Unknown hosts, non-chat paths, non-POST methods, and bad URLs all stay
|
|
# unnamed so nothing request-controlled becomes a telemetry label.
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"/v1/chat/completions",
|
|
"https://llm.example.internal/v1",
|
|
) == ("", "")
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"/v1/embeddings",
|
|
"https://api.z.ai/v1",
|
|
) == ("", "")
|
|
assert custom_base_passthrough_telemetry(
|
|
"GET",
|
|
"/v1/chat/completions",
|
|
"https://api.z.ai/v1",
|
|
) == ("", "")
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"/v1/chat/completions",
|
|
"://bad-url",
|
|
) == ("", "")
|
|
|
|
|
|
def test_custom_base_passthrough_telemetry_requires_a_chat_path_segment() -> None:
|
|
# A bare suffix match would also name ``notchat/completions``.
|
|
for path in ("/v1/notchat/completions", "/v1/xchat/completions", "notchat/completions"):
|
|
assert custom_base_passthrough_telemetry("POST", path, "https://api.z.ai/v1") == ("", "")
|
|
# The exact segment, bare or nested, still names the host.
|
|
for path in ("chat/completions", "/chat/completions", "/api/paas/v4/chat/completions"):
|
|
assert custom_base_passthrough_telemetry("POST", path, "https://api.z.ai/v1") == (
|
|
"chat/completions",
|
|
"zai",
|
|
)
|
|
|
|
|
|
def test_custom_base_passthrough_telemetry_recognizes_xai_chat() -> None:
|
|
# Grok Build routes plain OpenAI chat completions through custom-base
|
|
# routing with base_url https://api.x.ai; attribute the provider as xai
|
|
# so savings rollups separate grok traffic from OpenAI's.
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"/v1/chat/completions",
|
|
"https://api.x.ai",
|
|
) == ("chat/completions", "xai")
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"/v1/chat/completions",
|
|
"https://api.x.ai.",
|
|
) == ("chat/completions", "xai")
|
|
# Other xai paths and methods stay unlabelled.
|
|
assert custom_base_passthrough_telemetry(
|
|
"GET",
|
|
"/v1/chat/completions",
|
|
"https://api.x.ai",
|
|
) == ("", "")
|
|
assert custom_base_passthrough_telemetry(
|
|
"POST",
|
|
"/v1/models",
|
|
"https://api.x.ai",
|
|
) == ("", "")
|