Files
headroom/tests/test_proxy_passthrough.py
9ad16a77f8 feat(proxy): report DeepSeek as its own provider in dashboard (#2565)
## Description

Fixes #1832.

DeepSeek traffic through the OpenAI-compatible chat path
(chat/completions, GitHub Copilot, custom bases) is labelled provider
`openai` unless the upstream host is OpenCode Zen. So DeepSeek requests
collapse into the generic `openai` bucket in the dashboard `by_provider`
/ Prometheus `requests_by_provider` breakdown — the reporter sees no
separate DeepSeek line.

Closes #1832

## Type of Change

- [x] New feature (non-breaking change that adds functionality)

## Changes Made

- Added `provider_label_from_model()` in `passthrough.py`: `deepseek*`
model ids now report provider `deepseek`.
- Wired it into the OpenAI chat handler's provider fallback:
`custom_chat_provider or provider_label_from_model(model) or "openai"`.
- Vendor routing prefixes (`deepseek/deepseek-chat`,
`github/deepseek-v4`) resolve to the underlying model; the prefix table
is ordered so adding more OpenAI-compatible providers later is a
one-line change.
- Display-only: pricing stays keyed on the model, not this label, so
cost math is unchanged.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_proxy_passthrough.py -q
tests/test_proxy_passthrough.py ....                                     [100%]
4 passed in 0.30s

$ ruff check headroom/proxy/handlers/openai.py headroom/proxy/passthrough.py tests/test_proxy_passthrough.py
All checks passed!

$ mypy headroom
Success: no issues found in 509 source files
```

## Real Behavior Proof

- Environment: local checkout, Python venv, `pytest`/`ruff`/`mypy` as
above.
- Exact command / steps: classification is unit-tested in
`tests/test_proxy_passthrough.py` — DeepSeek ids (incl. vendor prefixes
and casing) map to provider `deepseek`; OpenAI/o-series/zen ids stay
unlabelled.
- Observed result: `deepseek*` requests now surface as a distinct
`deepseek` provider in the `by_provider` / `requests_by_provider`
breakdown instead of folding into `openai`.
- Not tested: no live dashboard round against a real DeepSeek endpoint;
verified at the classifier/handler boundary.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Documentation checklist item is N/A — this is a display-only
provider-label classification with no user-facing config or docs
surface. Manual testing box left unchecked: validated at the unit
boundary rather than a live DeepSeek dashboard round.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2565"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `ecc8dc149036ca65dd9828ab17e31e0a9f35c06c`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Current-main integration preserves the contributor's model-based
DeepSeek reporting and known-upstream/xAI precedence. One handler
conflict was mechanically reconciled with main's
_resolve_openai_chat_forward_path; the contributor attribution
assignment AST is identical to its published head, and the chat-path
calls match main's AST. All other413incoming patches (414paths total)
match old-main to current-main after diff-location normalization.
Main-relative scope remains only the intended three Python files. A
local UTF-8 reconstruction corrected an initial encoding mistake before
tests/final commit; no unintended text/comment changes remain.

Fresh validation:384 passed,27 skipped,2 warnings across
provider/passthrough/integration/routing/xAI and incoming-current-main
regressions. Three contributor Python files pass Ruff lint/format;
candidate offline uv lock check and main-relative whitespace pass.
Complete tracked native inputset/everyrecordedinputhash/artifactchecksum
verified before reuse of existing3982DLL; not a fresh build. Static
assignment/patch proof is not live provider, dashboard or HTTP
execution. Previous106attribution/cardinality probe results remain
historical; no new106caseclaim. Fresh tests use synthetic/mocked
upstreams, not a live DeepSeek credential/account.

Author discussion and all current inline thread states reviewed; no
outstanding inline findings or author replies. Recognized upstream
labels and fixed unknown-custom bucket remain ahead of model fallback;
no hostname-derived telemetry label introduced. Pricing unchanged.
Source/test delta contains no workflow/setup/dependency/CLI/native path
changes, and current-main publication security gate passes. Contributor
correctness/intent unchanged. Exact published-head hosted checks and
final fresh discussion/main/mergeability guard remain before approval.
Maintainers handle CI; no PR merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-07 20:45:18 -06:00

166 lines
5.9 KiB
Python

from headroom.proxy.passthrough import (
custom_base_passthrough_telemetry,
is_opencode_zen_base,
provider_label_from_model,
)
def test_custom_base_passthrough_telemetry_recognizes_opencode_zen_chat() -> None:
assert custom_base_passthrough_telemetry(
"POST",
"/zen/v1/chat/completions",
"https://opencode.ai/",
) == ("chat/completions", "zen")
assert custom_base_passthrough_telemetry(
"POST",
"zen/v1/chat/completions",
"https://www.opencode.ai",
) == ("chat/completions", "zen")
def test_custom_base_passthrough_telemetry_ignores_non_matching_traffic() -> None:
assert custom_base_passthrough_telemetry(
"GET",
"/zen/v1/chat/completions",
"https://opencode.ai/",
) == ("", "")
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"https://opencode.ai/",
) == ("", "")
assert custom_base_passthrough_telemetry(
"POST",
"/zen/v1/chat/completions",
"https://custom.example/",
) == ("", "")
assert custom_base_passthrough_telemetry(
"POST",
"/zen/v1/chat/completions",
"://bad-url",
) == ("", "")
def test_provider_label_from_model_classifies_deepseek() -> None:
assert provider_label_from_model("deepseek-chat") == "deepseek"
assert provider_label_from_model("deepseek-reasoner") == "deepseek"
assert provider_label_from_model("DeepSeek-V4-Flash") == "deepseek"
# Vendor-prefixed routing ids resolve to the underlying model.
assert provider_label_from_model("deepseek/deepseek-chat") == "deepseek"
assert provider_label_from_model("github/deepseek-v4") == "deepseek"
def test_provider_label_from_model_leaves_openai_models_unlabeled() -> None:
for model in ("gpt-5.4-mini", "openai/gpt-4o", "o3", "zen", "", None):
assert provider_label_from_model(model) == ""
def test_is_opencode_zen_base_recognizes_zen_origins() -> None:
assert is_opencode_zen_base("https://opencode.ai")
assert is_opencode_zen_base("https://www.opencode.ai")
assert is_opencode_zen_base("https://opencode.ai/zen")
def test_is_opencode_zen_base_rejects_other_or_missing_bases() -> None:
assert not is_opencode_zen_base(None)
assert not is_opencode_zen_base("")
assert not is_opencode_zen_base("https://custom.example")
assert not is_opencode_zen_base("https://opencode.ai.evil.example")
assert not is_opencode_zen_base("://bad-url")
def test_custom_base_passthrough_telemetry_names_known_chat_hosts() -> None:
# Exact hosts only, fixed labels: the taxonomy, not the request, decides.
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"https://api.z.ai/api/coding/paas/v4",
) == ("chat/completions", "zai")
assert custom_base_passthrough_telemetry(
"POST",
"v1/chat/completions",
"https://api.meta.ai/v1",
) == ("chat/completions", "meta")
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"https://api.openai.com/v1",
) == ("chat/completions", "openai")
# Host parsing is case-insensitive; the label stays fixed.
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"https://API.Z.AI/api/coding/paas/v4",
) == ("chat/completions", "zai")
def test_custom_base_passthrough_telemetry_keeps_everything_else_unnamed() -> None:
# Lookalike hosts never match: an exact-host set cannot be talked into
# naming an attacker-controlled upstream.
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"https://api.z.ai.evil.test/v1",
) == ("", "")
# Unknown hosts, non-chat paths, non-POST methods, and bad URLs all stay
# unnamed so nothing request-controlled becomes a telemetry label.
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"https://llm.example.internal/v1",
) == ("", "")
assert custom_base_passthrough_telemetry(
"POST",
"/v1/embeddings",
"https://api.z.ai/v1",
) == ("", "")
assert custom_base_passthrough_telemetry(
"GET",
"/v1/chat/completions",
"https://api.z.ai/v1",
) == ("", "")
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"://bad-url",
) == ("", "")
def test_custom_base_passthrough_telemetry_requires_a_chat_path_segment() -> None:
# A bare suffix match would also name ``notchat/completions``.
for path in ("/v1/notchat/completions", "/v1/xchat/completions", "notchat/completions"):
assert custom_base_passthrough_telemetry("POST", path, "https://api.z.ai/v1") == ("", "")
# The exact segment, bare or nested, still names the host.
for path in ("chat/completions", "/chat/completions", "/api/paas/v4/chat/completions"):
assert custom_base_passthrough_telemetry("POST", path, "https://api.z.ai/v1") == (
"chat/completions",
"zai",
)
def test_custom_base_passthrough_telemetry_recognizes_xai_chat() -> None:
# Grok Build routes plain OpenAI chat completions through custom-base
# routing with base_url https://api.x.ai; attribute the provider as xai
# so savings rollups separate grok traffic from OpenAI's.
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"https://api.x.ai",
) == ("chat/completions", "xai")
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"https://api.x.ai.",
) == ("chat/completions", "xai")
# Other xai paths and methods stay unlabelled.
assert custom_base_passthrough_telemetry(
"GET",
"/v1/chat/completions",
"https://api.x.ai",
) == ("", "")
assert custom_base_passthrough_telemetry(
"POST",
"/v1/models",
"https://api.x.ai",
) == ("", "")