fix(telemetry): label custom-base chat upstreams from a fixed provider set (#3759)

## Description

Unrecognized custom-base chat upstreams report
`outcome_provider="openai"`.
Dashboards and per-provider stores merge every OpenAI-compatible vendor
except Zen into `openai`.

### Fix

Labels come from a fixed set in `custom_base_passthrough_telemetry()`:

| Custom base host | Label |
| --- | --- |
| `api.z.ai` | `zai` |
| `api.meta.ai` | `meta` |
| `api.openai.com` | `openai` |
| anything else | `custom` |

- **Match:** exact host, `chat/completions` path
- **Zen:** unchanged
- **No base header:** `openai`, as before
- **Bounded:** the hostname is never a label

Formatting and the SSRF guard are unchanged.
Pricing tables are unchanged; provider-dependent savings estimates can
change.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Fix:** `CUSTOM_BASE_CHAT_PROVIDERS`, exact host to label
- **Fix:** `CUSTOM_BASE_PROVIDER = "custom"` fallback
- **Fix:** chat handler reports the fixed-set label
- **Removed:** `_custom_base_provider_label()`
- **Test:** known hosts, subpath bases, host case
- **Test:** lookalike hosts, non-chat paths, bad URLs
- **Test:** `zai`, `meta`, `custom`, no-header control
- **Test:** 1,000-request flood across every sink

### Flood regression

Through the real chat handler:

1. 980 distinct hosts: ok, upstream 502, local 429, upstream 429
2. 20 `api.z.ai` requests: ok and local 429
3. Exact counts: in-memory, lifetime, OTel
4. `/metrics` labels: inside `{openai, zen, zai, meta, custom}`

The pre-seeded zero `unknown` failure bucket is the one extra label, on
`main` too.

## Testing

- [x] Unit tests pass (`pytest`): no new failures; see baseline below
- [x] Linting passes (`ruff check .`), plus `ruff format --check .`
- [x] Type checking passes (`mypy headroom`):
`--ignore-missing-imports`, mypy 1.20.2 as in CI
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
new tests against main c0292984 code:
FAILED tests/test_proxy_openai_cache_stability.py::test_openai_chat_custom_base_reports_zai_as_outcome_provider
FAILED tests/test_proxy_openai_cache_stability.py::test_openai_chat_outcome_provider_uses_the_fixed_taxonomy
FAILED tests/test_proxy_openai_cache_stability.py::test_openai_chat_custom_base_flood_cannot_grow_the_provider_set
FAILED tests/test_proxy_passthrough.py::test_custom_base_passthrough_telemetry_names_known_chat_hosts
FAILED tests/test_proxy_passthrough.py::test_custom_base_passthrough_telemetry_requires_a_chat_path_segment
E   AssertionError: assert {'openai': 260} == {'zai': 10, 'custom': 250}
5 failed, 13 passed

branch: 18 passed

pytest, full suite:
  main c0292984   11 failed, 12975 passed, 580 skipped
  branch          11 failed, 12981 passed, 580 skipped
same 11 fail on both:
  tests/test_proxy_health.py (4 readyz Kompress tests)
  tests/test_read_maturation_handler_nobust.py (3)
  tests/test_upstream_guard.py::test_no_route_forwards_to_a_loopback_upstream
  tests/test_cli_proxy_improvements.py::TestMissingProxyDepsError (2)
  tests/test_image_compression.py::TestOnnxRouter::test_full_classify_with_image

ruff check .: All checks passed!
ruff format --check .: 1634 files already formatted
mypy 1.20.2 headroom --ignore-missing-imports: Success: no issues found in 539 source files
```

## Real Behavior Proof

- Environment: Linux, Python 3.12.3, model `glm-5`
- Exact command / steps: see Commands below
- Observed result: see table
- Not tested: live OTel collector; `api.meta.ai` live

### Setup

1. **Trees:** `main` c0292984, this branch
2. **Proxy:** one per tree, own workspace
3. **Allowlist:** every host in `HEADROOM_ALLOWED_BASE_URLS`

The SSRF guard drops a base header whose host does not resolve and falls
back to `openai`.
The flood hosts do not resolve, so without the allowlist they would
never reach the label code.

### Commands

Per tree, from its checkout:

```bash
hosts="api.z.ai,$UPSTREAM_HOST"
for i in $(seq -f %04g 1 100); do hosts="$hosts,flood-$i.label-flood.test"; done
HEADROOM_ALLOWED_BASE_URLS="$hosts" HEADROOM_WORKSPACE_DIR=ws PYTHONPATH=. \
  headroom proxy --port 8787 --log-file ws/requests.jsonl
```

Traffic:

```bash
chat() {  # chat [base-url] [key] [content]
  curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8787/v1/chat/completions \
    -H 'content-type: application/json' \
    ${1:+-H "x-headroom-base-url: $1"} ${2:+-H "authorization: Bearer $2"} \
    -d '{"model":"glm-5","messages":[{"role":"user","content":"'"${3:-provider label check}"'"}]}'
}
chat https://api.z.ai/api/coding/paas/v4        # 401, no key
chat "$UPSTREAM_BASE_URL" "$UPSTREAM_KEY"       # 200, OpenAI-compatible
chat "" sk-control                              # 200, Headroom response-cache hit
for i in $(seq -f %04g 1 100); do chat "https://flood-$i.label-flood.test/v1" sk-flood flood; done  # 502
curl -s http://127.0.0.1:8787/stats | jq '.requests | {by_provider, failed_by_provider}'
curl -s http://127.0.0.1:8787/metrics | grep -o 'provider="[a-z]*"' | sort | uniq -c
```

### Result

| | `main` c0292984 | this branch |
| --- | --- | --- |
| `by_provider` | `openai` 2 | `custom` 1, `openai` 1 |
| `failed_by_provider` | `openai` 101, `unknown` 0 | `custom` 100, `zai`
1, `unknown` 0 |
| `/metrics` provider series | 13 `openai`, 1 `unknown` | 13 `custom`, 1
`zai`, 1 `openai`, 1 `unknown` |

On `main`, z.ai, the real upstream and all 100 flood hosts count as
`openai`.
On this branch, z.ai stays `zai` and the flood lands in one `custom`
bucket.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, telemetry only. Custom-base chat
reports `zai`, `meta`, `openai` or `custom` instead of always `openai`.
Routing, auth, pricing tables and the SSRF guard are unchanged;
provider-dependent savings estimates can change.
- Kill switch / disable path: None dedicated.
- Unsafe override required: No.
- Qualification impact: None. Dashboards keyed on `openai` will see
`zai`, `meta` and `custom` series.
- Rollback path: Revert.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (no new
failures)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

**main**


https://github.com/user-attachments/assets/3a0790f0-36a2-4906-ab3b-049273547fa5

**this branch**


https://github.com/user-attachments/assets/c4bc678c-4c59-47d2-94b2-2e942a9e061c

## Additional Notes

- **Closes:** none; no issue tracks this
- **Related:** #2626, same convention for `xai`
- **Docs:** unchecked; no documented behavior
- **Trade-off:** proxied z.ai reports `custom`
- **Baseline:** 11 fail on `main` too; listed above

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
donjor
2026-10-01 12:17:49 -05:00
committed by GitHub
co-authored by JerrettDavis
parent a6d6c14cf7
commit f0ec2bb376
4 changed files with 327 additions and 6 deletions
+7 -2
View File
@@ -93,10 +93,10 @@ from headroom.proxy.handlers._debug_dump import _debug_dump_mode, _redact_debug_
from headroom.proxy.image_isolation import run_image_compression_isolated
from headroom.proxy.outcome import RequestOutcome
from headroom.proxy.output_shaper import shaper_enabled_for, steering_allowed_for
from headroom.proxy.passthrough import CUSTOM_BASE_PROVIDER, is_opencode_zen_base
from headroom.proxy.passthrough import (
custom_base_passthrough_telemetry as _custom_base_passthrough_telemetry,
)
from headroom.proxy.passthrough import is_opencode_zen_base
from headroom.proxy.project_context import (
classify_project,
get_current_project,
@@ -3642,7 +3642,12 @@ class OpenAIHandlerMixin:
handler_path,
custom_upstream_base_url or "",
)
openai_chat_outcome_provider = custom_chat_provider or "openai"
# Fixed taxonomy from the shared helper (zen, zai, meta, openai); any
# other custom base is the shared "custom" bucket. Never derive the
# label from the request-controlled hostname — see the review on #3759.
openai_chat_outcome_provider = custom_chat_provider or (
CUSTOM_BASE_PROVIDER if upstream_base_url else "openai"
)
# Memory: Get user ID when memory is enabled. Reads `request.headers`
# directly because `headers` was stripped of `x-headroom-*` for the
+27 -4
View File
@@ -6,6 +6,19 @@ from urllib.parse import urlparse
OPENCODE_ZEN_HOSTS = {"opencode.ai", "www.opencode.ai"}
# Chat upstreams we can name, keyed by exact host. Bounded by design: the
# request-controlled ``x-headroom-base-url`` must never become a telemetry
# label, or a client sending a fresh hostname per request grows the
# per-provider stores and exported series without limit (review on #3759).
CUSTOM_BASE_CHAT_PROVIDERS = {
"api.openai.com": "openai",
"api.z.ai": "zai",
"api.meta.ai": "meta",
}
# Fixed label for chat traffic on any other custom base.
CUSTOM_BASE_PROVIDER = "custom"
def is_opencode_zen_base(base_url: str | None) -> bool:
"""Return True when ``base_url`` targets the OpenCode Zen gateway.
@@ -35,9 +48,19 @@ def custom_base_passthrough_telemetry(method: str, path: str, base_url: str) ->
host = (urlparse(base_url.strip()).hostname or "").lower()
except ValueError:
return "", ""
if host not in OPENCODE_ZEN_HOSTS:
return "", ""
normalized_path = path[1:] if path.startswith("/") else path
if normalized_path == "zen/v1/chat/completions":
return "chat/completions", "zen"
if host in OPENCODE_ZEN_HOSTS:
if normalized_path == "zen/v1/chat/completions":
return "chat/completions", "zen"
return "", ""
# Known OpenAI-compatible chat hosts get a fixed name; chat-completions
# traffic on any other custom base is the shared "custom" bucket and
# everything else stays unnamed so tool traffic is not LLM telemetry.
# Match whole path segments so ``/v1/notchat/completions`` stays unnamed.
provider = CUSTOM_BASE_CHAT_PROVIDERS.get(host)
is_chat_path = normalized_path == "chat/completions" or normalized_path.endswith(
"/chat/completions"
)
if provider is not None and is_chat_path:
return "chat/completions", provider
return "", ""
+224
View File
@@ -4,6 +4,8 @@ from __future__ import annotations
import copy
import json
import pathlib
import re
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock, patch
@@ -13,9 +15,13 @@ import pytest
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import InMemoryMetricReader
from headroom.backends.base import BackendResponse
from headroom.observability import HeadroomOtelMetrics, reset_otel_metrics, set_otel_metrics
from headroom.proxy.server import ProxyConfig, create_app
from tests.test_observability_metrics import _collect_metrics
class _FakePrefixTracker:
@@ -1177,3 +1183,221 @@ def test_openai_cache_mode_forwards_last_turns_bytes_unchanged(stream: bool) ->
assert sent[0][2]["content"] == "[compressed call_1]"
assert sent[1][: len(sent[0])] == sent[0]
assert sent[2][: len(sent[1])] == sent[1]
def test_openai_chat_custom_base_reports_zai_as_outcome_provider(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""An OpenAI-compatible ``x-headroom-base-url`` upstream is not ``openai``."""
# Allowlist the host so the SSRF guard does not need live DNS.
monkeypatch.setenv("HEADROOM_ALLOWED_BASE_URLS", "api.z.ai")
captured = {}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
async def _fake_stream_response(*args, **kwargs): # noqa: ANN002, ANN003
captured.update(kwargs)
return httpx.Response(200, text="data: [DONE]\n\n")
proxy._stream_response = _fake_stream_response
client.post(
"/v1/chat/completions",
headers={
"authorization": "Bearer test-key",
"x-headroom-base-url": "https://api.z.ai/api/coding/paas/v4",
},
json={
"model": "glm-5",
"stream": True,
"messages": [{"role": "user", "content": "hi"}],
},
)
assert captured.get("outcome_provider") == "zai"
def test_openai_chat_outcome_provider_uses_the_fixed_taxonomy(
monkeypatch: pytest.MonkeyPatch, tmp_path: pathlib.Path
) -> None:
"""Buffered chat labels each custom base from the fixed set, never the host."""
# Keep the SSRF guard out of the way: these hosts are deliberately
# unresolvable, and a DNS miss makes the handler ignore the override.
monkeypatch.setattr("headroom.proxy.handlers.openai.is_safe_upstream_url", lambda url: True)
# Keep the lifetime savings store out of the user's real on-disk state.
monkeypatch.setenv("HEADROOM_SAVINGS_PATH", str(tmp_path / "savings.json"))
with _make_proxy_client() as client:
proxy = client.app.state.proxy
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001, ANN002, ANN003
return httpx.Response(
200,
json={
"id": "chatcmpl_1",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "ok"},
"finish_reason": "stop",
}
],
"usage": {"prompt_tokens": 10, "completion_tokens": 1, "total_tokens": 11},
},
)
proxy._retry_request = _fake_retry
def _chat(base: str | None) -> None:
headers = {"authorization": "Bearer test-key"}
if base is not None:
headers["x-headroom-base-url"] = base
response = client.post(
"/v1/chat/completions",
headers=headers,
json={"model": "glm-5", "messages": [{"role": "user", "content": "hi"}]},
)
assert response.status_code == 200
_chat("https://api.meta.ai/v1")
_chat("https://api.meta.ai/v1/sub")
_chat("https://llm.example.internal/v1")
# No header: the plain OpenAI path keeps its own label (control).
_chat(None)
by_provider = proxy.metrics.requests_by_provider
assert by_provider["meta"] == 2
assert by_provider["custom"] == 1
assert by_provider["openai"] == 1
def test_openai_chat_custom_base_flood_cannot_grow_the_provider_set(
monkeypatch: pytest.MonkeyPatch, tmp_path: pathlib.Path
) -> None:
"""Regression for the #3759 review: a host flood stays inside the fixed set.
The old code derived the outcome provider from the request-controlled
``x-headroom-base-url`` hostname, so one fresh hostname per request grew
the provider stores and exported series without bound. 980 distinct flood
hosts plus 20 z.ai requests cover success, upstream failure, local 429 and
upstream 429, and every sink must count exactly ``zai`` and ``custom``.
"""
fixed_providers = {"openai", "zen", "zai", "meta", "custom"}
zai_base = "https://api.z.ai/api/coding/paas/v4"
limited_auth = "Bearer rate-limited"
# The flood hosts are unresolvable on purpose; without this stub the SSRF
# guard would drop the override and relabel the flood as "openai".
monkeypatch.setattr("headroom.proxy.handlers.openai.is_safe_upstream_url", lambda url: True)
# Isolated sinks: a fresh lifetime store and workspace, so the exact
# counts below are this test's traffic and nothing reaches ~/.headroom.
monkeypatch.setenv("HEADROOM_SAVINGS_PATH", str(tmp_path / "savings.json"))
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path / "workspace"))
reader = InMemoryMetricReader()
set_otel_metrics(HeadroomOtelMetrics(meter_provider=MeterProvider(metric_readers=[reader])))
class _LimitOneKey:
"""Headroom's own limiter, refusing only ``limited_auth``."""
async def check_request(self, key: str = "default") -> tuple[bool, float]:
return key != limited_auth, 1.0
async def check_tokens(self, key: str, tokens: int) -> tuple[bool, float]:
return True, 0.0
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001, ANN002, ANN003
if "flood-down-" in url:
raise httpx.ConnectError("flood host down")
if "flood-upstream429-" in url:
return httpx.Response(429, json={"error": {"message": "slow down"}})
return httpx.Response(
200,
json={
"id": "chatcmpl_flood",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "ok"},
"finish_reason": "stop",
}
],
"usage": {"prompt_tokens": 5, "completion_tokens": 1, "total_tokens": 6},
},
)
expected_status = {"ok": 200, "down": 502, "limited": 429, "upstream429": 429}
sent: dict[tuple[str, str], int] = {}
flood_hosts: set[str] = set()
try:
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy._retry_request = _fake_retry
proxy.rate_limiter = _LimitOneKey()
for i in range(1000):
if i % 50 == 0:
provider, kind = "zai", ("ok" if i % 100 == 0 else "limited")
base = zai_base
else:
provider, kind = "custom", ("limited", "upstream429", "down", "ok")[i % 4]
host = f"flood-{kind}-{i}.example.invalid"
flood_hosts.add(host)
base = f"https://{host}/v1"
sent[provider, kind] = sent.get((provider, kind), 0) + 1
auth = limited_auth if kind == "limited" else "Bearer test-key"
response = client.post(
"/v1/chat/completions",
headers={"authorization": auth, "x-headroom-base-url": base},
json={"model": "glm-5", "messages": [{"role": "user", "content": f"turn {i}"}]},
)
assert response.status_code == expected_status[kind], (i, base)
# The traffic this test sends, pinned so the claim stays exact.
assert len(flood_hosts) == 980
assert sent == {
("zai", "ok"): 10,
("zai", "limited"): 10,
("custom", "ok"): 250,
("custom", "down"): 240,
("custom", "limited"): 240,
("custom", "upstream429"): 250,
}
metrics = proxy.metrics
# In-memory stores: exact counts; "unknown" is the seeded zero bucket.
assert dict(metrics.requests_by_provider) == {"zai": 10, "custom": 250}
assert dict(metrics.requests_failed_by_provider) == {"unknown": 0, "custom": 240}
assert metrics.requests_rate_limited_by_source == {"headroom": 250, "upstream": 250}
# Lifetime store, isolated under tmp_path.
lifetime = metrics.savings_tracker.lifetime_response()["requests"]
assert lifetime["by_provider"] == {"zai": 10, "custom": 250}
assert lifetime["failed_by_provider"] == {"custom": 240}
assert lifetime["rate_limited_by_source"] == {"headroom": 250, "upstream": 250}
# Prometheus export: every provider series is from the fixed set.
export = client.get("/metrics").text
assert set(re.findall(r'provider="([^"]+)"', export)) <= fixed_providers | {"unknown"}
assert 'headroom_requests_by_provider{provider="zai"} 10' in export
assert 'headroom_requests_by_provider{provider="custom"} 250' in export
# OTel: every proxy data point carries a fixed provider attribute, and
# z.ai stays its own series on the success and local-429 paths.
otel = _collect_metrics(reader)
otel_points: dict[tuple[str, str, str], float] = {}
for name, metric in otel.items():
if not name.startswith("headroom.proxy."):
continue
for point in metric.data.data_points:
label = point.attributes.get("provider")
assert label in fixed_providers, (name, label)
if name.startswith("headroom.proxy.requests"):
key = (name, label, point.attributes.get("source", ""))
otel_points[key] = otel_points.get(key, 0) + point.value
assert otel_points == {
("headroom.proxy.requests", "zai", ""): 10,
("headroom.proxy.requests", "custom", ""): 250,
("headroom.proxy.requests.failed", "custom", ""): 240,
("headroom.proxy.requests.rate_limited", "zai", "headroom"): 10,
("headroom.proxy.requests.rate_limited", "custom", "headroom"): 240,
("headroom.proxy.requests.rate_limited", "custom", "upstream"): 250,
}
finally:
reset_otel_metrics()
+69
View File
@@ -49,3 +49,72 @@ def test_is_opencode_zen_base_rejects_other_or_missing_bases() -> None:
assert not is_opencode_zen_base("https://custom.example")
assert not is_opencode_zen_base("https://opencode.ai.evil.example")
assert not is_opencode_zen_base("://bad-url")
def test_custom_base_passthrough_telemetry_names_known_chat_hosts() -> None:
# Exact hosts only, fixed labels: the taxonomy, not the request, decides.
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"https://api.z.ai/api/coding/paas/v4",
) == ("chat/completions", "zai")
assert custom_base_passthrough_telemetry(
"POST",
"v1/chat/completions",
"https://api.meta.ai/v1",
) == ("chat/completions", "meta")
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"https://api.openai.com/v1",
) == ("chat/completions", "openai")
# Host parsing is case-insensitive; the label stays fixed.
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"https://API.Z.AI/api/coding/paas/v4",
) == ("chat/completions", "zai")
def test_custom_base_passthrough_telemetry_keeps_everything_else_unnamed() -> None:
# Lookalike hosts never match: an exact-host set cannot be talked into
# naming an attacker-controlled upstream.
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"https://api.z.ai.evil.test/v1",
) == ("", "")
# Unknown hosts, non-chat paths, non-POST methods, and bad URLs all stay
# unnamed so nothing request-controlled becomes a telemetry label.
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"https://llm.example.internal/v1",
) == ("", "")
assert custom_base_passthrough_telemetry(
"POST",
"/v1/embeddings",
"https://api.z.ai/v1",
) == ("", "")
assert custom_base_passthrough_telemetry(
"GET",
"/v1/chat/completions",
"https://api.z.ai/v1",
) == ("", "")
assert custom_base_passthrough_telemetry(
"POST",
"/v1/chat/completions",
"://bad-url",
) == ("", "")
def test_custom_base_passthrough_telemetry_requires_a_chat_path_segment() -> None:
# A bare suffix match would also name ``notchat/completions``.
for path in ("/v1/notchat/completions", "/v1/xchat/completions", "notchat/completions"):
assert custom_base_passthrough_telemetry("POST", path, "https://api.z.ai/v1") == ("", "")
# The exact segment, bare or nested, still names the host.
for path in ("chat/completions", "/chat/completions", "/api/paas/v4/chat/completions"):
assert custom_base_passthrough_telemetry("POST", path, "https://api.z.ai/v1") == (
"chat/completions",
"zai",
)