mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 16:45:34 +08:00
## Description
This change isolates TOIN learning by tenant in shared-proxy
deployments. Each pattern key gains a tenant namespace derived from a
configurable sanitized header, a stable auth/token hash, or the
backwards-compatible global namespace. Legacy keys migrate to global and
every resolution path emits a structured event.
## Threat model
Pre-F3, every request's tool-call patterns aggregated into one global
pool keyed by `(auth_mode, model_family, sig_hash)`. That works for an
OSS single-tenant deploy, but in a multi-tenant SaaS / shared-proxy
deploy:
1. **Cross-tenant pattern leakage.** Tenant A's compression patterns
silently train recommendations the proxy then applies to tenant B's
requests, degrading both tenants' compression quality.
2. **Structural fingerprint exposure.** TOIN dumps include
`tool_signature_hash` per slice. With one shared pool, a user with read
access to a dump file learns the structural fingerprint of every other
tenant's tool surface.
F3 partitions: every TOIN store key gets a `tenant_key` prefix derived
from the inbound request. Two distinct tenants therefore train two
isolated pattern pools.
## Tenant key derivation rules
| Source | When | Key derivation |
|------------|--------------------------------------------------------------|-------------------------------------------------------------------|
| `header` | `HEADROOM_TENANT_KEY_HEADER` (default
`X-Headroom-Tenant-ID`) is present and survives sanitization | Sanitized
header value (ASCII alphanum + `-_`, max 64 chars) |
| `hash` | No header but bearer token + `auth_mode` (from F1) present |
`sha256(f"{auth_mode}:{api_key_prefix}")[:24]` (api_key_prefix = first 8
chars of bearer); same SHA-256[:24] idiom as PR #395 |
| `global` | Neither header nor bearer present (OSS single-tenant / dev)
| Literal `"global"` namespace; pre-F3 patterns migrate here on first
load |
Header path takes precedence over hash. Header values that sanitize to
empty (pure unicode / control chars only) fall through to the next rule.
## Configuration
Single env var: `HEADROOM_TENANT_KEY_HEADER`. Default
`X-Headroom-Tenant-ID`. The header NAME is configurable; the
sanitization rules and the `"global"` literal are NOT — they're
load-bearing for namespace isolation.
## Structured log fields
Every resolution path emits exactly one `tenant_key_resolved` log event
(no silent fallbacks per `feedback_no_silent_fallbacks.md`):
| Field | Always present | Notes |
|--------------|----------------|------------------------------------------------|
| `event` | yes | Always `"tenant_key_resolved"` |
| `tenant_key` | yes | Resolved key (or `"global"`) |
| `source` | yes | One of `header` / `hash` / `global` |
| `header` | header path | Header name that matched |
| `auth_mode` | hash path | `payg` / `oauth` / `subscription` |
Operators can alert on a sustained `source="global"` rate to spot
mis-configured upstream auth proxies.
## Implementation
- **NEW** `headroom/proxy/tenant_key.py`: pure resolver. Public surface
= `resolve_tenant_key(request) -> tuple[str, str]`. Plus
`set_request_tenant_key()` / `get_current_tenant_key()` ContextVar pair
so TOIN's deep-stack `record_compression` calls observe the right tenant
slice.
- `headroom/telemetry/toin.py`: extends `PatternKey` from a 3-tuple to a
4-tuple `(tenant_key, auth_mode, model_family, sig_hash)`.
`_deserialize_pattern_key` is backward-compat with both pre-F3 formats
(3-tuple from PR-B5, bare `sig_hash` from v1.0). `record_compression` /
`record_retrieval` / `get_pattern` accept an optional `tenant_key`
kwarg; when absent they read from the ContextVar.
- `headroom/cli/toin_publish.py`: each `[[recommendation]]` row now
carries `tenant_key`; rows sort by `(tenant_key, auth_mode,
model_family, sig_hash)` for clean diffs.
- `headroom/proxy/handlers/anthropic.py` +
`headroom/proxy/handlers/openai.py`: call `resolve_tenant_key(request)`
once at request entry, right after the F1 `classify_auth_mode` block.
Three call sites total (1 Anthropic, 2 OpenAI). Result attached to
`request.state.tenant_key` / `request.state.tenant_key_source` AND set
on the ContextVar.
## Realignment build constraints (per project memory)
- **configurable**: `HEADROOM_TENANT_KEY_HEADER` env var.
- **no hardcodes**: header name + `"global"` literal are the only
constants; both motivated and documented in code.
- **no regexes**: sanitization uses string ops +
`unicodedata.category()`.
- **no silent fallbacks**: every resolution emits a structured log
event.
- **comprehensive tests**: header / hash / global paths + sanitization
edges + tenant-isolation property test.
- **structured logs**: `event=tenant_key_resolved` with `tenant_key`,
`source`, optional `header` / `auth_mode`.
## Test plan
- [x] `pytest tests/test_tenant_key.py` (29 cases — header / hash /
global paths, structured-log emission, sanitization edges, ContextVar
round-trip)
- [x] `pytest tests/test_toin_observation_only.py` (6 new isolation
cases — two-tenant divergence, ContextVar read, explicit kwarg
precedence, default migration to `"global"`)
- [x] `pytest tests/test_toin*.py` (existing TOIN tests updated for the
4-tuple key shape)
- [x] `pytest tests/test_proxy_tenant_key_integration.py` (4 cases —
handler wires `request.state.tenant_key` + ContextVar through all three
resolution paths)
- [x] `pytest tests/test_critical_fixes.py tests/test_critical_gaps.py
tests/test_proxy_ccr.py` (legacy 3-tuple references updated)
- [x] `make ci-precheck-python` — 176 tests pass (passed via pre-push
hook)
- [x] `make ci-precheck-commitlint` — clean (passed via pre-push hook)
Refs: F1 (PR #399, auth_mode classifier), F2.1 (PR #400, per-auth-mode
CompressionPolicy gates).
## Type of Change
- [x] Bug fix / tenant-isolation hardening
- [ ] New feature
- [ ] Breaking change
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring
## Changes Made
- Add a pure tenant-key resolver and request ContextVar.
- Extend TOIN pattern keys and published recommendations with a tenant
namespace while preserving legacy deserialization.
- Resolve and attach tenant identity at Anthropic and OpenAI request
entry.
- Add isolation, migration, sanitization, header, CCR, handler, and
executor coverage.
## Testing
- [x] Focused tenant/TOIN/proxy tests pass
- [x] Python CI precheck passes
- [x] Commitlint precheck passes
### Test Output
```text
176 focused/precheck tests passed
Current-main isolation, migration, header, CCR, and executor validation: green
```
## Real Behavior Proof
- Environment: current-main reconciled Python proxy/TOIN test
environment.
- Exact command / steps: ran tenant resolver, TOIN observation/publish,
handler integration, header-isolation, CCR, and compression-executor
suites.
- Observed result: distinct tenants train isolated pattern pools; legacy
patterns migrate to global; request handlers propagate the resolved
namespace through deep TOIN calls.
- Not tested: production multi-tenant traffic volume or an external TOIN
dump consumer.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
---------
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
281 lines
11 KiB
Python
281 lines
11 KiB
Python
"""Unit tests for ``headroom.proxy.tenant_key.resolve_tenant_key``.
|
|
|
|
Phase F PR-F3 — covers the two resolution paths (header / global), the structured-log requirement, and the sanitization edge
|
|
cases (empty, too-long, unicode, control chars, hyphens, mixed
|
|
ASCII).
|
|
|
|
The handler-integration tests (``request.state`` plumbing) live with
|
|
the proxy handler tests; this file pins the pure resolver contract.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from types import SimpleNamespace
|
|
|
|
import pytest
|
|
|
|
from headroom.proxy.tenant_key import (
|
|
DEFAULT_TENANT_KEY_HEADER,
|
|
GLOBAL_TENANT_KEY,
|
|
SOURCE_GLOBAL,
|
|
SOURCE_HEADER,
|
|
TENANT_KEY_HEADER_ENV_VAR,
|
|
get_current_tenant_key,
|
|
resolve_tenant_key,
|
|
set_request_tenant_key,
|
|
)
|
|
|
|
|
|
def _request(
|
|
headers: dict[str, str] | None = None, auth_mode: str | None = None
|
|
) -> SimpleNamespace:
|
|
"""Build a minimal Starlette-shaped Request fixture.
|
|
|
|
SimpleNamespace + a plain dict for `headers` is sufficient because
|
|
``resolve_tenant_key`` only touches ``request.headers.get`` /
|
|
``.items`` and ``request.state.auth_mode``.
|
|
"""
|
|
state = SimpleNamespace(auth_mode=auth_mode)
|
|
return SimpleNamespace(headers=headers or {}, state=state)
|
|
|
|
|
|
# ── Header path ───────────────────────────────────────────────────────
|
|
|
|
|
|
def test_header_path_returns_header_source() -> None:
|
|
"""Header present + sanitized non-empty ⇒ header source."""
|
|
req = _request(headers={"X-Headroom-Tenant-ID": "cust_abc123"})
|
|
tenant_key, source = resolve_tenant_key(req)
|
|
assert tenant_key == "cust_abc123"
|
|
assert source == SOURCE_HEADER
|
|
|
|
|
|
def test_header_path_lowercase_header_lookup() -> None:
|
|
"""Header lookup is case-insensitive (Starlette norm)."""
|
|
req = _request(headers={"x-headroom-tenant-id": "tenant42"})
|
|
tenant_key, source = resolve_tenant_key(req)
|
|
assert tenant_key == "tenant42"
|
|
assert source == SOURCE_HEADER
|
|
|
|
|
|
def test_header_path_custom_header_via_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""``HEADROOM_TENANT_KEY_HEADER`` overrides the default name."""
|
|
monkeypatch.setenv(TENANT_KEY_HEADER_ENV_VAR, "X-Custom-Tenant")
|
|
req = _request(headers={"X-Custom-Tenant": "company9"})
|
|
tenant_key, source = resolve_tenant_key(req)
|
|
assert tenant_key == "company9"
|
|
assert source == SOURCE_HEADER
|
|
|
|
|
|
def test_header_path_default_name_is_x_headroom_tenant_id() -> None:
|
|
"""Default header name is documented as ``X-Headroom-Tenant-ID``."""
|
|
assert DEFAULT_TENANT_KEY_HEADER == "X-Headroom-Tenant-ID"
|
|
|
|
|
|
def test_header_path_falls_through_when_empty_after_sanitization() -> None:
|
|
"""Pure-non-allowed header (e.g. all unicode) ⇒ fall through, NOT header source."""
|
|
# All-emoji header sanitizes to empty string — neither header
|
|
# so we drop to the global fallback.
|
|
req = _request(headers={"X-Headroom-Tenant-ID": "🦀🚀"})
|
|
tenant_key, source = resolve_tenant_key(req)
|
|
assert source == SOURCE_GLOBAL
|
|
assert tenant_key == GLOBAL_TENANT_KEY
|
|
|
|
|
|
# ── Hash path ─────────────────────────────────────────────────────────
|
|
|
|
|
|
def test_bearer_without_header_uses_global_namespace() -> None:
|
|
req = _request(
|
|
headers={"authorization": "Bearer sk-ant-api03-abc123def"},
|
|
auth_mode="payg",
|
|
)
|
|
|
|
tenant_key, source = resolve_tenant_key(req)
|
|
|
|
assert tenant_key == GLOBAL_TENANT_KEY
|
|
assert source == SOURCE_GLOBAL
|
|
|
|
|
|
def test_auth_mode_without_header_does_not_partition_tenants() -> None:
|
|
bearer = "Bearer same-tok-aaabbbccc"
|
|
|
|
payg = resolve_tenant_key(_request(headers={"authorization": bearer}, auth_mode="payg"))
|
|
oauth = resolve_tenant_key(_request(headers={"authorization": bearer}, auth_mode="oauth"))
|
|
|
|
assert payg == (GLOBAL_TENANT_KEY, SOURCE_GLOBAL)
|
|
assert oauth == (GLOBAL_TENANT_KEY, SOURCE_GLOBAL)
|
|
|
|
|
|
def test_global_fallback_when_no_signals() -> None:
|
|
"""No header, no auth, no key ⇒ literal ``"global"``."""
|
|
req = _request()
|
|
tenant_key, source = resolve_tenant_key(req)
|
|
assert tenant_key == GLOBAL_TENANT_KEY == "global"
|
|
assert source == SOURCE_GLOBAL
|
|
|
|
|
|
def _capture_tenant_resolution_logs(monkeypatch: pytest.MonkeyPatch) -> list[SimpleNamespace]:
|
|
records: list[SimpleNamespace] = []
|
|
|
|
def _info(_message: str, *, extra: dict[str, object] | None = None) -> None:
|
|
records.append(SimpleNamespace(**(extra or {})))
|
|
|
|
monkeypatch.setattr("headroom.proxy.tenant_key.logger.info", _info)
|
|
return records
|
|
|
|
|
|
def test_global_fallback_emits_structured_log(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""The ``"global"`` source MUST log (no silent fallback)."""
|
|
req = _request()
|
|
matching = _capture_tenant_resolution_logs(monkeypatch)
|
|
resolve_tenant_key(req)
|
|
|
|
assert len(matching) == 1
|
|
assert matching[0].source == "global" # type: ignore[attr-defined]
|
|
assert matching[0].tenant_key == "global" # type: ignore[attr-defined]
|
|
|
|
|
|
def test_header_source_emits_structured_log(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Header path also emits the structured log."""
|
|
req = _request(headers={"X-Headroom-Tenant-ID": "tenant1"})
|
|
matching = _capture_tenant_resolution_logs(monkeypatch)
|
|
resolve_tenant_key(req)
|
|
assert len(matching) == 1
|
|
assert matching[0].source == "header" # type: ignore[attr-defined]
|
|
|
|
|
|
def test_bearer_without_header_emits_global_structured_log(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Bearer auth without a tenant header still emits the global log."""
|
|
req = _request(
|
|
headers={"authorization": "Bearer sk-ant-api03-abc123def"},
|
|
auth_mode="payg",
|
|
)
|
|
matching = _capture_tenant_resolution_logs(monkeypatch)
|
|
resolve_tenant_key(req)
|
|
assert len(matching) == 1
|
|
assert matching[0].source == "global" # type: ignore[attr-defined]
|
|
|
|
|
|
# ── Sanitization edges ────────────────────────────────────────────────
|
|
|
|
|
|
def test_sanitization_allows_alphanumeric() -> None:
|
|
"""ASCII alphanumerics survive untouched."""
|
|
req = _request(headers={"X-Headroom-Tenant-ID": "Cust42TenantA"})
|
|
tenant_key, _ = resolve_tenant_key(req)
|
|
assert tenant_key == "Cust42TenantA"
|
|
|
|
|
|
def test_sanitization_allows_hyphens_and_underscores() -> None:
|
|
"""Hyphens and underscores survive (common in SaaS tenant IDs)."""
|
|
req = _request(headers={"X-Headroom-Tenant-ID": "cust_a-b_c-9"})
|
|
tenant_key, _ = resolve_tenant_key(req)
|
|
assert tenant_key == "cust_a-b_c-9"
|
|
|
|
|
|
def test_sanitization_drops_control_chars() -> None:
|
|
"""Control chars (``\\x00`` / ``\\x1f``) get stripped."""
|
|
req = _request(headers={"X-Headroom-Tenant-ID": "ten\x00ant\x1f1"})
|
|
tenant_key, source = resolve_tenant_key(req)
|
|
assert tenant_key == "tenant1"
|
|
assert source == SOURCE_HEADER
|
|
|
|
|
|
def test_sanitization_drops_tab_and_newline() -> None:
|
|
"""Whitespace (``\\t`` / ``\\n``) gets stripped."""
|
|
req = _request(headers={"X-Headroom-Tenant-ID": "tenant\t\n1"})
|
|
tenant_key, source = resolve_tenant_key(req)
|
|
assert tenant_key == "tenant1"
|
|
assert source == SOURCE_HEADER
|
|
|
|
|
|
def test_sanitization_drops_punctuation() -> None:
|
|
"""`|`, `:`, `.`, `,`, `/` stripped (TOIN store-key separator safety)."""
|
|
req = _request(headers={"X-Headroom-Tenant-ID": "ten|ant:1.0/x"})
|
|
tenant_key, source = resolve_tenant_key(req)
|
|
assert tenant_key == "tenant10x"
|
|
assert source == SOURCE_HEADER
|
|
|
|
|
|
def test_sanitization_drops_unicode_falls_through_when_pure_unicode() -> None:
|
|
"""Pure-unicode header (CJK / emoji) sanitizes to empty ⇒ falls through."""
|
|
req = _request(headers={"X-Headroom-Tenant-ID": "测试テスト"})
|
|
_, source = resolve_tenant_key(req)
|
|
# No header source — sanitized to empty, dropped to global.
|
|
assert source == SOURCE_GLOBAL
|
|
|
|
|
|
def test_sanitization_truncates_at_64_chars() -> None:
|
|
"""Headers > 64 ASCII alphanum chars are TRUNCATED, not rejected."""
|
|
raw = "a" * 100
|
|
req = _request(headers={"X-Headroom-Tenant-ID": raw})
|
|
tenant_key, source = resolve_tenant_key(req)
|
|
assert tenant_key == "a" * 64
|
|
assert len(tenant_key) == 64
|
|
assert source == SOURCE_HEADER
|
|
|
|
|
|
def test_sanitization_empty_header_falls_through() -> None:
|
|
"""Empty header value ⇒ fall through (header treated as absent)."""
|
|
req = _request(headers={"X-Headroom-Tenant-ID": ""})
|
|
_, source = resolve_tenant_key(req)
|
|
assert source == SOURCE_GLOBAL
|
|
|
|
|
|
def test_sanitization_whitespace_only_falls_through() -> None:
|
|
"""Whitespace-only header ⇒ sanitizes to empty ⇒ falls through."""
|
|
req = _request(headers={"X-Headroom-Tenant-ID": " \t\n "})
|
|
_, source = resolve_tenant_key(req)
|
|
assert source == SOURCE_GLOBAL
|
|
|
|
|
|
def test_sanitization_mixed_unicode_and_ascii_preserves_ascii() -> None:
|
|
"""``cust_我_42`` sanitizes to ``cust__42`` (ASCII chars survive)."""
|
|
req = _request(headers={"X-Headroom-Tenant-ID": "cust_我_42"})
|
|
tenant_key, source = resolve_tenant_key(req)
|
|
assert tenant_key == "cust__42"
|
|
assert source == SOURCE_HEADER
|
|
|
|
|
|
# ── ContextVar surface ────────────────────────────────────────────────
|
|
|
|
|
|
def test_get_current_tenant_key_defaults_to_global() -> None:
|
|
"""Outside a request context, ``get_current_tenant_key`` ⇒ ``"global"``."""
|
|
set_request_tenant_key(None) # Make sure we're cleared.
|
|
assert get_current_tenant_key() == GLOBAL_TENANT_KEY
|
|
|
|
|
|
def test_set_request_tenant_key_round_trips() -> None:
|
|
"""``set`` then ``get`` returns the value."""
|
|
set_request_tenant_key("tenant_abc")
|
|
try:
|
|
assert get_current_tenant_key() == "tenant_abc"
|
|
finally:
|
|
set_request_tenant_key(None)
|
|
|
|
|
|
def test_set_request_tenant_key_none_resets_to_global() -> None:
|
|
"""Clearing returns to the global default."""
|
|
set_request_tenant_key("tenant_abc")
|
|
set_request_tenant_key(None)
|
|
assert get_current_tenant_key() == GLOBAL_TENANT_KEY
|
|
|
|
|
|
# ── Header precedence over auth headers ───────────────────────────────
|
|
|
|
|
|
def test_header_takes_precedence_over_auth_header() -> None:
|
|
"""When both a tenant header and bearer auth are present, the tenant header wins."""
|
|
req = _request(
|
|
headers={
|
|
"X-Headroom-Tenant-ID": "cust_explicit",
|
|
"authorization": "Bearer sk-ant-api03-aaaaa",
|
|
},
|
|
auth_mode="payg",
|
|
)
|
|
tenant_key, source = resolve_tenant_key(req)
|
|
assert tenant_key == "cust_explicit"
|
|
assert source == SOURCE_HEADER
|