Files
headroom/tests/test_tenant_key.py
f90a56b038 fix(proxy): F3 — per-tenant TOIN learning key (#404)
## Description

This change isolates TOIN learning by tenant in shared-proxy
deployments. Each pattern key gains a tenant namespace derived from a
configurable sanitized header, a stable auth/token hash, or the
backwards-compatible global namespace. Legacy keys migrate to global and
every resolution path emits a structured event.

## Threat model

Pre-F3, every request's tool-call patterns aggregated into one global
pool keyed by `(auth_mode, model_family, sig_hash)`. That works for an
OSS single-tenant deploy, but in a multi-tenant SaaS / shared-proxy
deploy:

1. **Cross-tenant pattern leakage.** Tenant A's compression patterns
silently train recommendations the proxy then applies to tenant B's
requests, degrading both tenants' compression quality.
2. **Structural fingerprint exposure.** TOIN dumps include
`tool_signature_hash` per slice. With one shared pool, a user with read
access to a dump file learns the structural fingerprint of every other
tenant's tool surface.

F3 partitions: every TOIN store key gets a `tenant_key` prefix derived
from the inbound request. Two distinct tenants therefore train two
isolated pattern pools.

## Tenant key derivation rules

| Source | When | Key derivation |

|------------|--------------------------------------------------------------|-------------------------------------------------------------------|
| `header` | `HEADROOM_TENANT_KEY_HEADER` (default
`X-Headroom-Tenant-ID`) is present and survives sanitization | Sanitized
header value (ASCII alphanum + `-_`, max 64 chars) |
| `hash` | No header but bearer token + `auth_mode` (from F1) present |
`sha256(f"{auth_mode}:{api_key_prefix}")[:24]` (api_key_prefix = first 8
chars of bearer); same SHA-256[:24] idiom as PR #395 |
| `global` | Neither header nor bearer present (OSS single-tenant / dev)
| Literal `"global"` namespace; pre-F3 patterns migrate here on first
load |

Header path takes precedence over hash. Header values that sanitize to
empty (pure unicode / control chars only) fall through to the next rule.

## Configuration

Single env var: `HEADROOM_TENANT_KEY_HEADER`. Default
`X-Headroom-Tenant-ID`. The header NAME is configurable; the
sanitization rules and the `"global"` literal are NOT — they're
load-bearing for namespace isolation.

## Structured log fields

Every resolution path emits exactly one `tenant_key_resolved` log event
(no silent fallbacks per `feedback_no_silent_fallbacks.md`):

| Field | Always present | Notes |

|--------------|----------------|------------------------------------------------|
| `event` | yes | Always `"tenant_key_resolved"` |
| `tenant_key` | yes | Resolved key (or `"global"`) |
| `source` | yes | One of `header` / `hash` / `global` |
| `header` | header path | Header name that matched |
| `auth_mode` | hash path | `payg` / `oauth` / `subscription` |

Operators can alert on a sustained `source="global"` rate to spot
mis-configured upstream auth proxies.

## Implementation

- **NEW** `headroom/proxy/tenant_key.py`: pure resolver. Public surface
= `resolve_tenant_key(request) -> tuple[str, str]`. Plus
`set_request_tenant_key()` / `get_current_tenant_key()` ContextVar pair
so TOIN's deep-stack `record_compression` calls observe the right tenant
slice.
- `headroom/telemetry/toin.py`: extends `PatternKey` from a 3-tuple to a
4-tuple `(tenant_key, auth_mode, model_family, sig_hash)`.
`_deserialize_pattern_key` is backward-compat with both pre-F3 formats
(3-tuple from PR-B5, bare `sig_hash` from v1.0). `record_compression` /
`record_retrieval` / `get_pattern` accept an optional `tenant_key`
kwarg; when absent they read from the ContextVar.
- `headroom/cli/toin_publish.py`: each `[[recommendation]]` row now
carries `tenant_key`; rows sort by `(tenant_key, auth_mode,
model_family, sig_hash)` for clean diffs.
- `headroom/proxy/handlers/anthropic.py` +
`headroom/proxy/handlers/openai.py`: call `resolve_tenant_key(request)`
once at request entry, right after the F1 `classify_auth_mode` block.
Three call sites total (1 Anthropic, 2 OpenAI). Result attached to
`request.state.tenant_key` / `request.state.tenant_key_source` AND set
on the ContextVar.

## Realignment build constraints (per project memory)

- **configurable**: `HEADROOM_TENANT_KEY_HEADER` env var.
- **no hardcodes**: header name + `"global"` literal are the only
constants; both motivated and documented in code.
- **no regexes**: sanitization uses string ops +
`unicodedata.category()`.
- **no silent fallbacks**: every resolution emits a structured log
event.
- **comprehensive tests**: header / hash / global paths + sanitization
edges + tenant-isolation property test.
- **structured logs**: `event=tenant_key_resolved` with `tenant_key`,
`source`, optional `header` / `auth_mode`.

## Test plan

- [x] `pytest tests/test_tenant_key.py` (29 cases — header / hash /
global paths, structured-log emission, sanitization edges, ContextVar
round-trip)
- [x] `pytest tests/test_toin_observation_only.py` (6 new isolation
cases — two-tenant divergence, ContextVar read, explicit kwarg
precedence, default migration to `"global"`)
- [x] `pytest tests/test_toin*.py` (existing TOIN tests updated for the
4-tuple key shape)
- [x] `pytest tests/test_proxy_tenant_key_integration.py` (4 cases —
handler wires `request.state.tenant_key` + ContextVar through all three
resolution paths)
- [x] `pytest tests/test_critical_fixes.py tests/test_critical_gaps.py
tests/test_proxy_ccr.py` (legacy 3-tuple references updated)
- [x] `make ci-precheck-python` — 176 tests pass (passed via pre-push
hook)
- [x] `make ci-precheck-commitlint` — clean (passed via pre-push hook)

Refs: F1 (PR #399, auth_mode classifier), F2.1 (PR #400, per-auth-mode
CompressionPolicy gates).

## Type of Change

- [x] Bug fix / tenant-isolation hardening
- [ ] New feature
- [ ] Breaking change
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- Add a pure tenant-key resolver and request ContextVar.
- Extend TOIN pattern keys and published recommendations with a tenant
namespace while preserving legacy deserialization.
- Resolve and attach tenant identity at Anthropic and OpenAI request
entry.
- Add isolation, migration, sanitization, header, CCR, handler, and
executor coverage.

## Testing

- [x] Focused tenant/TOIN/proxy tests pass
- [x] Python CI precheck passes
- [x] Commitlint precheck passes

### Test Output

```text
176 focused/precheck tests passed
Current-main isolation, migration, header, CCR, and executor validation: green
```

## Real Behavior Proof

- Environment: current-main reconciled Python proxy/TOIN test
environment.
- Exact command / steps: ran tenant resolver, TOIN observation/publish,
handler integration, header-isolation, CCR, and compression-executor
suites.
- Observed result: distinct tenants train isolated pattern pools; legacy
patterns migrate to global; request handlers propagate the resolved
namespace through deep TOIN calls.
- Not tested: production multi-tenant traffic volume or an external TOIN
dump consumer.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 22:03:59 -05:00

281 lines
11 KiB
Python

"""Unit tests for ``headroom.proxy.tenant_key.resolve_tenant_key``.
Phase F PR-F3 — covers the two resolution paths (header / global), the structured-log requirement, and the sanitization edge
cases (empty, too-long, unicode, control chars, hyphens, mixed
ASCII).
The handler-integration tests (``request.state`` plumbing) live with
the proxy handler tests; this file pins the pure resolver contract.
"""
from __future__ import annotations
from types import SimpleNamespace
import pytest
from headroom.proxy.tenant_key import (
DEFAULT_TENANT_KEY_HEADER,
GLOBAL_TENANT_KEY,
SOURCE_GLOBAL,
SOURCE_HEADER,
TENANT_KEY_HEADER_ENV_VAR,
get_current_tenant_key,
resolve_tenant_key,
set_request_tenant_key,
)
def _request(
headers: dict[str, str] | None = None, auth_mode: str | None = None
) -> SimpleNamespace:
"""Build a minimal Starlette-shaped Request fixture.
SimpleNamespace + a plain dict for `headers` is sufficient because
``resolve_tenant_key`` only touches ``request.headers.get`` /
``.items`` and ``request.state.auth_mode``.
"""
state = SimpleNamespace(auth_mode=auth_mode)
return SimpleNamespace(headers=headers or {}, state=state)
# ── Header path ───────────────────────────────────────────────────────
def test_header_path_returns_header_source() -> None:
"""Header present + sanitized non-empty ⇒ header source."""
req = _request(headers={"X-Headroom-Tenant-ID": "cust_abc123"})
tenant_key, source = resolve_tenant_key(req)
assert tenant_key == "cust_abc123"
assert source == SOURCE_HEADER
def test_header_path_lowercase_header_lookup() -> None:
"""Header lookup is case-insensitive (Starlette norm)."""
req = _request(headers={"x-headroom-tenant-id": "tenant42"})
tenant_key, source = resolve_tenant_key(req)
assert tenant_key == "tenant42"
assert source == SOURCE_HEADER
def test_header_path_custom_header_via_env(monkeypatch: pytest.MonkeyPatch) -> None:
"""``HEADROOM_TENANT_KEY_HEADER`` overrides the default name."""
monkeypatch.setenv(TENANT_KEY_HEADER_ENV_VAR, "X-Custom-Tenant")
req = _request(headers={"X-Custom-Tenant": "company9"})
tenant_key, source = resolve_tenant_key(req)
assert tenant_key == "company9"
assert source == SOURCE_HEADER
def test_header_path_default_name_is_x_headroom_tenant_id() -> None:
"""Default header name is documented as ``X-Headroom-Tenant-ID``."""
assert DEFAULT_TENANT_KEY_HEADER == "X-Headroom-Tenant-ID"
def test_header_path_falls_through_when_empty_after_sanitization() -> None:
"""Pure-non-allowed header (e.g. all unicode) ⇒ fall through, NOT header source."""
# All-emoji header sanitizes to empty string — neither header
# so we drop to the global fallback.
req = _request(headers={"X-Headroom-Tenant-ID": "🦀🚀"})
tenant_key, source = resolve_tenant_key(req)
assert source == SOURCE_GLOBAL
assert tenant_key == GLOBAL_TENANT_KEY
# ── Hash path ─────────────────────────────────────────────────────────
def test_bearer_without_header_uses_global_namespace() -> None:
req = _request(
headers={"authorization": "Bearer sk-ant-api03-abc123def"},
auth_mode="payg",
)
tenant_key, source = resolve_tenant_key(req)
assert tenant_key == GLOBAL_TENANT_KEY
assert source == SOURCE_GLOBAL
def test_auth_mode_without_header_does_not_partition_tenants() -> None:
bearer = "Bearer same-tok-aaabbbccc"
payg = resolve_tenant_key(_request(headers={"authorization": bearer}, auth_mode="payg"))
oauth = resolve_tenant_key(_request(headers={"authorization": bearer}, auth_mode="oauth"))
assert payg == (GLOBAL_TENANT_KEY, SOURCE_GLOBAL)
assert oauth == (GLOBAL_TENANT_KEY, SOURCE_GLOBAL)
def test_global_fallback_when_no_signals() -> None:
"""No header, no auth, no key ⇒ literal ``"global"``."""
req = _request()
tenant_key, source = resolve_tenant_key(req)
assert tenant_key == GLOBAL_TENANT_KEY == "global"
assert source == SOURCE_GLOBAL
def _capture_tenant_resolution_logs(monkeypatch: pytest.MonkeyPatch) -> list[SimpleNamespace]:
records: list[SimpleNamespace] = []
def _info(_message: str, *, extra: dict[str, object] | None = None) -> None:
records.append(SimpleNamespace(**(extra or {})))
monkeypatch.setattr("headroom.proxy.tenant_key.logger.info", _info)
return records
def test_global_fallback_emits_structured_log(monkeypatch: pytest.MonkeyPatch) -> None:
"""The ``"global"`` source MUST log (no silent fallback)."""
req = _request()
matching = _capture_tenant_resolution_logs(monkeypatch)
resolve_tenant_key(req)
assert len(matching) == 1
assert matching[0].source == "global" # type: ignore[attr-defined]
assert matching[0].tenant_key == "global" # type: ignore[attr-defined]
def test_header_source_emits_structured_log(monkeypatch: pytest.MonkeyPatch) -> None:
"""Header path also emits the structured log."""
req = _request(headers={"X-Headroom-Tenant-ID": "tenant1"})
matching = _capture_tenant_resolution_logs(monkeypatch)
resolve_tenant_key(req)
assert len(matching) == 1
assert matching[0].source == "header" # type: ignore[attr-defined]
def test_bearer_without_header_emits_global_structured_log(monkeypatch: pytest.MonkeyPatch) -> None:
"""Bearer auth without a tenant header still emits the global log."""
req = _request(
headers={"authorization": "Bearer sk-ant-api03-abc123def"},
auth_mode="payg",
)
matching = _capture_tenant_resolution_logs(monkeypatch)
resolve_tenant_key(req)
assert len(matching) == 1
assert matching[0].source == "global" # type: ignore[attr-defined]
# ── Sanitization edges ────────────────────────────────────────────────
def test_sanitization_allows_alphanumeric() -> None:
"""ASCII alphanumerics survive untouched."""
req = _request(headers={"X-Headroom-Tenant-ID": "Cust42TenantA"})
tenant_key, _ = resolve_tenant_key(req)
assert tenant_key == "Cust42TenantA"
def test_sanitization_allows_hyphens_and_underscores() -> None:
"""Hyphens and underscores survive (common in SaaS tenant IDs)."""
req = _request(headers={"X-Headroom-Tenant-ID": "cust_a-b_c-9"})
tenant_key, _ = resolve_tenant_key(req)
assert tenant_key == "cust_a-b_c-9"
def test_sanitization_drops_control_chars() -> None:
"""Control chars (``\\x00`` / ``\\x1f``) get stripped."""
req = _request(headers={"X-Headroom-Tenant-ID": "ten\x00ant\x1f1"})
tenant_key, source = resolve_tenant_key(req)
assert tenant_key == "tenant1"
assert source == SOURCE_HEADER
def test_sanitization_drops_tab_and_newline() -> None:
"""Whitespace (``\\t`` / ``\\n``) gets stripped."""
req = _request(headers={"X-Headroom-Tenant-ID": "tenant\t\n1"})
tenant_key, source = resolve_tenant_key(req)
assert tenant_key == "tenant1"
assert source == SOURCE_HEADER
def test_sanitization_drops_punctuation() -> None:
"""`|`, `:`, `.`, `,`, `/` stripped (TOIN store-key separator safety)."""
req = _request(headers={"X-Headroom-Tenant-ID": "ten|ant:1.0/x"})
tenant_key, source = resolve_tenant_key(req)
assert tenant_key == "tenant10x"
assert source == SOURCE_HEADER
def test_sanitization_drops_unicode_falls_through_when_pure_unicode() -> None:
"""Pure-unicode header (CJK / emoji) sanitizes to empty ⇒ falls through."""
req = _request(headers={"X-Headroom-Tenant-ID": "测试テスト"})
_, source = resolve_tenant_key(req)
# No header source — sanitized to empty, dropped to global.
assert source == SOURCE_GLOBAL
def test_sanitization_truncates_at_64_chars() -> None:
"""Headers > 64 ASCII alphanum chars are TRUNCATED, not rejected."""
raw = "a" * 100
req = _request(headers={"X-Headroom-Tenant-ID": raw})
tenant_key, source = resolve_tenant_key(req)
assert tenant_key == "a" * 64
assert len(tenant_key) == 64
assert source == SOURCE_HEADER
def test_sanitization_empty_header_falls_through() -> None:
"""Empty header value ⇒ fall through (header treated as absent)."""
req = _request(headers={"X-Headroom-Tenant-ID": ""})
_, source = resolve_tenant_key(req)
assert source == SOURCE_GLOBAL
def test_sanitization_whitespace_only_falls_through() -> None:
"""Whitespace-only header ⇒ sanitizes to empty ⇒ falls through."""
req = _request(headers={"X-Headroom-Tenant-ID": " \t\n "})
_, source = resolve_tenant_key(req)
assert source == SOURCE_GLOBAL
def test_sanitization_mixed_unicode_and_ascii_preserves_ascii() -> None:
"""``cust_我_42`` sanitizes to ``cust__42`` (ASCII chars survive)."""
req = _request(headers={"X-Headroom-Tenant-ID": "cust_我_42"})
tenant_key, source = resolve_tenant_key(req)
assert tenant_key == "cust__42"
assert source == SOURCE_HEADER
# ── ContextVar surface ────────────────────────────────────────────────
def test_get_current_tenant_key_defaults_to_global() -> None:
"""Outside a request context, ``get_current_tenant_key`` ⇒ ``"global"``."""
set_request_tenant_key(None) # Make sure we're cleared.
assert get_current_tenant_key() == GLOBAL_TENANT_KEY
def test_set_request_tenant_key_round_trips() -> None:
"""``set`` then ``get`` returns the value."""
set_request_tenant_key("tenant_abc")
try:
assert get_current_tenant_key() == "tenant_abc"
finally:
set_request_tenant_key(None)
def test_set_request_tenant_key_none_resets_to_global() -> None:
"""Clearing returns to the global default."""
set_request_tenant_key("tenant_abc")
set_request_tenant_key(None)
assert get_current_tenant_key() == GLOBAL_TENANT_KEY
# ── Header precedence over auth headers ───────────────────────────────
def test_header_takes_precedence_over_auth_header() -> None:
"""When both a tenant header and bearer auth are present, the tenant header wins."""
req = _request(
headers={
"X-Headroom-Tenant-ID": "cust_explicit",
"authorization": "Bearer sk-ant-api03-aaaaa",
},
auth_mode="payg",
)
tenant_key, source = resolve_tenant_key(req)
assert tenant_key == "cust_explicit"
assert source == SOURCE_HEADER