Files
headroom/tests/test_proxy_tenant_key_integration.py
f90a56b038 fix(proxy): F3 — per-tenant TOIN learning key (#404)
## Description

This change isolates TOIN learning by tenant in shared-proxy
deployments. Each pattern key gains a tenant namespace derived from a
configurable sanitized header, a stable auth/token hash, or the
backwards-compatible global namespace. Legacy keys migrate to global and
every resolution path emits a structured event.

## Threat model

Pre-F3, every request's tool-call patterns aggregated into one global
pool keyed by `(auth_mode, model_family, sig_hash)`. That works for an
OSS single-tenant deploy, but in a multi-tenant SaaS / shared-proxy
deploy:

1. **Cross-tenant pattern leakage.** Tenant A's compression patterns
silently train recommendations the proxy then applies to tenant B's
requests, degrading both tenants' compression quality.
2. **Structural fingerprint exposure.** TOIN dumps include
`tool_signature_hash` per slice. With one shared pool, a user with read
access to a dump file learns the structural fingerprint of every other
tenant's tool surface.

F3 partitions: every TOIN store key gets a `tenant_key` prefix derived
from the inbound request. Two distinct tenants therefore train two
isolated pattern pools.

## Tenant key derivation rules

| Source | When | Key derivation |

|------------|--------------------------------------------------------------|-------------------------------------------------------------------|
| `header` | `HEADROOM_TENANT_KEY_HEADER` (default
`X-Headroom-Tenant-ID`) is present and survives sanitization | Sanitized
header value (ASCII alphanum + `-_`, max 64 chars) |
| `hash` | No header but bearer token + `auth_mode` (from F1) present |
`sha256(f"{auth_mode}:{api_key_prefix}")[:24]` (api_key_prefix = first 8
chars of bearer); same SHA-256[:24] idiom as PR #395 |
| `global` | Neither header nor bearer present (OSS single-tenant / dev)
| Literal `"global"` namespace; pre-F3 patterns migrate here on first
load |

Header path takes precedence over hash. Header values that sanitize to
empty (pure unicode / control chars only) fall through to the next rule.

## Configuration

Single env var: `HEADROOM_TENANT_KEY_HEADER`. Default
`X-Headroom-Tenant-ID`. The header NAME is configurable; the
sanitization rules and the `"global"` literal are NOT — they're
load-bearing for namespace isolation.

## Structured log fields

Every resolution path emits exactly one `tenant_key_resolved` log event
(no silent fallbacks per `feedback_no_silent_fallbacks.md`):

| Field | Always present | Notes |

|--------------|----------------|------------------------------------------------|
| `event` | yes | Always `"tenant_key_resolved"` |
| `tenant_key` | yes | Resolved key (or `"global"`) |
| `source` | yes | One of `header` / `hash` / `global` |
| `header` | header path | Header name that matched |
| `auth_mode` | hash path | `payg` / `oauth` / `subscription` |

Operators can alert on a sustained `source="global"` rate to spot
mis-configured upstream auth proxies.

## Implementation

- **NEW** `headroom/proxy/tenant_key.py`: pure resolver. Public surface
= `resolve_tenant_key(request) -> tuple[str, str]`. Plus
`set_request_tenant_key()` / `get_current_tenant_key()` ContextVar pair
so TOIN's deep-stack `record_compression` calls observe the right tenant
slice.
- `headroom/telemetry/toin.py`: extends `PatternKey` from a 3-tuple to a
4-tuple `(tenant_key, auth_mode, model_family, sig_hash)`.
`_deserialize_pattern_key` is backward-compat with both pre-F3 formats
(3-tuple from PR-B5, bare `sig_hash` from v1.0). `record_compression` /
`record_retrieval` / `get_pattern` accept an optional `tenant_key`
kwarg; when absent they read from the ContextVar.
- `headroom/cli/toin_publish.py`: each `[[recommendation]]` row now
carries `tenant_key`; rows sort by `(tenant_key, auth_mode,
model_family, sig_hash)` for clean diffs.
- `headroom/proxy/handlers/anthropic.py` +
`headroom/proxy/handlers/openai.py`: call `resolve_tenant_key(request)`
once at request entry, right after the F1 `classify_auth_mode` block.
Three call sites total (1 Anthropic, 2 OpenAI). Result attached to
`request.state.tenant_key` / `request.state.tenant_key_source` AND set
on the ContextVar.

## Realignment build constraints (per project memory)

- **configurable**: `HEADROOM_TENANT_KEY_HEADER` env var.
- **no hardcodes**: header name + `"global"` literal are the only
constants; both motivated and documented in code.
- **no regexes**: sanitization uses string ops +
`unicodedata.category()`.
- **no silent fallbacks**: every resolution emits a structured log
event.
- **comprehensive tests**: header / hash / global paths + sanitization
edges + tenant-isolation property test.
- **structured logs**: `event=tenant_key_resolved` with `tenant_key`,
`source`, optional `header` / `auth_mode`.

## Test plan

- [x] `pytest tests/test_tenant_key.py` (29 cases — header / hash /
global paths, structured-log emission, sanitization edges, ContextVar
round-trip)
- [x] `pytest tests/test_toin_observation_only.py` (6 new isolation
cases — two-tenant divergence, ContextVar read, explicit kwarg
precedence, default migration to `"global"`)
- [x] `pytest tests/test_toin*.py` (existing TOIN tests updated for the
4-tuple key shape)
- [x] `pytest tests/test_proxy_tenant_key_integration.py` (4 cases —
handler wires `request.state.tenant_key` + ContextVar through all three
resolution paths)
- [x] `pytest tests/test_critical_fixes.py tests/test_critical_gaps.py
tests/test_proxy_ccr.py` (legacy 3-tuple references updated)
- [x] `make ci-precheck-python` — 176 tests pass (passed via pre-push
hook)
- [x] `make ci-precheck-commitlint` — clean (passed via pre-push hook)

Refs: F1 (PR #399, auth_mode classifier), F2.1 (PR #400, per-auth-mode
CompressionPolicy gates).

## Type of Change

- [x] Bug fix / tenant-isolation hardening
- [ ] New feature
- [ ] Breaking change
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- Add a pure tenant-key resolver and request ContextVar.
- Extend TOIN pattern keys and published recommendations with a tenant
namespace while preserving legacy deserialization.
- Resolve and attach tenant identity at Anthropic and OpenAI request
entry.
- Add isolation, migration, sanitization, header, CCR, handler, and
executor coverage.

## Testing

- [x] Focused tenant/TOIN/proxy tests pass
- [x] Python CI precheck passes
- [x] Commitlint precheck passes

### Test Output

```text
176 focused/precheck tests passed
Current-main isolation, migration, header, CCR, and executor validation: green
```

## Real Behavior Proof

- Environment: current-main reconciled Python proxy/TOIN test
environment.
- Exact command / steps: ran tenant resolver, TOIN observation/publish,
handler integration, header-isolation, CCR, and compression-executor
suites.
- Observed result: distinct tenants train isolated pattern pools; legacy
patterns migrate to global; request handlers propagate the resolved
namespace through deep TOIN calls.
- Not tested: production multi-tenant traffic volume or an external TOIN
dump consumer.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 22:03:59 -05:00

229 lines
7.8 KiB
Python

"""F3 handler-integration tests: tenant_key plumbing through the proxy.
Pins that the OpenAI / Anthropic handlers call ``resolve_tenant_key``
once at request entry, populate ``request.state.tenant_key`` /
``request.state.tenant_key_source``, AND set the request-scoped
ContextVar so downstream TOIN ``record_compression`` calls observe the
right tenant slice.
Uses the same ``_DummyOpenAIHandler`` pattern as
``tests/test_openai_codex_routing.py`` so we don't need the Rust core
or a live upstream HTTP server.
"""
from __future__ import annotations
import base64
import json
from types import SimpleNamespace
import anyio
import pytest
from fastapi import Request
from headroom.proxy.handlers.openai import OpenAIHandlerMixin
from headroom.proxy.tenant_key import (
GLOBAL_TENANT_KEY,
SOURCE_GLOBAL,
SOURCE_HEADER,
get_current_tenant_key,
set_request_tenant_key,
)
def _jwt(payload: dict) -> str:
"""Build a 3-segment JWT for OAuth classification."""
header = {"alg": "none", "typ": "JWT"}
def encode(part: dict) -> str:
raw = json.dumps(part, separators=(",", ":")).encode("utf-8")
return base64.urlsafe_b64encode(raw).decode("ascii").rstrip("=")
return f"{encode(header)}.{encode(payload)}."
# ── Test fixtures (Dummy handler) ─────────────────────────────────────────
class _DummyMetrics:
async def record_request(self, **kwargs): # noqa: ANN003
return None
async def record_failed(self, **kwargs): # noqa: ANN003
return None
class _DummyTokenizer:
def count_messages(self, messages):
return len(messages)
class _ResponseStub:
status_code = 200
headers = {"content-type": "application/json", "content-length": "42"}
content = b'{"id":"resp_123","output":[{"type":"message"}]}'
def json(self):
return {"usage": {"input_tokens": 2, "output_tokens": 1}}
class _DummyOpenAIHandler(OpenAIHandlerMixin):
"""Minimal handler that captures the request that flows through F3."""
OPENAI_API_URL = "https://api.openai.com"
def __init__(self) -> None:
self.rate_limiter = None
self.metrics = _DummyMetrics()
self.config = SimpleNamespace(
optimize=False,
retry_max_attempts=3,
retry_base_delay_ms=10,
retry_max_delay_ms=50,
connect_timeout_seconds=10,
openai_extra_headers=None,
)
self.usage_reporter = None
self.openai_provider = SimpleNamespace(get_context_limit=lambda model: 128_000)
self.openai_pipeline = SimpleNamespace(apply=lambda **kwargs: None)
self.anthropic_backend = None
self.cost_tracker = None
self.memory_handler = None
self.session_tracker_store = SimpleNamespace(
compute_session_id=lambda *a, **k: "sess-openai-1",
)
self.captured_request: Request | None = None
async def _next_request_id(self) -> str:
return "req-1"
def _extract_tags(self, headers: dict[str, str]) -> dict[str, str]:
return {}
async def _retry_request(self, method: str, url: str, headers: dict, body: dict):
return _ResponseStub()
async def _run_compression_in_executor(self, fn, *, timeout: float):
return fn()
async def _stream_response(self, **kwargs):
return SimpleNamespace(status_code=200)
def _build_request(headers: dict[str, str], body: dict) -> Request:
payload = json.dumps(body).encode("utf-8")
async def receive():
return {"type": "http.request", "body": payload, "more_body": False}
scope = {
"type": "http",
"asgi": {"version": "3.0"},
"http_version": "1.1",
"method": "POST",
"scheme": "https",
"path": "/v1/responses",
"raw_path": b"/v1/responses",
"query_string": b"",
"headers": [
(key.lower().encode("utf-8"), value.encode("utf-8")) for key, value in headers.items()
],
"client": ("127.0.0.1", 12345),
"server": ("testserver", 443),
}
return Request(scope, receive)
@pytest.fixture(autouse=True)
def _reset_tenant_key_contextvar():
"""Clean ContextVar after every test so they don't leak."""
set_request_tenant_key(None)
yield
set_request_tenant_key(None)
# ── F3 plumbing pins ──────────────────────────────────────────────────────
def test_handler_populates_request_state_tenant_key_for_header_path(monkeypatch):
"""Header-set tenant_id ⇒ request.state.tenant_key + source="header"."""
handler = _DummyOpenAIHandler()
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda model: _DummyTokenizer())
request = _build_request(
headers={
"Authorization": "Bearer sk-test-key",
"X-Headroom-Tenant-ID": "cust_acme",
},
body={"model": "gpt-5.4", "input": "hi"},
)
anyio.run(handler.handle_openai_responses, request)
assert getattr(request.state, "tenant_key", None) == "cust_acme"
assert getattr(request.state, "tenant_key_source", None) == SOURCE_HEADER
def test_handler_populates_request_state_tenant_key_for_global_fallback(monkeypatch):
"""No tenant header, even with bearer auth, ⇒ tenant_key is global."""
handler = _DummyOpenAIHandler()
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda model: _DummyTokenizer())
# PAYG bearer (sk-...). F1 will classify auth_mode=PAYG, but F3
# requires an explicit tenant header instead of deriving from secrets.
request = _build_request(
headers={"Authorization": "Bearer sk-ant-api03-aaaabbbbccccdddd"},
body={"model": "gpt-5.4", "input": "hi"},
)
anyio.run(handler.handle_openai_responses, request)
assert getattr(request.state, "tenant_key_source", None) == SOURCE_GLOBAL
tenant_key = getattr(request.state, "tenant_key", None)
assert tenant_key == GLOBAL_TENANT_KEY
def test_handler_populates_global_when_no_signals(monkeypatch):
"""No header, no bearer ⇒ tenant_key="global" with source="global"."""
handler = _DummyOpenAIHandler()
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda model: _DummyTokenizer())
# Neither auth nor explicit tenant header.
request = _build_request(
headers={},
body={"model": "gpt-5.4", "input": "hi"},
)
anyio.run(handler.handle_openai_responses, request)
assert getattr(request.state, "tenant_key", None) == GLOBAL_TENANT_KEY
assert getattr(request.state, "tenant_key_source", None) == SOURCE_GLOBAL
def test_handler_populates_contextvar_so_toin_records_under_tenant(monkeypatch):
"""Handler must call set_request_tenant_key — TOIN reads the ContextVar.
We verify by asserting the contextvar is populated AFTER the
handler returns (the handler doesn't reset it on success — that
would defeat the whole point, since downstream record_compression
happens deep inside the pipeline). The contextvar is per-task; the
next request's handler will overwrite it.
"""
handler = _DummyOpenAIHandler()
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda model: _DummyTokenizer())
request = _build_request(
headers={"X-Headroom-Tenant-ID": "tenant_for_toin"},
body={"model": "gpt-5.4", "input": "hi"},
)
# Run inside an explicit context so the test can read back the
# ContextVar the handler set. Without `anyio.run`'s default
# context isolation, we'd see a stale value across tests.
async def _drive() -> str:
await handler.handle_openai_responses(request)
return get_current_tenant_key()
seen = anyio.run(_drive)
assert seen == "tenant_for_toin"