mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 16:45:34 +08:00
## Description Removes the anonymous-telemetry **beacon** — the only external, third-party data flow Headroom ever initiated. When telemetry was opted in, it POSTed aggregate `/stats` to a hardcoded **Supabase** REST endpoint (with an embedded anon API key in the source). For enterprise/on-prem deployments this is exactly the kind of vendor-controlled data egress a security review flags, so it's gone entirely — **zero "Supabase" references remain in the codebase.** What stays (by design): the **local** telemetry collector + the `HEADROOM_TELEMETRY` opt-in (it only feeds `/stats` and `/v1/telemetry` — nothing leaves the process), **OpenTelemetry export** (`HEADROOM_OTEL_METRICS_*`, so operators send operational metrics to *their own* collector), and the license usage reporter (your own domain, license-key-gated). Also fixes the contact domain: `headroom.dev` → `headroomlabs.ai` everywhere. Closes # (no tracking issue) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) > Non-breaking: `HEADROOM_TELEMETRY` is still accepted (now gates local collection only). The only behavior change is that no telemetry is ever sent externally. ## Changes Made - **Deleted the Supabase beacon**: `TelemetryBeacon` class, `_SUPABASE_URL`/`_SUPABASE_KEY`/`_TABLE`/`_ENDPOINT`, the JSONB projection helper, the proxy-lifespan beacon wiring, the `SUPABASE_` install env passthrough, and `tests/test_strategy_stats_supabase.py`. - **Kept** the local opt-in predicate (`is_telemetry_enabled` etc.) in `beacon.py` — still used by the local collector + CLI — reworded to "local only". - **Retained** the single-worker-owner file lock (the cc-switch reconciler depends on it); updated its comments to drop the beacon framing. - `/stats` `anon_telemetry_shipping` is now always `False` (nothing ships externally); startup log reworded to "Local telemetry". - Reworded remaining "Supabase" comments in `collector.py`, `context.py`, `prometheus_metrics.py`, and two test docstrings. - Contact domain: `security@headroom.dev` → `security@headroomlabs.ai`, `conduct@headroom.dev` → `conduct@headroomlabs.ai`, FUNDING.yml sponsor URL. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ grep -rniI "supabase" --include=*.py --include=*.md --include=*.mdx . # (excl .venv/sbom) >>> ZERO Supabase references $ grep -rniI "headroom.dev" . >>> ZERO headroom.dev references $ ruff check <changed files> -> All checks passed! $ ruff format --check <changed files> -> 10 files already formatted $ mypy <changed telemetry files> -> Success: no issues found $ pytest tests/test_telemetry.py tests/test_telemetry_warning.py \ tests/test_proxy_telemetry_env.py tests/test_compression_observability.py \ tests/test_paths.py tests/test_paths_backward_compat.py -q ============================= 173 passed in 6.67s ============================== ``` ## Real Behavior Proof - **Environment:** macOS, Python 3.12 (`.venv`). - **Exact command / steps:** repo-wide grep for `supabase`/`headroom.dev`; `create_app(...)` driven through a full `TestClient` lifespan (startup + shutdown) in `test_proxy_telemetry_env.py`; `/stats` exercised in `test_telemetry_warning.py`. - **Observed result:** zero `supabase`/`headroom.dev` strings remain; the proxy starts and shuts down cleanly with the beacon removed (the worker-owner lock + reconciler still elect a single owner); `/stats.anon_telemetry_shipping` is `False` even with `HEADROOM_TELEMETRY=on`; local collector + OTEL paths unchanged. - **Not tested:** no live network call was ever made (the point — the external POST is gone). OTEL export and the license reporter were not exercised (unchanged by this PR). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The license usage reporter (`reporter.py` → `app.headroomlabs.ai`) is intentionally **kept** — it's license-key-gated (dormant for unlicensed/OSS deployments) and goes to your own domain, not a third party. - Docs/CHANGELOG left unchecked: a couple of docs mention the telemetry beacon and may want a follow-up note that it now collects locally only; happy to add.
416 lines
15 KiB
Python
416 lines
15 KiB
Python
"""Per-strategy compression observability tests.
|
|
|
|
These guard the forcing function: when any compressor runs in
|
|
production, a `CompressionObserver` notification fires once per real
|
|
compression event, and `PrometheusMetrics` accumulates per-strategy
|
|
counters that the test suite asserts on directly.
|
|
|
|
The TOIN→SmartCrusher silent disconnect (caught three weeks late by
|
|
manual audit) was invisible because no signal distinguished by
|
|
strategy. These tests exist so the next regression of that shape
|
|
fails the suite the day it lands instead of waiting on an audit.
|
|
|
|
The counters live ONLY as in-process state on the metrics instance;
|
|
they are deliberately NOT exported as new Prometheus metric names
|
|
(to avoid unbounded metric-series growth) — they remain observable
|
|
via /stats. CI-level
|
|
observability via these tests is enough to catch silent regressions;
|
|
production export waits on a non-column-adding pipeline.
|
|
|
|
Coverage:
|
|
|
|
1. `ContentRouter.compress(...)` calls observer once per RoutingDecision.
|
|
2. `SmartCrusher.apply(...)` calls observer once per crushed message.
|
|
3. Both transforms tolerate an observer that raises (compression must
|
|
still succeed).
|
|
4. `PrometheusMetrics` correctly satisfies the `CompressionObserver`
|
|
protocol — `record_compression` increments per-strategy counters
|
|
and `tokens_saved_by_strategy` accumulates only positive savings.
|
|
5. The Prometheus scrape output (`export()`) does NOT emit any new
|
|
metric names — the per-strategy state stays internal.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from dataclasses import dataclass, field
|
|
from typing import Any
|
|
|
|
import pytest
|
|
|
|
from headroom.transforms.content_detector import ContentType
|
|
from headroom.transforms.content_router import (
|
|
CompressionStrategy,
|
|
ContentRouter,
|
|
ContentRouterConfig,
|
|
RouterCompressionResult,
|
|
RoutingDecision,
|
|
)
|
|
from headroom.transforms.observability import CompressionObserver
|
|
from headroom.transforms.smart_crusher import SmartCrusher, SmartCrusherConfig
|
|
|
|
# ─── Test doubles ──────────────────────────────────────────────────────
|
|
|
|
|
|
@dataclass
|
|
class SpyObserver:
|
|
"""Captures every `record_compression` call for assertion."""
|
|
|
|
calls: list[tuple[str, int, int]] = field(default_factory=list)
|
|
|
|
def record_compression(
|
|
self,
|
|
strategy: str,
|
|
original_tokens: int,
|
|
compressed_tokens: int,
|
|
) -> None:
|
|
self.calls.append((strategy, original_tokens, compressed_tokens))
|
|
|
|
|
|
@dataclass
|
|
class ExplodingObserver:
|
|
"""Raises on every call. Used to assert observer failures don't
|
|
propagate out and break compression."""
|
|
|
|
raised: int = 0
|
|
|
|
def record_compression(self, *_a: Any, **_kw: Any) -> None:
|
|
self.raised += 1
|
|
raise RuntimeError("simulated observer outage")
|
|
|
|
|
|
# ─── Protocol conformance ──────────────────────────────────────────────
|
|
|
|
|
|
def test_spy_satisfies_observer_protocol():
|
|
spy = SpyObserver()
|
|
# `runtime_checkable` Protocol — isinstance check works.
|
|
assert isinstance(spy, CompressionObserver)
|
|
|
|
|
|
def test_prometheus_metrics_satisfies_observer_protocol():
|
|
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
|
|
|
m = PrometheusMetrics()
|
|
assert isinstance(m, CompressionObserver)
|
|
|
|
|
|
# ─── ContentRouter wiring ──────────────────────────────────────────────
|
|
|
|
|
|
def test_content_router_records_observer_call_per_routing_decision():
|
|
spy = SpyObserver()
|
|
router = ContentRouter(ContentRouterConfig(), observer=spy)
|
|
|
|
# Forge a routing log directly via the result object — the observer
|
|
# call site walks `result.routing_log`, so we assert the contract
|
|
# without depending on which compressor would actually fire.
|
|
result = RouterCompressionResult(
|
|
compressed="x",
|
|
original="x",
|
|
strategy_used=CompressionStrategy.SMART_CRUSHER,
|
|
routing_log=[
|
|
RoutingDecision(
|
|
content_type=ContentType.JSON_ARRAY,
|
|
strategy=CompressionStrategy.SMART_CRUSHER,
|
|
original_tokens=200,
|
|
compressed_tokens=50,
|
|
),
|
|
RoutingDecision(
|
|
content_type=ContentType.SOURCE_CODE,
|
|
strategy=CompressionStrategy.CODE_AWARE,
|
|
original_tokens=300,
|
|
compressed_tokens=300, # passthrough — still recorded
|
|
),
|
|
],
|
|
)
|
|
router._observe(result)
|
|
|
|
assert spy.calls == [
|
|
("smart_crusher", 200, 50),
|
|
("code_aware", 300, 300),
|
|
]
|
|
|
|
|
|
def test_content_router_with_no_observer_is_silent():
|
|
router = ContentRouter(ContentRouterConfig()) # observer defaults None
|
|
result = RouterCompressionResult(
|
|
compressed="x",
|
|
original="x",
|
|
strategy_used=CompressionStrategy.PASSTHROUGH,
|
|
routing_log=[
|
|
RoutingDecision(
|
|
content_type=ContentType.PLAIN_TEXT,
|
|
strategy=CompressionStrategy.TEXT,
|
|
original_tokens=10,
|
|
compressed_tokens=5,
|
|
)
|
|
],
|
|
)
|
|
# Should not raise.
|
|
router._observe(result)
|
|
|
|
|
|
def test_content_router_swallows_observer_failures():
|
|
boom = ExplodingObserver()
|
|
router = ContentRouter(ContentRouterConfig(), observer=boom)
|
|
result = RouterCompressionResult(
|
|
compressed="x",
|
|
original="x",
|
|
strategy_used=CompressionStrategy.TEXT,
|
|
routing_log=[
|
|
RoutingDecision(
|
|
content_type=ContentType.PLAIN_TEXT,
|
|
strategy=CompressionStrategy.TEXT,
|
|
original_tokens=10,
|
|
compressed_tokens=5,
|
|
)
|
|
],
|
|
)
|
|
# Must not raise — observability failures are not compression failures.
|
|
router._observe(result)
|
|
assert boom.raised == 1
|
|
|
|
|
|
# ─── SmartCrusher wiring (legacy direct-pipeline path) ─────────────────
|
|
|
|
|
|
def _bigger_array(n: int = 60) -> str:
|
|
import json as _json
|
|
|
|
items = [{"status": "ok", "tag": "x", "n": i} for i in range(n)]
|
|
return _json.dumps(items)
|
|
|
|
|
|
@pytest.fixture
|
|
def isolated_toin(tmp_path, monkeypatch):
|
|
"""Point TOIN at a tempdir for the duration of the test.
|
|
|
|
SmartCrusher.apply() feeds the global TOIN learning store via
|
|
`record_compression`. Its default storage path is
|
|
`~/.headroom/toin.json`, which persists across pytest invocations.
|
|
On Python 3.11 CI runs the suite twice (regular + coverage); a
|
|
pattern written in run #1 changes which rows the lossy sampler
|
|
keeps in run #2 and breaks `test_first_last_items_always_preserved`
|
|
in `test_evals.py`.
|
|
|
|
Isolating the TOIN file per test contains the side effect.
|
|
"""
|
|
from pathlib import Path
|
|
|
|
from headroom.telemetry.toin import TOIN_PATH_ENV_VAR, reset_toin
|
|
|
|
storage = str(Path(tmp_path) / "toin.json")
|
|
monkeypatch.setenv(TOIN_PATH_ENV_VAR, storage)
|
|
reset_toin()
|
|
yield
|
|
reset_toin()
|
|
|
|
|
|
def test_smart_crusher_apply_records_observer_per_crushed_message(isolated_toin):
|
|
"""End-to-end: SmartCrusher.apply() walks messages, crushes the
|
|
big tool_result, fires the observer with strategy='smart_crusher'."""
|
|
from headroom.providers.openai import OpenAITokenCounter
|
|
from headroom.tokenizer import Tokenizer
|
|
|
|
spy = SpyObserver()
|
|
crusher = SmartCrusher(SmartCrusherConfig(), observer=spy)
|
|
tok = Tokenizer(OpenAITokenCounter("gpt-4o-mini"), model="gpt-4o-mini")
|
|
|
|
messages = [
|
|
{"role": "user", "content": "what's in the data?"},
|
|
{"role": "tool", "content": _bigger_array(60)},
|
|
]
|
|
result = crusher.apply(messages, tok)
|
|
# If the analyzer chose passthrough this run, the observer wasn't
|
|
# fired; that's fine for the wiring test — we only assert it WAS
|
|
# fired in the case it crushed.
|
|
if "smart_crush:" in ",".join(result.transforms_applied):
|
|
assert spy.calls, "smart_crusher crushed but observer wasn't notified"
|
|
for strategy, original, compressed in spy.calls:
|
|
assert strategy == "smart_crusher"
|
|
assert original > 0
|
|
assert compressed >= 0
|
|
|
|
|
|
def test_smart_crusher_apply_swallows_observer_failures(isolated_toin):
|
|
"""Observer raises → compression still completes, returns valid
|
|
TransformResult, count of raises matches the crushed_count."""
|
|
from headroom.providers.openai import OpenAITokenCounter
|
|
from headroom.tokenizer import Tokenizer
|
|
|
|
boom = ExplodingObserver()
|
|
crusher = SmartCrusher(SmartCrusherConfig(), observer=boom)
|
|
tok = Tokenizer(OpenAITokenCounter("gpt-4o-mini"), model="gpt-4o-mini")
|
|
messages = [{"role": "tool", "content": _bigger_array(60)}]
|
|
result = crusher.apply(messages, tok)
|
|
# Either the analyzer didn't crush (boom.raised == 0) or it did
|
|
# (boom.raised >= 1) — but in both cases compression returned a
|
|
# valid TransformResult. No exception escaped.
|
|
assert result.messages is not None
|
|
|
|
|
|
# ─── PrometheusMetrics implementation ──────────────────────────────────
|
|
|
|
|
|
def test_prometheus_metrics_accumulates_per_strategy_counters():
|
|
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
|
|
|
m = PrometheusMetrics()
|
|
|
|
m.record_compression("smart_crusher", original_tokens=200, compressed_tokens=50)
|
|
m.record_compression("smart_crusher", original_tokens=100, compressed_tokens=40)
|
|
m.record_compression("diff", original_tokens=80, compressed_tokens=80) # no savings
|
|
m.record_compression("code_aware", original_tokens=50, compressed_tokens=70) # negative savings
|
|
|
|
assert m.compressions_by_strategy == {
|
|
"smart_crusher": 2,
|
|
"diff": 1,
|
|
"code_aware": 1,
|
|
}
|
|
# Tokens saved is `max(0, original - compressed)` per strategy.
|
|
# smart_crusher: 150 + 60 = 210; diff: 0 (no savings, dict entry omitted);
|
|
# code_aware: 0 (negative).
|
|
assert m.tokens_saved_by_strategy == {"smart_crusher": 210}
|
|
|
|
|
|
def test_prometheus_metrics_accumulates_codex_ws_unit_and_frame_counters():
|
|
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
|
|
|
m = PrometheusMetrics()
|
|
|
|
m.record_codex_ws_unit(
|
|
strategy="mixed",
|
|
reason_category="applied",
|
|
elapsed_ms=1250,
|
|
text_bytes=10_000,
|
|
tokens_before=2500,
|
|
tokens_after=1000,
|
|
tokens_saved=1500,
|
|
modified=True,
|
|
strategy_chain=["mixed", "kompress"],
|
|
content_type="text",
|
|
text_shape="jsonl_like",
|
|
)
|
|
m.record_codex_ws_unit(
|
|
strategy="passthrough",
|
|
reason_category="size_floor",
|
|
elapsed_ms=2,
|
|
text_bytes=100,
|
|
tokens_before=20,
|
|
tokens_after=20,
|
|
tokens_saved=0,
|
|
modified=False,
|
|
strategy_chain=["passthrough"],
|
|
content_type="unknown",
|
|
text_shape="plain_text_like",
|
|
)
|
|
m.record_codex_ws_frame(
|
|
elapsed_ms=1260,
|
|
bytes_before=20_000,
|
|
bytes_after=8_000,
|
|
attempted_tokens=2500,
|
|
tokens_saved=1500,
|
|
modified=True,
|
|
strategy_chain=["mixed", "kompress"],
|
|
final_strategies=["mixed"],
|
|
)
|
|
m.record_codex_ws_frame(
|
|
elapsed_ms=30_000,
|
|
bytes_before=426_318,
|
|
failed=True,
|
|
)
|
|
|
|
assert m.codex_ws_units_total == 2
|
|
assert m.codex_ws_units_modified_total == 1
|
|
assert m.codex_ws_units_by_strategy == {"mixed": 1, "passthrough": 1}
|
|
assert m.codex_ws_units_by_category == {"applied": 1, "size_floor": 1}
|
|
assert m.codex_ws_units_by_content_type == {"text": 1, "unknown": 1}
|
|
assert m.codex_ws_units_by_text_shape == {"jsonl_like": 1, "plain_text_like": 1}
|
|
assert m.codex_ws_units_to_kompress_total == 0
|
|
assert m.codex_ws_units_kompress_attempted_total == 1
|
|
assert m.codex_ws_unit_elapsed_ms_max == 1250
|
|
assert m.codex_ws_unit_tokens_saved_sum == 1500
|
|
|
|
assert m.codex_ws_frames_attempted_total == 2
|
|
assert m.codex_ws_frames_compressed_total == 1
|
|
assert m.codex_ws_frames_failed_total == 1
|
|
assert m.codex_ws_frames_to_kompress_total == 0
|
|
assert m.codex_ws_frames_kompress_attempted_total == 1
|
|
assert m.codex_ws_frame_elapsed_ms_max == 30_000
|
|
assert m.codex_ws_frame_tokens_saved_sum == 1500
|
|
|
|
|
|
def test_prometheus_export_does_not_leak_per_strategy_metrics():
|
|
"""Per-strategy state is tracked in-process only. The Prometheus
|
|
scrape output deliberately must NOT emit new metric names (to avoid
|
|
unbounded metric-series growth); the state stays observable via
|
|
/stats. This test guards that constraint: if a future change adds
|
|
the metric to the scrape, this fails and forces a conscious
|
|
decision."""
|
|
import asyncio
|
|
|
|
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
|
|
|
m = PrometheusMetrics()
|
|
m.record_compression("smart_crusher", original_tokens=200, compressed_tokens=50)
|
|
m.record_compression("diff", original_tokens=120, compressed_tokens=70)
|
|
|
|
output = asyncio.run(m.export())
|
|
|
|
assert "headroom_compressions_total" not in output
|
|
assert "headroom_tokens_saved_by_strategy_total" not in output
|
|
|
|
|
|
# ─── End-to-end smoke (router + metrics together) ──────────────────────
|
|
|
|
|
|
def test_router_with_prometheus_observer_increments_counters():
|
|
"""Plumbing test: a router wired to a real PrometheusMetrics
|
|
instance lights up the per-strategy counters as routing decisions
|
|
accumulate. This is the production wiring shape from
|
|
`headroom/proxy/server.py`."""
|
|
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
|
|
|
m = PrometheusMetrics()
|
|
router = ContentRouter(ContentRouterConfig(), observer=m)
|
|
|
|
fake_result = RouterCompressionResult(
|
|
compressed="x",
|
|
original="x",
|
|
strategy_used=CompressionStrategy.MIXED,
|
|
routing_log=[
|
|
RoutingDecision(
|
|
content_type=ContentType.JSON_ARRAY,
|
|
strategy=CompressionStrategy.SMART_CRUSHER,
|
|
original_tokens=300,
|
|
compressed_tokens=80,
|
|
),
|
|
RoutingDecision(
|
|
content_type=ContentType.SOURCE_CODE,
|
|
strategy=CompressionStrategy.CODE_AWARE,
|
|
original_tokens=200,
|
|
compressed_tokens=120,
|
|
),
|
|
RoutingDecision(
|
|
content_type=ContentType.JSON_ARRAY,
|
|
strategy=CompressionStrategy.SMART_CRUSHER,
|
|
original_tokens=100,
|
|
compressed_tokens=40,
|
|
),
|
|
],
|
|
)
|
|
router._observe(fake_result)
|
|
|
|
assert m.compressions_by_strategy == {"smart_crusher": 2, "code_aware": 1}
|
|
assert m.tokens_saved_by_strategy == {
|
|
"smart_crusher": (300 - 80) + (100 - 40), # 280
|
|
"code_aware": (200 - 120), # 80
|
|
}
|
|
|
|
|
|
# IntelligentContextManager observability tests retired with PR-B1 —
|
|
# the manager itself was deleted along with the message-dropping
|
|
# strategy. Inner-router observability is now exercised solely
|
|
# through ContentRouter, covered by
|
|
# `test_content_router_records_observer_call_per_routing_decision`.
|