fix(cli): report anonymous beacon status accurately in the banner, log, and /stats (#3533)

## Description

`HEADROOM_TELEMETRY` (local aggregate stats, opt-in, off by default) and
`HEADROOM_BEACON` (the anonymous session-summary upload beacon —
`telemetry/beacon.py`: `BEACON_DEFAULT_ON = True`, opt-**out** by
default) are two independent switches, by the module's own docstring.
Three places only ever checked the first one and never mentioned the
second:

- The proxy CLI startup banner (`cli/proxy.py`) printed a bare
`Telemetry: DISABLED` whenever `HEADROOM_TELEMETRY`/`--no-telemetry` was
off, with no mention that the beacon — the switch that actually ships
data externally — was still active by default.
- A startup log line (`proxy/server.py`) carried a comment and message
claiming telemetry is "collected locally only (the anonymous telemetry
beacon was removed)". This isn't true of the current codebase:
`telemetry/beacon.py` and `telemetry/session.py` fully implement and
wire the beacon end to end via `record_outcome()` → `SessionAggregator`
→ a POST to `headroom-beacon.headroom-beacon.workers.dev`.
- The `/stats` API's `anon_telemetry_shipping` field was hardcoded to
`False` on the same premise, so a caller polling `/stats` to confirm
nothing ships externally got a false assurance regardless of their
actual `HEADROOM_BEACON` setting.

Found auditing this deployment in front of Claude Code on a home server,
alongside #2683/#2684 (`SECURITY.md`'s credential-storage claim) — a
related but distinct gap: that PR made the true facts about the beacon
*discoverable* in a new docs page (and states explicitly "no code
changes"); this PR fixes the three places where the running proxy
*actively asserts something false* on its own status surfaces.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/proxy.py`: the startup banner now delegates to the
existing `format_telemetry_notice()` helper
(`headroom/telemetry/beacon.py`), which already gets the beacon-vs-local
distinction right and is already covered by `TestFormatTelemetryNotice`
— instead of re-deriving (and re-drifting from) the same wording a
second time.
- `headroom/proxy/server.py`: the startup log line and the `/stats`
`anon_telemetry_shipping` field now call `is_beacon_enabled()` for real
instead of a hardcoded value / a comment asserting the beacon was
removed.
- `tests/test_telemetry_warning.py`: updated
`TestProxyCLITelemetryBanner` and `TestStatsEndpointTelemetryFlag` (see
Testing below for why).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_telemetry_warning.py -v
...
40 passed in 0.84s

$ python -m ruff check headroom/cli/proxy.py headroom/proxy/server.py tests/test_telemetry_warning.py
All checks passed!

$ python -m ruff format --check headroom/cli/proxy.py headroom/proxy/server.py tests/test_telemetry_warning.py
3 files already formatted

$ python -m mypy headroom/cli/proxy.py headroom/proxy/server.py --ignore-missing-imports
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS (aarch64-darwin), Python 3.14.7, `pip install -e
".[proxy]"`, pytest 9.1.1, `click.testing.CliRunner` for the CLI banner
assertions.
- Exact command / steps: ran the full existing
`tests/test_telemetry_warning.py` suite before touching any test,
confirming every existing `TestProxyCLITelemetryBanner` test only ever
set `HEADROOM_TELEMETRY` and never `HEADROOM_BEACON` — meaning they
passed identically whether the beacon was on or off, which is how this
shipped uncaught. Updated those six tests to pin `HEADROOM_BEACON=off`
(isolating the local-only wording they were written to test, matching
the convention `TestFormatTelemetryNotice` already uses a few classes
above), added two new tests for the beacon-on-by-default case, and
rewrote the two `/stats` tests that previously asserted
`anon_telemetry_shipping is False` unconditionally under a comment
stating the same "beacon was removed" premise as the code it was
testing.
- Observed result: all 40 tests in the file pass against the patched
code. Reverting the source changes only (keeping the updated tests)
reproduces real failures — the two `TestProxyCLITelemetryBanner`
beacon-specific tests and both `TestStatsEndpointTelemetryFlag` tests
fail against the pre-fix banner/log/`/stats` code, confirming the tests
actually exercise the fix rather than passing vacuously.
- Not tested: the actual outbound POST to the beacon endpoint (out of
scope: this PR changes what the proxy *reports about* the beacon's
state, not the beacon's own send/receive logic, which is unchanged and
untouched here). (`mypy headroom/cli/proxy.py headroom/proxy/server.py
--ignore-missing-imports`, pinned to the CI toolchain versions —
`mypy==1.20.2`, `numpy==2.2.6` from `uv.lock` — now run and clean; see
Testing → Test Output above.)

## Runtime Rollout Safety

- Rollout-managed feature(s): none — this changes status-reporting
strings and a boolean field, not any behavior gated by a rollout flag.
- Minimum rollout channel: n/a
- Stable/default behavior changed: the *reported* telemetry status
changes (an operator who set only `--no-telemetry` now sees the beacon
disclosed instead of a bare "DISABLED"); the actual telemetry/beacon
behavior itself is unchanged by this PR.
- Kill switch / disable path: unaffected by this PR —
`HEADROOM_BEACON=off` / `DO_NOT_TRACK=1` continue to work exactly as
before.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit; the banner/log/`/stats` field return
to their previous (inaccurate) wording.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation — n/a
beyond the code comments themselves; #2684 already covers the docs-page
side of this gap
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Companion to #2683/#2684 (thank you @keshavlingala for the docs half of
this) — happy to rebase on top of that once it lands if there's any
overlap in `SECURITY.md`, though this PR doesn't touch that file.

This is one of two small, independently-scoped fixes from the same audit
pass; the other (an unhandled `ClientDisconnect` crash in
`handle_anthropic_messages`) is a separate PR per the "one logical
change per PR" guidance in CONTRIBUTING.md.

Opening as a draft while I give it a final look — will mark ready for
review shortly.

---------

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Bradley Allan Davis
2026-09-24 16:46:35 -05:00
committed by GitHub
co-authored by Tejas Chopra Claude Opus 5 JD Davis
parent 3e36aefb43
commit fa9edb3a29
4 changed files with 168 additions and 31 deletions
+27 -9
View File
@@ -1547,18 +1547,36 @@ Memory (Multi-Provider):
" Stateless: YES (no filesystem writes — memory, logs, TOIN disabled)\n"
)
from headroom.telemetry.beacon import is_telemetry_enabled
# Build telemetry section for the startup banner.
#
# HEADROOM_TELEMETRY (local aggregate stats, off by default) and
# HEADROOM_BEACON (the anonymous upload beacon, ON by default —
# see telemetry/beacon.py) are two independent switches. This banner
# used to check only is_telemetry_enabled() and print "DISABLED" for
# any operator who had merely turned local stats off, even though the
# beacon — the switch that actually ships data off the machine — was
# still on and unmentioned. Delegate to format_telemetry_notice(), the
# one place that already gets the beacon-vs-local distinction right,
# instead of re-deriving (and re-drifting from) the same wording here.
from headroom.telemetry.beacon import (
format_telemetry_notice,
is_beacon_enabled,
is_telemetry_enabled,
)
# Build telemetry section for the startup banner. Telemetry is opt-in
# (off by default); the disabled line surfaces how to opt in.
if is_telemetry_enabled():
telemetry_line = (
" Telemetry: ENABLED (anonymous aggregate stats — you opted in)\n"
" Disable: HEADROOM_TELEMETRY=off or headroom proxy --no-telemetry"
)
_notice = format_telemetry_notice(prefix=" ")
if _notice:
telemetry_line = _notice
elif is_beacon_enabled() or is_telemetry_enabled():
# format_telemetry_notice() returns "" when HEADROOM_TELEMETRY_WARN=off
# suppresses the notice text itself — still say ON/OFF plainly rather
# than silently showing nothing in the one place an operator is most
# likely to be checking.
telemetry_line = " Telemetry: ON (notice suppressed via HEADROOM_TELEMETRY_WARN=off)"
else:
telemetry_line = (
" Telemetry: DISABLED (opt in: HEADROOM_TELEMETRY=on or headroom proxy --telemetry)"
" Telemetry: OFF (local stats: HEADROOM_TELEMETRY=on to enable | "
"beacon: HEADROOM_BEACON=on to enable)"
)
# Discover proxy extensions (third-party packages registered via the
+1 -1
View File
@@ -143,7 +143,7 @@
</div>
<template x-if="stats.anon_telemetry_shipping">
<div class="inline-flex items-center gap-1.5 rounded-full border border-amber-500/40 bg-amber-500/10 px-2.5 py-1"
title="Anonymous aggregate telemetry is enabled. Disable with HEADROOM_TELEMETRY=off or --no-telemetry.">
title="The anonymous telemetry beacon is on: compression stats only, never prompts, code, or file paths. Turn it off with HEADROOM_BEACON=off or DO_NOT_TRACK=1.">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16" fill="currentColor" aria-hidden="true"
class="w-3 h-3 text-amber-400 shrink-0">
<path fill-rule="evenodd"
+23 -7
View File
@@ -189,7 +189,7 @@ from headroom.subscription.tracker import (
get_subscription_tracker,
)
from headroom.telemetry import get_telemetry_collector
from headroom.telemetry.beacon import is_telemetry_enabled
from headroom.telemetry.beacon import is_beacon_enabled, is_telemetry_enabled
from headroom.telemetry.toin import get_toin
from headroom.transforms import (
CacheAligner,
@@ -2203,9 +2203,13 @@ class HeadroomProxy(
)
# Log local telemetry status so operators can see it in the log stream.
# Nothing is sent externally — telemetry is collected locally only (the
# anonymous telemetry beacon was removed); operational metrics export
# only to your own OTEL collector via HEADROOM_OTEL_METRICS_*.
# This is HEADROOM_TELEMETRY only (local aggregate stats; nothing
# sent externally). It is NOT a statement about the anonymous upload
# beacon (HEADROOM_BEACON), which is a separate, ON-by-default switch
# (telemetry/beacon.py) still present in this codebase — the previous
# wording here ("the anonymous telemetry beacon was removed") was
# false and predates this fix. See is_beacon_enabled() below for the
# accurate beacon-shipping status.
if is_telemetry_enabled():
logger.info(
"Local telemetry: ENABLED (aggregate stats, local only — nothing sent "
@@ -2216,6 +2220,13 @@ class HeadroomProxy(
"Local telemetry: DISABLED (off by default — opt in: "
"HEADROOM_TELEMETRY=on or --telemetry)"
)
if is_beacon_enabled():
logger.info(
"Anonymous upload beacon: ENABLED (default — session summaries are sent to "
"Headroom Labs). Opt out: HEADROOM_BEACON=off or DO_NOT_TRACK=1"
)
else:
logger.info("Anonymous upload beacon: DISABLED")
self.pipeline_extensions.emit(
PipelineStage.POST_START,
@@ -4804,9 +4815,14 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
# Per-language AST compression pauses. Empty on a healthy install;
# non-empty is the explanation for a savings drop in one language.
"code_syntax_breaker": _code_syntax_breaker_status(),
# Always False: the anonymous telemetry beacon was removed, so no
# telemetry is ever shipped externally (local collection only).
"anon_telemetry_shipping": False,
# Reflects the actual live state of the anonymous upload beacon
# (HEADROOM_BEACON, on by default — see telemetry/beacon.py).
# This field previously hardcoded False on the incorrect premise
# that the beacon had been removed from the codebase; it had not,
# so an operator polling /stats to confirm nothing ships
# externally was given a false assurance regardless of their
# actual HEADROOM_BEACON setting.
"anon_telemetry_shipping": is_beacon_enabled(),
"telemetry": {
"enabled": telemetry_stats.get("enabled", False),
"total_compressions": telemetry_stats.get("total_compressions", 0),
+117 -14
View File
@@ -132,7 +132,19 @@ class TestFormatTelemetryNotice:
class TestProxyCLITelemetryBanner:
"""Proxy CLI startup banner must include telemetry status."""
"""Proxy CLI startup banner must include telemetry status.
Every scenario below sets HEADROOM_BEACON=off explicitly, isolating the
local-telemetry-only wording (mirrors the convention already used by
TestFormatTelemetryNotice, e.g. test_returns_notice_when_telemetry_on).
Before the beacon-disclosure fix the banner never looked at the beacon
at all, so these tests would have passed unchanged whether the beacon
was on or off -- which is exactly the bug: an operator relying on this
banner had no way to tell the two apart. The two tests at the bottom of
this class (*_beacon_default_on_is_surfaced,
*_beacon_off_local_off_says_fully_off) cover the beacon-on-by-default
case the old banner never surfaced.
"""
@pytest.fixture
def runner(self):
@@ -141,6 +153,7 @@ class TestProxyCLITelemetryBanner:
def test_banner_shows_telemetry_enabled(self, runner, monkeypatch):
# Telemetry is opt-in: it only shows ENABLED once explicitly turned on.
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
monkeypatch.setenv("HEADROOM_BEACON", "off")
from headroom.cli.main import main
@@ -151,9 +164,12 @@ class TestProxyCLITelemetryBanner:
assert "ENABLED" in result.output
def test_banner_disabled_by_default(self, runner, monkeypatch):
# The whole point of opt-in: unset env => telemetry off, banner says so
# and surfaces how to opt in.
# The whole point of opt-in: unset env => local telemetry off, banner
# says so and surfaces how to opt in. (Beacon pinned off here so this
# isolates the local-only wording; see the beacon-specific tests
# below for the on-by-default beacon case.)
monkeypatch.delenv("HEADROOM_TELEMETRY", raising=False)
monkeypatch.setenv("HEADROOM_BEACON", "off")
from headroom.cli.main import main
@@ -161,11 +177,12 @@ class TestProxyCLITelemetryBanner:
result = runner.invoke(main, ["proxy"])
assert "Telemetry:" in result.output
assert "DISABLED" in result.output
assert "HEADROOM_TELEMETRY=on" in result.output or "--telemetry" in result.output
assert "OFF" in result.output
assert "HEADROOM_TELEMETRY=on" in result.output
def test_telemetry_flag_opts_in(self, runner, monkeypatch):
monkeypatch.delenv("HEADROOM_TELEMETRY", raising=False)
monkeypatch.setenv("HEADROOM_BEACON", "off")
from headroom.cli.main import main
@@ -177,6 +194,7 @@ class TestProxyCLITelemetryBanner:
def test_banner_shows_telemetry_disabled(self, runner, monkeypatch):
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
monkeypatch.setenv("HEADROOM_BEACON", "off")
from headroom.cli.main import main
@@ -184,10 +202,11 @@ class TestProxyCLITelemetryBanner:
result = runner.invoke(main, ["proxy"])
assert "Telemetry:" in result.output
assert "DISABLED" in result.output
assert "OFF" in result.output
def test_no_telemetry_flag_disables(self, runner, monkeypatch):
monkeypatch.delenv("HEADROOM_TELEMETRY", raising=False)
monkeypatch.setenv("HEADROOM_BEACON", "off")
from headroom.cli.main import main
@@ -195,10 +214,11 @@ class TestProxyCLITelemetryBanner:
result = runner.invoke(main, ["proxy", "--no-telemetry"])
assert "Telemetry:" in result.output
assert "DISABLED" in result.output
assert "OFF" in result.output
def test_banner_shows_opt_out_instructions_when_enabled(self, runner, monkeypatch):
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
monkeypatch.setenv("HEADROOM_BEACON", "off")
from headroom.cli.main import main
@@ -207,6 +227,49 @@ class TestProxyCLITelemetryBanner:
assert "HEADROOM_TELEMETRY=off" in result.output or "--no-telemetry" in result.output
def test_banner_beacon_default_on_is_surfaced(self, runner, monkeypatch):
"""The actual bug: HEADROOM_TELEMETRY=off / --no-telemetry alone used
to make the banner print a bare "Telemetry: DISABLED" even though the
anonymous upload beacon (a separate, on-by-default switch) was still
active and never mentioned. After the fix, an operator who sets only
--no-telemetry still sees the beacon disclosed."""
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
monkeypatch.delenv("HEADROOM_BEACON", raising=False)
monkeypatch.delenv("DO_NOT_TRACK", raising=False)
from headroom.cli.main import main
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
result = runner.invoke(main, ["proxy", "--no-telemetry"])
assert "Telemetry:" in result.output
assert "compression stats" in result.output
assert "HEADROOM_BEACON=off" in result.output
# The old banner's bare "DISABLED" claim must not appear on the
# Telemetry line specifically (the banner has an unrelated
# "Memory: DISABLED" line that must not make this assertion a
# false pass) -- that combination is exactly the false assurance
# this fix removes.
telemetry_line = next(
line for line in result.output.splitlines() if line.strip().startswith("Telemetry:")
)
assert "DISABLED" not in telemetry_line
def test_banner_beacon_off_and_local_off_says_fully_off(self, runner, monkeypatch):
"""Only when BOTH switches are off does the banner claim nothing is
happening -- the one case where a bare OFF claim is actually true."""
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
monkeypatch.setenv("HEADROOM_BEACON", "off")
from headroom.cli.main import main
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
result = runner.invoke(main, ["proxy"])
assert "Telemetry:" in result.output
assert "OFF" in result.output
assert "compression stats" not in result.output
# ---------------------------------------------------------------------------
# wrap CLI telemetry notice
@@ -268,14 +331,53 @@ class TestWrapCLITelemetryNotice:
@pytest.mark.asyncio
class TestStatsEndpointTelemetryFlag:
"""The /stats endpoint must expose anon_telemetry_shipping."""
"""The /stats endpoint's anon_telemetry_shipping field must reflect the
live HEADROOM_BEACON state, not a hardcoded constant.
Previously this field was hardcoded to False on the premise that "the
anonymous telemetry beacon was removed" -- it was not (telemetry/beacon.py
and telemetry/session.py fully implement and wire it via
record_outcome() -> SessionAggregator -> a POST to
headroom-beacon.headroom-beacon.workers.dev, gated on is_beacon_enabled(),
which is ON BY DEFAULT). An operator polling /stats to confirm nothing
ships externally got a false assurance regardless of their actual
HEADROOM_BEACON setting. These two tests previously asserted `False`
unconditionally, encoding the same incorrect premise as the field they
were testing.
"""
pytest.importorskip("fastapi")
async def test_stats_anon_telemetry_shipping_always_false(self, monkeypatch):
# The anonymous telemetry beacon was removed, so nothing is ever shipped
# externally — even with telemetry explicitly enabled.
async def test_stats_anon_telemetry_shipping_reflects_beacon_on(self, monkeypatch):
# HEADROOM_BEACON is on by default (BEACON_DEFAULT_ON = True) and
# HEADROOM_TELEMETRY does not touch it -- setting local telemetry on
# must not by itself make this field misreport False.
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
monkeypatch.delenv("HEADROOM_BEACON", raising=False)
monkeypatch.delenv("DO_NOT_TRACK", raising=False)
from headroom.proxy.server import ProxyConfig, create_app
app = create_app(
ProxyConfig(
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
)
)
from httpx import ASGITransport, AsyncClient
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
resp = await client.get("/stats")
assert resp.status_code == 200
data = resp.json()
assert "anon_telemetry_shipping" in data
assert data["anon_telemetry_shipping"] is True
async def test_stats_anon_telemetry_shipping_false_when_beacon_off(self, monkeypatch):
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
monkeypatch.setenv("HEADROOM_BEACON", "off")
from headroom.proxy.server import ProxyConfig, create_app
app = create_app(
@@ -296,8 +398,10 @@ class TestStatsEndpointTelemetryFlag:
assert "anon_telemetry_shipping" in data
assert data["anon_telemetry_shipping"] is False
async def test_stats_includes_anon_telemetry_shipping_false(self, monkeypatch):
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
async def test_stats_anon_telemetry_shipping_false_when_do_not_track(self, monkeypatch):
# DO_NOT_TRACK silences the beacon regardless of HEADROOM_BEACON.
monkeypatch.setenv("DO_NOT_TRACK", "1")
monkeypatch.delenv("HEADROOM_BEACON", raising=False)
from headroom.proxy.server import ProxyConfig, create_app
app = create_app(
@@ -315,7 +419,6 @@ class TestStatsEndpointTelemetryFlag:
assert resp.status_code == 200
data = resp.json()
assert "anon_telemetry_shipping" in data
assert data["anon_telemetry_shipping"] is False