mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-23 02:49:17 +08:00
feat/https-compose-caddy
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
1a40b3ce67 |
fix(telemetry): let HEADROOM_TELEMETRY=off stop the anonymous beacon (#4235)
## Description
Make an explicit HEADROOM_TELEMETRY=off or --no-telemetry choice stop
the anonymous upload beacon as well as local telemetry, while preserving
the separately configured HEADROOM_BEACON=on override and existing
offline/DO_NOT_TRACK policy.
Existing persistent manifests already contain explicit off values
written by the old installer defaults. Those installs stay opted out
after upgrading unless deliberately reconfigured or reinstalled. The PR
does not guess which saved off values were intentional. New installs
omit an unspecified telemetry choice so they retain the existing default
beacon policy.
## Type of Change
- [x] Bug fix (telemetry opt-out behavior)
- [x] Documentation update
## Changes Made
- Apply an explicit local-telemetry opt-out to the beacon enable
decision and report the reason in telemetry status.
- Recheck enablement before sending buffered session telemetry.
- Preserve unspecified, enabled, and disabled telemetry choices when
generating installation manifests.
- Update operator documentation and regression coverage.
## Testing
- [x] Maintainer telemetry, beacon, and snapshot suites passed after the
current-main sync.
- [x] Whitespace check passed.
### Test Output
```text
python -m pytest tests/test_telemetry_warning.py tests/test_proxy_telemetry_env.py tests/test_beacon_schema_v2.py tests/test_beacon_gzip.py tests/test_original_snapshot_in_place_hook.py -q
122 passed, 1 warning in 21.19s
```
A previous broader installer run also reproduced the persistent-service
embedder assertion on current main independently; that baseline failure
was not attributed to this PR. The other 34 planner tests passed in that
qualification. Hosted CI is qualifying head
|
||
|
|
2e2758de95 |
Fix macOS install-native CI failure from stale restart manifest test stub (#3779)
The `install-native (macos-latest)` Actions job was failing in the
restart smoke test because the test-local manifest stub no longer
matched the fields required by the current install lifecycle code. This
updates the stub so the workflow exercises the real restart path instead
of failing on fixture drift.
- **Root cause**
- `test_install_restart_waits_for_old_process_before_starting` used a
minimal `Manifest` stub that was missing fields now consumed during
managed-env reconciliation after restart.
- **Change**
- Extend `_restart_race_manifest()` in
`tests/test_cli/test_install_cli.py` to include the current deployment
fields expected by restart:
- `port`
- `backend`
- `targets`
- `tool_envs`
- **Why this scope**
- Keeps the fix test-only.
- Preserves production behavior; the failure was caused by an outdated
fixture, not runtime logic.
```python
class Manifest:
profile = "default"
preset = "persistent-service"
runtime_kind = "python"
supervisor_kind = "service"
scope = "user"
port = 8787
health_url = "http://127.0.0.1:8787/readyz"
backend = "anthropic"
targets: list[str] = []
tool_envs: dict[str, dict[str, str]] = {}
mutations: list = []
```
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
fa9edb3a29 |
fix(cli): report anonymous beacon status accurately in the banner, log, and /stats (#3533)
## Description `HEADROOM_TELEMETRY` (local aggregate stats, opt-in, off by default) and `HEADROOM_BEACON` (the anonymous session-summary upload beacon — `telemetry/beacon.py`: `BEACON_DEFAULT_ON = True`, opt-**out** by default) are two independent switches, by the module's own docstring. Three places only ever checked the first one and never mentioned the second: - The proxy CLI startup banner (`cli/proxy.py`) printed a bare `Telemetry: DISABLED` whenever `HEADROOM_TELEMETRY`/`--no-telemetry` was off, with no mention that the beacon — the switch that actually ships data externally — was still active by default. - A startup log line (`proxy/server.py`) carried a comment and message claiming telemetry is "collected locally only (the anonymous telemetry beacon was removed)". This isn't true of the current codebase: `telemetry/beacon.py` and `telemetry/session.py` fully implement and wire the beacon end to end via `record_outcome()` → `SessionAggregator` → a POST to `headroom-beacon.headroom-beacon.workers.dev`. - The `/stats` API's `anon_telemetry_shipping` field was hardcoded to `False` on the same premise, so a caller polling `/stats` to confirm nothing ships externally got a false assurance regardless of their actual `HEADROOM_BEACON` setting. Found auditing this deployment in front of Claude Code on a home server, alongside #2683/#2684 (`SECURITY.md`'s credential-storage claim) — a related but distinct gap: that PR made the true facts about the beacon *discoverable* in a new docs page (and states explicitly "no code changes"); this PR fixes the three places where the running proxy *actively asserts something false* on its own status surfaces. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/proxy.py`: the startup banner now delegates to the existing `format_telemetry_notice()` helper (`headroom/telemetry/beacon.py`), which already gets the beacon-vs-local distinction right and is already covered by `TestFormatTelemetryNotice` — instead of re-deriving (and re-drifting from) the same wording a second time. - `headroom/proxy/server.py`: the startup log line and the `/stats` `anon_telemetry_shipping` field now call `is_beacon_enabled()` for real instead of a hardcoded value / a comment asserting the beacon was removed. - `tests/test_telemetry_warning.py`: updated `TestProxyCLITelemetryBanner` and `TestStatsEndpointTelemetryFlag` (see Testing below for why). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_telemetry_warning.py -v ... 40 passed in 0.84s $ python -m ruff check headroom/cli/proxy.py headroom/proxy/server.py tests/test_telemetry_warning.py All checks passed! $ python -m ruff format --check headroom/cli/proxy.py headroom/proxy/server.py tests/test_telemetry_warning.py 3 files already formatted $ python -m mypy headroom/cli/proxy.py headroom/proxy/server.py --ignore-missing-imports Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS (aarch64-darwin), Python 3.14.7, `pip install -e ".[proxy]"`, pytest 9.1.1, `click.testing.CliRunner` for the CLI banner assertions. - Exact command / steps: ran the full existing `tests/test_telemetry_warning.py` suite before touching any test, confirming every existing `TestProxyCLITelemetryBanner` test only ever set `HEADROOM_TELEMETRY` and never `HEADROOM_BEACON` — meaning they passed identically whether the beacon was on or off, which is how this shipped uncaught. Updated those six tests to pin `HEADROOM_BEACON=off` (isolating the local-only wording they were written to test, matching the convention `TestFormatTelemetryNotice` already uses a few classes above), added two new tests for the beacon-on-by-default case, and rewrote the two `/stats` tests that previously asserted `anon_telemetry_shipping is False` unconditionally under a comment stating the same "beacon was removed" premise as the code it was testing. - Observed result: all 40 tests in the file pass against the patched code. Reverting the source changes only (keeping the updated tests) reproduces real failures — the two `TestProxyCLITelemetryBanner` beacon-specific tests and both `TestStatsEndpointTelemetryFlag` tests fail against the pre-fix banner/log/`/stats` code, confirming the tests actually exercise the fix rather than passing vacuously. - Not tested: the actual outbound POST to the beacon endpoint (out of scope: this PR changes what the proxy *reports about* the beacon's state, not the beacon's own send/receive logic, which is unchanged and untouched here). (`mypy headroom/cli/proxy.py headroom/proxy/server.py --ignore-missing-imports`, pinned to the CI toolchain versions — `mypy==1.20.2`, `numpy==2.2.6` from `uv.lock` — now run and clean; see Testing → Test Output above.) ## Runtime Rollout Safety - Rollout-managed feature(s): none — this changes status-reporting strings and a boolean field, not any behavior gated by a rollout flag. - Minimum rollout channel: n/a - Stable/default behavior changed: the *reported* telemetry status changes (an operator who set only `--no-telemetry` now sees the beacon disclosed instead of a bare "DISABLED"); the actual telemetry/beacon behavior itself is unchanged by this PR. - Kill switch / disable path: unaffected by this PR — `HEADROOM_BEACON=off` / `DO_NOT_TRACK=1` continue to work exactly as before. - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit; the banner/log/`/stats` field return to their previous (inaccurate) wording. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation — n/a beyond the code comments themselves; #2684 already covers the docs-page side of this gap - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Companion to #2683/#2684 (thank you @keshavlingala for the docs half of this) — happy to rebase on top of that once it lands if there's any overlap in `SECURITY.md`, though this PR doesn't touch that file. This is one of two small, independently-scoped fixes from the same audit pass; the other (an unhandled `ClientDisconnect` crash in `handle_anthropic_messages`) is a separate PR per the "one logical change per PR" guidance in CONTRIBUTING.md. Opening as a draft while I give it a final look — will mark ready for review shortly. --------- Co-authored-by: Tejas Chopra <chopratejas@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
67eb910e38 |
feat(beacon): schema v2 — routing/compression training signal, gzip transport (#3253)
## What Nine new sections on the session heartbeat — `quality`, `hist`, `shapes`, `cache`, `trajectory`, `clients`, `errors`, `strata`, `config` — plus a gzip transport. Everything is a counter, a ratio, or a slug from a fixed list in the source. No prompts, code, file paths, tool names, project names, or error text can appear. ## No regression in v1 Every existing field keeps its name, type, and meaning. `tests/test_beacon_schema_v2.py` locks the v1 surface with literal key lists and a shape-stability invariant, so a v1 field that silently changes type fails there rather than in the corpus six weeks later. The v2 section is built in `_payload_v2()` behind `try/except`, and `_fold_extra()` likewise. A bug in new code can cost the new fields and nothing else. The first version built one dict literal — any raise dropped `tokens.saved` along with it. ## The STRUCT/MAP flip Four fields ship as lists of records rather than objects keyed by data (`clients`, `config`, `strata`, `waste`). DuckDB infers a JSON object as `STRUCT` when the keys are few and identifier-shaped, and as `MAP` otherwise — so a key set that grows with the fleet flips type mid-corpus and `BINDER ERROR`s the reader. Caught against the real 369k-session corpus, not in theory: `clients` inferred `STRUCT(claude_code, codex)` at two harnesses and `MAP(VARCHAR, BIGINT)` at 22, and `clients.cursor` errors on one of them. ## Transport Body is gzipped at level 6 (~6x). The receiver sniffs magic bytes rather than trusting `Content-Encoding`, enforces a decompression-bomb limit *during* the stream read, and leaves the plain path byte-identical for the existing fleet. A `400`/`415` disables gzip for the process and retries once. A `413` does not — an uncompressed retry is 6x larger and guaranteed to fail again. At current rates (~235k heartbeats/day): v1 today 892 MB/day, v2 uncompressed 1,802 MB/day, **v2 gzipped 295 MB/day** — a net reduction against what we send now. ## `headroom telemetry` Prints the real payload, built by the same `_Session.payload()` the beacon uses. \"Go read worker.js\" answers what happens to the data; it does not answer what leaves your machine. Uses `read_install_id()` and `resource_attributes(create_install_id=False)` — the default mints and persists the id, so building the report would itself be the act that identifies the machine. Filtering the id out of the *output* is not enough; by then the file exists. ## Receiver Allowlist gains the nine keys. `MAX_BODY_BYTES` 64KB → 256KB — that figure dated from \"a beacon event is ~2KB\", and worst-case v2 is 81,965 B uncompressed. ## Verification Against the production corpus (369,015 sessions): **14/14** `beacon.sh` reports pass, **18/18** dashboard queries pass, v1 totals byte-identical at `273,495,971,478` tokens saved. `sample-event.json` regenerated as a real v2 payload. The old v1-only version would have passed the post-deploy smoke test whether or not the new allowlist landed; the new one distinguishes the cases 9/9 vs 0/9. Suite: 11,655 passed, 1 failed — `test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`, which reproduces identically on clean `main`. ## Not verified here `node test-rollup.mjs` and `npx wrangler dev` have not been run — no `node` in the authoring environment. `DecompressionStream` is the one path with no fallback and has never been executed. **Both are required before deploy.** ## Deploy order **Worker first, client second.** The worker is backward-compatible with the v1 fleet (allowlist additive, plain path byte-identical, cap only raised). The reverse is not: the worker strips unknown keys at ingest, so v2 data arriving before the allowlist lands is gone, not recoverable by re-rolling. The corpus already carries 6 such rows from a dev proxy. Merging this PR does not ship the client — releases are separate commits. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
9cfb00838a |
fix(telemetry): anonymous compression stats — no prompts, no data (#2728)
## In one line
Headroom starts reporting **how well compression is working** — counters
and percentages only. **No prompts. No code. No file paths. Nothing
about what you're building.**
## Why
Right now nobody knows whether compression actually helps real users.
You can see your own numbers in `/stats`, but that's it — there's no way
to tell whether a given workload compresses well, or why it sometimes
doesn't. This closes that loop so we can make compression better for
everyone.
## Exactly what gets sent
One message per session, and every 5 minutes while you're active:
```json
{
"session": { "id": "random", "turns": 47, "duration_s": 4210, "seq": 3 },
"tokens": { "original": 890000, "attempted": 410000, "saved": 320000,
"tool_saved": 48000, "cache_read": 210000 },
"rates": { "saved_pct": 35.96, "eligible_pct": 46.07, "yield_pct": 78.05,
"cache_read_pct": 23.60, "overhead_pct": 1.96 },
"compression": { "transforms": {"crush": 47}, "passthrough_turns": 0 },
"skips": {},
"sources": { "proxy": 47 },
"providers": ["anthropic"],
"models": ["claude-sonnet-4-5-20250929"],
"failures": 2
}
```
Plus a random install ID, the Headroom version, and OS/architecture
(`darwin`, `arm64`).
That's the whole thing. A full example lives at
`deploy/beacon/sample-event.json`.
## What is never sent
- Your prompts or the model's responses
- Your code
- File paths, project names, repo names
- Tool names or MCP server names
- Hostname, username, or IP address
- Custom or fine-tuned model names (an id like `ft:gpt-4o:acme-corp:…`
contains a company name, so only models in a public registry are
reported)
**This is structural, not a pinky-swear.** Every value in the payload is
a number, a fixed word, or a random ID — there is no free-text field
anywhere for content to hide in. The receiver
(`deploy/beacon/worker.js`, in this repo so you can read it) drops
anything not on an explicit allowlist before storing.
## Turning it off
Any one of these:
```bash
HEADROOM_BEACON=off # or
DO_NOT_TRACK=1 # or
# offline mode
```
It's on by default, and Headroom says so at startup:
```
Telemetry: anonymous compression stats — never prompts, code, or file paths.
Helps us improve compression | Off: HEADROOM_BEACON=off
```
`HEADROOM_TELEMETRY` is a **separate** switch that still only affects
local stats. If you had turned that on, this change does not start
uploading anything — you answered a different question, and upgrading
should not change the answer.
## Why the percentages, not just "tokens saved"
"We saved 36%" hides the interesting part. In the example above only
**46% of tokens were eligible** for compression at all — the rest is
frozen cache prefix and system prompts we deliberately do not touch. Of
what we *could* touch, we removed **78%**.
Those are two separate problems. Raising eligibility is proxy work;
raising yield is compressor work. A single number cannot tell us which
to fix.
## Coverage
`emit_request_outcome` is a single chokepoint —
`handler.metrics.record_request` is called from exactly one place,
inside the funnel — so all 30 `RequestOutcome` construction sites are
covered: Anthropic, OpenAI, Gemini, Bedrock, batch, streaming, and the
long-lived Codex Responses-WS path.
The `headroom_compress` MCP path bypassed that funnel and is now wired
in separately. It has a different shape (no provider, no upstream
latency, and everything handed to the tool is eligible by construction),
so `sources` counts turns by origin — MCP turns always read
`eligible_pct: 100` and must not drag the proxy's real eligibility
ceiling upward.
**Subagents.** All subagent traffic through the proxy merges into one
session, which is correct for savings and retention but means `turns`
conflates fan-out with depth. Fan-out is still derivable —
`compression.latency_ms_total / session.duration_s` gives the
concurrency ratio (~1x serial, ~4x for four parallel agents), so no
extra field is needed. Verified no lost updates under 6-way concurrency
(1,200 turns).
**Known gap:** `--workers N` gives each process its own aggregator, so
one user session becomes up to N. Token totals and fleet rates stay
correct; session counts inflate. This matches the existing documented
limitation that TOIN state, CostTracker, and the prefix tracker are all
per-process.
## Notes for reviewers
- **Cumulative snapshots, not deltas.** Every report restates running
totals under one session ID, so the highest `seq` per `(install,
session)` is the complete session. Dedupe is a window function, and a
lost report costs nothing.
- **Never breaks the proxy.** Every path swallows its own exceptions;
uploads go out on a daemon thread so nothing blocks the request loop.
- **Explicit User-Agent is load-bearing.** urllib's default is blocked
by Cloudflare (error 1010). Combined with fire-and-forget error
handling, that would have failed every upload while looking perfectly
healthy.
- **The exit flush was broken and is fixed.** `atexit` handed the POST
to a daemon thread, and daemon threads are killed before they finish
during interpreter shutdown — so nothing was sent. That silently dropped
*every session shorter than the 5-minute heartbeat*, plus all
short-lived subagent MCP processes. The exit path now posts
synchronously with a 2s timeout.
- Receiver and query tooling are in `deploy/beacon/`.
## Testing
- `python -m headroom.telemetry.session` self-check: dedupe, cumulative
totals, dropped-report recovery, payload contains no model id or
prompt-derived string, allowlist coverage
- 175 telemetry/outcome tests pass; 6 new ones cover the opt-out notice
- Verified end to end against a live deployment: client → receiver →
storage → query
## Still to do before release
The default endpoint currently points at a temporary `workers.dev` URL.
It needs to move to a Headroom-owned hostname before this ships in a
tagged release — noted inline at `DEFAULT_ENDPOINT`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
e0ce4b1d48 |
fix: remove rtk and lean-ctx CLI context tools (#2677)
## Description Removes both third-party CLI context tools — **rtk** and **lean-ctx** — and with them the context-tool selector itself. Headroom no longer downloads, installs or configures either one, and there is no replacement. The previous pass (#2344) gated only three entry points inside `headroom/cli/wrap.py`. That left the feature reachable in practice: | Gap | Effect | |---|---| | `scripts/install.sh:1544`, `install.ps1:1681` | Ran `rtk init --global --auto-patch` from bash/PowerShell, **bypassing the Python gate entirely** — `curl \| sh` still wrote a Claude Code `PreToolUse` hook regardless of `HEADROOM_RTK` | | `wrap.py` `_setup_context_tool_for_agent` | **`wrap openhands` was broken by default**: `rtk_required=True` met a gate returning `None` → `SystemExit(1)`. Invisible because all 8 openhands tests patched `_ensure_rtk_binary` to a fake path | | `proxy/helpers.py`, `subscription/tracker.py` | Proxy shelled out to `rtk gain` from `/stats`, the dashboard and `headroom perf`; the tracker polled it per contribution (`_RTK_WIRING_DEFAULT = "enabled"`) | | No cleanup path | Nothing removed artifacts an earlier default had installed, so a machine that once ran the old default kept rtk in the loop forever (#1669, #1955) | Also worth noting: the rtk binary download had **no SHA or signature verification** — only `rtk --version` as a smoke test. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made **Removed** — `headroom/rtk/` and `headroom/lean_ctx/` packages, `headroom/cli/wrap_rtk_metrics.py`, `_selected_context_tool` / `_setup_context_tool_for_agent` / `_VALID_CONTEXT_TOOLS`, the `--rtk` / `--no-rtk` / `--no-project-rtk` / `--keep-rtk` flags across all 18 wrap subcommands, `HEADROOM_RTK*`, the proxy-side `rtk gain` polling, the dashboard CLI-filtering panel (rows + all 8 `cliFiltering*` Alpine getters), `paths.rtk_path()` / `lean_ctx_path()`, the SDK path helpers, `benchmarks/rtk_loop_learn_eval.py`, and the `headroom/rtk/**` CI path filters. **Fails loudly, not silently** — `--context-tool` / `--no-context-tool` / `HEADROOM_CONTEXT_TOOL` are kept solely to error out. They live in shell profiles, aliases and CI jobs, and accepting them as a no-op would read as Headroom having quietly stopped working. The installers reject them too, which matters more than it looks: their arg parsers forward the first unknown flag **and everything after it** to the wrapped tool, so a leftover `--no-rtk` would have silently swallowed a following `--port` and then been ignored downstream. **New `headroom/context_tool_cleanup.py`** — deleting the code cannot help a machine that already ran the old default, since the hooks, binaries and injected guidance are durable on disk. `purge_context_tool_artifacts()` runs once per `wrap`/`unwrap` and removes the registered hook entries, the generated hook scripts, the Headroom-managed `~/.local/bin` symlinks, the vendored `~/.headroom/bin/{rtk,lean-ctx}` binaries, the `lean-ctx` MCP server entry and the marker-fenced instruction blocks. Deliberately conservative: idempotent, **skips** a malformed config rather than overwriting it, and only unlinks a symlink resolving inside Headroom's own bin dir so a user's own build is untouched. It reports on **stderr**, because `wrap/unwrap openclaw --prepare-only` emit machine-readable JSON on stdout as their entire contract. Skipped for `wrap selfheal` (runs from a SessionStart hook; must not race Claude Code's writer for `~/.claude.json`) and for `--help`, which must stay read-only. **Client-config hardening** (discovered while investigating a "corrupted Serena settings file" report) — `wrap.py` reset a settings file to `{}` when an existing file would not parse, then wrote that back. One hand-edited typo or a transient `EACCES`/`EINTR` on a valid file destroyed the user's `permissions`, `env` and `hooks`, on **every `headroom wrap claude`**. It now refuses to write. Separately, `fsutil.write_text` is now atomic (temp file + `fsync` + `os.replace`), fixing all 14 non-atomic client-config writes at once; it follows symlinks rather than replacing them (dotfile managers) and preserves an existing file's mode. **Deliberately kept** — `rtk` stays in the wrapper-peel list in `transforms/content_router.py`. It sits beside `sudo`/`env`/`timeout` as shell-command grammar, so `rtk cat f` is still classified as a file read for anyone running their own rtk install, which the purge intentionally leaves alone. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates All checks passed! $ ruff format --check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates 1255 files already formatted $ mypy headroom/ Success: no issues found in 508 source files $ pytest tests/test_context_tool_cleanup.py -q 11 passed $ pytest tests/test_fsutil.py -q 12 passed $ pytest tests/test_cli/test_wrap_codex.py -q # 89 tests 89 passed in 431.68s $ pytest tests/test_cli/test_wrap_opencode.py -q 39 passed in 257.46s $ pytest tests/test_cli/test_wrap_helpers.py -q 45 passed $ pytest tests/test_paths.py -q 75 passed $ pytest tests/test_cli/test_unwrap_claude.py -q 14 passed $ pytest tests/test_proxy_savings_history.py -q 39 passed $ pytest tests/test_cli/test_wrap_copilot.py -q 27 passed $ pytest tests/test_cli/test_wrap_zcode.py -q 20 passed $ pytest tests/test_subscription_tracker.py -q 9 passed $ pytest tests/test_proxy_dashboard_stats_cache.py -q 5 passed, 1 skipped ``` Repo-wide grep for 14 removed symbols (`headroom.rtk`, `headroom.lean_ctx`, `_ensure_rtk_binary`, `_selected_context_tool`, `_get_context_tool_stats`, `rtk_path`, `lean_ctx_path`, `wrap_rtk_metrics`, `HEADROOM_RTK`, `cli_tokens_avoided`, `tokens_saved_rtk`, …) across `*.py`, `*.ts`, `*.sh`, `*.ps1`, `*.yml`, `*.html`: **zero hits**. Notable test changes: `test_wrap_openhands.py` no longer patches `_ensure_rtk_binary` and asserts `wrap openhands --prepare-only` exits 0 unpatched — the regression that was previously masked. `test_wrap_continue.py` and `test_wrap_hintfile_agents.py` were removed (every test drove RTK instruction injection). A new `test_subscription_tracker.py::test_load_state_written_before_cli_context_tools_were_removed` proves a pre-removal `subscription_state.json` still loads. ## Real Behavior Proof - **Environment:** macOS 15.4 (darwin 25.4.0), Python 3.12.6, Headroom @ this branch, real `~/.headroom` and `~/.claude` on the dev machine. - **Exact command / steps and observed result:** ```text # 1. Retired flag fails loudly instead of silently no-op'ing $ headroom wrap codex --prepare-only --context-tool rtk Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: they rewrote shell commands through a third-party binary Headroom no longer manages. Drop --context-tool / --no-context-tool and unset HEADROOM_CONTEXT_TOOL; `headroom wrap` uninstalls what they left behind on first run. $ HEADROOM_CONTEXT_TOOL=lean-ctx headroom wrap codex --prepare-only Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: ... # 2. install.sh rejects the retired flags (extracted parse_wrap_args harness) ['--no-rtk', '--port', '9999'] rc=1 ERROR: CLI context tools ... Drop --no-rtk ['--context-tool=rtk'] rc=1 ERROR: CLI context tools ... Drop --context-tool $ bash -n scripts/install.sh # syntax OK # 3. Purge ran against the real machine, which had all the orphaned artifacts $ python -c "from headroom.context_tool_cleanup import purge_context_tool_artifacts; ..." removed ~/.headroom/bin/lean-ctx (51 MB) removed ~/.headroom/bin/rtk (7.7 MB) removed ~/.local/bin/rtk (symlink into ~/.headroom/bin) removed ~/.claude/hooks/rtk-rewrite.sh removed 8 lean-ctx-* hook scripts # ~/.claude.json afterwards: 90 top-level keys, 19 projects, mcpServers unchanged # → ~59 MB reclaimed, no unrelated key touched # 4. stdout stays machine-readable while the purge reports (planted a fake artifact) $ headroom wrap openclaw --prepare-only --gateway-provider-id codex >out 2>err $ cat out {"enabled":true,"config":{"proxyPort":8787,...}} # parses as JSON $ cat err Retired CLI context tool cleanup: removed /Users/tcms/.headroom/bin/rtk # 5. --help is inert (planted artifact survives), a real run purges $ headroom wrap codex --help → artifact survived: CORRECT $ headroom wrap openclaw --prepare-only → purged: CORRECT # 6. MCP purge dry-run against a copy of the real 82 KB ~/.claude.json top-level keys 90 -> 90; projects 19 -> 19; LOST keys: none all content outside mcpServers byte-identical: True ``` Dashboard rendered via the Playwright test after the panel removal: "Token Savings" shows only `Proxy 0 (0.0%)` / `Of total wire: 36.86%`, and "Token Usage" reads Before Compression → Proxy Removed → After Compression with no "Filtered (this session)" row. Nothing below the removed panel broke. - **Not tested:** Windows and Linux (macOS only) — `install.ps1` is verified by brace-balance and inspection, not executed, since no `pwsh` is available locally. The wrap e2e suite (`e2e/wrap/run.py`) was updated but not run; it needs the Docker e2e image. `serena project index` interaction is exercised in the stacked base PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes **Stacked on #2676** (`tejas/serena-config-bootstrap`) — please merge that first; this PR's base should then be retargeted to `main`, or it will read as containing that fix too. **Breaking-change migration for users:** - Drop `--rtk`, `--no-rtk`, `--no-project-rtk`, `--keep-rtk`, `--context-tool`, `--no-context-tool` from any alias, script or CI job, and unset `HEADROOM_RTK*` / `HEADROOM_CONTEXT_TOOL`. They now error rather than being ignored, so the failure is immediate and self-explaining. - Previously-installed artifacts are purged automatically on the next `wrap`/`unwrap`; no manual cleanup needed. - `headroom perf --json` no longer carries a `cli_filtering` key, and `/stats` no longer returns a `context_tool` section. **Docs:** `docs/rtk-architecture.md` deleted; RTK/lean-ctx removed from `README.md`, `docs/content/docs/{configuration,opencode,grok-build,docker-install,filesystem-contract}.mdx`, `docs/observability.md` and the matching `wiki/` pages. `REALIGNMENT/09-phase-G-rtk-observability.md` is marked SUPERSEDED rather than deleted, to keep the planning record. **Follow-ups not in scope:** `_emit_wrap_interrupted` was deleted as dead code — its only caller was the `except KeyboardInterrupt` guarding the binary download, so with no download there is nothing slow left to interrupt. |
||
|
|
53be64ca12 |
chore(telemetry): remove Supabase anonymous beacon; fix contact domain to headroomlabs.ai (#1526)
## Description Removes the anonymous-telemetry **beacon** — the only external, third-party data flow Headroom ever initiated. When telemetry was opted in, it POSTed aggregate `/stats` to a hardcoded **Supabase** REST endpoint (with an embedded anon API key in the source). For enterprise/on-prem deployments this is exactly the kind of vendor-controlled data egress a security review flags, so it's gone entirely — **zero "Supabase" references remain in the codebase.** What stays (by design): the **local** telemetry collector + the `HEADROOM_TELEMETRY` opt-in (it only feeds `/stats` and `/v1/telemetry` — nothing leaves the process), **OpenTelemetry export** (`HEADROOM_OTEL_METRICS_*`, so operators send operational metrics to *their own* collector), and the license usage reporter (your own domain, license-key-gated). Also fixes the contact domain: `headroom.dev` → `headroomlabs.ai` everywhere. Closes # (no tracking issue) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) > Non-breaking: `HEADROOM_TELEMETRY` is still accepted (now gates local collection only). The only behavior change is that no telemetry is ever sent externally. ## Changes Made - **Deleted the Supabase beacon**: `TelemetryBeacon` class, `_SUPABASE_URL`/`_SUPABASE_KEY`/`_TABLE`/`_ENDPOINT`, the JSONB projection helper, the proxy-lifespan beacon wiring, the `SUPABASE_` install env passthrough, and `tests/test_strategy_stats_supabase.py`. - **Kept** the local opt-in predicate (`is_telemetry_enabled` etc.) in `beacon.py` — still used by the local collector + CLI — reworded to "local only". - **Retained** the single-worker-owner file lock (the cc-switch reconciler depends on it); updated its comments to drop the beacon framing. - `/stats` `anon_telemetry_shipping` is now always `False` (nothing ships externally); startup log reworded to "Local telemetry". - Reworded remaining "Supabase" comments in `collector.py`, `context.py`, `prometheus_metrics.py`, and two test docstrings. - Contact domain: `security@headroom.dev` → `security@headroomlabs.ai`, `conduct@headroom.dev` → `conduct@headroomlabs.ai`, FUNDING.yml sponsor URL. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ grep -rniI "supabase" --include=*.py --include=*.md --include=*.mdx . # (excl .venv/sbom) >>> ZERO Supabase references $ grep -rniI "headroom.dev" . >>> ZERO headroom.dev references $ ruff check <changed files> -> All checks passed! $ ruff format --check <changed files> -> 10 files already formatted $ mypy <changed telemetry files> -> Success: no issues found $ pytest tests/test_telemetry.py tests/test_telemetry_warning.py \ tests/test_proxy_telemetry_env.py tests/test_compression_observability.py \ tests/test_paths.py tests/test_paths_backward_compat.py -q ============================= 173 passed in 6.67s ============================== ``` ## Real Behavior Proof - **Environment:** macOS, Python 3.12 (`.venv`). - **Exact command / steps:** repo-wide grep for `supabase`/`headroom.dev`; `create_app(...)` driven through a full `TestClient` lifespan (startup + shutdown) in `test_proxy_telemetry_env.py`; `/stats` exercised in `test_telemetry_warning.py`. - **Observed result:** zero `supabase`/`headroom.dev` strings remain; the proxy starts and shuts down cleanly with the beacon removed (the worker-owner lock + reconciler still elect a single owner); `/stats.anon_telemetry_shipping` is `False` even with `HEADROOM_TELEMETRY=on`; local collector + OTEL paths unchanged. - **Not tested:** no live network call was ever made (the point — the external POST is gone). OTEL export and the license reporter were not exercised (unchanged by this PR). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The license usage reporter (`reporter.py` → `app.headroomlabs.ai`) is intentionally **kept** — it's license-key-gated (dormant for unlicensed/OSS deployments) and goes to your own domain, not a third party. - Docs/CHANGELOG left unchecked: a couple of docs mention the telemetry beacon and may want a follow-up note that it now collects locally only; happy to add. |
||
|
|
b99869778b |
fix(telemetry): switch anonymous telemetry to opt-in (off by default) (#1223)
## Description
Anonymous usage telemetry was **on by default** (opt-out). This flips it
to **opt-in**: nothing is collected or shipped unless the user
explicitly turns it on. Small change, but it makes "no data leaves the
proxy by default" the actual default rather than something users have to
discover and disable.
Closes # <!-- N/A: no tracking issue -->
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality) — adds
`--telemetry` opt-in flag
- [x] Breaking change (fix or feature that would cause existing
functionality to change) — telemetry no longer runs unless opted in
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `is_telemetry_enabled()` is now **fail-closed**: only explicit
on-values (`on`/`true`/`1`/`yes`/`enable`/`enabled`) enable telemetry;
unset, empty, or unrecognized values stay disabled. This single
predicate gates both the Supabase beacon and the local `/v1/telemetry`
collector.
- Added `--telemetry` opt-in flag to `headroom proxy` and `headroom
install apply`; kept `--no-telemetry` and `HEADROOM_TELEMETRY=off` for
back-compat. If both are passed, opt-out wins.
- Install manifests now write `HEADROOM_TELEMETRY` explicitly
(`on`/`off`) plus the matching flag, so generated systemd/docker/launchd
deployments are unambiguous and don't rely on the runtime default.
- Startup banner and proxy log show `DISABLED` by default and surface
how to opt in.
- Updated tests for opt-in defaults; added coverage for the default-off
banner, the `--telemetry` flag, and the explicit-on manifest path.
- Updated docs
(proxy/configuration/installation/benchmarks/community-savings mdx, spec
011/015, wiki proxy/cli/benchmarks/metrics) and `CHANGELOG.md`.
## Testing
- [x] Unit tests pass (`pytest`) — targeted telemetry + install-planner
suites
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_telemetry_warning.py tests/test_telemetry.py tests/test_install/test_planner.py -q
81 passed
$ ruff check <changed source + test files>
All checks passed!
$ mypy headroom/telemetry/beacon.py headroom/cli/proxy.py headroom/cli/install.py \
headroom/install/planner.py headroom/proxy/server.py
Success: no issues found in 5 source files
```
## Real Behavior Proof
- **Environment:** macOS (darwin 25.4.0), project `.venv`, `headroom`
CLI.
- **Exact command / steps:**
```
$ python -c "import os; from headroom.telemetry.beacon import
is_telemetry_enabled; \
os.environ.pop('HEADROOM_TELEMETRY', None); print('unset ->',
is_telemetry_enabled()); \
[ (os.environ.__setitem__('HEADROOM_TELEMETRY', v), print(repr(v), '->',
is_telemetry_enabled())) \
for v in ['on','TRUE','1','yes','off','0','garbage',''] ]"
$ headroom proxy --help | grep -i telemetry
$ headroom install apply --help | grep -i telemetry
```
- **Observed result:**
```
unset -> False # off by default
'on' -> True 'TRUE' -> True '1' -> True 'yes' -> True
'off' -> False '0' -> False
'garbage' -> False '' -> False # fail-closed
proxy: --telemetry Opt in to anonymous usage telemetry — off by default
(env: HEADROOM_TELEMETRY=on)
--no-telemetry Force anonymous usage telemetry off (already the default;
env: HEADROOM_TELEMETRY=off)
install: --telemetry Opt in to anonymous telemetry in the runtime (off
by default).
--no-telemetry Force anonymous telemetry off in the runtime (already the
default).
```
- **Not tested:** live Supabase beacon network round-trip (no opt-in
network call was made); full `pytest` suite was not run — only the
telemetry + install-planner targeted suites.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- "Breaking change" is checked because the default behavior changes
(telemetry stops running unless opted in). It is **not** an API break —
`--no-telemetry` and `HEADROOM_TELEMETRY=off` still work, so existing
opt-out configs are unaffected.
- No tracking issue, so `Closes #` is left N/A.
|
||
|
|
4b061792b2 | feat: add lean-ctx context tool support | ||
|
|
217bf16d3c |
fix: format telemetry warning tests for CI
Rebase the anonymous telemetry transparency warning work onto the latest main branch and format the new telemetry warning tests so the Python 3.12 lint lane passes cleanly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
de311f7da4 |
feat: add anonymous telemetry warning across all user-facing paths
- Add is_telemetry_warn_enabled() feature flag (HEADROOM_TELEMETRY_WARN=off to suppress at build/pack time) and format_telemetry_notice() helper to beacon.py - Export new helpers from headroom.telemetry.__init__ - Proxy CLI startup banner now shows Telemetry: ENABLED/DISABLED with opt-out instructions when enabled - Log telemetry status at proxy server startup() so it appears in log stream - wrap CLI _launch_tool() and the bespoke claude wrap command both call _print_telemetry_notice() so users see the notice before the tool launches - /stats endpoint now includes anon_telemetry_shipping boolean flag - Dashboard header shows amber Anon Telemetry indicator chip when anon_telemetry_shipping is true (theme-matching, with tooltip showing opt-out) - 30 new tests covering all paths; all 82 tests pass Agent-Logs-Url: https://github.com/JerrettDavis/headroom/sessions/880a7bb3-3ad9-49f4-a0b6-3ffdde233e48 Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com> |