mirror of
https://github.com/alphaXiv/OpenResearch.git
synced 2026-11-11 09:59:34 +08:00
Daniel Kim
08d30a8e61
* Add anonymous, opt-out usage telemetry (PostHog)
`orx` shipped with no telemetry, so there was no way to measure installs,
DAU/WAU, retention, or command usage. This adds anonymous, opt-out analytics
sent to a dedicated PostHog project.
What's collected: a per-invocation `cli_command` event and explicit
`experiment_started` events (create + run, with a coarse compute target),
carrying only a random per-install UUID, CLI version, os/arch, and CI flag.
Never any code, prompts, file contents, ids, names, paths, or tokens.
Design:
- New `src/telemetry.rs`: settings (install id + opt-out) at
`config_dir()/settings.json`, opt-out policy, event send, and a
fire-and-forget session modeled on `updates::UpdateWarning`.
- Opt-out: `DO_NOT_TRACK`, `ORX_TELEMETRY=off`, `CI`, a `--no-telemetry`
global flag, and a persistent `orx telemetry on|off|status` subcommand.
A one-time disclosure prints after the first `orx login` (shown even when
disabled, so it's never permanently suppressed).
- Never blocks or crashes the CLI: events are spawned and flushed once at
exit within a bounded window, so success output is never delayed; all
errors are swallowed.
- Cross-process safe: settings writes use an `fd_lock` advisory lock plus
atomic temp+rename, so a concurrent write can't revert a persisted opt-out.
- musl-safe: reuses the rustls `reqwest` client; no new TLS/C dependency.
Events land at PostHog `/i/v0/e/` with a public write-only `phc_` key and a
client-side ISO-8601 timestamp. `$process_person_profile:false` keeps events
anonymous. Unit tests cover opt-out precedence, id stability, payload shape
(PII-free), timestamp math, corrupt-file fail-safe, and the non-blocking
flush path.
* Prefix all CLI events with `cli_` to separate from website analytics
This PostHog project is shared with the website/cloud-agent analytics, so CLI
events must be distinguishable by event name alone — relying only on the
`source: "cli"` property is fragile (one forgotten dashboard filter leaks CLI
events into website metrics).
`build_payload` now stamps a `cli_` prefix onto every event name centrally, so
every current and future CLI event is `cli_*` by construction. Wire names:
- `cli_command` (was `cli_command`)
- `cli_experiment_started` (was `experiment_started`)
Added `every_event_name_is_cli_prefixed` to lock the guarantee.
* Telemetry: trim opt-out surface to --no-telemetry + `orx telemetry off`
Drop the `DO_NOT_TRACK`, `ORX_TELEMETRY`, and `CI` environment-variable
opt-out/disable checks. They were redundant with the two mechanisms we keep:
the `--no-telemetry` per-run flag and the persistent `orx telemetry off`.
The `CI` auto-disable in particular guarded a scenario that's mostly
hypothetical for an interactive research CLI, and it only ever fired for
hosted CI providers that set `CI` (missing Jenkins, agent boxes, and ad-hoc
scripts — the automation `orx` actually runs under). Every event still carries
a `ci` property, so automated runs can be filtered at query time in analytics
without being dropped up front.
Updates the opt-out policy, first-login notice text, `--no-telemetry` help,
README, and tests accordingly.
* orx up: add usage-analytics step to onboarding
Adds a third onboarding step ("Usage analytics") to the `orx up` dashboard
first-run walkthrough, after the coding-agents and git/GitHub steps. It shows
what is and isn't collected and lets the user toggle anonymous analytics On/Off
before creating their first project. On by default (matches the CLI's opt-out
model); the choice persists via the same `telemetry_disabled` setting as
`orx telemetry off`.
- Backend: `GET/POST /api/settings/telemetry` in up.rs, reusing
telemetry::disabled_reason / set_persisted_disabled (mirrors the git-settings
handler pattern). Returns `{ enabled, reason }`.
- UI: getTelemetry/setTelemetry in api.ts; Onboarding.tsx grows to a 3-step
flow with a TelemetryCard (On/Off segmented control).
Verified end-to-end in orx up: toggling Off in the wizard persists
`telemetryDisabled: true` to settings.json; toggling On clears it.
* Telemetry: drop the CLI login stderr disclosure
The one-time `orx login` stderr notice (show_notice_once) is removed now that
the disclosure lives in the `orx up` onboarding step, which is the right place
for it. Also removes the now-unused `notice_shown` settings field and updates
the module doc + README to point at the onboarding step instead of a login
notice.
* Telemetry: record consent decisions (cli_telemetry_consent)
Adds one event, `cli_telemetry_consent { agreed: bool }`, that records whether
the user opts into or out of analytics — so we can count agrees vs rejects.
Unlike every other telemetry event, this one is UNCONDITIONAL: it bypasses the
opt-out check and the persisted install id, and uses a fresh anonymous
distinct_id per call. If it honored the opt-out, every rejection would be
invisible and the ratio would be meaningless.
Fired once when the user leaves the analytics choice:
- `orx up` onboarding: on "Create your first project" (records the final On/Off
state, so users who accept the default On are counted too) via a dedicated
POST /api/settings/telemetry/consent endpoint.
- CLI: on `orx telemetry on` / `orx telemetry off`.
Tests: consent payload shape + name prefix, and that record_consent sends even
when persisted-off without generating an install id.
* Bump version to 0.1.57
Ships the anonymous opt-out telemetry subsystem, the orx up onboarding
usage-analytics step, and the cli_telemetry_consent event. Merging this PR
cuts release v0.1.57.