Daniel Kim
08d30a8e61
* Add anonymous, opt-out usage telemetry (PostHog)
`orx` shipped with no telemetry, so there was no way to measure installs,
DAU/WAU, retention, or command usage. This adds anonymous, opt-out analytics
sent to a dedicated PostHog project.
What's collected: a per-invocation `cli_command` event and explicit
`experiment_started` events (create + run, with a coarse compute target),
carrying only a random per-install UUID, CLI version, os/arch, and CI flag.
Never any code, prompts, file contents, ids, names, paths, or tokens.
Design:
- New `src/telemetry.rs`: settings (install id + opt-out) at
`config_dir()/settings.json`, opt-out policy, event send, and a
fire-and-forget session modeled on `updates::UpdateWarning`.
- Opt-out: `DO_NOT_TRACK`, `ORX_TELEMETRY=off`, `CI`, a `--no-telemetry`
global flag, and a persistent `orx telemetry on|off|status` subcommand.
A one-time disclosure prints after the first `orx login` (shown even when
disabled, so it's never permanently suppressed).
- Never blocks or crashes the CLI: events are spawned and flushed once at
exit within a bounded window, so success output is never delayed; all
errors are swallowed.
- Cross-process safe: settings writes use an `fd_lock` advisory lock plus
atomic temp+rename, so a concurrent write can't revert a persisted opt-out.
- musl-safe: reuses the rustls `reqwest` client; no new TLS/C dependency.
Events land at PostHog `/i/v0/e/` with a public write-only `phc_` key and a
client-side ISO-8601 timestamp. `$process_person_profile:false` keeps events
anonymous. Unit tests cover opt-out precedence, id stability, payload shape
(PII-free), timestamp math, corrupt-file fail-safe, and the non-blocking
flush path.
* Prefix all CLI events with `cli_` to separate from website analytics
This PostHog project is shared with the website/cloud-agent analytics, so CLI
events must be distinguishable by event name alone — relying only on the
`source: "cli"` property is fragile (one forgotten dashboard filter leaks CLI
events into website metrics).
`build_payload` now stamps a `cli_` prefix onto every event name centrally, so
every current and future CLI event is `cli_*` by construction. Wire names:
- `cli_command` (was `cli_command`)
- `cli_experiment_started` (was `experiment_started`)
Added `every_event_name_is_cli_prefixed` to lock the guarantee.
* Telemetry: trim opt-out surface to --no-telemetry + `orx telemetry off`
Drop the `DO_NOT_TRACK`, `ORX_TELEMETRY`, and `CI` environment-variable
opt-out/disable checks. They were redundant with the two mechanisms we keep:
the `--no-telemetry` per-run flag and the persistent `orx telemetry off`.
The `CI` auto-disable in particular guarded a scenario that's mostly
hypothetical for an interactive research CLI, and it only ever fired for
hosted CI providers that set `CI` (missing Jenkins, agent boxes, and ad-hoc
scripts — the automation `orx` actually runs under). Every event still carries
a `ci` property, so automated runs can be filtered at query time in analytics
without being dropped up front.
Updates the opt-out policy, first-login notice text, `--no-telemetry` help,
README, and tests accordingly.
* orx up: add usage-analytics step to onboarding
Adds a third onboarding step ("Usage analytics") to the `orx up` dashboard
first-run walkthrough, after the coding-agents and git/GitHub steps. It shows
what is and isn't collected and lets the user toggle anonymous analytics On/Off
before creating their first project. On by default (matches the CLI's opt-out
model); the choice persists via the same `telemetry_disabled` setting as
`orx telemetry off`.
- Backend: `GET/POST /api/settings/telemetry` in up.rs, reusing
telemetry::disabled_reason / set_persisted_disabled (mirrors the git-settings
handler pattern). Returns `{ enabled, reason }`.
- UI: getTelemetry/setTelemetry in api.ts; Onboarding.tsx grows to a 3-step
flow with a TelemetryCard (On/Off segmented control).
Verified end-to-end in orx up: toggling Off in the wizard persists
`telemetryDisabled: true` to settings.json; toggling On clears it.
* Telemetry: drop the CLI login stderr disclosure
The one-time `orx login` stderr notice (show_notice_once) is removed now that
the disclosure lives in the `orx up` onboarding step, which is the right place
for it. Also removes the now-unused `notice_shown` settings field and updates
the module doc + README to point at the onboarding step instead of a login
notice.
* Telemetry: record consent decisions (cli_telemetry_consent)
Adds one event, `cli_telemetry_consent { agreed: bool }`, that records whether
the user opts into or out of analytics — so we can count agrees vs rejects.
Unlike every other telemetry event, this one is UNCONDITIONAL: it bypasses the
opt-out check and the persisted install id, and uses a fresh anonymous
distinct_id per call. If it honored the opt-out, every rejection would be
invisible and the ratio would be meaningless.
Fired once when the user leaves the analytics choice:
- `orx up` onboarding: on "Create your first project" (records the final On/Off
state, so users who accept the default On are counted too) via a dedicated
POST /api/settings/telemetry/consent endpoint.
- CLI: on `orx telemetry on` / `orx telemetry off`.
Tests: consent payload shape + name prefix, and that record_consent sends even
when persisted-off without generating an install id.
* Bump version to 0.1.57
Ships the anonymous opt-out telemetry subsystem, the orx up onboarding
usage-analytics step, and the cli_telemetry_consent event. Merging this PR
cuts release v0.1.57.
OpenResearch CLI (orx)
A small command-line interface for OpenResearch. Log in via the browser, then query your projects from the terminal.
Requirements
-
Rust (stable) with Cargo. Install via rustup:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Build & run
# Build (debug):
cargo build
# Run during development:
cargo run -- login
cargo run -- projects
# Build an optimized release binary at target/release/orx:
cargo build --release
./target/release/orx login
# Or install `orx` onto your PATH (~/.cargo/bin):
cargo install --path .
orx login
orx projects
Run the tests with cargo test.
Commands
| Command | Description |
|---|---|
orx login [--api-url <url>] |
Opens your browser, authenticates, and stores a personal access token. |
orx logout |
Removes the stored token. |
orx install-skills [--agent claude|codex|opencode|cursor|all] |
Installs the orx skill shim into local coding agents (Claude Code, Codex, OpenCode, Cursor) so they auto-discover the CLI (orx login also offers this). |
orx projects [--all] |
Lists your projects, grouped by organization. --all includes archived. |
orx compute [--gpu <id>] [--count <n>] [--provider <name>] | --cpu] |
Lists the GPU compute catalog across all providers and regions, or CPU-only offers with --cpu, sorted by price. |
orx instance create <orgId> (--gpu <id> [--count <n>] [--disk <gb>] [--provider <name>] | --cpu <flavor> [--vcpus <n>]) |
Spins up a standalone instance in an organization (not tied to an experiment), like the dashboard's "Spin up". |
orx exp status <expId> |
Shows an experiment's status, branch, parent, run command, and latest run, plus a local git diff recipe for what the run changed. |
orx exp cmd <expId> [--set <command>] |
Views or sets the experiment's run command. |
orx exp run <expId> (--gpu <id> [--count <n>] [--disk <gb>] [--provider <name>] | --cpu <flavor> [--vcpus <n>] | --sandbox <id>) |
Launches a run on new GPU, new CPU-only, or existing compute. |
orx exp cancel <expId> |
Cancels the in-flight run. |
orx lit "<query>" [--limit <n>] [--json] |
Full-text search alphaXiv's paper corpus (no login required). |
orx paper <id|url> [--full] |
Fetch a paper's machine-readable report, or its full text with --full. |
orx version [--check] [--json] |
Prints the CLI version; --check compares it to the latest release. |
orx update [--dry-run] [--force] |
Updates orx by re-running the release installer (installer installs only). |
Updating
orx update only manages binaries installed by the release installer script
(it checks the cargo-dist install receipt at
${XDG_CONFIG_HOME:-~/.config}/openresearch-cli/). If you installed with
cargo install, update the same way instead. In interactive terminals orx
also checks for new releases in the background at most once a day and prints
a one-line notice to stderr; set ORX_NO_UPDATE_CHECK=1 (or
OPENRESEARCH_CLI_DISABLE_UPDATE=1) to turn that off. Agents, scripts, pipes,
and CI never see the notice.
Configuration
-
API URL resolves from
--api-url→OPENRESEARCH_API_URL→ the built-in default (http://localhost:4000). Point it at the production API host for real use:export OPENRESEARCH_API_URL=https://api.openresearch.sh -
Credentials are stored at
${XDG_CONFIG_HOME:-~/.config}/openresearch/credentials.json(mode0600).
How login works
orx login starts a temporary HTTP listener on a random 127.0.0.1 port, opens
{api}/auth/cli/login in your browser, and waits for the API to redirect back
with a freshly minted personal access token after you authenticate. The token is
sent as Authorization: Bearer … on every subsequent request.
Usage analytics
orx sends anonymous usage analytics to help us understand which features
are used and prioritize accordingly. This is opt-out; the orx up dashboard
surfaces the choice as a step in first-run onboarding.
What is collected — the command name that was run, a random per-install UUID,
the CLI version, your OS/architecture, whether the run is in CI, and a small set
of coarse labels for key events (e.g. an experiment was created or a run was
launched, and a broad compute target like modal/k8s/local).
What is never collected — no code, prompt text, file contents, file paths, project/experiment ids or names, repo names, tokens, emails, or any other personal or identifying data. The install UUID is random and not tied to your account.
How to opt out — either:
orx telemetry off # persistent, per-machine
orx telemetry status # see current state + the anonymous install id
orx <cmd> --no-telemetry # per-run
Events are sent fire-and-forget on a background task with a short timeout, so
they never meaningfully block or fail a command (at most a brief flush window on
exit). Every event carries a ci property, so automated runs can be filtered in
analytics without being dropped up front.