mirror of
https://github.com/alphaXiv/OpenResearch.git
synced 2026-11-11 09:59:34 +08:00
Daniel Kim
30e2ab18f4
* Make reasoning/effort choices model-aware (#123) The composer exposed one static reasoning list per harness, which hid valid choices and silently overrode the CLI's own configured effort. Reasoning choices now ride on each ModelInfo, with the harness-wide list kept only as the fallback for a session with no model selected: * Codex — per-model tiers from a curated table (codex exposes no queryable effort metadata), kept in the same table as the model ids so a model can't be added without stating its tiers. Verified live against codex-cli 0.144: all three 5.6 models accept `ultra`, and 5.5 rejects `max` with a 400 — issue #123's table claims Luna stops at `max`, which did not reproduce. This matters because codex forwards the value unvalidated, so a wrong tier kills the turn. * OpenCode — real per-model `variants` parsed from `opencode models --verbose`, sent as a top-level `variant`. A model with an empty variants map hides the picker entirely, which is why absent and empty are distinct states. Unknown variants are forwarded rather than filtered: the catalog is the authority and already ships ids outside our vocabulary (`thinking` on minimax-m3). * Claude Code — `ultracode` gated on CLI >= 2.1.217. The installed 2.1.197 does not error on an unknown effort; it warns and silently runs at the default, so advertising it unconditionally would offer a choice that does nothing. Every list leads with a `default` sentinel that sends no override at all, and it is the default selection — so a configured `model_reasoning_effort = "max"` is no longer clobbered by an implicit per-turn `high`. Switching models reconciles a stored effort the new model doesn't support back to `default`, both in the composer and again in each harness mapper before it reaches a CLI. Deliberately no store migration for existing rows: every value older builds wrote is still a value the picker offers, so a blanket reset would be indistinguishable from a level the user just chose — and Store::open() is a per-request constructor, so it would re-run and clobber fresh picks. * Lower the ultracode gate to 2.1.203, the measured boundary The constant was 2.1.217 — the version issue #123 happened to report — with a comment admitting the true cutoff was somewhere in (197, 217] and that lowering it needed a version actually tested. It has now been tested: every published release in 2.1.197..=2.1.212 was installed in isolation and probed. 2.1.198-2.1.202 reject (warn on stderr, run at default effort); 2.1.203 and every version above accept. So a 2.1.212 install was being denied a choice its CLI supports. The acceptance is specific to this value rather than newer CLIs dropping the warning: on 2.1.206, `ultra`, `ultrathink`, and `ultracode123` all still warn while `ultracode` does not. `--help` lists only the five ordinary tiers on every version tested, so it can't be used to detect support. The tests now pin the 2.1.202/2.1.203 pair, so moving the constant without re-probing fails loudly. * Discover the Codex catalog from the CLI instead of curating it The model list and per-model effort tiers were hard-coded. That was wrong on the facts, not just the principle: the installed codex actually serves seven models (gpt-5.4, gpt-5.4-mini, gpt-5.3-codex-spark were missing), and the curated Luna entry advertised `ultra`, which the catalog does not offer — codex's own picker doesn't show it, so ours shouldn't either. (An earlier live `codex exec` turn on Luna tolerated `ultra`, which is what the table was built from; the catalog is authoritative for what to offer.) Detection now spawns `codex app-server`, does the initialize handshake, and calls `model/list` — the same data codex's TUI picker renders, with each model's `supportedReasoningEfforts`. This is the codex analogue of the opencode `models --verbose` path. The static table survives only as the fallback for a codex too old to answer (legacy exec era), with Luna corrected to the catalog's tiers. Send-time validation follows the opencode precedent for catalog-discovered models: a model outside the frozen fallback table forwards its level rather than clamping to the conservative intersection, because the composer only offered what `model/list` reported — clamping would drop genuinely supported tiers. A bad value still surfaces as codex's 400 in the chat. Claude stays gated rather than discovered, now with evidence there is nothing to query: no models command, a static `--help`, and the "Valid values" warning on 2.1.212 still omits `ultracode` even though that version accepts it — the CLI's only enumeration surface is inconsistent with its own parser. * Discover the Claude catalog and ultracode support from the CLI Claude Code was the last harness with hard-coded models and effort tiers. It turns out the CLI can be asked for both, via two surfaces: * `list_models` — a control request over `--print` stream-json (the channel run_turn already speaks for set_model), returning the same catalog the CLI's own /model menu renders: each model's alias, and its per-model `supportedEffortLevels`. This immediately corrected the static list a third time: Haiku supports no effort control at all, yet the table had been offering all five tiers on it. Its picker now hides, like a variant-less opencode model. * `--effort ultracode --version` — the CLI's own argument parser as a capability probe (~0.2s, no network). `--version` still runs the parser, which warns `Unknown --effort value` for a value it doesn't know; absence of the warning is acceptance. This replaces the bisected version constant entirely: the 2.1.202/2.1.203 boundary is now discovered per install rather than pinned. The parser is the only truthful surface — --help, the warning's own "Valid values" list, and list_models all omit ultracode on versions that accept it, because the CLI models it as a session mode, not an effort level. It is appended to models that reach xhigh (the mode is documented as xhigh + dynamic workflows), only when the probe passes. claude_effort now forwards any non-sentinel value, matching the catalog policy in codex_reasoning and opencode_variant: the composer only offers what the CLI reported, an allowlist would drop a future catalog tier, and Claude only warns-and-defaults on an unknown value rather than failing the turn. The static model/effort tables survive only as the fallback for a CLI too old to answer list_models, mirroring the codex and opencode fallbacks. * Carry the catalogs' display names and blurbs into the picker The Claude aliases the CLI's picker submits (`opus[1m]`, `sonnet`) are unversioned, so the model menu couldn't say which Opus or Sonnet you'd get. The catalogs already answer this: list_models has displayName + description (the description is where the resolved version lives — "Opus 4.8 with 1M context · …"), codex model/list has the same pair, and opencode's verbose output has name. ModelInfo now carries display_name/description, and the picker renders the catalog's own strings — the same text each CLI's native picker shows — with the raw id demoted to a hover title. Statically-listed fallback models keep deriving a label from the id. * Lead with versioned names, drop the Default row, allow free-form model ids Three picker fixes from using it: * The Claude rows read "Fable / Fable 5 · Most capable…" — the name repeated because the catalog's displayName is unversioned and the version lives in the description's first `·` segment. That segment is now promoted to the display name ("Fable 5") with the rest as the blurb, so rows lead with the resolved version, exactly once. * The per-harness "Default model · CLI configuration" row is gone wherever the CLI advertises a catalog — it was redundant noise next to a discovered list that already leads with the CLI's default. It survives only for custom providers, which advertise no models and where "let the CLI decide" is the only sensible row. New selections seed the catalog's first model instead of null. * Typing an id the catalog doesn't list offers "Use "<id>"". The catalogs are curated menus, not the set of accepted ids — `--model claude-opus-5` works today on a CLI whose menu still resolves `opus` to 4.8 — so the menu can't be the ceiling. A custom id gets no per-model effort list (unknowable), so the reasoning picker falls back to the conservative harness-wide tiers. * Preselect the effort that actually runs instead of a Default sentinel The reasoning menu led every list with "Default · CLI configuration" — a pseudo-value that told the user nothing about what would run. Replace it with the real answer wherever the CLI can give one: * Codex reports each model's defaultReasoningEffort, and config.toml's model_reasoning_effort outranks it (codex's own resolution order — verified: this machine's config sets "high" and the picker now preselects High on every model, not the catalog's low/medium). The sentinel row is gone for catalog models; the resolved tier is preselected and marked "· Default". Sending it explicitly is equivalent to sending nothing, by construction. * Claude reports no default tier because its unset default isn't one: the catalog advertises supportsAdaptiveThinking, and with no --effort the CLI scales effort per request. Preselecting a fixed tier would pin behavior the user never chose, so the sentinel stays — now labeled "Adaptive", which is what actually runs, instead of "Default". * OpenCode reports no per-model default, so its sentinel row stays as is. reasoningFor resolves in that order: the model's reported concrete default, then the sentinel when offered, then the harness-wide default. Stale stored "default" sentinels on codex sessions reconcile to the resolved tier. * Render a concrete default tier inline instead of pinning it Pinning above a separator now applies only to the `default` sentinel, which isn't a point on the tier ramp (Claude's Adaptive, opencode's model default). A concrete default — codex's resolved tier, a permission mode — stays in its natural ramp position with just the "· Default" marker, so the codex menu reads Low → Ultra in order. Side effect on the shared mode menu: Auto renders inline with the marker and the 1/2/3 number hints cover the full list again. * Make every picker change update the sticky new-session preference The mode/model/reasoning selection saved for new sessions only updated when the pickers were changed with no session open. A change made inside a session landed as that session's unsent tweak and the global preference never learned about it — so configuring auto/sonnet/high mid-session and clicking New session showed the old config. selectModel now always writes the merged selection to the sticky preference (state + localStorage), and additionally layers it onto the open session's override as before. New session therefore opens with whatever was chosen last, wherever it was chosen — verified across a full page reload.
OpenResearch CLI (orx)
Important
If you are an OpenResearch user or someone who is interested in autoresearch, we'd love to chat with you. Please email contact@alphaxiv.org if interested.
Run autoresearch on your machine
- Run research agents in parallel. Spins up agents in different worktrees so you can investigate several different directions at once.
- Works with Claude Code, Codex, and OpenCode
- Bring your own compute. Works with SSH, Slurm, Kubernetes, Modal, HuggingFace and more.
- Give it a goal. Can run the entire autoresearch loop from literature review to experiment analysis.
- Local and private. Your code and your data stays on your machine.
https://github.com/user-attachments/assets/33b62182-0795-490d-9366-0fb0b4bd49fd
Quick start
curl -LsSf https://openresearch.sh/install.sh | sh
orx up
The dashboard opens at http://127.0.0.1:4791. Give the agent a goal — for
example, ask it to reproduce a paper:
/reproduce-paper <paper URL or title> on <compute>
or turn one into an interactive marimo notebook:
/paper-to-marimo <paper URL or title> on <compute>
The dashboard
orx up runs a single local process on 127.0.0.1 — an embedded web UI plus a
JSON/SSE API over a local SQLite store. From there you get:
- Agent chat — a research assistant with full project context, backed by your locally installed harness: Claude Code, Codex, or OpenCode (pick the harness and model in the UI). Ask it to analyze runs, dig into results, edit code, and spin up new experiments.
- The experiment tree — every experiment is a git branch: a runnable snapshot of your code. The root is your baseline; children are variants measured against it, so lineage stays explicit.
- Runs — launch on your compute (Modal, Hugging Face Jobs, Kubernetes, Slurm, any SSH box, your own machine) or OpenResearch managed GPUs, and watch live logs, statuses, git diffs, files, and linked W&B runs stream in.
- Autoresearch — describe a goal and let the agent run autonomously toward it: proposing, launching, and analyzing experiments.
Everything binds to loopback only; nothing on the dashboard's paths leaves your machine except the compute and paper-search calls you initiate.
On a remote machine
Develop from your laptop while the dashboard runs next to your GPUs:
orx up --remote user@host # or an ~/.ssh/config alias; append :PORT for a custom SSH port
This starts orx up on the remote box over SSH, tunnels the port back, and
opens your browser locally. Note the remote server is unauthenticated on that
host's loopback, so other users on the same box can reach it.
Commands
Run orx --help (or orx <command> --help) for full usage. The highlights:
| Area | Commands |
|---|---|
| Dashboard | up |
| Auth | login, logout |
| Projects | projects, explore, project, create-project, env |
| Experiments | experiments, create-experiment, exp status/cmd/run/cancel |
| Runs & evidence | runs, logs, search-logs, artifacts, artifact, wandb, query, chart, report |
| Compute | compute, instance create |
| Literature | lit, paper (alphaXiv full-text search — no login required) |
| Agent integration | install-skills, skill |
| Maintenance | version, update, telemetry |
orx install-skills drops the OpenResearch skill into your local coding agents
(Claude Code, Codex, OpenCode, Cursor) so they can drive orx themselves —
orx login offers this too.
Installing
The install script above fetches the latest prebuilt release (macOS and Linux, x86_64 and arm64) and is the same as:
curl -LsSf https://github.com/alphaXiv/openresearch-cli/releases/latest/download/openresearch-cli-installer.sh | sh
orx update keeps script-installed binaries current; interactive terminals
also get a once-a-day background check with a one-line stderr notice (silence
it with ORX_NO_UPDATE_CHECK=1).
From source
Requires Rust (stable) via rustup. The prebuilt dashboard
UI is committed at ui/dist, so a plain build works:
cargo build --release # binary at target/release/orx
cargo install --path . # or install onto your PATH (~/.cargo/bin)
To hack on the dashboard UI itself (Vite + React, embedded into the binary at build time):
cd ui && pnpm install && pnpm build
Run the tests with cargo test.
Configuration
- API URL — defaults to production (
https://api.openresearch.sh); override with--api-urlorOPENRESEARCH_API_URL. - Credentials —
orx loginopens your browser, mints a personal access token, and stores it at${XDG_CONFIG_HOME:-~/.config}/openresearch/credentials.json(mode0600). Sent asAuthorization: Bearer …on every request.
Usage analytics
orx sends anonymous usage analytics to help prioritize features. It's
opt-out, and the orx up onboarding surfaces the choice on first run.
- Collected: command name, a random per-install UUID, CLI version, OS/arch,
a CI flag, and coarse event labels (e.g. "a run launched on
modal"). - Never collected: code, prompts, file contents or paths, project or experiment ids/names, repo names, tokens, emails — nothing identifying. The install UUID is not tied to your account.
orx telemetry off # persistent, per-machine
orx telemetry status # current state + the anonymous install id
orx <cmd> --no-telemetry # per-run
Events are fire-and-forget on a background task and never block a command.