trending/OpenResearch
1
0
Fork 0
mirror of https://github.com/alphaXiv/OpenResearch.git synced 2026-10-02 01:34:34 +08:00
Code Issues Packages Projects Releases Wiki Activity

OR-158 Decouple compute runs from GitHub

Browse Source 
Ships direct source snapshots across compute backends, optional GitHub experiment publication, and CLI v0.1.98.
This commit is contained in:
Daniel Kim
2026-08-11 13:10:24 -07:00
committed by GitHub
parent 4738715e94
commit 82ea072291
53 changed files with 2775 additions and 1958 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Cargo.lock
Generated
+3 -1
View File
@@ -989,7 +989,7 @@ checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
[[package]]
name = "openresearch-cli"
version = "0.1.97"
version = "0.1.98"
dependencies = [
"anyhow",
"async-trait",
@@ -1006,7 +1006,9 @@ dependencies = [
"semver",
"serde",
"serde_json",
"sha2",
"tokio",
"tokio-util",
"toml",
"urlencoding",
"uuid",
Cargo.toml
+3 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "openresearch-cli"
version = "0.1.97"
version = "0.1.98"
edition = "2021"
description = "OpenResearch CLI (orx) — Rust port"
repository = "https://github.com/alphaXiv/openresearch-cli"
@@ -12,6 +12,7 @@ path = "src/main.rs"
[dependencies]
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net", "io-util", "io-std", "time", "fs", "process", "sync", "signal"] }
tokio-util = { version = "0.7", features = ["io"] }
# rustls instead of the default native-tls: avoids an OpenSSL dependency so the
# static musl Linux builds link cleanly (native-tls/openssl won't build for musl).
reqwest = { version = "0.12", default-features = false, features = ["json", "charset", "http2", "rustls-tls", "stream"] }
@@ -39,6 +40,7 @@ rust-embed = "8"
futures = "0.3"
# async fns in the Harness trait (object-safe `dyn Harness` in the registry).
async-trait = "0.1"
sha2 = "0.10"
[target.'cfg(unix)'.dependencies]
# Free-space pre-check for the data-dir move (statvfs). Already in the tree
README.md
+6 -7
View File
@@ -55,16 +55,15 @@ JSON/SSE API over a local SQLite store. From there you get:
- **The experiment tree** — every experiment is a git branch: a runnable
snapshot of your code. The root is your baseline; children are variants
measured against it, so lineage stays explicit.
- **Runs** — projects and experiments stay local by default and run on this
machine from their recorded Git commit. Enable GitHub syncing for a project
when you want Modal, Hugging Face Jobs, Kubernetes, Slurm, SSH, Ray, or
OpenResearch compute; remote launches push the exact branch first.
- **Runs** — every backend receives the same immutable archive of the recorded
Git commit. Modal, Hugging Face Jobs, Kubernetes, Slurm, SSH, Ray,
OpenResearch, and local runs do not require a hosted repository or push.
- **Autoresearch** — describe a goal and let the agent run autonomously toward
it: proposing, launching, and analyzing experiments.
Everything binds to loopback only. Creating a local project does not call
GitHub; public paper repositories and paper search use the network only when
you choose those flows.
Everything binds to loopback only. Creating a local project and launching
compute do not publish code; upstream repositories and paper search use the
network only when you choose those import flows.
### On a remote machine
agent-skills/orx-compute-k8s/SKILL.md
+8 -12
View File
@@ -1,6 +1,6 @@
---
name: orx-compute-k8s
description: "Run an experiment on your own Kubernetes cluster (`orx exp run --backend k8s`): the committed-manifest contract orx enforces at submit. Use when the user names k8s, kubernetes, or a cluster, before writing or editing `.orx/k8s.yaml`, for multi-node or Indexed Jobs, or when a k8s submit is rejected."
description: "Run an experiment on your own Kubernetes cluster (`orx exp run --backend k8s`): the single-pod committed-manifest contract orx enforces at submit. Use when the user names k8s, kubernetes, or a cluster, before writing or editing `.orx/k8s.yaml`, or when a k8s submit is rejected."
---
**Use `--backend k8s` ONLY when the user explicitly asks to run on their
@@ -12,8 +12,9 @@ come from the user's configured Kubernetes profile.
**There are no flavors: the run's shape is a Kubernetes manifest you commit
on the experiment branch** (default `.orx/k8s.yaml`, or `--manifest <path>`).
Inspect the cluster yourself (`kubectl get nodes`, allocatable resources, GPU
products) and write whatever the run needs — a single-pod GPU Job, an Indexed
Job spanning nodes with a headless Service, an auxiliary inference Deployment.
products) and write whatever the run needs around one primary pod — a GPU Job,
PVCs, ConfigMaps, Services, or an auxiliary inference Deployment. Parallel and
Indexed Jobs are rejected because the immutable archive is staged into one pod.
The manifest inherits through the experiment tree like all code; changing it is
a commit, visible in the diff like any experimental variable.
@@ -48,23 +49,18 @@ The contract orx enforces at submit (loud, before anything runs):
(Services, Deployments, ConfigMaps) ride along; cancel deletes exactly what
the manifest created.
- **Some container of that Job must run `$ORX_SCRIPT`** — the env var orx
injects with the clone-and-run script (branch tip + the experiment's fixed
injects with the snapshot-and-run script (recorded revision + the experiment's fixed
run command). Set `command: ["bash", "-c", "$ORX_SCRIPT"]`. The manifest
shapes *where* the command runs, never *what* runs.
- Every resource needs `metadata.name` (no `generateName`) and no foreign
`metadata.namespace`. Use `{{ORX_RUN}}` in names — orx substitutes a
run-unique token so re-runs don't collide.
- orx injects run labels, the `orx-env` Secret (synced env + `HF_TOKEN` /
`GITHUB_TOKEN`) on the primary Job's containers, and defaults for
- orx injects run labels, the `orx-env` Secret (synced environment variables)
on the primary Job's containers, and defaults for
`activeDeadlineSeconds` (from `--timeout`, default 4h; a manifest-set value
wins), `ttlSecondsAfterFinished`, and `backoffLimit: 0`. Auxiliary
resources that need the env reference the `orx-env` Secret themselves.
- The run log follows the primary Job's **leader pod** (completion index 0
for Indexed Jobs, else its sole pod) — make it print the evidence; other
pods stay reachable via `kubectl logs`. Cross-node traffic rides the pod
network — fine for loosely-coupled work (async RL, parameter-server);
tightly-coupled per-step all-reduce wants a fast fabric the cluster may not
have.
- The run log follows the primary Job's sole pod; make it print the evidence.
- Everything downstream is identical (`orx exp wait` / `orx runs` /
`orx logs`, cancel via `orx exp cancel`). A detached `orx supervise`
watches the Job via kubectl; don't kill it.
agent-skills/orx-compute/SKILL.local.md
+13 -15
View File
@@ -3,11 +3,9 @@ name: orx-compute
description: "Launch experiment runs with `orx exp run`: backends (hf, modal, k8s, ssh, slurm, ray, openresearch, local), flavors, timeouts, images, sizing, and `orx exp wait`. Use before launching or re-launching any run, when choosing or switching a backend or GPU flavor, when a job OOMs, stalls, or times out, or when deciding GPU vs CPU."
---
Projects are local-only by default. In that state, commit the experiment and
use the `local` backend; never push. Remote backends stay unavailable until the
user explicitly enables GitHub syncing for the project. Once enabled,
`orx exp run` pushes the selected branch before provisioning and aborts the
launch if that push fails.
Commit the experiment before launching. `orx` archives that exact revision and
transfers the immutable source snapshot directly to the selected backend. No push,
repository clone, GitHub token, or public repository is required.
In local mode (`orx up`) every run launches with `orx exp run <expId>` onto a
**backend**: `hf`, `modal`, `k8s`, `ssh`, `slurm`, `ray`, `openresearch`, or `local`.
@@ -30,9 +28,9 @@ orx exp cancel <expId> # cancel the in-flight run
code/config on the child's branch instead (see the `orx-experiment-tree`
skill). A node with no command refuses to launch and points at
`orx project edit`.
- **Push before launching.** Every backend clones the experiment branch's tip
**as it is on GitHub** and runs the fixed command — uncommitted or unpushed
edits won't be in the run (see the `orx-git` skill).
- **Commit before launching.** Every backend runs the same content-addressed
archive of the recorded local revision. Uncommitted edits are excluded; a
push is unrelated to launching (see the `orx-git` skill).
- **`orx exp run` queues the run and returns immediately** — it does not wait.
Follow progress with `orx runs <projectId>` and `orx logs <runId>`, or block
with `orx exp wait` (below).
@@ -110,10 +108,10 @@ orx exp run <expId> --backend ssh --host my-gpu-box
- **`--host <alias>` is required on every launch** — a machine, not a hardware
shape, so there is no `--flavor`, `--image`, or `--timeout` (the process runs
until it exits or is cancelled). Use one of the user's configured SSH host
aliases; OpenResearch validates reachability and git separately.
aliases; OpenResearch validates reachability before upload.
- Auth is your own ssh keys/agent — orx never reads a key, it just shells out
to `ssh <alias>`. The host needs `git` and `bash`; private repos clone via
the `GITHUB_TOKEN` passed in the run's env. The run lives under
to `ssh <alias>`. The host needs `tar` and `bash`; orx uploads the snapshot
over SSH. The run lives under
`~/.orx/runs/<runId>/` on the host; cancel kills the remote process group.
## Your Slurm cluster — `--backend slurm`
@@ -134,9 +132,9 @@ orx exp run <expId> --backend slurm # CPU-only, settings defa
## A Ray Jobs cluster — `--backend ray`
Submits via the Ray Jobs / Dashboard API (same contract as Hugging Face Jobs:
clone the experiment branch tip from GitHub and run the fixed command). Needs a
reachable Ray head (Dashboard, usually port 8265).
Submits via the Ray Jobs / Dashboard API. The snapshot is uploaded as Ray's
`working_dir` package. Needs a reachable Ray head (Dashboard, usually port
8265).
```sh
orx exp run <expId> --backend ray
@@ -192,7 +190,7 @@ orx exp run <expId> --backend local
machine has. It shares CPU/RAM/GPU with OpenResearch and your editing tools:
prefer it for small or CPU-scale runs and a remote backend for anything
heavy.
- Still the full run contract: it clones the branch tip into its own run dir
- Still the full run contract: it extracts the recorded snapshot into its own run dir
(never your checkout), supervised and tracked by OpenResearch — never run
training directly in your shell instead. The run lives under
`<orx data dir>/local-runs/<runId>/`; cancel TERMs the process group.
agent-skills/orx-compute/SKILL.md
+38 -39
View File
@@ -33,11 +33,11 @@ Rules and notes:
touch a command is the baseline having none yet.
- **Set a run command before launching.** `orx exp run` fails with a pointer to
`orx exp cmd --set` if the node has none.
- **Push your edits before launching.** A run trains the branch's tip **as it is
on GitHub** — so commit and push first (see the `orx-git` skill). As a
safety net, `orx exp run` refuses a child whose branch has **no changes over its
parent** (the tell-tale of "queued before pushing") — push and retry, or pass
`--force` to run the unchanged code deliberately.
- **Commit your edits before launching.** Local projects run an immutable source
snapshot of the committed branch and never need a GitHub push. Managed server
projects still run their recorded remote revision. As a safety net, `orx exp
run` refuses a child whose branch has no changes over its parent; commit a
meaningful change and retry, or pass `--force` deliberately.
- **Pick compute with exactly one of `--gpu`, `--cpu`, or `--sandbox`.** With
`--gpu`, `--count` defaults to `1` and `--disk` to `100` (GB). A new GPU
instance defaults to **RunPod** (the cheapest matching RunPod offer for the
@@ -59,10 +59,10 @@ Rules and notes:
## The default compute target (local projects)
A local-only project always launches on this machine with the `local` backend.
After GitHub syncing is enabled, local projects can also use remote backends;
the user may configure a **default compute target** that is machine-wide and
shared by those projects. When one is set, `orx exp run <expId>` with no
A local project can launch on this machine or transfer its immutable source
snapshot directly to a remote backend. The user may configure a **default
compute target** that is machine-wide and shared by local projects. When one is
set, `orx exp run <expId>` with no
`--backend` launches there with the saved default flavor — omitting the flag is
how you use it (flavor-required backends still need `--flavor` if no default
flavor is saved). When none is set, ask the user to choose an explicit backend.
@@ -71,8 +71,9 @@ stays their default.
## Running on Hugging Face Jobs — `--backend hf`
**Managed compute (`--gpu`/`--cpu`/`--sandbox`) is the default. Use
`--backend hf` ONLY when the user explicitly asks for Hugging Face Jobs**
**Managed compute (`--gpu`/`--cpu`/`--sandbox`) is the default for server
projects. `--backend hf` requires a local project (`orx up`); use it ONLY when
the user explicitly asks for Hugging Face Jobs**
(e.g. "run this on HF", "use my huggingface account"), it is the configured
default target, or the project context says to
prefer it. A connected HF token by itself is NOT a signal to switch — it just
@@ -97,12 +98,9 @@ Rules and notes:
as managed GPUs.
- **Set `--timeout` to cover the whole run** (default `4h`). HF kills the job
at the timeout; a killed job reads as a failed run.
- The job clones the experiment branch's **GitHub tip** and runs the fixed run
command, same contract as managed runs — commit and push first. Private
repos work automatically: the platform mints a repo-scoped clone token from
the project's connected GitHub app and passes it to the job as a secret.
Never ask the user to provision a `GITHUB_TOKEN`; setting one (env or
project env var) is only an override for repos outside the connected app.
- For a local project, `orx` uploads the committed source snapshot into a
private job volume before starting the fixed run command. The job never
clones a repository and does not need repository credentials.
- `--image` overrides the container (default: a CUDA pytorch image on GPU
flavors, `python:3.12` on cpu flavors). Pick an image with your deps baked
in when pip-install time dominates the run.
@@ -113,10 +111,10 @@ Rules and notes:
## Running on Modal — `--backend modal`
**Same rule as HF: managed compute is the default. Use `--backend modal` ONLY
when the user explicitly asks for Modal** ("run this on Modal", "use my Modal
account") or it is the configured default target. Modal runs on the user's
own Modal account, billed there per second; no OpenResearch balance is spent.
**Same rule as HF: `--backend modal` requires a local project. Use it ONLY when
the user explicitly asks for Modal** ("run this on Modal", "use my Modal
account") or it is the configured default target. Modal runs on the user's own
Modal account, billed there per second; no OpenResearch balance is spent.
It runs the job in a Modal **Sandbox** (an ephemeral container that scales to
zero when the run ends).
@@ -139,9 +137,8 @@ Rules and notes:
Prefer the smallest flavor that fits.
- **Set `--timeout` to cover the whole run** (default `4h`). Modal kills the
sandbox at the timeout; a killed sandbox reads as a failed run.
- Same clone contract as HF/managed: the sandbox clones the experiment branch's
**GitHub tip** and runs the fixed command — commit and push first. Private
repos work automatically via the platform's repo-scoped clone token.
- `orx` copies the committed source snapshot into the sandbox before starting
the fixed command; Modal never needs repository access.
- `--image` overrides the container (default: a CUDA pytorch image on GPU
flavors, `python:3.12` on cpu). Pick one with your deps baked in when
pip-install time dominates.
@@ -151,9 +148,10 @@ Rules and notes:
## Running on your Kubernetes cluster — `--backend k8s`
Runs the experiment on your own Kubernetes cluster from a manifest committed on
the experiment branch. The full manifest contract lives in the `orx-compute-k8s`
skill — fetch it (`orx skill compute-k8s`) before your first k8s launch.
Requires a local project (`orx up`). Runs the experiment on your own Kubernetes
cluster from a manifest committed on the experiment branch. The full manifest
contract lives in the `orx-compute-k8s` skill — fetch it (`orx skill compute-k8s`)
before your first k8s launch.
## Running on your own box — `--backend ssh`
@@ -171,11 +169,11 @@ orx exp run <expId> --backend ssh --host my-gpu-box # ~/.ssh/config alias
Rules and notes:
- **`--host` is the ssh host alias** (from `~/.ssh/config`) — a machine, not a
hardware shape, so there is no `--flavor` here. Use one of the user's
configured aliases; OpenResearch validates reachability and git separately.
configured aliases; OpenResearch validates reachability and required tools.
- Auth is your ssh keys/agent — orx never reads a key, it just shells out to
`ssh <alias>`. The host needs `git` and `bash`; it clones the experiment
branch's GitHub tip (private repos via the `GITHUB_TOKEN` passed in the run's
env) and runs the fixed command. Commit and push first, same as the others.
`ssh <alias>`. The host needs `bash` and `tar`; orx streams the committed
source snapshot over SSH, extracts it into the run directory, and starts the
fixed command.
- No `--image` (the host's environment is used as-is) and no `--timeout` (the
process runs until it exits or you cancel).
- The run lives under `~/.orx/runs/<runId>/` on the host (`run.sh`, `log`,
@@ -204,9 +202,9 @@ Rules and notes:
environment (modules, conda, whatever the login profile provides).
- `--timeout` (default `4h`) applies — size it to cover the whole run; a job
killed at the timeout reads as a failed run.
- Same clone contract as every backend: the job clones the experiment branch's
GitHub tip and runs the fixed command — commit and push first. Everything
downstream (`orx exp wait` / `orx runs` / `orx logs` / `orx exp cancel`) is
- `orx` streams the committed source snapshot to the login node before `sbatch`
starts the fixed command. Everything downstream (`orx exp wait` / `orx runs`
/ `orx logs` / `orx exp cancel`) is
identical; a detached `orx supervise` mirrors status and logs — don't kill it.
## Running on a Ray Jobs cluster — `--backend ray`
@@ -233,7 +231,8 @@ Rules and notes:
that avoids Pending on small heads.
- No `--image`, `--host`, or `--timeout` — the job runs in the cluster's
runtime env until it finishes; size and bound work in the run command itself.
- Same clone contract and downstream commands as every backend; a detached
- Ray receives the committed snapshot as its `working_dir` package; downstream
commands stay the same, and a detached
`orx supervise` mirrors status and logs — don't kill it.
## Running on an OpenResearch box — `--backend openresearch`
@@ -258,7 +257,8 @@ Rules and notes:
`--provider <P>`. No `--image` — the platform's image is fixed.
- `--timeout` (default `4h`) applies — the box is deleted when the run ends
either way, so nothing persists on it; everything you need must be in the log.
- Same clone contract and downstream commands as every backend; a detached
- `orx` streams the committed source snapshot to the provisioned box;
downstream commands stay the same, and a detached
`orx supervise` mirrors status and logs — don't kill it.
## Running on this machine — `--backend local`
@@ -279,9 +279,8 @@ Rules and notes:
this machine has; prefer it for small or CPU-scale runs and use a remote
backend for anything heavy — it shares CPU/RAM/GPU with everything else on
the machine.
- Same clone contract as every backend: the run clones the experiment
branch's GitHub tip into its own run dir (never your checkout) and runs the
fixed command — commit and push first. Never run training directly in your
- The run extracts the committed source snapshot into its own run dir (never
your checkout) and runs the fixed command. Never run training directly in your
shell instead: that would be unsupervised and untracked by OpenResearch.
- The run lives under `<orx data dir>/local-runs/<runId>/` (`run.sh`, `log`,
`pid`, `exit_code`). Cancellation through OpenResearch or `orx exp cancel` TERMs the
agent-skills/orx-experiment-tree/SKILL.local.md
+3 -4
View File
@@ -85,7 +85,7 @@ To drive a project toward a goal (e.g. "best convergence for d=8"), this is the
intended flow — do **not** edit a frozen node or rewrite the run command:
1. **Read the baseline's code.** You already sit in a private git worktree of the
project's repo — `git fetch origin && git checkout <branch>` and read it with
project's repo — `git checkout <branch>` and read it with
your normal tools (see the `orx-git` skill). See the node's run command with
`orx exp status <expId>` and find where the knobs live (config files,
hyperparameters, model defs).
@@ -117,10 +117,9 @@ intended flow — do **not** edit a frozen node or rewrite the run command:
4. **Implement each child's change on its git branch** — `orx create-experiment`
prints the child's branch (`orx/<slug>`); in your worktree:
```sh
git fetch origin && git checkout orx/<child-slug>
git merge --ff-only origin/orx/<child-slug>
git checkout orx/<child-slug>
# …edit only the files that idea touches…
git commit -am "cosine LR + warmup" && git push
git commit -am "cosine LR + warmup"
```
**Leave the run command alone.** While you're in the code, **make the run
print the evidence you'll need to judge it** — final metrics, a compact
agent-skills/orx-experiment-tree/SKILL.md
+5 -9
View File
@@ -114,17 +114,13 @@ the run command:
The child inherits its parent's run command automatically — you don't set it,
and you never give siblings different commands or env vars (cardinal rule 2).
4. **Implement each child's change on its git branch** — `orx create-experiment`
prints the child's branch (`orx/<slug>`); sync the project's clone (in the
openresearch cache dir — see the `orx-git` skill), check the branch out,
edit only the files that idea touches, commit, and push. **Leave the run
prints the child's branch (`orx/<slug>`); check the branch out in the
project worktree, edit only the files that idea touches, and commit. **Leave the run
command alone:**
```sh
DIR=~/.cache/openresearch/repos/<owner>/<repo> # owner/repo from `orx projects`
[ -d "$DIR" ] || git clone https://github.com/<owner>/<repo> "$DIR"
git -C "$DIR" fetch origin && git -C "$DIR" checkout orx/<child-slug>
git -C "$DIR" merge --ff-only origin/orx/<child-slug>
# …edit config.yaml under "$DIR": schedule: constant → cosine …
git -C "$DIR" commit -am "cosine LR + warmup" && git -C "$DIR" push
git checkout orx/<child-slug>
# …edit config.yaml: schedule: constant → cosine …
git commit -am "cosine LR + warmup"
```
While you're in the code, **make the run emit the evidence you'll need to judge
it.** Have it write rollout transcripts, per-sample eval breakdowns, generated
agent-skills/orx-git/SKILL.local.md
+35
View File
@@ -0,0 +1,35 @@
---
name: orx-git
description: "Read, edit, commit, and diff experiment code with local Git. Use whenever you touch an experiment branch, compare nodes, prepare a run, or diagnose stale code."
---
Git records every experiment locally. GitHub publication may be enabled for
collaborator visibility, but it is never part of compute transport. Follow the
project playbook's publication status; do not push merely to launch compute and
do not fetch from or publish back to the paper's upstream repository.
Each experiment node has a local `orx/<slug>` branch. `orx
create-experiment` creates it from its parent. Work in the session worktree,
check out the printed branch, make only that experiment's change, and commit it:
```sh
git checkout orx/<slug>
git status --short
git add <changed files>
git commit -m "describe the experiment change"
```
The runner builds an immutable source archive from the recorded commit, so
committed work is sufficient on every backend. Uncommitted files are never included in a
run. Before launching, confirm `git status --short` is empty and inspect the
recorded commit with `git show --stat --oneline HEAD`.
To compare a child with its parent, use local refs only:
```sh
git diff <parent-branch>...orx/<child-slug>
git log --oneline <parent-branch>..orx/<child-slug>
```
Once a run answers an experiment, treat that branch as immutable and create a
child for the next change.
agent-skills/orx-git/SKILL.md
+13 -15
View File
@@ -1,20 +1,18 @@
---
name: orx-git
description: "Read, edit, and diff a node's code with plain git: sync, commit, and push before running. Use whenever you touch experiment code — before editing any branch, when a checkout or push fails, when comparing two nodes' code, or when a run seems to have picked up stale code."
description: "Read, edit, commit, and diff experiment code with Git. Use whenever you touch a branch, compare nodes, prepare a run, diagnose stale code, or publish changes."
---
Every experiment node **is a git branch** (`orx/<slug>`) on the project's GitHub
repo — `orx create-experiment` prints it. There is no dev box and no `orx` code
command: the **local clone in the cache dir is the standard way to interface
with code** — reading a node's files, diffing what a run changed, and editing —
all with plain git and your own tools.
Every experiment node is a git branch (`orx/<slug>`), but source transport
depends on the project type:
(In a local `orx up` session you already sit in a private git worktree of the
project repo, so you can edit the checked-out branch in place — `git fetch origin
&& git checkout <branch>`, edit, commit, push. The cache-dir clone below is the
flow for everything outside a live session, and for cloud/full-set contexts.)
- For a local project, work in its session worktree, check out the branch, edit,
and commit. Every compute backend receives an immutable snapshot of that local
commit. A GitHub push is optional publication and is never required to run.
- For a managed server project, GitHub remains the source of record. Use the
cache-dir clone flow below and push the branch before running.
**Clone into the openresearch cache dir, not your cwd.** The canonical location,
**For managed server projects, clone into the openresearch cache dir, not your cwd.** The canonical location,
keyed by repo so the same clone is reused across all of a project's experiments:
```
@@ -25,7 +23,7 @@ keyed by repo so the same clone is reused across all of a project's experiments:
directory or the user's project folders — clones accreting in `~/projects` is the
failure mode this avoids.
This is how you **realize a child's hypothesis**: after `create-experiment
For a managed server project, this is how you **realize a child's hypothesis**: after `create-experiment
--parent`, check out the child's branch and make the specific code/config edits
its description calls for — then commit, push, and run. Edit only the files that
idea touches, and **don't touch the run command** (it's inherited; see the
@@ -63,9 +61,9 @@ Rules and notes:
already has — the repo lives under your account or your org, so access is the
same as any of your repos. If a clone or push fails on auth, authenticate git
for github.com (e.g. `gh auth login` or an SSH key) and retry.
- **Push before you run.** `orx exp run` launches from the branch's pushed tip on
GitHub — uncommitted or unpushed edits won't be in the run. Commit and push
first.
- **Push before managed server runs.** Local projects run committed snapshots
directly and do not require a push. Managed server projects launch from the
pushed GitHub tip, so commit and push those branches first.
- **Never merge or rebase a branch once its node is frozen** (cardinal rule):
its history is the code those results came from. Bring changes in on a
**child** instead. On a *provisional* node a plain `git merge
src/commands/exp.rs
+2 -44
View File
@@ -10,8 +10,8 @@
//!
//! This module is now thin: it parses args and resolves the id to a
//! `ControlPlane`, then calls one verb. The per-plane bodies live in
//! `crate::plane::{server_plane, local_plane}`. Only the three job-launch helpers
//! (`hf_clone_script` / `default_hf_image` / `spawn_detached_supervise`) stay
//! `crate::plane::{server_plane, local_plane}`. Only the two job-launch helpers
//! (`default_hf_image` / `spawn_detached_supervise`) stay
//! here — every `src/local/*` backend imports them as `crate::commands::exp::*`.
use std::time::{Duration, Instant};
@@ -98,34 +98,6 @@ async fn wait(
// --- job-launch helpers shared with the src/local/* backends -----------------
/// Fetch one recorded commit and run the fixed command. GITHUB_TOKEN
/// (passed as a job secret when present locally) authenticates private repos
/// through an ephemeral credential helper; the URL and git config stay tokenless.
pub(crate) fn hf_clone_script(git_ref: &str, owner: &str, repo: &str, cmd: &str) -> String {
let url = shell_quote(&format!("https://github.com/{owner}/{repo}.git"));
format!(
"set -eo pipefail; command -v git >/dev/null 2>&1 || (apt-get update -qq && apt-get install -y -qq git); \
git init -q repo; cd repo; \
git remote add origin {url}; \
git -c credential.helper= -c credential.helper='!f() {{ echo username=x-access-token; echo \"password=$GITHUB_TOKEN\"; }}; f' \
fetch --depth 1 origin {git_ref}; git checkout --detach FETCH_HEAD; {cmd}",
git_ref = shell_quote(git_ref),
)
}
fn shell_quote(value: &str) -> String {
format!("'{}'", value.replace('\'', "'\"'\"'"))
}
pub(crate) fn local_clone_script(repo_path: &str, commit_sha: &str, cmd: &str) -> String {
format!(
"set -eo pipefail; git clone --no-checkout --local {} repo; cd repo; git checkout --detach {}; {}",
shell_quote(repo_path),
shell_quote(commit_sha),
cmd
)
}
/// Default docker image per flavor family: plain python for CPU flavors, a
/// CUDA-ready pytorch image for GPU flavors. Override with --image.
pub(crate) fn default_hf_image(flavor: &str) -> String {
@@ -309,17 +281,3 @@ mod tests {
std::fs::remove_dir_all(dir).unwrap();
}
}
#[cfg(test)]
mod clone_script_tests {
use super::*;
#[test]
fn remote_clone_fetches_recorded_commit_detached() {
let script = hf_clone_script("abc123", "owner", "repo", "python run.py");
assert!(script.contains("fetch --depth 1 origin 'abc123'"));
assert!(script.contains("checkout --detach FETCH_HEAD"));
assert!(!script.contains("--branch"));
assert!(!script.contains("x-access-token:${GITHUB_TOKEN}@"));
}
}
src/commands/projects.rs
+1 -10
View File
@@ -47,9 +47,6 @@ pub async fn run(args: crate::ProjectsArgs) -> Result<()> {
"paperId": p.paper_id,
"path": p.repo_path,
"baselineBranch": p.baseline_branch,
"repo": p.has_github_repository().then(|| format!("{}/{}", p.github_owner, p.github_repo)),
"githubEnabled": p.github_enabled(),
"githubUrl": p.github_url(),
"archived": false,
"orgId": serde_json::Value::Null,
"orgName": "Local (orx up)",
@@ -92,19 +89,13 @@ pub async fn run(args: crate::ProjectsArgs) -> Result<()> {
.unwrap_or(0);
for p in &local {
let pad = id_width.saturating_sub(p.id.chars().count());
let publication = if p.github_enabled() {
format!(" · GitHub {}/{}", p.github_owner, p.github_repo)
} else {
String::new()
};
println!(
" {}{} {} (local) {} · baseline {}{}",
" {}{} {} (local) {} · baseline {}",
p.id,
" ".repeat(pad),
p.name,
p.repo_path,
p.baseline_branch,
publication,
);
}
}
src/commands/skill.rs
+4 -58
View File
@@ -1,5 +1,5 @@
use crate::config;
use crate::error::{anyhow, require_credentials, Result};
use crate::error::{require_credentials, Result};
use crate::local::agent_skills::{self, SkillSet};
// Bundled top-level overview, shipped with the CLI so `orx skill` works without
@@ -17,51 +17,14 @@ fn current_skill_set() -> SkillSet {
}
}
fn local_project_publication() -> Option<bool> {
if !crate::local::chat::in_local_session() {
return None;
}
let enabled = (|| {
let session_id = crate::local::chat::launching_chat_session()?;
let store = crate::store::Store::open().ok()?;
let session = store.get_chat_session(&session_id).ok().flatten()?;
store
.get_local_project(&session.project_id)
.ok()
.flatten()
.map(|project| project.github_enabled())
})()
.unwrap_or(false);
Some(enabled)
}
pub async fn run(args: crate::SkillArgs) -> Result<()> {
let publication = local_project_publication();
if let Some(path) = args.path {
// First: a bundled module (with or without the `orx-` prefix). These
// ship in the binary, so they resolve offline and never drift.
if let Some(skill) = agent_skills::find(&path, current_skill_set()) {
if let Some(github_enabled) = publication {
if !agent_skills::available_in_session(skill, github_enabled) {
return Err(anyhow!(
"{} requires GitHub. Enable GitHub syncing for this project first.",
skill.name
));
}
println!(
"{}",
agent_skills::session_content(skill, github_enabled).trim_end()
);
} else {
println!("{}", skill.content.trim_end());
}
println!("{}", skill.content.trim_end());
return Ok(());
}
if publication == Some(false) {
return Err(anyhow!(
"Only bundled local-safe skills are available while this project is local-only. Enable GitHub syncing for this project before loading remote references."
));
}
// Otherwise fetch the canonical doc from the API (same docs the assistant
// reads), so the schema never drifts from a hand-maintained copy.
let creds = require_credentials().await;
@@ -72,28 +35,11 @@ pub async fn run(args: crate::SkillArgs) -> Result<()> {
// No path: print the bundled overview, then the bundled module index, then
// list API-fetchable deep references (best effort — skip if unreachable).
if publication == Some(false) {
println!(
"OpenResearch local-only skills. Commit experiment branches locally and run them on this machine. GitHub and external compute remain unavailable until the user enables GitHub syncing for this project."
);
} else {
println!("{}", SKILL_MD);
}
println!("{}", SKILL_MD);
println!("\nBundled modules (orx skill <name>):");
for s in agent_skills::skills(current_skill_set()) {
let github_enabled = publication.unwrap_or(true);
if agent_skills::available_in_session(s, github_enabled) {
println!(
" {:<20} {}",
s.name,
agent_skills::session_description(s, github_enabled)
);
}
}
if publication == Some(false) {
return Ok(());
println!(" {:<20} {}", s.name, s.description);
}
let creds = match config::load_credentials().await? {
src/commands/supervise.rs
+37 -33
View File
@@ -70,7 +70,25 @@ pub async fn run(args: crate::SuperviseArgs) -> Result<()> {
} else {
Some(require_credentials().await)
};
let descriptor = BackendDescriptor::parse(&stored.backend_json)?;
let mut descriptor = BackendDescriptor::parse(&stored.backend_json)?;
if descriptor.job_id.is_none() {
if let Some(recovered) = crate::compute::recover_submission_handle(&run_id)? {
store.set_backend_json(&run_id, &recovered.to_json())?;
descriptor = recovered;
}
}
if descriptor.job_id.is_none() {
store.update_status(&run_id, "failed", Some(now_ms()), None)?;
store.set_result_markdown(
&run_id,
&format!(
"Submission was interrupted before the {} provider handle was recorded. \
Inspect the provider for resources labelled or_run={run_id} before retrying.",
descriptor.kind.trim_end_matches("_job")
),
)?;
return Ok(());
}
if descriptor.kind == "k8s_job" {
return run_k8s(store, stored, descriptor, creds, run_id).await;
}
@@ -980,35 +998,19 @@ async fn run_openresearch(
.await
.unwrap_or(false);
if !already_launched {
// The payload is re-derivable from the store + config, so a restart
// that died before launching can rebuild it exactly.
let Some(exp) = store.get_local_experiment(&stored.experiment_id)? else {
store.update_status(&run_id, "failed", Some(now_ms()), None)?;
store.set_result_markdown(
&run_id,
"Local experiment vanished from the store before launch.",
)?;
teardown_box(&store, &lifecycle, &sandbox_id, &run_id).await;
return Ok(());
let source = match crate::compute::SourceSnapshot::from_run(&stored, &descriptor) {
Ok(source) => source,
Err(error) => {
store.update_status(&run_id, "failed", Some(now_ms()), None)?;
store.set_result_markdown(
&run_id,
&format!("The recorded source snapshot could not be loaded: {error}"),
)?;
teardown_box(&store, &lifecycle, &sandbox_id, &run_id).await;
return Ok(());
}
};
let Some(project) = store.get_local_project(&exp.project_id)? else {
store.update_status(&run_id, "failed", Some(now_ms()), None)?;
store.set_result_markdown(
&run_id,
"Local project vanished from the store before launch.",
)?;
teardown_box(&store, &lifecycle, &sandbox_id, &run_id).await;
return Ok(());
};
let script = crate::commands::exp::hf_clone_script(
stored
.commit_sha
.as_deref()
.ok_or_else(|| anyhow!("Remote run is missing its recorded commit SHA."))?,
&project.github_owner,
&project.github_repo,
&stored.command,
);
let script = crate::compute::staged_script(&stored.command);
let script =
openresearch::wrap_with_timeout(&script, descriptor.timeout_secs.unwrap_or(4 * 3600));
let mut env: std::collections::HashMap<String, String> =
@@ -1016,10 +1018,6 @@ async fn run_openresearch(
if let Ok(hf_token) = hf::resolve_token() {
env.entry("HF_TOKEN".to_string()).or_insert(hf_token);
}
if let Some(gh) = crate::local::git::resolve_github_token() {
env.insert("GITHUB_TOKEN".to_string(), gh);
}
// sshd and the org key sync can lag a freshly-online box, so the
// launch retries for ~2 minutes before giving up.
let mut launch_err = None;
@@ -1033,6 +1031,12 @@ async fn run_openresearch(
teardown_box(&store, &lifecycle, &sandbox_id, &run_id).await;
return Ok(());
}
let staged = ssh::stage_source(&target, &run_id, &source.path, &source.digest).await;
if let Err(err) = staged {
eprintln!("supervise {run_id}: source staging failed (will retry): {err}");
launch_err = Some(err);
continue;
}
match ssh::run_job(&ssh::SshJobSpec {
target: target.clone(),
run_id: run_id.clone(),
src/commands/up.rs
+255 -189
View File
@@ -274,13 +274,23 @@ fn router(state: AppState) -> Router {
post(disable_project_github),
)
.route("/api/projects/{id}/github/push", post(push_project_github))
.route("/api/github/account", get(github_account))
.route(
"/api/github/project-repo-preview",
get(github_project_repo_preview),
)
.route("/api/github/repo-access", get(github_repo_access))
.route("/api/projects/{id}/experiments", get(list_experiments))
.route("/api/projects/{id}/runs", get(list_project_runs))
.route("/api/papers/search", get(search_papers_api))
.route("/api/papers/resolve", get(resolve_paper_api))
.route("/api/compute/backends", get(compute_backends))
.route("/api/runs", post(create_run))
.route("/api/runs/{id}", get(get_run))
.route("/api/instances", get(list_instances))
.route("/api/runs/{id}/cancel", post(cancel_run))
.route("/api/runs/{id}/log", get(run_log))
.route("/api/runs/{id}/logs", get(run_logs))
.route("/api/runs/{id}/diff", get(run_diff))
.route("/api/experiments/{id}/diff", get(experiment_diff))
.route("/api/experiments/{id}/commits", get(experiment_commits))
@@ -315,8 +325,6 @@ fn router(state: AppState) -> Router {
)
.route("/api/settings/data-dir/validate", post(validate_data_dir))
.route("/api/settings/data-dir/move", post(move_data_dir))
.route("/api/github/repo-access", get(github_repo_access))
.route("/api/github/account", get(github_account))
.route(
"/api/settings/git",
get(git_settings).post(set_git_settings),
@@ -827,6 +835,9 @@ async fn project_path_status(Query(q): Query<ProjectPathStatusQ>) -> ApiResult {
};
let initialized =
git_version.is_some() && directory && local::git::is_repository(&resolved);
let github_publication = initialized
.then(|| local::git::github_publication(&resolved))
.flatten();
Ok(Json(json!({
"gitVersion": git_version,
"resolvedPath": resolved.to_string_lossy(),
@@ -834,6 +845,8 @@ async fn project_path_status(Query(q): Query<ProjectPathStatusQ>) -> ApiResult {
"directory": directory,
"empty": empty,
"initialized": initialized,
"githubOwner": github_publication.as_ref().map(|(owner, _)| owner),
"githubRepo": github_publication.as_ref().map(|(_, repo)| repo),
})))
})
.await
@@ -897,6 +910,7 @@ struct CreateProjectReq {
create_folder: bool,
#[serde(default)]
initialize_git: bool,
github_sync_enabled: Option<bool>,
}
async fn create_project(
@@ -916,9 +930,22 @@ async fn create_project(
let path = req.path;
let create_folder = req.create_folder;
let initialize_git = req.initialize_git;
let clone_url = req.clone_url;
let clone_url = req.clone_url.filter(|url| !url.trim().is_empty());
let paper_id = req.paper_id.filter(|paper_id| !paper_id.trim().is_empty());
if paper_id.is_some() && clone_url.is_none() {
return Err(bad_request(
"A paper project requires a linked public code repository.",
));
}
let github_sync_enabled = req
.github_sync_enabled
.unwrap_or_else(crate::config::github_for_new_projects);
let repo_size_kb = match clone_url.as_deref() {
Some(url) => local::github::public_repo_size_kb(url).await,
None => None,
};
let shallow_clone = local::github::should_shallow_clone(repo_size_kb);
let run_command = req.run_command;
let paper_id = req.paper_id.filter(|p| !p.trim().is_empty());
let result = tokio::task::spawn_blocking(move || {
let store = Store::open()?;
local::projects::create_project(
@@ -929,6 +956,7 @@ async fn create_project(
create_folder,
initialize_git,
clone_url,
shallow_clone,
run_command,
paper_id,
},
@@ -942,7 +970,19 @@ async fn create_project(
.admit(&project.id)
.ok_or_else(|| bad_request("project deletion is in progress"))?;
drop(create_admission);
let (project, github_publication_error) = publish_project_by_default(project).await;
let (project, github_publication_error) = if github_sync_enabled {
match push_project_for_sync(project.clone()).await {
Ok(project) => (project, None),
Err(error) => {
let project = Store::open()?
.get_local_project(&project.id)?
.unwrap_or(project);
(project, Some(error.to_string()))
}
}
} else {
(project, None)
};
crate::telemetry::capture_project_created(true);
Ok(Json(json!({
"project": project_json(&project),
@@ -1066,7 +1106,9 @@ async fn initialize_project_git(
fn push_project(project: &local::model::LocalProject) -> Result<()> {
if !project.github_enabled() {
return Err(anyhow!("Connect GitHub for this project before pushing."));
return Err(anyhow!(
"Enable GitHub syncing for this project before pushing."
));
}
let path = std::path::Path::new(&project.repo_path);
local::git::add_github_remote(path, &project.github_owner, &project.github_repo)?;
@@ -1100,7 +1142,20 @@ async fn create_independent_project_repository(
.map(|session| session.id)
.collect::<Vec<_>>();
local::git::migrate_legacy_project_worktrees(&project, &session_ids)?;
let source_repository = project
.has_github_repository()
.then(|| (project.github_owner.clone(), project.github_repo.clone()));
let reroot_shallow = local::git::prepare_shallow_repository_for_publication(
std::path::Path::new(&project.repo_path),
)?;
let (owner, repo, _) = local::github::create_project_repo(&project.slug).await?;
if reroot_shallow {
local::git::reroot_shallow_repository(
std::path::Path::new(&project.repo_path),
&project.baseline_branch,
source_repository.as_ref(),
)?;
}
project.github_owner = owner;
project.github_repo = repo;
project.github_sync_enabled = false;
@@ -1113,19 +1168,18 @@ async fn push_project_for_sync(
) -> Result<local::model::LocalProject> {
if local::git::resolve_github_token().is_none() {
return Err(anyhow!(
"Connect GitHub first with gh auth login or a GitHub token."
"Connect GitHub first with `gh auth login` or a GitHub token."
));
}
let mut using_existing_repository = project.has_github_repository();
if using_existing_repository {
if let Some(meta) =
local::github::repo_meta(&project.github_owner, &project.github_repo).await
{
if !meta.can_push || meta.archived {
project = create_independent_project_repository(project).await?;
using_existing_repository = false;
}
let can_push = local::github::repo_meta(&project.github_owner, &project.github_repo)
.await
.is_some_and(|meta| meta.can_push && !meta.archived);
if !can_push {
project = create_independent_project_repository(project).await?;
using_existing_repository = false;
}
} else {
project = create_independent_project_repository(project).await?;
@@ -1133,11 +1187,10 @@ async fn push_project_for_sync(
}
let push_once = |project: &local::model::LocalProject| {
let mut project_for_push = project.clone();
project_for_push.github_sync_enabled = true;
tokio::task::spawn_blocking(move || push_project(&project_for_push))
let mut project = project.clone();
project.github_sync_enabled = true;
tokio::task::spawn_blocking(move || push_project(&project))
};
let first_push = push_once(&project)
.await
.map_err(|error| anyhow!("Git push task failed: {error}"))?;
@@ -1156,25 +1209,6 @@ async fn push_project_for_sync(
Ok(project)
}
async fn publish_project_by_default(
project: local::model::LocalProject,
) -> (local::model::LocalProject, Option<String>) {
if !crate::config::github_for_new_projects() || project.github_enabled() {
return (project, None);
}
match push_project_for_sync(project.clone()).await {
Ok(project) => (project, None),
Err(error) => {
let project = Store::open()
.and_then(|store| store.get_local_project(&project.id))
.ok()
.flatten()
.unwrap_or(project);
(project, Some(error.to_string()))
}
}
}
async fn enable_project_github(State(state): State<AppState>, Path(id): Path<String>) -> ApiResult {
reject_if_moving(&state)?;
let _admission = state
@@ -1184,16 +1218,11 @@ async fn enable_project_github(State(state): State<AppState>, Path(id): Path<Str
reject_if_moving(&state)?;
let _lock = project_publication_lock(&state, &id).await;
let store = Store::open()?;
let mut project = store
let project = store
.get_local_project(&id)?
.ok_or_else(|| not_found("project"))?;
project = push_project_for_sync(project).await.map_err(bad_request)?;
let git_status = tokio::task::spawn_blocking({
let project = project.clone();
move || project_git_json(&project)
})
.await
.map_err(|error| ApiError::from(anyhow!("git task failed: {error}")))?;
let project = push_project_for_sync(project).await.map_err(bad_request)?;
let git_status = project_git_json(&project);
Ok(Json(
json!({ "project": project_json(&project), "git": git_status }),
))
@@ -1246,6 +1275,41 @@ async fn push_project_github(State(state): State<AppState>, Path(id): Path<Strin
))
}
async fn github_account() -> ApiResult {
Ok(Json(
json!({ "login": local::github::viewer_login().await }),
))
}
#[derive(Deserialize)]
struct ProjectRepoPreviewQuery {
name: String,
}
async fn github_project_repo_preview(Query(q): Query<ProjectRepoPreviewQuery>) -> ApiResult {
let candidate = local::projects::project_slug_preview(&Store::open()?, q.name.trim())?;
let repo = local::github::available_project_repo_name(&candidate).await;
Ok(Json(json!({ "repo": repo })))
}
#[derive(Deserialize)]
struct RepoAccessQuery {
owner: String,
repo: String,
}
async fn github_repo_access(Query(q): Query<RepoAccessQuery>) -> ApiResult {
let owner = q.owner.trim();
let repo = q.repo.trim();
if owner.is_empty() || repo.is_empty() {
return Err(bad_request("owner and repo are required"));
}
let meta = local::github::repo_meta(owner, repo).await;
Ok(Json(json!({
"canPush": meta.is_some_and(|meta| meta.can_push && !meta.archived),
})))
}
/// Mark a project visited: bumps updated_at, which drives the recency sort
/// and the SSE project.updated diff.
async fn open_project(Path(id): Path<String>) -> ApiResult {
@@ -1314,7 +1378,7 @@ async fn update_project(
/// Delete a project and everything hanging off it. Refuses while runs are in
/// flight (deleting their rows would strand the supervisor mid-job) — but
/// requests their cancellation, so a retry shortly after goes through. The
/// GitHub repo and the cache clone are left untouched.
/// The registered repository folder is left untouched.
async fn delete_project(State(state): State<AppState>, Path(id): Path<String>) -> ApiResult {
reject_if_moving(&state)?;
let _deleting_project = state
@@ -1397,6 +1461,87 @@ async fn list_project_runs(Path(id): Path<String>) -> ApiResult {
Ok(Json(json!({ "runs": runs })))
}
async fn compute_backends() -> Json<Value> {
Json(json!({ "backends": crate::compute::capabilities() }))
}
#[derive(Deserialize)]
#[serde(rename_all = "camelCase")]
struct CreateRunReq {
experiment_id: String,
backend: Option<String>,
flavor: Option<String>,
host: Option<String>,
manifest: Option<String>,
image: Option<String>,
timeout: Option<String>,
org: Option<String>,
provider: Option<String>,
disk: Option<i64>,
#[serde(default)]
force: bool,
}
async fn create_run(State(state): State<AppState>, Json(req): Json<CreateRunReq>) -> ApiResult {
reject_if_moving(&state)?;
let store = Store::open()?;
let experiment = store
.get_local_experiment(&req.experiment_id)?
.ok_or_else(|| not_found("experiment"))?;
let _admission = state
.project_lifecycle
.admit(&experiment.project_id)
.ok_or_else(|| bad_request("project deletion is in progress"))?;
let mut backend = req.backend;
let mut flavor = req.flavor;
local::apply_compute_default(&mut backend, &mut flavor);
let args = crate::ExpRunArgs {
exp_id: req.experiment_id,
gpu: None,
count: None,
disk: req.disk,
provider: req.provider,
cpu: None,
vcpus: None,
sandbox: None,
backend: Some(backend.unwrap_or_else(|| "local".to_string())),
flavor,
org: req.org,
host: req.host,
manifest: req.manifest,
image: req.image,
timeout: req.timeout,
force: req.force,
};
let run = crate::compute::submit(&args).await.map_err(bad_request)?;
Ok(Json(json!({ "run": ApiRun::from(&run) })))
}
fn backend_for_run(
run: &StoredRun,
) -> std::result::Result<Box<dyn crate::compute::ComputeBackend>, ApiError> {
let descriptor =
crate::jobs::BackendDescriptor::parse(&run.backend_json).map_err(bad_request)?;
let id = descriptor
.kind
.strip_suffix("_job")
.unwrap_or(&descriptor.kind);
let id = if id == "k8s" { "k8s" } else { id };
crate::compute::backend(id).map_err(bad_request)
}
async fn get_run(Path(id): Path<String>) -> ApiResult {
let run = Store::open()?
.get_run(&id)?
.ok_or_else(|| not_found("run"))?;
let backend = backend_for_run(&run)?;
let run = backend.status(&run).await.map_err(bad_request)?;
if is_terminal(&run.status) {
backend.cleanup(&run).await.map_err(bad_request)?;
}
Ok(Json(json!({ "run": ApiRun::from(&run) })))
}
/// Newest-first cap for the cross-project instances list. Generous: the store
/// is a local single-user SQLite db, so this only bounds pathological history.
const INSTANCES_LIMIT: usize = 500;
@@ -1433,10 +1578,27 @@ async fn cancel_run(Path(id): Path<String>) -> ApiResult {
if is_terminal(&run.status) {
return Err(bad_request(format!("run already {}", run.status)));
}
crate::commands::exp::request_local_run_cancel(&store, &run.id)?;
let backend = backend_for_run(&run)?;
backend.cancel(&run).await.map_err(bad_request)?;
Ok(Json(json!({ "ok": true })))
}
#[derive(Deserialize)]
struct LogsQuery {
cursor: Option<u64>,
}
async fn run_logs(Path(id): Path<String>, Query(q): Query<LogsQuery>) -> ApiResult {
let run = Store::open()?
.get_run(&id)?
.ok_or_else(|| not_found("run"))?;
let batch = backend_for_run(&run)?
.logs(&run, crate::compute::LogCursor(q.cursor.unwrap_or(0)))
.await
.map_err(bad_request)?;
Ok(Json(json!(batch)))
}
#[derive(Deserialize)]
struct LogQuery {
offset: Option<u64>,
@@ -2271,36 +2433,6 @@ fn data_dir_json() -> Value {
})
}
/// The signed-in GitHub login, so "creates github.com/you/x" can name the real
/// account. `null` when there's no usable token — the UI keeps saying "you".
async fn github_account() -> ApiResult {
Ok(Json(
json!({ "login": local::github::viewer_login().await }),
))
}
#[derive(Deserialize)]
struct RepoAccessQuery {
owner: String,
repo: String,
}
/// Whether the stored credentials can push to `owner/repo`, so the New project
/// form can drop the fork choice when it isn't one. Mirrors the create path:
/// unknown (no token / API hiccup) counts as access, so the UI never nags about
/// a fork the server wouldn't make.
async fn github_repo_access(Query(q): Query<RepoAccessQuery>) -> ApiResult {
let owner = q.owner.trim().to_string();
let repo = q.repo.trim().to_string();
if owner.is_empty() || repo.is_empty() {
return Err(bad_request("owner and repo are required"));
}
let meta = local::github::repo_meta(&owner, &repo).await;
Ok(Json(json!({
"canPush": meta.map(|m| m.can_push && !m.archived).unwrap_or(true),
})))
}
async fn data_dir_settings() -> ApiResult {
tokio::task::spawn_blocking(|| Ok(Json(data_dir_json())))
.await
@@ -2551,25 +2683,16 @@ fn git_out(args: &[&str]) -> Option<String> {
}
fn git_settings_json() -> Value {
// Spawn failure means gh isn't installed — distinct from installed-but-
// signed-out, so the UI can lead with the right fix.
let gh = std::process::Command::new("gh")
.args(["auth", "token"])
.output();
let gh_installed = gh.is_ok();
let github_source = if std::env::var("GITHUB_TOKEN").is_ok_and(|t| !t.trim().is_empty()) {
Some("env")
} else if crate::config::synced_env_var("GITHUB_TOKEN").is_some() {
Some("stored")
} else {
matches!(gh, Ok(out) if out.status.success() && !out.stdout.is_empty()).then_some("gh")
};
let gh_installed = std::process::Command::new("gh")
.arg("--version")
.output()
.is_ok_and(|output| output.status.success());
json!({
"gitVersion": git_out(&["--version"]),
"userName": git_out(&["config", "--global", "user.name"]),
"userEmail": git_out(&["config", "--global", "user.email"]),
"ghInstalled": gh_installed,
"githubTokenSource": github_source,
"githubTokenSource": github_token_source(),
})
}
@@ -2584,9 +2707,7 @@ fn project_defaults_json() -> Value {
}
async fn project_defaults() -> ApiResult {
tokio::task::spawn_blocking(|| Ok(Json(project_defaults_json())))
.await
.map_err(|error| ApiError::from(anyhow!("project defaults task failed: {error}")))?
Ok(Json(project_defaults_json()))
}
#[derive(Deserialize)]
@@ -2598,21 +2719,16 @@ struct SetProjectDefaultsReq {
}
async fn set_project_defaults(Json(req): Json<SetProjectDefaultsReq>) -> ApiResult {
tokio::task::spawn_blocking(move || -> Result<Json<Value>> {
if req.github_for_new_projects && github_token_source().is_none() {
return Err(anyhow!(
"Connect GitHub before enabling it by default for new projects."
));
}
crate::config::set_github_for_new_projects(req.github_for_new_projects)?;
if let Some(seen) = req.github_default_prompt_seen {
crate::config::set_github_default_prompt_seen(seen)?;
}
Ok(Json(project_defaults_json()))
})
.await
.map_err(|error| ApiError::from(anyhow!("project defaults task failed: {error}")))?
.map_err(bad_request)
if req.github_for_new_projects && github_token_source().is_none() {
return Err(bad_request(
"Connect GitHub before enabling it by default for new projects.",
));
}
crate::config::set_github_for_new_projects(req.github_for_new_projects)?;
if let Some(seen) = req.github_default_prompt_seen {
crate::config::set_github_default_prompt_seen(seen)?;
}
Ok(Json(project_defaults_json()))
}
#[derive(Deserialize)]
@@ -2620,60 +2736,42 @@ struct SetGitTokenReq {
token: String,
}
/// Validate a pasted GitHub token against the API, then persist it to the
/// synced env file — the same store job launches already read, so local git
/// ops and remote compute both pick it up.
async fn set_git_token(Json(req): Json<SetGitTokenReq>) -> ApiResult {
let token = req.token.trim().to_string();
if token.is_empty() {
return Err(bad_request("token is required"));
}
let resp = reqwest::Client::new()
let response = reqwest::Client::new()
.get("https://api.github.com/user")
.header("User-Agent", "orx")
.header("Authorization", format!("Bearer {token}"))
.bearer_auth(&token)
.send()
.await
.map_err(|e| bad_request(format!("Could not reach api.github.com: {e}")))?;
if resp.status() == reqwest::StatusCode::UNAUTHORIZED {
return Err(bad_request(
"GitHub rejected the token — check it was copied fully.",
));
}
if !resp.status().is_success() {
.map_err(|error| bad_request(format!("Could not reach api.github.com: {error}")))?;
if !response.status().is_success() {
return Err(bad_request(format!(
"GitHub returned {} validating the token.",
resp.status()
"GitHub rejected the token ({}).",
response.status()
)));
}
// Classic PATs list scopes; fine-grained tokens send an empty header, so
// only enforce when scopes are reported.
let scopes = resp
let scopes = response
.headers()
.get("x-oauth-scopes")
.and_then(|v| v.to_str().ok())
.and_then(|value| value.to_str().ok())
.unwrap_or("")
.to_string();
if !scopes.trim().is_empty() && !scopes.split(',').any(|s| s.trim() == "repo") {
if !scopes.trim().is_empty() && !scopes.split(',').any(|scope| scope.trim() == "repo") {
return Err(bad_request(
"Token is valid but lacks the `repo` scope — private clones and branch pushes would fail.",
"Token is valid but lacks the `repo` scope needed for private repositories.",
));
}
tokio::task::spawn_blocking(move || {
crate::config::write_synced_env_var("GITHUB_TOKEN", &token)?;
Ok(Json(git_settings_json()))
})
.await
.map_err(|e| ApiError::from(anyhow!("git task failed: {e}")))?
crate::config::write_synced_env_var("GITHUB_TOKEN", &token)?;
Ok(Json(git_settings_json()))
}
async fn delete_git_token() -> ApiResult {
tokio::task::spawn_blocking(|| {
crate::config::remove_synced_env_var("GITHUB_TOKEN")?;
Ok(Json(git_settings_json()))
})
.await
.map_err(|e| ApiError::from(anyhow!("git task failed: {e}")))?
crate::config::remove_synced_env_var("GITHUB_TOKEN")?;
Ok(Json(git_settings_json()))
}
async fn git_settings() -> ApiResult {
@@ -3002,7 +3100,7 @@ struct SshPreflightReq {
host: String,
}
/// Live check for one host: can we reach it (BatchMode ssh), and is `git` there?
/// Live check for one host: can we reach it and run bash/tar snapshots?
async fn ssh_preflight(Json(req): Json<SshPreflightReq>) -> ApiResult {
let host = req.host.trim().to_string();
if host.is_empty() {
@@ -3012,7 +3110,7 @@ async fn ssh_preflight(Json(req): Json<SshPreflightReq>) -> ApiResult {
let test = SshHostTest {
host,
reachable: p.reachable,
git_found: p.git_found,
tools_found: p.tools_found,
error: p.error,
tested_at: now_ms(),
};
@@ -3099,7 +3197,7 @@ struct SlurmPreflightReq {
host: String,
}
/// Live check for one login node: reachable, Slurm CLI + git present, and
/// Live check for one login node: reachable, Slurm CLI + snapshot tools, and
/// which partitions exist (feeds the partition picker).
async fn slurm_preflight(Json(req): Json<SlurmPreflightReq>) -> ApiResult {
let host = req.host.trim().to_string();
@@ -3110,7 +3208,7 @@ async fn slurm_preflight(Json(req): Json<SlurmPreflightReq>) -> ApiResult {
Ok(Json(json!({
"reachable": p.reachable,
"slurmFound": p.slurm_found,
"gitFound": p.git_found,
"toolsFound": p.tools_found,
"partitions": p.partitions,
"error": p.error,
})))
@@ -3262,7 +3360,7 @@ fn openresearch_summary(logged_in: bool, ssh: &SshReadiness) -> String {
}
}
fn compute_settings_json(ssh: SshReadiness, github_enabled: bool) -> Value {
fn compute_settings_json(ssh: SshReadiness) -> Value {
let default = crate::config::compute_default();
let (default_backend, default_flavor) = match &default {
Some((b, f)) => (Some(b.as_str()), f.as_deref()),
@@ -3369,30 +3467,15 @@ fn compute_settings_json(ssh: SshReadiness, github_enabled: bool) -> Value {
]);
if let Some(targets) = targets.as_array_mut() {
for target in targets {
let local_target = target.get("id").and_then(Value::as_str) == Some("local");
let enabled = local_target || github_enabled;
if let Some(target) = target.as_object_mut() {
target.insert("enabled".to_string(), Value::Bool(enabled));
target.insert(
"disabledReason".to_string(),
if enabled {
Value::Null
} else {
Value::String("Connect GitHub to enable".to_string())
},
);
target.insert("enabled".to_string(), Value::Bool(true));
target.insert("disabledReason".to_string(), Value::Null);
}
}
}
let effective_backend = if github_enabled {
default_backend.unwrap_or("local")
} else {
"local"
};
let effective_flavor = github_enabled.then_some(default_flavor).flatten();
json!({
"defaultBackend": effective_backend,
"defaultFlavor": effective_flavor,
"defaultBackend": default_backend.unwrap_or("local"),
"defaultFlavor": default_flavor,
"configuredDefaultBackend": default_backend,
"configuredDefaultFlavor": default_flavor,
"targets": targets,
@@ -3405,26 +3488,14 @@ struct ComputeSettingsQuery {
project_id: Option<String>,
}
fn project_github_enabled(project_id: Option<&str>) -> Result<bool> {
let Some(project_id) = project_id else {
return Ok(false);
};
Ok(Store::open()?
.get_local_project(project_id)?
.ok_or_else(|| anyhow!("project not found"))?
.github_enabled())
}
async fn compute_settings(Query(query): Query<ComputeSettingsQuery>) -> ApiResult {
let ssh = openresearch_ssh_readiness().await;
let project_id = query.project_id;
let _project_id = query.project_id;
// fs/env probes only, but keep them off the async runtime anyway.
let payload = tokio::task::spawn_blocking(move || -> Result<Value> {
let github_enabled = project_github_enabled(project_id.as_deref())?;
Ok(compute_settings_json(ssh, github_enabled))
})
.await
.map_err(|e| ApiError::from(anyhow!("compute settings task failed: {e}")))??;
let payload =
tokio::task::spawn_blocking(move || -> Result<Value> { Ok(compute_settings_json(ssh)) })
.await
.map_err(|e| ApiError::from(anyhow!("compute settings task failed: {e}")))??;
Ok(Json(payload))
}
@@ -3441,7 +3512,7 @@ struct SetComputeDefaultReq {
/// (config state fluctuates outside orx; the UI warns instead) — only unknown
/// backends and meaningless flavors are rejected.
async fn set_compute_default(Json(req): Json<SetComputeDefaultReq>) -> ApiResult {
let github_enabled = project_github_enabled(req.project_id.as_deref())?;
let _project_id = req.project_id;
let backend = req
.backend
.map(|b| b.trim().to_string())
@@ -3452,11 +3523,6 @@ async fn set_compute_default(Json(req): Json<SetComputeDefaultReq>) -> ApiResult
.filter(|f| !f.is_empty());
if let Some(b) = &backend {
local::validate_compute_default(b, flavor.as_deref()).map_err(bad_request)?;
if b != "local" && !github_enabled {
return Err(bad_request(
"Connect GitHub for this project before selecting remote compute.",
));
}
}
// Picking openresearch as the default is the moment to answer "will this
// actually work?", so the row that comes back is honest about the SSH key.
@@ -3466,7 +3532,7 @@ async fn set_compute_default(Json(req): Json<SetComputeDefaultReq>) -> ApiResult
// the plain ApiError conversion, not as a 400 blaming the request.
let payload = tokio::task::spawn_blocking(move || -> Result<Value> {
crate::config::set_compute_default(backend, flavor)?;
Ok(compute_settings_json(ssh, github_enabled))
Ok(compute_settings_json(ssh))
})
.await
.map_err(|e| ApiError::from(anyhow!("compute default task failed: {e}")))??;
src/compute.rs
+807
View File
@@ -0,0 +1,807 @@
//! Backend-agnostic compute lifecycle and immutable source snapshots.
//!
//! Local Git remains the experiment-history database. A launch never asks a
//! remote backend to clone that history: it archives the exact recorded commit
//! once, addresses the archive by SHA-256, and hands that immutable payload to
//! the selected provider adapter.
use std::io::Read as _;
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
#[cfg(unix)]
use std::os::unix::fs::{OpenOptionsExt, PermissionsExt};
use async_trait::async_trait;
use serde::Serialize;
use sha2::{Digest as _, Sha256};
use crate::error::{anyhow, Result};
use crate::jobs::BackendDescriptor;
use crate::local::model::{LocalExperiment, LocalProject};
use crate::store::{log_path, Store, StoredRun};
#[derive(Debug, Clone)]
pub struct SourceSnapshot {
pub revision: String,
pub digest: String,
pub size: u64,
pub path: PathBuf,
pub ray_package: Option<(String, PathBuf)>,
}
impl SourceSnapshot {
pub fn create(
project: &LocalProject,
experiment: &LocalExperiment,
include_ray_package: bool,
) -> Result<Self> {
let repo = Path::new(&project.repo_path);
let revision = crate::local::git::local_head_sha(repo, &experiment.branch_name)?;
let dir = crate::store::data_dir().join("source-snapshots");
prepare_snapshot_dir(&dir)?;
let nonce = uuid::Uuid::new_v4();
let tar_tmp = dir.join(format!(".{nonce}.tar"));
archive(repo, &revision, "tar", &tar_tmp)?;
let (digest, size) = digest_file(&tar_tmp)?;
let path = dir.join(format!("{digest}.tar"));
install_content_addressed(&tar_tmp, &path, &digest, size)?;
let ray_package = if include_ray_package {
let zip_tmp = dir.join(format!(".{nonce}.zip"));
archive(repo, &revision, "zip", &zip_tmp)?;
let (zip_digest, zip_size) = digest_file(&zip_tmp)?;
let zip_path = dir.join(format!("{zip_digest}.zip"));
install_content_addressed(&zip_tmp, &zip_path, &zip_digest, zip_size)?;
Some((zip_digest, zip_path))
} else {
None
};
Ok(Self {
revision,
digest,
size,
path,
ray_package,
})
}
pub fn apply_to_descriptor(&self, descriptor: &mut BackendDescriptor) {
descriptor.source_digest = Some(self.digest.clone());
descriptor.source_path = Some(self.path.to_string_lossy().into_owned());
descriptor.source_size = Some(self.size);
}
pub fn from_run(run: &StoredRun, descriptor: &BackendDescriptor) -> Result<Self> {
let revision = run
.commit_sha
.clone()
.ok_or_else(|| anyhow!("Run {} has no recorded source revision.", run.id))?;
let digest = descriptor
.source_digest
.clone()
.ok_or_else(|| anyhow!("Run {} has no recorded source digest.", run.id))?;
let recorded_path = descriptor
.source_path
.as_deref()
.map(PathBuf::from)
.ok_or_else(|| anyhow!("Run {} has no recorded source archive.", run.id))?;
let path = if recorded_path.is_file() {
recorded_path
} else {
crate::store::data_dir()
.join("source-snapshots")
.join(format!("{digest}.tar"))
};
if !path.is_file() {
return Err(anyhow!(
"Run {} source archive is missing at {}.",
run.id,
path.display()
));
}
let size = descriptor
.source_size
.or_else(|| std::fs::metadata(&path).ok().map(|m| m.len()))
.unwrap_or(0);
let (actual_digest, actual_size) = digest_file(&path)?;
if actual_digest != digest || (size != 0 && actual_size != size) {
return Err(anyhow!(
"Run {} source archive failed its digest check.",
run.id
));
}
Ok(Self {
revision,
digest,
size,
path,
ray_package: None,
})
}
}
fn archive(repo: &Path, revision: &str, format: &str, destination: &Path) -> Result<()> {
let mut options = std::fs::OpenOptions::new();
options.write(true).create_new(true);
#[cfg(unix)]
options.mode(0o600);
let file = options.open(destination)?;
let output = Command::new("git")
.current_dir(repo)
.args(["archive", &format!("--format={format}"), revision])
.stdout(Stdio::from(file))
.stderr(Stdio::piped())
.output()
.map_err(|e| anyhow!("Could not run git archive: {e}"))?;
if output.status.success() {
return Ok(());
}
let _ = std::fs::remove_file(destination);
Err(anyhow!(
"git archive failed for {}: {}",
revision,
String::from_utf8_lossy(&output.stderr).trim()
))
}
fn prepare_snapshot_dir(path: &Path) -> Result<()> {
std::fs::create_dir_all(path)?;
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt;
let metadata = std::fs::symlink_metadata(path)?;
if !metadata.file_type().is_dir() || metadata.uid() != unsafe { libc::geteuid() } {
return Err(anyhow!(
"Source snapshot directory {} is not owned by the current user.",
path.display()
));
}
std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o700))?;
}
Ok(())
}
fn digest_file(path: &Path) -> Result<(String, u64)> {
let mut file = std::fs::File::open(path)?;
let mut hasher = Sha256::new();
let mut size = 0u64;
let mut buf = [0u8; 128 * 1024];
loop {
let read = file.read(&mut buf)?;
if read == 0 {
break;
}
hasher.update(&buf[..read]);
size += read as u64;
}
Ok((format!("{:x}", hasher.finalize()), size))
}
fn install_content_addressed(
source: &Path,
destination: &Path,
expected_digest: &str,
expected_size: u64,
) -> Result<()> {
if destination.exists() {
let (digest, size) = digest_file(destination)?;
if digest == expected_digest && size == expected_size {
std::fs::remove_file(source)?;
restrict_snapshot_file(destination)?;
return Ok(());
}
std::fs::remove_file(destination)?;
}
match std::fs::rename(source, destination) {
Ok(()) => restrict_snapshot_file(destination),
Err(_err) if destination.exists() => {
let (digest, size) = digest_file(destination)?;
std::fs::remove_file(source)?;
if digest == expected_digest && size == expected_size {
restrict_snapshot_file(destination)
} else {
Err(anyhow!(
"Cached source snapshot {} failed its digest check.",
destination.display()
))
}
}
Err(err) => Err(err.into()),
}
}
fn restrict_snapshot_file(path: &Path) -> Result<()> {
#[cfg(unix)]
std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o600))?;
Ok(())
}
pub fn snapshot_script(archive_path: &str, command: &str) -> String {
format!(
"set -eo pipefail; mkdir -p repo; tar -xf {} -C repo; cd repo; {}",
shell_quote(archive_path),
command
)
}
pub fn staged_script(command: &str) -> String {
format!("set -eo pipefail; cd repo; {command}")
}
pub fn gated_script(archive_path: &str, command: &str) -> String {
format!(
"set -eo pipefail; mkdir -p \"$(dirname -- {archive})\"; while [ ! -f {ready} ]; do sleep 0.1; done; mkdir -p repo; tar -xf {archive} -C repo; cd repo; {command}",
archive = shell_quote(archive_path),
ready = shell_quote(&format!("{archive_path}.ready")),
)
}
fn shell_quote(value: &str) -> String {
format!("'{}'", value.replace('\'', "'\"'\"'"))
}
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct Capabilities {
pub id: &'static str,
pub label: &'static str,
pub remote: bool,
pub flavors: bool,
pub requires_flavor: bool,
pub source_transport: &'static str,
}
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct Preflight {
pub ready: bool,
pub detail: Option<String>,
}
#[derive(Debug, Clone)]
pub struct StagedSource(pub SourceSnapshot);
#[derive(Debug, Clone, Copy, Default)]
pub struct LogCursor(pub u64);
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct LogBatch {
pub data_base64: String,
pub next_cursor: u64,
pub eof: bool,
}
#[async_trait]
pub trait ComputeBackend: Send + Sync {
fn capabilities(&self) -> Capabilities;
async fn preflight(&self, args: &crate::ExpRunArgs) -> Result<Preflight>;
async fn stage_source(
&self,
project: &LocalProject,
experiment: &LocalExperiment,
) -> Result<StagedSource>;
async fn submit(
&self,
args: &crate::ExpRunArgs,
source: StagedSource,
run_id: String,
) -> Result<StoredRun>;
async fn status(&self, handle: &StoredRun) -> Result<StoredRun> {
Store::open()?
.get_run(&handle.id)?
.ok_or_else(|| anyhow!("Run {} not found.", handle.id))
}
async fn logs(&self, handle: &StoredRun, cursor: LogCursor) -> Result<LogBatch> {
use base64::Engine as _;
use std::io::{Read as _, Seek as _};
let path = log_path(&handle.id);
let mut file = match std::fs::File::open(&path) {
Ok(file) => file,
Err(_) => {
return Ok(LogBatch {
data_base64: String::new(),
next_cursor: cursor.0,
eof: crate::local::is_terminal(&handle.status),
})
}
};
let len = file.metadata()?.len();
let start = cursor.0.min(len);
file.seek(std::io::SeekFrom::Start(start))?;
let mut data = Vec::new();
file.take(256 * 1024).read_to_end(&mut data)?;
let next = start + data.len() as u64;
Ok(LogBatch {
data_base64: base64::engine::general_purpose::STANDARD.encode(data),
next_cursor: next,
eof: crate::local::is_terminal(&handle.status) && next >= len,
})
}
async fn cancel(&self, handle: &StoredRun) -> Result<()> {
crate::commands::exp::request_local_run_cancel(&Store::open()?, &handle.id)
}
async fn cleanup(&self, handle: &StoredRun) -> Result<()> {
if self.capabilities().id == "hf" && crate::local::is_terminal(&handle.status) {
let descriptor = BackendDescriptor::parse(&handle.backend_json)?;
if let (Some(path), Some(digest)) = (descriptor.source_path, descriptor.source_digest) {
let staging = Path::new(&path)
.parent()
.unwrap_or_else(|| Path::new("."))
.join(format!("{digest}.hf"));
if staging.is_dir() {
std::fs::remove_dir_all(staging)?;
}
}
}
Ok(())
}
}
async fn stage_snapshot(
project: &LocalProject,
experiment: &LocalExperiment,
include_ray_package: bool,
) -> Result<StagedSource> {
let project = project.clone();
let experiment = experiment.clone();
tokio::task::spawn_blocking(move || {
SourceSnapshot::create(&project, &experiment, include_ray_package)
})
.await
.map_err(|error| anyhow!("source snapshot task failed: {error}"))?
.map(StagedSource)
}
fn ready() -> Result<Preflight> {
Ok(Preflight {
ready: true,
detail: None,
})
}
macro_rules! backend_adapter {
(
$name:ident, $id:literal, $label:literal, $remote:literal, $flavors:literal,
$requires_flavor:literal, $transport:literal, $ray_package:literal,
preflight |$preflight_args:ident| $preflight:expr,
submit |$submit_args:ident, $source:ident, $run_id:ident| $submit:expr
) => {
pub struct $name;
#[async_trait]
impl ComputeBackend for $name {
fn capabilities(&self) -> Capabilities {
Capabilities {
id: $id,
label: $label,
remote: $remote,
flavors: $flavors,
requires_flavor: $requires_flavor,
source_transport: $transport,
}
}
async fn preflight(&self, args: &crate::ExpRunArgs) -> Result<Preflight> {
let $preflight_args = args;
$preflight
}
async fn stage_source(
&self,
project: &LocalProject,
experiment: &LocalExperiment,
) -> Result<StagedSource> {
stage_snapshot(project, experiment, $ray_package).await
}
async fn submit(
&self,
args: &crate::ExpRunArgs,
staged: StagedSource,
id: String,
) -> Result<StoredRun> {
let $submit_args = args;
let $source = staged.0;
let $run_id = id;
$submit
}
}
};
}
backend_adapter!(
LocalCompute,
"local",
"This machine",
false,
false,
false,
"local archive",
false,
preflight | _args | ready(),
submit | args,
source,
run_id | crate::local::localrun::submit_local_run_with_source(args, source, run_id).await
);
backend_adapter!(
HuggingFaceCompute,
"hf",
"Hugging Face Jobs",
true,
true,
true,
"private job volume",
false,
preflight | args | {
if args.flavor.is_none() {
return Ok(not_ready("Hugging Face Jobs requires --flavor."));
}
let token = crate::jobs::huggingface::resolve_token()?;
crate::jobs::huggingface::whoami(&token).await?;
ready()
},
submit | args,
source,
run_id | crate::local::hf::submit_local_hf_with_source(args, source, run_id).await
);
backend_adapter!(
ModalCompute,
"modal",
"Modal",
true,
true,
true,
"sandbox filesystem",
false,
preflight | args | {
if args.flavor.is_none() {
return Ok(not_ready("Modal requires --flavor."));
}
crate::jobs::modal::preflight().await?;
ready()
},
submit | args,
source,
run_id | crate::local::modal::submit_local_modal_with_source(args, source, run_id).await
);
backend_adapter!(
KubernetesCompute,
"k8s",
"Kubernetes",
true,
false,
false,
"kubectl cp",
false,
preflight | _args | {
let settings = crate::jobs::kubernetes::load_settings()?.unwrap_or_default();
let check =
crate::jobs::kubernetes::preflight(settings.context.as_deref(), &settings.namespace)
.await;
if !check.kubectl_found || !check.reachable || !check.can_create_jobs {
return Ok(not_ready(check.error.as_deref().unwrap_or(
"kubectl cannot reach the cluster or create Jobs in the namespace.",
)));
}
ready()
},
submit | args,
source,
run_id | crate::local::k8s::submit_local_k8s_with_source(args, source, run_id).await
);
backend_adapter!(
SshCompute,
"ssh",
"SSH",
true,
false,
false,
"SSH tar stream",
false,
preflight | args | {
let host = args
.host
.as_deref()
.ok_or_else(|| anyhow!("SSH requires --host <alias>."))?;
let check = crate::jobs::ssh::preflight(&crate::jobs::ssh::SshTarget::alias(host)).await;
if !check.reachable || !check.tools_found {
return Ok(not_ready(
check
.error
.as_deref()
.unwrap_or("The SSH host needs bash and tar."),
));
}
ready()
},
submit | args,
source,
run_id | crate::local::ssh::submit_local_ssh_with_source(args, source, run_id).await
);
backend_adapter!(
SlurmCompute,
"slurm",
"Slurm",
true,
false,
false,
"SSH tar stream",
false,
preflight | args | {
let settings = crate::jobs::slurm::load_settings()?.unwrap_or_default();
let host = args
.host
.as_deref()
.or(settings.host.as_deref())
.ok_or_else(|| anyhow!("Slurm requires --host or a configured host."))?;
let check = crate::jobs::slurm::preflight(host).await;
if !check.reachable || !check.slurm_found || !check.tools_found {
return Ok(not_ready(check.error.as_deref().unwrap_or(
"The Slurm host needs bash, tar, sbatch, squeue, and scancel.",
)));
}
ready()
},
submit | args,
source,
run_id | crate::local::slurm::submit_local_slurm_with_source(args, source, run_id).await
);
backend_adapter!(
RayCompute,
"ray",
"Ray Jobs",
true,
false,
false,
"working_dir package",
true,
preflight | _args | {
let address = crate::jobs::ray::resolve_address(None);
crate::jobs::ray::preflight(&address).await?;
ready()
},
submit | args,
source,
run_id | crate::local::ray::submit_local_ray_with_source(args, source, run_id).await
);
backend_adapter!(
OpenResearchCompute,
"openresearch",
"OpenResearch",
true,
true,
true,
"SSH tar stream",
false,
preflight | args | {
if args.flavor.is_none() {
return Ok(not_ready("OpenResearch requires --flavor."));
}
if crate::config::load_credentials().await?.is_none() {
return Ok(not_ready("OpenResearch requires `orx login`."));
}
ready()
},
submit | args,
source,
run_id
| crate::local::openresearch::submit_local_openresearch_with_source(args, source, run_id)
.await
);
pub fn backend(id: &str) -> Result<Box<dyn ComputeBackend>> {
match id {
"local" => Ok(Box::new(LocalCompute)),
"hf" => Ok(Box::new(HuggingFaceCompute)),
"modal" => Ok(Box::new(ModalCompute)),
"k8s" => Ok(Box::new(KubernetesCompute)),
"ssh" => Ok(Box::new(SshCompute)),
"slurm" => Ok(Box::new(SlurmCompute)),
"ray" => Ok(Box::new(RayCompute)),
"openresearch" => Ok(Box::new(OpenResearchCompute)),
_ => Err(anyhow!("Unknown compute backend '{id}'.")),
}
}
pub fn capabilities() -> Vec<Capabilities> {
crate::local::BACKENDS
.iter()
.filter_map(|id| backend(id).ok())
.map(|backend| backend.capabilities())
.collect()
}
pub async fn submit(args: &crate::ExpRunArgs) -> Result<StoredRun> {
let backend_id = args.backend.as_deref().unwrap_or("local");
let backend = backend(backend_id)?;
let store = Store::open()?;
let experiment = store
.get_local_experiment(&args.exp_id)?
.ok_or_else(|| anyhow!("Local experiment {} not found.", args.exp_id))?;
let project = store
.get_local_project(&experiment.project_id)?
.ok_or_else(|| anyhow!("Local project {} not found.", experiment.project_id))?;
let preflight = backend.preflight(args).await?;
if !preflight.ready {
return Err(anyhow!(
"{}",
preflight
.detail
.unwrap_or_else(|| "Compute backend is not ready.".to_string())
));
}
let source = backend.stage_source(&project, &experiment).await?;
let run_id = uuid::Uuid::new_v4().to_string();
let command = Some(experiment.run_command.clone())
.filter(|value| !value.trim().is_empty())
.or_else(|| {
project
.run_command
.clone()
.filter(|value| !value.trim().is_empty())
})
.unwrap_or_default();
let mut descriptor = BackendDescriptor {
kind: format!("{}_job", backend_id),
namespace: None,
job_id: None,
flavor: args.flavor.clone(),
image: args.image.clone(),
url: None,
context: None,
manifest: args.manifest.clone(),
resources: None,
ssh_host: None,
ssh_port: None,
ssh_user: None,
timeout_secs: None,
source_digest: None,
source_path: None,
source_size: None,
};
source.0.apply_to_descriptor(&mut descriptor);
let now = crate::store::now_ms();
let pending = StoredRun {
id: run_id.clone(),
experiment_id: experiment.id.clone(),
project_id: project.id.clone(),
status: "starting".to_string(),
backend_json: descriptor.to_json(),
command,
created_at: now,
updated_at: now,
ended_at: None,
exit_code: None,
commit_sha: Some(source.0.revision.clone()),
result_markdown: None,
cancel_requested: false,
chat_session_id: crate::local::chat::launching_chat_session(),
};
reserve_run(&store, &pending, args.force)?;
let pending_backend_json = descriptor.to_json();
match backend.submit(args, source, run_id.clone()).await {
Ok(run) => {
if project.github_enabled() {
if let Err(error) = crate::local::git::spawn_branch_publication(
Path::new(&project.repo_path),
&experiment.branch_name,
&project.github_owner,
&project.github_repo,
) {
eprintln!(
"GitHub sync could not start; compute is already running from the source snapshot: {error}"
);
}
}
Ok(run)
}
Err(error) => {
let current = store.get_run(&run_id)?;
let handle_was_persisted = current
.as_ref()
.is_some_and(|run| run.backend_json != pending_backend_json);
if !handle_was_persisted {
store.update_status(&run_id, "failed", Some(crate::store::now_ms()), None)?;
store
.set_result_markdown(&run_id, &format!("Compute submission failed: {error}"))?;
}
Err(error)
}
}
}
fn reserve_run(store: &Store, pending: &StoredRun, force: bool) -> Result<()> {
let dir = crate::store::data_dir().join("submission-locks");
std::fs::create_dir_all(&dir)?;
let file = std::fs::OpenOptions::new()
.create(true)
.read(true)
.write(true)
.truncate(false)
.open(dir.join(&pending.experiment_id))?;
let mut lock = fd_lock::RwLock::new(file);
let _guard = lock.write()?;
if !force {
if let Some(run) = store
.list_runs_by_experiment(&pending.experiment_id)?
.into_iter()
.find(|run| !crate::local::is_terminal(&run.status))
{
return Err(anyhow!(
"Run {} is already in flight for this experiment ({}). Cancel it with \
`orx exp cancel {}` or pass --force to launch anyway.",
run.id,
run.status,
pending.experiment_id
));
}
}
store.upsert_run(pending)
}
pub fn record_submission_handle(run_id: &str, descriptor: &BackendDescriptor) -> Result<()> {
let database_error = Store::open()
.and_then(|store| store.set_backend_json(run_id, &descriptor.to_json()))
.err();
let dir = crate::store::data_dir().join("submission-handles");
if let Err(error) = std::fs::create_dir_all(&dir) {
return match database_error {
None => {
eprintln!("warning: could not create submission recovery directory: {error}");
Ok(())
}
Some(database_error) => Err(anyhow!(
"Could not persist provider handle in SQLite ({database_error}) or the recovery directory ({error})."
)),
};
}
let destination = dir.join(format!("{run_id}.json"));
let temporary = dir.join(format!(".{run_id}.{}.tmp", uuid::Uuid::new_v4()));
let file_error = std::fs::write(&temporary, descriptor.to_json())
.and_then(|()| std::fs::rename(&temporary, destination))
.err();
if let Some(error) = file_error {
let _ = std::fs::remove_file(temporary);
if let Some(database_error) = database_error {
return Err(anyhow!(
"Could not persist provider handle in SQLite ({database_error}) or its recovery file ({error})."
));
}
eprintln!("warning: could not write redundant submission recovery record: {error}");
}
Ok(())
}
pub fn recover_submission_handle(run_id: &str) -> Result<Option<BackendDescriptor>> {
let path = crate::store::data_dir()
.join("submission-handles")
.join(format!("{run_id}.json"));
match std::fs::read_to_string(path) {
Ok(json) => BackendDescriptor::parse(&json).map(Some),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(None),
Err(error) => Err(error.into()),
}
}
fn not_ready(detail: impl Into<String>) -> Preflight {
Preflight {
ready: false,
detail: Some(detail.into()),
}
}
src/jobs/huggingface.rs
+172 -20
View File
@@ -10,6 +10,8 @@
//! plain `Bearer` header on every call including the log stream.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::process::Stdio;
use std::sync::OnceLock;
use std::time::Duration;
@@ -19,6 +21,115 @@ use serde_json::json;
use crate::error::{anyhow, Result};
const SOURCE_LAUNCHER: &str = r#"
import json, sys
from huggingface_hub import HfApi
spec = json.load(sys.stdin)
api = HfApi(endpoint=spec["endpoint"], token=spec["token"])
volume = api.sync_job_volume(
spec["sourceDir"],
"/orx-source",
namespace=spec["namespace"],
remote_name="orx-" + spec["digest"],
read_only=True,
)
job = api.run_job(
image=spec["image"],
command=spec["command"],
env=spec["environment"],
secrets=spec["secrets"],
flavor=spec["flavor"],
timeout=spec["timeoutSeconds"],
labels=spec["labels"],
volumes=[volume],
namespace=spec["namespace"],
)
print(json.dumps({"id": job.id}))
"#;
fn managed_python() -> PathBuf {
let env = crate::config::config_dir().join("envs").join("huggingface");
if cfg!(windows) {
env.join("Scripts").join("python.exe")
} else {
env.join("bin").join("python")
}
}
async fn ensure_client_env() -> Result<PathBuf> {
static INSTALL_LOCK: OnceLock<tokio::sync::Mutex<()>> = OnceLock::new();
let _install = INSTALL_LOCK
.get_or_init(|| tokio::sync::Mutex::new(()))
.lock()
.await;
let python = managed_python();
if python.exists() {
let ready = tokio::process::Command::new(&python)
.args([
"-c",
"from huggingface_hub import HfApi; assert hasattr(HfApi, 'sync_job_volume')",
])
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null())
.status()
.await
.map(|status| status.success())
.unwrap_or(false);
if ready {
return Ok(python);
}
}
let base = ["python3", "python"]
.into_iter()
.find(|candidate| {
std::process::Command::new(candidate)
.args(["-c", "import venv"])
.status()
.map(|status| status.success())
.unwrap_or(false)
})
.ok_or_else(|| anyhow!("Python 3 is required to stage source for Hugging Face Jobs."))?;
let env_dir = python
.parent()
.and_then(Path::parent)
.ok_or_else(|| anyhow!("Invalid managed Hugging Face environment path."))?;
if !python.exists() {
if let Some(parent) = env_dir.parent() {
std::fs::create_dir_all(parent)?;
}
let status = tokio::process::Command::new(base)
.args(["-m", "venv"])
.arg(env_dir)
.status()
.await?;
if !status.success() {
return Err(anyhow!(
"Could not create the Hugging Face client environment."
));
}
}
eprintln!("orx: installing the Hugging Face source-transfer client (one time)…");
let status = tokio::process::Command::new(&python)
.args([
"-m",
"pip",
"install",
"--quiet",
"--disable-pip-version-check",
"huggingface_hub>=1.8.0",
])
.status()
.await?;
if !status.success() {
return Err(anyhow!(
"Could not install the Hugging Face source-transfer client."
));
}
Ok(python)
}
pub fn endpoint() -> String {
std::env::var("HF_ENDPOINT").unwrap_or_else(|_| "https://huggingface.co".to_string())
}
@@ -188,31 +299,72 @@ pub struct JobSubmission {
pub labels: HashMap<String, String>,
}
pub async fn run_job(token: &str, namespace: &str, spec: &JobSubmission) -> Result<JobInfo> {
// Mirror the python client: arguments/environment always present.
let mut body = json!({
/// Sync the immutable archive into the private `jobs-artifacts` bucket and
/// mount it into the Job. The digest-derived remote name makes retries upload
/// nothing when the exact source was already staged.
pub async fn run_job_with_source(
token: &str,
namespace: &str,
spec: &JobSubmission,
archive: &Path,
digest: &str,
) -> Result<JobInfo> {
let python = ensure_client_env().await?;
let source_dir = archive
.parent()
.unwrap_or_else(|| Path::new("."))
.join(format!("{digest}.hf"));
std::fs::create_dir_all(&source_dir)?;
let staged = source_dir.join("source.tar");
if !staged.exists() && std::fs::hard_link(archive, &staged).is_err() {
std::fs::copy(archive, &staged)?;
}
let body = json!({
"endpoint": endpoint(),
"token": token,
"namespace": namespace,
"sourceDir": source_dir,
"digest": digest,
"image": spec.docker_image,
"command": spec.command,
"arguments": [],
"environment": spec.environment,
"environment": super::default_unbuffered(&spec.environment),
"secrets": spec.secrets,
"flavor": spec.flavor,
"dockerImage": spec.docker_image,
"timeoutSeconds": spec.timeout_seconds,
"labels": spec.labels,
});
if !spec.secrets.is_empty() {
body["secrets"] = json!(spec.secrets);
let mut child = tokio::process::Command::new(python)
.args(["-c", SOURCE_LAUNCHER])
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()?;
use tokio::io::AsyncWriteExt as _;
child
.stdin
.take()
.expect("piped stdin")
.write_all(body.to_string().as_bytes())
.await?;
let output = child.wait_with_output().await?;
if !output.status.success() {
return Err(anyhow!(
"Hugging Face source staging failed: {}",
String::from_utf8_lossy(&output.stderr).trim()
));
}
if !spec.labels.is_empty() {
body["labels"] = json!(spec.labels);
}
let res = http()
.post(format!("{}/api/jobs/{}", endpoint(), namespace))
.bearer_auth(token)
.json(&body)
.send()
.await
.map_err(|e| anyhow!("Could not reach Hugging Face: {}", e))?;
let job: JobInfo = check(res, "job submit").await?.json().await?;
Ok(job)
let value: serde_json::Value = serde_json::from_slice(&output.stdout)?;
let id = value["id"]
.as_str()
.filter(|id| !id.is_empty())
.ok_or_else(|| anyhow!("Hugging Face source launch returned no job id."))?;
Ok(JobInfo {
id: id.to_string(),
status: JobStatus {
stage: "SCHEDULING".to_string(),
message: None,
},
})
}
pub async fn inspect_job(token: &str, namespace: &str, job_id: &str) -> Result<JobInfo> {
src/jobs/kubernetes.rs
+100 -12
View File
@@ -13,7 +13,7 @@
//! - the manifest must contain exactly one Job (or mark one of several with
//! the `orx-primary: "true"` label) — its completion/failure is the run's;
//! - the Job's container command must reference `$ORX_SCRIPT`, the injected
//! env var holding the clone-and-run script (the run command stays the
//! env var holding the snapshot-and-run script (the run command stays the
//! experiment's fixed contract);
//! - orx injects run labels, the `orx-env` Secret ref, and defaults for
//! `activeDeadlineSeconds` / `ttlSecondsAfterFinished` / `backoffLimit`
@@ -28,6 +28,7 @@
//! `is_terminal_stage` in `jobs/mod.rs` apply unchanged.
use std::collections::HashMap;
use std::path::PathBuf;
use std::process::Stdio;
use std::time::Duration;
@@ -231,7 +232,7 @@ pub struct ManifestSpec {
/// Raw manifest text as committed on the experiment branch (YAML or JSON,
/// multi-document fine).
pub manifest: String,
/// Clone-and-run script; injected as the `ORX_SCRIPT` env var on the
/// Snapshot-and-run script; injected as the `ORX_SCRIPT` env var on the
/// primary Job's containers.
pub script: String,
/// Run-unique DNS-safe token substituted for `{{ORX_RUN}}`.
@@ -241,6 +242,7 @@ pub struct ManifestSpec {
/// Injected as `activeDeadlineSeconds` when the manifest doesn't set one.
pub timeout_seconds: u64,
pub labels: HashMap<String, String>,
pub source_archive: PathBuf,
}
pub struct Submitted {
@@ -318,12 +320,78 @@ pub async fn run_manifest(
}
created.push(handle);
}
let staged = stage_source(context, namespace, &job_name, &spec.source_archive).await;
if let Err(error) = staged {
for resource in created.iter().rev() {
let _ = delete_resources(context, namespace, std::slice::from_ref(resource)).await;
}
return Err(error);
}
Ok(Submitted {
job_name,
resources: created,
})
}
async fn stage_source(
context: Option<&str>,
namespace: &str,
job_name: &str,
archive: &std::path::Path,
) -> Result<()> {
let selector = format!("job-name={job_name}");
let deadline = std::time::Instant::now() + Duration::from_secs(120);
let mut last_error = String::new();
while std::time::Instant::now() < deadline {
let pod = kubectl(
context,
&[
"get",
"pods",
"-n",
namespace,
"-l",
&selector,
"-o",
"jsonpath={.items[0].metadata.name}",
],
None,
)
.await
.unwrap_or_default();
let pod = pod.trim();
if !pod.is_empty() {
let local = archive.to_string_lossy().into_owned();
let remote = format!("{namespace}/{pod}:/tmp/orx-source/source.tar");
match kubectl(context, &["cp", &local, &remote], None).await {
Ok(_) => match kubectl(
context,
&[
"exec",
"-n",
namespace,
pod,
"--",
"touch",
"/tmp/orx-source/source.tar.ready",
],
None,
)
.await
{
Ok(_) => return Ok(()),
Err(error) => last_error = error.to_string(),
},
Err(error) => last_error = error.to_string(),
}
}
tokio::time::sleep(Duration::from_secs(1)).await;
}
Err(anyhow!(
"Kubernetes source staging timed out for Job {namespace}/{job_name}: {last_error}"
))
}
fn resource_handle(doc: &Value) -> String {
format!(
"{}/{}",
@@ -436,6 +504,15 @@ fn prepare_docs(
let job = &mut docs[primary];
let job_name = job["metadata"]["name"].as_str().unwrap_or("").to_string();
let completions = job["spec"]["completions"].as_u64().unwrap_or(1);
let parallelism = job["spec"]["parallelism"].as_u64().unwrap_or(1);
if completions > 1 || parallelism > 1 || job["spec"]["completionMode"] == "Indexed" {
return Err(anyhow!(
"Job '{}' must run a single pod; parallel and Indexed Jobs cannot receive a \
pod-local source snapshot safely",
job_name
));
}
label_map(&mut job["spec"]["template"]["metadata"]["labels"]);
if job["spec"]["activeDeadlineSeconds"].is_null() {
job["spec"]["activeDeadlineSeconds"] = json!(timeout_seconds);
@@ -443,15 +520,14 @@ fn prepare_docs(
if job["spec"]["ttlSecondsAfterFinished"].is_null() {
job["spec"]["ttlSecondsAfterFinished"] = json!(86400);
}
if job["spec"]["backoffLimit"].is_null() {
// Silent retries would splice two attempts into one run log.
job["spec"]["backoffLimit"] = json!(0);
}
// A replacement pod would not share the pod-local snapshot staged by kubectl cp.
job["spec"]["backoffLimit"] = json!(0);
let containers = job["spec"]["template"]["spec"]["containers"]
.as_array_mut()
.ok_or_else(|| anyhow!("the Job has no containers"))?;
let mut references_script = false;
let mut script_container = None;
let mut script_container_count = 0usize;
for c in containers.iter_mut() {
for field in ["command", "args"] {
if let Some(items) = c[field].as_array() {
@@ -459,7 +535,10 @@ fn prepare_docs(
.iter()
.any(|a| a.as_str().is_some_and(|s| s.contains("ORX_SCRIPT")))
{
references_script = true;
script_container_count += 1;
if script_container.is_none() {
script_container = c["name"].as_str().map(str::to_string);
}
}
}
}
@@ -492,14 +571,23 @@ fn prepare_docs(
}
c["envFrom"] = json!(env_from);
}
if !references_script {
let Some(script_container) = script_container else {
return Err(anyhow!(
"no container in Job '{}' runs the experiment: reference the injected script, \
e.g. command: [\"bash\", \"-c\", \"$ORX_SCRIPT\"] — it clones the branch tip \
e.g. command: [\"bash\", \"-c\", \"$ORX_SCRIPT\"] — it extracts the source snapshot \
and runs the experiment's fixed run command",
job_name
));
};
if script_container_count != 1 {
return Err(anyhow!(
"Job '{}' must have exactly one container that references ORX_SCRIPT; found {}",
job_name,
script_container_count
));
}
job["spec"]["template"]["metadata"]["annotations"]["kubectl.kubernetes.io/default-container"] =
json!(script_container);
Ok((docs, job_name))
}
@@ -781,13 +869,13 @@ mod tests {
}
#[test]
fn author_settings_win_over_defaults() {
fn author_deadline_wins_but_retries_are_disabled() {
let mut j = job("train");
j["spec"]["activeDeadlineSeconds"] = json!(60);
j["spec"]["backoffLimit"] = json!(2);
let (docs, _) = prepare(j).unwrap();
assert_eq!(docs[0]["spec"]["activeDeadlineSeconds"], 60);
assert_eq!(docs[0]["spec"]["backoffLimit"], 2);
assert_eq!(docs[0]["spec"]["backoffLimit"], 0);
}
#[test]
src/jobs/localbox.rs
+22 -4
View File
@@ -141,10 +141,28 @@ pub fn inspect_job(dir: &Path) -> JobState {
},
// Dead pid: run.sh may have written exit_code and exited between the
// check above and the ps probe — re-read before calling it killed.
Ok(_) => exit_code_state(dir).unwrap_or(JobState {
stage: "ERROR".into(),
message: Some("process died without an exit code (killed?)".into()),
}),
Ok(_) => {
for _ in 0..3 {
if let Some(state) = exit_code_state(dir) {
return state;
}
std::thread::sleep(std::time::Duration::from_millis(10));
}
if std::fs::metadata(dir.join("pid"))
.and_then(|metadata| metadata.modified())
.and_then(|modified| modified.elapsed().map_err(std::io::Error::other))
.is_ok_and(|age| age < std::time::Duration::from_secs(1))
{
return JobState {
stage: "RUNNING".into(),
message: None,
};
}
JobState {
stage: "ERROR".into(),
message: Some("process died without an exit code (killed?)".into()),
}
}
// pid not written yet — just starting.
Err(_) => JobState {
stage: "RUNNING".into(),
src/jobs/mod.rs
+11
View File
@@ -80,6 +80,14 @@ pub struct BackendDescriptor {
/// the supervisor, long after the `--timeout` flag is gone.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub timeout_secs: Option<u64>,
/// Immutable local source archive used for this run. These fields make a
/// delayed or restarted supervisor independent of the working tree.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub source_digest: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub source_path: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub source_size: Option<u64>,
}
impl BackendDescriptor {
@@ -249,6 +257,9 @@ mod tests {
ssh_port: None,
ssh_user: None,
timeout_secs: Some(14_400),
source_digest: None,
source_path: None,
source_size: None,
}
}
src/jobs/modal.rs
+11 -2
View File
@@ -66,6 +66,13 @@ def submit():
if tags:
kwargs["tags"] = tags
sb = modal.Sandbox.create("bash", "-c", spec["script"], **kwargs)
if spec.get("sourceArchive"):
try:
sb.filesystem.copy_from_local(spec["sourceArchive"], "/tmp/orx-source.tar")
sb.exec("touch", "/tmp/orx-source.tar.ready").wait()
except Exception:
sb.terminate()
raise
print(json.dumps({"sandboxId": sb.object_id}))
def status(sid):
@@ -389,7 +396,7 @@ pub fn default_image(gpu: bool) -> String {
}
pub struct ModalJobSpec {
/// `bash -c` payload (the shared clone-and-run script).
/// `bash -c` payload (the shared snapshot-and-run script).
pub script: String,
pub image: String,
pub gpu: Option<String>,
@@ -402,6 +409,7 @@ pub struct ModalJobSpec {
pub app: String,
/// Sandbox tags (or_run / or_experiment / or_project) for observability.
pub tags: HashMap<String, String>,
pub source_archive: Option<PathBuf>,
}
/// Submit the sandbox; returns its object id (the reattach handle).
@@ -422,6 +430,7 @@ pub async fn run_job(spec: &ModalJobSpec) -> Result<String> {
"env": env,
"timeoutSeconds": spec.timeout_seconds,
"tags": spec.tags,
"sourceArchive": spec.source_archive,
});
let out = launcher_capture(&["submit"], Some(&body.to_string())).await?;
let v: Value = serde_json::from_slice(&out).map_err(|e| {
@@ -523,7 +532,7 @@ pub struct ModalStatus {
pub error: Option<String>,
}
/// Fail fast before doing any launch work (git push, api registration) if
/// Fail fast before doing any source staging or provider submission if
/// Modal isn't usable — provisioning the orx-managed env on first use. Returns
/// a friendly, actionable error otherwise.
pub async fn preflight() -> Result<()> {
src/jobs/ray.rs
+38 -1
View File
@@ -246,6 +246,7 @@ pub struct JobSubmission {
pub resources: RayResources,
pub env: HashMap<String, String>,
pub metadata: HashMap<String, String>,
pub working_dir: Option<String>,
}
#[derive(Debug, Clone)]
@@ -278,10 +279,14 @@ fn map_ray_status(raw: &str) -> String {
/// so a success needs nothing from the response body.
pub async fn run_job(address: &str, spec: &JobSubmission) -> Result<()> {
let env = super::default_unbuffered(&spec.env);
let mut runtime_env = json!({ "env_vars": env });
if let Some(working_dir) = &spec.working_dir {
runtime_env["working_dir"] = json!(working_dir);
}
let mut body = json!({
"entrypoint": spec.entrypoint,
"submission_id": spec.submission_id,
"runtime_env": { "env_vars": env },
"runtime_env": runtime_env,
"metadata": spec.metadata,
"entrypoint_num_cpus": spec.resources.cpus,
"entrypoint_num_gpus": spec.resources.gpus,
@@ -299,6 +304,38 @@ pub async fn run_job(address: &str, spec: &JobSubmission) -> Result<()> {
Ok(())
}
/// Upload a Ray `working_dir` package through the Jobs server. Ray's own SDK
/// uses this exact content-addressed package endpoint before submitting the
/// job; using the snapshot digest makes the existence check restart-safe.
pub async fn stage_working_dir(
address: &str,
digest: &str,
zip: &std::path::Path,
) -> Result<String> {
let name = format!("_ray_pkg_{digest}.zip");
let url = format!("{address}/api/packages/gcs/{name}");
let existing = http()
.get(&url)
.send()
.await
.map_err(|e| anyhow!("Could not query Ray source package at {address}: {e}"))?;
if existing.status() != reqwest::StatusCode::OK {
if existing.status() != reqwest::StatusCode::NOT_FOUND {
check(existing, "source package lookup").await?;
}
let file = tokio::fs::File::open(zip).await?;
let body = reqwest::Body::wrap_stream(tokio_util::io::ReaderStream::new(file));
let uploaded = http()
.put(&url)
.body(body)
.send()
.await
.map_err(|e| anyhow!("Could not upload Ray source package at {address}: {e}"))?;
check(uploaded, "source package upload").await?;
}
Ok(format!("gcs://{name}"))
}
pub async fn inspect_job(address: &str, submission_id: &str) -> Result<JobInfo> {
let res = http()
.get(format!("{address}/api/jobs/{submission_id}"))
src/jobs/slurm.rs
+8 -7
View File
@@ -359,12 +359,12 @@ pub async fn cancel_job(host: &str, job_id: &str) -> Result<()> {
// --- preflight ----------------------------------------------------------------
/// Per-host readiness for the Settings UI: reachable, Slurm CLI + git
/// Per-host readiness for the Settings UI: reachable, Slurm CLI + snapshot tools
/// present, and which partitions exist.
pub struct SlurmPreflight {
pub reachable: bool,
pub slurm_found: bool,
pub git_found: bool,
pub tools_found: bool,
/// From `sinfo` (default partition's trailing `*` stripped).
pub partitions: Vec<String>,
pub error: Option<String>,
@@ -373,17 +373,18 @@ pub struct SlurmPreflight {
pub async fn preflight(host: &str) -> SlurmPreflight {
let cmd = "if command -v sbatch >/dev/null 2>&1 && command -v squeue >/dev/null 2>&1 \
&& command -v scancel >/dev/null 2>&1; then echo SLURM_OK; fi; \
if command -v git >/dev/null 2>&1; then echo GIT_OK; fi; \
if command -v bash >/dev/null 2>&1 && command -v tar >/dev/null 2>&1; \
then echo TOOLS_OK; fi; \
sinfo -h -o %P 2>/dev/null || true";
match ssh_run(&SshTarget::alias(host), cmd, None).await {
Ok(out) => {
let mut slurm_found = false;
let mut git_found = false;
let mut tools_found = false;
let mut partitions = Vec::new();
for line in out.lines().map(str::trim).filter(|l| !l.is_empty()) {
match line {
"SLURM_OK" => slurm_found = true,
"GIT_OK" => git_found = true,
"TOOLS_OK" => tools_found = true,
p => {
let p = p.trim_end_matches('*').to_string();
if !p.is_empty() && !partitions.contains(&p) {
@@ -395,7 +396,7 @@ pub async fn preflight(host: &str) -> SlurmPreflight {
SlurmPreflight {
reachable: true,
slurm_found,
git_found,
tools_found,
partitions,
error: None,
}
@@ -403,7 +404,7 @@ pub async fn preflight(host: &str) -> SlurmPreflight {
Err(e) => SlurmPreflight {
reachable: false,
slurm_found: false,
git_found: false,
tools_found: false,
partitions: Vec::new(),
error: Some(e.to_string()),
},
src/jobs/ssh.rs
+148 -16
View File
@@ -7,7 +7,7 @@
//! polls reuse one TCP session instead of a handshake apiece.
//!
//! The handle is a remote run directory `~/.orx/runs/<run_id>/` holding:
//! run.sh the launcher (exported env + clone-and-run payload)
//! run.sh the launcher (exported env + snapshot-and-run payload)
//! log merged stdout/stderr
//! pid the detached process-group leader
//! exit_code written when the payload finishes
@@ -22,11 +22,49 @@ use tokio::process::Command;
use crate::error::{anyhow, Result};
/// Where ssh keeps its ControlMaster sockets. Created on first use.
/// Keep sockets out of config paths, which can exceed macOS's 104-byte limit.
#[cfg(unix)]
fn control_dir() -> PathBuf {
use std::hash::{Hash as _, Hasher as _};
let uid = unsafe { libc::geteuid() };
let mut namespace = std::collections::hash_map::DefaultHasher::new();
crate::config::config_dir().hash(&mut namespace);
PathBuf::from("/tmp").join(format!("orx-ssh-{uid}-{:08x}", namespace.finish() as u32))
}
#[cfg(not(unix))]
fn control_dir() -> PathBuf {
crate::config::config_dir().join("ssh-cm")
}
fn prepare_control_dir() -> Result<()> {
let dir = control_dir();
std::fs::create_dir_all(&dir).map_err(|e| {
anyhow!(
"Could not create SSH control directory {}: {e}",
dir.display()
)
})?;
#[cfg(unix)]
{
use std::os::unix::fs::{MetadataExt as _, PermissionsExt as _};
let metadata = std::fs::symlink_metadata(&dir)?;
let uid = unsafe { libc::geteuid() };
if !metadata.file_type().is_dir() || metadata.uid() != uid {
return Err(anyhow!(
"SSH control path {} is not an owner-controlled directory.",
dir.display()
));
}
let mut permissions = metadata.permissions();
permissions.set_mode(0o700);
std::fs::set_permissions(&dir, permissions)?;
}
Ok(())
}
/// An ssh endpoint. The classic ssh backend connects by `~/.ssh/config` alias
/// (`SshTarget::alias`); backends that learn an endpoint at runtime (an
/// OpenResearch box on a provider-assigned host:port) pass an explicit
@@ -96,12 +134,8 @@ impl SshTarget {
/// Shared ssh options: BatchMode (never hang on a prompt) + connection
/// multiplexing so repeated polls are cheap.
fn ssh_opts(target: &SshTarget) -> Vec<String> {
// Not ssh's %C token: the expanded path must fit in sun_path (104 bytes
// on macOS) and `<config dir>/ssh-cm/<40-hex>.<12-char tmp suffix>`
// overflows it for ordinary home dirs — ssh then fails outright rather
// than skip multiplexing. A 16-hex hash keeps it short. It folds in the
// extra opts (where %C folds in user/host/port) so `user@host -p 2222`
// and `user@host -p 2223` never share a control socket.
// A 16-hex hash leaves room for ssh's temporary bind suffix. It folds in
// the extra opts so different ports never share a control socket.
use std::hash::{Hash, Hasher};
let mut h = std::collections::hash_map::DefaultHasher::new();
target.dest.hash(&mut h);
@@ -132,7 +166,15 @@ pub(crate) async fn ssh_run(
remote_cmd: &str,
stdin: Option<&str>,
) -> Result<String> {
let _ = std::fs::create_dir_all(control_dir());
ssh_run_bytes(target, remote_cmd, stdin.map(str::as_bytes)).await
}
async fn ssh_run_bytes(
target: &SshTarget,
remote_cmd: &str,
stdin: Option<&[u8]>,
) -> Result<String> {
prepare_control_dir()?;
let mut cmd = Command::new("ssh");
cmd.args(ssh_opts(target))
.arg("--")
@@ -155,7 +197,7 @@ pub(crate) async fn ssh_run(
if let Some(input) = stdin {
use tokio::io::AsyncWriteExt as _;
if let Some(mut pipe) = child.stdin.take() {
let _ = pipe.write_all(input.as_bytes()).await;
let _ = pipe.write_all(input).await;
drop(pipe); // EOF
}
}
@@ -179,6 +221,78 @@ pub(crate) async fn ssh_run(
Ok(String::from_utf8_lossy(&out.stdout).into_owned())
}
async fn ssh_run_file(
target: &SshTarget,
remote_cmd: &str,
source: &std::path::Path,
) -> Result<String> {
prepare_control_dir()?;
let mut child = Command::new("ssh")
.args(ssh_opts(target))
.arg("--")
.arg(&target.dest)
.arg(remote_cmd)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.map_err(|e| anyhow!("Could not run ssh: {e}"))?;
let mut file = tokio::fs::File::open(source).await?;
if let Some(mut pipe) = child.stdin.take() {
tokio::io::copy(&mut file, &mut pipe).await?;
drop(pipe);
}
let out = child
.wait_with_output()
.await
.map_err(|e| anyhow!("ssh wait failed: {e}"))?;
if !out.status.success() {
return Err(anyhow!(
"ssh {} failed: {}",
target.dest,
String::from_utf8_lossy(&out.stderr).trim()
));
}
Ok(String::from_utf8_lossy(&out.stdout).into_owned())
}
/// Upload a content-addressed tar once, then materialize it into this run's
/// private `repo/` directory. Both the cache write and extraction are safe to
/// repeat after a client or supervisor restart.
pub async fn stage_source(
target: &SshTarget,
run_id: &str,
archive: &std::path::Path,
digest: &str,
) -> Result<String> {
let dir = format!(".orx/runs/{run_id}");
let cache = format!(".orx/source/{digest}.tar");
let present = ssh_run(
target,
&format!("test -f \"$HOME/{cache}\" && echo present || true"),
None,
)
.await?;
if present.trim() != "present" {
let upload = format!(
"umask 077; mkdir -p \"$HOME/.orx/source\"; \
tmp=\"$HOME/{cache}.tmp.$$\"; cat > \"$tmp\" && mv \"$tmp\" \"$HOME/{cache}\""
);
ssh_run_file(target, &upload, archive).await?;
}
ssh_run(
target,
&format!(
"umask 077; mkdir -p \"$HOME/.orx/runs\" \"$HOME/{dir}/repo\"; \
chmod 700 \"$HOME/.orx/runs\" \"$HOME/{dir}\" \"$HOME/{dir}/repo\"; \
tar -xf \"$HOME/{cache}\" -C \"$HOME/{dir}/repo\""
),
None,
)
.await?;
Ok(dir)
}
/// Single-quote a value for safe embedding in the remote bash script.
pub(crate) fn sh_quote(s: &str) -> String {
format!("'{}'", s.replace('\'', "'\\''"))
@@ -189,7 +303,7 @@ pub struct SshJobSpec {
pub target: SshTarget,
/// Names the remote run dir `~/.orx/runs/<run_id>`.
pub run_id: String,
/// The shared clone-and-run payload (`bash` script body).
/// The shared snapshot-and-run payload (`bash` script body).
pub script: String,
/// Exported inside run.sh on the remote (tokens, synced env).
pub env: HashMap<String, String>,
@@ -320,29 +434,30 @@ pub async fn cancel_job(target: &SshTarget, dir: &str) -> Result<()> {
Ok(())
}
/// Per-host readiness for the Settings UI: can we reach it, and is `git` there?
/// Per-host readiness for the Settings UI: can we reach it and execute snapshots?
pub struct SshPreflight {
pub reachable: bool,
pub git_found: bool,
pub tools_found: bool,
pub error: Option<String>,
}
pub async fn preflight(target: &SshTarget) -> SshPreflight {
match ssh_run(
target,
"command -v git >/dev/null 2>&1 && echo GIT_OK || echo NO_GIT",
"command -v bash >/dev/null 2>&1 && command -v tar >/dev/null 2>&1 \
&& echo TOOLS_OK || echo NO_TOOLS",
None,
)
.await
{
Ok(out) => SshPreflight {
reachable: true,
git_found: out.contains("GIT_OK"),
tools_found: out.contains("TOOLS_OK"),
error: None,
},
Err(e) => SshPreflight {
reachable: false,
git_found: false,
tools_found: false,
error: Some(e.to_string()),
},
}
@@ -410,4 +525,21 @@ mod tests {
assert_ne!(control_path(&mk("22022")), control_path(&mk("22023")));
assert_eq!(control_path(&mk("22022")), control_path(&mk("22022")));
}
#[cfg(unix)]
#[test]
fn control_path_fits_macos_unix_socket_limit() {
let option = ssh_opts(&SshTarget::host_port(
"root@ssh3.vast.ai".into(),
22,
HostKeyPolicy::Ephemeral,
))
.into_iter()
.find(|o| o.starts_with("ControlPath="))
.unwrap();
let path = option.strip_prefix("ControlPath=").unwrap();
assert!(path.starts_with("/tmp/orx-ssh-"));
assert!(path.len() + 17 < 104, "{path}");
}
}
src/local/agent_skills.rs
+25 -39
View File
@@ -55,16 +55,13 @@ pub enum SkillSet {
// public skill name) ----------------------------------------------------------
const COMPUTE_LOCAL: &str = include_str!("../../agent-skills/orx-compute/SKILL.local.md");
const COMPUTE_LOCAL_ONLY: &str = include_str!("../../agent-skills/orx-compute/SKILL.local-only.md");
const COMPUTE_CLOUD: &str = include_str!("../../agent-skills/orx-compute/SKILL.md");
const COMPUTE_K8S: &str = include_str!("../../agent-skills/orx-compute-k8s/SKILL.md");
const EXPERIMENT_TREE_LOCAL: &str =
include_str!("../../agent-skills/orx-experiment-tree/SKILL.local.md");
const EXPERIMENT_TREE_CLOUD: &str = include_str!("../../agent-skills/orx-experiment-tree/SKILL.md");
const GIT_EDITING: &str = include_str!("../../agent-skills/orx-git/SKILL.md");
const EXPERIMENT_TREE_LOCAL_ONLY: &str =
include_str!("../../agent-skills/orx-experiment-tree/SKILL.local-only.md");
const GIT_LOCAL_ONLY: &str = include_str!("../../agent-skills/orx-git/SKILL.local-only.md");
const GIT_LOCAL: &str = include_str!("../../agent-skills/orx-git/SKILL.local.md");
const GIT_CLOUD: &str = include_str!("../../agent-skills/orx-git/SKILL.md");
const LIT: &str = include_str!("../../agent-skills/orx-lit/SKILL.md");
const CREATE: &str = include_str!("../../agent-skills/orx-create/SKILL.md");
const REPORTS_LOCAL: &str = include_str!("../../agent-skills/orx-reports/SKILL.local.md");
@@ -96,7 +93,7 @@ const S_COMPUTE_CLOUD: AgentSkill = AgentSkill {
};
const S_COMPUTE_K8S: AgentSkill = AgentSkill {
name: "orx-compute-k8s",
description: "Run an experiment on your own Kubernetes cluster (`orx exp run --backend k8s`): the committed-manifest contract orx enforces at submit. Use when the user names k8s, kubernetes, or a cluster, before writing or editing `.orx/k8s.yaml`, for multi-node or Indexed Jobs, or when a k8s submit is rejected.",
description: "Run an experiment on your own Kubernetes cluster (`orx exp run --backend k8s`): the single-pod committed-manifest contract orx enforces at submit. Use when the user names k8s, kubernetes, or a cluster, before writing or editing `.orx/k8s.yaml`, or when a k8s submit is rejected.",
content: COMPUTE_K8S,
};
const S_EXPERIMENT_TREE_LOCAL: AgentSkill = AgentSkill {
@@ -109,10 +106,15 @@ const S_EXPERIMENT_TREE_CLOUD: AgentSkill = AgentSkill {
description: D_EXPERIMENT_TREE,
content: EXPERIMENT_TREE_CLOUD,
};
const S_GIT: AgentSkill = AgentSkill {
const S_GIT_LOCAL: AgentSkill = AgentSkill {
name: "orx-git",
description: "Read, edit, and diff a node's code with plain git: sync, commit, and push before running. Use whenever you touch experiment code — before editing any branch, when a checkout or push fails, when comparing two nodes' code, or when a run seems to have picked up stale code.",
content: GIT_EDITING,
description: "Read, edit, commit, and diff experiment code with local Git. Use whenever you touch an experiment branch, compare nodes, prepare a run, or diagnose stale code.",
content: GIT_LOCAL,
};
const S_GIT_CLOUD: AgentSkill = AgentSkill {
name: "orx-git",
description: "Read, edit, commit, and diff experiment code with Git. Use whenever you touch a branch, compare nodes, prepare a run, diagnose stale code, or publish changes.",
content: GIT_CLOUD,
};
const S_LIT: AgentSkill = AgentSkill {
name: "orx-lit",
@@ -152,7 +154,7 @@ pub fn skills(set: SkillSet) -> Vec<&'static AgentSkill> {
match set {
SkillSet::Local => vec![
&S_EXPERIMENT_TREE_LOCAL,
&S_GIT,
&S_GIT_LOCAL,
&S_COMPUTE_LOCAL,
&S_COMPUTE_K8S,
&S_EVIDENCE_LOCAL,
@@ -162,7 +164,7 @@ pub fn skills(set: SkillSet) -> Vec<&'static AgentSkill> {
SkillSet::Full => vec![
&S_CREATE,
&S_EXPERIMENT_TREE_CLOUD,
&S_GIT,
&S_GIT_CLOUD,
&S_COMPUTE_CLOUD,
&S_COMPUTE_K8S,
&S_EVIDENCE_CLOUD,
@@ -183,31 +185,16 @@ pub fn find(name: &str, set: SkillSet) -> Option<&'static AgentSkill> {
.find(|s| s.name == want || s.name.strip_prefix("orx-") == Some(want))
}
pub fn available_in_session(skill: &AgentSkill, github_enabled: bool) -> bool {
github_enabled || skill.name != "orx-compute-k8s"
pub fn available_in_session(_skill: &AgentSkill, _github_enabled: bool) -> bool {
true
}
pub fn session_content(skill: &AgentSkill, github_enabled: bool) -> &'static str {
if github_enabled {
return skill.content;
}
match skill.name {
"orx-git" => GIT_LOCAL_ONLY,
"orx-compute" => COMPUTE_LOCAL_ONLY,
"orx-experiment-tree" => EXPERIMENT_TREE_LOCAL_ONLY,
_ => skill.content,
}
pub fn session_content(skill: &AgentSkill, _github_enabled: bool) -> &'static str {
skill.content
}
pub fn session_description(skill: &AgentSkill, github_enabled: bool) -> &'static str {
if github_enabled {
return skill.description;
}
match skill.name {
"orx-git" => "Read, edit, commit, and diff experiment code with local Git only.",
"orx-compute" => "Launch committed experiments on this machine and inspect their logs.",
_ => skill.description,
}
pub fn session_description(skill: &AgentSkill, _github_enabled: bool) -> &'static str {
skill.description
}
/// Write the [`SkillSet::Local`] modules as `<worktree>/<skills_dir_rel>/<name>/SKILL.md`,
@@ -438,7 +425,7 @@ mod tests {
}
#[test]
fn local_only_session_skills_override_push_instructions() {
fn github_disabled_sessions_keep_snapshot_compute_available() {
let tmp = std::env::temp_dir().join(format!(
"orx-local-only-skills-test-{}",
uuid::Uuid::new_v4()
@@ -448,13 +435,12 @@ mod tests {
assert!(tmp.join(rel).join("orx-compute-k8s").exists());
ensure_session_skills(&tmp, rel, false).unwrap();
let git = std::fs::read_to_string(tmp.join(rel).join("orx-git/SKILL.md")).unwrap();
assert!(git.contains("This project is local-only"));
assert!(!git.contains("git push"));
assert!(!git.contains("origin/"));
assert!(git.contains("never part of compute transport"));
assert!(git.contains("do not push merely to launch compute"));
let compute = std::fs::read_to_string(tmp.join(rel).join("orx-compute/SKILL.md")).unwrap();
assert!(!compute.contains("branch tip"));
assert!(!compute.contains("git push"));
assert!(!tmp.join(rel).join("orx-compute-k8s").exists());
assert!(compute.contains("immutable source snapshot"));
assert!(compute.contains("Hugging Face Jobs"));
assert!(tmp.join(rel).join("orx-compute-k8s").exists());
let _ = std::fs::remove_dir_all(tmp);
}
src/local/codex.rs
-6
View File
@@ -496,12 +496,6 @@ async fn spawn_client(session_id: &str) -> Result<Arc<CodexClient>> {
if let Some(dir) = ensure_orx_data_dir() {
cmd.env("ORX_DATA_DIR", &dir);
}
// The sandbox blocks the keyring `gh` keeps its token in; resolve it out
// here and pass it down. Resolved once per child, not per turn.
if let Some(token) = crate::local::git::resolve_github_token() {
cmd.env("GH_TOKEN", &token);
cmd.env("GITHUB_TOKEN", token);
}
// Own process group: a terminal SIGINT reaches orx up alone, which then
// tears the child down deliberately (kill_on_drop / shutdown()).
#[cfg(unix)]
src/local/experiments.rs
+3 -6
View File
@@ -64,8 +64,8 @@ pub fn legacy_root_warning(project: &LocalProject, experiment: &LocalExperiment)
})
}
/// Create a local experiment. Every node gets its own `orx/<slug>` branch,
/// pushed to origin: a child forks off its parent's tip, a baseline/root off
/// Create a local experiment. Every node gets its own `orx/<slug>` branch:
/// a child forks off its parent's tip, a baseline/root off
/// the project's base branch. The base branch itself is never an experiment
/// node — it stays mutable (README, notebooks, publication surface) while
/// `orx/*` branches hold the experiment nodes' recorded code. Matches the
@@ -98,10 +98,7 @@ pub fn create_experiment(
.map(|p| p.branch_name.as_str())
.unwrap_or(&project.baseline_branch);
let branch_name = format!("orx/{slug}");
let publication = project
.github_enabled()
.then_some((project.github_owner.as_str(), project.github_repo.as_str()));
git::create_experiment_branch(repo, fork_point, &branch_name, publication)?;
git::create_experiment_branch(repo, fork_point, &branch_name)?;
// Inherit: explicit > parent's command > project default > "".
let run_command = run_command
src/local/git.rs
+252 -305
View File
@@ -3,8 +3,13 @@
//! `~/.cache/openresearch/repos/<owner>/<repo>`, the same convention SKILL.md
//! documents for manual diffing.
use std::io::Read;
use std::path::{Path, PathBuf};
use std::process::Command;
use std::process::{Child, Command, Stdio};
use std::time::{Duration, Instant};
#[cfg(unix)]
use std::os::unix::process::CommandExt;
use crate::error::{anyhow, Result};
@@ -170,7 +175,15 @@ pub fn initialize_repository(path: &Path) -> Result<()> {
Ok(())
}
pub fn clone_public(url: &str, path: &Path) -> Result<()> {
fn public_clone_history_args(shallow: bool) -> &'static [&'static str] {
if shallow {
&["--depth=1", "--single-branch"]
} else {
&[]
}
}
pub fn clone_public(url: &str, path: &Path, shallow: bool) -> Result<()> {
let url = public_clone_url(url)?;
let empty_config = std::env::temp_dir().join(format!(
"orx-public-clone-{}.gitconfig",
@@ -180,7 +193,8 @@ pub fn clone_public(url: &str, path: &Path) -> Result<()> {
.write(true)
.create_new(true)
.open(&empty_config)?;
let output = Command::new("git")
let mut command = Command::new("git");
command
.current_dir(std::env::temp_dir())
.env("GIT_TERMINAL_PROMPT", "0")
.env("GIT_ASKPASS", "")
@@ -197,16 +211,11 @@ pub fn clone_public(url: &str, path: &Path) -> Result<()> {
.env_remove("GIT_OBJECT_DIRECTORY")
.env_remove("GIT_ALTERNATE_OBJECT_DIRECTORIES")
.env("GIT_SSH_COMMAND", "false")
.args([
"-c",
"credential.helper=",
"-c",
"core.askPass=",
"clone",
&url,
&path.to_string_lossy(),
])
.output();
.args(["-c", "credential.helper=", "-c", "core.askPass=", "clone"])
.args(public_clone_history_args(shallow))
.arg(&url)
.arg(path);
let output = command.output();
let _ = std::fs::remove_file(&empty_config);
let out = output.map_err(|error| anyhow!("Could not run git clone: {error}"))?;
if !out.status.success() {
@@ -276,7 +285,14 @@ pub fn validate_project_repository(path: &Path) -> Result<()> {
}
pub fn local_head_sha(path: &Path, branch: &str) -> Result<String> {
git(Some(path), &["rev-parse", &format!("{branch}^{{commit}}")])
git(
Some(path),
&[
"rev-parse",
"--verify",
&format!("refs/heads/{branch}^{{commit}}"),
],
)
}
pub fn remotes(path: &Path) -> Result<Vec<(String, String)>> {
@@ -315,11 +331,13 @@ fn sanitize_remote_url(url: &str) -> String {
}
pub fn github_publication(path: &Path) -> Option<(String, String)> {
[GITHUB_REMOTE, "origin"].into_iter().find_map(|remote| {
let url = git(Some(path), &["remote", "get-url", remote]).ok()?;
let (owner, repo) = parse_github_url(&url)?;
remote_matches_publication(path, remote, &owner, &repo).then_some((owner, repo))
})
[GITHUB_REMOTE, "origin", "upstream"]
.into_iter()
.find_map(|remote| {
let url = git(Some(path), &["remote", "get-url", remote]).ok()?;
let (owner, repo) = parse_github_url(&url)?;
remote_matches_publication(path, remote, &owner, &repo).then_some((owner, repo))
})
}
fn parse_github_url(url: &str) -> Option<(String, String)> {
@@ -349,6 +367,10 @@ fn parse_github_url(url: &str) -> Option<(String, String)> {
.then(|| (owner.to_string(), repo.to_string()))
}
pub fn github_repository(url: &str) -> Option<(String, String)> {
parse_github_url(url)
}
fn github_repository_matches(url: &str, owner: &str, repo: &str) -> bool {
parse_github_url(url).is_some_and(|(remote_owner, remote_repo)| {
remote_owner.eq_ignore_ascii_case(owner) && remote_repo.eq_ignore_ascii_case(repo)
@@ -413,24 +435,22 @@ pub fn identity(
(name, email, name_source, email_source)
}
/// `GITHUB_TOKEN` env, else the synced env file (UI-pasted token), else
/// `gh auth token`, else None (public-repo fallback).
pub fn resolve_github_token() -> Option<String> {
if let Ok(t) = std::env::var("GITHUB_TOKEN") {
let t = t.trim().to_string();
if !t.is_empty() {
return Some(t);
if let Ok(token) = std::env::var("GITHUB_TOKEN") {
let token = token.trim().to_string();
if !token.is_empty() {
return Some(token);
}
}
if let Some(t) = crate::config::synced_env_var("GITHUB_TOKEN") {
return Some(t);
if let Some(token) = crate::config::synced_env_var("GITHUB_TOKEN") {
return Some(token);
}
let out = Command::new("gh").args(["auth", "token"]).output().ok()?;
if !out.status.success() {
let output = Command::new("gh").args(["auth", "token"]).output().ok()?;
if !output.status.success() {
return None;
}
let t = String::from_utf8_lossy(&out.stdout).trim().to_string();
(!t.is_empty()).then_some(t)
let token = String::from_utf8_lossy(&output.stdout).trim().to_string();
(!token.is_empty()).then_some(token)
}
/// Fail early on a typo'd baseline branch — otherwise it only surfaces much
@@ -738,8 +758,19 @@ fn seed_copy_in(
Ok(())
}
pub fn local_branches(repo_path: &Path) -> Result<Vec<String>> {
Ok(git(
Some(repo_path),
&["for-each-ref", "--format=%(refname:short)", "refs/heads"],
)?
.lines()
.filter(|branch| !branch.is_empty())
.map(str::to_string)
.collect())
}
fn publication_remote(repo_path: &Path, owner: &str, repo: &str) -> Result<String> {
for remote in [GITHUB_REMOTE, "origin"] {
for remote in [GITHUB_REMOTE, "origin", "upstream"] {
if remote_matches_publication(repo_path, remote, owner, repo) {
return Ok(remote.to_string());
}
@@ -749,10 +780,82 @@ fn publication_remote(repo_path: &Path, owner: &str, repo: &str) -> Result<Strin
))
}
pub fn is_shallow_repository(repo_path: &Path) -> Result<bool> {
Ok(git(Some(repo_path), &["rev-parse", "--is-shallow-repository"])? == "true")
}
pub fn reroot_shallow_repository(
repo_path: &Path,
baseline_branch: &str,
source: Option<&(String, String)>,
) -> Result<()> {
if !is_shallow_repository(repo_path)? {
return Ok(());
}
if !is_clean(repo_path)? {
return Err(anyhow!(
"Cannot prepare a shallow paper import for publication because the working tree has changes."
));
}
let temporary = format!("orx-import-{}", uuid::Uuid::new_v4().simple());
git(Some(repo_path), &["checkout", "--orphan", &temporary])?;
git(Some(repo_path), &["add", "-A"])?;
let source_name = source
.map(|(owner, repo)| format!("{owner}/{repo}"))
.unwrap_or_else(|| "paper repository".to_string());
git(
Some(repo_path),
&[
"-c",
"user.name=OpenResearch",
"-c",
"user.email=local@openresearch.sh",
"commit",
"--allow-empty",
"-m",
&format!("Import snapshot from {source_name}"),
],
)?;
git(Some(repo_path), &["branch", "-M", baseline_branch])?;
Ok(())
}
pub fn prepare_shallow_repository_for_publication(repo_path: &Path) -> Result<bool> {
if !is_shallow_repository(repo_path)? {
return Ok(false);
}
if local_branches(repo_path)?
.iter()
.any(|branch| branch.starts_with("orx/"))
{
let remote = ["upstream", "origin"]
.into_iter()
.find(|remote| git(Some(repo_path), &["remote", "get-url", remote]).is_ok())
.ok_or_else(|| anyhow!("The shallow project has no source remote to deepen."))?;
git(Some(repo_path), &["fetch", "--unshallow", remote])?;
return Ok(false);
}
Ok(true)
}
const GITHUB_CREDENTIAL_HELPER: &str =
"!f() { host=; while IFS='=' read key value; do [ \"$key\" = host ] && host=$value; done; [ \"$host\" = github.com ] || exit 0; echo username=x-access-token; echo \"password=$ORX_GITHUB_TOKEN\"; }; f";
fn authenticated_git(repo_path: &Path, args: &[&str]) -> Result<String> {
fn redact_remote_urls(text: &str) -> String {
text.split_whitespace()
.map(|word| {
let bare = word.trim_matches(|ch: char| "'\"`()[]{}<>,".contains(ch));
if bare.to_ascii_lowercase().contains("://") {
word.replace(bare, &sanitize_remote_url(bare))
} else {
word.to_string()
}
})
.collect::<Vec<_>>()
.join(" ")
}
fn authenticated_git(repo_path: &Path, args: &[&str], timeout: Duration) -> Result<String> {
let mut command = Command::new("git");
command
.current_dir(repo_path)
@@ -772,93 +875,82 @@ fn authenticated_git(repo_path: &Path, args: &[&str]) -> Result<String> {
.env("GIT_CONFIG_VALUE_2", "/dev/null")
.env("ORX_GITHUB_TOKEN", token);
}
let output = command
#[cfg(unix)]
command.process_group(0);
let mut child = command
.args(args)
.output()
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.map_err(|error| anyhow!("Could not run git: {error}"))?;
if !output.status.success() {
let mut error = String::from_utf8_lossy(&output.stderr).trim().to_string();
if let Some(token) = token {
error = error.replace(&token, "[redacted]");
let mut stdout = child
.stdout
.take()
.ok_or_else(|| anyhow!("Could not capture git stdout"))?;
let mut stderr = child
.stderr
.take()
.ok_or_else(|| anyhow!("Could not capture git stderr"))?;
let stdout_reader = std::thread::spawn(move || {
let mut output = Vec::new();
stdout.read_to_end(&mut output).map(|_| output)
});
let stderr_reader = std::thread::spawn(move || {
let mut output = Vec::new();
stderr.read_to_end(&mut output).map(|_| output)
});
let deadline = Instant::now() + timeout;
let (status, timed_out) = loop {
if let Some(status) = child.try_wait()? {
break (status, false);
}
error = redact_remote_urls(&error);
if Instant::now() >= deadline {
terminate_git_process_tree(&mut child);
break (child.wait()?, true);
}
std::thread::sleep(Duration::from_millis(50));
};
let stdout = stdout_reader
.join()
.map_err(|_| anyhow!("Could not collect git stdout"))??;
let stderr = stderr_reader
.join()
.map_err(|_| anyhow!("Could not collect git stderr"))??;
if timed_out {
return Err(anyhow!(
"git {} failed: {error}",
args.first().copied().unwrap_or("command")
"git {} timed out after {} seconds",
args.first().copied().unwrap_or("command"),
timeout.as_secs()
));
}
Ok(String::from_utf8_lossy(&output.stdout).trim().to_string())
}
fn isolated_github_refs(url: &str, branch: &str) -> Result<String> {
let scratch =
std::env::temp_dir().join(format!("orx-github-preflight-{}", uuid::Uuid::new_v4()));
std::fs::create_dir(&scratch)?;
let empty_config = scratch.join("global.gitconfig");
std::fs::OpenOptions::new()
.write(true)
.create_new(true)
.open(&empty_config)?;
let token = resolve_github_token();
let mut command = Command::new("git");
command
.current_dir(&scratch)
.env("GIT_TERMINAL_PROMPT", "0")
.env("GIT_ASKPASS", "")
.env("SSH_ASKPASS", "")
.env("SSH_ASKPASS_REQUIRE", "never")
.env("GIT_CONFIG_NOSYSTEM", "1")
.env("GIT_CONFIG_GLOBAL", &empty_config)
.env_remove("GIT_CONFIG_PARAMETERS")
.env_remove("GIT_DIR")
.env_remove("GIT_WORK_TREE")
.env_remove("GIT_COMMON_DIR")
.env("GIT_SSH_COMMAND", "false");
if let Some(token) = &token {
command
.env("GIT_CONFIG_COUNT", "2")
.env("GIT_CONFIG_KEY_0", "credential.helper")
.env("GIT_CONFIG_VALUE_0", "")
.env("GIT_CONFIG_KEY_1", "credential.helper")
.env("GIT_CONFIG_VALUE_1", GITHUB_CREDENTIAL_HELPER)
.env("ORX_GITHUB_TOKEN", token);
} else {
command.env_remove("GIT_CONFIG_COUNT");
}
let output = command.args(["ls-remote", "--heads", url, branch]).output();
let _ = std::fs::remove_dir_all(&scratch);
let output = output.map_err(|error| anyhow!("Could not run GitHub preflight: {error}"))?;
if !output.status.success() {
let mut error = String::from_utf8_lossy(&output.stderr).trim().to_string();
if !status.success() {
let mut error = String::from_utf8_lossy(&stderr).trim().to_string();
if let Some(token) = token {
error = error.replace(&token, "[redacted]");
}
return Err(anyhow!(
"GitHub HTTPS preflight failed: {}",
"git {} failed: {}",
args.first().copied().unwrap_or("command"),
redact_remote_urls(&error)
));
}
Ok(String::from_utf8_lossy(&output.stdout).trim().to_string())
Ok(String::from_utf8_lossy(&stdout).trim().to_string())
}
fn redact_remote_urls(text: &str) -> String {
text.split_whitespace()
.map(|word| {
let bare = word.trim_matches(|ch: char| "'\"`()[]{}<>,".contains(ch));
if bare.to_ascii_lowercase().contains("://") {
word.replace(bare, &sanitize_remote_url(bare))
} else {
word.to_string()
}
})
.collect::<Vec<_>>()
.join(" ")
fn terminate_git_process_tree(child: &mut Child) {
#[cfg(unix)]
unsafe {
// Git owns this process group, including credential-bearing transport children.
libc::kill(-(child.id() as i32), libc::SIGKILL);
}
#[cfg(not(unix))]
let _ = child.kill();
}
fn push(repo_path: &Path, args: &[&str]) -> Result<()> {
let mut command = vec!["push"];
command.extend_from_slice(args);
authenticated_git(repo_path, &command)?;
authenticated_git(repo_path, &command, Duration::from_secs(600))?;
Ok(())
}
@@ -900,31 +992,48 @@ pub fn add_github_remote(repo_path: &Path, owner: &str, repo: &str) -> Result<()
pub fn push_all(repo_path: &Path, baseline_branch: &str, owner: &str, repo: &str) -> Result<()> {
let remote = publication_remote(repo_path, owner, repo)?;
push_all_to(repo_path, baseline_branch, &remote)
}
fn push_all_to(repo_path: &Path, baseline_branch: &str, remote: &str) -> Result<()> {
let mut branches = local_branches(repo_path)?;
if let Some(index) = branches.iter().position(|branch| branch == baseline_branch) {
let baseline = branches.remove(index);
branches.insert(0, baseline);
}
let mut branches = local_branches(repo_path)?
.into_iter()
.filter(|branch| branch == baseline_branch || branch.starts_with("orx/"))
.collect::<Vec<_>>();
branches.sort_by_key(|branch| branch != baseline_branch);
for branch in branches {
push(repo_path, &["-u", remote, &branch])?;
push(repo_path, &["-u", &remote, &branch])?;
}
push(repo_path, &[remote, "--tags"])?;
Ok(())
}
pub fn local_branches(repo_path: &Path) -> Result<Vec<String>> {
Ok(git(
Some(repo_path),
&["for-each-ref", "--format=%(refname:short)", "refs/heads"],
)?
.lines()
.filter(|branch| !branch.is_empty())
.map(str::to_string)
.collect())
pub fn push_branch(repo_path: &Path, branch: &str, owner: &str, repo: &str) -> Result<()> {
let remote = publication_remote(repo_path, owner, repo)?;
push(repo_path, &["-u", &remote, branch])
}
pub fn spawn_branch_publication(
repo_path: &Path,
branch: &str,
owner: &str,
repo: &str,
) -> Result<()> {
let executable = std::env::current_exe()?;
let mut command = Command::new(executable);
command
.arg("publish-branch")
.arg(repo_path)
.arg(branch)
.arg(owner)
.arg(repo)
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null());
#[cfg(unix)]
command.process_group(0);
let mut child = command
.spawn()
.map_err(|error| anyhow!("Could not start GitHub publication worker: {error}"))?;
std::thread::spawn(move || {
let _ = child.wait();
});
Ok(())
}
pub fn publication_sync_status(
@@ -936,18 +1045,20 @@ pub fn publication_sync_status(
let Ok(remote) = publication_remote(repo_path, owner, repo) else {
return "not configured";
};
let Ok(remote_refs) = authenticated_git(
repo_path,
&["ls-remote", "--heads", &remote],
Duration::from_secs(30),
) else {
return "unknown";
};
let Ok(branches) = local_branches(repo_path) else {
return "unknown";
};
if !branches.iter().any(|branch| branch == baseline_branch) {
return "unknown";
}
let Ok(remote_refs) =
authenticated_git(repo_path, &["ls-remote", "--heads", "--tags", &remote])
else {
return "unknown";
};
for branch in branches {
for branch in branches
.into_iter()
.filter(|branch| branch == baseline_branch || branch.starts_with("orx/"))
{
let Ok(local) = local_head_sha(repo_path, &branch) else {
return "unknown";
};
@@ -963,72 +1074,22 @@ pub fn publication_sync_status(
};
}
}
let Ok(local_tags) = git(Some(repo_path), &["tag", "--list"]) else {
return "unknown";
};
if local_tags.lines().all(|tag| {
let Ok(sha) = git(
Some(repo_path),
&["rev-parse", &format!("{tag}^{{commit}}")],
) else {
return false;
};
let direct = format!("refs/tags/{tag}");
let peeled = format!("{direct}^{{}}");
remote_refs.lines().any(|remote_line| {
remote_line == format!("{sha}\t{direct}") || remote_line == format!("{sha}\t{peeled}")
})
}) {
"synced"
} else {
"local changes to push"
}
"synced"
}
/// Create `new_branch` from `parent_branch`'s local tip and optionally publish it.
/// Create `new_branch` from `parent_branch`'s local tip.
pub fn create_experiment_branch(
repo_path: &Path,
parent_branch: &str,
new_branch: &str,
publication: Option<(&str, &str)>,
) -> Result<()> {
git(
Some(repo_path),
&["branch", "--no-track", new_branch, parent_branch],
)?;
let Some((owner, repo)) = publication else {
return Ok(());
};
if let Err(err) = push_branch(repo_path, new_branch, owner, repo) {
// Leave nothing behind — a retry re-picks the same slug.
let _ = git(Some(repo_path), &["branch", "-D", new_branch]);
return Err(err);
}
Ok(())
}
/// Head SHA of a branch — the *remote* tip when it exists (that's what a job
/// clones), the local ref otherwise. The opposite preference of
/// `resolve_branch_commit`, which serves the code browser and wants the
/// agent's not-yet-pushed local work.
pub fn branch_head_sha(repo_path: &Path, branch: &str, owner: &str, repo: &str) -> Result<String> {
let remote = format!(
"refs/remotes/{}/{branch}",
publication_remote(repo_path, owner, repo)?
);
if let Ok(sha) = git(Some(repo_path), &["rev-parse", &remote]) {
return Ok(sha);
}
git(Some(repo_path), &["rev-parse", branch])
}
/// Whether origin already has the branch (a cheap network check).
pub fn branch_on_remote(repo_path: &Path, branch: &str, owner: &str, repo: &str) -> Result<bool> {
let remote = publication_remote(repo_path, owner, repo)?;
let out = git(Some(repo_path), &["ls-remote", "--heads", &remote, branch])?;
Ok(!out.is_empty())
}
/// A file's content at a specific commit (`git show <sha>:<path>`), i.e.
/// exactly what a job cloning that sha will see — not the working tree.
pub fn file_at(repo_path: &Path, sha: &str, path: &str) -> Result<String> {
@@ -1044,56 +1105,6 @@ pub fn is_tracked(repo_path: &Path, path: &str) -> bool {
.is_ok()
}
pub fn push_branch(repo_path: &Path, branch: &str, owner: &str, repo: &str) -> Result<()> {
let remote = publication_remote(repo_path, owner, repo)?;
push(repo_path, &["-u", &remote, branch])?;
Ok(())
}
pub fn publish_branch_commit(
project: &crate::local::model::LocalProject,
branch: &str,
) -> Result<String> {
if !project.github_enabled() {
return Err(anyhow!(
"Remote compute requires this project's GitHub repository. Enable GitHub syncing for this project, then retry."
));
}
let repo_path = Path::new(&project.repo_path);
if !is_repository(repo_path) {
return Err(anyhow!("{} is not a Git repository", repo_path.display()));
}
push_branch(
repo_path,
branch,
&project.github_owner,
&project.github_repo,
)?;
let commit_sha = branch_head_sha(
repo_path,
branch,
&project.github_owner,
&project.github_repo,
)?;
let url = format!(
"https://github.com/{}/{}.git",
project.github_owner, project.github_repo
);
let remote = isolated_github_refs(&url, branch)?;
let reference = format!("refs/heads/{branch}");
if !remote
.lines()
.any(|line| line == format!("{commit_sha}\t{reference}"))
{
return Err(anyhow!(
"The recorded commit cannot be cloned through the HTTPS credentials used by remote jobs. Connect a GitHub token with access to {}/{}.",
project.github_owner,
project.github_repo
));
}
Ok(commit_sha)
}
// --- diffs ------------------------------------------------------------------
/// Whole-diff cap, mirroring the OpenResearch api's MAX_DIFF_BYTES.
@@ -1688,38 +1699,6 @@ mod tests {
let _ = std::fs::remove_dir_all(&repo);
}
#[test]
fn push_all_publishes_every_branch_and_tag() {
let repo = temp_repo();
run(&repo, &["branch", "orx/experiment"]);
run(&repo, &["branch", "notes"]);
run(&repo, &["tag", "v1"]);
let remote = repo.with_extension("bare.git");
let remote_string = remote.to_string_lossy().into_owned();
run(&repo, &["init", "--bare", &remote_string]);
run(&repo, &["remote", "add", GITHUB_REMOTE, &remote_string]);
push_all_to(&repo, "main", GITHUB_REMOTE).unwrap();
let heads = run(&repo, &["ls-remote", "--heads", GITHUB_REMOTE]);
assert!(heads.contains("refs/heads/main"));
assert!(heads.contains("refs/heads/orx/experiment"));
assert!(heads.contains("refs/heads/notes"));
assert_eq!(run(&repo, &["config", "branch.main.remote"]), GITHUB_REMOTE);
assert_eq!(
run(&repo, &["config", "branch.orx/experiment.remote"]),
GITHUB_REMOTE
);
assert_eq!(
run(&repo, &["config", "branch.notes.remote"]),
GITHUB_REMOTE
);
let tags = run(&repo, &["ls-remote", "--tags", GITHUB_REMOTE]);
assert!(tags.contains("refs/tags/v1"));
let _ = std::fs::remove_dir_all(&repo);
let _ = std::fs::remove_dir_all(&remote);
}
#[test]
fn experiment_branch_collision_never_rewrites_existing_work() {
let repo = temp_repo();
@@ -1729,52 +1708,11 @@ mod tests {
run(&repo, &["add", "later.txt"]);
run(&repo, &["commit", "-q", "-m", "later"]);
assert!(create_experiment_branch(&repo, "main", "orx/existing", None).is_err());
assert!(create_experiment_branch(&repo, "main", "orx/existing").is_err());
assert_eq!(run(&repo, &["rev-parse", "orx/existing"]), before);
let _ = std::fs::remove_dir_all(&repo);
}
#[test]
fn publication_remote_must_match_exact_repository() {
let repo = temp_repo();
run(
&repo,
&[
"remote",
"add",
GITHUB_REMOTE,
"https://github.com/owner/project-backup.git",
],
);
run(
&repo,
&[
"remote",
"add",
"origin",
"git@github.com:owner/project.git",
],
);
assert_eq!(
publication_remote(&repo, "owner", "project").unwrap(),
"origin"
);
run(
&repo,
&[
"remote",
"set-url",
"--add",
"--push",
"origin",
"https://github.com/owner/elsewhere.git",
],
);
assert!(publication_remote(&repo, "owner", "project").is_err());
assert!(publication_remote(&repo, "owner", "missing").is_err());
let _ = std::fs::remove_dir_all(&repo);
}
#[test]
fn public_clone_urls_reject_credentials_and_secret_suffixes() {
assert!(public_clone_url("https://user:token@example.com/repo.git").is_err());
@@ -1789,4 +1727,13 @@ mod tests {
"SSH://example.com/repo.git"
);
}
#[test]
fn public_clone_history_is_shallow_only_when_requested() {
assert_eq!(
public_clone_history_args(true),
&["--depth=1", "--single-branch"]
);
assert!(public_clone_history_args(false).is_empty());
}
}
src/local/github.rs
+83 -129
View File
@@ -1,7 +1,4 @@
//! Minimal GitHub REST calls for local mode — create a repo on the signed-in
//! user's account or in an organization, check push access, fork-by-copy.
//! Token from `GITHUB_TOKEN` or `gh auth token`, same resolution the clone path
//! uses.
//! Minimal GitHub REST calls for optional project publication.
use std::time::Duration;
@@ -11,6 +8,11 @@ use super::git::resolve_github_token;
use crate::error::{anyhow, Error, Result};
const UA: &str = concat!("orx/", env!("CARGO_PKG_VERSION"));
pub const SHALLOW_CLONE_THRESHOLD_KB: u64 = 250 * 1024;
pub fn should_shallow_clone(size_kb: Option<u64>) -> bool {
size_kb.is_some_and(|size| size >= SHALLOW_CLONE_THRESHOLD_KB)
}
#[derive(Debug)]
struct RepositoryNameExists;
@@ -23,35 +25,43 @@ impl std::fmt::Display for RepositoryNameExists {
impl std::error::Error for RepositoryNameExists {}
/// Create a blank private repo with an auto-init commit so the clone/branch
/// flow works immediately. An organization target is optional; without one,
/// GitHub creates the repo under the token's user.
pub async fn create_repo(
repo: &str,
organization: Option<&str>,
) -> Result<(String, String, String)> {
create_repo_api(repo, true, organization).await
}
/// Create an empty private repository for a local project. The local history
/// remains authoritative and becomes the remote's first history on push.
pub async fn create_project_repo(repo: &str) -> Result<(String, String, String)> {
match create_repo_api(repo, false, None).await {
Err(error) if error.downcast_ref::<RepositoryNameExists>().is_some() => {
let suffix = &uuid::Uuid::new_v4().simple().to_string()[..8];
create_repo_api(&format!("{repo}-{suffix}"), false, None).await
for suffix in 1..=100 {
let candidate = if suffix == 1 {
repo.to_string()
} else {
format!("{repo}-{suffix}")
};
match create_repo_api(&candidate, false).await {
Err(error) if error.downcast_ref::<RepositoryNameExists>().is_some() => continue,
result => return result,
}
result => result,
}
Err(anyhow!(
"Could not find an available GitHub repository name for '{repo}'."
))
}
pub async fn available_project_repo_name(repo: &str) -> String {
let Some(owner) = viewer_login().await else {
return repo.to_string();
};
for suffix in 1..=100 {
let candidate = if suffix == 1 {
repo.to_string()
} else {
format!("{repo}-{suffix}")
};
if repo_meta(&owner, &candidate).await.is_none() {
return candidate;
}
}
repo.to_string()
}
/// GET an api.github.com URL with the resolved token. `None` when there's no
/// token or the request fails — callers decide what that means.
async fn authed_get(url: &str) -> Option<reqwest::Response> {
let token = resolve_github_token()?;
reqwest::Client::builder()
// Without a timeout a black-holed connection hangs the caller — and
// the New project form blocks on this check.
.timeout(Duration::from_secs(10))
.build()
.ok()?
@@ -64,49 +74,60 @@ async fn authed_get(url: &str) -> Option<reqwest::Response> {
.ok()
}
/// What the New project form needs about a repo, from one API call.
pub async fn public_repo_size_kb(url: &str) -> Option<u64> {
let (owner, repo) = super::git::github_repository(url)?;
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(10))
.build()
.ok()?;
let mut request = client
.get(format!(
"https://api.github.com/repos/{}/{}",
urlencoding::encode(&owner),
urlencoding::encode(&repo)
))
.header("user-agent", UA)
.header("accept", "application/vnd.github+json")
.header("x-github-api-version", "2022-11-28");
if let Some(token) = resolve_github_token() {
request = request.bearer_auth(token);
}
let response = request.send().await.ok()?;
if !response.status().is_success() {
return None;
}
let body: Value = response.json().await.ok()?;
body.get("size").and_then(Value::as_u64)
}
pub struct RepoMeta {
pub can_push: bool,
pub archived: bool,
/// The repo's own default branch — the honest baseline when the user
/// doesn't pick one (`create_project` would otherwise assume "main").
pub default_branch: Option<String>,
}
/// The signed-in GitHub login, so the UI can name the account a new repo lands
/// on instead of guessing "you". `None` when there's no usable token.
pub async fn viewer_login() -> Option<String> {
let res = authed_get("https://api.github.com/user").await?;
if !res.status().is_success() {
let response = authed_get("https://api.github.com/user").await?;
if !response.status().is_success() {
return None;
}
let body: Value = res.json().await.ok()?;
let body: Value = response.json().await.ok()?;
body.get("login")
.and_then(Value::as_str)
.filter(|s| !s.is_empty())
.filter(|login| !login.is_empty())
.map(str::to_string)
}
/// Repo permissions + default branch. `None` means we couldn't tell (no token,
/// API hiccup); callers treat that as "assume access" so a check outage never
/// forces a surprise fork.
pub async fn repo_meta(owner: &str, repo: &str) -> Option<RepoMeta> {
// Encoded: owner/repo reach here straight from a text field, and a stray
// `/` or `..` would otherwise re-point the request at another endpoint.
let res = authed_get(&format!(
let response = authed_get(&format!(
"https://api.github.com/repos/{}/{}",
urlencoding::encode(owner),
urlencoding::encode(repo)
))
.await?;
match res.status() {
// Invisible to *this token* — which is also what a private repo looks
// like to an SSH-only user who can push to it perfectly well. Report
// unknown rather than "can't push": a wrong `false` silently snapshots
// their repo into a new one and drops its history.
match response.status() {
reqwest::StatusCode::NOT_FOUND => None,
s if s.is_success() => {
let body: Value = res.json().await.ok()?;
status if status.is_success() => {
let body: Value = response.json().await.ok()?;
Some(RepoMeta {
can_push: body
.pointer("/permissions/push")
@@ -116,89 +137,28 @@ pub async fn repo_meta(owner: &str, repo: &str) -> Option<RepoMeta> {
.get("archived")
.and_then(Value::as_bool)
.unwrap_or(false),
default_branch: body
.get("default_branch")
.and_then(Value::as_str)
.filter(|s| !s.is_empty())
.map(str::to_string),
})
}
_ => None,
}
}
/// Fork-by-copy: snapshot `src_owner/src_repo` (at `src_branch`, or its
/// default) into a fresh private repo `<slug>-<hash>` on the token's user —
/// the same import convention the platform uses — so the project always ends
/// up on a repo the user can push to. Returns (owner, repo, default_branch).
pub async fn fork_copy_repo(
src_owner: &str,
src_repo: &str,
src_branch: Option<String>,
destination_organization: Option<&str>,
) -> Result<(String, String, String)> {
let hash = &uuid::Uuid::new_v4().simple().to_string()[..8];
let name = format!("{}-{hash}", crate::local::slugify(src_repo));
let (owner, name, _) = create_repo_api(&name, false, destination_organization).await?;
let (src_owner, src_repo) = (src_owner.to_string(), src_repo.to_string());
let (dst_owner, dst_repo) = (owner.clone(), name.clone());
tokio::task::spawn_blocking(move || {
super::git::seed_copy(
&src_owner,
&src_repo,
src_branch.as_deref(),
&dst_owner,
&dst_repo,
)
})
.await
.map_err(|e| anyhow!("seed task failed: {e}"))??;
Ok((owner, name, "main".to_string()))
}
fn create_repo_endpoint(organization: Option<&str>) -> Result<String> {
let Some(organization) = organization
.map(str::trim)
.filter(|value| !value.is_empty())
else {
return Ok("https://api.github.com/user/repos".to_string());
};
let valid = organization.len() <= 39
&& !organization.starts_with('-')
&& !organization.ends_with('-')
&& !organization.contains("--")
&& organization
.chars()
.all(|character| character.is_ascii_alphanumeric() || character == '-');
if !valid {
return Err(anyhow!("Invalid GitHub organization login: {organization}"));
}
Ok(format!("https://api.github.com/orgs/{organization}/repos"))
}
async fn create_repo_api(
repo: &str,
auto_init: bool,
organization: Option<&str>,
) -> Result<(String, String, String)> {
async fn create_repo_api(repo: &str, auto_init: bool) -> Result<(String, String, String)> {
let token = resolve_github_token().ok_or_else(|| {
anyhow!(
"Creating a GitHub repo needs credentials — run `gh auth login` or set GITHUB_TOKEN."
)
anyhow!("Creating a GitHub repo needs credentials — run `gh auth login` or connect a GitHub token.")
})?;
let res = reqwest::Client::new()
.post(create_repo_endpoint(organization)?)
let response = reqwest::Client::new()
.post("https://api.github.com/user/repos")
.bearer_auth(&token)
.header("user-agent", UA)
.header("accept", "application/vnd.github+json")
.json(&json!({ "name": repo, "private": true, "auto_init": auto_init }))
.send()
.await
.map_err(|e| anyhow!("GitHub API unreachable: {e}"))?;
let status = res.status();
let body: Value = res.json().await.unwrap_or_default();
.map_err(|error| anyhow!("GitHub API unreachable: {error}"))?;
let status = response.status();
let body: Value = response.json().await.unwrap_or_default();
if status == reqwest::StatusCode::UNPROCESSABLE_ENTITY {
// Typically "name already exists on this account".
let detail = body
.pointer("/errors/0/message")
.and_then(Value::as_str)
@@ -211,11 +171,11 @@ async fn create_repo_api(
return Err(anyhow!("Could not create '{repo}': {detail}."));
}
if !status.is_success() {
let msg = body
let message = body
.get("message")
.and_then(Value::as_str)
.unwrap_or("unknown error");
return Err(anyhow!("GitHub repo create failed ({status}): {msg}"));
return Err(anyhow!("GitHub repo create failed ({status}): {message}"));
}
let owner = body
.pointer("/owner/login")
@@ -237,18 +197,12 @@ async fn create_repo_api(
#[cfg(test)]
mod tests {
use super::create_repo_endpoint;
use super::*;
#[test]
fn repository_endpoint_targets_user_or_organization() {
assert_eq!(
create_repo_endpoint(None).unwrap(),
"https://api.github.com/user/repos"
);
assert_eq!(
create_repo_endpoint(Some("alphaXiv")).unwrap(),
"https://api.github.com/orgs/alphaXiv/repos"
);
assert!(create_repo_endpoint(Some("not/an/org")).is_err());
fn shallow_clone_is_reserved_for_large_repositories() {
assert!(!should_shallow_clone(None));
assert!(!should_shallow_clone(Some(SHALLOW_CLONE_THRESHOLD_KB - 1)));
assert!(should_shallow_clone(Some(SHALLOW_CLONE_THRESHOLD_KB)));
}
}
src/local/harness/codex.rs
-8
View File
@@ -2491,14 +2491,6 @@ async fn run_turn_exec(ctx: &mut TurnCtx) -> Result<()> {
if let Some(dir) = &data_dir_pin {
cmd.env("ORX_DATA_DIR", dir);
}
// The sandbox blocks the keyring `gh` keeps its token in ("stored token is
// invalid" from inside the workspace), so resolve it out here and pass it
// down; both `gh` and its git credential helper prefer these env vars.
if let Some(token) = crate::local::git::resolve_github_token() {
cmd.env("GH_TOKEN", &token);
cmd.env("GITHUB_TOKEN", token);
}
let mut child = cmd
.spawn()
.map_err(|e| anyhow!("Could not spawn {}: {}", bin.display(), e))?;
src/local/hf.rs
+27 -43
View File
@@ -4,10 +4,10 @@
use std::collections::HashMap;
use crate::commands::exp::{default_hf_image, hf_clone_script, spawn_detached_supervise};
use crate::commands::exp::{default_hf_image, spawn_detached_supervise};
use crate::compute::SourceSnapshot;
use crate::error::{anyhow, Result};
use crate::jobs::{huggingface as hf, BackendDescriptor};
use crate::local::git;
use crate::store::{now_ms, Store, StoredRun};
/// CLI wrapper around `submit_local_hf`: submit, then print the summary.
@@ -34,6 +34,14 @@ pub async fn launch_local_hf(args: &crate::ExpRunArgs) -> Result<()> {
/// supervisor. `args.exp_id` must exist in `local_experiments`; requires
/// `--backend hf` and `--flavor`. Shared by the CLI and the `orx up` API.
pub async fn submit_local_hf(args: &crate::ExpRunArgs) -> Result<StoredRun> {
crate::compute::submit(args).await
}
pub async fn submit_local_hf_with_source(
args: &crate::ExpRunArgs,
source: SourceSnapshot,
run_id: String,
) -> Result<StoredRun> {
if args.sandbox.is_some() || args.gpu.is_some() || args.cpu.is_some() {
return Err(anyhow!(
"Local experiments run on Hugging Face Jobs; drop --gpu/--cpu/--sandbox \
@@ -78,60 +86,24 @@ pub async fn submit_local_hf(args: &crate::ExpRunArgs) -> Result<StoredRun> {
// One run in flight per experiment unless the caller deliberately forces
// a concurrent launch — the double-click / double-submit guard.
if !args.force {
if let Some(r) = store
.list_runs_by_experiment(&exp.id)?
.into_iter()
.find(|r| !crate::local::is_terminal(&r.status))
{
return Err(anyhow!(
"Run {} is already in flight for this experiment ({}). \
Cancel it with `orx exp cancel {}` or pass --force to launch anyway.",
r.id,
r.status,
exp.id
));
}
}
let token = hf::resolve_token()?;
let namespace = hf::whoami(&token).await?;
// The job clones from GitHub, so the branch tip must exist there. Fetch
// via ensure_clone so branch_head_sha matches what the job will check out.
// Git shells out (network, can stall) — keep it off the async workers.
let commit_sha = {
let project = project.clone();
let branch = exp.branch_name.clone();
tokio::task::spawn_blocking(move || git::publish_branch_commit(&project, &branch))
.await
.map_err(|e| anyhow!("git task failed: {e}"))??
};
let run_id = uuid::Uuid::new_v4().to_string();
let image = args
.image
.clone()
.unwrap_or_else(|| default_hf_image(&flavor));
let script = hf_clone_script(
&commit_sha,
&project.github_owner,
&project.github_repo,
&run_command,
);
let script = crate::compute::snapshot_script("/orx-source/source.tar", &run_command);
// Tokens travel as job secrets only — the command line stays tokenless.
let mut secrets = HashMap::new();
secrets.insert("HF_TOKEN".to_string(), token.clone());
if let Some(gh) = git::resolve_github_token() {
secrets.insert("GITHUB_TOKEN".to_string(), gh);
}
let mut labels = HashMap::new();
labels.insert("or_run".to_string(), run_id.clone());
labels.insert("or_experiment".to_string(), exp.id.clone());
labels.insert("or_project".to_string(), project.id.clone());
let job = hf::run_job(
let job = hf::run_job_with_source(
&token,
&namespace,
&hf::JobSubmission {
@@ -143,10 +115,12 @@ pub async fn submit_local_hf(args: &crate::ExpRunArgs) -> Result<StoredRun> {
timeout_seconds,
labels,
},
&source.path,
&source.digest,
)
.await?;
let descriptor = BackendDescriptor {
let mut descriptor = BackendDescriptor {
kind: "hf_job".to_string(),
namespace: Some(namespace.clone()),
job_id: Some(job.id.clone()),
@@ -160,7 +134,15 @@ pub async fn submit_local_hf(args: &crate::ExpRunArgs) -> Result<StoredRun> {
ssh_port: None,
ssh_user: None,
timeout_secs: None,
source_digest: None,
source_path: None,
source_size: None,
};
source.apply_to_descriptor(&mut descriptor);
if let Err(error) = crate::compute::record_submission_handle(&run_id, &descriptor) {
let _ = hf::cancel_job(&token, &namespace, &job.id).await;
return Err(error);
}
let run = StoredRun {
id: run_id.clone(),
experiment_id: exp.id.clone(),
@@ -172,9 +154,11 @@ pub async fn submit_local_hf(args: &crate::ExpRunArgs) -> Result<StoredRun> {
updated_at: now_ms(),
ended_at: None,
exit_code: None,
commit_sha: Some(commit_sha),
commit_sha: Some(source.revision),
result_markdown: None,
cancel_requested: false,
cancel_requested: store
.get_run(&run_id)?
.is_some_and(|run| run.cancel_requested),
chat_session_id: crate::local::chat::launching_chat_session(),
};
store.upsert_run(&run)?;
src/local/k8s.rs
+40 -51
View File
@@ -5,13 +5,14 @@
//!
//! There are no flavors: the run's shape is a **manifest committed on the
//! experiment branch** (default `.orx/k8s.yaml`, or `--manifest <path>`),
//! read at the branch tip — the same commit the job clones — so unpushed
//! manifest edits never run. See `jobs/kubernetes.rs` for the contract orx
//! read at the recorded revision — the same commit the job receives — so
//! uncommitted manifest edits never run. See `jobs/kubernetes.rs` for the contract orx
//! enforces on it.
use std::collections::HashMap;
use crate::commands::exp::{hf_clone_script, spawn_detached_supervise};
use crate::commands::exp::spawn_detached_supervise;
use crate::compute::SourceSnapshot;
use crate::error::{anyhow, Result};
use crate::jobs::{huggingface as hf, kubernetes as k8s, BackendDescriptor};
use crate::local::git;
@@ -49,6 +50,14 @@ pub async fn launch_local_k8s(args: &crate::ExpRunArgs) -> Result<()> {
/// Submit the local experiment's run from its committed manifest and detach a
/// supervisor.
pub async fn submit_local_k8s(args: &crate::ExpRunArgs) -> Result<StoredRun> {
crate::compute::submit(args).await
}
pub async fn submit_local_k8s_with_source(
args: &crate::ExpRunArgs,
source: SourceSnapshot,
run_id: String,
) -> Result<StoredRun> {
if args.sandbox.is_some() || args.gpu.is_some() || args.cpu.is_some() {
return Err(anyhow!(
"--backend k8s runs from a manifest committed on the experiment branch; \
@@ -101,62 +110,31 @@ pub async fn submit_local_k8s(args: &crate::ExpRunArgs) -> Result<StoredRun> {
})?;
// One run in flight per experiment unless deliberately forced.
if !args.force {
if let Some(r) = store
.list_runs_by_experiment(&exp.id)?
.into_iter()
.find(|r| !crate::local::is_terminal(&r.status))
{
return Err(anyhow!(
"Run {} is already in flight for this experiment ({}). \
Cancel it with `orx exp cancel {}` or pass --force to launch anyway.",
r.id,
r.status,
exp.id
));
}
}
// The job clones from GitHub, so the branch tip must exist there — and the
// manifest is read from that same tip, not the working tree.
let (commit_sha, manifest) = {
let project = project.clone();
let branch = exp.branch_name.clone();
let manifest = {
let repo_path = project.repo_path.clone();
let revision = source.revision.clone();
let path = manifest_path.clone();
tokio::task::spawn_blocking(move || -> Result<(String, String)> {
let sha = git::publish_branch_commit(&project, &branch)?;
let manifest = git::file_at(std::path::Path::new(&project.repo_path), &sha, &path)
.map_err(|_| {
tokio::task::spawn_blocking(move || -> Result<String> {
git::file_at(std::path::Path::new(&repo_path), &revision, &path).map_err(|_| {
anyhow!(
"No manifest at '{path}' on branch '{branch}' — write one, commit, \
and push (jobs run the branch tip, so an uncommitted manifest \
doesn't exist yet). Pass --manifest <path> if it lives elsewhere."
"No manifest at '{path}' in committed revision {revision} — write one \
and commit it before running. Pass --manifest <path> if it lives elsewhere."
)
})?;
Ok((sha, manifest))
})
})
.await
.map_err(|e| anyhow!("git task failed: {e}"))??
};
let run_id = uuid::Uuid::new_v4().to_string();
let script = hf_clone_script(
&commit_sha,
&project.github_owner,
&project.github_repo,
&run_command,
);
let script = crate::compute::gated_script("/tmp/orx-source/source.tar", &run_command);
// The pod's env: everything the user synced (API keys), plus the tokens
// the clone script and common tooling expect. Travels via a k8s Secret,
// the run script and common tooling expect. Travels via a k8s Secret,
// never on a command line.
let mut env: HashMap<String, String> = crate::config::list_synced_env().into_iter().collect();
if let Ok(hf_token) = hf::resolve_token() {
env.entry("HF_TOKEN".to_string()).or_insert(hf_token);
}
if let Some(gh) = git::resolve_github_token() {
env.insert("GITHUB_TOKEN".to_string(), gh);
}
let mut labels = HashMap::new();
labels.insert("or_run".to_string(), run_id.clone());
labels.insert("or_experiment".to_string(), exp.id.clone());
@@ -181,25 +159,34 @@ pub async fn submit_local_k8s(args: &crate::ExpRunArgs) -> Result<StoredRun> {
env,
timeout_seconds,
labels,
source_archive: source.path.clone(),
},
)
.await?;
let descriptor = BackendDescriptor {
let mut descriptor = BackendDescriptor {
kind: "k8s_job".to_string(),
namespace: Some(namespace),
job_id: Some(submitted.job_name),
namespace: Some(namespace.clone()),
job_id: Some(submitted.job_name.clone()),
flavor: None,
image: None,
url: None,
context,
context: context.clone(),
manifest: Some(manifest_path),
resources: Some(submitted.resources),
resources: Some(submitted.resources.clone()),
ssh_host: None,
ssh_port: None,
ssh_user: None,
timeout_secs: None,
source_digest: None,
source_path: None,
source_size: None,
};
source.apply_to_descriptor(&mut descriptor);
if let Err(error) = crate::compute::record_submission_handle(&run_id, &descriptor) {
let _ = k8s::delete_resources(context.as_deref(), &namespace, &submitted.resources).await;
return Err(error);
}
let run = StoredRun {
id: run_id.clone(),
experiment_id: exp.id.clone(),
@@ -211,9 +198,11 @@ pub async fn submit_local_k8s(args: &crate::ExpRunArgs) -> Result<StoredRun> {
updated_at: now_ms(),
ended_at: None,
exit_code: None,
commit_sha: Some(commit_sha),
commit_sha: Some(source.revision),
result_markdown: None,
cancel_requested: false,
cancel_requested: store
.get_run(&run_id)?
.is_some_and(|run| run.cancel_requested),
chat_session_id: crate::local::chat::launching_chat_session(),
};
store.upsert_run(&run)?;
src/local/localrun.rs
+27 -36
View File
@@ -1,15 +1,15 @@
//! Local launch — the on-this-machine twin of `local/ssh.rs`: run the
//! experiment as a detached process on the machine running orx. Same clone
//! contract as every backend — the run clones the branch's GitHub tip into
//! experiment as a detached process on the machine running orx. Same snapshot
//! contract as every backend — the run extracts the recorded revision into
//! its own run dir, never the agent's worktree. The run row lives in the
//! local store only; a detached `orx supervise` watches the process.
use std::collections::HashMap;
use crate::commands::exp::{local_clone_script, spawn_detached_supervise};
use crate::commands::exp::spawn_detached_supervise;
use crate::compute::SourceSnapshot;
use crate::error::{anyhow, Result};
use crate::jobs::{localbox, BackendDescriptor};
use crate::local::git;
use crate::store::{now_ms, Store, StoredRun};
/// CLI wrapper around `submit_local_run`: submit, then print the summary.
@@ -30,6 +30,14 @@ pub async fn launch_local_run(args: &crate::ExpRunArgs) -> Result<()> {
/// and detach a supervisor. Requires `--backend local`; there is nothing else
/// to pick — the hardware is whatever this machine has.
pub async fn submit_local_run(args: &crate::ExpRunArgs) -> Result<StoredRun> {
crate::compute::submit(args).await
}
pub async fn submit_local_run_with_source(
args: &crate::ExpRunArgs,
source: SourceSnapshot,
run_id: String,
) -> Result<StoredRun> {
if args.sandbox.is_some() || args.gpu.is_some() || args.cpu.is_some() {
return Err(anyhow!(
"--backend local runs on this machine; drop --gpu/--cpu/--sandbox — \
@@ -71,37 +79,10 @@ pub async fn submit_local_run(args: &crate::ExpRunArgs) -> Result<StoredRun> {
})?;
// One run in flight per experiment unless deliberately forced.
if !args.force {
if let Some(r) = store
.list_runs_by_experiment(&exp.id)?
.into_iter()
.find(|r| !crate::local::is_terminal(&r.status))
{
return Err(anyhow!(
"Run {} is already in flight for this experiment ({}). \
Cancel it with `orx exp cancel {}` or pass --force to launch anyway.",
r.id,
r.status,
exp.id
));
}
}
let commit_sha = {
let repo_path = project.repo_path.clone();
let branch = exp.branch_name.clone();
tokio::task::spawn_blocking(move || -> Result<String> {
git::local_head_sha(std::path::Path::new(&repo_path), &branch)
})
.await
.map_err(|e| anyhow!("git task failed: {e}"))??
};
let run_id = uuid::Uuid::new_v4().to_string();
let script = local_clone_script(&project.repo_path, &commit_sha, &run_command);
let script = crate::compute::snapshot_script(&source.path.to_string_lossy(), &run_command);
// The run's env: everything the user synced (API keys), plus the tokens
// the clone script expects. Exported inside run.sh (written owner-only).
// the run script expects. Exported inside run.sh (written owner-only).
let mut env: HashMap<String, String> = crate::config::list_synced_env().into_iter().collect();
if let Ok(hf_token) = crate::jobs::huggingface::resolve_token() {
env.entry("HF_TOKEN".to_string()).or_insert(hf_token);
@@ -113,7 +94,7 @@ pub async fn submit_local_run(args: &crate::ExpRunArgs) -> Result<StoredRun> {
env,
})?;
let descriptor = BackendDescriptor {
let mut descriptor = BackendDescriptor {
kind: "local_job".to_string(),
namespace: None,
job_id: Some(dir.to_string_lossy().into_owned()),
@@ -127,7 +108,15 @@ pub async fn submit_local_run(args: &crate::ExpRunArgs) -> Result<StoredRun> {
ssh_port: None,
ssh_user: None,
timeout_secs: None,
source_digest: None,
source_path: None,
source_size: None,
};
source.apply_to_descriptor(&mut descriptor);
if let Err(error) = crate::compute::record_submission_handle(&run_id, &descriptor) {
let _ = localbox::cancel_job(&dir);
return Err(error);
}
let run = StoredRun {
id: run_id.clone(),
experiment_id: exp.id.clone(),
@@ -139,9 +128,11 @@ pub async fn submit_local_run(args: &crate::ExpRunArgs) -> Result<StoredRun> {
updated_at: now_ms(),
ended_at: None,
exit_code: None,
commit_sha: Some(commit_sha),
commit_sha: Some(source.revision),
result_markdown: None,
cancel_requested: false,
cancel_requested: store
.get_run(&run_id)?
.is_some_and(|run| run.cancel_requested),
chat_session_id: crate::local::chat::launching_chat_session(),
};
store.upsert_run(&run)?;
src/local/modal.rs
+28 -43
View File
@@ -6,10 +6,10 @@
use std::collections::HashMap;
use crate::commands::exp::{hf_clone_script, spawn_detached_supervise};
use crate::commands::exp::spawn_detached_supervise;
use crate::compute::SourceSnapshot;
use crate::error::{anyhow, Result};
use crate::jobs::{huggingface as hf, modal, BackendDescriptor};
use crate::local::git;
use crate::store::{now_ms, Store, StoredRun};
/// Modal app all orx sandboxes are grouped under (visible in the Modal dashboard).
@@ -38,6 +38,14 @@ pub async fn launch_local_modal(args: &crate::ExpRunArgs) -> Result<()> {
/// is a Modal GPU (t4, l4, a10g, a100, a100-80gb, l40s, h100, h200, …) or
/// `cpu` / `cpu-large` for CPU-only.
pub async fn submit_local_modal(args: &crate::ExpRunArgs) -> Result<StoredRun> {
crate::compute::submit(args).await
}
pub async fn submit_local_modal_with_source(
args: &crate::ExpRunArgs,
source: SourceSnapshot,
run_id: String,
) -> Result<StoredRun> {
if args.sandbox.is_some() || args.gpu.is_some() || args.cpu.is_some() {
return Err(anyhow!(
"--backend modal runs on Modal serverless GPUs; drop --gpu/--cpu/--sandbox \
@@ -52,7 +60,7 @@ pub async fn submit_local_modal(args: &crate::ExpRunArgs) -> Result<StoredRun> {
)
})?;
let resources = modal::resolve_flavor(&flavor_name);
// Fail before the git push if Modal plainly isn't set up on this box.
// Fail before source staging if Modal plainly isn't set up on this box.
modal::preflight().await?;
// Same default as the HF/k8s paths — no-timeout jobs are a footgun.
let timeout_seconds = match &args.timeout {
@@ -83,53 +91,19 @@ pub async fn submit_local_modal(args: &crate::ExpRunArgs) -> Result<StoredRun> {
})?;
// One run in flight per experiment unless deliberately forced.
if !args.force {
if let Some(r) = store
.list_runs_by_experiment(&exp.id)?
.into_iter()
.find(|r| !crate::local::is_terminal(&r.status))
{
return Err(anyhow!(
"Run {} is already in flight for this experiment ({}). \
Cancel it with `orx exp cancel {}` or pass --force to launch anyway.",
r.id,
r.status,
exp.id
));
}
}
// The sandbox clones from GitHub, so the branch tip must exist there.
let commit_sha = {
let project = project.clone();
let branch = exp.branch_name.clone();
tokio::task::spawn_blocking(move || git::publish_branch_commit(&project, &branch))
.await
.map_err(|e| anyhow!("git task failed: {e}"))??
};
let run_id = uuid::Uuid::new_v4().to_string();
let image = args
.image
.clone()
.unwrap_or_else(|| modal::default_image(resources.gpu.is_some()));
let script = hf_clone_script(
&commit_sha,
&project.github_owner,
&project.github_repo,
&run_command,
);
let script = crate::compute::gated_script("/tmp/orx-source.tar", &run_command);
// The sandbox's env: everything the user synced (API keys), plus the tokens
// the clone script and common tooling expect. Rides an ephemeral Modal
// the run script and common tooling expect. Rides an ephemeral Modal
// Secret, never the plain env arg.
let mut env: HashMap<String, String> = crate::config::list_synced_env().into_iter().collect();
if let Ok(hf_token) = hf::resolve_token() {
env.entry("HF_TOKEN".to_string()).or_insert(hf_token);
}
if let Some(gh) = git::resolve_github_token() {
env.insert("GITHUB_TOKEN".to_string(), gh);
}
let mut tags = HashMap::new();
tags.insert("or_run".to_string(), run_id.clone());
tags.insert("or_experiment".to_string(), exp.id.clone());
@@ -145,13 +119,14 @@ pub async fn submit_local_modal(args: &crate::ExpRunArgs) -> Result<StoredRun> {
timeout_seconds,
app: MODAL_APP.to_string(),
tags,
source_archive: Some(source.path.clone()),
})
.await?;
let descriptor = BackendDescriptor {
let mut descriptor = BackendDescriptor {
kind: "modal_job".to_string(),
namespace: Some(MODAL_APP.to_string()),
job_id: Some(sandbox_id),
job_id: Some(sandbox_id.clone()),
flavor: Some(flavor_name),
image: Some(image),
url: None,
@@ -162,7 +137,15 @@ pub async fn submit_local_modal(args: &crate::ExpRunArgs) -> Result<StoredRun> {
ssh_port: None,
ssh_user: None,
timeout_secs: None,
source_digest: None,
source_path: None,
source_size: None,
};
source.apply_to_descriptor(&mut descriptor);
if let Err(error) = crate::compute::record_submission_handle(&run_id, &descriptor) {
let _ = modal::cancel_job(&sandbox_id).await;
return Err(error);
}
let run = StoredRun {
id: run_id.clone(),
experiment_id: exp.id.clone(),
@@ -174,9 +157,11 @@ pub async fn submit_local_modal(args: &crate::ExpRunArgs) -> Result<StoredRun> {
updated_at: now_ms(),
ended_at: None,
exit_code: None,
commit_sha: Some(commit_sha),
commit_sha: Some(source.revision),
result_markdown: None,
cancel_requested: false,
cancel_requested: store
.get_run(&run_id)?
.is_some_and(|run| run.cancel_requested),
chat_session_id: crate::local::chat::launching_chat_session(),
};
store.upsert_run(&run)?;
src/local/model.rs
+1 -1
View File
@@ -17,7 +17,7 @@ pub struct LocalProject {
/// where any experiment lives (legacy roots predating per-baseline
/// branches may still ride it).
pub baseline_branch: String,
/// Local clone path (`~/.cache/openresearch/repos/<owner>/<repo>`).
/// Local repository path.
pub repo_path: String,
pub run_command: Option<String>,
/// arXiv id the project starts from (versionless, e.g. `2401.12345`).
src/local/opencode.rs
+25 -49
View File
@@ -110,33 +110,17 @@ fn opencode_config_json(model: Option<&str>, instructions: &str) -> String {
const SYSTEM_PROMPT: &str = include_str!("../../SYSTEM_PROMPT.md");
fn playbook_md(project: &LocalProject) -> String {
let all_backends_enabled = true;
let id = &project.id;
let name = &project.name;
let publication_line = if project.github_enabled() {
format!(
"- GitHub repository: {}",
project
.github_url()
.expect("enabled project has publication metadata")
)
"- Source: immutable snapshots from committed local Git revisions\n- GitHub publication: enabled for experiment visibility; never used for compute transport"
} else {
"- GitHub: not enabled — this project is local-only".to_string()
};
let experiment_publish_clause = if project.github_enabled() {
"created locally and pushed to GitHub"
} else {
"created locally and never pushed"
};
let edit_step = if project.github_enabled() {
"2. **Edit** in this worktree: check out `<branch>`, change the code, commit, and `git push`. Remote runs use the pushed commit."
} else {
"2. **Edit** in this worktree: check out `<branch>`, change the code, and commit. Never push; local runs clone the recorded local commit."
};
let compute_contract = if project.github_enabled() {
"Remote backends clone the pushed GitHub commit; the local backend clones the recorded commit directly from the project folder."
} else {
"This project is local-only: only the `local` backend is available, and it clones the recorded commit directly from the project folder."
"- Source: immutable snapshots from committed local Git revisions\n- GitHub publication: disabled"
};
let experiment_publish_clause = "created locally";
let edit_step = "2. **Edit** in this worktree: check out `<branch>`, change the code, and commit. Do not push just to launch compute; `orx` snapshots the committed revision directly.";
let compute_contract = "Every backend receives the same immutable archive of the recorded local Git revision; no backend clones a repository or needs GitHub credentials.";
let baseline = &project.baseline_branch;
let artifacts = super::files::files_dir(project)
.to_string_lossy()
@@ -157,7 +141,7 @@ fn playbook_md(project: &LocalProject) -> String {
// told to OMIT `--backend`, never to echo the default back, so even a
// stale prompt launches on the current default.
let configured_compute_default = crate::config::compute_default();
let compute_default = if project.github_enabled() {
let compute_default = if all_backends_enabled {
Some(
configured_compute_default
.clone()
@@ -171,9 +155,8 @@ fn playbook_md(project: &LocalProject) -> String {
} else {
"the local fallback"
};
let compute_bullet = if !project.github_enabled() {
"- Compute: **local only** — run recorded commits on this machine. External backends remain unavailable until the user enables GitHub syncing for this project."
.to_string()
let compute_bullet = if !all_backends_enabled {
"- Compute: run immutable snapshots of recorded commits on configured backends.".to_string()
} else {
match &compute_default {
Some((b, f)) => {
@@ -194,9 +177,8 @@ fn playbook_md(project: &LocalProject) -> String {
}
}
};
let backends_intro = if !project.github_enabled() {
"`orx exp run` uses only the `local` backend for this project. Do not select, configure, or contact an external backend."
.to_string()
let backends_intro = if !all_backends_enabled {
"`orx exp run` transfers the recorded source snapshot to the selected backend.".to_string()
} else {
match &compute_default {
Some((b, f)) => {
@@ -235,7 +217,7 @@ fn playbook_md(project: &LocalProject) -> String {
.to_string(),
}
};
let (run_invocation, run_guidance, compute_guidance) = if project.github_enabled() {
let (run_invocation, run_guidance, compute_guidance) = if all_backends_enabled {
(
"`orx exp run <expId> [--backend <hf|modal|k8s|ssh|slurm|openresearch|local>] [flags]`",
"Launch the node's run. Backend flags, flavors, and sizing: **`orx-compute` skill** (k8s manifest: **`orx-compute-k8s`**).",
@@ -243,18 +225,18 @@ fn playbook_md(project: &LocalProject) -> String {
)
} else {
(
"`orx exp run <expId> --backend local`",
"Launch the recorded commit on this machine. External backends are unavailable until the user enables GitHub.",
"Load `orx-compute` for the local launch/wait contract. Do not configure or contact external providers.",
"`orx exp run <expId> [--backend <name>] [flags]`",
"Launch the recorded commit on a configured backend.",
"Load `orx-compute` for the launch and wait contract.",
)
};
let skills_scope = if project.github_enabled() {
let skills_scope = if all_backends_enabled {
"backend flags and sizing, the k8s manifest, tree shaping, git recipes, log analysis, and artifact naming"
} else {
"local runs, tree shaping, local git recipes, log analysis, and artifact naming"
};
let launch_step = if !project.github_enabled() {
"3. **Launch locally**: `orx exp run <expId> --backend local`. External backends are unavailable until the user enables GitHub syncing for this project."
let launch_step = if !all_backends_enabled {
"3. **Launch**: `orx exp run <expId>` using the configured backend."
} else if compute_default.is_some() {
"3. **Launch**: `orx exp run <expId>` — omitting `--backend` uses the default\n \
target (flags the default still needs are listed under \"Compute backends\") —\n \
@@ -270,11 +252,7 @@ fn playbook_md(project: &LocalProject) -> String {
// the playbook index and the files on disk can never drift.
let skills_list = super::agent_skills::skills(super::agent_skills::SkillSet::Local)
.iter()
.filter(|skill| super::agent_skills::available_in_session(skill, project.github_enabled()))
.map(|s| {
let description = super::agent_skills::session_description(s, project.github_enabled());
format!("- **{}** — {description}", s.name)
})
.map(|s| format!("- **{}** — {}", s.name, s.description))
.collect::<Vec<_>>()
.join("\n");
let template = SYSTEM_PROMPT
@@ -284,7 +262,7 @@ fn playbook_md(project: &LocalProject) -> String {
template
.replace("{name}", name)
.replace("{id}", id)
.replace("{publication_line}", &publication_line)
.replace("{publication_line}", publication_line)
.replace("{experiment_publish_clause}", experiment_publish_clause)
.replace("{edit_step}", edit_step)
.replace("{compute_contract}", compute_contract)
@@ -770,17 +748,15 @@ mod tests {
}
#[test]
fn local_only_playbook_requires_commits_without_pushes() {
fn playbook_requires_commits_without_pushes() {
let mut project = sample_project();
project.github_owner.clear();
project.github_repo.clear();
let md = playbook_md(&project);
assert!(md.contains("GitHub: not enabled"));
assert!(md.contains("Never push; local runs clone the recorded local commit"));
assert!(md.contains("Compute: **local only**"));
assert!(md.contains("`orx exp run <expId> --backend local`"));
assert!(!md.contains("orx-compute-k8s"));
assert!(!md.contains("--backend <hf|modal"));
assert!(md.contains("immutable archive"));
assert!(md.contains("Do not push just to launch compute"));
assert!(md.contains("orx-compute-k8s"));
assert!(md.contains("--backend <hf|modal"));
}
#[test]
src/local/openresearch.rs
+22 -31
View File
@@ -6,9 +6,9 @@
use crate::client::{create_sandbox, list_orgs, CreateSandboxBody};
use crate::commands::exp::spawn_detached_supervise;
use crate::compute::SourceSnapshot;
use crate::error::{anyhow, require_credentials, Result};
use crate::jobs::{openresearch, BackendDescriptor};
use crate::local::git;
use crate::local::ssh_identity;
use crate::store::{now_ms, Store, StoredRun};
@@ -41,6 +41,14 @@ pub async fn launch_local_openresearch(args: &crate::ExpRunArgs) -> Result<()> {
/// supervisor. Requires `--backend openresearch` and `--flavor <shape>`
/// (`h100_sxm[:count]` or `cpu5c|cpu5g|cpu5m[:vcpus]`), plus `orx login`.
pub async fn submit_local_openresearch(args: &crate::ExpRunArgs) -> Result<StoredRun> {
crate::compute::submit(args).await
}
pub async fn submit_local_openresearch_with_source(
args: &crate::ExpRunArgs,
source: SourceSnapshot,
run_id: String,
) -> Result<StoredRun> {
if args.sandbox.is_some() || args.gpu.is_some() || args.cpu.is_some() {
return Err(anyhow!(
"--gpu/--cpu/--sandbox are the managed server-experiment flags; with \
@@ -157,32 +165,6 @@ pub async fn submit_local_openresearch(args: &crate::ExpRunArgs) -> Result<Store
})?;
// One run in flight per experiment unless deliberately forced.
if !args.force {
if let Some(r) = store
.list_runs_by_experiment(&exp.id)?
.into_iter()
.find(|r| !crate::local::is_terminal(&r.status))
{
return Err(anyhow!(
"Run {} is already in flight for this experiment ({}). \
Cancel it with `orx exp cancel {}` or pass --force to launch anyway.",
r.id,
r.status,
exp.id
));
}
}
// The box clones from GitHub, so the branch tip must exist there — push
// BEFORE provisioning so a git failure never bills a box.
let commit_sha = {
let project = project.clone();
let branch = exp.branch_name.clone();
tokio::task::spawn_blocking(move || git::publish_branch_commit(&project, &branch))
.await
.map_err(|e| anyhow!("git task failed: {e}"))??
};
let sandbox = create_sandbox(
&creds,
&CreateSandboxBody {
@@ -194,8 +176,7 @@ pub async fn submit_local_openresearch(args: &crate::ExpRunArgs) -> Result<Store
.map_err(billing_friendly)?
.sandbox;
let run_id = uuid::Uuid::new_v4().to_string();
let descriptor = BackendDescriptor {
let mut descriptor = BackendDescriptor {
kind: "openresearch_job".to_string(),
namespace: Some(org_id),
job_id: Some(sandbox.id.clone()),
@@ -209,7 +190,15 @@ pub async fn submit_local_openresearch(args: &crate::ExpRunArgs) -> Result<Store
ssh_port: None,
ssh_user: None,
timeout_secs: Some(timeout_secs),
source_digest: None,
source_path: None,
source_size: None,
};
source.apply_to_descriptor(&mut descriptor);
if let Err(error) = crate::compute::record_submission_handle(&run_id, &descriptor) {
let _ = openresearch::teardown(&creds, &sandbox.id).await;
return Err(error);
}
let run = StoredRun {
id: run_id.clone(),
experiment_id: exp.id.clone(),
@@ -221,9 +210,11 @@ pub async fn submit_local_openresearch(args: &crate::ExpRunArgs) -> Result<Store
updated_at: now_ms(),
ended_at: None,
exit_code: None,
commit_sha: Some(commit_sha),
commit_sha: Some(source.revision),
result_markdown: None,
cancel_requested: false,
cancel_requested: store
.get_run(&run_id)?
.is_some_and(|run| run.cancel_requested),
chat_session_id: crate::local::chat::launching_chat_session(),
};
src/local/projects.rs
+21 -7
View File
@@ -33,6 +33,10 @@ fn unique_project_slug(store: &Store, base: &str) -> Result<String> {
}
}
pub fn project_slug_preview(store: &Store, name: &str) -> Result<String> {
unique_project_slug(store, &slugify(name))
}
pub(crate) fn expand_path(path: &str) -> Result<PathBuf> {
let trimmed = path.trim();
if trimmed.is_empty() {
@@ -60,6 +64,7 @@ fn prepare_path(
create_folder: bool,
initialize_git: bool,
clone_url: Option<&str>,
shallow_clone: bool,
) -> Result<PathBuf> {
let path = expand_path(path)?;
if let Some(url) = clone_url.map(str::trim).filter(|url| !url.is_empty()) {
@@ -74,7 +79,7 @@ fn prepare_path(
} else if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)?;
}
git::clone_public(url, &path)?;
git::clone_public(url, &path, shallow_clone)?;
git::rename_origin_to_upstream(&path)?;
} else if !path.exists() {
if !create_folder {
@@ -115,11 +120,18 @@ pub fn create_project(
create_folder,
initialize_git,
clone_url,
shallow_clone,
run_command,
paper_id,
} = options;
let slug = unique_project_slug(store, &slugify(name))?;
let repo_path = prepare_path(path, create_folder, initialize_git, clone_url.as_deref())?;
let repo_path = prepare_path(
path,
create_folder,
initialize_git,
clone_url.as_deref(),
shallow_clone,
)?;
if store
.list_local_projects()?
.iter()
@@ -131,9 +143,9 @@ pub fn create_project(
));
}
let baseline_branch = git::require_current_branch(&repo_path)?;
let publication = git::github_publication(&repo_path);
let (github_owner, github_repo) = publication.unwrap_or_default();
let now = now_ms();
let project = LocalProject {
id: uuid::Uuid::new_v4().to_string(),
@@ -158,6 +170,7 @@ pub struct CreateProjectOptions {
pub create_folder: bool,
pub initialize_git: bool,
pub clone_url: Option<String>,
pub shallow_clone: bool,
pub run_command: Option<String>,
pub paper_id: Option<String>,
}
@@ -282,12 +295,12 @@ mod tests {
let remotes = git::remotes(Path::new(&project.repo_path)).unwrap();
assert_eq!(remotes[0].0, "upstream");
assert!(!remotes.iter().any(|(name, _)| name == "origin"));
assert!(!project.github_enabled());
assert!(project.github_owner.is_empty());
std::fs::remove_dir_all(root).unwrap();
}
#[test]
fn ordinary_github_origin_remains_local_only() {
fn ordinary_github_origin_remains_opt_in() {
let root = root();
let project_path = root.join("project");
initialized(&project_path);
@@ -315,7 +328,7 @@ mod tests {
}
#[test]
fn dedicated_github_remote_is_recognized() {
fn dedicated_github_remote_is_recognized_but_remains_opt_in() {
let root = root();
let project_path = root.join("project");
initialized(&project_path);
@@ -324,7 +337,7 @@ mod tests {
&[
"remote",
"add",
git::GITHUB_REMOTE,
"github",
"git@github.com:example/research.git",
],
);
@@ -336,6 +349,7 @@ mod tests {
CreateProjectOptions::default(),
)
.unwrap();
assert!(!project.github_enabled());
assert_eq!(project.github_owner, "example");
assert_eq!(project.github_repo, "research");
std::fs::remove_dir_all(root).unwrap();
src/local/ray.rs
+32 -44
View File
@@ -3,11 +3,11 @@
use std::collections::HashMap;
use crate::commands::exp::{hf_clone_script, spawn_detached_supervise};
use crate::commands::exp::spawn_detached_supervise;
use crate::compute::SourceSnapshot;
use crate::error::{anyhow, Result};
use crate::jobs::ssh::sh_quote;
use crate::jobs::{huggingface, ray, BackendDescriptor};
use crate::local::git;
use crate::store::{now_ms, Store, StoredRun};
/// CLI wrapper: submit, then print the summary.
@@ -31,6 +31,14 @@ pub async fn launch_local_ray(args: &crate::ExpRunArgs) -> Result<()> {
/// Submit the local experiment's run as a Ray Job and detach a supervisor.
pub async fn submit_local_ray(args: &crate::ExpRunArgs) -> Result<StoredRun> {
crate::compute::submit(args).await
}
pub async fn submit_local_ray_with_source(
args: &crate::ExpRunArgs,
source: SourceSnapshot,
run_id: String,
) -> Result<StoredRun> {
if args.sandbox.is_some() || args.gpu.is_some() || args.cpu.is_some() {
return Err(anyhow!(
"--backend ray submits to your Ray cluster; drop --gpu/--cpu/--sandbox and \
@@ -84,22 +92,6 @@ pub async fn submit_local_ray(args: &crate::ExpRunArgs) -> Result<StoredRun> {
)
})?;
if !args.force {
if let Some(r) = store
.list_runs_by_experiment(&exp.id)?
.into_iter()
.find(|r| !crate::local::is_terminal(&r.status))
{
return Err(anyhow!(
"Run {} is already in flight for this experiment ({}). \
Cancel it with `orx exp cancel {}` or pass --force to launch anyway.",
r.id,
r.status,
exp.id
));
}
}
// Reachability check before we touch git / allocate a run id.
ray::preflight(&address).await.map_err(|e| {
anyhow!(
@@ -109,58 +101,44 @@ pub async fn submit_local_ray(args: &crate::ExpRunArgs) -> Result<StoredRun> {
)
})?;
let commit_sha = {
let project = project.clone();
let branch = exp.branch_name.clone();
tokio::task::spawn_blocking(move || git::publish_branch_commit(&project, &branch))
.await
.map_err(|e| anyhow!("git task failed: {e}"))??
};
let run_id = uuid::Uuid::new_v4().to_string();
// Ray submission ids: letters, digits, dashes, underscores.
let submission_id = format!("orx-{}", run_id.replace('-', ""));
let script = hf_clone_script(
&commit_sha,
&project.github_owner,
&project.github_repo,
&run_command,
);
// The job env: everything the user synced (API keys), plus the tokens the
// clone step expects. Ray renders runtime_env in its dashboard, but anyone
// run step expects. Ray renders runtime_env in its dashboard, but anyone
// with dashboard access can submit jobs anyway — same trust boundary.
let mut env: HashMap<String, String> = crate::config::list_synced_env().into_iter().collect();
if let Ok(hf_token) = huggingface::resolve_token() {
env.entry("HF_TOKEN".to_string()).or_insert(hf_token);
}
if let Some(gh) = git::resolve_github_token() {
// Overrides any synced GITHUB_TOKEN: the clone URL embeds exactly this
// variable, and it must be the token the branch was pushed with.
env.insert("GITHUB_TOKEN".to_string(), gh);
}
let mut metadata = HashMap::new();
metadata.insert("or_run".to_string(), run_id.clone());
metadata.insert("or_experiment".to_string(), exp.id.clone());
metadata.insert("or_project".to_string(), project.id.clone());
let (package_digest, package_path) = source
.ray_package
.as_ref()
.ok_or_else(|| anyhow!("Ray source package was not created."))?;
let working_dir = ray::stage_working_dir(&address, package_digest, package_path).await?;
ray::run_job(
&address,
&ray::JobSubmission {
entrypoint: format!("bash -c {}", sh_quote(&script)),
entrypoint: format!("bash -c {}", sh_quote(&run_command)),
submission_id: submission_id.clone(),
resources,
env,
metadata,
working_dir: Some(working_dir),
},
)
.await?;
let watch = ray::job_url(&address, &submission_id);
let descriptor = BackendDescriptor {
let mut descriptor = BackendDescriptor {
kind: "ray_job".to_string(),
namespace: Some(address.clone()),
job_id: Some(submission_id),
job_id: Some(submission_id.clone()),
flavor: args.flavor.clone(),
image: None,
url: Some(watch),
@@ -171,7 +149,15 @@ pub async fn submit_local_ray(args: &crate::ExpRunArgs) -> Result<StoredRun> {
ssh_port: None,
ssh_user: None,
timeout_secs: None,
source_digest: None,
source_path: None,
source_size: None,
};
source.apply_to_descriptor(&mut descriptor);
if let Err(error) = crate::compute::record_submission_handle(&run_id, &descriptor) {
let _ = ray::stop_job(&address, &submission_id).await;
return Err(error);
}
let run = StoredRun {
id: run_id.clone(),
experiment_id: exp.id.clone(),
@@ -183,9 +169,11 @@ pub async fn submit_local_ray(args: &crate::ExpRunArgs) -> Result<StoredRun> {
updated_at: now_ms(),
ended_at: None,
exit_code: None,
commit_sha: Some(commit_sha),
commit_sha: Some(source.revision),
result_markdown: None,
cancel_requested: false,
cancel_requested: store
.get_run(&run_id)?
.is_some_and(|run| run.cancel_requested),
chat_session_id: crate::local::chat::launching_chat_session(),
};
store.upsert_run(&run)?;
src/local/skills.rs
+19 -21
View File
@@ -55,16 +55,13 @@ Workflow:
4. Publish: `trackio logbook publish <hf-username>/<openreview-id>` (the OpenReview ID from the paper reference above; after later edits, `trackio logbook sync`).
"#;
const ICML_REPRO_LOCAL_TEMPLATE: &str = r#"Prepare an ICML 2026 reproduction in this local-only project.
const ICML_REPRO_LOCAL_TEMPLATE: &str = r#"Prepare an ICML 2026 reproduction in this project.
Paper: {args}
The challenge workflow requires Hugging Face Jobs and publication, which are
not available while this project is local-only. Do not launch `hf jobs` or
bypass the OpenResearch compute gate. Explain that the user must explicitly
enable GitHub syncing for this project before the challenge workflow can
run. Until then, you may read the paper and outline a local experiment plan,
but do not provision external compute or publish a logbook.
Use `orx exp run --backend hf` for formal runs; source snapshots are uploaded
directly and do not require repository publication. Publishing the Trackio
logbook is a separate explicit workflow.
"#;
const REPRODUCE_PAPER_TEMPLATE: &str = r#"Reproduce a research paper claim by claim on the user's compute.
@@ -133,7 +130,7 @@ orx never binds a new experiment root to `main` — every node gets its own `orx
Reproduce the claim:
1. Create a child experiment for the selected claim.
2. Encode all parameters in committed code or configuration and keep the inherited run command unchanged.
3. Commit and push before launching; remote jobs clone the pushed branch.
3. Commit before launching; `orx` transfers an immutable source snapshot directly.
4. Launch with `orx exp run <experiment-id> --backend <backend> ...` (omit `--backend` to use the configured default target).
5. Hold the turn open with `orx exp wait` until the run is terminal, then read the evidence with `orx logs <run-id>`.
6. Record findings immediately with `orx exp desc`.
@@ -223,16 +220,17 @@ Finish by reporting:
Do not stop after creating the notebook. Finish only after the reproduction is analyzed, the notebook is validated, public links work, and provenance is recorded in the experiment tree.
"#;
const REPRODUCE_PAPER_LOCAL_TEMPLATE: &str = r#"Reproduce a research paper claim by claim in this local-only project.
const REPRODUCE_PAPER_LOCAL_TEMPLATE: &str = r#"Reproduce a research paper claim by claim in this unpublished project.
Paper and compute: {args}
Use the configured local repository and `orx` experiment tree. Read the paper,
enumerate its empirical claims, and choose the smallest honest reproduction
that fits this machine. Create experiment nodes for meaningful variants, make
each change on its printed local branch, commit it, and launch only with
`orx exp run <experiment-id> --backend local`. Keep the inherited run command
fixed. Wait with `orx exp wait --project <project-id>`, inspect `orx runs`, and
that fits the available compute. Create experiment nodes for meaningful variants,
make each change on its printed local branch, and commit it. Use the configured
default compute target, or the explicit backend and flavor/host the user chose;
source snapshots do not require GitHub. Keep the inherited run command fixed.
Wait with `orx exp wait --project <project-id>`, inspect `orx runs`, and
read every terminal run with `orx logs <run-id>`.
Record the paper number, observed number, scale or substitutions, sample size,
@@ -244,17 +242,17 @@ Produce a self-contained local report under the project's Artifacts directory,
with figures in an adjacent `images/` folder. Open with the strongest result,
separate paper evidence from observed evidence, and link experiment branches by
name without assuming hosted URLs. Do not publish, change repository visibility,
or contact a Git hosting service. If the user later wants hosted artifacts or
external compute, ask them to enable GitHub syncing for this project first.
or contact a Git hosting service unless they explicitly request publication.
"#;
const PAPER_TO_MARIMO_LOCAL_TEMPLATE: &str = r#"Reproduce a paper's main illustrative claim and create a self-contained local marimo tutorial.
Paper, compute, and preferences: {args}
This project is local-only. Read the paper, choose one illustrative empirical
claim, and use committed `orx/*` experiment branches with
`orx exp run <experiment-id> --backend local`. Keep formal evidence in run
This project is unpublished. Read the paper, choose one illustrative empirical
claim, and use committed `orx/*` experiment branches on the configured default
compute target or the explicit backend the user chose. Source snapshots do not
require GitHub. Keep formal evidence in run
logs, record conclusions with `orx exp desc`, and state every downscaling or
substitution honestly.
@@ -426,15 +424,15 @@ mod tests {
#[test]
fn local_publication_skills_keep_artifacts_local() {
let reproduce = super::expand("/reproduce-paper example", false).unwrap();
assert!(reproduce.contains("local-only"));
assert!(reproduce.contains("unpublished"));
assert!(!reproduce.contains("git push"));
assert!(!reproduce.contains("public GitHub"));
let marimo = super::expand("/paper-to-marimo example", false).unwrap();
assert!(marimo.contains("Artifacts directory"));
assert!(!marimo.contains("git push"));
let icml = super::expand("/icml-repro example", false).unwrap();
assert!(icml.contains("local-only"));
assert!(icml.contains("Do not launch `hf jobs`"));
assert!(icml.contains("source snapshots"));
assert!(!icml.contains("GitHub syncing"));
}
#[test]
src/local/slurm.rs
+33 -60
View File
@@ -7,9 +7,9 @@
use std::collections::HashMap;
use crate::commands::exp::spawn_detached_supervise;
use crate::compute::SourceSnapshot;
use crate::error::{anyhow, Result};
use crate::jobs::{huggingface, slurm, BackendDescriptor};
use crate::local::git;
use crate::store::{now_ms, Store, StoredRun};
/// CLI wrapper around `submit_local_slurm`: submit, then print the summary.
@@ -34,6 +34,14 @@ pub async fn launch_local_slurm(args: &crate::ExpRunArgs) -> Result<()> {
/// supervisor. Requires `--backend slurm`; the login node comes from
/// `--host <alias>` or the slurm settings default.
pub async fn submit_local_slurm(args: &crate::ExpRunArgs) -> Result<StoredRun> {
crate::compute::submit(args).await
}
pub async fn submit_local_slurm_with_source(
args: &crate::ExpRunArgs,
source: SourceSnapshot,
run_id: String,
) -> Result<StoredRun> {
if args.sandbox.is_some() || args.gpu.is_some() || args.cpu.is_some() {
return Err(anyhow!(
"--backend slurm runs on your own cluster; drop --gpu/--cpu/--sandbox and \
@@ -101,68 +109,23 @@ pub async fn submit_local_slurm(args: &crate::ExpRunArgs) -> Result<StoredRun> {
})?;
// One run in flight per experiment unless deliberately forced.
if !args.force {
if let Some(r) = store
.list_runs_by_experiment(&exp.id)?
.into_iter()
.find(|r| !crate::local::is_terminal(&r.status))
{
return Err(anyhow!(
"Run {} is already in flight for this experiment ({}). \
Cancel it with `orx exp cancel {}` or pass --force to launch anyway.",
r.id,
r.status,
exp.id
));
}
}
// The login node clones from GitHub, so the branch tip must exist there.
let commit_sha = {
let project = project.clone();
let branch = exp.branch_name.clone();
tokio::task::spawn_blocking(move || git::publish_branch_commit(&project, &branch))
.await
.map_err(|e| anyhow!("git task failed: {e}"))??
};
// Clone on the login node at submit time (compute nodes often lack
// internet); no apt-get fallback — nobody has root there, and preflight
// reports missing git. Unlike hf_clone_script the token must NOT ride in
// the URL: login nodes are multi-tenant (`/proc/<pid>/cmdline` is world-
// readable during the clone) and git would persist the credentialed URL
// in repo/.git/config on the shared filesystem. The inline credential
// helper reads the env at auth time instead.
let setup_script = format!(
"set -eo pipefail; \
git init -q repo; cd repo; git remote add origin {url}; \
git -c credential.helper= -c credential.helper='!f() {{ echo username=x-access-token; echo \"password=$GITHUB_TOKEN\"; }}; f' \
fetch --depth 1 origin {commit}; git checkout --detach FETCH_HEAD",
commit = crate::jobs::ssh::sh_quote(&commit_sha),
url = crate::jobs::ssh::sh_quote(&format!(
"https://github.com/{}/{}.git",
project.github_owner, project.github_repo
)),
);
// The job env: everything the user synced (API keys), plus the tokens the
// clone step expects. Exported in the setup script and job.sbatch.
// run step expects. Exported in the setup script and job.sbatch.
let mut env: HashMap<String, String> = crate::config::list_synced_env().into_iter().collect();
if let Ok(hf_token) = huggingface::resolve_token() {
env.entry("HF_TOKEN".to_string()).or_insert(hf_token);
}
if let Some(gh) = git::resolve_github_token() {
// Overrides any synced GITHUB_TOKEN (unlike HF_TOKEN's or_insert):
// the credential helper reads exactly this variable, and it must be
// the token the branch was pushed with.
env.insert("GITHUB_TOKEN".to_string(), gh);
}
let run_id = uuid::Uuid::new_v4().to_string();
crate::jobs::ssh::stage_source(
&crate::jobs::ssh::SshTarget::alias(&host),
&run_id,
&source.path,
&source.digest,
)
.await?;
let job_id = slurm::run_job(&slurm::SlurmJobSpec {
host: host.clone(),
run_id: run_id.clone(),
setup_script,
setup_script: "test -d repo".to_string(),
command: run_command.clone(),
env,
gres: args.flavor.as_deref().and_then(slurm::resolve_gres),
@@ -172,10 +135,10 @@ pub async fn submit_local_slurm(args: &crate::ExpRunArgs) -> Result<StoredRun> {
})
.await?;
let descriptor = BackendDescriptor {
let mut descriptor = BackendDescriptor {
kind: "slurm_job".to_string(),
namespace: Some(host),
job_id: Some(job_id),
namespace: Some(host.clone()),
job_id: Some(job_id.clone()),
flavor: args.flavor.clone(),
image: None,
url: None,
@@ -186,7 +149,15 @@ pub async fn submit_local_slurm(args: &crate::ExpRunArgs) -> Result<StoredRun> {
ssh_port: None,
ssh_user: None,
timeout_secs: None,
source_digest: None,
source_path: None,
source_size: None,
};
source.apply_to_descriptor(&mut descriptor);
if let Err(error) = crate::compute::record_submission_handle(&run_id, &descriptor) {
let _ = slurm::cancel_job(&host, &job_id).await;
return Err(error);
}
let run = StoredRun {
id: run_id.clone(),
experiment_id: exp.id.clone(),
@@ -198,9 +169,11 @@ pub async fn submit_local_slurm(args: &crate::ExpRunArgs) -> Result<StoredRun> {
updated_at: now_ms(),
ended_at: None,
exit_code: None,
commit_sha: Some(commit_sha),
commit_sha: Some(source.revision),
result_markdown: None,
cancel_requested: false,
cancel_requested: store
.get_run(&run_id)?
.is_some_and(|run| run.cancel_requested),
chat_session_id: crate::local::chat::launching_chat_session(),
};
store.upsert_run(&run)?;
src/local/ssh.rs
+29 -43
View File
@@ -6,10 +6,10 @@
use std::collections::HashMap;
use crate::commands::exp::{hf_clone_script, spawn_detached_supervise};
use crate::commands::exp::spawn_detached_supervise;
use crate::compute::SourceSnapshot;
use crate::error::{anyhow, Result};
use crate::jobs::{ssh, BackendDescriptor};
use crate::local::git;
use crate::store::{now_ms, Store, StoredRun};
/// CLI wrapper around `submit_local_ssh`: submit, then print the summary.
@@ -34,6 +34,14 @@ pub async fn launch_local_ssh(args: &crate::ExpRunArgs) -> Result<()> {
/// detach a supervisor. Requires `--backend ssh` and `--flavor <host>` where
/// the host is an `~/.ssh/config` alias.
pub async fn submit_local_ssh(args: &crate::ExpRunArgs) -> Result<StoredRun> {
crate::compute::submit(args).await
}
pub async fn submit_local_ssh_with_source(
args: &crate::ExpRunArgs,
source: SourceSnapshot,
run_id: String,
) -> Result<StoredRun> {
if args.sandbox.is_some() || args.gpu.is_some() || args.cpu.is_some() {
return Err(anyhow!(
"--backend ssh runs on your own box; drop --gpu/--cpu/--sandbox and pass \
@@ -81,61 +89,29 @@ pub async fn submit_local_ssh(args: &crate::ExpRunArgs) -> Result<StoredRun> {
})?;
// One run in flight per experiment unless deliberately forced.
if !args.force {
if let Some(r) = store
.list_runs_by_experiment(&exp.id)?
.into_iter()
.find(|r| !crate::local::is_terminal(&r.status))
{
return Err(anyhow!(
"Run {} is already in flight for this experiment ({}). \
Cancel it with `orx exp cancel {}` or pass --force to launch anyway.",
r.id,
r.status,
exp.id
));
}
}
// The remote clones from GitHub, so the branch tip must exist there.
let commit_sha = {
let project = project.clone();
let branch = exp.branch_name.clone();
tokio::task::spawn_blocking(move || git::publish_branch_commit(&project, &branch))
.await
.map_err(|e| anyhow!("git task failed: {e}"))??
};
let run_id = uuid::Uuid::new_v4().to_string();
let script = hf_clone_script(
&commit_sha,
&project.github_owner,
&project.github_repo,
&run_command,
);
let target = ssh::SshTarget::alias(&host);
ssh::stage_source(&target, &run_id, &source.path, &source.digest).await?;
let script = crate::compute::staged_script(&run_command);
// The remote env: everything the user synced (API keys), plus the tokens
// the clone script expects. Exported inside run.sh (written owner-only).
// the run script expects. Exported inside run.sh (written owner-only).
let mut env: HashMap<String, String> = crate::config::list_synced_env().into_iter().collect();
if let Ok(hf_token) = crate::jobs::huggingface::resolve_token() {
env.entry("HF_TOKEN".to_string()).or_insert(hf_token);
}
if let Some(gh) = git::resolve_github_token() {
env.insert("GITHUB_TOKEN".to_string(), gh);
}
let remote_dir = ssh::run_job(&ssh::SshJobSpec {
target: ssh::SshTarget::alias(&host),
target: target.clone(),
run_id: run_id.clone(),
script,
env,
})
.await?;
let descriptor = BackendDescriptor {
let mut descriptor = BackendDescriptor {
kind: "ssh_job".to_string(),
namespace: Some(host),
job_id: Some(remote_dir),
job_id: Some(remote_dir.clone()),
flavor: None,
image: None,
url: None,
@@ -146,7 +122,15 @@ pub async fn submit_local_ssh(args: &crate::ExpRunArgs) -> Result<StoredRun> {
ssh_port: None,
ssh_user: None,
timeout_secs: None,
source_digest: None,
source_path: None,
source_size: None,
};
source.apply_to_descriptor(&mut descriptor);
if let Err(error) = crate::compute::record_submission_handle(&run_id, &descriptor) {
let _ = ssh::cancel_job(&target, &remote_dir).await;
return Err(error);
}
let run = StoredRun {
id: run_id.clone(),
experiment_id: exp.id.clone(),
@@ -158,9 +142,11 @@ pub async fn submit_local_ssh(args: &crate::ExpRunArgs) -> Result<StoredRun> {
updated_at: now_ms(),
ended_at: None,
exit_code: None,
commit_sha: Some(commit_sha),
commit_sha: Some(source.revision),
result_markdown: None,
cancel_requested: false,
cancel_requested: store
.get_run(&run_id)?
.is_some_and(|run| run.cancel_requested),
chat_session_id: crate::local::chat::launching_chat_session(),
};
store.upsert_run(&run)?;
src/main.rs
+25
View File
@@ -14,6 +14,7 @@ mod browser;
#[allow(dead_code)]
mod client;
mod commands;
mod compute;
mod config;
mod error;
mod folder_picker;
@@ -177,6 +178,18 @@ enum Command {
/// an approval card and blocks until answered. Not a user command.
#[command(name = "mcp-gate", hide = true)]
McpGate,
/// Internal: detached worker for optional local-project publication.
#[command(name = "publish-branch", hide = true)]
PublishBranch(PublishBranchArgs),
}
#[derive(Args, Debug)]
struct PublishBranchArgs {
repo_path: std::path::PathBuf,
branch: String,
owner: String,
repo: String,
}
#[derive(Args, Debug)]
@@ -897,6 +910,15 @@ async fn main() {
}
return;
}
if let Command::PublishBranch(args) = &command {
if let Err(err) =
local::git::push_branch(&args.repo_path, &args.branch, &args.owner, &args.repo)
{
eprintln!("orx publish-branch: {err}");
std::process::exit(1);
}
return;
}
let warning = (!matches!(
command,
@@ -964,6 +986,7 @@ fn command_name(command: &Command) -> &'static str {
Command::Telemetry(_) => "telemetry",
Command::PlanGate => "plan-gate",
Command::McpGate => "mcp-gate",
Command::PublishBranch(_) => "publish-branch",
}
}
@@ -1019,6 +1042,7 @@ async fn dispatch(command: Command) -> error::Result<()> {
// Handled before dispatch (fast path, no telemetry/update check).
Command::PlanGate => commands::plan_gate::run().await,
Command::McpGate => commands::mcp_gate::run().await,
Command::PublishBranch(_) => unreachable!("handled before dispatch"),
}
}
@@ -1035,5 +1059,6 @@ fn command_uses_lifecycle_lock(command: &Command) -> bool {
| Command::Telemetry(_)
| Command::PlanGate
| Command::McpGate
| Command::PublishBranch(_)
)
}
src/plane/local_plane.rs
+17 -34
View File
@@ -137,12 +137,6 @@ impl ControlPlane for LocalPlane {
println!(" id: {}", project.id);
println!(" repo: {}", project.repo_path);
println!(" branch: {} (baseline)", project.baseline_branch);
if project.github_enabled() {
println!(
" GitHub: {}/{}",
project.github_owner, project.github_repo
);
}
match project
.run_command
.as_deref()
@@ -296,29 +290,9 @@ impl ControlPlane for LocalPlane {
// BEFORE the flag validations below, so e.g. `--host box1` with a default
// of `ssh` is a valid launch, and before `backend_label` is captured, so
// telemetry records the resolved backend.
let project_id = &self.experiment()?.project_id;
let github_enabled = self
.store
.get_local_project(project_id)?
.ok_or_else(|| anyhow!("Local project {project_id} not found."))?
.github_enabled();
if !github_enabled {
match args.backend.as_deref() {
None => {
args.backend = Some("local".to_string());
}
Some("local") => {}
Some(_) => {
return Err(anyhow!(
"Remote compute requires this project's GitHub repository. Enable GitHub syncing for this project, then retry."
));
}
}
} else {
crate::local::apply_compute_default(&mut args.backend, &mut args.flavor);
if args.backend.is_none() {
args.backend = Some("local".to_string());
}
crate::local::apply_compute_default(&mut args.backend, &mut args.flavor);
if args.backend.is_none() {
args.backend = Some("local".to_string());
}
if args.manifest.is_some() && args.backend.as_deref() != Some("k8s") {
return Err(anyhow!("--manifest only applies with --backend k8s."));
@@ -527,6 +501,19 @@ impl ControlPlane for LocalPlane {
run_command,
)?;
if project.github_enabled() {
if let Err(error) = crate::local::git::spawn_branch_publication(
std::path::Path::new(&project.repo_path),
&experiment.branch_name,
&project.github_owner,
&project.github_repo,
) {
eprintln!(
" warning: experiment created locally, but GitHub sync could not start: {error}"
);
}
}
println!("\u{2713} Created local {} experiment", kind);
if defaulted_to_root {
let root = parent_exp.as_ref().unwrap();
@@ -555,11 +542,7 @@ impl ControlPlane for LocalPlane {
println!(" cd {}", project.repo_path);
println!(" git checkout {}", experiment.branch_name);
println!(" # …edit, then…");
if project.github_enabled() {
println!(" git commit -am \"<msg>\" && git push");
} else {
println!(" git commit -am \"<msg>\"");
}
println!(" git commit -am \"<msg>\"");
Ok(())
}
src/plane/server_plane.rs
+15 -294
View File
@@ -17,19 +17,16 @@ use super::{
ControlPlane, CreateExperimentSpec, DescInput, LogRequest, ProjectEdit, Run, RunListing, RunLog,
};
use crate::client::{
cancel_experiment_run, create_baseline_experiment, create_child_experiment,
create_external_run, create_report, download_report_file, find_project, get_experiment,
get_project, get_report, list_experiments, list_reports, list_runs, read_run_log,
start_experiment_run, update_experiment, update_project, upload_to_presigned,
CreateBaselineExperimentBody, CreateChildBody, CreateReportBody, RunTarget,
UpdateExperimentBody, UpdateProjectBody,
cancel_experiment_run, create_baseline_experiment, create_child_experiment, create_report,
download_report_file, find_project, get_experiment, get_project, get_report, list_experiments,
list_reports, list_runs, read_run_log, start_experiment_run, update_experiment, update_project,
upload_to_presigned, CreateBaselineExperimentBody, CreateChildBody, CreateReportBody,
RunTarget, UpdateExperimentBody, UpdateProjectBody,
};
use crate::commands::experiments::print_tree;
use crate::config::Credentials;
use crate::error::{anyhow, require_credentials, Result};
use crate::jobs::{huggingface as hf, BackendDescriptor};
use crate::output::format_duration;
use crate::store::{now_ms, Store, StoredRun};
use crate::{ExpRunArgs, ReportCommand};
/// The cloud-api plane. `id` is the project/experiment/run id the command
@@ -703,8 +700,14 @@ impl ServerPlane {
));
}
match args.backend.as_deref() {
Some("hf") => return self.launch_hf(args).await,
Some("modal") => return self.launch_modal(args).await,
Some("hf" | "modal") => {
return Err(anyhow!(
"--backend {} requires a local experiment (`orx up`) so orx can stage its \
immutable source snapshot directly. Server experiments run through managed \
OpenResearch compute with --gpu/--cpu/--sandbox.",
args.backend.as_deref().unwrap_or_default()
));
}
Some("k8s") => {
return Err(anyhow!(
"--backend k8s is supported for local experiments (`orx up`) only for now."
@@ -738,9 +741,8 @@ impl ServerPlane {
}
Some(other) => {
return Err(anyhow!(
"Unknown --backend '{}'. Supported: hf (Hugging Face Jobs), \
modal (Modal serverless GPUs), k8s/ssh/slurm/ray/openresearch/local \
(local experiments only).",
"Unknown --backend '{}'. External backends are available for local \
experiments (`orx up`) only.",
other
));
}
@@ -824,287 +826,6 @@ impl ServerPlane {
Ok(())
}
/// `--backend hf` — run the experiment as a Hugging Face Job on the user's
/// own HF account. (Former `exp::launch_hf`.)
async fn launch_hf(&self, args: ExpRunArgs) -> Result<()> {
let creds = &self.creds;
if args.sandbox.is_some() || args.gpu.is_some() || args.cpu.is_some() {
return Err(anyhow!(
"--backend hf runs on Hugging Face Jobs; drop --gpu/--cpu/--sandbox \
and pass --flavor instead (e.g. --flavor a10g-small)."
));
}
let flavor = args.flavor.clone().ok_or_else(|| {
anyhow!(
"--backend hf requires --flavor: t4-small, a10g-small/large, l4x1, \
l40sx1, a100-large, h200, … (cpu-basic/cpu-upgrade for CPU). \
Priced per minute on your Hugging Face account."
)
})?;
// HF's own default is 30 minutes — a footgun for training runs, so
// default generously and let --timeout tighten it.
let timeout_seconds = match &args.timeout {
Some(t) => hf::parse_timeout(t)?,
None => 4 * 3600,
};
let token = hf::resolve_token()?;
let namespace = hf::whoami(&token).await?;
// Register first: the run must exist in the tree before compute starts,
// and the response carries the repo/branch/command orx needs to submit.
let mut descriptor = BackendDescriptor {
kind: "hf_job".to_string(),
namespace: Some(namespace.clone()),
job_id: None,
flavor: Some(flavor.clone()),
image: args.image.clone(),
url: None,
context: None,
manifest: None,
resources: None,
ssh_host: None,
ssh_port: None,
ssh_user: None,
timeout_secs: None,
};
let created =
create_external_run(creds, &args.exp_id, serde_json::to_value(&descriptor)?).await?;
let run_id = created.run.id.clone();
let image = args
.image
.clone()
.unwrap_or_else(|| crate::commands::exp::default_hf_image(&flavor));
let script = crate::commands::exp::hf_clone_script(
&created.branch_name,
&created.github_owner,
&created.github_repo,
&created.run_command,
);
let mut secrets = HashMap::new();
secrets.insert("HF_TOKEN".to_string(), token.clone());
// Clone credential precedence: explicit GITHUB_TOKEN (env, then the box's
// synced env file) overrides; otherwise the api's repo-scoped
// installation token flows automatically from the org's connected GitHub
// app — a private repo needs zero extra setup beyond having connected it.
let github_token = std::env::var("GITHUB_TOKEN")
.ok()
.filter(|t| !t.trim().is_empty())
.or_else(|| crate::config::synced_env_var("GITHUB_TOKEN"))
.or_else(|| created.github_token.clone());
if let Some(gh) = github_token {
secrets.insert("GITHUB_TOKEN".to_string(), gh);
}
let mut labels = HashMap::new();
labels.insert("or_run".to_string(), run_id.clone());
labels.insert("or_experiment".to_string(), args.exp_id.clone());
labels.insert("or_project".to_string(), created.project_id.clone());
let job = hf::run_job(
&token,
&namespace,
&hf::JobSubmission {
command: vec!["bash".to_string(), "-c".to_string(), script],
docker_image: image.clone(),
flavor: flavor.clone(),
environment: HashMap::new(),
secrets,
timeout_seconds,
labels,
},
)
.await?;
// Record the job handle: local store (the truth orx serve exposes), then
// the api mirror (display + reconciliation). Local write must not be lost
// even if the PATCH fails — supervise needs it to reattach.
descriptor.job_id = Some(job.id.clone());
descriptor.url = Some(hf::job_url(&namespace, &job.id));
descriptor.image = Some(image);
let store = Store::open()?;
store.upsert_run(&StoredRun {
id: run_id.clone(),
experiment_id: args.exp_id.clone(),
project_id: created.project_id.clone(),
status: "starting".to_string(),
backend_json: descriptor.to_json(),
command: created.run_command.clone(),
created_at: now_ms(),
updated_at: now_ms(),
ended_at: None,
exit_code: None,
commit_sha: None,
result_markdown: None,
cancel_requested: false,
chat_session_id: crate::local::chat::launching_chat_session(),
})?;
if let Err(err) = crate::client::update_external_run(
creds,
&run_id,
serde_json::json!({ "backend": serde_json::to_value(&descriptor)? }),
)
.await
{
eprintln!("warning: could not mirror the job handle to the api: {err}");
}
// Detach the supervisor: it tails logs, mirrors transitions, and uploads
// the final log. Survives this process exiting (new process group).
crate::commands::exp::spawn_detached_supervise(&run_id)?;
println!("\u{2713} Hugging Face job submitted.");
println!(" run {run_id}");
println!(" job {}/{} ({flavor})", namespace, job.id);
println!(" watch {}", descriptor.url.as_deref().unwrap_or(""));
println!(
" Follow it with `orx exp wait {}` or `orx logs {run_id}`.",
args.exp_id
);
// Key event, fired only on success. Managed run on the user's HF account.
crate::telemetry::capture_experiment_started("run", false, Some("hf"));
Ok(())
}
/// `--backend modal` — run the experiment as a Modal Sandbox on the user's
/// own Modal account. (Former `exp::launch_modal`.)
async fn launch_modal(&self, args: ExpRunArgs) -> Result<()> {
use crate::jobs::modal;
let creds = &self.creds;
if args.sandbox.is_some() || args.gpu.is_some() || args.cpu.is_some() {
return Err(anyhow!(
"--backend modal runs on Modal serverless GPUs; drop --gpu/--cpu/--sandbox \
and pass --flavor instead (e.g. --flavor a10g, --flavor a100-80gb, --flavor cpu)."
));
}
let flavor = args.flavor.clone().ok_or_else(|| {
anyhow!(
"--backend modal requires --flavor: a Modal GPU (t4, l4, a10g, a100, a100-80gb, \
l40s, h100, h200, or e.g. h100:2) — or cpu / cpu-large for CPU-only. \
Priced per second on your Modal account."
)
})?;
let resources = modal::resolve_flavor(&flavor);
// Fail before registering the run with the api if Modal isn't set up.
modal::preflight().await?;
let timeout_seconds = match &args.timeout {
Some(t) => hf::parse_timeout(t)?,
None => 4 * 3600,
};
const MODAL_APP: &str = "openresearch";
// Register first: the run must exist in the tree before compute starts,
// and the response carries the repo/branch/command orx needs to submit.
let mut descriptor = BackendDescriptor {
kind: "modal_job".to_string(),
namespace: Some(MODAL_APP.to_string()),
job_id: None,
flavor: Some(flavor.clone()),
image: args.image.clone(),
url: None,
context: None,
manifest: None,
resources: None,
ssh_host: None,
ssh_port: None,
ssh_user: None,
timeout_secs: None,
};
let created =
create_external_run(creds, &args.exp_id, serde_json::to_value(&descriptor)?).await?;
let run_id = created.run.id.clone();
let image = args
.image
.clone()
.unwrap_or_else(|| modal::default_image(resources.gpu.is_some()));
let script = crate::commands::exp::hf_clone_script(
&created.branch_name,
&created.github_owner,
&created.github_repo,
&created.run_command,
);
// Same clone-credential precedence as the HF path: explicit GITHUB_TOKEN
// (env, then the box's synced env file) overrides the api's repo-scoped
// installation token.
let mut env = HashMap::new();
if let Ok(hf_token) = hf::resolve_token() {
env.insert("HF_TOKEN".to_string(), hf_token);
}
let github_token = std::env::var("GITHUB_TOKEN")
.ok()
.filter(|t| !t.trim().is_empty())
.or_else(|| crate::config::synced_env_var("GITHUB_TOKEN"))
.or_else(|| created.github_token.clone());
if let Some(gh) = github_token {
env.insert("GITHUB_TOKEN".to_string(), gh);
}
let mut tags = HashMap::new();
tags.insert("or_run".to_string(), run_id.clone());
tags.insert("or_experiment".to_string(), args.exp_id.clone());
tags.insert("or_project".to_string(), created.project_id.clone());
let sandbox_id = modal::run_job(&modal::ModalJobSpec {
script,
image: image.clone(),
gpu: resources.gpu.clone(),
cpu: resources.cpu,
memory: resources.memory,
env,
timeout_seconds,
app: MODAL_APP.to_string(),
tags,
})
.await?;
// Record the sandbox handle: local store (the truth orx serve exposes),
// then the api mirror. The local write must not be lost even if PATCH
// fails.
descriptor.job_id = Some(sandbox_id.clone());
descriptor.image = Some(image);
let store = Store::open()?;
store.upsert_run(&StoredRun {
id: run_id.clone(),
experiment_id: args.exp_id.clone(),
project_id: created.project_id.clone(),
status: "starting".to_string(),
backend_json: descriptor.to_json(),
command: created.run_command.clone(),
created_at: now_ms(),
updated_at: now_ms(),
ended_at: None,
exit_code: None,
commit_sha: None,
result_markdown: None,
cancel_requested: false,
chat_session_id: crate::local::chat::launching_chat_session(),
})?;
if let Err(err) = crate::client::update_external_run(
creds,
&run_id,
serde_json::json!({ "backend": serde_json::to_value(&descriptor)? }),
)
.await
{
eprintln!("warning: could not mirror the sandbox handle to the api: {err}");
}
crate::commands::exp::spawn_detached_supervise(&run_id)?;
println!("\u{2713} Modal sandbox submitted.");
println!(" run {run_id}");
println!(" sandbox {sandbox_id} ({flavor})");
println!(
" Follow it with `orx exp wait {}` or `orx logs {run_id}`.",
args.exp_id
);
// Key event, fired only on success. Managed run on the user's Modal
// account.
crate::telemetry::capture_experiment_started("run", false, Some("modal"));
Ok(())
}
// --- report subcommands (former commands::report server fns) ----------
async fn report_show(&self, project_id: &str, report: &str) -> Result<()> {
src/store.rs
+9 -7
View File
@@ -263,6 +263,7 @@ impl Store {
host TEXT PRIMARY KEY,
reachable INTEGER NOT NULL,
git_found INTEGER NOT NULL,
tools_found INTEGER NOT NULL DEFAULT 0,
error TEXT,
tested_at INTEGER NOT NULL
);
@@ -292,6 +293,7 @@ impl Store {
"ALTER TABLE local_projects ADD COLUMN paper_id TEXT",
"ALTER TABLE local_projects ADD COLUMN github_sync_enabled INTEGER NOT NULL DEFAULT 1",
"ALTER TABLE local_experiments ADD COLUMN chat_session_id TEXT",
"ALTER TABLE ssh_host_tests ADD COLUMN tools_found INTEGER NOT NULL DEFAULT 0",
] {
let _ = conn.execute(ddl, []);
}
@@ -1085,14 +1087,14 @@ impl Store {
pub fn upsert_ssh_host_test(&self, t: &SshHostTest) -> Result<()> {
self.conn.execute(
"INSERT INTO ssh_host_tests (host, reachable, git_found, error, tested_at)
VALUES (?1, ?2, ?3, ?4, ?5)
"INSERT INTO ssh_host_tests (host, reachable, git_found, tools_found, error, tested_at)
VALUES (?1, ?2, 0, ?3, ?4, ?5)
ON CONFLICT(host) DO UPDATE SET
reachable = excluded.reachable,
git_found = excluded.git_found,
tools_found = excluded.tools_found,
error = excluded.error,
tested_at = excluded.tested_at",
params![t.host, t.reachable, t.git_found, t.error, t.tested_at],
params![t.host, t.reachable, t.tools_found, t.error, t.tested_at],
)?;
Ok(())
}
@@ -1100,12 +1102,12 @@ impl Store {
pub fn list_ssh_host_tests(&self) -> Result<Vec<SshHostTest>> {
let mut stmt = self
.conn
.prepare("SELECT host, reachable, git_found, error, tested_at FROM ssh_host_tests")?;
.prepare("SELECT host, reachable, tools_found, error, tested_at FROM ssh_host_tests")?;
let rows = stmt.query_map([], |row| {
Ok(SshHostTest {
host: row.get(0)?,
reachable: row.get(1)?,
git_found: row.get(2)?,
tools_found: row.get(2)?,
error: row.get(3)?,
tested_at: row.get(4)?,
})
@@ -1123,7 +1125,7 @@ pub struct SshHostTest {
#[serde(skip_serializing)]
pub host: String,
pub reachable: bool,
pub git_found: bool,
pub tools_found: bool,
pub error: Option<String>,
/// Unix millis.
pub tested_at: i64,
ui/dist/assets/index-9-xIg_xY.css
Vendored
-1
View File
File diff suppressed because one or more lines are too long
ui/dist/assets/index-HF2E1wNS.js → ui/dist/assets/index-B4SKGmyd.js
Vendored
+150 -145
View File
File diff suppressed because one or more lines are too long
ui/dist/assets/index-CDKn-8TR.css
Vendored
+1
View File
File diff suppressed because one or more lines are too long
ui/dist/index.html
Vendored
+2 -2
View File
@@ -26,8 +26,8 @@
html { background: #ffffff; }
html[data-theme="dark"] { background: #0e0c0c; }
</style>
<script type="module" crossorigin src="/assets/index-HF2E1wNS.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-9-xIg_xY.css">
<script type="module" crossorigin src="/assets/index-B4SKGmyd.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-CDKn-8TR.css">
</head>
<body>
<div
ui/src/api.ts
+11 -7
View File
@@ -135,6 +135,8 @@ export interface ProjectPathStatus {
directory: boolean | null;
empty: boolean | null;
initialized: boolean | null;
githubOwner?: string | null;
githubRepo?: string | null;
}
export const getProjectPathStatus = (path = "") => {
@@ -153,6 +155,7 @@ export interface NewProject {
cloneUrl?: string;
createFolder?: boolean;
initializeGit?: boolean;
githubSyncEnabled?: boolean;
}
export interface CreateProjectResult {
@@ -185,9 +188,10 @@ export const searchPapers = (q: string) =>
* `login` is null when there's no usable token. */
export const githubAccount = () => get<{ login: string | null }>("/api/github/account");
/** Whether the stored credentials can push to a repo. An unanswerable check
* (no token / API hiccup) reports `true`, matching the server's own fallback,
* so an outage never shows a fork choice the server wouldn't honour. */
export const githubProjectRepoPreview = (name: string) =>
get<{ repo: string }>(`/api/github/project-repo-preview?name=${encodeURIComponent(name)}`);
/** Whether the stored credentials are explicitly confirmed to push to a repo. */
export const repoAccess = (owner: string, repo: string) =>
get<{ canPush: boolean }>(
`/api/github/repo-access?owner=${encodeURIComponent(owner)}&repo=${encodeURIComponent(repo)}`,
@@ -478,13 +482,13 @@ export const getSshHosts = () =>
export interface SshPreflight {
reachable: boolean;
gitFound: boolean;
toolsFound: boolean;
error: string | null;
/** Unix millis. */
testedAt: number;
}
/** Live-test a host: reachable over ssh (BatchMode) and has `git`. */
/** Live-test a host: reachable over ssh and has bash + tar for snapshots. */
export const sshPreflight = (host: string) =>
post<SshPreflight>("/api/settings/ssh/preflight", { host });
@@ -514,12 +518,12 @@ export const saveSlurmSettings = (body: {
export interface SlurmPreflight {
reachable: boolean;
slurmFound: boolean;
gitFound: boolean;
toolsFound: boolean;
partitions: string[];
error: string | null;
}
/** Live-test a login node: reachable, Slurm CLI + git present, partitions. */
/** Live-test a login node: reachable, Slurm CLI + snapshot tools present. */
export const slurmPreflight = (host: string) =>
post<SlurmPreflight>("/api/settings/slurm/preflight", { host });
ui/src/components/NewProjectForm.tsx
+118 -6
View File
File diff suppressed because one or more lines are too long
ui/src/components/SettingsPage.tsx
+17 -41
View File
@@ -86,7 +86,7 @@ import { GitTokenForm } from "./GitTokenForm";
import { BackendBadge, BackendLogo } from "./BackendLogos";
import { ProgressBar } from "./ProgressBar";
import { StatusBadge } from "./StatusBadge";
import { BADGE_CLASS_NAME, BUTTON_CLASS_NAME, ERROR_BADGE_CLASS_NAME, ICON_BUTTON_CLASS_NAME, MONO_CLASS_NAME, PRIMARY_BUTTON_CLASS_NAME, SETTINGS_LOADING_CLASS_NAME, SMALL_BUTTON_CLASS_NAME, SMALL_PRIMARY_BUTTON_CLASS_NAME, SPINNER_CLASS_NAME, SUCCESS_BADGE_CLASS_NAME, WARNING_BADGE_CLASS_NAME } from "../styleClasses";
import { BADGE_CLASS_NAME, BUTTON_CLASS_NAME, ERROR_BADGE_CLASS_NAME, ICON_BUTTON_CLASS_NAME, MONO_CLASS_NAME, PRIMARY_BUTTON_CLASS_NAME, SETTINGS_LOADING_CLASS_NAME, SMALL_BUTTON_CLASS_NAME, SPINNER_CLASS_NAME, SUCCESS_BADGE_CLASS_NAME, WARNING_BADGE_CLASS_NAME } from "../styleClasses";
const SETTINGS_CARD_CLASS_NAME = [
"settings-card [&_>_.error]:text-accent-red [&_>_.error]:text-md",
@@ -578,8 +578,8 @@ function HostTestCell({ test }: { test: HostTest | undefined }) {
if (test === "testing") return <span className={SPINNER_CLASS_NAME} />;
const badge = !test.reachable ? (
<span className={ERROR_BADGE_CLASS_NAME} title={test.error ?? undefined}>Unreachable</span>
) : !test.gitFound ? (
<span className={ERROR_BADGE_CLASS_NAME}>No git</span>
) : !test.toolsFound ? (
<span className={ERROR_BADGE_CLASS_NAME}>Missing bash/tar</span>
) : (
<span className={SUCCESS_BADGE_CLASS_NAME}>Ready</span>
);
@@ -611,7 +611,7 @@ function SshSection() {
...t,
[host]: {
reachable: false,
gitFound: false,
toolsFound: false,
error: err instanceof Error ? err.message : String(err),
testedAt: Date.now(),
},
@@ -688,7 +688,7 @@ function SlurmTestBadge({ test }: { test: "testing" | SlurmPreflight | null }) {
</span>
);
if (!test.slurmFound) return <span className={ERROR_BADGE_CLASS_NAME}>No Slurm CLI</span>;
if (!test.gitFound) return <span className={ERROR_BADGE_CLASS_NAME}>No git</span>;
if (!test.toolsFound) return <span className={ERROR_BADGE_CLASS_NAME}>Missing bash/tar</span>;
return <span className={SUCCESS_BADGE_CLASS_NAME}>Ready</span>;
}
@@ -754,7 +754,7 @@ function SlurmSection() {
setTest({
reachable: false,
slurmFound: false,
gitFound: false,
toolsFound: false,
partitions: [],
error: err instanceof Error ? err.message : String(err),
});
@@ -1377,11 +1377,9 @@ function TargetRow({
function ComputeTab({
project,
onOpenGit,
onViewHistory,
}: {
project: Project | null;
onOpenGit: () => void;
onViewHistory: () => void;
}) {
const [settings, setSettings] = useState<ComputeSettings | null>(null);
@@ -1438,14 +1436,6 @@ function ComputeTab({
settings.configuredDefaultBackend !== null &&
settings.configuredDefaultBackend !== undefined &&
settings.configuredDefaultBackend !== "local";
const githubBlocksRemoteCompute = Boolean(
targets?.some(
(target) =>
target.id !== "local" &&
!target.enabled &&
target.disabledReason === "Connect GitHub to enable",
),
);
const renderTarget = (target: ComputeTargetSummary) => (
<TargetRow
key={`${project?.id ?? "none"}:${target.id}`}
@@ -1481,25 +1471,11 @@ function ComputeTab({
{error && <div className="error">{error}</div>}
<div className="compute-list flex flex-col gap-2.5 mb-3.5">
{targets.filter((target) => target.id === "local").map(renderTarget)}
{githubBlocksRemoteCompute && (
<div className="compute-github-gate flex items-center justify-between gap-6 mt-5.5 mx-0.5 mb-0.5 [&_h3]:m-0 [&_h3]:text-md [&_h3]:font-semibold [&_p]:mt-[3px] [&_p]:mx-0 [&_p]:mb-0 [&_p]:text-subtext [&_p]:text-sm [&_.btn]:flex-none [@media((max-width:_640px))]:items-stretch [@media((max-width:_640px))]:flex-col [@media((max-width:_640px))]:gap-3">
<div>
<h3>Remote targets</h3>
<p>
Enable GitHub syncing for this project to push experiment branches and run
them on remote compute.
</p>
</div>
<button type="button" className={SMALL_PRIMARY_BUTTON_CLASS_NAME} onClick={onOpenGit}>
Enable GitHub syncing
</button>
</div>
)}
{targets.filter((target) => target.id !== "local").map(renderTarget)}
</div>
{fallbackDefault && (
<p className={SETTINGS_NOTE_CLASS_NAME}>
Using this machine while the project is local-only. Your saved {settings?.configuredDefaultBackend} default will return after GitHub is enabled.
Using this machine because the saved {settings?.configuredDefaultBackend} default is not currently configured.
</p>
)}
<p className="compute-footnote flex items-start gap-1.5 mt-0.5 mx-0 mb-0 text-sm text-muted [&_svg]:flex-none [&_svg]:mt-px">
@@ -2056,7 +2032,8 @@ function ProjectDefaultsTab() {
<div className="project-default-title text-md font-semibold">Enable GitHub syncing for new projects</div>
<p>
When enabled, each new project gets a private GitHub repository. Experiment
branches are pushed automatically so their code can run on remote compute.
branches are pushed automatically for collaborator visibility. Compute always
uses direct source snapshots.
</p>
</div>
<button
@@ -2203,7 +2180,7 @@ function GitTab({
{!status.github.authenticated && (
<>
<p className="git-card-helper text-muted text-sm mt-3.5 mx-0 mb-0">
GitHub is optional. Connect only when you want remote compute or a hosted copy.
GitHub is optional. Connect only when you want a hosted copy for collaboration.
</p>
<GitTokenForm onSaved={() => load()} />
</>
@@ -2212,8 +2189,8 @@ function GitTab({
<>
<p className="git-card-helper text-muted text-sm mt-3.5 mx-0 mb-0">
{hasGithubRepository
? "Use this repository for automatic experiment-branch pushes when your connected account can write to it. Otherwise, OpenResearch creates a separate private repository for syncing and remote compute."
: "Create a private repository for this project and automatically push experiment branches so they can run on remote compute."}
? "Use this repository for automatic experiment-branch pushes when your connected account can write to it. Otherwise, OpenResearch creates a separate private repository for collaboration."
: "Create a private repository and automatically push experiment branches for collaborator visibility."}
</p>
<div className={GIT_CARD_ACTIONS_CLASS_NAME}>
{hasGithubRepository && status.github.url && <a className={BUTTON_CLASS_NAME} href={status.github.url} target="_blank" rel="noreferrer">Open on GitHub <ExternalLink size={12} /></a>}
@@ -2224,8 +2201,8 @@ function GitTab({
{status.github.enabled && (
<>
<p className="git-card-helper text-muted text-sm mt-3.5 mx-0 mb-0">
Disabling syncing stops automatic pushes and remote compute. It does not delete
the GitHub repository or any code already pushed there.
Disabling syncing stops automatic pushes. Compute continues to use direct source
snapshots. This does not delete the GitHub repository or code already pushed.
</p>
<div className={GIT_CARD_ACTIONS_CLASS_NAME}>
{status.github.url && <a className={BUTTON_CLASS_NAME} href={status.github.url} target="_blank" rel="noreferrer">Open on GitHub <ExternalLink size={12} /></a>}
@@ -2249,9 +2226,9 @@ function GitTab({
>
<h2 id="github-default-title">Make GitHub syncing the default?</h2>
<p>
This is useful if you expect to regularly run projects on remote compute. New
projects will enable GitHub syncing automatically, creating a private repository
when needed and pushing experiment branches for remote runs.
This is useful when collaborators follow project changes on GitHub. New projects
will enable syncing automatically, creating a private repository when needed and
pushing experiment branches for visibility.
</p>
{defaultPromptError && <div className="error">{defaultPromptError}</div>}
<div className="github-default-actions flex justify-end gap-2.5 mt-5.5">
@@ -2754,7 +2731,6 @@ export function SettingsView({
{tab === "compute" && (
<ComputeTab
project={project}
onOpenGit={() => onSelectTab("git")}
onViewHistory={() => onSelectTab("instances")}
/>
)}