mirror of
https://github.com/alphaXiv/OpenResearch.git
synced 2026-11-11 09:59:34 +08:00
Retire eager baseline: projects start with an empty experiment tree (#42)
* Retire eager baseline: projects start with an empty experiment tree
Creating/importing a project no longer auto-creates the baseline. The first
no-parent create-experiment becomes the baseline (the control); a partial
unique index (SQLite) hardens one-root-per-project against races. The hosted
client now calls POST /projects/{id}/baseline-experiment (renamed from
import-baseline; server keeps a deprecated alias) and can seed --run-command.
Playbooks, SKILL.md, --help text, and the dashboard empty states now teach the
create-the-baseline-first flow.
* Allow multiple baselines per project
Drop the one-root-per-project unique index (plus a DROP INDEX guard for dbs
that ran earlier builds of this branch) and the create_experiment pre-check.
New --baseline flag on orx create-experiment (and a baseline field on the
local HTTP body) forces another root when one already exists; plain no-parent
creates keep attaching under the oldest root. Playbooks/SKILL.md now describe
multi-root trees (each root is the control for its own subtree).
This commit is contained in:
@@ -21,12 +21,15 @@ These four govern everything below. Breaking any one silently invalidates your
|
||||
results — they are not style preferences. The detailed "experiment-tree model"
|
||||
section expands on the why; these are the non-negotiables.
|
||||
|
||||
1. **Never edit the baseline (the root).** The root is the control every variant
|
||||
is measured against. To try an idea, **branch a child** and edit the child.
|
||||
Editing the root moves the goalposts and destroys every comparison. (One
|
||||
exception, like rule 2's single legitimate `--set`: **seeding an *empty*
|
||||
baseline** with starting code when it has none yet — see "Seeding an empty
|
||||
baseline." Once the root holds real code, this rule is absolute.)
|
||||
1. **Never edit a baseline (root) once it holds real code.** A root is the
|
||||
control its variants are measured against. Projects start with an empty
|
||||
tree — **you create the baseline** (the first `orx create-experiment`, no
|
||||
`--parent`) and, on a blank repo, seed it with starting code before its first
|
||||
run (see "Seeding an empty baseline"). That setup window is the one
|
||||
exception, like rule 2's single legitimate `--set`. From the moment a root
|
||||
holds real code this rule is absolute: to try an idea, **branch a child**
|
||||
and edit the child. Editing a root moves the goalposts and destroys every
|
||||
comparison under it.
|
||||
2. **The run command *and* the environment are a fixed contract — identical on
|
||||
every node.** A child inherits its parent's run command verbatim; leave it
|
||||
alone. Do **not** give nodes different start commands, and do **not** vary
|
||||
@@ -90,7 +93,7 @@ orx logout # remove the stored token
|
||||
### Create and run experiments (write)
|
||||
| Command | What it does |
|
||||
|---|---|
|
||||
| `orx create-project <orgId> --name "<n>" [--repo <owner/repo>]` | Create a project **and its baseline (root node)**: bound to a GitHub repo, or on a fresh blank repo when `--repo` is omitted. See below. |
|
||||
| `orx create-project <orgId> --name "<n>" [--repo <owner/repo>]` | Create a project bound to a GitHub repo (or a fresh blank repo when `--repo` is omitted). Starts with an **empty experiment tree** — create the baseline next. See below. |
|
||||
| `orx project edit <projectId> [--name "<n>"] [--description "<text>" \| --description-stdin]` | Edit a project's name and/or description (pass at least one). `--description-stdin` overwrites the description from stdin (long markdown). |
|
||||
| `orx create-experiment <projectId> --title "<t>" [...]` | Add an experiment node; prints its git branch. See below. |
|
||||
| `orx compute [--gpu <id>] [--count <n>] [--provider <name>] \| --cpu]` | List the GPU compute catalog across all providers, or CPU-only offers with `--cpu` (price-sorted). See below. |
|
||||
@@ -295,9 +298,9 @@ references images by relative path (``).
|
||||
## `orx create-project` — start a new project
|
||||
|
||||
Creates a project in an organization (org ids are printed next to the org names
|
||||
in `orx projects`), then its **baseline (root node)** on the project's repo —
|
||||
one command yields a project ready to hang child experiments off. Every project
|
||||
is backed by exactly one git repo; `--repo` picks where that repo comes from:
|
||||
in `orx projects`), bound to a git repo. The project starts with an **empty
|
||||
experiment tree** — no baseline yet. Every project is backed by exactly one git
|
||||
repo; `--repo` picks where that repo comes from:
|
||||
|
||||
```sh
|
||||
# From an existing repo — yours (bound directly) or any readable repo
|
||||
@@ -309,23 +312,24 @@ orx create-project <orgId> --name "My new idea"
|
||||
```
|
||||
|
||||
- `--repo` takes `owner/repo` or a github.com URL. `--branch` (only with
|
||||
`--repo`) imports from a non-default branch. `--description` is optional.
|
||||
- The command prints the project id, repo, and the baseline's id + git branch.
|
||||
Next steps: hang children off the baseline with
|
||||
`--repo`) binds a non-default branch — the baseline will branch off it.
|
||||
`--description` is optional.
|
||||
- The command prints the project id + repo. **Next step: create the baseline**
|
||||
(the root node, the control every variant is measured against):
|
||||
`orx create-experiment <projectId> --title "Baseline"` (no `--parent`).
|
||||
Run it once for reference numbers, then hang children off it with
|
||||
`orx create-experiment <projectId> --title "<t>" --parent <baselineId>`.
|
||||
- For a **blank** project the baseline starts empty (a stub README): seed it
|
||||
from existing code before launching runs — for a paper or a known idea there is
|
||||
almost always a repo that implements it, and starting from that is faster and a
|
||||
far better control than code written from scratch. See "Seeding an empty
|
||||
baseline" below.
|
||||
- If the baseline step fails after the project was created, the command prints
|
||||
the `orx create-experiment <projectId> --title "<t>"` recovery — run that
|
||||
rather than re-running `create-project` (which would mint a second project).
|
||||
- For a **blank** project the baseline you create starts empty (a stub README):
|
||||
seed it from existing code before launching runs — for a paper or a known idea
|
||||
there is almost always a repo that implements it, and starting from that is
|
||||
faster and a far better control than code written from scratch. See "Seeding
|
||||
an empty baseline" below.
|
||||
|
||||
## Seeding an empty baseline — start from existing code, not from scratch
|
||||
|
||||
A **blank** project's baseline is an empty stub (just a `README.md`) — there's no
|
||||
code to run yet. The right move is almost never to **write the implementation by
|
||||
On a **blank** project, the baseline you create (`orx create-experiment`, no
|
||||
`--parent`) starts as an empty stub (just a `README.md`) — there's no code to
|
||||
run yet. The right move is almost never to **write the implementation by
|
||||
hand.** For nearly any paper or idea there is already a repo that implements it,
|
||||
and seeding the baseline from that repo is faster, more faithful, and a far
|
||||
better control than something typed from memory. Reproductions should start from
|
||||
@@ -397,12 +401,19 @@ orx create-experiment <projectId> --title "Larger batch size" --parent <experime
|
||||
|
||||
# Baseline (root) node on the project's bound repo:
|
||||
orx create-experiment <projectId> --title "Baseline"
|
||||
|
||||
# Additional baseline (another root) when the project already has one:
|
||||
orx create-experiment <projectId> --title "Baseline v2" --baseline
|
||||
```
|
||||
|
||||
- `--parent` selects the shape: with `--parent` ⇒ child; without it ⇒ baseline
|
||||
(root) on the repo the project is already bound to. The repo a project works on
|
||||
is chosen when the **project** is created (`orx create-project` or the web), so
|
||||
there is no `--repo` flag here.
|
||||
- `--parent` selects the shape: with `--parent` ⇒ child; without it, on an
|
||||
**empty project**, ⇒ the baseline (root) on the repo the project is already
|
||||
bound to. Once a root exists, a parentless create attaches under the oldest
|
||||
root on local projects (server projects create another baseline); pass
|
||||
`--baseline` to explicitly add another root — projects may hold multiple
|
||||
baselines, each the control for its own subtree. The repo a project works
|
||||
on is chosen when the **project** is created (`orx create-project` or the
|
||||
web), so there is no `--repo` flag here.
|
||||
- **A `--parent` child inherits the parent's run command** (and branches off its
|
||||
code). You do **not** set a run command on the child — keep it and vary the code
|
||||
on the child's git branch (see "the experiment-tree model" above).
|
||||
|
||||
+13
-11
@@ -482,15 +482,15 @@ pub struct CreateChildBody {
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct ImportBaselineBody {
|
||||
pub struct CreateBaselineExperimentBody {
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub title: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub description: Option<String>,
|
||||
/// Auto-generate suggested first experiments off the baseline. Defaults to
|
||||
/// true server-side when omitted.
|
||||
/// Run command seeded onto the baseline so it's launchable immediately.
|
||||
/// Omit to set it later (`orx exp cmd`).
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub generate_suggestions: Option<bool>,
|
||||
pub run_command: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
@@ -504,8 +504,8 @@ pub struct CreateProjectBody {
|
||||
/// project on a fresh blank repo (a stub root commit on `main`).
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub repo_full_name: Option<String>,
|
||||
/// Branch the baseline imports from (only with `repo_full_name`). Omit for
|
||||
/// the repo's default branch.
|
||||
/// Branch of the repo the project binds to (only with `repo_full_name`) —
|
||||
/// the baseline experiment branches off it. Omit for the repo's default.
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub branch: Option<String>,
|
||||
}
|
||||
@@ -877,17 +877,19 @@ pub async fn create_child_experiment(
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn import_baseline(
|
||||
pub async fn create_baseline_experiment(
|
||||
creds: &Credentials,
|
||||
project_id: &str,
|
||||
body: &ImportBaselineBody,
|
||||
body: &CreateBaselineExperimentBody,
|
||||
) -> Result<ExperimentEnvelope> {
|
||||
// Repo is bound at project creation; this just materializes the baseline on
|
||||
// it. `None` fields are omitted so the server applies its defaults.
|
||||
// Repo is bound at project creation; this materializes a baseline (root
|
||||
// node) on it. `None` fields are omitted so the server applies its
|
||||
// defaults. Repeat calls create additional roots — projects may hold
|
||||
// multiple baselines.
|
||||
let json = serde_json::to_value(body)?;
|
||||
api_post(
|
||||
creds,
|
||||
&format!("/projects/{}/import-baseline", project_id),
|
||||
&format!("/projects/{}/baseline-experiment", project_id),
|
||||
json,
|
||||
)
|
||||
.await
|
||||
|
||||
@@ -1,7 +1,13 @@
|
||||
//!
|
||||
//! Creates an experiment node. Two shapes, picked by flags:
|
||||
//! Creates an experiment node. Shapes, picked by flags:
|
||||
//! --parent <id> -> child experiment branched off that parent
|
||||
//! (no parent) -> baseline (root) experiment on the project's bound repo
|
||||
//! --baseline -> a new baseline (root), even when roots already exist —
|
||||
//! projects may hold multiple baselines
|
||||
//! (no flags) -> the oldest project root when one exists (local
|
||||
//! projects), or the baseline (root) when the tree is
|
||||
//! empty — projects start with no experiments, so the
|
||||
//! first create is the baseline: the control its
|
||||
//! variants are measured against
|
||||
//! A title is always required.
|
||||
//!
|
||||
//! Note: the repo a project works on is chosen when the PROJECT is created
|
||||
@@ -9,7 +15,8 @@
|
||||
//! The baseline is materialized on whatever repo the project is already bound to.
|
||||
|
||||
use crate::client::{
|
||||
create_child_experiment, import_baseline, CreateChildBody, Experiment, ImportBaselineBody,
|
||||
create_baseline_experiment, create_child_experiment, CreateBaselineExperimentBody,
|
||||
CreateChildBody, Experiment,
|
||||
};
|
||||
use crate::error::{anyhow, require_credentials, Result};
|
||||
use crate::store::Store;
|
||||
@@ -33,17 +40,19 @@ pub async fn run(args: crate::CreateExperimentArgs) -> Result<()> {
|
||||
&project,
|
||||
title,
|
||||
args.parent,
|
||||
args.baseline,
|
||||
args.description,
|
||||
args.run_command,
|
||||
);
|
||||
}
|
||||
|
||||
// The server create APIs carry no run command field — refuse rather than
|
||||
// silently drop it.
|
||||
if args.run_command.is_some() {
|
||||
// The server child-create API carries no run command field — refuse rather
|
||||
// than silently drop it. (The baseline create below does accept one.)
|
||||
if args.run_command.is_some() && args.parent.is_some() {
|
||||
return Err(anyhow!(
|
||||
"--run-command is supported for local projects only. For server \
|
||||
projects, set it after creation with `orx exp cmd <expId> --set '<cmd>'`."
|
||||
"--run-command is supported for local projects and server baselines \
|
||||
only. For server child experiments, set it after creation with \
|
||||
`orx exp cmd <expId> --set '<cmd>'`."
|
||||
));
|
||||
}
|
||||
|
||||
@@ -67,14 +76,15 @@ pub async fn run(args: crate::CreateExperimentArgs) -> Result<()> {
|
||||
kind = "child".to_string();
|
||||
} else {
|
||||
// Baseline on the project's already-bound GitHub repo. The server
|
||||
// branches `orx/<slug>` off the repo's default branch.
|
||||
let envelope = import_baseline(
|
||||
// branches `orx/<slug>` off the branch picked at project creation
|
||||
// (the repo's default unless one was chosen).
|
||||
let envelope = create_baseline_experiment(
|
||||
&creds,
|
||||
&args.project_id,
|
||||
&ImportBaselineBody {
|
||||
&CreateBaselineExperimentBody {
|
||||
title: Some(title),
|
||||
description,
|
||||
generate_suggestions: None,
|
||||
run_command: args.run_command,
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
@@ -102,13 +112,15 @@ pub async fn run(args: crate::CreateExperimentArgs) -> Result<()> {
|
||||
}
|
||||
|
||||
/// Local-mode create: child = branch `orx/<slug>` off the parent (pushed to
|
||||
/// origin so HF jobs can clone it). No parent = child of the project's root
|
||||
/// experiment — a second root is never created here.
|
||||
/// origin so HF jobs can clone it). No parent = child of the project's oldest
|
||||
/// root when one exists; on an empty project (or with `--baseline`) the new
|
||||
/// row becomes a baseline root. Projects may hold multiple baselines.
|
||||
fn run_local(
|
||||
store: &Store,
|
||||
project: &crate::local::model::LocalProject,
|
||||
title: String,
|
||||
parent: Option<String>,
|
||||
baseline: bool,
|
||||
description: Option<String>,
|
||||
run_command: Option<String>,
|
||||
) -> Result<()> {
|
||||
@@ -121,6 +133,7 @@ fn run_local(
|
||||
parent_id
|
||||
)
|
||||
})?),
|
||||
None if baseline => None,
|
||||
None => {
|
||||
let root = crate::local::experiments::project_root(store, &project.id)?;
|
||||
defaulted_to_root = root.is_some();
|
||||
|
||||
@@ -1,18 +1,18 @@
|
||||
//!
|
||||
//! Creates a project in an organization, then its baseline (root node) on the
|
||||
//! bound repo — one command yields a project ready to hang experiments off.
|
||||
//! Two shapes, picked by flags:
|
||||
//! Creates a project in an organization, bound to a GitHub repo. Two shapes,
|
||||
//! picked by flags:
|
||||
//! --repo <owner/repo> -> bound to that GitHub repo (your own repo, or a
|
||||
//! fresh copy when it's only readable)
|
||||
//! (no repo) -> a fresh blank repo (stub root commit on `main`),
|
||||
//! for projects that start from scratch
|
||||
//! A name is always required.
|
||||
//!
|
||||
//! The baseline import is a separate API call; if it fails after the project
|
||||
//! was created, we print the `orx create-experiment` retry instead of failing
|
||||
//! the whole command (so a re-run doesn't mint a duplicate project).
|
||||
//! The project starts with an empty experiment tree. The first experiment —
|
||||
//! created with `orx create-experiment <projectId> --title "<title>"` (no
|
||||
//! `--parent`), by you or by the agent — is the baseline: the control every
|
||||
//! variant is measured against.
|
||||
|
||||
use crate::client::{create_project, import_baseline, CreateProjectBody, ImportBaselineBody};
|
||||
use crate::client::{create_project, CreateProjectBody};
|
||||
use crate::error::{require_credentials, Result};
|
||||
|
||||
const USAGE: &str = "Usage: orx create-project <orgId> --name \"<name>\" [--repo <owner/repo|url>] [--branch <branch>] [--description \"<text>\"]";
|
||||
@@ -57,53 +57,28 @@ pub async fn run(args: crate::CreateProjectArgs) -> Result<()> {
|
||||
println!(" name: {}", project.name);
|
||||
println!(" repo: {}/{}", project.github_owner, project.github_repo);
|
||||
|
||||
// The root node: a baseline experiment on the repo we just bound.
|
||||
let baseline = import_baseline(
|
||||
&creds,
|
||||
&project.id,
|
||||
&ImportBaselineBody {
|
||||
title: None,
|
||||
description: None,
|
||||
generate_suggestions: None,
|
||||
},
|
||||
)
|
||||
.await;
|
||||
let experiment = match baseline {
|
||||
Ok(envelope) => envelope.experiment,
|
||||
Err(err) => {
|
||||
// The project exists; failing here would invite a duplicate-project
|
||||
// retry. Surface the recovery command instead.
|
||||
eprintln!();
|
||||
eprintln!(
|
||||
"Project created, but its baseline (root node) failed: {}",
|
||||
err
|
||||
);
|
||||
eprintln!("Create it with:");
|
||||
eprintln!(" orx create-experiment {} --title \"<title>\"", project.id);
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
|
||||
println!();
|
||||
println!("\u{2713} Created baseline (root node)");
|
||||
println!(" id: {}", experiment.id);
|
||||
println!(" title: {}", experiment.title);
|
||||
println!(" branch: {}", experiment.branch_name);
|
||||
println!();
|
||||
if from_repo {
|
||||
println!("Add child experiments off the baseline with:");
|
||||
println!("The project starts with an empty experiment tree. Create the baseline");
|
||||
println!("experiment (the root node — the control variants are measured against) with:");
|
||||
println!(
|
||||
" orx create-experiment {} --title \"<title>\" --parent {}",
|
||||
project.id, experiment.id
|
||||
" orx create-experiment {} --title \"Baseline\"",
|
||||
project.id
|
||||
);
|
||||
println!("or ask the agent to create and run it.");
|
||||
} else {
|
||||
println!("The baseline starts empty (a stub README). Push your starting code to it:");
|
||||
println!("The repo starts empty (a stub README on its default branch). Push your");
|
||||
println!("starting code first:");
|
||||
println!(
|
||||
" git clone https://github.com/{}/{} && git checkout {}",
|
||||
project.github_owner, project.github_repo, experiment.branch_name
|
||||
" git clone https://github.com/{}/{}",
|
||||
project.github_owner, project.github_repo
|
||||
);
|
||||
println!(" # …add code, commit, push…");
|
||||
println!("then create the baseline experiment (the root node) with:");
|
||||
println!(
|
||||
" orx create-experiment {} --title \"Baseline\"",
|
||||
project.id
|
||||
);
|
||||
println!(" # …add code, commit, then…");
|
||||
println!(" git push -u origin {}", experiment.branch_name);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
+6
-2
@@ -319,7 +319,6 @@ async fn create_project(Json(req): Json<CreateProjectReq>) -> ApiResult {
|
||||
let clone = move || {
|
||||
let store = Store::open()?;
|
||||
local::projects::create_project(&store, &name, &owner, &repo, baseline_branch, run_command)
|
||||
.map(|(project, _baseline)| project)
|
||||
};
|
||||
let mut result = tokio::task::spawn_blocking(clone.clone())
|
||||
.await
|
||||
@@ -472,6 +471,9 @@ async fn list_instances() -> ApiResult {
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct CreateExperimentReq {
|
||||
parent_experiment_id: Option<String>,
|
||||
/// Force a new baseline root even when the project already has one.
|
||||
#[serde(default)]
|
||||
baseline: bool,
|
||||
slug: Option<String>,
|
||||
title: Option<String>,
|
||||
description: Option<String>,
|
||||
@@ -494,7 +496,9 @@ async fn create_experiment(
|
||||
.get_local_experiment(pid)?
|
||||
.ok_or_else(|| not_found("parent experiment"))?,
|
||||
),
|
||||
// No parent -> the project root; never a second root.
|
||||
// `baseline` forces a new root; otherwise no parent -> the oldest
|
||||
// project root when one exists (empty project: a new baseline).
|
||||
None if req.baseline => None,
|
||||
None => local::experiments::project_root(&store, &project.id)?,
|
||||
};
|
||||
local::experiments::create_experiment(
|
||||
|
||||
@@ -32,7 +32,9 @@ fn unique_slug(store: &Store, project_id: &str, base: &str) -> Result<String> {
|
||||
}
|
||||
|
||||
/// The project's root experiment (parent NULL) — oldest first when several
|
||||
/// exist. CLI/API creates without a parent attach here instead of adding roots.
|
||||
/// exist. `None` on a fresh project: the tree starts empty and the first
|
||||
/// no-parent create becomes the baseline. CLI/API creates without a parent
|
||||
/// attach here once a root exists (pass `--baseline` to add another root).
|
||||
pub fn project_root(store: &Store, project_id: &str) -> Result<Option<LocalExperiment>> {
|
||||
// list is ordered created_at ASC, so `find` picks the oldest root.
|
||||
Ok(store
|
||||
@@ -67,8 +69,8 @@ pub fn create_experiment(
|
||||
};
|
||||
let slug = unique_slug(store, &project.id, &base)?;
|
||||
|
||||
// Git only on the parented path: a baseline row needs no branch, and
|
||||
// create_project calls this inside a store transaction (keep it network-free).
|
||||
// Git only on the parented path: a baseline row needs no branch — it rides
|
||||
// the project's baseline branch, which was validated at clone time.
|
||||
let branch_name = match parent {
|
||||
Some(p) => {
|
||||
let repo = git::ensure_clone(
|
||||
|
||||
@@ -244,9 +244,10 @@ the paths listed at the end of that output.
|
||||
|
||||
## 2. Carry out the user's research goal
|
||||
|
||||
Follow the auto-research loop from the guide: branch experiments off the baseline,
|
||||
launch runs within the user's GPU budget, wait on completions, and analyze each
|
||||
result before deciding to refill, promote, or stop.
|
||||
Follow the auto-research loop from the guide: create the baseline experiment
|
||||
first when the project is empty, branch variants off it, launch runs within the
|
||||
user's GPU budget, wait on completions, and analyze each result before deciding
|
||||
to refill, promote, or stop.
|
||||
|
||||
## Prerequisite
|
||||
|
||||
@@ -265,9 +266,9 @@ auto-research loop. It changes often, so always read it fresh rather than relyin
|
||||
on memory or a cached copy.
|
||||
|
||||
Then carry out the user's research goal, following the auto-research loop from that
|
||||
guide: branch experiments off the baseline, launch runs within the GPU budget,
|
||||
wait on completions, and analyze each result before deciding to refill, promote,
|
||||
or stop.
|
||||
guide: create the baseline experiment first when the project is empty, branch
|
||||
variants off it, launch runs within the GPU budget, wait on completions, and
|
||||
analyze each result before deciding to refill, promote, or stop.
|
||||
|
||||
If any command reports `Not logged in`, ask the user to run `orx login` first.
|
||||
|
||||
|
||||
+12
-3
@@ -133,13 +133,19 @@ only the commands listed below exist; use this project id (`{id}`) for every
|
||||
|
||||
Orient with `orx projects` and `orx runs {id}`.
|
||||
|
||||
**If the experiment tree is empty** (a fresh project), create the baseline
|
||||
first: `orx create-experiment {id} --title "Baseline"` (no `--parent`). Give it
|
||||
the run command, run it once for reference numbers, then branch children off it.
|
||||
|
||||
## Cardinal rules
|
||||
|
||||
Breaking any of these silently invalidates results — they are not style
|
||||
preferences.
|
||||
|
||||
1. **Never edit the baseline (the root experiment).** The root is the control
|
||||
every variant is measured against. To try an idea, **branch a child**
|
||||
1. **Never edit a baseline (root experiment) once it exists.** A root is the
|
||||
control its variants are measured against — on a fresh project you create
|
||||
it (first `orx create-experiment`, no `--parent`), and from then on it is
|
||||
frozen. To try an idea, **branch a child**
|
||||
(`orx create-experiment … --parent <expId>`) and edit the child's branch.
|
||||
2. **The run command and the environment are a fixed contract — identical on
|
||||
every node.** Children inherit it verbatim. If the project has no run
|
||||
@@ -160,7 +166,7 @@ preferences.
|
||||
| Command | What it does |
|
||||
|---|---|
|
||||
| `orx projects` | List projects; local ones are tagged `(local)`. |
|
||||
| `orx create-experiment {id} --title "<t>" [--description "<d>"] [--parent <expId>] [--run-command "<cmd>"]` | New node, branched `orx/<slug>` off the parent's tip (project root when `--parent` is omitted) and pushed to GitHub. |
|
||||
| `orx create-experiment {id} --title "<t>" [--description "<d>"] [--parent <expId> \| --baseline] [--run-command "<cmd>"]` | New node, branched `orx/<slug>` off the parent's tip and pushed to GitHub. Omit `--parent`: attaches under the oldest project root — or, on an empty project, becomes the baseline root itself. `--baseline` forces another root (multiple baselines are allowed). |
|
||||
| `orx project view {id}` / `orx project edit {id} --run-command "<cmd>"` | Inspect the project / set its default run command. |
|
||||
| `orx exp status <expId>` | Node's branch, command, and latest run. |
|
||||
| `orx exp desc <expId> [--set "<text>" \| --stdin]` | Read/overwrite the node's notes. Record findings here. |
|
||||
@@ -183,6 +189,9 @@ they hit public hosts and need no login.
|
||||
|
||||
Carry one goal across many runs:
|
||||
|
||||
0. **Round 0 — the baseline** (empty project only): `orx create-experiment {id}
|
||||
--title "Baseline"` (no `--parent`), set the run command, launch it once.
|
||||
Its numbers are the reference every variant is judged against.
|
||||
1. **Branch**: `orx create-experiment {id} --title "<idea>" --parent <parentId>`
|
||||
— one child per distinct thing you try.
|
||||
2. **Edit** in this clone: `git fetch origin && git checkout <branch>`, change
|
||||
|
||||
+13
-21
@@ -1,13 +1,17 @@
|
||||
//! Local project creation — clone the repo, insert the project row and its
|
||||
//! baseline root experiment. Used by the `orx up` HTTP API (`POST /api/projects`).
|
||||
//! Local project creation — clone the repo and insert the project row. Used by
|
||||
//! the `orx up` HTTP API (`POST /api/projects`).
|
||||
//!
|
||||
//! The project starts with an empty experiment tree. The first experiment
|
||||
//! created without a parent (via `orx create-experiment` or the HTTP API)
|
||||
//! becomes the baseline root — the control every variant is measured against.
|
||||
|
||||
use std::collections::HashSet;
|
||||
|
||||
use crate::error::Result;
|
||||
use crate::store::{now_ms, Store};
|
||||
|
||||
use super::model::{LocalExperiment, LocalProject};
|
||||
use super::{experiments, git, slugify};
|
||||
use super::model::LocalProject;
|
||||
use super::{git, slugify};
|
||||
|
||||
fn unique_project_slug(store: &Store, base: &str) -> Result<String> {
|
||||
let taken: HashSet<String> = store
|
||||
@@ -28,8 +32,9 @@ fn unique_project_slug(store: &Store, base: &str) -> Result<String> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Clone the repo and create the project plus its root "baseline" experiment
|
||||
/// (parent NULL, branch = baseline branch, command = the project default).
|
||||
/// Clone the repo and create the project row. No experiments are created —
|
||||
/// the tree starts empty and the baseline is created lazily (first no-parent
|
||||
/// `create_experiment`).
|
||||
pub fn create_project(
|
||||
store: &Store,
|
||||
name: &str,
|
||||
@@ -37,7 +42,7 @@ pub fn create_project(
|
||||
github_repo: &str,
|
||||
baseline_branch: Option<String>,
|
||||
run_command: Option<String>,
|
||||
) -> Result<(LocalProject, LocalExperiment)> {
|
||||
) -> Result<LocalProject> {
|
||||
let baseline_branch = baseline_branch
|
||||
.filter(|b| !b.trim().is_empty())
|
||||
.unwrap_or_else(|| "main".to_string());
|
||||
@@ -57,19 +62,6 @@ pub fn create_project(
|
||||
created_at: now,
|
||||
updated_at: now,
|
||||
};
|
||||
// One transaction: a project without its baseline root must never persist
|
||||
// (a failed baseline insert would orphan the project and burn its slug).
|
||||
let tx = store.begin()?;
|
||||
store.create_local_project(&project)?;
|
||||
let baseline = experiments::create_experiment(
|
||||
store,
|
||||
&project,
|
||||
None,
|
||||
Some("baseline"),
|
||||
Some("Baseline".to_string()),
|
||||
None,
|
||||
project.run_command.clone(),
|
||||
)?;
|
||||
tx.commit()?;
|
||||
Ok((project, baseline))
|
||||
Ok(project)
|
||||
}
|
||||
|
||||
+11
-6
@@ -308,8 +308,8 @@ pub struct CreateProjectArgs {
|
||||
/// Omit to start the project on a fresh blank repo.
|
||||
#[arg(long)]
|
||||
pub repo: Option<String>,
|
||||
/// Branch the baseline imports from (with `--repo`; defaults to the repo's
|
||||
/// default branch).
|
||||
/// Branch of the repo the project binds to (with `--repo`; defaults to the
|
||||
/// repo's default branch). The baseline experiment branches off it.
|
||||
#[arg(long)]
|
||||
pub branch: Option<String>,
|
||||
/// Project description.
|
||||
@@ -326,12 +326,17 @@ pub struct CreateExperimentArgs {
|
||||
/// Experiment description.
|
||||
#[arg(long)]
|
||||
pub description: Option<String>,
|
||||
/// Parent experiment id -> create a child. Omit to create a baseline on the
|
||||
/// project's bound repo (local projects: a child of the project root).
|
||||
/// Parent experiment id -> create a child. Omit on an empty project to
|
||||
/// create the baseline (root); once a root exists, local projects attach
|
||||
/// under the oldest root (server projects create another baseline).
|
||||
#[arg(long)]
|
||||
pub parent: Option<String>,
|
||||
/// Run command for the node (local projects only). Omit to inherit from
|
||||
/// the parent / project default.
|
||||
/// Create a new baseline (root) even when the project already has one.
|
||||
/// Conflicts with --parent. Projects may hold multiple baselines.
|
||||
#[arg(long, conflicts_with = "parent")]
|
||||
pub baseline: bool,
|
||||
/// Run command for the node (local projects and server baselines). Omit to
|
||||
/// inherit from the parent / project default.
|
||||
#[arg(long = "run-command")]
|
||||
pub run_command: Option<String>,
|
||||
}
|
||||
|
||||
@@ -160,6 +160,12 @@ impl Store {
|
||||
] {
|
||||
let _ = conn.execute(ddl, []);
|
||||
}
|
||||
// Older builds of this branch created a one-root-per-project unique
|
||||
// index; multiple baselines are allowed, so make sure it's gone.
|
||||
let _ = conn.execute(
|
||||
"DROP INDEX IF EXISTS uidx_local_experiments_project_baseline",
|
||||
[],
|
||||
);
|
||||
// Data migration: the chat_sessions.permission_mode wire ids were
|
||||
// neutralized off Claude Code's `--permission-mode` spelling (`default`,
|
||||
// `acceptEdits`, `bypassPermissions`) onto harness-agnostic ids (`ask`,
|
||||
|
||||
File diff suppressed because one or more lines are too long
Vendored
+1
-1
@@ -8,7 +8,7 @@
|
||||
html { background: #ffffff; }
|
||||
@media (prefers-color-scheme: dark) { html { background: #0e0c0c; } }
|
||||
</style>
|
||||
<script type="module" crossorigin src="/assets/index-kEYVZ_Rj.js"></script>
|
||||
<script type="module" crossorigin src="/assets/index-DTKGATIK.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-BeHHFtbu.css">
|
||||
</head>
|
||||
<body>
|
||||
|
||||
+6
-2
@@ -122,8 +122,12 @@ export const listInstances = () =>
|
||||
get<{ instances: Instance[] }>("/api/instances").then((r) => r.instances);
|
||||
|
||||
export interface NewExperiment {
|
||||
parentExperimentId: string;
|
||||
slug: string;
|
||||
/** Omit on an empty project to create the baseline root; once a root
|
||||
* exists, an omitted parent attaches the node under the oldest root. */
|
||||
parentExperimentId?: string;
|
||||
/** Force a new baseline root even when the project already has one. */
|
||||
baseline?: boolean;
|
||||
slug?: string;
|
||||
title?: string;
|
||||
description?: string;
|
||||
runCommand?: string;
|
||||
|
||||
@@ -870,7 +870,10 @@ export function ChatPanel({
|
||||
<h2>
|
||||
Open<span>Research</span>
|
||||
</h2>
|
||||
<p>Ask the agent to explore your codebase, branch experiments, and launch runs.</p>
|
||||
<p>
|
||||
Ask the agent to explore your codebase, create and run your baseline experiment, and
|
||||
branch variants off it.
|
||||
</p>
|
||||
<EmptyStateAgentHint
|
||||
harnesses={harnesses}
|
||||
selection={selection ?? defaultSelection(harnesses)}
|
||||
|
||||
@@ -152,7 +152,7 @@ export function NewProjectForm({
|
||||
/>
|
||||
</label>
|
||||
<label>
|
||||
Baseline branch
|
||||
Branch
|
||||
<input
|
||||
value={branch}
|
||||
onChange={(e) => setBranch(e.target.value)}
|
||||
|
||||
@@ -205,7 +205,7 @@ export function TreeView({
|
||||
if (experiments.length === 0) {
|
||||
return (
|
||||
<div className="empty-state">
|
||||
<p>No experiments yet — ask the agent to branch one from the baseline.</p>
|
||||
<p>No experiments yet — ask the agent in chat to create and run your baseline experiment.</p>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user