mirror of
https://github.com/alphaXiv/OpenResearch.git
synced 2026-11-13 10:19:05 +08:00
Daniel Kim
c09ccb6808
* Repair a provisional experiment instead of branching a child
An agent that hit a dependency error on its root experiment created a
child node to fix it, then another for the next error — a 14-node chain of
pure engineering remediation with no scientific content. The cause was a
missing affordance, not a missing rule: rule 1 forbade editing a baseline,
the loop offered only refill/promote/stop, and `orx create-experiment` was
the only exit from a failed state.
A node is now an evidence contract, not a code revision. It stays
provisional — editable on its own branch — until one of its runs puts the
intended measurement in the log, and only then freezes. Getting code to
execute is setup, and setup happens on the node.
- SYSTEM_PROMPT.md (+5 net lines): rule 1 freezes on measurement and covers
every node, not just the baseline (the chain reproduced at depth 3 as
easily as at the root); rule 6 inherits that freeze test instead of
restating "completed non-failed run", a trigger a `done`-but-diagnostic
run satisfied with zero evidence; the loop gains a fourth move, Repair;
the 2-strike env rule becomes the named repair cap and closes the dodge
of resetting it by creating a node.
- orx-experiment-tree (both variants): the freeze test, "remediation is
neither width nor depth", and a classification table.
- orx-git: repair-in-place workflow, worktree branch-ownership handling,
and a safe sync recipe — `checkout -B <b> origin/<b>` silently discards
unpushed commits, which a repair flow hits when a push has failed.
- orx-compute (both) + k8s: provisioning failures never create nodes.
- orx-create and root SKILL.md: drop "frozen the moment it holds real
code", which contradicted the new model.
Also fixes `orx skill <name>` serving cloud bodies inside local sessions.
The playbook points there as the fallback, but `find` searched the Full set
only, so a local agent was told to use `orx artifact`/`query`/`chart` —
commands local mode does not have. It now resolves against the set implied
by ORX_CHAT_SESSION_ID.
* Review round 1: close repair loopholes, fix the -B hazard
Four independent reviewers found 15 blockers. The two that mattered most
were mine:
- The old `checkout -B <b> origin/<b>` code fences survived in orx-git and
orx-create while a new bullet warned against that exact command eight
lines below. An agent copies the fence, not the prose — so the hazard was
documented, not fixed. All fences now use `checkout` + `merge --ff-only`,
which fails loudly instead of discarding unpushed commits.
- The "safe form" I had added (`checkout X 2>/dev/null || checkout -b X`)
was itself wrong: plain `checkout` already DWIM-creates the tracking
branch, so the fallback only fired on the worktree-lock error, where the
redirect masked "already checked out at ..." as "already exists" and made
the triage block unreachable.
Also:
- orx-git:32 still said "Edit children, never the baseline" — the sentence
this change exists to remove, in the skill an agent loads to repair.
- The cap read "two consecutive ... the same error twice", so a sequence of
*distinct* errors (the actual 14-node case) never tripped it. It now says
different errors still count, and that flavor/provider/backend switches
are repairs — only a measured run resets the count.
- Freezing's permanence lived only in the on-demand skill; it is now in both
always-on rule 1 copies, where the anti-tampering defence belongs.
- Dropped "a wrong flag" from the Repair list: it licensed in-place knob
edits against cardinal rules 2 and 3.
- k8s said a GPU-type change is "a real experimental variable" while the
tree table called flavor changes "provisioning, not science". Reconciled
on the freeze boundary.
- Partial measurement (3 of 5 metrics) was undefined; any real measurement
now freezes, and the node's expected metrics belong in `orx exp desc`.
- Harness shims still enumerated the three-move loop.
Rust: module doc described the old always-Full lookup; the variant test
pinned an English sentence from a file this branch doesn't touch (replaced
with a dispatch check over all four swapping skills); the mode inference
moved to `chat::in_local_session()` next to the env contract it reads, with
a direct test — it was the crux of the fix and untested.
* Review round 2: make the freeze test evidential, not causal
Round 2's four reviewers found 10 blockers. The most serious was a
regression my round-1 compression introduced.
The classification row read "anything that stops the code MEASURING at all
-> Repair". That is a causal test where an evidential one was needed: a run
that diverges to NaN, OOMs on the node whose hypothesis IS "wider MLP", or
times out on the slower config never reaches its eval block — so an agent
would repair the branch and silently delete a real negative result. That is
worse than the bug this branch fixes, because a wrong repair destroys
evidence irrecoverably while a wrong child costs one cheap node.
The freeze test now asks whether the failure came from the node's own
change, with a mechanical discriminator: would this have happened on the
parent too? If no, it is a result — freeze it.
Two reviewers disagreed on partial metrics (3 of 5 printed): one wanted
FROZEN kept, one wanted provisional because it contradicted the definition
above it. Resolved on the same asymmetry — evidence-valid now reads "any of
the numbers this node owes", so partial output freezes and the definition
carries it instead of a patch line below the table.
Also:
- `merge --ff-only` had been added only to orx-git; the orx-create seed
recipe and both experiment-tree child recipes kept a bare `checkout`,
which silently resolves to a stale local branch on a reused clone — and
the seed recipe then rm -rf's the tree and pushes from it. Fixed at all
four sites. Flagged independently by three of the four reviewers.
- src/local/skills.rs:117, the live `/reproduce` slash template, still said
"never mutate an experiment root after it has been run" — the old model,
in a Rust string my markdown-only grep gate never covered. The gate now
spans src/ too.
- Both orx-compute bodies had zero mentions of frozen/provisional, so an
agent loading them after a failure got an unconditional repair with the
freeze concept out of scope.
- orx-compute-k8s cited "the repair cap" without ever stating its value.
- The cap lacked "on the same node" in compute and create, reading as a
global counter.
- Root SKILL.md had neither the four moves nor the cap.
- in_local_session's doc asserted "only local projects can own a chat
session" as fact; there is no FK enforcing it and it is not the operative
reason. Re-grounded on the actual mechanism.
- The variant test's ptr::eq loop was redundant with the assert_ne above
it; it now pins each body against the embedded const, outside the lookup
under test.
* Review round 3: stop classifying OOM both ways
Round 3's four reviewers found 17 blockers, converging hard on one: my
round-2 freeze test named "OOM on the bigger model" as the canonical FROZEN
case, while six other sites still listed a bare "OOM" as a repair trigger —
including the Repair bullet 190 lines below it in the same file, the
always-on prompt, and both compute bodies. The compute skill is the one an
agent loads when a job OOMs, and it said "OOM or timeout -> bigger flavor,
same node". So a node whose whole hypothesis was "the bigger model" would
have its result repaired away by weight of repetition. Every bare OOM is now
qualified by whether the node's own change caused it.
Round 2 also overshot on partial output. "Any of the numbers this node owes"
froze a node that printed `step 0 | loss 11.83` and then died on a missing
path — and since a child whose purpose is "finally find the checkpoint" is
explicitly not a node, the agent was deadlocked: forbidden to repair,
forbidden to branch. Evidence-valid is now floored on the metric the node's
`orx exp desc` names; a partial trace is not the measurement.
The counterfactual ("would this have happened on the parent too?") was also
undecidable as written — a claim about a run that never happened, with the
burden falling on proving the parent would have failed, which pushes
uncertainty toward Repair. It now points at the parent's actual run log and
states the tie-break explicitly: uncertainty freezes, because freezing
wrongly costs one node while repairing wrongly destroys a measurement.
Also:
- SYSTEM_PROMPT rule 1 and the freeze table both said an all-failed node is
provisional, unconditionally — contradicting the freeze test two lines
later. The always-on copy matters most: an agent that never loads the
skill only ever saw the permissive rule.
- orx-git:37 still said "Always fetch + reset before editing", the exact
idiom the -B removal was meant to eliminate, 19 lines above a bullet
forbidding it.
- The repair recipe itself — the flow explicitly about returning to a branch
you already left — was the one checkout site still missing
`merge --ff-only`. I had claimed all four were fixed; this was the fourth.
- legacy_root_warning used "frozen" for branch immutability, now a term of
art meaning "has produced evidence". Reworded to "immutable".
* Review round 4: a typo is not a hypothesis
Round 4's four reviewers found 17 blockers. The important one is a hole in
the model itself, not a stray sentence.
An agent's own coding bug had no classification. A NameError or a shape
mismatch in the hypothesis edit is literally "caused by the change this node
makes", so round 3's causal clause froze it — and the child inherits the same
typo, and the same reasoning freezes the child. That is the 14-node chain
reintroduced under a different label, and the "freezing wrongly costs one
extra node" accounting was wrong for exactly this case: the cost is a chain.
The freeze test now separates the change's semantics from its spelling —
freeze only when a *correct* implementation of the node's idea would have
failed the same way.
The freeze test also rested on a field nothing produced. Evidence-valid is
floored on "the metric its `orx exp desc` names", but every instruction and
all four worked examples told the agent to write only the *change*, and
orx-create called the description optional. Step 3, the examples, and
orx-create now carry the measurement.
Also:
- `cancelled` was classified two ways: a run cancelled after the metric
landed matched both the provisional row and the frozen row. Keyed on
content now, not status.
- A run with no usable log at all (spin-up failure, preemption, empty
output) fell through "uncertainty freezes" and froze, contradicting
orx-compute's transient-retry guidance. It is now explicitly not a result.
- The tie-break told the agent to read the parent's run — but a root has no
parent, so a baseline that OOMs on its first launch (which orx-create says
to *expect* and repair) would have frozen permanently.
- A seventh unqualified OOM survived round 3's sweep, in orx-compute's
sizing section: "OOM or hopelessly-slow -> move up a tier. That's expected,
not a mistake" — an unconditional imperative to do the thing the freeze
test forbids, 35 lines below the qualified bullet, in the file an agent
loads when a job OOMs.
- Root SKILL.md rule 1 never got the "own change" qualifier round 3 added to
SYSTEM_PROMPT.md rule 1 — the same defect in the same slot on the cloud
side.
- skills.rs:113 still said "Immutable `orx/*` branches hold experiment
nodes", contradicting line 117 four lines below it.
- `--baseline` was an unguarded cap-evasion route; the cap now says a second
baseline is a repair wearing a root's hat.
- The cap terminated in "ask the user" with no unattended fallback. The
agent now records the diagnosis and carries on with other nodes.
* Review round 5: bound the counterfactual to the diff, not the idea
Round 5's four reviewers found 17 blockers. The decisive one: my round-4
defect carve-out reopened round 3's bug through a new door.
The carve-out asked whether "a correct implementation of the same idea"
would have failed. But an idea is a phrase the agent itself wrote, so the
agent controls the abstraction level it reasons against — and for any
resource failure the natural reading favours repair: "a correct
implementation of 'wider MLP' wouldn't have OOMed, it would use gradient
checkpointing." Rounds 1-3 built an evidential wall; round 4 punched a
narrative door through it, and one narrative hatch outweighs four evidential
ones because an agent that wants to keep working needs only one.
The counterfactual is now over **the diff actually written**, not the idea:
ask "is this diff malformed?", never "could a better version have avoided
it?" Resource and dynamics failures — OOM, timeout, NaN, divergence — are
explicitly never defects. A kwarg the callee *rejects* is a typo; a kwarg
value you regret is the hypothesis. A shape mismatch inside the edited lines
is a typo; one that collides with code you didn't touch is the finding that
the idea doesn't fit.
Also:
- The carve-out lived only in the on-demand skill, so it was off by default
for any agent that hadn't loaded it. Added to the always-on Repair move.
- SYSTEM_PROMPT step 1 (the branch step the agent takes most often) created
nodes with no --description at all — producing exactly the un-judgeable
node the freeze test can't classify. Round 4 fixed the skills and missed
the prompt.
- No fallback existed for a node created without a named metric; now infer
from title and siblings, write it, and freeze if it still can't be pinned.
- orx-create's seed recipe had no `set -e`: after this branch replaced the
force-creating `checkout -B` with a fallible `checkout` + `merge --ff-only`,
a failed sync fell through to `rm -rf` + `cp -a` + push on whatever branch
the clone was on — usually the project's README/notebook surface. The
hardening introduced the hazard; it is now guarded.
- k8s never classified a provisional-phase GPU-type or node-count change.
Cuts (quality review): the permanence paragraph stated one rule four times;
orx-compute cross-referenced the freeze test twice in four lines; root
SKILL.md inlined the whole taxonomy it tells the reader to go read; the
always-turn cap paragraph and rule 1 carried gloss the skills already own.
Retired `execution-invalid` (coined once, never reused) and reworded
"nothing ran" for truncated output, which is factually wrong.
The env test now restores via a Drop guard, matching telemetry's EnvGuard,
so a failing assert can't leak the mutated var into the test binary.
* Review round 6: judge the evidence, not the cause
Round 6's four reviewers found 15 blockers, and one of them named the reason
this section has been rewritten in every single round: the test was
adjudicating **causation**. "Did this node's own change cause the failure?"
is not answerable from a log, so each round swung between too-permissive and
too-restrictive, and each swing needed a new qualifier to patch the last.
Round 5's swing was too restrictive. Bounding defects to a closed list of
four syntactic symptoms left every *semantic* self-inflicted defect
unclassified — an added import needing an incompatible torch, a config key
the edit references but never adds, a wrong path inside the edited code.
Those fall through "uncertainty freezes" and freeze; a child to fix them is
"not a node"; the agent is trapped. That is round 4's deadlock re-entering
through round 5's door. Meanwhile "resource and dynamics failures are never
defects" was written as an unconditional universal, so a node that changed
only an LR and OOMed on a pre-existing leak also froze — contradicting
orx-compute's own instruction to retry exactly that.
The freeze test now asks what the log **holds**, not what caused it:
- numbers (the measurement, or a NaN/OOM/timeout of what this node changed)
-> FROZEN, that is the answer
- an error and no numbers -> provisional, repair in place
- nothing to judge -> provisional, repair or relaunch
Causes are arguable; evidence is on disk. This collapses six contradictory
sites into one rule and kills the "a correct implementation wouldn't have
OOMed" argument structurally rather than by prohibition — the question is
what the log contains, so there is nothing to argue about.
Also fixed this round:
- The cap reset on every sibling, so one setup bug could burn 2N runs across
a round without ever asking. Added a project-wide backstop.
- A bare relaunch didn't obviously count toward the cap, leaving an
unbounded retry loop on infrastructure flakiness.
- "A mis-sized flavor" sat unqualified one row above the OOM row — a
standing licence to reclassify any OOM as a sizing problem.
- The round-5 warning I added to the seed recipe said "the lines below wipe
the tree"; they are above it. Folded into the `set -e` comment where it
travels with the code.
- Root SKILL.md had lost its freeze-side clause in round 5's cuts, leaving
that document with a repair trigger and no freeze rule.
- The "freezing wrongly costs one extra node" accounting was still wrong
(the cost is a chain when a child inherits the defect) and had become
load-bearing again.
- Deduped the classify table against the freeze table; trimmed k8s paste.
* Review round 7: delete the causal predicate, don't reword it
Round 7's four reviewers found 17 blockers and one of them landed the
diagnosis that matters: the oscillation was not resolved by round 6, it was
relocated. The freeze test's numbers bucket still read "a quantitative
failure of *the thing this node changed*" — so an OOM landed in FROZEN or
Repair on exactly the causal judgement round 6 declared unanswerable, and
the compute skills' complementary clause ("OOM the node's own change didn't
cause -> bigger flavor, same node") made the pair a coin flip. Round 6 said
the argument was unavailable, then handed it back two lines up.
The causal predicate is now deleted rather than reworded, at all six sites.
Numbers freeze a node **whatever ended the run**; the only question is
whether the run got far enough to emit them. "Numbers" is defined as the
run's own measurement stream — byte counts inside an allocator error are not
numbers, digits in an error string are part of the error. That was the other
half of the problem: every CUDA OOM message carries a watermark, so on a
literal reading every OOM froze.
Two escape hatches this opened, both closed:
- The cap creates a standing incentive to declare "the OOM is numbers" to
escape a hard debug and avoid asking the user. Named explicitly: if that
reading only became attractive after a repair, it is the wrong reading.
- "Not the measurement I owed" let an agent repair a node that produced real
metrics. Numbers that aren't the ones you named still freeze it; a missing
eval block is a reporting defect, fixed on a child.
Also:
- orx-evidence — the skill whose entire job is judging a finished run, loaded
at exactly the repair-vs-child decision point — had no mention of the rule,
not even a cross-reference. Added to both variants.
- orx-create's copy-paste examples omitted --description entirely, so the doc
that most often creates nodes created unjudgeable ones; its own rule sat 25
lines below the code an agent copies.
- The "nothing to judge" class existed in one skill pair only. A preempted
run had no documented classification for an agent on the always-on docs.
Added to SYSTEM_PROMPT and root SKILL.md.
- SYSTEM_PROMPT step 0 (baseline creation) named no metric and implied the
first launch succeeds — the local-mode gap, since orx-create is Full-only.
- Dropped the freeze table: after round 6 it restated the prose buckets
almost verbatim. Shipping both was the restructure half-applied.
* Review round 8: one bright line instead of two soft ones
Round 8's four reviewers found 9 blockers, converging from two directions on
the same root problem: the rule still had two discretionary predicates, and
each was one paraphrase away from the causation test round 7 deleted.
"the **intended** numbers" (cardinal rule 1 in both top-level docs, the
always-on prompt, orx-git, and a Rust prompt string) restores exactly the
adjudication the tree skill kills — were these the numbers I meant? no ->
repair. "any **substantive** metric" (root SKILL.md and both orx-evidence
variants, the skill loaded at the moment a log is judged) invites the mirror
abuse: rule real metrics insubstantial and repair a node that produced
results. The tree skill neutralised both locally; none of that reached the
five other sites.
Both are now one bright line, stated identically at all six: **two or more
metric lines this run computed**. One banner, one config echo, or a single
step-0 line is not numbers; two logged intervals are, whatever they say. The
agent cannot tune that boundary after reading the log.
That also fixed a hole round 7 opened: "a throughput line" counted as
numbers, and a tqdm bar emits `3.6it/s` — so a *preempted* run froze
permanently on a progress bar, which bucket 3 is supposed to route to
repair. Progress bars, ETAs, and figures carried over from a resumed
checkpoint are now explicitly excluded, along with allocator byte counts.
Deleted the partial-trace carve-out: it contradicted the bullet three lines
above it ("losses past warmup" freeze vs "a warmup value" doesn't), left
"warmup" undefined, and was the last place the agent picked a boundary
post-hoc.
Also:
- The over-freeze is now acknowledged rather than left implicit. A node that
OOMed on a pre-existing leak after emitting losses is frozen even though
its change was innocent; saying so stops an agent reading it as an
oversight to reason around.
- Stated the asymmetry that seven rounds kept re-deriving: a spurious freeze
costs one node, a spurious repair rewrites the branch a recorded
commit_sha points at. When ambiguous, freeze.
- "If the only available repair would change what the node is testing, it is
not a repair" — a 2x-wider-MLP node that OOMs at allocation cannot be
fixed by shrinking the width.
- The hard-debug guard asked the agent to date its own beliefs. Now
mechanical and two-directional: the count of metric lines does not change
between readings.
- The cap's reset clause was dead text (measuring something freezes the
node, so repair ends there anyway).
- k8s said a GPU-type change is a repair while cardinal rule 2 calls the
environment a fixed contract; reconciled.
* Review round 9: kind, not count — one metric freezes
Round 9's reviewers found that round 8's bright line broke the normal
success path, and both correctness reviewers landed on it independently.
"Two or more metric lines" made a clean eval that prints exactly one
`accuracy: 0.83` **provisional and editable in place** — as would a single
summary block carrying five metrics, or a run that completed one epoch.
Those are successful, complete runs. Worse, orx-evidence explicitly tells
agents to "print final metrics and a compact summary block at the end,
not just scattered mid-training lines" — so following the repo's own advice
produced exactly the log shape the freeze test then refused to protect. The
better a run's output hygiene, the less protected it was.
That inverted the whole risk gradient: a crashed run emitting two loss lines
froze, while a clean run emitting one final accuracy did not. It was also
the first version in nine rounds where a run that fully answered its node's
question could be repaired — the one harm the asymmetry paragraph exists to
prevent, contradicted by the rule it was attached to.
The threshold is now **one**, and the discrimination moved to kind: any
metric the run computed freezes the node, one line or many. The exclusion
list carries the weight — banners, config echoes, progress bars and it/s,
allocator byte counts, timeout durations, resumed-checkpoint figures are
never numbers however often they appear. That also dissolves three ways the
count was gameable (summary block vs lines, redrawn tqdm lines in the byte
stream, PTY wrapping) and retires a false axiom that claimed line counts
don't vary between readings — they do.
Also:
- "If the only available repair would change what the node is testing" was a
general-purpose freeze excuse: pinning a dep, fixing a driver, or
correcting a shape error all "change what it tests", so an agent could
freeze any node it didn't want to debug and branch a child — the 14-node
chain rebuilt with each link relabelled "frozen". It now applies only to a
node the freeze test already froze.
- Reconciled the over-freeze paragraph with "remediation is neither width
nor depth": a confound-fix child is depth only because the parent holds a
measurement.
- The always-on prompt asserted the rule with no definition at all, so the
default rule an agent ran under was undefined. Inlined it.
- The cloud variant judged the PTY log only, but cloud runs put metrics in
EVAL.md and W&B — a documented best practice would have left a node
provisional with its measurement sitting in the artifact store.
- Dropped the last two causal qualifiers ("of what this node changed").
* Review round 9 (quality): retire the dead metric-naming machinery
The two quality reviewers reviewed the committed state before the kind-not-
count fix landed, so most of their findings were already resolved. Two
survived, and both are real.
**orx-compute contradicted the exclusion list.** It said "if the run put
numbers there — an OOM at a known memory watermark, a timeout at a known
step rate — that *is* the node's result", while the freeze test says byte
counts inside an allocator error and durations in a timeout message are
*never* numbers. Same log, opposite verdicts, in the skill an agent loads
when a job OOMs — the OOM-both-ways loophole reopened at the one site most
likely to be read alone. It now defers to the freeze test instead of
restating a rival criterion.
**"The freeze test reads that metric" is false.** Round 4 added the metric-
naming chain when bucket 1 genuinely read the named metric. Round 8 made the
test metric-agnostic ("numbers that aren't the ones you named still freeze
it"), so six surfaces were left asserting a lookup that no longer happens —
including "a node without one cannot be judged", which is flatly untrue: a
node with an empty description is judged exactly like any other. Naming the
measurement is still worth doing, so the guidance stays with an honest
reason (a later reader needs to know what the node owed); the false
justification is gone.
Two clauses died with it:
- The "No metric named?" fallback resolved a lookup that no longer occurs,
and its own last sentence just restated the bright line.
- The reporting-defect guard duplicated "Numbers that aren't the ones you
named still freeze it — record what actually landed" one bucket above.
Neither deletion reopens the semantic-defect case; that bucket-1 sentence
closes it on its own.
* Review round 10: close the inclusion list, not the exclusion list
Round 10's four reviewers found 23 blockers. The structural one is a flaw in
how I had been writing the rule for three rounds.
The exclusion list was closed and the inclusion side open ("a loss, an eval
figure, a score" — plus anything else you'd call a metric). That is backwards,
because the tie-breaker already pushes toward freezing: the enumeration guarded
the disfavoured direction while the open list guarded the favoured one. So
`lr=3e-5`, gradient norm, `tokens/s`, `epoch took 42.1s`, and dataset counts
were all undecidable, and an agent that didn't want to debug an OOM could call
the LR trace "the metric my node owes" and freeze. That is the fifth home for
the same discretion: idea -> cause -> evidence class -> line count -> kind.
The inclusion side is now the closed one. Numbers means a **quality**
measurement — how good the model or its output is: loss, perplexity,
accuracy/F1/reward/win-rate, eval score. Nothing else, whatever it is called.
Operational figures are never numbers however central to the node's idea, and
a node whose stated `Measures:` is operational is mis-specified. That resolves
the LR-node scenario and every case the enumeration had missed.
Round 9's own regression, found here: I widened bucket 1 to read artifacts and
W&B but left buckets 2 and 3 saying "the log". On the normal cloud path — where
the run command writes EVAL.md — a run whose metrics landed in an artifact
satisfied bucket 1 (frozen) *and* bucket 2 (provisional, "repair it in place").
The destructive verb was on the wrong branch. Worse, widening where evidence
counts without widening where you must *look* made "I read the log, only a
traceback" fully compliant while EVAL.md held the metrics — a repair licence.
Buckets 2 and 3 now require all channels, and a "look before you classify" rule
makes the check mandatory. Same fix in orx-evidence and root SKILL.md; the
local variant states explicitly that the log is its only channel, so a later
round doesn't "harmonize" the divergence away.
Also:
- "produced from this node's own commit" was a re-entry point for the causal
argument (inherited code emitting a loss isn't "my change producing it").
Now "this run computed", with the irrelevance of authorship stated.
- orx-compute told the agent to bump a node's flavor while cardinal rule 2 and
orx-compute-k8s call hardware shape a fixed contract. Re-sizing is now a
repair only until some node in the round has measured.
- The project-wide cap existed only in the on-demand skill, so an agent on the
always-on prompt could repair node A twice, B twice, C twice indefinitely —
the "N node problems instead of one setup problem" the clause exists to stop.
- The deleted reporting-defect guard carried a remedy, not just a
classification; restored the remedy clause alone.
- Cut "Why this way round" and two other restatements: after the principle
landed they said what the opening line and the over-freeze paragraph already
say.
* Cut the review scar tissue: 681 -> 402 added lines
Ten review rounds each closed a loophole by adding a clause, and I let that
framing override the context budget. The result was a rule that argued with a
hypothetical adversarial reader instead of stating itself — an 86-line freeze
test for something that fits in three lines, with the definition restated
three times and six paragraphs of pre-empted rationalizations.
The rule is unchanged. What went is the arguing:
- Freeze test: 86 -> 25 lines. Kept the two buckets, the operational-figures
exclusion, permanence, and the ambiguity tie-break. Cut "Why this way
round", the re-reading guard, the over-freeze acknowledgement, the
hypothesis-repair clause, the per-value/per-line gloss, and two restatements
of the definition — every one of them written to pre-empt a reading rather
than to state the rule.
- Classify section: 38 -> 17. The table's first row was a pointer to the
freeze test; the cap prose restated itself twice.
- SYSTEM_PROMPT (every turn, 3 harnesses): +44 -> +24. The exclusion list and
the error taxonomy belong in the skill the rule already links to.
- orx-git: +78 -> +62. The repair section duplicated the sync recipe 60 lines
above it. The `merge --ff-only` fix and the worktree-lock case stay.
- orx-evidence: the new section was a 17-line restatement of the freeze test
in the file that links to it. Now a 7-line pointer.
- orx-create, orx-compute (x2), k8s: same treatment — one statement of the
rule at its home, cross-references elsewhere.
- Dropped session_env_tests: 45 lines of mutex, Drop guard, and save/restore
to test `launching_chat_session().is_some()` on a one-line wrapper. The
behaviour that actually changed is covered by
find_serves_the_requested_variant_body, which stays.
fmt, clippy, and 277 tests green; no old-model phrasing survives.
* Collapse the duplicate sections: markdown now +217/-173 (net +44)
The last pass shortened each section but kept the structure, so the same rule
was still taught in three places. The real cut is structural.
`orx-experiment-tree` had two sections — "A node is an evidence contract" and
"Classify before you create" — that both answered repair-vs-child, plus a
Repair bullet in the loop restating it a third time with its own cap. Merged
into one 23-line section; the loop's bullet is now a pointer. Both variants
go net negative: +39/-51 and +37/-50.
Same test applied everywhere else — is this the rule's home, or a restatement?
- orx-compute (x2): 18-line block re-teaching the freeze rule and cap -> 7
lines of the three compute-specific routes, deferring the rest.
- Root SKILL.md: cardinal rule 1 was 15 lines saying it twice -> 6; the
four-move paragraph restated the freeze rule -> dropped.
- orx-git: the repair section duplicated the sync recipe above it -> 10 lines
for what is genuinely git-specific (same branch, worktree lock).
- orx-create: prose trimmed to the baseline-specific facts.
Nothing lost: the freeze rule, the repair cap, `merge --ff-only`, and the
`set -e` seed guard are each stated once at their home and cross-referenced
elsewhere. fmt/clippy clean, 277 tests green.
* PR review: reframe on baseline/hypothesis, restore Shape the tree
Two review comments, both right.
**Restored "Shape the tree".** My slimming pass deleted the whole section —
the flat-fan/noodle/stacked-bushes diagram and the width-vs-depth rule — as
collateral from merging two adjacent sections. That is pre-existing content
unrelated to this PR and should never have been touched. Restored verbatim
from main in both variants.
**Dropped the metric framing.** The rule leaned on "did the log carry a
quality metric", which is a log-parsing heuristic standing in for the actual
criterion: every node exists to establish a baseline or test a hypothesis, and
a run that dies on an error does neither. That is *why* successive runs on one
node are right — nothing was established, so there is nothing to protect. The
section now says that directly, and the exclusion list of operational figures
goes away with it: unintended behaviour (OOM, timeout, a divergence from a
bug, a missing dep) is an implementation or hardware detail, not an answer —
unless the node's hypothesis is itself about memory or runtime.
Reconciled the same framing across all eleven sites that had said "metric":
SYSTEM_PROMPT, root SKILL.md, both tree variants, orx-git, and both compute
variants.
**k8s bullet** cut to five lines and OOMKilled moved to the repair side with
the other implementation details, per the comment.
Added markdown lines barely move (218 -> 228); the tree skills shrink
(+66/+63 -> +45/+43). Net rises to +134 only because restoring Shape the tree
gives back 79 deletions I had wrongly banked as savings.
* Stop editing skills that aren't about the experiment tree
Fair challenge on the PR: k8s and compute are about launching runs, not about
tree shape or node creation. Re-read those edits cold and most of them were
tree semantics squatting in the wrong file.
- **orx-compute-k8s: dropped entirely.** The bullet was pure tree vocabulary
("not a result", "same branch", "same expId") in a file whose job is the
manifest contract. A k8s failure routes through orx-compute anyway, and a
manifest fix is an ordinary branch edit — nothing k8s-specific to say.
- **orx-compute: 7 lines -> 3.** Kept one clause, because this skill's own
description says "use when a job OOMs" and its pre-existing bullet already
says "re-launch, or pick a different flavor or backend" — an agent following
that needs to know the node is still repairable. Everything else deferred.
- **Reverted "Let a real failure escalate you" outright.** I had rewritten a
sizing rule to carry my terminology; sizing advice doesn't need it.
- **orx-evidence: reverted both variants.** The section restated the tree rule
in the skill that links to it, and still used the metric framing dropped in
the last commit.
Also removed the "Measures: <metric>" scaffolding from the branch step, the
worked examples, and the exp desc bullet. That existed to feed the metric
test; with the rule now on baseline-vs-hypothesis it has no consumer, so the
create examples go back to their original form.
Markdown drops to +166/-75 across 8 files (from +228 across 11). The change
now touches only the tree skills, the playbook, orx-git, orx-create, and the
one compute clause.