Files
openbot/scripts/workflow-loop-e2e.ts
T
NBandClaude Opus 5.5 39b2a95052 Start workflows from GitHub events and change GitHub from workflows (#4288)
* Fix workflow node fades and ports in browsers without scroll timelines

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add the GitHub workflow contract and the workflows-github-v1 capability

GitHub is a trigger type, an output target and a run source. The new
capability carries the GitHub catalog, repository options, event history
and a run with the last event. A client shows a workflow with a trigger
type or an output target that it does not know as read-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add migration 49 for GitHub workflow runs, deliveries and actions

Migration 49 rebuilds workflow_runs to accept the github run source and
keeps every run id. It adds three tables:

- workflow_github_runs: the item of each run that a GitHub event started.
- workflow_github_deliveries: the event history of each GitHub trigger,
  and the per-item queue (a waiting row keeps its payload until it starts).
- workflow_github_actions: each change that a GitHub output made, so a
  retry does not repeat it and the loop guard can find its own events.

The latest schema takes the same run source line and tables. Tests cover
schemas 46 to 48, a late failure and retry, newer and missing histories,
constraint refusals and cascades.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Route GitHub App events through Signal to the hosts that watch them

Signal gets POST /v1/github/events. It checks the App signature, reads only
the installation and repository IDs, and sends the exact body to every host
whose GitHub route ticket names the repository. It keeps no body.

The account service signs the 5-minute route ticket after it checks, with
the host's GitHub sign-in, which watched repositories that sign-in can see.
D1 keeps no GitHub route.

The host ingress socket holds a GitHub route through the new GitHubRelay,
answers github-delivery frames, and records the offline gaps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Start workflow runs from GitHub events, one run per item

- Match GitHub events to triggers by repository, event and filters.
- Queue an event when a run of its item is open; start it when that run ends.
  Runs of other items start at the same time.
- Skip echoes of the workflow's own actions (self) and the fourth run in
  10 minutes that OpenBot's bot caused on one item (chain).
- Keep the newest 50 events per trigger and record offline gaps.
- Add the refused skip reason and caused_by_bot to the unreleased v49 table.
- Add scripts/github-workflow-e2e.ts (host level).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Make the changes of GitHub outputs as OpenBot's bot

A GitHub output makes one of the 10 changes through the GitHub REST API with
the App's installation token. "Agent chooses" asks a model once and keeps only
the repository's own labels and people. Each change writes a `sending` row
before the call and a `done` row after it, so a retry does not post twice and
the loop guard knows the change's events. A restart sends again only a change
that a second call does not repeat. "Run with last event" starts a run with
the newest matching item.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Show GitHub triggers and outputs on the workflow canvas

The trigger and output cards get GitHub rows: repository, event,
filters, action, labels and people, with "Agent chooses". The Events
row lists what GitHub sent. Access and event alerts open GitHub, and
Connect opens the connect dialog of the GitHub settings page.

The run bar lists the open runs and the waiting events, and the canvas
shows the steps of the open run that the user views.

The renderer reads the catalog, labels, people and event history from
the host through new IPC reads and Team API routes. GitHub shows only
for an owner or admin on a host with workflows-github-v1.

Stories replace the prototype stories.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Translate the GitHub workflow text into 12 languages

Name the GitHub event and action labels as literal catalog keys, and
remove two keys that no code uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test the redaction of GitHub errors, and document GitHub workflows

Add a Vitest that checks that GitHub's error text and a failed request
put no token into a step's failure. Document the GitHub trigger path,
the delivery outcomes and the Team API routes, and add release notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remove unused GitHub exports, and fix an optional field in the e2e

Knip and the exactOptionalPropertyTypes ratchet failed the Check job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Send not_planned to GitHub when an output closes an issue as not planned

The saved close reason is notPlanned. GitHub refused it with
"Please specify a valid state reason." The adapter now maps each saved
reason to the state_reason that GitHub accepts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep trigger events off GitHub, and add repository search and loading rows

- A GitHub output removes the framed trigger event from its text. A comment
  that has only the event fails with a clear error and posts nothing.
- The repository menus have a search field.
- The Labels and Person menus show "Loading…" while the host reads them.
- The log says that the GitHub output failed, not that it made no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Make each loop pass change GitHub, and keep Telegram when a ticket fails

- A GitHub output keys its change by run, node and pass. A loop pass
  makes its own change. A retry or a restart of the same pass does not
  make the change again.
- When the trigger event comes back in a loop, the output also removes
  the empty "Input N:" names of the event.
- A failed GitHub or webhook ticket does not stop Telegram or the other
  routes on the shared socket.
- The connector reads the App events from GITHUB_ROUTE_EVENTS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Let Signal pass GitHub events of up to 512 KiB

An issue_comment or pull_request event can hold two texts of 65,536
characters each. Signal refused such an event with 413, and GitHub does
not send it again.

- GITHUB_DELIVERY_BODY_BYTES_LIMIT is 512 KiB.
- The pending budget for each host is 768 KiB, and the socket
  backpressure limit is 1 MiB, so the largest event fits.
- nginx gets a /v1/github/events location with the same body limit, no
  access log, and the 10-second wait of GitHub.
- The Signal e2e sends the largest event and one above the limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep an end marker in a comment inside its event, and stop loading after a failed options read

The event frame ends only at the marker on its own line. The JSON of the event keeps no line
break, so a commenter cannot end the frame early and get text posted as the bot.

A failed read of repository labels and people now stores empty options. The menus stop
showing "Loading". A change on GitHub reads them again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 00:19:46 +01:00

201 lines
9.4 KiB
TypeScript

// Checks workflow loops on the real SQLite and the real workflow service, on a temporary profile. The
// graph is a prompt, two approvals and an output, with a connection from the second approval back to
// the first. Approvals need no provider, so no model, agent, account or network is started. A save
// keeps the loop; each approval of the second step starts the next pass with its answer as one more
// input, and the output runs once for each pass; a rejection ends the loop and stops the run. A run
// from a node inside the loop starts at that node and takes the earlier answer of the node before it.
//
// Run: bun scripts/workflow-loop-e2e.ts
// Writes .openbot-build/workflow-loop-e2e/report.json and exits non-zero on a failure.
import assert from "node:assert/strict";
import { mkdirSync, writeFileSync } from "node:fs";
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { type WorkflowGraph, type WorkflowStep, workflowLoopEdges } from "@openbot/contracts/ipc";
import { Effect } from "effect";
import { runCauseEffect } from "../src/backend/effect-boundary";
import { OpenBotDatabase } from "../src/backend/openbot-database";
import { WorkflowGitHubStore } from "../src/backend/workflows/workflow-github-store";
import { WorkflowRunStore } from "../src/backend/workflows/workflow-run-store";
import { WorkflowStore } from "../src/backend/workflows/workflow-store";
import { workflowScheduleDueAt } from "../src/backend/workflows/workflow-triggers";
import { createWorkflows, type WorkflowsHandle } from "../src/backend/workflows/workflows";
const OUT = resolve(import.meta.dirname, "../.openbot-build/workflow-loop-e2e");
const WAIT_MS = 5_000;
interface Report {
checks: string[];
outputs: string[];
passed: boolean;
failure?: string;
}
const report: Report = { checks: [], outputs: [], passed: false };
const check = (name: string, ok: boolean, detail?: unknown) => {
assert.ok(ok, `${name}${detail === undefined ? "" : `: ${JSON.stringify(detail)}`}`);
report.checks.push(name);
};
const GRAPH: WorkflowGraph = {
nodes: [
{ id: "prompt", kind: "prompt", at: { x: 0, y: 0 }, fields: { text: "Draft" } },
{ id: "review", kind: "approval", at: { x: 300, y: 0 }, fields: { question: "Review?" } },
{ id: "check", kind: "approval", at: { x: 600, y: 0 }, fields: { question: "Again?" } },
{ id: "output", kind: "output", at: { x: 900, y: 0 }, fields: { target: "notify" } },
],
edges: [
{ id: "prompt>review", from: "prompt", to: "review", type: "text" },
{ id: "review>check", from: "review", to: "check", type: "text" },
{ id: "check>output", from: "check", to: "output", type: "text" },
{ id: "check>review", from: "check", to: "review", type: "text" },
],
strokes: [],
};
const directory = await mkdtemp(join(tmpdir(), "openbot-workflow-loop-"));
const database = new OpenBotDatabase(directory);
let service: WorkflowsHandle | null = null;
async function open() {
await Effect.runPromise(database.initialize());
const workflows = await Effect.runPromise(
createWorkflows({
store: new WorkflowStore({ database, nextDueAt: workflowScheduleDueAt(() => "UTC") }),
runs: new WorkflowRunStore({ database }),
github: new WorkflowGitHubStore({ database }),
githubApp: () => null,
delivery: () => null,
turnAnswer: () => null,
sendHandoff: () => Effect.fail({ cause: new Error("No agent step runs in this test.") }),
stopDelivery: () => Effect.void,
generateText: () => Effect.fail({ message: "No model step runs in this test." }),
notify: (notice) => {
if (notice.kind === "output") report.outputs.push(notice.text);
},
connectionOn: () => false,
post: () => Effect.die(new Error("No post runs in this test.")),
changed: () => {},
rearm: () => {},
timezone: () => "UTC",
}),
);
service = workflows;
return workflows;
}
async function close() {
if (service) await Effect.runPromise(service.close()).catch(() => {});
service = null;
database.close();
}
/** Waits until the run has a step that waits for a decision, and gives it. */
async function waiting(workflows: WorkflowsHandle, runId: string): Promise<WorkflowStep> {
const deadline = Date.now() + WAIT_MS;
for (;;) {
const step = (await runCauseEffect(workflows.run(runId))).steps.find((each) => each.status === "approval");
if (step) return step;
if (Date.now() > deadline) throw new Error("No step waits for a decision.");
await new Promise((done) => setImmediate(done));
}
}
/** The input of a step, as the store keeps it; the public step does not show it. */
const inputOf = (step: WorkflowStep) => new WorkflowRunStore({ database }).step(step.id)?.input;
const runStatus = async (workflows: WorkflowsHandle, runId: string) =>
(await runCauseEffect(workflows.run(runId))).run.status;
try {
const workflows = await open();
check(
"the graph has one loop: the connection back",
[...workflowLoopEdges(GRAPH.nodes, GRAPH.edges)].join() === "check>review",
);
const created = await runCauseEffect(workflows.create({ title: "Review loop" }));
const workflowId = created.summary.id;
const saved = await runCauseEffect(workflows.saveGraph({ workflowId, baseRevision: created.revision, graph: GRAPH }));
check("a graph with a loop saves", saved.ok, saved);
const detail = await runCauseEffect(workflows.get(workflowId));
check(
"the saved graph keeps the loop",
detail.graph.edges.some((edge) => edge.id === "check>review"),
);
const run = (await runCauseEffect(workflows.startRun({ workflowId, requestId: "loop-1" }))).run;
let step = await waiting(workflows, run.id);
check(
"pass 1 waits at the first approval",
step.nodeId === "review" && step.attempt === 1 && inputOf(step) === "Draft",
step,
);
check("the run waits for a decision", (await runStatus(workflows, run.id)) === "approval");
await runCauseEffect(workflows.decide({ stepId: step.id, approved: true }));
step = await waiting(workflows, run.id);
check("pass 1 goes on to the second approval", step.nodeId === "check" && step.attempt === 1, step);
await runCauseEffect(workflows.decide({ stepId: step.id, approved: true }));
step = await waiting(workflows, run.id);
check(
"the loop starts pass 2 at the first approval, with the loop's answer as one more input",
step.nodeId === "review" && step.attempt === 2 && inputOf(step) === "Input 1:\n\nDraft\n\nInput 2:\n\nDraft",
step,
);
check("the output runs after pass 1", report.outputs.length === 1, report.outputs);
await runCauseEffect(workflows.decide({ stepId: step.id, approved: true }));
step = await waiting(workflows, run.id);
check("pass 2 goes on to the second approval", step.nodeId === "check" && step.attempt === 2, step);
check("the output does not run before the pass ends", report.outputs.length === 1, report.outputs);
await runCauseEffect(workflows.decide({ stepId: step.id, approved: true }));
step = await waiting(workflows, run.id);
check("pass 3 starts", step.nodeId === "review" && step.attempt === 3, step);
check("the output runs after pass 2", report.outputs.length === 2, report.outputs);
await runCauseEffect(workflows.decide({ stepId: step.id, approved: false }));
check("a rejection ends the loop and stops the run", (await runStatus(workflows, run.id)) === "stopped");
const steps = (await runCauseEffect(workflows.run(run.id))).steps;
check(
"each pass is the next attempt of its node",
steps.filter((each) => each.nodeId === "review").length === 3 &&
steps.filter((each) => each.nodeId === "check").length === 2 &&
steps.filter((each) => each.nodeId === "prompt").length === 1,
steps.map((each) => `${each.nodeId}:${each.attempt}:${each.status}`),
);
const from = (await runCauseEffect(workflows.startRun({ workflowId, requestId: "loop-2", fromNodeId: "check" }))).run;
step = await waiting(workflows, from.id);
const reused = (await runCauseEffect(workflows.run(from.id))).steps.find((each) => each.nodeId === "review");
check(
"a run from a node in the loop starts at that node with the earlier answer before it",
step.nodeId === "check" && step.attempt === 1 && reused?.status === "done" && reused.attempt === 1,
{ step, reused },
);
const checked = step;
await runCauseEffect(workflows.decide({ stepId: checked.id, approved: true }));
step = await waiting(workflows, from.id);
const answer = (await runCauseEffect(workflows.run(from.id))).steps.find((each) => each.id === checked.id)?.output;
check(
"its loop goes back to the node before it, with only the loop's answer",
step.nodeId === "review" && step.attempt === 2 && answer !== undefined && inputOf(step) === answer?.trim(),
{ step, answer, input: inputOf(step) },
);
await runCauseEffect(workflows.stopRun(from.id));
check("a stop ends the loop", (await runStatus(workflows, from.id)) === "stopped");
report.passed = true;
} catch (error) {
report.failure = error instanceof Error ? (error.stack ?? error.message) : String(error);
} finally {
await close();
await rm(directory, { recursive: true, force: true });
mkdirSync(OUT, { recursive: true });
writeFileSync(join(OUT, "report.json"), `${JSON.stringify(report, null, 2)}\n`);
process.stdout.write(`${report.passed ? "passed" : "FAILED"}: ${report.checks.length} checks; ${OUT}/report.json\n`);
if (report.failure) process.stderr.write(`${report.failure}\n`);
process.exitCode = report.passed ? 0 : 1;
}