mirror of
https://github.com/nightly-labs/openbot.git
synced 2026-11-23 02:01:26 +08:00
* Plan 012: archive ended pool servers in B2; prove export and import on a pool machine Plan 012 moves the data of a Leaseweb pool server whose plan ended to an encrypted archive in Backblaze B2, so the machine goes back to the pool within a day. It follows #12721 and the dispute rule: 14 days of data, and a refunded or early disputed server is deleted with no archive. Item 1 is done. move-from-boat.sh export and import already work on a pool machine; no pool mode is needed. scripts/hosting/pool-archive-e2e.ts proves it on the test VPS pool-test-01: prepare, seed, export, reinstall, import, start. 6 of 6 checks pass, with 446 file facts the same after the import. - export copied the cron spool file with its "DO NOT EDIT" header, and the import kept those lines as customer text. It now uses `crontab -l`. Moves from boat had the same bug. - leaseweb-pool.ts packs the hosting folder with --no-xattrs: macOS tar added headers that GNU tar on the VPS warned about. - apps/auth-api/.env.production: encrypted B2 keys for the EU bucket (operator and Worker). The Worker secrets come in item 2. * Plan 012: encrypt pool archives with age and upload them to B2 from the machine The E2E now runs the full archive path on a pool machine: sync makes an age key and presigns B2 multipart part URLs, the machine encrypts the export and uploads each part, sync checks each part MD5 and the size, then the machine is reinstalled, downloads with a presigned URL, decrypts and imports. leaseweb-bootstrap.sh installs age. The plan records the developer decisions of 2026-11-19: one EU bucket for all locations, and age encryption. A 5.75 GB server passes 9 of 9 checks (54,094 file facts). Upload 661 s, download and decrypt 269 s, import 349 s. The import time is the disk speed of the VPS: the same extract alone took 47 to 67 s. Bash ignores PS4 from the environment as root, so the import trace sets it in the shell. * Plan 012 item 2: D1 table and Worker secrets for pool server archives Migration 0061 adds hosted_server_archives: one row for each verified archive in B2, with the wrapped age key, the B2 version id for a permanent delete, and the time when the archive must be deleted. The table has no foreign key, so an archive stays findable after a server or account delete. A deleted archive keeps no key. 0060 is taken by #12845. HOSTED_ARCHIVE_KEY_SECRET, B2_WORKER_KEY_ID_EU and B2_WORKER_APPLICATION_KEY_EU are a set of three for the local deploy and the CI deploy. The values are in .env.production and in the cloudflare-production Environment. hosted_servers.plan_end_reason is not added: Stripe keeps the reason in cancellation_details.reason, and sync reads the subscription fresh. * Plan 012 items 3, 5, 6: archive ended pool servers in B2 from sync sync archives a stopped pool server whose Stripe subscription is canceled (3 days later after a failed payment), stores the wrapped age key in hosted_server_archives, frees the machine and reinstalls it. Each sync deletes archives that are due, of a deleted server, or restored. The Worker 14-day delete also finds an archived server, and deletes the old boat sandbox of a moved server with no machine. PRIVACY.md and the DPA name Backblaze B2 for the encrypted archive. Hold the first production archive until the restore (item 4) is live: a renewal of an archived server now gets a new, empty server. * Plan 012 item 4: restore an archived pool server when its plan is renewed A renewal of a pool server with a live archive takes a ready pool machine. The assignment answer gives the machine a presigned B2 URL, the age key and the checksum. openbot-pool-wait runs move-from-boat.sh restore, reports "restored", and then starts OpenBot. A machine that does not send restoreVersion 1 gets no restore. A restore with no report after 3 hours goes back to awaiting_payment, and the machine leaves the pool. The status shows state "starting" with "restoring". Clients show the start step "restore" ("Restoring data") with no 15-minute limit, and settings and mobile show until which date the data of an ended pool server is kept. Tests: hosted-pool-restore-e2e (Worker flow, 25 checks) and pool-archive-e2e on pool-test-01 (real B2 archive, 10 checks). * Remove a code comment and an old plan dependency AGENTS.md forbids code comments. The reason for crontab -l stays here: the spool file starts with the "DO NOT EDIT" header of cron, and crontab on the VPS would keep those lines as text of the customer. Plan 012 no longer waits for a B2 US account or a Stripe decision. * Share the pool machine setup between the pool script and the E2E Move the hosting copy and the prepare and upgrade scripts to pool-machine.ts. The pool script and the archive E2E use this module. Remove the copies. * Give fetch a null body when the B2 request has no body The types ratchet does not allow an undefined value for an optional property. * Clear the pre-stop marker before a pool start and share the size command An archive that stops early leaves /run/openbot-pre-stop on the machine. openbot.service then skips its start, but sync records the server as running. The start command now removes the marker first. SERVER_BYTES moves to pool-archive.ts. The pool script and the E2E use it. * Let knip read the auth-api E2E scripts The restore E2E is the only user of the archive restore helpers, so knip reported them as unused exports. * Refuse each 401 to a restore report Before, openbot-pool-wait read a 401 after a lost answer as success, and could start OpenBot on a machine that the Worker took back. The Worker now keeps the offer token of a restored machine for 2 hours, which is longer than the report retries. A retry after a lost answer gets "restored", and each 401 stops the start.
224 lines
8.4 KiB
TypeScript
224 lines
8.4 KiB
TypeScript
import { createCipheriv, createHash, createHmac, generateKeyPairSync, randomBytes } from "node:crypto";
|
|
import { z } from "zod";
|
|
|
|
export const B2_HOST = "s3.eu-central-003.backblazeb2.com";
|
|
const B2_REGION = "eu-central-003";
|
|
export const B2_BUCKET = "openbot-pool-archive-eu";
|
|
export const ARCHIVE_REGION = "eu";
|
|
export const PART_BYTES = 100 * 1024 * 1024;
|
|
export const SERVER_BYTES = `du -sb --one-file-system /home/user /srv/openbot-hosted | awk '{ total += $1 } END { print total }'`;
|
|
export const UPLOAD_URL_SECONDS = 6 * 60 * 60;
|
|
const BECH32_CHARSET = "qpzry9x8gf2tvdw0s3jn54khce6mua7l";
|
|
const BECH32_GENERATOR = [0x3b6a57b2, 0x26508e6d, 0x1ea119fa, 0x3d4233dd, 0x2a1462b3];
|
|
const WRAP_VERSION = "v1";
|
|
|
|
export interface B2Key {
|
|
id: string;
|
|
secret: string;
|
|
}
|
|
|
|
export interface UploadedPart {
|
|
number: number;
|
|
size: number;
|
|
md5: string;
|
|
etag: string;
|
|
}
|
|
|
|
export const archiveScript = (machineHosting: string) => `
|
|
set -euo pipefail
|
|
recipient=$1 part_bytes=$2
|
|
dir=$(mktemp -d /var/tmp/openbot-archive.XXXXXX)
|
|
trap 'rm -rf "$dir"' EXIT
|
|
cat >"$dir/urls"
|
|
cat >"$dir/part.sh" <<'PART'
|
|
set -euo pipefail
|
|
number=$((10#\${FILE##*.} + 1))
|
|
url=$(sed -n "\${number}p" "$URLS")
|
|
[ -n "$url" ] || { echo "No upload URL for part $number." >&2; exit 3; }
|
|
cat >"$FILE"
|
|
size=$(stat -c %s "$FILE")
|
|
md5=$(md5sum "$FILE" | cut -c1-32)
|
|
etag=$(curl -fsS --retry 5 --retry-all-errors -T "$FILE" -D - -o /dev/null "$url" | tr -d '\\r"' | sed -n 's/^[Ee][Tt][Aa][Gg]: *//p')
|
|
rm -f "$FILE"
|
|
echo "part $number $size $md5 $etag"
|
|
PART
|
|
mkfifo "$dir/sum"
|
|
sha256sum <"$dir/sum" | cut -c1-64 >"$dir/sha256" &
|
|
summer=$!
|
|
export URLS="$dir/urls"
|
|
bash ${machineHosting}/move-from-boat.sh export - | age -r "$recipient" | tee "$dir/sum" |
|
|
split -b "$part_bytes" -d -a 5 --filter="bash $dir/part.sh" - "$dir/part."
|
|
wait "$summer"
|
|
echo "sha256 $(cat "$dir/sha256")"
|
|
`;
|
|
|
|
export function shellWord(value: string) {
|
|
return `'${value.replaceAll("'", `'"'"'`)}'`;
|
|
}
|
|
|
|
export function uploadedParts(output: string) {
|
|
const parts: UploadedPart[] = output
|
|
.split("\n")
|
|
.filter((line) => line.startsWith("part "))
|
|
.map((line) => {
|
|
const [, number = "", size = "", md5 = "", etag = ""] = line.split(" ");
|
|
return { number: Number(number), size: Number(size), md5, etag };
|
|
});
|
|
const sha256 = /^sha256 ([0-9a-f]{64})$/mu.exec(output)?.[1] ?? "";
|
|
return { parts, sha256, bytes: parts.reduce((total, part) => total + part.size, 0) };
|
|
}
|
|
|
|
export function partCount(serverBytes: number) {
|
|
return Math.ceil((serverBytes * 1.05 + 256 * 1024 * 1024) / PART_BYTES) + 1;
|
|
}
|
|
|
|
function bech32Polymod(values: number[]) {
|
|
let checksum = 1;
|
|
for (const value of values) {
|
|
const top = checksum >>> 25;
|
|
checksum = ((checksum & 0x1ffffff) << 5) ^ value;
|
|
for (const [bit, generator] of BECH32_GENERATOR.entries()) {
|
|
if ((top >>> bit) & 1) checksum ^= generator;
|
|
}
|
|
}
|
|
return checksum;
|
|
}
|
|
|
|
function bech32(prefix: string, data: Buffer) {
|
|
const words: number[] = [];
|
|
let accumulator = 0;
|
|
let bits = 0;
|
|
for (const byte of data) {
|
|
accumulator = ((accumulator << 8) | byte) & 0xfff;
|
|
bits += 8;
|
|
while (bits >= 5) {
|
|
bits -= 5;
|
|
words.push((accumulator >>> bits) & 31);
|
|
}
|
|
}
|
|
if (bits > 0) words.push((accumulator << (5 - bits)) & 31);
|
|
const codes = [...prefix].map((character) => character.charCodeAt(0));
|
|
const expanded = [...codes.map((code) => code >>> 5), 0, ...codes.map((code) => code & 31)];
|
|
const mod = bech32Polymod([...expanded, ...words, 0, 0, 0, 0, 0, 0]) ^ 1;
|
|
const checksum = Array.from({ length: 6 }, (_, index) => (mod >>> (5 * (5 - index))) & 31);
|
|
return `${prefix}1${[...words, ...checksum].map((word) => BECH32_CHARSET.charAt(word)).join("")}`;
|
|
}
|
|
|
|
export function newArchiveKey() {
|
|
const { privateKey, publicKey } = generateKeyPairSync("x25519");
|
|
const secret = z.object({ d: z.string() }).parse(privateKey.export({ format: "jwk" })).d;
|
|
const shared = z.object({ x: z.string() }).parse(publicKey.export({ format: "jwk" })).x;
|
|
return {
|
|
identity: bech32("age-secret-key-", Buffer.from(secret, "base64url")).toUpperCase(),
|
|
recipient: bech32("age", Buffer.from(shared, "base64url")),
|
|
};
|
|
}
|
|
|
|
export function wrapKeySecret(value: string | undefined) {
|
|
const secret = Buffer.from(value ?? "", "base64");
|
|
if (secret.length !== 32) throw new Error("HOSTED_ARCHIVE_KEY_SECRET is not 32 bytes in base64.");
|
|
return secret;
|
|
}
|
|
|
|
export function wrapArchiveKey(secret: Buffer, identity: string, serverId: string, objectKey: string) {
|
|
const iv = randomBytes(12);
|
|
const cipher = createCipheriv("aes-256-gcm", secret, iv);
|
|
cipher.setAAD(Buffer.from(`${serverId}\n${objectKey}`, "utf8"));
|
|
const sealed = Buffer.concat([cipher.update(identity, "utf8"), cipher.final(), cipher.getAuthTag()]);
|
|
return `${WRAP_VERSION}.${iv.toString("base64url")}.${sealed.toString("base64url")}`;
|
|
}
|
|
|
|
function uriPart(value: string) {
|
|
return encodeURIComponent(value).replace(
|
|
/[!'()*]/gu,
|
|
(character) => `%${character.charCodeAt(0).toString(16).toUpperCase()}`,
|
|
);
|
|
}
|
|
|
|
export function presign(
|
|
b2: B2Key,
|
|
method: string,
|
|
objectKey: string,
|
|
parameters: Record<string, string>,
|
|
seconds: number,
|
|
) {
|
|
const now = new Date().toISOString().replace(/[-:]|\.\d{3}/gu, "");
|
|
const day = now.slice(0, 8);
|
|
const scope = `${day}/${B2_REGION}/s3/aws4_request`;
|
|
const query = Object.entries({
|
|
...parameters,
|
|
"X-Amz-Algorithm": "AWS4-HMAC-SHA256",
|
|
"X-Amz-Credential": `${b2.id}/${scope}`,
|
|
"X-Amz-Date": now,
|
|
"X-Amz-Expires": String(seconds),
|
|
"X-Amz-SignedHeaders": "host",
|
|
})
|
|
.map(([name, value]) => `${uriPart(name)}=${uriPart(value)}`)
|
|
.sort()
|
|
.join("&");
|
|
const path = `/${B2_BUCKET}/${objectKey.split("/").map(uriPart).join("/")}`;
|
|
const request = [method, path, query, `host:${B2_HOST}`, "", "host", "UNSIGNED-PAYLOAD"].join("\n");
|
|
const text = ["AWS4-HMAC-SHA256", now, scope, createHash("sha256").update(request).digest("hex")].join("\n");
|
|
const signingKey = [day, B2_REGION, "s3", "aws4_request"].reduce<Buffer>(
|
|
(key, part) => createHmac("sha256", key).update(part).digest(),
|
|
Buffer.from(`AWS4${b2.secret}`),
|
|
);
|
|
const signature = createHmac("sha256", signingKey).update(text).digest("hex");
|
|
return `https://${B2_HOST}${path}?${query}&X-Amz-Signature=${signature}`;
|
|
}
|
|
|
|
export async function b2Request(
|
|
b2: B2Key,
|
|
method: string,
|
|
objectKey: string,
|
|
parameters: Record<string, string>,
|
|
body?: string,
|
|
) {
|
|
const response = await fetch(presign(b2, method, objectKey, parameters, 900), {
|
|
method,
|
|
body: body ?? null,
|
|
signal: AbortSignal.timeout(120_000),
|
|
});
|
|
const text = method === "HEAD" ? "" : await response.text();
|
|
return { status: response.status, headers: response.headers, text };
|
|
}
|
|
|
|
export async function startUpload(b2: B2Key, objectKey: string) {
|
|
const created = await b2Request(b2, "POST", objectKey, { uploads: "" });
|
|
const uploadId = /<UploadId>([^<]+)<\/UploadId>/u.exec(created.text)?.[1] ?? "";
|
|
if (created.status !== 200 || uploadId === "") throw new Error(`B2 did not start the upload: ${created.status}.`);
|
|
return uploadId;
|
|
}
|
|
|
|
export function partUrls(b2: B2Key, objectKey: string, uploadId: string, count: number) {
|
|
return Array.from({ length: count }, (_, index) =>
|
|
presign(b2, "PUT", objectKey, { partNumber: String(index + 1), uploadId }, UPLOAD_URL_SECONDS),
|
|
).join("\n");
|
|
}
|
|
|
|
export async function completeUpload(b2: B2Key, objectKey: string, uploadId: string, parts: UploadedPart[]) {
|
|
const body = `<CompleteMultipartUpload>${parts
|
|
.map((part) => `<Part><PartNumber>${part.number}</PartNumber><ETag>"${part.etag}"</ETag></Part>`)
|
|
.join("")}</CompleteMultipartUpload>`;
|
|
const done = await b2Request(b2, "POST", objectKey, { uploadId }, body);
|
|
if (done.status !== 200 || done.text.includes("<Error>")) {
|
|
throw new Error(`B2 did not complete the upload: ${done.status}.`);
|
|
}
|
|
const head = await b2Request(b2, "HEAD", objectKey, {});
|
|
return {
|
|
status: head.status,
|
|
versionId: head.headers.get("x-amz-version-id") ?? "",
|
|
size: Number(head.headers.get("content-length") ?? "0"),
|
|
};
|
|
}
|
|
|
|
export async function abortUpload(b2: B2Key, objectKey: string, uploadId: string) {
|
|
await b2Request(b2, "DELETE", objectKey, { uploadId }).catch(() => undefined);
|
|
}
|
|
|
|
export async function deleteArchive(b2: B2Key, objectKey: string, versionId: string) {
|
|
const answer = await b2Request(b2, "DELETE", objectKey, versionId === "" ? {} : { versionId });
|
|
const head = await b2Request(b2, "HEAD", objectKey, {});
|
|
return { deleted: (answer.status === 204 || answer.status === 404) && head.status === 404, status: answer.status };
|
|
}
|