Files
NB 4fb65f5fea Plan 012 items 3, 5, 6: archive ended pool servers in B2 (#12928)
* Plan 012: archive ended pool servers in B2; prove export and import on a pool machine

Plan 012 moves the data of a Leaseweb pool server whose plan ended to an
encrypted archive in Backblaze B2, so the machine goes back to the pool within
a day. It follows #12721 and the dispute rule: 14 days of data, and a refunded
or early disputed server is deleted with no archive.

Item 1 is done. move-from-boat.sh export and import already work on a pool
machine; no pool mode is needed. scripts/hosting/pool-archive-e2e.ts proves it
on the test VPS pool-test-01: prepare, seed, export, reinstall, import, start.
6 of 6 checks pass, with 446 file facts the same after the import.

- export copied the cron spool file with its "DO NOT EDIT" header, and the
  import kept those lines as customer text. It now uses `crontab -l`. Moves
  from boat had the same bug.
- leaseweb-pool.ts packs the hosting folder with --no-xattrs: macOS tar added
  headers that GNU tar on the VPS warned about.
- apps/auth-api/.env.production: encrypted B2 keys for the EU bucket
  (operator and Worker). The Worker secrets come in item 2.

* Plan 012: encrypt pool archives with age and upload them to B2 from the machine

The E2E now runs the full archive path on a pool machine: sync makes an age
key and presigns B2 multipart part URLs, the machine encrypts the export and
uploads each part, sync checks each part MD5 and the size, then the machine is
reinstalled, downloads with a presigned URL, decrypts and imports.

leaseweb-bootstrap.sh installs age. The plan records the developer decisions
of 2026-11-19: one EU bucket for all locations, and age encryption.

A 5.75 GB server passes 9 of 9 checks (54,094 file facts). Upload 661 s, download
and decrypt 269 s, import 349 s. The import time is the disk speed of the VPS:
the same extract alone took 47 to 67 s.

Bash ignores PS4 from the environment as root, so the import trace sets it in
the shell.

* Plan 012 item 2: D1 table and Worker secrets for pool server archives

Migration 0061 adds hosted_server_archives: one row for each verified archive
in B2, with the wrapped age key, the B2 version id for a permanent delete, and
the time when the archive must be deleted. The table has no foreign key, so an
archive stays findable after a server or account delete. A deleted archive
keeps no key. 0060 is taken by #12845.

HOSTED_ARCHIVE_KEY_SECRET, B2_WORKER_KEY_ID_EU and B2_WORKER_APPLICATION_KEY_EU
are a set of three for the local deploy and the CI deploy. The values are in
.env.production and in the cloudflare-production Environment.

hosted_servers.plan_end_reason is not added: Stripe keeps the reason in
cancellation_details.reason, and sync reads the subscription fresh.

* Plan 012 items 3, 5, 6: archive ended pool servers in B2 from sync

sync archives a stopped pool server whose Stripe subscription is canceled (3 days later after a failed
payment), stores the wrapped age key in hosted_server_archives, frees the machine and reinstalls it.
Each sync deletes archives that are due, of a deleted server, or restored. The Worker 14-day delete
also finds an archived server, and deletes the old boat sandbox of a moved server with no machine.
PRIVACY.md and the DPA name Backblaze B2 for the encrypted archive.

Hold the first production archive until the restore (item 4) is live: a renewal of an archived
server now gets a new, empty server.

* Plan 012 item 4: restore an archived pool server when its plan is renewed

A renewal of a pool server with a live archive takes a ready pool machine.
The assignment answer gives the machine a presigned B2 URL, the age key
and the checksum. openbot-pool-wait runs move-from-boat.sh restore, reports
"restored", and then starts OpenBot. A machine that does not send
restoreVersion 1 gets no restore. A restore with no report after 3 hours
goes back to awaiting_payment, and the machine leaves the pool.

The status shows state "starting" with "restoring". Clients show the start
step "restore" ("Restoring data") with no 15-minute limit, and settings and
mobile show until which date the data of an ended pool server is kept.

Tests: hosted-pool-restore-e2e (Worker flow, 25 checks) and
pool-archive-e2e on pool-test-01 (real B2 archive, 10 checks).

* Remove a code comment and an old plan dependency

AGENTS.md forbids code comments. The reason for crontab -l stays here:
the spool file starts with the "DO NOT EDIT" header of cron, and crontab
on the VPS would keep those lines as text of the customer. Plan 012 no
longer waits for a B2 US account or a Stripe decision.

* Share the pool machine setup between the pool script and the E2E

Move the hosting copy and the prepare and upgrade scripts to pool-machine.ts.
The pool script and the archive E2E use this module. Remove the copies.

* Give fetch a null body when the B2 request has no body

The types ratchet does not allow an undefined value for an optional property.

* Clear the pre-stop marker before a pool start and share the size command

An archive that stops early leaves /run/openbot-pre-stop on the machine.
openbot.service then skips its start, but sync records the server as running.
The start command now removes the marker first.

SERVER_BYTES moves to pool-archive.ts. The pool script and the E2E use it.

* Let knip read the auth-api E2E scripts

The restore E2E is the only user of the archive restore helpers, so knip
reported them as unused exports.

* Refuse each 401 to a restore report

Before, openbot-pool-wait read a 401 after a lost answer as success, and
could start OpenBot on a machine that the Worker took back. The Worker
now keeps the offer token of a restored machine for 2 hours, which is
longer than the report retries. A retry after a lost answer gets
"restored", and each 401 stops the start.
2026-11-19 21:41:39 +01:00

224 lines
8.4 KiB
TypeScript

import { createCipheriv, createHash, createHmac, generateKeyPairSync, randomBytes } from "node:crypto";
import { z } from "zod";
export const B2_HOST = "s3.eu-central-003.backblazeb2.com";
const B2_REGION = "eu-central-003";
export const B2_BUCKET = "openbot-pool-archive-eu";
export const ARCHIVE_REGION = "eu";
export const PART_BYTES = 100 * 1024 * 1024;
export const SERVER_BYTES = `du -sb --one-file-system /home/user /srv/openbot-hosted | awk '{ total += $1 } END { print total }'`;
export const UPLOAD_URL_SECONDS = 6 * 60 * 60;
const BECH32_CHARSET = "qpzry9x8gf2tvdw0s3jn54khce6mua7l";
const BECH32_GENERATOR = [0x3b6a57b2, 0x26508e6d, 0x1ea119fa, 0x3d4233dd, 0x2a1462b3];
const WRAP_VERSION = "v1";
export interface B2Key {
id: string;
secret: string;
}
export interface UploadedPart {
number: number;
size: number;
md5: string;
etag: string;
}
export const archiveScript = (machineHosting: string) => `
set -euo pipefail
recipient=$1 part_bytes=$2
dir=$(mktemp -d /var/tmp/openbot-archive.XXXXXX)
trap 'rm -rf "$dir"' EXIT
cat >"$dir/urls"
cat >"$dir/part.sh" <<'PART'
set -euo pipefail
number=$((10#\${FILE##*.} + 1))
url=$(sed -n "\${number}p" "$URLS")
[ -n "$url" ] || { echo "No upload URL for part $number." >&2; exit 3; }
cat >"$FILE"
size=$(stat -c %s "$FILE")
md5=$(md5sum "$FILE" | cut -c1-32)
etag=$(curl -fsS --retry 5 --retry-all-errors -T "$FILE" -D - -o /dev/null "$url" | tr -d '\\r"' | sed -n 's/^[Ee][Tt][Aa][Gg]: *//p')
rm -f "$FILE"
echo "part $number $size $md5 $etag"
PART
mkfifo "$dir/sum"
sha256sum <"$dir/sum" | cut -c1-64 >"$dir/sha256" &
summer=$!
export URLS="$dir/urls"
bash ${machineHosting}/move-from-boat.sh export - | age -r "$recipient" | tee "$dir/sum" |
split -b "$part_bytes" -d -a 5 --filter="bash $dir/part.sh" - "$dir/part."
wait "$summer"
echo "sha256 $(cat "$dir/sha256")"
`;
export function shellWord(value: string) {
return `'${value.replaceAll("'", `'"'"'`)}'`;
}
export function uploadedParts(output: string) {
const parts: UploadedPart[] = output
.split("\n")
.filter((line) => line.startsWith("part "))
.map((line) => {
const [, number = "", size = "", md5 = "", etag = ""] = line.split(" ");
return { number: Number(number), size: Number(size), md5, etag };
});
const sha256 = /^sha256 ([0-9a-f]{64})$/mu.exec(output)?.[1] ?? "";
return { parts, sha256, bytes: parts.reduce((total, part) => total + part.size, 0) };
}
export function partCount(serverBytes: number) {
return Math.ceil((serverBytes * 1.05 + 256 * 1024 * 1024) / PART_BYTES) + 1;
}
function bech32Polymod(values: number[]) {
let checksum = 1;
for (const value of values) {
const top = checksum >>> 25;
checksum = ((checksum & 0x1ffffff) << 5) ^ value;
for (const [bit, generator] of BECH32_GENERATOR.entries()) {
if ((top >>> bit) & 1) checksum ^= generator;
}
}
return checksum;
}
function bech32(prefix: string, data: Buffer) {
const words: number[] = [];
let accumulator = 0;
let bits = 0;
for (const byte of data) {
accumulator = ((accumulator << 8) | byte) & 0xfff;
bits += 8;
while (bits >= 5) {
bits -= 5;
words.push((accumulator >>> bits) & 31);
}
}
if (bits > 0) words.push((accumulator << (5 - bits)) & 31);
const codes = [...prefix].map((character) => character.charCodeAt(0));
const expanded = [...codes.map((code) => code >>> 5), 0, ...codes.map((code) => code & 31)];
const mod = bech32Polymod([...expanded, ...words, 0, 0, 0, 0, 0, 0]) ^ 1;
const checksum = Array.from({ length: 6 }, (_, index) => (mod >>> (5 * (5 - index))) & 31);
return `${prefix}1${[...words, ...checksum].map((word) => BECH32_CHARSET.charAt(word)).join("")}`;
}
export function newArchiveKey() {
const { privateKey, publicKey } = generateKeyPairSync("x25519");
const secret = z.object({ d: z.string() }).parse(privateKey.export({ format: "jwk" })).d;
const shared = z.object({ x: z.string() }).parse(publicKey.export({ format: "jwk" })).x;
return {
identity: bech32("age-secret-key-", Buffer.from(secret, "base64url")).toUpperCase(),
recipient: bech32("age", Buffer.from(shared, "base64url")),
};
}
export function wrapKeySecret(value: string | undefined) {
const secret = Buffer.from(value ?? "", "base64");
if (secret.length !== 32) throw new Error("HOSTED_ARCHIVE_KEY_SECRET is not 32 bytes in base64.");
return secret;
}
export function wrapArchiveKey(secret: Buffer, identity: string, serverId: string, objectKey: string) {
const iv = randomBytes(12);
const cipher = createCipheriv("aes-256-gcm", secret, iv);
cipher.setAAD(Buffer.from(`${serverId}\n${objectKey}`, "utf8"));
const sealed = Buffer.concat([cipher.update(identity, "utf8"), cipher.final(), cipher.getAuthTag()]);
return `${WRAP_VERSION}.${iv.toString("base64url")}.${sealed.toString("base64url")}`;
}
function uriPart(value: string) {
return encodeURIComponent(value).replace(
/[!'()*]/gu,
(character) => `%${character.charCodeAt(0).toString(16).toUpperCase()}`,
);
}
export function presign(
b2: B2Key,
method: string,
objectKey: string,
parameters: Record<string, string>,
seconds: number,
) {
const now = new Date().toISOString().replace(/[-:]|\.\d{3}/gu, "");
const day = now.slice(0, 8);
const scope = `${day}/${B2_REGION}/s3/aws4_request`;
const query = Object.entries({
...parameters,
"X-Amz-Algorithm": "AWS4-HMAC-SHA256",
"X-Amz-Credential": `${b2.id}/${scope}`,
"X-Amz-Date": now,
"X-Amz-Expires": String(seconds),
"X-Amz-SignedHeaders": "host",
})
.map(([name, value]) => `${uriPart(name)}=${uriPart(value)}`)
.sort()
.join("&");
const path = `/${B2_BUCKET}/${objectKey.split("/").map(uriPart).join("/")}`;
const request = [method, path, query, `host:${B2_HOST}`, "", "host", "UNSIGNED-PAYLOAD"].join("\n");
const text = ["AWS4-HMAC-SHA256", now, scope, createHash("sha256").update(request).digest("hex")].join("\n");
const signingKey = [day, B2_REGION, "s3", "aws4_request"].reduce<Buffer>(
(key, part) => createHmac("sha256", key).update(part).digest(),
Buffer.from(`AWS4${b2.secret}`),
);
const signature = createHmac("sha256", signingKey).update(text).digest("hex");
return `https://${B2_HOST}${path}?${query}&X-Amz-Signature=${signature}`;
}
export async function b2Request(
b2: B2Key,
method: string,
objectKey: string,
parameters: Record<string, string>,
body?: string,
) {
const response = await fetch(presign(b2, method, objectKey, parameters, 900), {
method,
body: body ?? null,
signal: AbortSignal.timeout(120_000),
});
const text = method === "HEAD" ? "" : await response.text();
return { status: response.status, headers: response.headers, text };
}
export async function startUpload(b2: B2Key, objectKey: string) {
const created = await b2Request(b2, "POST", objectKey, { uploads: "" });
const uploadId = /<UploadId>([^<]+)<\/UploadId>/u.exec(created.text)?.[1] ?? "";
if (created.status !== 200 || uploadId === "") throw new Error(`B2 did not start the upload: ${created.status}.`);
return uploadId;
}
export function partUrls(b2: B2Key, objectKey: string, uploadId: string, count: number) {
return Array.from({ length: count }, (_, index) =>
presign(b2, "PUT", objectKey, { partNumber: String(index + 1), uploadId }, UPLOAD_URL_SECONDS),
).join("\n");
}
export async function completeUpload(b2: B2Key, objectKey: string, uploadId: string, parts: UploadedPart[]) {
const body = `<CompleteMultipartUpload>${parts
.map((part) => `<Part><PartNumber>${part.number}</PartNumber><ETag>"${part.etag}"</ETag></Part>`)
.join("")}</CompleteMultipartUpload>`;
const done = await b2Request(b2, "POST", objectKey, { uploadId }, body);
if (done.status !== 200 || done.text.includes("<Error>")) {
throw new Error(`B2 did not complete the upload: ${done.status}.`);
}
const head = await b2Request(b2, "HEAD", objectKey, {});
return {
status: head.status,
versionId: head.headers.get("x-amz-version-id") ?? "",
size: Number(head.headers.get("content-length") ?? "0"),
};
}
export async function abortUpload(b2: B2Key, objectKey: string, uploadId: string) {
await b2Request(b2, "DELETE", objectKey, { uploadId }).catch(() => undefined);
}
export async function deleteArchive(b2: B2Key, objectKey: string, versionId: string) {
const answer = await b2Request(b2, "DELETE", objectKey, versionId === "" ? {} : { versionId });
const head = await b2Request(b2, "HEAD", objectKey, {});
return { deleted: (answer.status === 204 || answer.status === 404) && head.status === 404, status: answer.status };
}