Files
NB 4fb65f5fea Plan 012 items 3, 5, 6: archive ended pool servers in B2 (#12928)
* Plan 012: archive ended pool servers in B2; prove export and import on a pool machine

Plan 012 moves the data of a Leaseweb pool server whose plan ended to an
encrypted archive in Backblaze B2, so the machine goes back to the pool within
a day. It follows #12721 and the dispute rule: 14 days of data, and a refunded
or early disputed server is deleted with no archive.

Item 1 is done. move-from-boat.sh export and import already work on a pool
machine; no pool mode is needed. scripts/hosting/pool-archive-e2e.ts proves it
on the test VPS pool-test-01: prepare, seed, export, reinstall, import, start.
6 of 6 checks pass, with 446 file facts the same after the import.

- export copied the cron spool file with its "DO NOT EDIT" header, and the
  import kept those lines as customer text. It now uses `crontab -l`. Moves
  from boat had the same bug.
- leaseweb-pool.ts packs the hosting folder with --no-xattrs: macOS tar added
  headers that GNU tar on the VPS warned about.
- apps/auth-api/.env.production: encrypted B2 keys for the EU bucket
  (operator and Worker). The Worker secrets come in item 2.

* Plan 012: encrypt pool archives with age and upload them to B2 from the machine

The E2E now runs the full archive path on a pool machine: sync makes an age
key and presigns B2 multipart part URLs, the machine encrypts the export and
uploads each part, sync checks each part MD5 and the size, then the machine is
reinstalled, downloads with a presigned URL, decrypts and imports.

leaseweb-bootstrap.sh installs age. The plan records the developer decisions
of 2026-11-19: one EU bucket for all locations, and age encryption.

A 5.75 GB server passes 9 of 9 checks (54,094 file facts). Upload 661 s, download
and decrypt 269 s, import 349 s. The import time is the disk speed of the VPS:
the same extract alone took 47 to 67 s.

Bash ignores PS4 from the environment as root, so the import trace sets it in
the shell.

* Plan 012 item 2: D1 table and Worker secrets for pool server archives

Migration 0061 adds hosted_server_archives: one row for each verified archive
in B2, with the wrapped age key, the B2 version id for a permanent delete, and
the time when the archive must be deleted. The table has no foreign key, so an
archive stays findable after a server or account delete. A deleted archive
keeps no key. 0060 is taken by #12845.

HOSTED_ARCHIVE_KEY_SECRET, B2_WORKER_KEY_ID_EU and B2_WORKER_APPLICATION_KEY_EU
are a set of three for the local deploy and the CI deploy. The values are in
.env.production and in the cloudflare-production Environment.

hosted_servers.plan_end_reason is not added: Stripe keeps the reason in
cancellation_details.reason, and sync reads the subscription fresh.

* Plan 012 items 3, 5, 6: archive ended pool servers in B2 from sync

sync archives a stopped pool server whose Stripe subscription is canceled (3 days later after a failed
payment), stores the wrapped age key in hosted_server_archives, frees the machine and reinstalls it.
Each sync deletes archives that are due, of a deleted server, or restored. The Worker 14-day delete
also finds an archived server, and deletes the old boat sandbox of a moved server with no machine.
PRIVACY.md and the DPA name Backblaze B2 for the encrypted archive.

Hold the first production archive until the restore (item 4) is live: a renewal of an archived
server now gets a new, empty server.

* Plan 012 item 4: restore an archived pool server when its plan is renewed

A renewal of a pool server with a live archive takes a ready pool machine.
The assignment answer gives the machine a presigned B2 URL, the age key
and the checksum. openbot-pool-wait runs move-from-boat.sh restore, reports
"restored", and then starts OpenBot. A machine that does not send
restoreVersion 1 gets no restore. A restore with no report after 3 hours
goes back to awaiting_payment, and the machine leaves the pool.

The status shows state "starting" with "restoring". Clients show the start
step "restore" ("Restoring data") with no 15-minute limit, and settings and
mobile show until which date the data of an ended pool server is kept.

Tests: hosted-pool-restore-e2e (Worker flow, 25 checks) and
pool-archive-e2e on pool-test-01 (real B2 archive, 10 checks).

* Remove a code comment and an old plan dependency

AGENTS.md forbids code comments. The reason for crontab -l stays here:
the spool file starts with the "DO NOT EDIT" header of cron, and crontab
on the VPS would keep those lines as text of the customer. Plan 012 no
longer waits for a B2 US account or a Stripe decision.

* Share the pool machine setup between the pool script and the E2E

Move the hosting copy and the prepare and upgrade scripts to pool-machine.ts.
The pool script and the archive E2E use this module. Remove the copies.

* Give fetch a null body when the B2 request has no body

The types ratchet does not allow an undefined value for an optional property.

* Clear the pre-stop marker before a pool start and share the size command

An archive that stops early leaves /run/openbot-pre-stop on the machine.
openbot.service then skips its start, but sync records the server as running.
The start command now removes the marker first.

SERVER_BYTES moves to pool-archive.ts. The pool script and the E2E use it.

* Let knip read the auth-api E2E scripts

The restore E2E is the only user of the archive restore helpers, so knip
reported them as unused exports.

* Refuse each 401 to a restore report

Before, openbot-pool-wait read a 401 after a lost answer as success, and
could start OpenBot on a machine that the Worker took back. The Worker
now keeps the offer token of a restored machine for 2 hours, which is
longer than the report retries. A retry after a lost answer gets
"restored", and each 401 stops the start.
2026-11-19 21:41:39 +01:00

102 lines
4.1 KiB
Bash
Executable File

#!/bin/bash
set -euo pipefail
export LC_ALL=C
umask 077
STATE=/etc/openbot-pool
INSTALL=/opt/OpenBot/hosted
WORK=${RUNTIME_DIRECTORY:-/run/openbot-pool}
INTERVAL=10
MOVE=$(dirname "$(readlink -f "$0")")/move-from-boat.sh
REPORT_TRIES=90
fail() {
echo "$*" >&2
exit 1
}
finish() {
rm -f "$STATE/token"
systemctl disable --quiet openbot-pool-wait.service || true
echo "$*"
exit 0
}
[ "$(id -u)" -eq 0 ] || fail "Run this as root."
[ -s "$STATE/token" ] || finish "This machine has no offer."
host=$(sed -n '1p' "$STATE/host" 2>/dev/null || true)
[[ $host =~ ^leaseweb:[0-9]{1,20}$ ]] || fail "$STATE/host has no machine ID."
token=$(sed -n '1p' "$STATE/token")
[[ $token =~ ^[A-Za-z0-9_-]{32,128}$ ]] || fail "$STATE/token is not a token."
user=$(cat "$INSTALL/service-user" 2>/dev/null || true)
home=$(getent passwd "$user" | cut -d: -f6 || true)
[ -n "$user" ] && [ "$user" != root ] && [ -d "$home" ] || fail "This machine has no service user. Prepare it first."
url=$(sed -n 's/^OPENBOT_AUTH_API_URL=//p' "$INSTALL/openbot.env" 2>/dev/null | head -n 1)
[[ $url =~ ^https://[A-Za-z0-9.-]+$ ]] || fail "$INSTALL/openbot.env has no account service."
install -d -m 0700 "$WORK"
request=$WORK/request
answer=$WORK/answer
restore_version=""
if [ -f "$MOVE" ] && command -v age >/dev/null; then restore_version=',"restoreVersion":1'; fi
printf '{"hostId":"%s","token":"%s"%s}' "$host" "$token" "$restore_version" >"$request"
echo "Machine $host waits for a server."
while :; do
: >"$answer"
status=$(curl --silent --max-time 30 --proto '=https' --header 'content-type: application/json' \
--data-binary "@$request" --output "$answer" --write-out '%{http_code}' \
"$url/v2/hosting/pool/assignment" || true)
if [ "$status" = 401 ]; then
finish "The account service does not know the token of this machine. The offer is over."
fi
if [ "$status" = 200 ] && grep -q '"state":"assigned"' "$answer"; then
server=$(sed -n 's/.*"serverId":"\([0-9a-fA-F-]\{36\}\)".*/\1/p' "$answer" | head -n 1)
claim=$(sed -n 's/.*"claim":"\([A-Za-z0-9_-]\{16,512\}\)".*/\1/p' "$answer" | head -n 1)
archive_url="" key="" sum=""
if grep -q '"restore":{' "$answer"; then
archive_url=$(sed -n 's/.*"url":"\(https:\/\/[^"]*\)".*/\1/p' "$answer" | head -n 1)
key=$(sed -n 's/.*"key":"\(AGE-SECRET-KEY-1[0-9A-Z]\{58\}\)".*/\1/p' "$answer" | head -n 1)
sum=$(sed -n 's/.*"sha256":"\([0-9a-f]\{64\}\)".*/\1/p' "$answer" | head -n 1)
[ -n "$restore_version" ] && [ -n "$archive_url" ] && [ -n "$key" ] && [ -n "$sum" ] || server=""
fi
rm -f "$answer"
if [ -n "$server" ] && [ -n "$claim" ]; then break; fi
echo "The answer has no server ID, no claim or no complete restore. Ask again." >&2
fi
sleep "$INTERVAL"
done
report_restore() {
local report=$WORK/report status
printf '{"hostId":"%s","token":"%s","restoreVersion":1,"restored":"%s"}' "$host" "$token" "$server" >"$report"
for _ in $(seq "$REPORT_TRIES"); do
: >"$answer"
status=$(curl --silent --max-time 30 --proto '=https' --header 'content-type: application/json' \
--data-binary "@$report" --output "$answer" --write-out '%{http_code}' \
"$url/v2/hosting/pool/assignment" || true)
if [ "$status" = 200 ] && grep -q '"state":"restored"' "$answer"; then return 0; fi
if [ "$status" = 401 ]; then
fail "The account service took server $server back. Its data is on this machine. Do not start OpenBot."
fi
sleep "$INTERVAL"
done
fail "The account service did not get the restore of server $server. Its data is on this machine."
}
if [ -n "$key" ]; then
echo "Restore server $server from its archive."
printf '%s\n%s\n' "$key" "$archive_url" | OPENBOT_POOL_RESTORE=1 bash "$MOVE" restore "$sum"
unset key archive_url
fi
OPENBOT_HOSTED_HOST_ID=$server OPENBOT_HOSTED_CLAIM=$claim HOME=$home \
setpriv --reuid "$user" --regid "$(id -gn "$user")" --init-groups "$INSTALL/openbot-hosted-env"
unset claim
if [ -n "$sum" ]; then
report_restore
bash "$MOVE" start
finish "Server $server is restored and starts on machine $host."
fi
systemctl enable --now openbot.service
finish "Server $server starts on machine $host."