mirror of
https://github.com/nightly-labs/openbot.git
synced 2026-11-23 02:01:26 +08:00
* Start a hosted server again when its main process stops responding On 2026-11-21 a hosted server became short of memory. The main process then stopped in a zygote read and did not respond again. systemd did not start the service again, because the process did not exit. The server stayed offline until an operator restarted it. The launcher now starts a watchdog. Each 30 seconds it reads the run marker, which main writes each 60 seconds. When the file does not change for 20 reads (10 minutes), the watchdog stops main with SIGKILL, and systemd starts the service again. The watchdog compares the file content, not clocks, so a boat resume does not start it. It does not kill when main did not write the file in this run, when the disk has less than 1 MB free, or when the pid is not openbot. scripts/hosted-watchdog-e2e.ts tests the watchdog with fake apps, and CI runs it in the reverse-actions-e2e job. * Keep the watchdog away from the start and the server safe mode The local review found two cases where the watchdog stopped a process that did not hang: - The server safe mode calls recordExit, which stops the run marker heartbeat, and then stays alive with no time limit on a hosted server. The watchdog killed it every 10 minutes, and the SIGKILL got past RestartPreventExitStatus=78. - Migrations and the database quick_check run on the main thread. On a large database they can block the heartbeat for more than 10 minutes. The kill rolled back the migration, and the next start tried again. Now the watchdog counts a read only when the run marker has "state":"running" and openbot-start-attempts-v1.json has "attempt":null, which StartAttempts writes after the renderer loads. The watchdog gets the userData folder in place of the marker path. --stop and the watchdog now use one helper that checks the pid, the process name and the cgroup. The E2E test adds the starting and quitting cases, and starts each fake app only after the first watchdog read, so the watchdog always sees a change. * Name the release notes file after the PR branch