mirror of
https://github.com/Tencent/BrowserSkill.git
synced 2026-10-02 07:34:35 +08:00
fix(update): verify the takeover and own the restart before rolling back
- `bsk update` restarts the daemon as a process it owns. One that is not ready in time is stopped and reaped before the previous version is restarted, so the rollback no longer meets it holding the daemon lock. - A handover or restart succeeds only when a daemon of the release's version serves the original port. `--version` must print that version. - A failed handover records a serving daemon only once the resumed one has published daemon.json, and appends the error if it cannot serve again. - A daemon keeps the executable path it started from, so it can update again after a rollback on Linux, where `current_exe` then names a deleted file. - The update lock sits next to the installed executable, so updates from different bsk homes are serialized through installation and rollback.
This commit is contained in:
drakezhang
+15
-7
@@ -11,17 +11,25 @@ Starting from 0.2.0, CLI / Extension / DSH Plugin share the same version number.
|
||||
|
||||
- A failed auto-update no longer leaves the browser disconnected
|
||||
([#336](https://github.com/Tencent/BrowserSkill/issues/336)). The daemon
|
||||
runs the new executable once before relying on it, starts a daemon from it,
|
||||
and exits only after that daemon answers. If the new daemon exits or is not
|
||||
ready within 20 seconds, the previous executable is put back and the running
|
||||
daemon serves again on the same port; the release is retried after 6 hours.
|
||||
checks that the new executable reports the release's version, starts a
|
||||
daemon from it, and exits only after a daemon of that version serves the
|
||||
same port. If the new daemon exits or is not ready within 20 seconds, it is
|
||||
stopped, the previous executable is put back and the running daemon serves
|
||||
again on the same port; the release is retried after 6 hours. The update
|
||||
record says a daemon is serving only once it has published `daemon.json`.
|
||||
This applies on all platforms.
|
||||
- Windows self-update no longer depends on a detached script: the running
|
||||
`bsk.exe` is renamed aside and the new one takes its place.
|
||||
- `bsk update` installs and checks the new executable before stopping the
|
||||
daemon, and puts the previous executable back if the restarted daemon does
|
||||
not become ready. It restarts the daemon from the installed path, which
|
||||
Linux no longer reports as the current executable once it is replaced.
|
||||
daemon. If the restarted daemon is not ready in time, it is stopped, the
|
||||
previous executable is put back and the previous version restarted. It
|
||||
restarts the daemon from the installed path, which Linux no longer reports
|
||||
as the current executable once it is replaced.
|
||||
- A daemon keeps using the executable path it started from, so it can update
|
||||
again after a rolled-back update on Linux.
|
||||
- Updates of one executable are serialized through a lock file next to it, so
|
||||
daemons or `bsk update` runs with different bsk homes cannot overwrite each
|
||||
other's installation or rollback.
|
||||
|
||||
### Changed
|
||||
|
||||
|
||||
@@ -250,9 +250,9 @@ bsk update --yes
|
||||
|
||||
For the default local setup, this installs the new executable and checks that it runs while the daemon keeps serving, then restarts the daemon from it on the same port. If the restarted daemon does not become ready, the previous executable is put back and the daemon is restarted from it. A daemon started with `--foreground` is left running on the previous version; restart it in its terminal or supervisor. If you use the installer to replace the binary, restart the daemon afterwards with `bsk daemon restart`.
|
||||
|
||||
A daemon that `bsk` started in the background also checks for a new release every 30 minutes and, while no agent session is active, installs it the same way. It then starts a daemon from the new executable and exits only once that daemon answers. If the new daemon exits or is not ready within 20 seconds, the running daemon puts the previous executable back, serves again on the same port, and retries that release after 6 hours. Browser connections drop for a moment during a handover and reconnect. A daemon started with `--foreground` belongs to its terminal or supervisor, so it only reports new releases: run `bsk update`, then restart the daemon there. If bsk cannot write next to its executable, it only reports new releases; update it with the installer or package manager you used. Set `BSK_AUTO_UPDATE=off` to disable daemon-side auto-update while keeping manual `bsk update` available.
|
||||
A daemon that `bsk` started in the background also checks for a new release every 30 minutes and, while no agent session is active, installs it the same way. It then starts a daemon from the new executable and exits only once a daemon of the new version serves the same port. If the new daemon exits or is not ready within 20 seconds, it is stopped, and the running daemon puts the previous executable back, serves again on the same port, and retries that release after 6 hours. Browser connections drop for a moment during a handover and reconnect. A daemon started with `--foreground` belongs to its terminal or supervisor, so it only reports new releases: run `bsk update`, then restart the daemon there. If bsk cannot write next to its executable, it only reports new releases; update it with the installer or package manager you used. Set `BSK_AUTO_UPDATE=off` to disable daemon-side auto-update while keeping manual `bsk update` available.
|
||||
|
||||
`bsk doctor` shows the last update attempt; `update-state.json` in the bsk home keeps its stage, result and error. Until an update is confirmed, the previous executable stays next to the new one as `.bsk.old-*` (`.bsk.exe.old-*` on Windows); one that is still running remains until its process exits, and a later daemon start removes it. On Windows, self-update renames the running executable, which NTFS supports; where the file system refuses, the update fails and leaves the installation unchanged. bsk supports Windows 10 and Windows Server 2016 or later.
|
||||
`bsk doctor` shows the last update attempt; `update-state.json` in the bsk home keeps its stage, result and error. Updates of one executable run one at a time, even from different bsk homes, through `.bsk.update.lock` (`.bsk.exe.update.lock` on Windows), which stays next to it. Until an update is confirmed, the previous executable stays next to the new one as `.bsk.old-*` (`.bsk.exe.old-*` on Windows); one that is still running remains until its process exits, and a later daemon start removes it. On Windows, self-update renames the running executable, which NTFS supports; where the file system refuses, the update fails and leaves the installation unchanged. bsk supports Windows 10 and Windows Server 2016 or later.
|
||||
|
||||
Update the extension through its browser store. Update the DSH plugin separately, then restart its profile:
|
||||
|
||||
|
||||
+2
-2
@@ -250,9 +250,9 @@ bsk update --yes
|
||||
|
||||
默认本地配置下，这条命令会在 daemon 继续服务的同时安装新版本并检查它能否运行，然后用新版本在原端口重启 daemon。重启后的 daemon 没有就绪时，会放回旧版本并用它重启 daemon。通过 `--foreground` 启动的 daemon 不会被停止，会继续运行旧版本，请在它所在的终端或进程管理器中重启。如果使用安装脚本替换了二进制，请随后执行 `bsk daemon restart`。
|
||||
|
||||
由 `bsk` 在后台启动的 daemon 还会每 30 分钟检查一次新版本。没有 Agent 会话时，它按同样方式安装新版本，再用新版本启动一个 daemon，等新 daemon 开始响应后才退出。新 daemon 退出，或 20 秒内没有就绪时，当前 daemon 会放回旧版本，在原端口继续服务，6 小时后再尝试这个版本。交接期间浏览器连接会短暂断开并自动重连。通过 `--foreground` 启动的 daemon 归所在终端或进程管理器管理，只提示新版本：执行 `bsk update`，再在那里重启 daemon。bsk 无法在可执行文件所在目录写入时，也只提示新版本，请用原来的安装脚本或包管理器升级。设置 `BSK_AUTO_UPDATE=off` 可关闭 daemon 自动升级，手动 `bsk update` 仍然可用。
|
||||
由 `bsk` 在后台启动的 daemon 还会每 30 分钟检查一次新版本。没有 Agent 会话时，它按同样方式安装新版本，再用新版本启动一个 daemon，确认新版本的 daemon 已在原端口服务后才退出。新 daemon 退出，或 20 秒内没有就绪时，它会被停止，当前 daemon 放回旧版本，在原端口继续服务，6 小时后再尝试这个版本。交接期间浏览器连接会短暂断开并自动重连。通过 `--foreground` 启动的 daemon 归所在终端或进程管理器管理，只提示新版本：执行 `bsk update`，再在那里重启 daemon。bsk 无法在可执行文件所在目录写入时，也只提示新版本，请用原来的安装脚本或包管理器升级。设置 `BSK_AUTO_UPDATE=off` 可关闭 daemon 自动升级，手动 `bsk update` 仍然可用。
|
||||
|
||||
`bsk doctor` 会显示最近一次更新的结果；bsk home 下的 `update-state.json` 记录了它的阶段、结果和错误原因。更新确认成功前，旧版本会以 `.bsk.old-*`（Windows 上为 `.bsk.exe.old-*`）的名字留在新版本旁边；仍在运行的旧版本会保留到对应进程退出，之后再启动 daemon 时会清理。Windows 上的自更新依赖重命名正在运行的可执行文件，NTFS 支持这一操作；文件系统不支持时，更新会失败，已安装的版本保持不变。bsk 支持 Windows 10 和 Windows Server 2016 及以上版本。
|
||||
`bsk doctor` 会显示最近一次更新的结果；bsk home 下的 `update-state.json` 记录了它的阶段、结果和错误原因。同一个可执行文件的更新会依次进行，即使来自不同的 bsk home 也是如此，靠的是它旁边的 `.bsk.update.lock`（Windows 上为 `.bsk.exe.update.lock`），这个文件会一直保留。更新确认成功前，旧版本会以 `.bsk.old-*`（Windows 上为 `.bsk.exe.old-*`）的名字留在新版本旁边；仍在运行的旧版本会保留到对应进程退出，之后再启动 daemon 时会清理。Windows 上的自更新依赖重命名正在运行的可执行文件，NTFS 支持这一操作；文件系统不支持时，更新会失败，已安装的版本保持不变。bsk 支持 Windows 10 和 Windows Server 2016 及以上版本。
|
||||
|
||||
扩展通过浏览器商店更新。DSH 插件需要单独更新，完成后重启对应 profile：
|
||||
|
||||
|
||||
@@ -375,7 +375,7 @@ fn install_candidate_with_client(
|
||||
restart_daemon: bool,
|
||||
client: &reqwest::blocking::Client,
|
||||
) -> Result<DaemonRestart> {
|
||||
let target = std::env::current_exe().context("locate current bsk executable")?;
|
||||
let target = installed_executable()?.to_path_buf();
|
||||
if let Err(err) = ensure_replaceable(&target) {
|
||||
UpdateRecord::skipped(
|
||||
UpdateSource::Command,
|
||||
@@ -387,7 +387,7 @@ fn install_candidate_with_client(
|
||||
.save();
|
||||
return Err(err.context(installer_hint(&target)));
|
||||
}
|
||||
let _lock = UpdateLock::try_acquire()?;
|
||||
let _lock = UpdateLock::try_acquire(&target)?;
|
||||
let mut record = UpdateRecord::start(UpdateSource::Command, &candidate.latest, &target);
|
||||
record.save();
|
||||
let installed = install_verified(candidate, &target, client, &mut record)?;
|
||||
@@ -424,10 +424,12 @@ fn install_candidate_with_client(
|
||||
port: running.map(|daemon| daemon.ws_port),
|
||||
..StartArgs::default()
|
||||
};
|
||||
// Start from `target`: once it is replaced, Linux no longer reports it as
|
||||
// the current executable.
|
||||
let start = || crate::daemon::start::start_detached(&target, &args);
|
||||
match start() {
|
||||
// The started process is ours to stop if it is not ready in time, so a
|
||||
// restart of the previous version never meets a stuck one holding the
|
||||
// daemon lock.
|
||||
let start =
|
||||
|version: &Version| crate::daemon::start::start_owned(&target, &args, &version.to_string());
|
||||
match start(&candidate.latest) {
|
||||
Ok(daemon) => {
|
||||
record.succeed(Some((daemon.pid, daemon.version)));
|
||||
installed.discard();
|
||||
@@ -436,7 +438,13 @@ fn install_candidate_with_client(
|
||||
Err(err) => {
|
||||
let err = err.context("restart the daemon from the new version");
|
||||
let recovery = installed.roll_back(|| Recovery::Restored {
|
||||
daemon_serving: start().is_ok(),
|
||||
daemon_serving: match start(&candidate.current) {
|
||||
Ok(_) => true,
|
||||
Err(restart) => {
|
||||
tracing::error!(error = %format_args!("{restart:#}"), "could not restart the previous version");
|
||||
false
|
||||
}
|
||||
},
|
||||
});
|
||||
let restored = matches!(recovery, Recovery::Restored { .. });
|
||||
record.fail(&err, recovery);
|
||||
@@ -498,7 +506,7 @@ pub(crate) fn install_verified(
|
||||
};
|
||||
record.previous_executable = Some(installed.previous.clone());
|
||||
record.save();
|
||||
if let Err(err) = verify_executable(target) {
|
||||
if let Err(err) = verify_executable(target, &candidate.latest) {
|
||||
let err = err.context("the new executable failed its self-check");
|
||||
record.fail(&err, installed.roll_back(|| Recovery::Unchanged));
|
||||
return Err(err);
|
||||
@@ -1050,9 +1058,10 @@ pub(crate) fn ensure_replaceable(target: &Path) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Run the new executable once, so a binary that cannot start on this system
|
||||
/// is caught before any daemon depends on it.
|
||||
fn verify_executable(exe: &Path) -> Result<()> {
|
||||
/// Run the new executable once, so a binary that cannot start on this system,
|
||||
/// or is not the version the manifest names, is caught before any daemon
|
||||
/// depends on it.
|
||||
fn verify_executable(exe: &Path, version: &Version) -> Result<()> {
|
||||
let mut command = std::process::Command::new(exe);
|
||||
command
|
||||
.arg("--version")
|
||||
@@ -1089,14 +1098,25 @@ fn verify_executable(exe: &Path) -> Result<()> {
|
||||
let _ = pipe.read_to_string(&mut stdout);
|
||||
}
|
||||
anyhow::ensure!(
|
||||
status.success() && stdout.starts_with("bsk "),
|
||||
"`{} --version` exited with {status} and printed {:?}",
|
||||
status.success() && stdout.trim() == format!("bsk {version}"),
|
||||
"`{} --version` exited with {status} and printed {:?}; expected \"bsk {version}\"",
|
||||
exe.display(),
|
||||
stdout.trim()
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The executable this process was started from, captured on first use. On
|
||||
/// Linux `current_exe` names a replaced executable `<path> (deleted)` from then
|
||||
/// on, even after a rollback puts one back at the path, so a daemon captures it
|
||||
/// at startup and uses it for its whole life.
|
||||
pub(crate) fn installed_executable() -> Result<&'static Path> {
|
||||
static EXE: std::sync::OnceLock<Option<PathBuf>> = std::sync::OnceLock::new();
|
||||
EXE.get_or_init(|| std::env::current_exe().ok())
|
||||
.as_deref()
|
||||
.context("locate current bsk executable")
|
||||
}
|
||||
|
||||
/// Best-effort removal of what earlier updates left next to `exe`: previous
|
||||
/// executables and staged binaries whose owning process has exited, plus
|
||||
/// files of the former Windows script updater. A previous executable that is
|
||||
@@ -1451,20 +1471,94 @@ mod tests {
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn self_check_requires_a_bsk_version_line() {
|
||||
fn self_check_requires_the_manifest_version() {
|
||||
let tmp = tempfile::TempDir::new().unwrap();
|
||||
verify_executable(&script(tmp.path(), "echo 'bsk 9.9.9'")).unwrap();
|
||||
for body in ["echo 'bsk 9.9.9'; exit 3", "echo 'not bsk'", "exit 0"] {
|
||||
let error = verify_executable(&script(tmp.path(), body)).unwrap_err();
|
||||
assert!(
|
||||
format!("{error:#}").contains("--version"),
|
||||
"{body}: {error:#}"
|
||||
);
|
||||
let version = Version::new(9, 9, 9);
|
||||
verify_executable(&script(tmp.path(), "echo 'bsk 9.9.9'"), &version).unwrap();
|
||||
for body in [
|
||||
"echo 'bsk 9.9.9'; exit 3",
|
||||
"echo 'bsk 9.9.8'",
|
||||
"echo 'bsk 9.9.9-rc.1'",
|
||||
"echo 'not bsk'",
|
||||
"exit 0",
|
||||
] {
|
||||
let error = verify_executable(&script(tmp.path(), body), &version).unwrap_err();
|
||||
let error = format!("{error:#}");
|
||||
assert!(error.contains("expected \"bsk 9.9.9\""), "{body}: {error}");
|
||||
}
|
||||
let error = verify_executable(&tmp.path().join("missing")).unwrap_err();
|
||||
let error = verify_executable(&tmp.path().join("missing"), &version).unwrap_err();
|
||||
assert!(format!("{error:#}").contains("missing"), "{error:#}");
|
||||
}
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
#[test]
|
||||
#[ignore = "subprocess entry point"]
|
||||
fn rolled_back_process() {
|
||||
// Captured before the parent replaces this executable, as a daemon
|
||||
// does at startup.
|
||||
let captured = installed_executable().unwrap().to_path_buf();
|
||||
let dir = PathBuf::from(std::env::var_os("BSK_TEST_DIR").unwrap());
|
||||
std::fs::write(dir.join("ready"), "").unwrap();
|
||||
while !dir.join("go").exists() {
|
||||
std::thread::sleep(Duration::from_millis(20));
|
||||
}
|
||||
let current = std::env::current_exe().unwrap();
|
||||
let installed = install_binary(&captured, b"second release");
|
||||
std::fs::write(
|
||||
dir.join("result"),
|
||||
format!(
|
||||
"{}\n{}\n{}",
|
||||
current.display(),
|
||||
captured.display(),
|
||||
installed.map_or_else(|err| format!("{err:#}"), |_| "installed".into())
|
||||
),
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
#[test]
|
||||
fn a_rolled_back_process_can_install_again() {
|
||||
let tmp = tempfile::TempDir::new().unwrap();
|
||||
let target = tmp.path().join("bsk");
|
||||
std::fs::copy(std::env::current_exe().unwrap(), &target).unwrap();
|
||||
let original = std::fs::read(&target).unwrap();
|
||||
let mut child = std::process::Command::new(&target)
|
||||
.args([
|
||||
"--exact",
|
||||
"cli::update::tests::rolled_back_process",
|
||||
"--ignored",
|
||||
])
|
||||
.env("BSK_TEST_DIR", tmp.path())
|
||||
.stdout(std::process::Stdio::null())
|
||||
.spawn()
|
||||
.unwrap();
|
||||
let deadline = Instant::now() + Duration::from_secs(30);
|
||||
while !tmp.path().join("ready").exists() {
|
||||
assert!(Instant::now() < deadline, "helper did not start");
|
||||
std::thread::sleep(Duration::from_millis(20));
|
||||
}
|
||||
|
||||
// An update of the running process fails and is rolled back.
|
||||
install_binary(&target, b"first release")
|
||||
.unwrap()
|
||||
.restore()
|
||||
.unwrap();
|
||||
assert_eq!(std::fs::read(&target).unwrap(), original);
|
||||
std::fs::write(tmp.path().join("go"), "").unwrap();
|
||||
assert!(child.wait().unwrap().success());
|
||||
|
||||
let result = std::fs::read_to_string(tmp.path().join("result")).unwrap();
|
||||
let lines: Vec<_> = result.lines().collect();
|
||||
assert!(
|
||||
lines[0].ends_with(" (deleted)"),
|
||||
"Linux no longer names the rolled-back executable: {result}"
|
||||
);
|
||||
assert_eq!(lines[1], target.display().to_string(), "{result}");
|
||||
assert_eq!(lines[2], "installed", "{result}");
|
||||
assert_eq!(std::fs::read(&target).unwrap(), b"second release");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn leftovers_belong_to_their_process_until_it_exits() {
|
||||
let tmp = tempfile::TempDir::new().unwrap();
|
||||
|
||||
@@ -175,6 +175,40 @@ impl UpdateRecord {
|
||||
self.save();
|
||||
}
|
||||
|
||||
/// A failed attempt's previous version serves again: its daemon has bound
|
||||
/// its endpoints and published `daemon.json`.
|
||||
pub(crate) fn confirm_serving(&mut self) {
|
||||
if self.mark_serving() {
|
||||
self.save();
|
||||
}
|
||||
}
|
||||
|
||||
fn mark_serving(&mut self) -> bool {
|
||||
let Some(Recovery::Restored { daemon_serving }) = &mut self.recovery else {
|
||||
return false;
|
||||
};
|
||||
*daemon_serving = true;
|
||||
self.updated_at_epoch_secs = now_epoch_secs();
|
||||
true
|
||||
}
|
||||
|
||||
/// A failed attempt's previous version could not serve again either.
|
||||
pub(crate) fn serving_failed(&mut self, error: &anyhow::Error) {
|
||||
self.note_serving_failure(error);
|
||||
self.save();
|
||||
}
|
||||
|
||||
fn note_serving_failure(&mut self, error: &anyhow::Error) {
|
||||
let handover = self.error.take().unwrap_or_default();
|
||||
self.error = Some(format!(
|
||||
"{handover}; the previous version could not serve again: {error:#}"
|
||||
));
|
||||
if let Some(Recovery::Restored { daemon_serving }) = &mut self.recovery {
|
||||
*daemon_serving = false;
|
||||
}
|
||||
self.updated_at_epoch_secs = now_epoch_secs();
|
||||
}
|
||||
|
||||
/// When a daemon may next try `target`, if an earlier failure defers it.
|
||||
pub(crate) fn retry_blocked_until(&self, target: &Version, now: u64) -> Option<u64> {
|
||||
let retry_after = self.retry_after_epoch_secs?;
|
||||
@@ -227,15 +261,15 @@ pub fn write(path: &Path, record: &UpdateRecord) -> Result<()> {
|
||||
super::write_json_atomically(path, record)
|
||||
}
|
||||
|
||||
/// Serializes update attempts that share a bsk home, so an automatic and a
|
||||
/// manual update never swap the executable at the same time.
|
||||
/// Serializes update attempts on one installed executable, whichever bsk
|
||||
/// home they run for, from installation through confirmation or rollback.
|
||||
/// The lock file (`.<name>.update.lock`) stays next to the executable.
|
||||
#[derive(Debug)]
|
||||
pub(crate) struct UpdateLock(File);
|
||||
|
||||
impl UpdateLock {
|
||||
pub(crate) fn try_acquire() -> Result<Self> {
|
||||
paths::ensure_bsk_home()?;
|
||||
Self::try_acquire_at(&paths::update_lock_path()?)
|
||||
pub(crate) fn try_acquire(target: &Path) -> Result<Self> {
|
||||
Self::try_acquire_at(&super::sibling(target, "update.lock")?)
|
||||
}
|
||||
|
||||
fn try_acquire_at(path: &Path) -> Result<Self> {
|
||||
@@ -333,6 +367,24 @@ mod tests {
|
||||
assert_eq!(succeeded.retry_blocked_until(&TARGET, 0), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_lock_belongs_to_the_installation_not_the_bsk_home() {
|
||||
let tmp = tempfile::TempDir::new().unwrap();
|
||||
let shared = tmp.path().join("bin").join("bsk");
|
||||
let other = tmp.path().join("other").join("bsk");
|
||||
for exe in [&shared, &other] {
|
||||
std::fs::create_dir_all(exe.parent().unwrap()).unwrap();
|
||||
}
|
||||
// Two daemons with different bsk homes update the same executable.
|
||||
let first = UpdateLock::try_acquire(&shared).unwrap();
|
||||
let error = UpdateLock::try_acquire(&shared).unwrap_err();
|
||||
assert!(format!("{error:#}").contains("another bsk update is in progress"));
|
||||
let _unrelated = UpdateLock::try_acquire(&other).unwrap();
|
||||
drop(first);
|
||||
UpdateLock::try_acquire(&shared).unwrap();
|
||||
assert!(tmp.path().join("bin").join(".bsk.update.lock").exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_one_update_attempt_holds_the_lock_at_a_time() {
|
||||
let tmp = tempfile::TempDir::new().unwrap();
|
||||
@@ -359,6 +411,44 @@ mod tests {
|
||||
UpdateLock::try_acquire_at(&path).expect("released after the winner finishes");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_resumed_service_is_confirmed_or_its_failure_appended() {
|
||||
let restored = |serving| UpdateRecord {
|
||||
result: UpdateResult::Failed,
|
||||
error: Some("replacement exited".into()),
|
||||
recovery: Some(Recovery::Restored {
|
||||
daemon_serving: serving,
|
||||
}),
|
||||
..UpdateRecord::start(UpdateSource::Daemon, &TARGET, Path::new("bsk"))
|
||||
};
|
||||
let mut confirmed = restored(false);
|
||||
assert!(confirmed.mark_serving());
|
||||
assert_eq!(
|
||||
confirmed.recovery,
|
||||
Some(Recovery::Restored {
|
||||
daemon_serving: true
|
||||
})
|
||||
);
|
||||
|
||||
let mut unchanged = UpdateRecord {
|
||||
recovery: Some(Recovery::Unchanged),
|
||||
..restored(false)
|
||||
};
|
||||
assert!(!unchanged.mark_serving(), "only a restored daemon resumes");
|
||||
|
||||
let mut failed = restored(false);
|
||||
failed.note_serving_failure(&anyhow::anyhow!("bind WS server: address in use"));
|
||||
assert_eq!(
|
||||
failed.recovery,
|
||||
Some(Recovery::Restored {
|
||||
daemon_serving: false
|
||||
})
|
||||
);
|
||||
let error = failed.error.unwrap();
|
||||
assert!(error.starts_with("replacement exited; "), "{error}");
|
||||
assert!(error.contains("address in use"), "{error}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn skip_records_name_their_version_and_reason() {
|
||||
let record = UpdateRecord::skipped(
|
||||
|
||||
@@ -110,11 +110,6 @@ pub fn update_state_path() -> Result<PathBuf> {
|
||||
Ok(bsk_home()?.join("update-state.json"))
|
||||
}
|
||||
|
||||
/// Held for the whole of one update attempt (`update.lock`).
|
||||
pub fn update_lock_path() -> Result<PathBuf> {
|
||||
Ok(bsk_home()?.join("update.lock"))
|
||||
}
|
||||
|
||||
/// Why a replacement daemon failed to start, for the daemon handing over to it.
|
||||
pub fn replacement_failure_path(pid: u32) -> Result<PathBuf> {
|
||||
Ok(bsk_home()?
|
||||
|
||||
@@ -23,6 +23,7 @@ use tracing::{debug, error, info, warn};
|
||||
|
||||
use crate::cli::daemon::StartArgs;
|
||||
use crate::cli::ensure_daemon::SPAWN_DEADLINE;
|
||||
use crate::cli::update::state::UpdateRecord;
|
||||
use crate::daemon::{
|
||||
browsers::{BROWSER_LIVENESS_TICK, BROWSER_LIVENESS_TIMEOUT, EXTENSION_CONNECT_WAIT},
|
||||
info as daemon_info, ipc, lockfile, paths,
|
||||
@@ -172,6 +173,28 @@ pub(crate) fn start_detached(exe: &Path, args: &StartArgs) -> Result<daemon_info
|
||||
start_background_at(exe, args, deadline)
|
||||
}
|
||||
|
||||
/// Start a daemon from `exe` for `bsk update`, and wait until a daemon of
|
||||
/// `version` serves `args`' port. Unlike [`start_detached`], the process is
|
||||
/// this caller's: one that is not ready in time is stopped and reaped, so a
|
||||
/// restart of the previous version never meets it holding the daemon lock.
|
||||
pub(crate) fn start_owned(
|
||||
exe: &Path,
|
||||
args: &StartArgs,
|
||||
version: &str,
|
||||
) -> Result<daemon_info::DaemonInfo> {
|
||||
let mut child = spawn_detached_at(exe, args, None)?;
|
||||
let pid = child.id();
|
||||
let expected = handover::Expected {
|
||||
port: Some(args.resolved_port()).filter(|port| *port != 0),
|
||||
version,
|
||||
};
|
||||
let daemon = handover::wait_until_serving(&mut child, pid, handover::HANDOVER_TIMEOUT, || {
|
||||
handover::observe(None, &expected)
|
||||
})?;
|
||||
disown_daemon(child);
|
||||
Ok(daemon)
|
||||
}
|
||||
|
||||
/// Shared explicit/automatic startup, without an intermediate launcher or
|
||||
/// captured pipe. The deadline limits this caller's wait, not daemon lifetime.
|
||||
pub(crate) fn start_background(
|
||||
@@ -346,6 +369,8 @@ fn wait_for_stopped(expected: &daemon_info::DaemonInfo, timeout: Duration) -> Re
|
||||
/// Run the daemon in the foreground of the current process: acquire
|
||||
/// the lock, bind IPC, publish `daemon.json`, and serve until shutdown.
|
||||
pub fn run_foreground(cfg: DaemonConfig) -> Result<()> {
|
||||
// Before any update can replace the executable (see the function).
|
||||
let _ = crate::cli::update::installed_executable();
|
||||
paths::ensure_bsk_home()?;
|
||||
let _log_guard = init_tracing();
|
||||
let result = run_daemon(&cfg);
|
||||
@@ -368,7 +393,7 @@ fn run_daemon(cfg: &DaemonConfig) -> Result<()> {
|
||||
);
|
||||
let startup = acquire_daemon_lock(cfg.replaces).and_then(|lock| {
|
||||
info!(?lock, "daemon lock acquired");
|
||||
serve(cfg).map(|stopped| (lock, stopped))
|
||||
serve(cfg, None).map(|stopped| (lock, stopped))
|
||||
});
|
||||
let (mut lock, mut stopped) = match startup {
|
||||
Ok(started) => started,
|
||||
@@ -384,11 +409,23 @@ fn run_daemon(cfg: &DaemonConfig) -> Result<()> {
|
||||
return Ok(());
|
||||
};
|
||||
drop(lock);
|
||||
match handover::finish(*pending) {
|
||||
let mut record = match handover::finish(*pending) {
|
||||
handover::Finished::Exit => return Ok(()),
|
||||
handover::Finished::Resume(reclaimed) => lock = reclaimed,
|
||||
}
|
||||
stopped = serve(cfg)?;
|
||||
handover::Finished::Resume {
|
||||
lock: reclaimed,
|
||||
record,
|
||||
} => {
|
||||
lock = reclaimed;
|
||||
record
|
||||
}
|
||||
};
|
||||
stopped = match serve(cfg, Some(&mut record)) {
|
||||
Ok(stopped) => stopped,
|
||||
Err(err) => {
|
||||
record.serving_failed(&err);
|
||||
return Err(err);
|
||||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -433,7 +470,9 @@ enum StopReason {
|
||||
|
||||
/// Bind IPC and WS, publish `daemon.json`, and serve until shutdown. Returns
|
||||
/// with every endpoint released except the daemon lock, which the caller holds.
|
||||
fn serve(cfg: &DaemonConfig) -> Result<Stopped> {
|
||||
/// `resumed` records a failed handover whose previous version serves again
|
||||
/// here; it is confirmed once `daemon.json` is published.
|
||||
fn serve(cfg: &DaemonConfig, resumed: Option<&mut UpdateRecord>) -> Result<Stopped> {
|
||||
let runtime = tokio::runtime::Builder::new_multi_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
@@ -460,20 +499,22 @@ fn serve(cfg: &DaemonConfig) -> Result<Stopped> {
|
||||
.context("initialize transfer staging")?;
|
||||
let session_idle_task = spawn_session_idle_reaper(Arc::clone(&state));
|
||||
let browser_liveness_task = spawn_browser_liveness_reaper(Arc::clone(&state));
|
||||
// Fired by the update check task once it has started a replacement
|
||||
// daemon, which waits for this process to release the lock.
|
||||
let restart_notify = Arc::new(tokio::sync::Notify::new());
|
||||
let update_check_task = spawn_update_check_task(
|
||||
Arc::clone(&state),
|
||||
Arc::clone(&restart_notify),
|
||||
Arc::clone(&handover_slot),
|
||||
);
|
||||
let ws_addr = SocketAddr::new(cfg.listen_ip(), cfg.ws_port);
|
||||
let ws_handle = ws::WsServer::new(Arc::clone(&state))
|
||||
.bind(ws_addr)
|
||||
.await
|
||||
.with_context(|| format!("bind WS server on {ws_addr}"))?;
|
||||
let ws_port = ws_handle.local_addr.port();
|
||||
// Fired by the update check task once it has started a replacement
|
||||
// daemon, which waits for this process to release the lock. The
|
||||
// replacement must serve the port bound here.
|
||||
let restart_notify = Arc::new(tokio::sync::Notify::new());
|
||||
let update_check_task = spawn_update_check_task(
|
||||
Arc::clone(&state),
|
||||
ws_port,
|
||||
Arc::clone(&restart_notify),
|
||||
Arc::clone(&handover_slot),
|
||||
);
|
||||
|
||||
let info = daemon_info::DaemonInfo::now(
|
||||
std::process::id(),
|
||||
@@ -489,6 +530,9 @@ fn serve(cfg: &DaemonConfig) -> Result<Stopped> {
|
||||
sock = %sock_path.display(),
|
||||
"daemon ready"
|
||||
);
|
||||
if let Some(record) = resumed {
|
||||
record.confirm_serving();
|
||||
}
|
||||
remove_update_leftovers_after(cfg.replaces);
|
||||
|
||||
// Best-effort: keep installed agent skills in step with this
|
||||
@@ -671,7 +715,7 @@ fn serve(cfg: &DaemonConfig) -> Result<Stopped> {
|
||||
/// Remove files earlier updates left next to this executable. A predecessor
|
||||
/// still runs from its previous executable until it exits, so wait for it.
|
||||
fn remove_update_leftovers_after(predecessor: Option<u32>) {
|
||||
let Ok(exe) = std::env::current_exe() else {
|
||||
let Ok(exe) = crate::cli::update::installed_executable() else {
|
||||
return;
|
||||
};
|
||||
// A plain thread, so a slow predecessor never delays this daemon's exit.
|
||||
@@ -682,7 +726,7 @@ fn remove_update_leftovers_after(predecessor: Option<u32>) {
|
||||
std::thread::sleep(Duration::from_millis(200));
|
||||
}
|
||||
}
|
||||
crate::cli::update::remove_update_leftovers(&exe);
|
||||
crate::cli::update::remove_update_leftovers(exe);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -819,6 +863,7 @@ pub(crate) fn spawn_session_idle_reaper(state: Arc<DaemonState>) -> tokio::task:
|
||||
/// after [`crate::cli::update::state::RETRY_AFTER_FAILURE`].
|
||||
pub(crate) fn spawn_update_check_task(
|
||||
state: Arc<DaemonState>,
|
||||
ws_port: u16,
|
||||
restart: Arc<tokio::sync::Notify>,
|
||||
handover_slot: Arc<Mutex<Option<handover::Pending>>>,
|
||||
) -> tokio::task::JoinHandle<()> {
|
||||
@@ -837,12 +882,11 @@ pub(crate) fn spawn_update_check_task(
|
||||
return;
|
||||
}
|
||||
};
|
||||
// Capture our own executable path once, up front: after an
|
||||
// auto-update replaces the binary, `current_exe` on Linux starts
|
||||
// returning a " (deleted)"-suffixed path that can neither be
|
||||
// replaced again nor spawned.
|
||||
let exe_path = match std::env::current_exe() {
|
||||
Ok(exe) => Some(exe),
|
||||
// Captured when the process started: after an update replaced the
|
||||
// binary, `current_exe` on Linux returns a " (deleted)"-suffixed path
|
||||
// for good, even once a rollback resumes this process.
|
||||
let exe_path = match update::installed_executable() {
|
||||
Ok(exe) => Some(exe.to_path_buf()),
|
||||
Err(err) => {
|
||||
warn!(error = %err, "auto-update install disabled: cannot locate current executable");
|
||||
None
|
||||
@@ -997,7 +1041,7 @@ pub(crate) fn spawn_update_check_task(
|
||||
let exe = exe_path.clone();
|
||||
let config = state.config.clone();
|
||||
let started = tokio::task::spawn_blocking(move || {
|
||||
start_replacement(installed, exe.as_deref(), &config)
|
||||
start_replacement(installed, exe.as_deref(), &config, ws_port)
|
||||
})
|
||||
.await;
|
||||
match started {
|
||||
@@ -1038,7 +1082,7 @@ fn prepare_handover(
|
||||
state::{UpdateLock, UpdateRecord, UpdateSource},
|
||||
};
|
||||
let exe = exe.context("current executable unknown")?;
|
||||
let lock = UpdateLock::try_acquire()?;
|
||||
let lock = UpdateLock::try_acquire(exe)?;
|
||||
let mut record = UpdateRecord::start(UpdateSource::Daemon, &candidate.latest, exe);
|
||||
record.save();
|
||||
let installed = update::self_install_candidate(candidate, exe, &mut record)?;
|
||||
@@ -1049,12 +1093,14 @@ fn prepare_handover(
|
||||
})
|
||||
}
|
||||
|
||||
/// Spawn the replacement daemon from the new executable. If it cannot even
|
||||
/// be started, restore the previous executable and keep serving. Blocking.
|
||||
/// Spawn the replacement daemon from the new executable, on the port this
|
||||
/// daemon serves. If it cannot even be started, restore the previous
|
||||
/// executable and keep serving. Blocking.
|
||||
fn start_replacement(
|
||||
prepared: Prepared,
|
||||
exe: Option<&Path>,
|
||||
config: &DaemonConfig,
|
||||
ws_port: u16,
|
||||
) -> Option<handover::Pending> {
|
||||
use crate::cli::update::state::{Recovery, UpdateStage};
|
||||
let Prepared {
|
||||
@@ -1066,7 +1112,7 @@ fn start_replacement(
|
||||
let spawned = exe
|
||||
.context("current executable unknown")
|
||||
.and_then(|exe| {
|
||||
let args = restart_start_args(config)?;
|
||||
let args = restart_start_args(config, ws_port)?;
|
||||
spawn_detached_at(exe, &args, Some(std::process::id()))
|
||||
})
|
||||
.context("start the replacement daemon");
|
||||
@@ -1074,6 +1120,7 @@ fn start_replacement(
|
||||
Ok(child) => Some(handover::Pending {
|
||||
child_pid: child.id(),
|
||||
child,
|
||||
port: ws_port,
|
||||
installed,
|
||||
record,
|
||||
_update_lock: lock,
|
||||
@@ -1123,14 +1170,15 @@ fn auto_update_policy(enabled: bool, detached: bool) -> crate::cli::update::Auto
|
||||
}
|
||||
|
||||
/// Rebuild the `StartArgs` for the replacement daemon from the running
|
||||
/// config so the respawn keeps the same port and idle timeouts.
|
||||
fn restart_start_args(cfg: &DaemonConfig) -> Result<StartArgs> {
|
||||
/// config so the respawn keeps the port it serves (`ws_port`, which differs
|
||||
/// from the configured one for `--port 0`) and its idle timeouts.
|
||||
fn restart_start_args(cfg: &DaemonConfig, ws_port: u16) -> Result<StartArgs> {
|
||||
anyhow::ensure!(
|
||||
cfg.server.is_none(),
|
||||
"server restart is managed by the deployment supervisor"
|
||||
);
|
||||
Ok(StartArgs {
|
||||
port: Some(cfg.ws_port),
|
||||
port: Some(ws_port),
|
||||
foreground: false,
|
||||
session_idle: Some(cfg.session_idle),
|
||||
daemon_idle: Some(cfg.daemon_idle),
|
||||
@@ -1676,12 +1724,13 @@ mod tests {
|
||||
#[test]
|
||||
fn restart_start_args_preserve_the_running_config() {
|
||||
let cfg = DaemonConfig {
|
||||
ws_port: 1234,
|
||||
ws_port: 0,
|
||||
session_idle: Duration::from_secs(11),
|
||||
daemon_idle: Duration::from_secs(22),
|
||||
..DaemonConfig::new(0)
|
||||
};
|
||||
let args = restart_start_args(&cfg).unwrap();
|
||||
// The port actually bound, not the configured `--port 0`.
|
||||
let args = restart_start_args(&cfg, 1234).unwrap();
|
||||
assert_eq!(args.port, Some(1234));
|
||||
assert!(!args.foreground);
|
||||
assert_eq!(args.session_idle, Some(Duration::from_secs(11)));
|
||||
@@ -1712,7 +1761,7 @@ mod tests {
|
||||
}
|
||||
.server_config()
|
||||
.unwrap();
|
||||
assert!(restart_start_args(&cfg).is_err());
|
||||
assert!(restart_start_args(&cfg, 52800).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -16,6 +16,7 @@ use tracing::{error, info, warn};
|
||||
use super::DaemonChild;
|
||||
use crate::cli::update::Installed;
|
||||
use crate::cli::update::state::{Recovery, UpdateLock, UpdateRecord};
|
||||
use crate::daemon::info::DaemonInfo;
|
||||
use crate::daemon::lockfile::{self, DaemonLock};
|
||||
use crate::daemon::paths;
|
||||
use crate::daemon::probe::{self, PROBE_TIMEOUT, Probe};
|
||||
@@ -35,6 +36,8 @@ const FAILURE_REPORT_LIMIT: usize = 4096;
|
||||
pub(crate) struct Pending {
|
||||
pub(super) child: DaemonChild,
|
||||
pub(super) child_pid: u32,
|
||||
/// The WS port this daemon serves, which the replacement must serve too.
|
||||
pub(super) port: u16,
|
||||
pub(super) installed: Installed,
|
||||
pub(super) record: UpdateRecord,
|
||||
pub(super) _update_lock: UpdateLock,
|
||||
@@ -44,16 +47,47 @@ pub(super) enum Finished {
|
||||
/// Another daemon serves, or none can; this process exits.
|
||||
Exit,
|
||||
/// The handover failed and this process serves again under the lock.
|
||||
Resume(DaemonLock),
|
||||
/// `record` says so, and is confirmed once serving has resumed.
|
||||
Resume {
|
||||
lock: DaemonLock,
|
||||
record: Box<UpdateRecord>,
|
||||
},
|
||||
}
|
||||
|
||||
/// The daemon a waiter accepts as serving.
|
||||
pub(crate) struct Expected<'a> {
|
||||
/// The WS port browsers connect to; `None` accepts any.
|
||||
pub(crate) port: Option<u16>,
|
||||
pub(crate) version: &'a str,
|
||||
}
|
||||
|
||||
/// What one status probe found, judged against [`Expected`].
|
||||
pub(super) enum Found {
|
||||
Serving(DaemonInfo),
|
||||
/// A daemon answers, but not the expected one.
|
||||
Other(String),
|
||||
Nothing,
|
||||
}
|
||||
|
||||
/// Complete a handover once this process has stopped serving and released
|
||||
/// the daemon lock.
|
||||
pub(super) fn finish(mut pending: Pending) -> Finished {
|
||||
match wait_for_replacement(&mut pending.child, pending.child_pid, HANDOVER_TIMEOUT) {
|
||||
Ok((pid, version)) => {
|
||||
info!(pid, %version, "replacement daemon is serving; exiting");
|
||||
pending.record.succeed(Some((pid, version)));
|
||||
let own_pid = std::process::id();
|
||||
let target = pending.record.target_version.clone();
|
||||
let expected = Expected {
|
||||
port: Some(pending.port),
|
||||
version: &target,
|
||||
};
|
||||
let waited = wait_until_serving(
|
||||
&mut pending.child,
|
||||
pending.child_pid,
|
||||
HANDOVER_TIMEOUT,
|
||||
|| observe(Some(own_pid), &expected),
|
||||
);
|
||||
match waited {
|
||||
Ok(daemon) => {
|
||||
info!(pid = daemon.pid, version = %daemon.version, "replacement daemon is serving; exiting");
|
||||
pending.record.succeed(Some((daemon.pid, daemon.version)));
|
||||
pending.installed.discard();
|
||||
Finished::Exit
|
||||
}
|
||||
@@ -69,10 +103,13 @@ pub(super) fn finish(mut pending: Pending) -> Finished {
|
||||
|
||||
/// Put the previous executable back and serve again. Restoring comes first,
|
||||
/// so a daemon another client starts meanwhile runs the previous version too.
|
||||
/// The record claims a serving daemon only for one that already serves the
|
||||
/// original port; a resumed one confirms it after publishing `daemon.json`.
|
||||
fn recover(pending: Pending, err: &anyhow::Error) -> Finished {
|
||||
let Pending {
|
||||
installed,
|
||||
mut record,
|
||||
port,
|
||||
..
|
||||
} = pending;
|
||||
let restored = installed.restore();
|
||||
@@ -86,23 +123,28 @@ fn recover(pending: Pending, err: &anyhow::Error) -> Finished {
|
||||
None
|
||||
}
|
||||
};
|
||||
let daemon_serving = lock.is_some() || serving_daemon(std::process::id()).is_some();
|
||||
let recovery = match restored {
|
||||
Ok(()) => Recovery::Restored { daemon_serving },
|
||||
Err(_) => installed.restore_failed(),
|
||||
};
|
||||
record.fail(err, recovery);
|
||||
let daemon_serving = lock.is_none() && serving_on(port);
|
||||
record.fail(
|
||||
err,
|
||||
match restored {
|
||||
Ok(()) => Recovery::Restored { daemon_serving },
|
||||
Err(_) => installed.restore_failed(),
|
||||
},
|
||||
);
|
||||
match lock {
|
||||
Some(lock) => {
|
||||
info!("resuming service with the previous version");
|
||||
Finished::Resume(lock)
|
||||
Finished::Resume {
|
||||
lock,
|
||||
record: Box::new(record),
|
||||
}
|
||||
}
|
||||
None if daemon_serving => {
|
||||
info!("another daemon is serving; exiting");
|
||||
info!("another daemon serves the port; exiting");
|
||||
Finished::Exit
|
||||
}
|
||||
None => {
|
||||
error!("no daemon is serving after the failed handover; run `bsk daemon start`");
|
||||
error!("no daemon serves the port after the failed handover; run `bsk daemon start`");
|
||||
Finished::Exit
|
||||
}
|
||||
}
|
||||
@@ -119,41 +161,91 @@ pub(super) fn abandon(mut pending: Pending, reason: &str) {
|
||||
pending.record.fail(&err, recovery);
|
||||
}
|
||||
|
||||
/// Wait until a daemon other than this process answers, failing as soon as
|
||||
/// the replacement exits or `timeout` passes. Returns the serving daemon's
|
||||
/// pid and version.
|
||||
pub(super) fn wait_for_replacement(
|
||||
/// Wait until `observe` finds the expected daemon serving, failing as soon as
|
||||
/// `child` exits or `timeout` passes; a child still running then is stopped
|
||||
/// and reaped, so it releases whatever it holds. Another client may start
|
||||
/// the expected daemon meanwhile, which counts; one that answers with another
|
||||
/// version or port does not, and is named in the error.
|
||||
pub(super) fn wait_until_serving(
|
||||
child: &mut DaemonChild,
|
||||
child_pid: u32,
|
||||
timeout: Duration,
|
||||
) -> Result<(u32, String)> {
|
||||
let own_pid = std::process::id();
|
||||
mut observe: impl FnMut() -> Found,
|
||||
) -> Result<DaemonInfo> {
|
||||
let deadline = Instant::now() + timeout;
|
||||
loop {
|
||||
if let Some(serving) = serving_daemon(own_pid) {
|
||||
return Ok(serving);
|
||||
let mut other = None;
|
||||
let mut look = |other: &mut Option<String>| match observe() {
|
||||
Found::Serving(daemon) => Some(daemon),
|
||||
Found::Other(found) => {
|
||||
*other = Some(found);
|
||||
None
|
||||
}
|
||||
if let Some(status) = child.try_wait().context("check the replacement daemon")? {
|
||||
// Another client may have started a daemon from the new binary.
|
||||
if let Some(serving) = serving_daemon(own_pid) {
|
||||
return Ok(serving);
|
||||
Found::Nothing => None,
|
||||
};
|
||||
loop {
|
||||
if let Some(daemon) = look(&mut other) {
|
||||
return Ok(daemon);
|
||||
}
|
||||
if let Some(status) = child.try_wait().context("check the new daemon")? {
|
||||
if let Some(daemon) = look(&mut other) {
|
||||
return Ok(daemon);
|
||||
}
|
||||
anyhow::bail!(
|
||||
"the replacement daemon (pid {child_pid}) exited with {status} before it was ready{}",
|
||||
startup_failure(child_pid)
|
||||
"the new daemon (pid {child_pid}) exited with {status} before it was ready{}{}",
|
||||
startup_failure(child_pid),
|
||||
describe_other(other.as_deref())
|
||||
);
|
||||
}
|
||||
if Instant::now() >= deadline {
|
||||
stop(child);
|
||||
anyhow::bail!(
|
||||
"the replacement daemon (pid {child_pid}) was not ready within {timeout:?} and was stopped{}",
|
||||
startup_failure(child_pid)
|
||||
"the new daemon (pid {child_pid}) was not ready within {timeout:?} and was stopped{}{}",
|
||||
startup_failure(child_pid),
|
||||
describe_other(other.as_deref())
|
||||
);
|
||||
}
|
||||
std::thread::sleep(POLL);
|
||||
}
|
||||
}
|
||||
|
||||
fn describe_other(other: Option<&str>) -> String {
|
||||
other.map_or_else(String::new, |other| {
|
||||
format!("; meanwhile a different daemon answered: {other}")
|
||||
})
|
||||
}
|
||||
|
||||
/// Probe the IPC endpoint once. `exclude` is a daemon that never counts,
|
||||
/// such as the one handing over.
|
||||
pub(super) fn observe(exclude: Option<u32>, expected: &Expected<'_>) -> Found {
|
||||
match probe::probe(PROBE_TIMEOUT) {
|
||||
Ok(Probe::Ready(daemon)) if Some(daemon.status.pid) != exclude => judge(
|
||||
daemon.status.pid,
|
||||
&daemon.status.daemon_version,
|
||||
daemon.status.ws_port,
|
||||
expected,
|
||||
)
|
||||
.map_or_else(Found::Other, |()| Found::Serving(daemon.info)),
|
||||
_ => Found::Nothing,
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether a daemon answering as `pid`, `version` on `port` is the expected
|
||||
/// one; if not, a description of what answered instead.
|
||||
fn judge(pid: u32, version: &str, port: u16, expected: &Expected<'_>) -> Result<(), String> {
|
||||
let port_matches = expected.port.is_none_or(|expected| expected == port);
|
||||
if version == expected.version && port_matches {
|
||||
return Ok(());
|
||||
}
|
||||
Err(format!(
|
||||
"pid {pid}, bsk {version} on port {port}, expected bsk {}{}",
|
||||
expected.version,
|
||||
expected
|
||||
.port
|
||||
.map(|port| format!(" on port {port}"))
|
||||
.unwrap_or_default()
|
||||
))
|
||||
}
|
||||
|
||||
/// A replacement that fails to start leaves the reason for its predecessor,
|
||||
/// which reports it in the update record.
|
||||
pub(super) fn report_startup_failure(err: &anyhow::Error) {
|
||||
@@ -178,13 +270,13 @@ fn startup_failure(pid: u32) -> String {
|
||||
}
|
||||
}
|
||||
|
||||
fn serving_daemon(own_pid: u32) -> Option<(u32, String)> {
|
||||
match probe::probe(PROBE_TIMEOUT) {
|
||||
Ok(Probe::Ready(daemon)) if daemon.status.pid != own_pid => {
|
||||
Some((daemon.status.pid, daemon.status.daemon_version))
|
||||
}
|
||||
_ => None,
|
||||
}
|
||||
/// Whether some daemon other than this process serves `port`.
|
||||
fn serving_on(port: u16) -> bool {
|
||||
matches!(
|
||||
probe::probe(PROBE_TIMEOUT),
|
||||
Ok(Probe::Ready(daemon))
|
||||
if daemon.status.pid != std::process::id() && daemon.status.ws_port == port
|
||||
)
|
||||
}
|
||||
|
||||
/// `None` when another process keeps the lock: either it serves already, or
|
||||
@@ -195,7 +287,7 @@ fn reclaim_lock() -> Result<Option<DaemonLock>> {
|
||||
match lockfile::acquire() {
|
||||
Ok(lock) => return Ok(Some(lock)),
|
||||
Err(err) if err.is::<lockfile::AlreadyLocked>() => {
|
||||
if Instant::now() >= deadline || serving_daemon(std::process::id()).is_some() {
|
||||
if Instant::now() >= deadline {
|
||||
return Ok(None);
|
||||
}
|
||||
std::thread::sleep(POLL);
|
||||
@@ -210,6 +302,36 @@ fn stop(child: &mut DaemonChild) {
|
||||
let _ = child.wait();
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod judge_tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn only_the_expected_version_on_the_expected_port_counts() {
|
||||
let expected = Expected {
|
||||
port: Some(52719),
|
||||
version: "999.0.0",
|
||||
};
|
||||
assert!(judge(7, "999.0.0", 52719, &expected).is_ok());
|
||||
let other = judge(7, "0.3.1", 52720, &expected).unwrap_err();
|
||||
assert_eq!(
|
||||
other,
|
||||
"pid 7, bsk 0.3.1 on port 52720, expected bsk 999.0.0 on port 52719"
|
||||
);
|
||||
assert!(
|
||||
judge(7, "0.3.1", 52719, &expected).is_err(),
|
||||
"wrong version"
|
||||
);
|
||||
assert!(judge(7, "999.0.0", 52800, &expected).is_err(), "wrong port");
|
||||
|
||||
let any_port = Expected {
|
||||
port: None,
|
||||
version: "999.0.0",
|
||||
};
|
||||
assert!(judge(7, "999.0.0", 40000, &any_port).is_ok());
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(all(test, unix))]
|
||||
mod tests {
|
||||
use super::*;
|
||||
@@ -222,6 +344,58 @@ mod tests {
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
fn serving(pid: u32) -> DaemonInfo {
|
||||
DaemonInfo::now(pid, "sock".into(), 52719, "999.0.0")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_expected_daemon_counts_even_when_another_client_started_it() {
|
||||
let mut child = spawn("sleep 30");
|
||||
let pid = child.id();
|
||||
let mut probes = 0;
|
||||
|
||||
let daemon = wait_until_serving(&mut child, pid, Duration::from_secs(10), || {
|
||||
probes += 1;
|
||||
if probes < 3 {
|
||||
Found::Nothing
|
||||
} else {
|
||||
Found::Serving(serving(4242))
|
||||
}
|
||||
})
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(daemon.pid, 4242);
|
||||
stop(&mut child);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_different_daemon_answering_is_not_a_successful_handover() {
|
||||
isolated(
|
||||
concat!(
|
||||
module_path!(),
|
||||
"::a_different_daemon_answering_is_not_a_successful_handover"
|
||||
),
|
||||
|| {
|
||||
let mut child = spawn("sleep 0.3; exit 1");
|
||||
let pid = child.id();
|
||||
|
||||
let error = wait_until_serving(&mut child, pid, Duration::from_secs(10), || {
|
||||
Found::Other(
|
||||
"pid 9, bsk 0.3.1 on port 52720, expected bsk 999.0.0 on port 52719".into(),
|
||||
)
|
||||
})
|
||||
.unwrap_err();
|
||||
|
||||
let error = format!("{error:#}");
|
||||
assert!(error.contains("exited with"), "{error}");
|
||||
assert!(
|
||||
error.contains("a different daemon answered: pid 9, bsk 0.3.1 on port 52720"),
|
||||
"{error}"
|
||||
);
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_replacement_that_exits_early_fails_with_its_reported_reason() {
|
||||
isolated(
|
||||
@@ -237,7 +411,8 @@ mod tests {
|
||||
std::fs::write(&report, "bind WS server: address in use").unwrap();
|
||||
|
||||
let error =
|
||||
wait_for_replacement(&mut child, pid, Duration::from_secs(10)).unwrap_err();
|
||||
wait_until_serving(&mut child, pid, Duration::from_secs(10), || Found::Nothing)
|
||||
.unwrap_err();
|
||||
|
||||
let error = format!("{error:#}");
|
||||
assert!(error.contains(&format!("pid {pid}")), "{error}");
|
||||
@@ -261,8 +436,10 @@ mod tests {
|
||||
let pid = child.id();
|
||||
let started = Instant::now();
|
||||
|
||||
let error =
|
||||
wait_for_replacement(&mut child, pid, Duration::from_millis(300)).unwrap_err();
|
||||
let error = wait_until_serving(&mut child, pid, Duration::from_millis(300), || {
|
||||
Found::Nothing
|
||||
})
|
||||
.unwrap_err();
|
||||
|
||||
assert!(started.elapsed() < Duration::from_secs(10));
|
||||
let error = format!("{error:#}");
|
||||
|
||||
@@ -1,11 +1,15 @@
|
||||
//! A detached daemon that auto-updates hands over to a daemon started from
|
||||
//! the new executable, and exits only once that daemon serves. When the new
|
||||
//! executable fails its self-check, or its daemon cannot start, the previous
|
||||
//! executable is put back and the running daemon keeps serving on its port.
|
||||
//! the new executable, and exits only once that daemon serves the release's
|
||||
//! version on its port. When the new executable fails its self-check, or its
|
||||
//! daemon cannot start, the previous executable is put back and the running
|
||||
//! daemon keeps serving on its port. `bsk update` restarts the daemon with the
|
||||
//! same guarantees.
|
||||
//!
|
||||
//! On Windows these tests need a host that permits Job breakaway; CI runs
|
||||
//! them from `scripts/test-windows-daemon.ps1`.
|
||||
|
||||
mod release_fixture;
|
||||
|
||||
use std::cell::RefCell;
|
||||
use std::fs;
|
||||
use std::io::{Cursor, Read, Write};
|
||||
@@ -22,7 +26,8 @@ use sha2::{Digest, Sha256};
|
||||
const EXE: &str = if cfg!(windows) { "bsk.exe" } else { "bsk" };
|
||||
const MARKER: &[u8] = b"auto-update-handover-fixture";
|
||||
|
||||
/// Serves a manifest naming release 999.0.0 and an archive holding `binary`.
|
||||
/// Serves a manifest naming [`release_fixture::newer_version`] and an
|
||||
/// archive holding `binary`.
|
||||
struct ReleaseServer {
|
||||
url: String,
|
||||
downloads: Arc<AtomicUsize>,
|
||||
@@ -46,9 +51,10 @@ impl ReleaseServer {
|
||||
"sha256": Sha256::digest(&archive).iter().map(|byte| format!("{byte:02x}")).collect::<String>(),
|
||||
}),
|
||||
);
|
||||
let manifest =
|
||||
serde_json::to_vec(&serde_json::json!({"version": "999.0.0", "assets": assets}))
|
||||
.unwrap();
|
||||
let manifest = serde_json::to_vec(
|
||||
&serde_json::json!({"version": release_fixture::newer_version(), "assets": assets}),
|
||||
)
|
||||
.unwrap();
|
||||
let stop = Arc::new(AtomicBool::new(false));
|
||||
let downloads = Arc::new(AtomicUsize::new(0));
|
||||
let worker = {
|
||||
@@ -281,13 +287,15 @@ impl Fixture {
|
||||
}
|
||||
}
|
||||
|
||||
/// Files next to the executable other than the executable itself.
|
||||
/// Files next to the executable other than the executable itself and
|
||||
/// the update lock, which stays.
|
||||
fn leftovers(&self) -> Vec<String> {
|
||||
let lock = format!(".{EXE}.update.lock");
|
||||
fs::read_dir(self.exe.parent().unwrap())
|
||||
.unwrap()
|
||||
.flatten()
|
||||
.map(|entry| entry.file_name().to_string_lossy().into_owned())
|
||||
.filter(|name| name != EXE)
|
||||
.filter(|name| name != EXE && *name != lock)
|
||||
.collect()
|
||||
}
|
||||
|
||||
@@ -333,48 +341,65 @@ impl Drop for Fixture {
|
||||
}
|
||||
}
|
||||
|
||||
/// The current bsk with a trailing marker: the same program, but a
|
||||
/// different file, so the tests can tell which one is installed.
|
||||
fn marked_bsk(_: &Path) -> Vec<u8> {
|
||||
/// The release: the bsk under test, reporting the newer version.
|
||||
fn newer_bsk(dir: &Path) -> Vec<u8> {
|
||||
release_fixture::newer_bsk(dir)
|
||||
}
|
||||
|
||||
/// The bsk under test with a trailing marker: a different file that still
|
||||
/// reports the current version, not the one its manifest names.
|
||||
fn mislabelled_bsk(_: &Path) -> Vec<u8> {
|
||||
let mut binary = fs::read(env!("CARGO_BIN_EXE_bsk")).unwrap();
|
||||
binary.extend_from_slice(MARKER);
|
||||
binary
|
||||
}
|
||||
|
||||
/// Compile a stand-in for a broken release.
|
||||
fn compiled(dir: &Path, name: &str, main: &str) -> Vec<u8> {
|
||||
let source = dir.join(format!("{name}.rs"));
|
||||
fs::write(&source, format!("fn main() {{ {main} }}")).unwrap();
|
||||
let output = dir.join(format!("{name}{}", std::env::consts::EXE_SUFFIX));
|
||||
let rustc = std::env::var_os("RUSTC").unwrap_or_else(|| "rustc".into());
|
||||
let status = Command::new(rustc)
|
||||
.args(["--edition", "2021", "-o"])
|
||||
.arg(&output)
|
||||
.arg(&source)
|
||||
.status()
|
||||
.unwrap();
|
||||
assert!(status.success(), "compile {name}");
|
||||
fs::read(output).unwrap()
|
||||
/// Answers `--version` like the release; its daemon then runs `daemon`.
|
||||
fn stand_in(dir: &Path, name: &str, daemon: &str) -> Vec<u8> {
|
||||
release_fixture::compiled(
|
||||
dir,
|
||||
name,
|
||||
&format!(
|
||||
r#"
|
||||
if std::env::args().nth(1).as_deref() == Some("--version") {{
|
||||
println!("bsk {}");
|
||||
return;
|
||||
}}
|
||||
{daemon}
|
||||
"#,
|
||||
release_fixture::newer_version()
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
/// Answers `--version` like bsk, but its daemon fails to start.
|
||||
/// Its daemon fails to start.
|
||||
fn release_whose_daemon_fails(dir: &Path) -> Vec<u8> {
|
||||
compiled(
|
||||
stand_in(dir, "daemon_fails", "std::process::exit(3);")
|
||||
}
|
||||
|
||||
/// Its daemon takes the daemon lock, then never serves.
|
||||
fn release_whose_daemon_hangs(dir: &Path) -> Vec<u8> {
|
||||
stand_in(
|
||||
dir,
|
||||
"daemon_fails",
|
||||
"daemon_hangs",
|
||||
r#"
|
||||
if std::env::args().nth(1).as_deref() == Some("--version") {
|
||||
println!("bsk 999.0.0");
|
||||
return;
|
||||
}
|
||||
std::process::exit(3);
|
||||
let home = std::env::var_os("BSK_HOME").unwrap();
|
||||
let lock = std::fs::OpenOptions::new()
|
||||
.read(true)
|
||||
.write(true)
|
||||
.create(true)
|
||||
.truncate(false)
|
||||
.open(std::path::Path::new(&home).join("daemon.lock"))
|
||||
.unwrap();
|
||||
lock.lock().unwrap();
|
||||
std::thread::sleep(std::time::Duration::from_secs(600));
|
||||
"#,
|
||||
)
|
||||
}
|
||||
|
||||
/// Cannot even report its version.
|
||||
fn release_that_cannot_run(dir: &Path) -> Vec<u8> {
|
||||
compiled(dir, "cannot_run", "std::process::exit(1);")
|
||||
release_fixture::compiled(dir, "cannot_run", "std::process::exit(1);")
|
||||
}
|
||||
|
||||
fn unused_port() -> u16 {
|
||||
@@ -387,7 +412,7 @@ fn unused_port() -> u16 {
|
||||
|
||||
#[test]
|
||||
fn auto_update_exits_only_after_the_new_daemon_serves() {
|
||||
let fixture = Fixture::new(marked_bsk);
|
||||
let fixture = Fixture::new(newer_bsk);
|
||||
let port = unused_port();
|
||||
let old_pid = fixture.start_daemon(port);
|
||||
|
||||
@@ -403,7 +428,16 @@ fn auto_update_exits_only_after_the_new_daemon_serves() {
|
||||
assert_eq!(record["source"], "daemon", "{record}");
|
||||
assert_eq!(record["result"], "succeeded", "{record}");
|
||||
assert_eq!(record["stage"], "handover", "{record}");
|
||||
assert_eq!(record["target_version"], "999.0.0", "{record}");
|
||||
assert_eq!(
|
||||
record["target_version"],
|
||||
release_fixture::newer_version(),
|
||||
"{record}"
|
||||
);
|
||||
assert_eq!(
|
||||
record["daemon_version"],
|
||||
release_fixture::newer_version(),
|
||||
"{record}"
|
||||
);
|
||||
assert_eq!(record["daemon_pid"], new_pid, "{record}");
|
||||
assert!(record.get("previous_executable").is_none(), "{record}");
|
||||
assert!(fixture.installed() == fixture.release);
|
||||
@@ -433,14 +467,15 @@ fn a_failed_handover_restores_the_previous_executable_and_keeps_serving() {
|
||||
.info()
|
||||
.is_some_and(|info| info["pid"] == old_pid && info["ws_port"] == port)
|
||||
});
|
||||
// Confirmed only once the resumed daemon has published daemon.json.
|
||||
fixture.wait_for("the resumed service to be confirmed", || {
|
||||
fixture.record().is_some_and(|record| {
|
||||
record["recovery"] == serde_json::json!({"state": "restored", "daemon_serving": true})
|
||||
})
|
||||
});
|
||||
|
||||
let record = fixture.record().unwrap();
|
||||
assert_eq!(record["stage"], "handover", "{record}");
|
||||
assert_eq!(
|
||||
record["recovery"],
|
||||
serde_json::json!({"state": "restored", "daemon_serving": true}),
|
||||
"{record}"
|
||||
);
|
||||
let error = record["error"].as_str().unwrap();
|
||||
assert!(error.contains("before it was ready"), "{error}");
|
||||
assert!(record["retry_after_epoch_secs"].is_u64(), "{record}");
|
||||
@@ -463,7 +498,7 @@ fn a_failed_handover_restores_the_previous_executable_and_keeps_serving() {
|
||||
|
||||
#[test]
|
||||
fn manual_update_leaves_a_host_managed_daemon_to_its_owner() {
|
||||
let fixture = Fixture::new(marked_bsk);
|
||||
let fixture = Fixture::new(newer_bsk);
|
||||
let port = unused_port();
|
||||
let pid = fixture.start_foreground_daemon(port);
|
||||
fixture.wait_for("the foreground daemon", || {
|
||||
@@ -492,7 +527,7 @@ fn manual_update_leaves_a_host_managed_daemon_to_its_owner() {
|
||||
|
||||
#[test]
|
||||
fn manual_update_restarts_a_background_daemon_on_its_port() {
|
||||
let fixture = Fixture::new(marked_bsk);
|
||||
let fixture = Fixture::new(newer_bsk);
|
||||
let port = unused_port();
|
||||
let start = fixture
|
||||
.command()
|
||||
@@ -522,33 +557,126 @@ fn manual_update_restarts_a_background_daemon_on_its_port() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_release_that_cannot_run_is_never_handed_over_to() {
|
||||
let fixture = Fixture::new(release_that_cannot_run);
|
||||
fn a_release_that_cannot_run_or_reports_another_version_is_never_handed_over_to() {
|
||||
for (release, expected_error) in [
|
||||
(
|
||||
release_that_cannot_run as fn(&Path) -> Vec<u8>,
|
||||
"exited with",
|
||||
),
|
||||
(
|
||||
mislabelled_bsk,
|
||||
concat!("printed \"bsk ", env!("CARGO_PKG_VERSION"), "\""),
|
||||
),
|
||||
] {
|
||||
let fixture = Fixture::new(release);
|
||||
let port = unused_port();
|
||||
let old_pid = fixture.start_daemon(port);
|
||||
fixture.wait_for("the daemon to serve", || {
|
||||
fixture.info().is_some_and(|info| info["pid"] == old_pid)
|
||||
});
|
||||
|
||||
fixture.wait_for("the failed update to be recorded", || {
|
||||
fixture
|
||||
.record()
|
||||
.is_some_and(|record| record["result"] == "failed")
|
||||
});
|
||||
|
||||
let record = fixture.record().unwrap();
|
||||
assert_eq!(record["stage"], "install", "{record}");
|
||||
assert_eq!(
|
||||
record["recovery"],
|
||||
serde_json::json!({"state": "unchanged"}),
|
||||
"{record}"
|
||||
);
|
||||
let error = record["error"].as_str().unwrap();
|
||||
assert!(error.contains("self-check"), "{error}");
|
||||
assert!(error.contains(expected_error), "{error}");
|
||||
assert!(fixture.installed() == fixture.original);
|
||||
assert_eq!(fixture.info().unwrap()["pid"], old_pid, "never stopped");
|
||||
assert!(!fixture.daemon_exited());
|
||||
fixture.status_succeeds();
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn manual_update_stops_a_new_daemon_stuck_on_the_lock_and_restores_the_service() {
|
||||
let fixture = Fixture::new(release_whose_daemon_hangs);
|
||||
let port = unused_port();
|
||||
let old_pid = fixture.start_daemon(port);
|
||||
fixture.wait_for("the daemon to serve", || {
|
||||
fixture.info().is_some_and(|info| info["pid"] == old_pid)
|
||||
});
|
||||
let start = fixture
|
||||
.command()
|
||||
.args(["daemon", "start", "--port", &port.to_string()])
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(
|
||||
start.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&start.stderr)
|
||||
);
|
||||
|
||||
fixture.wait_for("the failed update to be recorded", || {
|
||||
fixture
|
||||
.record()
|
||||
.is_some_and(|record| record["result"] == "failed")
|
||||
});
|
||||
let out = fixture
|
||||
.command()
|
||||
.args(["--json", "update", "--yes"])
|
||||
.output()
|
||||
.unwrap();
|
||||
|
||||
// `--json` reports the error on stdout.
|
||||
let output = format!(
|
||||
"{}{}",
|
||||
String::from_utf8_lossy(&out.stdout),
|
||||
String::from_utf8_lossy(&out.stderr)
|
||||
);
|
||||
assert!(!out.status.success(), "the update must fail: {output}");
|
||||
assert!(output.contains("rolled back"), "{output}");
|
||||
assert!(fixture.installed() == fixture.original);
|
||||
// The stuck daemon was stopped, so the previous version got the lock and
|
||||
// serves the original port again.
|
||||
let info = fixture.info().expect("a daemon serves again");
|
||||
assert_eq!(info["ws_port"], port, "{info}");
|
||||
assert_eq!(info["version"], env!("CARGO_PKG_VERSION"), "{info}");
|
||||
fixture.status_succeeds();
|
||||
let record = fixture.record().unwrap();
|
||||
assert_eq!(record["stage"], "install", "{record}");
|
||||
assert_eq!(record["stage"], "restart", "{record}");
|
||||
assert_eq!(
|
||||
record["recovery"],
|
||||
serde_json::json!({"state": "unchanged"}),
|
||||
serde_json::json!({"state": "restored", "daemon_serving": true}),
|
||||
"{record}"
|
||||
);
|
||||
assert!(
|
||||
record["error"].as_str().unwrap().contains("self-check"),
|
||||
let error = record["error"].as_str().unwrap();
|
||||
assert!(error.contains("was not ready within"), "{error}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_resumed_daemon_that_cannot_serve_again_is_recorded_as_not_serving() {
|
||||
let fixture = Fixture::new(release_whose_daemon_fails);
|
||||
let port = unused_port();
|
||||
fixture.start_daemon(port);
|
||||
fixture.wait_for("the daemon to serve", || {
|
||||
fixture.info().is_some_and(|info| info["ws_port"] == port)
|
||||
});
|
||||
|
||||
// Take the port the moment the daemon releases it for the handover, so
|
||||
// the previous version cannot bind it again after the replacement fails.
|
||||
let listener = loop {
|
||||
if let Ok(listener) = TcpListener::bind(("127.0.0.1", port)) {
|
||||
break listener;
|
||||
}
|
||||
assert!(!fixture.daemon_exited(), "the daemon exited early");
|
||||
};
|
||||
fixture.wait_for("the daemon to give up", || fixture.daemon_exited());
|
||||
|
||||
let record = fixture.record().unwrap();
|
||||
assert_eq!(record["result"], "failed", "{record}");
|
||||
assert_eq!(
|
||||
record["recovery"],
|
||||
serde_json::json!({"state": "restored", "daemon_serving": false}),
|
||||
"{record}"
|
||||
);
|
||||
let error = record["error"].as_str().unwrap();
|
||||
assert!(error.contains("before it was ready"), "{error}");
|
||||
assert!(
|
||||
error.contains("the previous version could not serve again"),
|
||||
"{error}"
|
||||
);
|
||||
assert!(fixture.installed() == fixture.original);
|
||||
assert_eq!(fixture.info().unwrap()["pid"], old_pid, "never stopped");
|
||||
assert!(!fixture.daemon_exited());
|
||||
fixture.status_succeeds();
|
||||
drop(listener);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
//! Releases of the bsk under test for update tests. A release must report the
|
||||
//! version its manifest names, so tests serve a copy whose version string is
|
||||
//! replaced by a newer one of the same length.
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use std::fs;
|
||||
use std::path::Path;
|
||||
|
||||
/// Newer than the bsk under test and of the same length: every digit becomes
|
||||
/// 9, so `0.3.1` becomes `9.9.9`.
|
||||
pub fn newer_version() -> String {
|
||||
let current = env!("CARGO_PKG_VERSION");
|
||||
let newer: String = current
|
||||
.chars()
|
||||
.map(|c| if c.is_ascii_digit() { '9' } else { c })
|
||||
.collect();
|
||||
assert_ne!(newer, current, "the bsk under test is already {current}");
|
||||
newer
|
||||
}
|
||||
|
||||
/// The bsk under test, reporting [`newer_version`]. On macOS the copy is
|
||||
/// signed again (ad hoc), since the kernel refuses modified signed code.
|
||||
pub fn newer_bsk(dir: &Path) -> Vec<u8> {
|
||||
let current = env!("CARGO_PKG_VERSION").as_bytes();
|
||||
let newer = newer_version();
|
||||
let original = fs::read(env!("CARGO_BIN_EXE_bsk")).unwrap();
|
||||
let mut binary = Vec::with_capacity(original.len());
|
||||
let mut rest = original.as_slice();
|
||||
while let Some(at) = rest
|
||||
.windows(current.len())
|
||||
.position(|window| window == current)
|
||||
{
|
||||
binary.extend_from_slice(&rest[..at]);
|
||||
binary.extend_from_slice(newer.as_bytes());
|
||||
rest = &rest[at + current.len()..];
|
||||
}
|
||||
binary.extend_from_slice(rest);
|
||||
assert_ne!(binary, original, "the version string was not found");
|
||||
resign(dir, binary)
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn resign(dir: &Path, binary: Vec<u8>) -> Vec<u8> {
|
||||
let path = dir.join("newer-bsk");
|
||||
fs::write(&path, binary).unwrap();
|
||||
let status = std::process::Command::new("codesign")
|
||||
.args(["--force", "--sign", "-"])
|
||||
.arg(&path)
|
||||
.stderr(std::process::Stdio::null())
|
||||
.status()
|
||||
.unwrap();
|
||||
assert!(status.success(), "codesign {}", path.display());
|
||||
fs::read(path).unwrap()
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
fn resign(_dir: &Path, binary: Vec<u8>) -> Vec<u8> {
|
||||
binary
|
||||
}
|
||||
|
||||
/// Compile a stand-in for a broken release from the body of its `main`.
|
||||
pub fn compiled(dir: &Path, name: &str, main: &str) -> Vec<u8> {
|
||||
let source = dir.join(format!("{name}.rs"));
|
||||
fs::write(&source, format!("fn main() {{ {main} }}")).unwrap();
|
||||
let output = dir.join(format!("{name}{}", std::env::consts::EXE_SUFFIX));
|
||||
let rustc = std::env::var_os("RUSTC").unwrap_or_else(|| "rustc".into());
|
||||
let status = std::process::Command::new(rustc)
|
||||
.args(["--edition", "2021", "-o"])
|
||||
.arg(&output)
|
||||
.arg(&source)
|
||||
.status()
|
||||
.unwrap();
|
||||
assert!(status.success(), "compile {name}");
|
||||
fs::read(output).unwrap()
|
||||
}
|
||||
@@ -1,6 +1,8 @@
|
||||
//! Exercise self-update with a real executable and a local release server.
|
||||
#![cfg(windows)]
|
||||
|
||||
mod release_fixture;
|
||||
|
||||
use std::fs;
|
||||
use std::io::{Cursor, Read, Write};
|
||||
use std::net::{TcpListener, TcpStream};
|
||||
@@ -15,8 +17,6 @@ use std::time::{Duration, Instant};
|
||||
use bsk::daemon::info::DaemonInfo;
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
const MARKER: &[u8] = b"windows-update-regression-fixture";
|
||||
|
||||
struct ReleaseServer {
|
||||
url: String,
|
||||
requests: Arc<AtomicUsize>,
|
||||
@@ -40,7 +40,7 @@ impl ReleaseServer {
|
||||
listener.set_nonblocking(true).unwrap();
|
||||
let url = format!("http://{}", listener.local_addr().unwrap());
|
||||
let manifest = serde_json::to_vec(&serde_json::json!({
|
||||
"version": "999.0.0",
|
||||
"version": release_fixture::newer_version(),
|
||||
"assets": {"windows-x64": {
|
||||
"url": format!("{url}/bsk.zip"),
|
||||
"sha256": Sha256::digest(&archive).iter().map(|byte| format!("{byte:02x}")).collect::<String>(),
|
||||
@@ -138,7 +138,7 @@ fn release_server_waits_for_delayed_and_fragmented_request_headers() {
|
||||
assert!(response.starts_with("HTTP/1.1 200 OK\r\n"), "{response}");
|
||||
let (_, body) = response.split_once("\r\n\r\n").unwrap();
|
||||
let manifest: serde_json::Value = serde_json::from_str(body).unwrap();
|
||||
assert_eq!(manifest["version"], "999.0.0");
|
||||
assert_eq!(manifest["version"], release_fixture::newer_version());
|
||||
}
|
||||
|
||||
struct Fixture {
|
||||
@@ -159,10 +159,8 @@ impl Fixture {
|
||||
let home = tmp.path().join("home");
|
||||
fs::create_dir(&home).unwrap();
|
||||
fs::copy(env!("CARGO_BIN_EXE_bsk"), &exe).unwrap();
|
||||
// A PE overlay distinguishes the replacement without requiring a
|
||||
// second build or changing the executable's behavior/version.
|
||||
let mut binary = fs::read(&exe).unwrap();
|
||||
binary.extend_from_slice(MARKER);
|
||||
// The same program reporting a newer version, as a release must.
|
||||
let binary = release_fixture::newer_bsk(tmp.path());
|
||||
let server = ReleaseServer::new(&binary);
|
||||
Self {
|
||||
_tmp: tmp,
|
||||
@@ -223,13 +221,14 @@ impl Fixture {
|
||||
fs::read(&self.exe).is_ok_and(|binary| binary == self.binary)
|
||||
}
|
||||
|
||||
/// Files next to the executable other than the executable itself.
|
||||
/// Files next to the executable other than the executable itself and
|
||||
/// the update lock, which stays.
|
||||
fn leftovers(&self) -> Vec<String> {
|
||||
fs::read_dir(self.exe.parent().unwrap())
|
||||
.unwrap()
|
||||
.flatten()
|
||||
.map(|entry| entry.file_name().to_string_lossy().into_owned())
|
||||
.filter(|name| name != "bsk.exe")
|
||||
.filter(|name| name != "bsk.exe" && name != ".bsk.exe.update.lock")
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user