mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-31 07:44:57 +08:00
Matthew GrossmanandEvan Lezar
33138516ad
* feat(tmachine): add the Session API and a test-runner package
Add a library API that boots an environment, runs commands in it with timeouts,
copies files in and out, collects diagnostics and tears it down, so a caller can
drive a machine without writing an Ansible playbook per suite. SSH and exec
timeouts and QEMU cleanup on SIGTERM keep a hung guest or a cancelled CI job from
leaving QEMU running.
Add the `openshell-test-runner` flake package, which runs tests/runner with the
toolchains and VM tools it needs.
The Session API and test-runner package are adapted from NVIDIA/OpenShell#4629.
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* build(cargo): add a check that aligned workspaces lock shared crates like the root
`cargo --locked` only checks a lockfile against its own manifest. A workspace
outside the root that builds root crates through path dependencies, such as the
testkit suites building `openshell-sdk`, has its own Cargo.lock, so the two
lockfiles can resolve a shared crate to different versions without any check
failing. That is how aws-lc-sys 0.46 was picked up in #4629, which does not link
against the AWS-LC that Nix provides.
check_cargo_lockfile_alignment.py fails when an aligned workspace locks a
registry crate that the root lockfile also contains at a version the root
lockfile does not have. With --fix it re-locks the workspace starting from the
root lockfile. tests/testkit is the first aligned workspace.
The script and its tests come from #4629 (a37ebe917).
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(conformance): add a sandbox smoke suite built on testkit
Add the `sandbox` crate to the conformance workspace with one testkit test that
creates a sandbox, runs `echo` in it with exec, deletes it and checks it is gone.
It is deliberately small: it exists so the runner can run a real test in a
tmachine environment on CI before the existing suites move to testkit.
The workspace's other crate, the CLI conformance suite, keeps running as it does
today. The conformance workspace joins the lockfile alignment check, so its
lockfile is realigned with the root lockfile.
The test is adapted from the `sandbox-lifecycle` suite in #4629.
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* feat(runner): add a test runner that runs a test family on tmachine
`openshell-test-runner` builds a family's packages (a workspace under
tests/suites/) on the host as a nextest archive for the guest's musl target,
boots one tmachine environment by name with the named installer through the
Session API, runs nextest in it and brings back the JUnit report and log. No
per-suite Ansible playbook is involved:
nix run .#openshell-test-runner -- --environment ubuntu-docker-rootful \
--installer deb --family conformance --package sandbox
The runner points the tests at the system under test with environment
variables: OPENSHELL_GATEWAY (the gateway the installer registered, `--gateway`,
default `openshell`) and OPENSHELL_BIN (the installed CLI). It waits for the
gateway with `testkit-ready` before the tests start. Nextest runs with
`--test-threads 1` and a slow-timeout that terminates a hung test.
A run passes if nextest succeeds and its JUnit report shows at least one test
and no failures. `--filter` narrows the tests with a nextest filterset, and
`--keep` leaves the VM up for debugging. Results from an earlier run are
removed first, and the guest's diagnostics are collected after a failure.
Which tests run in which environments is left to the caller for now; there are
no plans, environment registry or per-test requirements, and a broken test is
`#[ignore]`d.
Most of this comes from #4629, adapted to testkit finding the gateway the way the
CLI does instead of reading a target file.
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(tests): lint and test the runner, tmachine and conformance suite
Run fmt, clippy and nextest for tests/tmachine and tests/runner, and fmt and
clippy for the new `sandbox` suite crate, wherever tests/testkit already runs
them: the mise tasks and Branch Checks. The suite's tests need a gateway, so they
run through `mise run e2e:testkit` against a local Docker gateway and, on CI,
through the test runner rather than the unit-test job.
The lockfile task and the Cargo lockfiles job also run the alignment check, and
its own tests run with `mise run test`.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(tests): run testkit suites on tmachine from a reusable E2E Suites workflow
Add a reusable workflow with an explicit matrix: each row names a tmachine
environment and installer, a family and package, and the runner label. Each
row runs `nix run .#openshell-test-runner` once. The jobs use the same
integration inputs as the existing tmachine lanes, cache the tmachine disks
keyed on the Ansible tree, upload the JUnit report and logs whether or not the
run passes, and an aggregator job reports the overall result.
The only row runs the conformance sandbox suite on amd64 Ubuntu with rootful
Docker and the Debian package, which CI can provision today. Branch E2E calls
the workflow when the test:e2e label is set. The job is not in the Core E2E
result's needs yet, so it starts out advisory. The existing conformance,
feature and driver lanes are unchanged.
Modeled on the E2E Suites workflow in #4629.
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(tests): document the runner and the conformance suite
Describe how the runner runs a family in a tmachine environment, locally and
on CI, in TESTING.md, add a README for the sandbox suite, point the testkit
README at the runner, and add rows for tests/suites, tests/tmachine and
tests/runner to the project-structure tables in AGENTS.md and CONTRIBUTING.md.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(tests): simplify the test runner and tmachine
- Replace tmachine's global QEMU PID registry with until_signal, which
drops the work on SIGTERM/SIGINT/SIGHUP so kill_on_drop stops QEMU.
This also lets Ctrl-C end a --keep run.
- Decide a run's status from nextest's exit code (with --no-tests fail)
instead of parsing JUnit; print the end of the nextest log on failure.
- Move gateway diagnostics from tmachine's Session into the runner, drop
Session::run_playbooks, and make Session own its VM directly.
- Read TMACHINE_SSH_PORT once and reject invalid values; drop the racy
port pre-check.
- Build the runner with Nix so tmachine's firmware paths are compiled in,
and set ANSIBLE_CONFIG only in the wrapper.
- Copy the bundle directory in directly, fold lib.rs into main.rs, and
make the gateway name required.
- Drop the semver hint from the lockfile alignment check and trim
duplicated docs.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(tmachine): keep the SSH port check and name a run with no tests
Restore the check that the SSH forward port is free before QEMU starts. QEMU
reports a taken port only by exiting, so a layer build would otherwise run its
playbooks against the VM that already holds the port and cache the result.
Report nextest's no-tests exit code as "no test ran" instead of a bare exit
status.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(runner): keep logs from a timed-out run and gate on the E2E suites
nextest now writes its output to a file in the guest, and the runner copies
nextest.log and junit.xml out after every run, so a run that times out or
loses SSH still leaves a log. A failed JUnit copy no longer turns a test
failure into an error.
The runner exits with 1 when a test failed and 2 when the run could not
complete, and annotates the GitHub Actions run with which it was.
The E2E suites lane is now part of the Core E2E result. Its disk cache key
also covers tmachine's cache versions in setup.rs and install.rs.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(e2e): push test runner builds to Cachix
E2E Suites set up Nix without a Cachix token, so every job rebuilt
openshell-test-runner from source. Accept an optional CACHIX_AUTH_TOKEN
secret, pass it to setup-nix, and pass it from Branch E2E, as Branch Checks
and the binary builds already do.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>