e30c938bb1
* test(server): add a local PostgreSQL test runner
The openshell-server tests that need a real PostgreSQL server are
ignored by default and had no shared way to run. The mutation-replay
test reads its own OPENSHELL_REPLAY_TEST_DATABASE_URL, so every
contributor had to provision a database by hand, and the advisory-lock
tests that follow in this series need the same setup.
Add mise run test:rust:postgres. It runs every ignored postgres_* test
in openshell-server against OPENSHELL_TEST_POSTGRES_URL, or starts a
disposable PostgreSQL container with Docker or Podman (CONTAINER_ENGINE
selects one) on the image pinned by the Kubernetes e2e fixture and
removes it on exit. Tests run one at a time because advisory locks are
database-wide. The runner always points the legacy replay variable at
the selected database, so an inherited URL cannot send that test to a
different server. test:postgres-runner checks that selection with a fake
cargo and needs no database or container engine. CI does not run the
PostgreSQL tests; TESTING.md documents the task.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): hold the mutation guard while deleting a provider
DeleteProvider checked that no sandbox referenced the provider and then
deleted the record without holding the sandbox mutation guard. Sandbox
create and provider attach take that guard while they write provider
references, so one of them could add a reference after the attached
sandbox check passed, and the delete then removed a provider that a
sandbox spec still named.
Take the guard before the attached sandbox check, as provider create and
update already do, so the check and the delete run against a stable
set of sandbox references, and reject an empty name after authorization
but before the guard. A new test holds the guard, attaches the provider
while the delete waits, and asserts that the delete fails with
FailedPrecondition and leaves the provider in place.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): release staged credentials when a refresh fails early
A provider credential refresh stages the minted values under new
credential handles before it commits the provider update, and deletes
those handles when validation or persistence fails. Two earlier returns
skipped that cleanup. The minted expiry was converted to a protobuf
timestamp only after staging, so an expiry outside the timestamp range
failed the refresh and left the staged values in the credential driver.
A failure to acquire the sandbox mutation guard returned the same way.
Convert the expiry before staging anything, so a bad value fails before
any handle exists, and delete the staged handles when the guard cannot
be acquired. A new test refreshes a stored credential with an expiry of
i64::MAX and asserts that the call fails, the provider keeps its
original handles, and the credential driver holds the same number of
values as before. The guard failure path has no test here, because the
SQLite guard used by unit tests cannot fail.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): acquire mutation locks on a dedicated PostgreSQL lock pool
The cross-object mutation guard took its advisory-lock session from the
10-connection data pool and closed it on every release, so each guarded
mutation opened a new database connection, and lock holders competed
with their own critical sections for data connections. A lock wait that
hit the 10 second lock_timeout surfaced as INTERNAL.
Add a lazy, dedicated 4-connection lock pool and a new
persistence::mutation_lock module for the lock key, mode and set types.
Store::acquire_distributed_mutation_guard now takes a lock set and a
deadline, and each lock statement sets lock_timeout to the time left
before that deadline, so a cancelled acquisition never waits past it on
the server. Returned connections are scrubbed with
pg_advisory_unlock_all() and reused, and each return is bounded by 5
seconds so a stalled session cannot keep a pool permit.
A lock statement that PostgreSQL ends with SQLSTATE 55P03 fails with a
new LockTimeout error, and so does a wait for a lock connection during
which every one was checked out at some point, or that started with less
than a second left, too little to open one. The API returns it as
UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT and a 1 second retry
delay, because the mutation lock was not acquired, so the request's
guarded writes did not run. Only the guard acquisition classifies 55P03;
any other 55P03 stays a database error. A lock connection that
PostgreSQL does not open by the deadline despite at least a second to do
so (refused, out of connection slots, starting up, or not answering, all
of which SQLx retries silently) is a database error, not a lock timeout.
Mutations still take the legacy global key exclusively, so old and new
replicas keep excluding each other during a rolling upgrade. Each
PostgreSQL-backed gateway pod can now open up to 14 database
connections (10 data plus 4 lock) instead of 10, and the chart's
autoscaling.maxReplicas comment says what its default of 4 needs. The
SSH host identity lock stays on a data connection, because sandbox
creation takes it while holding its mutation guard and a second
lock-pool connection there could exhaust the pool. The provider
credential refresh lock stays there too, so a refresh still progresses
while every lock connection is in use.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): partition mutation locks by workspace and sandbox
Every cross-object mutation, and every lifecycle and supervisor update,
took one process-wide mutex and, on PostgreSQL, one fleet-wide advisory
lock. Unrelated sandboxes and providers therefore serialized across all
replicas, and one slow critical section delayed every other mutation in
the fleet.
Replace that lock with hierarchical intention locks on a global key, one
key per workspace and one key per sandbox, each held shared or
exclusive. Global settings and policy writers and platform-scope profile
writers hold the global key exclusively. Provider and workspace-scoped
profile writers, including DeleteProvider and credential refresh, hold
their workspace key exclusively. Sandbox writers and supervisor reports
hold their sandbox key exclusively under shared global and workspace
keys. Lifecycle, driver-watch and reconcile paths take only
process-local keys and keep relying on compare-and-swap across replicas.
Workspace and sandbox keys are domain-separated SHA-256 prefixes. Each
guard takes its local keys and then its advisory locks in ascending
order under one 10 second deadline, and no path nests guards. The
settings mutex is removed, since the global and sandbox keys cover it.
Sandbox settings are keyed by sandbox name, which a sandbox key no
longer covers once another sandbox reuses the name, so a settings
update returns NOT_FOUND unless its sandbox still holds the name after
the read, and ABORTED if the settings row was recreated before the
write.
The global key keeps its legacy value, so older replicas, which hold it
exclusively for every mutation, still exclude new ones during a rolling
upgrade. Guards now also give up after 10 seconds on SQLite, and
provider refresh reports a guard timeout as UNAVAILABLE instead of
INTERNAL. A supervisor report for a missing sandbox returns NOT_FOUND.
Startup endpoint-status reconciliation still holds the global key for
its whole scan.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): reconcile endpoint status on startup one sandbox at a time
Startup endpoint-status reconciliation held the global mutation key
exclusively for its whole scan. On PostgreSQL that blocked every guarded
mutation on every replica until the starting gateway had walked all
sandboxes, and the scan first waited behind every in-flight sandbox
mutation. It also paged with an offset, so sandboxes deleted mid-scan
shifted rows between pages, and a write that raced the scan (a Conflict
from a lifecycle update on another replica, or a sandbox deleted after
it was listed) failed gateway startup.
Walk the sandboxes with keyset paging instead, and reset each candidate
under its own sandbox-scoped guard, up to four at a time to match the
lock pool. A candidate with a fresh shared owner is skipped before any
guard is taken. Under the guard the reconciliation re-checks the owner,
re-reads the sandbox, skips it when it is gone or no longer carries
endpoint status, and writes against the version it just read. A Conflict
or a database error at write time, which is how a row deleted after the
re-read surfaces, is retried with a fresh read, at most five attempts
per sandbox. A lock timeout on the guard is retried the same way, so a
holder of the sandbox, its workspace or the global key that releases
within the retry window cannot stop a replica from starting. No
lock-timeout retry starts once the scan has run for three lock timeouts,
so a scan stuck behind a long holder fails within about 40 seconds,
inside the chart's 60-second startup probe, and logs the sandbox it gave
up on. Store::list_all_messages loses its only caller and is removed.
Reconciliation stays fail-closed: a holder that outlasts the retry
window, an owner lookup failure, or a sandbox that keeps changing still
fails startup, so stale endpoint success is never served. User-visible
change: a restarting gateway no longer stalls mutations of unrelated
sandboxes and providers on its peers while it reconciles.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): skip the endpoint-status lock when a session was replaced
When a supervisor session ends, the gateway resets that sandbox's tool
server endpoint evidence unless a replacement session already owns
endpoint observation. The replacement check ran only after the reset
had read the sandbox and taken its sandbox mutation guard, so a reset
whose supervisor had already reconnected still queued behind other
mutations of that sandbox, and on PostgreSQL it used a lock-pool
connection, only to find it had nothing to do.
Check for a replacement first: a live local session, or a fresh owner
record on a peer. When one exists, return without reading the sandbox
or taking the guard. Otherwise take the guard and repeat the same check
under it, because a replacement's pre-acknowledgement reset runs under
the same sandbox key and resetting after it would wipe the
replacement's fresh evidence. Both checks share one helper, so they
cannot drift apart. When no replacement exists, the reset costs one
extra owner-row read.
This helps most when a replica notices a dead stream late, after the
supervisor has already reconnected elsewhere (a network partition or a
keepalive expiry).
Deferring to a replacement leaves the old session's evidence to the
replacement's pre-acknowledgement reset, which can time out on its
bounded mutation lock wait. That reset's failure path removed the
replacement and released its owner record but scheduled no
invalidation, so the old session's success evidence could outlive both
sessions. An old session that ended after its replacement registered
already deferred this way, and the unguarded check now makes every
disconnect that finds the replacement defer too. When the reset fails
while the replacement is still current, also schedule the retried
invalidation that follows a session end. That failure path keeps the
lifecycle demotion from abandon_session_setup, but runs it in the
spawned task before the retry and only while this session is still
current, so a timed-out reset returns promptly.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* test(server): cover staged credential cleanup on a lock timeout
A provider credential refresh deletes its staged handles when it cannot
acquire the provider mutation guard, but that path had no test: the
SQLite guard used by unit tests could not fail. Mutation locks now
acquire under a deadline that tests can shorten, so the timeout path
can be exercised directly.
The new test shortens the lock deadline to 50 ms, holds the workspace
mutation guard, and refreshes a provider that already stores an AWS
access key. The refresh returns UNAVAILABLE with the
MUTATION_LOCK_TIMEOUT reason, the provider keeps its original handle,
which still resolves to the old key, and the credential driver holds
the same number of values as before the refresh.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): export mutation lock wait and timeout metrics
A handler mutation guard can now wait up to 10 seconds for its keys and
then fail with UNAVAILABLE, but nothing showed operators how long guards
waited or how often they gave up, short of reading warning logs on each
replica.
Export openshell_server_mutation_lock_wait_seconds, a histogram of the
time one guard took to acquire all of its keys (local registry plus
PostgreSQL advisory locks), recorded on success, and
openshell_server_mutation_lock_timeouts_total, a counter of guard
acquisitions that timed out. Both carry a bounded scope label: global,
workspace or sandbox. The platform scope, whose workspace is empty,
counts as global. The histogram uses the same 1 ms to 15 s buckets as
the relay and peer histograms, and the timeout counter starts at 0 for
every scope, so rate() works on an idle replica. The timeout warning
now also logs the scope. A guard that fails for another reason, such as
a lock connection PostgreSQL did not open, is not counted as a timeout
and logs a "mutation lock acquisition failed" warning instead.
Lifecycle, driver-watch and reconcile paths take only process-local
locks and never time out, so they are not recorded.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* test(server): add PostgreSQL mutation lock integration tests
The scoped mutation locks were only unit-tested on SQLite, where the
database layer is a no-op. Add ignored postgres_* tests that run the
advisory-lock layer against a real PostgreSQL through mise run
test:rust:postgres, each in its own disposable schema (TestSchema).
persistence/mutation_lock_pg_tests.rs uses two stores on one database
as two replicas. It covers disjoint sandbox scopes holding at once, a
workspace key blocking only sandboxes in that workspace, the global key
blocking every scope, and exclusion in both directions against a raw
holder of the legacy global key, whose 55P03 stays a database error
outside a guard acquisition. It checks that timed-out and cancelled
acquisitions release the keys they already took, that a released lock
connection returns to the pool without residual locks, that a release
stalled on the network closes its session and the pool recovers, and
that the lock pool never grows past its size. It also checks that a
full lock pool times out as lock contention, while a lock connection
that PostgreSQL never answers fails as a database error.
compute/mutation_guard.rs runs the random scope mix across two
PostgreSQL runtimes and checks that a by-id guard waits for a workspace
writer on the other replica. The provider refresh lock test moves onto
the runner too, and checks that a refresh proceeds while every lock
connection is held.
A bench_postgres_* capacity check measures a paced burst of 500
reconnects 12 ms apart into one replica and fails when the p99 lock
wait reaches a fifth of the lock timeout. Machine load moves those
waits, so mise run test:rust:postgres skips it and mise run
test:rust:postgres:bench runs it alone, through a new
OPENSHELL_TEST_POSTGRES_PREFIX setting in the runner.
The tests add test-only helpers: the lock pool's size and idle count
and the backend pid of a held lock session. Nothing changes outside
tests and the test runner.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* ci: run the PostgreSQL-backed server tests
Branch Checks now runs the postgres_* tests through the same runner
developers use, on an x86_64 Linux runner with Docker. They cover lock
scope exclusion, interop with replicas still on the old global key,
cancellation, lock-pool bounds and connection failures, which no other
CI lane checks directly. The load-sensitive bench_postgres_* capacity
checks stay local.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* docs: document scoped mutation locks and lock-pool sizing
In the High Availability guide, size PostgreSQL for 14 connections per
gateway pod, 10 for data and 4 for the new lock pool, so three or more
replicas now exceed what the PostgreSQL default leaves, and keep
headroom for lock sessions that cancelled requests leave behind.
Explain which changes lock a sandbox, a workspace, or the whole fleet,
and that a lock wait over 10 seconds returns a retryable UNAVAILABLE
with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to
raise max_connections to the 14-per-pod size before upgrading, because
the rollout already runs new pods at that count, and warns that older
replicas keep serializing every mutation until the rollout finishes.
Monitor Capacity gains the lock metrics and their queries, and names
slow driver, credential, middleware, or profile source calls as another
cause of lock waits.
List the lock wait histogram and timeout counter, with their scope
label, on the Gateway Metrics page. The counter also counts background
work and leaves out lock connections that PostgreSQL does not open
although at least a second of the wait remained.
In the cluster debugging skill, replace the single advisory-lock
paragraph with the lock levels and pool, what each timeout detail
means, how a lock connection that PostgreSQL does not open shows up,
when terminating a lock backend is safe, a pg_locks query that shows
holders and waiters, and how to spot the global key. The per-pod greps
also read the timeout counter, and new rows cover lock timeouts and
lock connections that PostgreSQL does not open.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): check mutation lock ordering in debug builds
The mutation lock ordering rules were prose only, so a nested guard on
a different key passed every test and could only deadlock under
contention. Debug builds now register each held guard with the Tokio
task (or the thread, outside a task) that acquired it, and panic before
waiting when a local key, a blocking lifecycle gate or the SSH identity
lock would break rules 1, 4 or 5. Startup endpoint-status
reconciliation and the tests that hold a guard while driving guarded
code run under lock_order::branch(), which marks deliberate concurrency
within one task. Release builds compile the check out.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): do not hold gateway shutdown for the endpoint-status retry
When a replacement supervisor session failed its endpoint-status reset,
the spawned cleanup task kept the session lifetime guard through
retry_endpoint_status_after_supervisor_disconnect. That retry only ends
once the reset is durable, so a gateway shutdown during a database or
lock outage waited the full 10 s and reported ownership cleanup as
incomplete. The guard now covers only the lifecycle demotion, as in
finish_supervisor_session, and a SQLite test checks that shutdown
completes while the retry is still blocked.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* refactor(server): make the platform profile lock scope explicit
Platform-scope provider profile writers (import, update, delete) pass an
empty workspace on purpose and need X(global), because they validate
sandboxes in every workspace. MutationScope::profiles() now names that
escalation and picks Global for the platform scope, instead of relying
on MutationScope::Workspace("") to mean it. An empty workspace name now
trips a debug assertion, and release builds keep X(global) as a
fail-safe that over-locks rather than under-locks.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): amortize local mutation lock table sweeps
LocalMutationLocks swept its whole table on every key lookup, so a
burst of N concurrent acquisitions on one replica cost O(N^2) under a
blocking mutex. The table is now swept only when a new key grows it
past the larger of 64 and twice the live entries left by the last
sweep, which makes acquisition amortized O(1). Released entries,
including those of cancelled waiters, stay bounded at about twice the
peak live set. The lifecycle gate registry keeps its per-call sweep.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): make the database pool ceiling configurable
Both stores hardcoded their pool ceiling: 10 connections for Postgres, 5
for on-disk SQLite. One pool is shared by every database-backed RPC, so
that number is also the gateway's ceiling on concurrent database work.
Once every connection is checked out, callers queue on acquire and sqlx
logs "time to acquire exceeded slow threshold"; sandbox creates then time
out and are retried, which adds load rather than shedding it.
The right ceiling is deployment-specific — it depends on how many gateway
replicas share the database and what max_connections the server itself
allows — so it cannot be a single number baked into the binary.
Expose it on the same config surface as the rest of the gateway's
settings: --db-max-connections, OPENSHELL_DB_MAX_CONNECTIONS, and the
TOML key database_max_connections, resolved in that precedence order.
Helm deployments set the key through gatewayConfig. Omitting it keeps
each backend's previous value, so existing deployments do not move.
Values below 1 are rejected rather than silently replaced by the default:
a zero pool would block every acquire, and quietly ignoring a typo would
reproduce the ceiling the operator is trying to lift. An in-memory SQLite
database stays pinned to one connection, since the database lives inside
that connection.
Refs: #2561
Signed-off-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com>
(cherry picked from commit 33517b32bd)
[emacchi: dropped the server.dbMaxConnections Helm value; operators set
the key through gatewayConfig]
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): make the mutation lock pool size configurable
The mutation lock pool was fixed at 4 connections per replica. That
number rests on one reconnect-burst bench rather than scale testing, so
a gateway that serves many more sandboxes needs a way to raise it
without a rebuild. Add --db-lock-max-connections,
OPENSHELL_DB_LOCK_MAX_CONNECTIONS and the database_lock_max_connections
gateway.toml key next to the data pool knob, resolved in the same order;
Helm deployments set it through gatewayConfig. The default stays 4,
zero is rejected, SQLite ignores it with a warning, and the gateway logs
both pool sizes at startup. PostgreSQL now needs at least 2 data
connections, because the SSH identity lock keeps one while it queries
the pool. The connection sizing docs, the HA guide formula and the
debug skill now count data plus lock connections.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): export mutation lock errors and lock connections in use
A mutation lock acquisition that failed without timing out, such as a
lock connection PostgreSQL never opened or a failed lock statement,
only logged a warning, and readiness stays healthy because it pings the
data pool. Count those failures in
openshell_server_mutation_lock_errors_total{scope}, kept apart from
timeouts because they point at PostgreSQL capacity or connectivity
rather than contention. Export
openshell_server_mutation_lock_connections_in_use as a gauge slot held
by each checked-out lock connection, next to
openshell_server_mutation_lock_connections_capacity, which reads the
configured lock pool size; both exist only with PostgreSQL. The metrics
catalog, the HA guide's signals and PromQL examples, and the debug
skill cover the new series.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): time mutation lock holds and test a stalled holder
Nothing bounds how long a mutation guard is held, so a holder stuck in
a compute driver, credential backend or middleware call keeps its scope
locked while waiters time out. Each guard now records its hold in
openshell_server_mutation_lock_hold_seconds{scope} when it is released,
and logs a warning when the hold outlasted the lock wait timeout, so
operators can find the replica and scope behind those timeouts. A
PostgreSQL test with a stalled holder pins the current contract:
waiters on its sandbox, workspace and global scopes fail by their
deadline, other scopes proceed, the holder keeps its key, and a full
lock pool on one replica leaves the other working. The metrics catalog,
the HA guide and the debug skill describe the new signals, which
observe holds without bounding them.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
---------
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
Co-authored-by: divesh <dgude@nvidia.com>
Co-authored-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com>
23 KiB
Testing
Running Tests
mise run test # Rust + Python unit tests
mise run e2e # End-to-end tests (starts a Docker-backed gateway)
mise run ci # Everything: lint, compile checks, and tests
Test Layout
crates/*/src/ # Inline #[cfg(test)] modules
crates/*/tests/ # Rust integration tests
python/openshell/ # Python unit tests (*_test.py suffix)
e2e/python/ # Python E2E tests (test_*.py prefix)
e2e/rust/ # Rust CLI E2E tests
Rust Tests
Unit tests live inline with #[cfg(test)] mod tests blocks. Integration tests
go in crates/*/tests/ and are named *_integration.rs.
Use #[tokio::test] for anything async:
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn store_round_trip() {
let store = Store::connect("sqlite::memory:").await.unwrap();
store.put("sandbox", "abc", "my-sandbox", b"payload").await.unwrap();
let record = store.get("sandbox", "abc").await.unwrap().unwrap();
assert_eq!(record.payload, b"payload");
}
}
Run Rust tests only:
mise run test:rust # cargo test --workspace
Rust validation checks tracked Cargo lockfiles; run mise run rust:lockfiles:check to check them directly. If one is stale, refresh it with Cargo using its adjacent manifest, review the diff, and commit the update.
PostgreSQL-backed tests
Tests that need a real PostgreSQL server, such as advisory-lock concurrency
across two stores, are #[ignore]d and named postgres_*. Run them with:
mise run test:rust:postgres
The task starts a disposable PostgreSQL container with Docker or Podman
(set CONTAINER_ENGINE to choose), runs the tests one at a time, and removes
the container. Each test works in its own temporary schema, except the replay
and provider refresh lock tests, which use the default schema. To use your own
disposable database, set OPENSHELL_TEST_POSTGRES_URL. The task overrides
OPENSHELL_REPLAY_TEST_DATABASE_URL so legacy tests use that same database.
Never point it at a database that a running gateway uses: the tests take
fleet-wide advisory locks.
Branch Checks runs them on x86_64 Linux.
Load-sensitive capacity checks are named bench_postgres_* instead, so that
task skips them. Run them on an idle machine with
mise run test:rust:postgres:bench.
Native Windows validation
Use mise run --skip-tools pre-commit with the existing Rust/MSVC toolchain.
Windows now checks tracked Cargo lockfiles through PowerShell rather than
skipping them.
mise run --skip-tools sdk:ts:ci selects the x64 Biome executable on Windows
(including ARM64 hosts running it under emulation), resolves the protobuf
plugin through its Windows .cmd launcher, and installs the matching locked
ARM64 Rolldown binding when Node itself is ARM64. The helper preserves lockfile
resolution; it must not upgrade unrelated test dependencies.
mise run --skip-tools go:ci retains race detection except on Windows ARM64,
where Go does not support it. Windows token-file tests explicitly skip POSIX
mode-bit assertions; those skips do not establish Windows ACL protection.
Use a checkout with LF text files when running Unix-shell fixture checks.
Python Unit Tests
Python unit tests use the *_test.py suffix convention (not test_* prefix)
and live alongside the source in python/openshell/. They use mock-based
patterns with fake gRPC stubs:
def test_exec_python_serializes_callable_payload() -> None:
stub = _FakeStub()
client = _client_with_fake_stub(stub)
def add(a: int, b: int) -> int:
return a + b
result = client.exec_python("sandbox-1", add, args=(2, 3))
assert result.exit_code == 0
Run Python unit tests only:
mise run test:python # uv run pytest python/
E2E Tests
E2E tests run against a live gateway. By default, mise run e2e starts an
ephemeral standalone gateway with the Docker compute driver, runs the suite,
and cleans it up afterward. To run the suite against an existing plaintext
gateway, set OPENSHELL_GATEWAY_ENDPOINT:
OPENSHELL_GATEWAY_ENDPOINT=http://127.0.0.1:18080 mise run e2e
Raw endpoint mode is HTTP-only. Use a named gateway config when a gateway requires mTLS.
mise run test:gateway-config validates generated gateway TOML without a live
runtime. It covers the current schema and the Podman in-tree versus external
driver configuration boundary.
Python E2E (e2e/python/)
mise run e2e:python builds openshell/e2e-python:dev from
e2e/python/Dockerfile.workload and selects it only for the test gateway.
This Noble-based fixture supplies the sandbox user, Python tooling, Git,
and a writable /sandbox/.venv. Its Python version comes from .python-version
so cloudpickle code objects match the test runner. The production workload
default remains the unmodified NVIDIA Ubuntu image.
The Rust Docker harness also selects this fixture for tests that need tools
or the named user. Explicit --from or --template arguments and the default-image
tests retain their own image selection. Docker sandbox and support-container
fixtures use the fixed image name; no workload-image override is needed.
Podman, VM, and Kubernetes retain their existing pinned, pullable fixture and
image setup. Conformance-only runs do not need the Docker fixture.
Build the Docker fixture separately with mise run e2e:workload:build.
Tests use the sandbox fixture from conftest.py to create real sandboxes:
def test_exec_returns_stdout(sandbox):
with sandbox(delete_on_exit=True) as sb:
result = sb.exec(["echo", "hello"])
assert result.exit_code == 0
assert "hello" in result.stdout
Sandbox.exec_python
exec_python serializes a Python callable with cloudpickle, sends it to the
sandbox, and returns the result. Because cloudpickle serializes module-level
functions by reference (which fails inside the sandbox), use one of these
patterns:
Closures from factory functions:
def _make_adder():
def add(a, b):
return a + b
return add
def test_addition(sandbox):
with sandbox(delete_on_exit=True) as sb:
result = sb.exec_python(_make_adder(), args=(2, 3))
assert result.stdout.strip() == "5"
Bound methods on local classes:
def test_multiply(sandbox):
class Calculator:
def multiply(self, a, b):
return a * b
with sandbox(delete_on_exit=True) as sb:
result = sb.exec_python(Calculator().multiply, args=(6, 7))
assert result.stdout.strip() == "42"
Shared Fixtures (e2e/python/conftest.py)
| Fixture | Scope | Purpose |
|---|---|---|
sandbox_client |
session | gRPC client connected to the active gateway |
sandbox |
function | Factory returning a Sandbox context manager |
Rust CLI E2E (e2e/rust/)
Rust-based e2e tests that exercise the openshell CLI binary as a subprocess.
They live in the openshell-e2e crate and use a shared harness for sandbox
lifecycle management, output parsing, and cleanup.
Exposed service URLs use virtual hostnames for gateway routing. Host-side tests
must connect the TCP socket directly to a reachable gateway listener address,
normally loopback, and send the service URL authority in the HTTP Host
header. Do not resolve *.openshell.localhost; resolver support for arbitrary
.localhost subdomains varies across local and CI environments.
Treat the advertised service URL scheme as authoritative. For HTTPS, use the
virtual service hostname for TLS SNI and the configured gateway trust roots.
When the listener requires mTLS, present the active gateway client identity;
the local e2e wrappers register these materials under
$XDG_CONFIG_HOME/openshell/gateways/$OPENSHELL_GATEWAY/mtls/. Do not downgrade
an HTTPS service URL to plaintext when dialing loopback. Parse the URL and load
TLS material before entering a readiness loop so permanent configuration
errors fail immediately. Retry only transient connection failures and
documented readiness responses, and include the last observation in timeout
diagnostics.
Verify exposed-service tests in both the default local mode and the CI-equivalent HTTPS mode:
mise run e2e:rust
OPENSHELL_ENABLE_LOOPBACK_SERVICE_HTTP=false mise run e2e:rust
When more than one test needs this behavior, put the transport in the shared
Rust e2e harness and require callers to use it instead of duplicating DNS,
HTTP Host, TLS SNI, and mTLS handling.
Suites:
- Common suite (
--features e2e) - driver-neutral CLI behavior, sandbox lifecycle, sync, port forwarding, policy, and provider tests. - CLI conformance (
tests/suites/conformance) - portable Cargo tests for lifecycle, mechanistic drafts, file transfer, and the sandbox-local API, including agent-authored permission requests. Driver E2E runs the complete Cargo test package. The installed-artifact conformance suite runs the same tests from a nextest archive and offers a focusedpolicy-advisortestsuite for manual integration runs. - Driver suites (
--features e2e-docker,e2e-podman,e2e-kubernetes, ore2e-vm) - CLI conformance plus the common and driver-specific coverage for the selected deployment. - Docker suite (
--features e2e-docker) - includes Docker-only coverage such as Dockerfile image builds, Docker preflight checks, and managed Docker gateway start. - Docker GPU suite (
--features e2e-docker-gpu) - Docker suite plus GPU sandbox smoke coverage. - VM suite (
--features e2e-vm) - runs e2e tests on a VM. - Kubernetes credential-driver suite (
--features e2e-kubernetes-credential-drivers) - targeted Kubernetes Secrets and Vault provider credential storage coverage.
GPU device-selection tests compare OpenShell sandboxes against a plain Docker or
Podman container that requests --device nvidia.com/gpu=all. The probe image
defaults to the image used by the gateway stage in
deploy/docker/Dockerfile.images; set OPENSHELL_E2E_GPU_PROBE_IMAGE to
override it. Per-device checks run only for NVIDIA CDI device IDs reported by
the runtime's discovered devices list, so WSL2 hosts that expose only
nvidia.com/gpu=all skip the index-based cases. Exact CDI device selection is
passed through --driver-config-json with the active Docker or Podman driver
key.
Run the Docker-backed Rust CLI e2e suite:
mise run e2e:docker
Run the portable CLI conformance suite against the gateway selected in your OpenShell CLI configuration:
cargo build --package openshell-cli
OPENSHELL_BIN="$PWD/target/debug/openshell" \
cargo test \
--locked \
--manifest-path tests/suites/conformance/Cargo.toml \
--package openshell-test-conformance-cli \
--no-fail-fast \
-- \
--test-threads=1 \
--nocapture
The gateway must already be installed, reachable, and selected before the tests
start. The test suite does not provision a gateway or select a compute driver.
Set OPENSHELL_BIN to another executable to test a different prebuilt CLI.
The phase-1 scenario verifies the complete CLI-to-gateway-to-driver path without
depending on how the gateway was installed or which driver is configured. It
requires machine-readable gRPC status, creates a uniquely named detached
sandbox with the configured default image, verifies the sandbox is Ready by finding its
unique name in paginated JSON list output, executes echo with a run-specific
marker, deletes the sandbox, and verifies that its name no longer appears.
Driver suites enable the same profile
instead of maintaining a separate smoke implementation. Sandbox lifecycle,
label matrices, VM overlay, and TLS-key permission assertions remain regular
E2E coverage.
Each invocation prints a ten-character run ID before creating resources.
Conformance sandboxes use names such as ct-<run-id>-01. The runner tracks the
exact name and uses it for cleanup; phase 1 does not add ownership labels.
The runner deletes owned resources after both success and failure. If the test process is interrupted before cleanup, locate leftovers without touching unrelated gateway state:
openshell sandbox list --output json
openshell sandbox delete <sandbox-name>
Gateway-backed Rust E2E tasks run the CLI conformance Cargo tests against the configured gateway, then run any lane-specific Rust tests that still apply. Run the Podman-backed Rust CLI e2e suite:
mise run e2e:podman
Run the portable subset in a disposable rootless Podman guest:
nix run .#build-artifacts
nix run .#tmachine -- test fedora-podman-rootless binaries e2e-podman
nix run .#tmachine -- test fedora-podman-rootless binaries driver-podman
Print the exact tmachine archive selection as a shell PODMAN_CI_TESTS array:
nix run .#generate-podman-e2e-ci-tests
The e2e-podman testsuite runs a nextest archive built with the corresponding
Rust feature and preloads its Python workload image into the rootless Podman
store. The separate driver-podman testsuite compares OpenShell and direct
Podman user-namespace mappings for the default, auto, keep-id, and private
profiles. The E2E archive excludes binaries that still depend on wrapper-owned
gateway controls, SPIFFE fixtures, the caller driver-config setting, or other
guest setup. The driver-podman suite replaces the removed podman_userns E2E
binary. tests/artifacts.nix keeps the follow-up exclusions explicit and uses
the same filter for the generated inventory, so excluded binaries cannot appear
as false passes or silently re-enter the archive.
The oci-image feature testsuite (tests/suites/features/oci-image) checks
OCI image identity and working-directory behavior shared by the Docker and
Podman drivers against installed deb packages on Ubuntu and rpm packages on
Fedora. CI runs it on Docker rootful, Podman rootful, and Podman rootless guests:
nix run .#tmachine -- test ubuntu-docker-rootful deb oci-image
The shared suite also runs in the existing ARM64 Docker E2E workflow for
branch checks and both release workflows. That interim lane compiles the same
test crate natively and uses the architecture-matched prebuilt candidate CLI
and gateway plus the candidate multi-architecture sandbox and supervisor
images through e2e/with-docker-gateway.sh. The tmachine lanes still use
x86_64 artifacts; they do not replace ARM64 image coverage.
Run it against a local Docker or Podman gateway:
mise run e2e:oci-image
mise run e2e:oci-image:podman
Both tasks set OPENSHELL_TEST_CONTAINER_ENGINE, the command that builds test
images into the gateway's image store.
Run the VM-backed Rust CLI e2e suite:
mise run e2e:vm
Run the targeted Kubernetes credential-driver e2e suite. This deploys an OpenBao fixture for the Vault-compatible driver path and validates Kubernetes Secrets and Vault storage backends one at a time:
mise run e2e:kubernetes:credential-drivers
Kubernetes E2E (e2e/rust/e2e-kubernetes.sh)
Kubernetes e2e tests deploy an OpenShell gateway into a real Kubernetes cluster
via Helm and run the Rust e2e suite against it. On vanilla Kubernetes the harness
reaches the gateway through kubectl port-forward; on OpenShift it instead uses a
passthrough Route secured with mandatory mTLS (see the OpenShift note below).
Run with an ephemeral k3d cluster (macOS; created and torn down automatically):
mise run e2e:kubernetes
Target an existing cluster (kind, k3d, or OpenShift):
OPENSHELL_E2E_KUBE_CONTEXT=my-context mise run e2e:kubernetes
Scope to a single test for local debugging:
OPENSHELL_E2E_KUBE_TEST=smoke mise run e2e:kubernetes
OpenShift: when the target cluster exposes the route.openshift.io API
group, the harness automatically applies SCC-compatible Helm overrides, grants
the required SCCs (privileged to openshell-sandbox, and anyuid to the
PostgreSQL fixture for DB scenarios), and drives the gateway through a
passthrough Route with mandatory mTLS instead of port-forward. No extra flags are
needed, but oc must be installed and authenticated against the target cluster
with permission to modify SCC bindings (oc adm policy add-scc-to-user) — the
harness exits early if oc is missing. The SCC grants and extracted client
mTLS material are removed during cleanup, including on failure or interrupt.
On a remote cluster, drop the e2e-host-gateway feature. Those tests rely
on the sandbox-side host.openshell.internal alias reaching the machine running
the tests, which is unreachable from pods on a remote cluster, so they fail.
Left enabled, the host_gateway_alias suite fails because
host.openshell.internal does not resolve inside the pod, so the gateway
SSRF-denies the request (DNS resolution failed / ssrf_denied) — a networking
property of remote pods, not a gateway or transport fault. Override
OPENSHELL_E2E_KUBERNETES_FEATURES to exclude it:
OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \
OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \
mise run e2e:kubernetes
On an existing cluster the harness builds the CLI from your branch but pulls the
published gateway/supervisor image (default tag latest). The CLI and the
image can therefore be different versions. If tests fail because of this version
difference — for example, sandbox tests fail with Pod exists with phase: Failed
or connect-based tests stall because the deployed image predates a feature your
branch CLI needs — set IMAGE_TAG to an image that matches your branch.
The latest tag lags to the last semver release, so it is often older than
main. Two better choices:
IMAGE_TAG=dev— a floating tag that tracks the latestmainbuild. Good for an ad-hoc run when your branch is close tomainHEAD. Because it floats, two runs on different days can pull different images, so it is not reproducible.- Pin the exact commit your branch is based on — deterministic and immune to
a floating tag moving. Published tags are the full 40-char git SHA (semver tags
without a
vprefix also exist but only for released versions):
OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \
OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \
IMAGE_TAG=$(git rev-parse "$(git merge-base HEAD upstream/main)") \
mise run e2e:kubernetes
To pin a specific released version, use its semver tag without a v prefix
(0.0.115, not v0.0.115):
OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \
OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \
IMAGE_TAG=0.0.115 \
mise run e2e:kubernetes
A semver tag matches a released commit, which may be behind main; if your
branch CLI needs a newer feature, pin the SHA of your branch's base instead.
Confirm a tag exists before relying on it (set TAG to the tag you plan to use):
TAG=0.0.115
skopeo inspect "docker://ghcr.io/nvidia/openshell/gateway:${TAG}"
IMAGE_TAG sets the default tag for the gateway/supervisor image pair; the CLI
under test is always built from your branch. To validate against images from
your exact commit instead, build and push them and point
OPENSHELL_REGISTRY/IMAGE_TAG at them.
Test wrappers accept independent image overrides:
GATEWAY_IMAGE=registry.example.com/custom/gateway:test \
SUPERVISOR_IMAGE=registry.example.com/custom/supervisor:test \
SANDBOX_IMAGE=registry.example.com/custom/sandbox:test \
mise run e2e:kubernetes
GATEWAY_IMAGE applies to the Kubernetes gateway container. SUPERVISOR_IMAGE
applies to the trusted supervisor image selected by the Kubernetes, Docker, and
Podman wrappers. SANDBOX_IMAGE applies to the trusted workload-side runtime
image that stages the openshell-sandbox binary. A repository-only value
inherits IMAGE_TAG; a value with an explicit tag or @sha256: digest is used
as-is. When these variables are unset, the existing OPENSHELL_REGISTRY plus
IMAGE_TAG behavior is retained.
The Docker and Podman wrappers continue to give
OPENSHELL_DOCKER_SUPERVISOR_IMAGE and OPENSHELL_SUPERVISOR_IMAGE precedence
over SUPERVISOR_IMAGE.
Digest-pinned Kubernetes overrides require disabling local image builds, because Docker cannot tag a locally built image with a digest reference:
OPENSHELL_E2E_KUBE_BUILD_IMAGES=0 \
GATEWAY_IMAGE=registry.example.com/custom/gateway@sha256:<digest> \
SUPERVISOR_IMAGE=registry.example.com/custom/supervisor@sha256:<digest> \
mise run e2e:kubernetes
Available task variants:
| Task | Purpose |
|---|---|
e2e:kubernetes |
Default Rust e2e against Helm-deployed gateway |
e2e:kubernetes:db |
All database backend scenarios (SQLite + external PostgreSQL) |
e2e:kubernetes:sidecar |
Supervisor sidecar topology overlay |
e2e:kubernetes:credential-drivers |
Kubernetes Secrets and Vault credential storage |
e2e:kubernetes:workspace-managed |
Managed workspace mode (auto-created namespaces) |
e2e:kubernetes:workspace-operator |
Operator workspace mode (pre-provisioned namespaces) |
e2e:kubernetes:v1alpha1 |
Agent Sandbox v1alpha1 compatibility |
e2e:kubernetes:external-driver |
External Kubernetes driver sidecar |
Kubernetes e2e environment variables:
| Variable | Purpose |
|---|---|
OPENSHELL_E2E_KUBE_CONTEXT |
kubectl context for an existing cluster (skips k3d creation) |
OPENSHELL_E2E_KUBE_TEST |
Scope to a single test (e.g. smoke) |
OPENSHELL_E2E_KUBE_EXTRA_VALUES |
Colon-separated additional Helm values files |
OPENSHELL_E2E_KUBERNETES_FEATURES |
Cargo feature flags (default: e2e,e2e-host-gateway,e2e-kubernetes) |
IMAGE_TAG |
Gateway/supervisor image tag (default: latest for existing clusters) |
OPENSHELL_REGISTRY |
Image registry prefix (default: ghcr.io/nvidia/openshell) |
GATEWAY_IMAGE |
Kubernetes gateway image repository or complete tagged/digest-pinned image reference; digests require OPENSHELL_E2E_KUBE_BUILD_IMAGES=0 |
SUPERVISOR_IMAGE |
Gateway/supervisor image repository or complete tagged/digest-pinned image reference; Kubernetes digests require OPENSHELL_E2E_KUBE_BUILD_IMAGES=0 |
SANDBOX_IMAGE |
Trusted sandbox runtime image repository or complete tagged/digest-pinned image reference |
Run a single test directly with cargo:
cargo test --manifest-path e2e/rust/Cargo.toml --features e2e --test sync
Run a single Docker-only test directly with cargo:
cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test docker_preflight
The harness (e2e/rust/src/harness/) provides:
| Module | Purpose |
|---|---|
binary |
Builds and resolves the openshell binary from the workspace |
container |
Container-engine selection and support containers for proxy tests |
gateway |
Managed gateway restart controls for gateway-owned e2e runs |
sandbox |
SandboxGuard RAII type — creates sandboxes and deletes them on drop |
output |
ANSI stripping and field extraction from CLI output |
port |
wait_for_port() and find_free_port() for TCP testing |
Environment Variables
| Variable | Purpose |
|---|---|
OPENSHELL_GATEWAY |
Override active gateway name for E2E tests |
OPENSHELL_GATEWAY_ENDPOINT |
Run E2E tests against an existing plaintext HTTP gateway endpoint |
OPENSHELL_E2E_DRIVER |
Driver name exported by the e2e gateway wrapper (docker, podman, or vm) |
OPENSHELL_E2E_CREDENTIAL_DRIVERS |
Enables the Kubernetes credential-driver fixture path in e2e/with-kube-gateway.sh |
OPENSHELL_E2E_KUBE_CONTEXT |
kubectl context for Kubernetes e2e (skips ephemeral k3d) |
OPENSHELL_E2E_KUBE_TEST |
Scope Kubernetes e2e to a single test by name |