Describe how Docker GPU sandboxes derive filesystem access from the
selected CDI devices, which CDI mounts are rejected, the NVIDIA MPS pipe
directory exception, missing additionalGids handling on Docker releases
before 29.5, and SELinux requirements for the projected spec
directories. Add cluster debugging entries for CDI projection,
admission, probe and restart failures, and document the CDI context in
the core, sandbox and Docker driver READMEs.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
For GPU sandboxes, record the selected CDI device IDs and the spec
directories the Docker daemon currently reports in the protected
boundary configuration, and project those directories read-only under
/.openshell/cdi-specs so the sandbox runtime can resolve the same
specs. User mounts, image volumes and the workspace cannot overlap the
projection path.
Before creating the sandbox, probe each directory with a stopped
container from the prepared workload image and skip directories that
do not exist on the Docker host. Probes are bounded by a timeout,
labelled for cleanup with their sandbox, and remove their anonymous
volumes. A probe that cannot be removed no longer fails creation.
With resource admission enabled, containers with a CDI context must
keep a CDI device request matching the recorded selection and one
read-only, non-recursive, private bind per recorded directory.
Admission reads the bootstrap only for CDI containers, and boundary
files are replaced atomically so it never observes a partial write.
The Docker e2e gateway enables resource admission in GPU mode.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
When the boundary configuration carries a CDI context, resolve it at
agent start and use the result instead of the hard-coded GPU baseline:
- add derived read-only paths, and read-write access for device nodes
and IPC directories, unless the authored policy already covers them;
- reject an authored read-only entry for a device node or IPC directory
instead of overriding it;
- validate every CDI target through the descriptor that anchors its
Landlock rule at each launch and exec, rejecting broad-root aliases,
type changes and changed mount access;
- require Landlock enforcement for CDI targets even when the policy is
best effort.
Missing CDI additionalGids are reported as a warning, because some
runtimes do not apply them and devices that do not depend on them stay
usable. A required root group remains fatal. Enrichment, failures and
the effective Landlock mode are reported as OCSF events.
Sandboxes without a CDI context keep the existing GPU baseline.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Resolve the selected CDI devices against the spec directories projected
into the workload and derive the filesystem requirements their edits
imply:
- device nodes, which need read-write access;
- read-only mounts, expanded to the parent directory for shared
libraries;
- Unix sockets, kept as exact targets without filesystem grants;
- the NVIDIA MPS pipe directory, accepted as a writable IPC directory
only when mounted nosuid, nodev and noexec;
- additional GIDs, excluding root.
Effective mount access comes from the workload's mount table rather
than spec options. Ordinary writable mounts, broad roots and
conflicting uses of one path are rejected.
Captured Orin, Spark and multi-GPU specs drive the production resolver
against reviewed baselines. The resolver is wired into launch in the
next commit.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Define the shared CDI context schema: the selected CDI device IDs and
the workload-local paths at which a compute driver projects CDI spec
directories. Carry it as an optional field in the sandbox boundary
configuration, omitted from serialized output when absent so non-GPU
bootstraps stay readable by older sandbox runtimes. No driver sets it
yet.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(cli): pick forward test ports below the ephemeral range
The forward tests released an OS-assigned port before the CLI bound it,
so a concurrent test's bind to port 0 could claim it. Pick ports below
the Linux and macOS ephemeral ranges from a random offset instead.
Closes#4569
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(cli): retry forward tests when another process takes the port
Replace the fixed port range with a retry. The CLI checks a forward port
before binding it, so a concurrent test can claim it in between; on that
error, pick a new OS-assigned port and run the forward again.
Refs #4569
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Supervisor OCSF events reached only the sandbox's local JSONL file and
the shorthand log lines sent to the gateway, so collectors had no
structured copy of most of the security record. When the gateway has
`[openshell.gateway.otlp.ocsf]` configured, compute drivers now pass its
endpoint and schema version to supervisors, which export their own OCSF
events directly as OTLP log records. Supervisors already have egress,
so the gateway and its supervisor connection carry no extra load.
Supervisors use the same `openshell-otel` exporter as the gateway, with
a supervisor resource identity that includes `openshell.sandbox.id` and
`openshell.gateway.name`, so telemetry from sandboxes of different
gateways sharing a collector can be told apart. This applies to
supervisor spans too. Drivers pass the settings to supervisors as
`OPENSHELL_OTLP_OCSF_ENDPOINT`, `OPENSHELL_OTLP_OCSF_SCHEMA_VERSION`, and
`OPENSHELL_GATEWAY_NAME`; the gateway passes them to the VM driver as
arguments.
Collector headers that the gateway uses for OCSF export
(`OTEL_EXPORTER_OTLP_LOGS_HEADERS`, else `OTEL_EXPORTER_OTLP_HEADERS`)
travel in each sandbox's supervisor authentication bundle, which only
the supervisor reads, so authenticated collectors accept supervisor
records without exposing credentials to the workload.
The VM driver now sends SIGTERM and waits for the supervisor to exit
before closing its liveness pipe. Closing the pipe first made the
supervisor exit at once and drop queued records.
Closes#4544
Refs #2892
Signed-off-by: Kris Hicks <khicks@nvidia.com>
The gateway only persisted its OCSF events to the JSONL file, so collectors had
to tail that file. Adding `[openshell.gateway.otlp.ocsf]` now also exports each
gateway OCSF event as an OTLP log record, using the same resource identity as
gateway traces. `schema_version` optionally downgrades the exported JSON to
OCSF 1.1 or 1.3.
OTLP endpoints are now configured per signal, so OCSF events can go to a
collector that feeds a SIEM while spans go elsewhere. The existing
`[openshell.gateway.otlp]` `endpoint` is the default for every signal and on
its own still enables trace export. `[otlp.traces]` overrides the trace
endpoint, and `[otlp.ocsf]` takes its own `endpoint` or falls back to the
default.
Each record carries the OCSF JSON as its body, the event time as its timestamp,
a severity mapped from `severity_id`, and `ocsf.*` and `openshell.sandbox.id`
attributes for filtering without parsing the body.
Records flow through a queue bounded by count and bytes to the SDK OTLP log
exporter in batches bounded the same way. Failed exports retry with backoff;
records lost to a full queue, an oversized body, or exhausted retries are
counted in metrics and reported to the collector in an `openshell.ocsf.gap`
record on the next successful export. The exporter lives in `openshell-otel` so
other processes can share it.
Helm deployments set the `traces` and `ocsf` tables through `gatewayConfig`,
and the local development gateway tasks enable OCSF export.
Closes#4499
Refs #2892
Signed-off-by: Kris Hicks <khicks@nvidia.com>
rcgen 0.13.2 declares its optional x509-parser dependency with the verify
feature, which pulls in ring. The workspace enabled rcgen's x509-parser feature
for every crate, so the CLI and gateway linked ring alongside aws-lc even though
nothing in them verifies signatures with x509-parser.
Only SandboxCa::from_pem in openshell-supervisor-network needs
CertificateParams::from_ca_cert_pem. Enable the feature there alone. ring no
longer appears in the dependency trees of openshell, openshell-gateway or
openshell-prover; it remains in openshell-supervisor until rcgen is upgraded.
Release builds compile one package at a time, so the feature does not leak into
the other binaries.
Signed-off-by: politerealism <burdcat17@gmail.com>
The core, server, and sandbox-backend build scripts always used the vendored protoc binary and ignored PROTOC, so a distro build could not use its system compiler. When PROTOC is set, use it and let prost resolve the includes (PROTOC_INCLUDE still applies); otherwise behave exactly as before. Contributes to #3510.
Signed-off-by: politerealism <burdcat17@gmail.com>
In operator workspace mode the namespace label watcher fills the allowlist
in the background after the driver starts. Workspace checks that ran before
its first full listing rejected labeled namespaces with PermissionDenied, so
provider refreshes due at gateway startup were parked as configuration
errors for an hour.
Track whether the allowlist has received an authoritative listing. Operator
workspace checks now wait up to 10 seconds for it and return Unavailable if
it is still missing, which callers such as the provider refresh worker
retry shortly. An unreadable namespace file still yields a listed, empty
allowlist that fails closed.
Closes#4526
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
The seccomp notification probes install their listener on a launcher
thread and receive it over a channel. When the install fails, the
launcher drops the sender and exits, so the broker only sees a closed
channel and returns a generic "launcher disappeared" error. The thread
is never joined, so the kernel's errno is discarded.
That collapses several distinct failures into one opaque string: a
notification struct size mismatch, a failed PR_SET_NO_NEW_PRIVS, and
any non-EINVAL SECCOMP_SET_MODE_FILTER rejection all look identical to
a pre-5.19 kernel refusing WAIT_KILLABLE_RECV, which the install path
already retries without the flag.
Join the launcher on the recv() failure path and propagate its error
unchanged, so both the ErrorKind and raw_os_error survive to the
qualification report. Apply this to all three probes.
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* test(server): add a local PostgreSQL test runner
The openshell-server tests that need a real PostgreSQL server are
ignored by default and had no shared way to run. The mutation-replay
test reads its own OPENSHELL_REPLAY_TEST_DATABASE_URL, so every
contributor had to provision a database by hand, and the advisory-lock
tests that follow in this series need the same setup.
Add mise run test:rust:postgres. It runs every ignored postgres_* test
in openshell-server against OPENSHELL_TEST_POSTGRES_URL, or starts a
disposable PostgreSQL container with Docker or Podman (CONTAINER_ENGINE
selects one) on the image pinned by the Kubernetes e2e fixture and
removes it on exit. Tests run one at a time because advisory locks are
database-wide. The runner always points the legacy replay variable at
the selected database, so an inherited URL cannot send that test to a
different server. test:postgres-runner checks that selection with a fake
cargo and needs no database or container engine. CI does not run the
PostgreSQL tests; TESTING.md documents the task.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): hold the mutation guard while deleting a provider
DeleteProvider checked that no sandbox referenced the provider and then
deleted the record without holding the sandbox mutation guard. Sandbox
create and provider attach take that guard while they write provider
references, so one of them could add a reference after the attached
sandbox check passed, and the delete then removed a provider that a
sandbox spec still named.
Take the guard before the attached sandbox check, as provider create and
update already do, so the check and the delete run against a stable
set of sandbox references, and reject an empty name after authorization
but before the guard. A new test holds the guard, attaches the provider
while the delete waits, and asserts that the delete fails with
FailedPrecondition and leaves the provider in place.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): release staged credentials when a refresh fails early
A provider credential refresh stages the minted values under new
credential handles before it commits the provider update, and deletes
those handles when validation or persistence fails. Two earlier returns
skipped that cleanup. The minted expiry was converted to a protobuf
timestamp only after staging, so an expiry outside the timestamp range
failed the refresh and left the staged values in the credential driver.
A failure to acquire the sandbox mutation guard returned the same way.
Convert the expiry before staging anything, so a bad value fails before
any handle exists, and delete the staged handles when the guard cannot
be acquired. A new test refreshes a stored credential with an expiry of
i64::MAX and asserts that the call fails, the provider keeps its
original handles, and the credential driver holds the same number of
values as before. The guard failure path has no test here, because the
SQLite guard used by unit tests cannot fail.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): acquire mutation locks on a dedicated PostgreSQL lock pool
The cross-object mutation guard took its advisory-lock session from the
10-connection data pool and closed it on every release, so each guarded
mutation opened a new database connection, and lock holders competed
with their own critical sections for data connections. A lock wait that
hit the 10 second lock_timeout surfaced as INTERNAL.
Add a lazy, dedicated 4-connection lock pool and a new
persistence::mutation_lock module for the lock key, mode and set types.
Store::acquire_distributed_mutation_guard now takes a lock set and a
deadline, and each lock statement sets lock_timeout to the time left
before that deadline, so a cancelled acquisition never waits past it on
the server. Returned connections are scrubbed with
pg_advisory_unlock_all() and reused, and each return is bounded by 5
seconds so a stalled session cannot keep a pool permit.
A lock statement that PostgreSQL ends with SQLSTATE 55P03 fails with a
new LockTimeout error, and so does a wait for a lock connection during
which every one was checked out at some point, or that started with less
than a second left, too little to open one. The API returns it as
UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT and a 1 second retry
delay, because the mutation lock was not acquired, so the request's
guarded writes did not run. Only the guard acquisition classifies 55P03;
any other 55P03 stays a database error. A lock connection that
PostgreSQL does not open by the deadline despite at least a second to do
so (refused, out of connection slots, starting up, or not answering, all
of which SQLx retries silently) is a database error, not a lock timeout.
Mutations still take the legacy global key exclusively, so old and new
replicas keep excluding each other during a rolling upgrade. Each
PostgreSQL-backed gateway pod can now open up to 14 database
connections (10 data plus 4 lock) instead of 10, and the chart's
autoscaling.maxReplicas comment says what its default of 4 needs. The
SSH host identity lock stays on a data connection, because sandbox
creation takes it while holding its mutation guard and a second
lock-pool connection there could exhaust the pool. The provider
credential refresh lock stays there too, so a refresh still progresses
while every lock connection is in use.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): partition mutation locks by workspace and sandbox
Every cross-object mutation, and every lifecycle and supervisor update,
took one process-wide mutex and, on PostgreSQL, one fleet-wide advisory
lock. Unrelated sandboxes and providers therefore serialized across all
replicas, and one slow critical section delayed every other mutation in
the fleet.
Replace that lock with hierarchical intention locks on a global key, one
key per workspace and one key per sandbox, each held shared or
exclusive. Global settings and policy writers and platform-scope profile
writers hold the global key exclusively. Provider and workspace-scoped
profile writers, including DeleteProvider and credential refresh, hold
their workspace key exclusively. Sandbox writers and supervisor reports
hold their sandbox key exclusively under shared global and workspace
keys. Lifecycle, driver-watch and reconcile paths take only
process-local keys and keep relying on compare-and-swap across replicas.
Workspace and sandbox keys are domain-separated SHA-256 prefixes. Each
guard takes its local keys and then its advisory locks in ascending
order under one 10 second deadline, and no path nests guards. The
settings mutex is removed, since the global and sandbox keys cover it.
Sandbox settings are keyed by sandbox name, which a sandbox key no
longer covers once another sandbox reuses the name, so a settings
update returns NOT_FOUND unless its sandbox still holds the name after
the read, and ABORTED if the settings row was recreated before the
write.
The global key keeps its legacy value, so older replicas, which hold it
exclusively for every mutation, still exclude new ones during a rolling
upgrade. Guards now also give up after 10 seconds on SQLite, and
provider refresh reports a guard timeout as UNAVAILABLE instead of
INTERNAL. A supervisor report for a missing sandbox returns NOT_FOUND.
Startup endpoint-status reconciliation still holds the global key for
its whole scan.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): reconcile endpoint status on startup one sandbox at a time
Startup endpoint-status reconciliation held the global mutation key
exclusively for its whole scan. On PostgreSQL that blocked every guarded
mutation on every replica until the starting gateway had walked all
sandboxes, and the scan first waited behind every in-flight sandbox
mutation. It also paged with an offset, so sandboxes deleted mid-scan
shifted rows between pages, and a write that raced the scan (a Conflict
from a lifecycle update on another replica, or a sandbox deleted after
it was listed) failed gateway startup.
Walk the sandboxes with keyset paging instead, and reset each candidate
under its own sandbox-scoped guard, up to four at a time to match the
lock pool. A candidate with a fresh shared owner is skipped before any
guard is taken. Under the guard the reconciliation re-checks the owner,
re-reads the sandbox, skips it when it is gone or no longer carries
endpoint status, and writes against the version it just read. A Conflict
or a database error at write time, which is how a row deleted after the
re-read surfaces, is retried with a fresh read, at most five attempts
per sandbox. A lock timeout on the guard is retried the same way, so a
holder of the sandbox, its workspace or the global key that releases
within the retry window cannot stop a replica from starting. No
lock-timeout retry starts once the scan has run for three lock timeouts,
so a scan stuck behind a long holder fails within about 40 seconds,
inside the chart's 60-second startup probe, and logs the sandbox it gave
up on. Store::list_all_messages loses its only caller and is removed.
Reconciliation stays fail-closed: a holder that outlasts the retry
window, an owner lookup failure, or a sandbox that keeps changing still
fails startup, so stale endpoint success is never served. User-visible
change: a restarting gateway no longer stalls mutations of unrelated
sandboxes and providers on its peers while it reconciles.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): skip the endpoint-status lock when a session was replaced
When a supervisor session ends, the gateway resets that sandbox's tool
server endpoint evidence unless a replacement session already owns
endpoint observation. The replacement check ran only after the reset
had read the sandbox and taken its sandbox mutation guard, so a reset
whose supervisor had already reconnected still queued behind other
mutations of that sandbox, and on PostgreSQL it used a lock-pool
connection, only to find it had nothing to do.
Check for a replacement first: a live local session, or a fresh owner
record on a peer. When one exists, return without reading the sandbox
or taking the guard. Otherwise take the guard and repeat the same check
under it, because a replacement's pre-acknowledgement reset runs under
the same sandbox key and resetting after it would wipe the
replacement's fresh evidence. Both checks share one helper, so they
cannot drift apart. When no replacement exists, the reset costs one
extra owner-row read.
This helps most when a replica notices a dead stream late, after the
supervisor has already reconnected elsewhere (a network partition or a
keepalive expiry).
Deferring to a replacement leaves the old session's evidence to the
replacement's pre-acknowledgement reset, which can time out on its
bounded mutation lock wait. That reset's failure path removed the
replacement and released its owner record but scheduled no
invalidation, so the old session's success evidence could outlive both
sessions. An old session that ended after its replacement registered
already deferred this way, and the unguarded check now makes every
disconnect that finds the replacement defer too. When the reset fails
while the replacement is still current, also schedule the retried
invalidation that follows a session end. That failure path keeps the
lifecycle demotion from abandon_session_setup, but runs it in the
spawned task before the retry and only while this session is still
current, so a timed-out reset returns promptly.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* test(server): cover staged credential cleanup on a lock timeout
A provider credential refresh deletes its staged handles when it cannot
acquire the provider mutation guard, but that path had no test: the
SQLite guard used by unit tests could not fail. Mutation locks now
acquire under a deadline that tests can shorten, so the timeout path
can be exercised directly.
The new test shortens the lock deadline to 50 ms, holds the workspace
mutation guard, and refreshes a provider that already stores an AWS
access key. The refresh returns UNAVAILABLE with the
MUTATION_LOCK_TIMEOUT reason, the provider keeps its original handle,
which still resolves to the old key, and the credential driver holds
the same number of values as before the refresh.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): export mutation lock wait and timeout metrics
A handler mutation guard can now wait up to 10 seconds for its keys and
then fail with UNAVAILABLE, but nothing showed operators how long guards
waited or how often they gave up, short of reading warning logs on each
replica.
Export openshell_server_mutation_lock_wait_seconds, a histogram of the
time one guard took to acquire all of its keys (local registry plus
PostgreSQL advisory locks), recorded on success, and
openshell_server_mutation_lock_timeouts_total, a counter of guard
acquisitions that timed out. Both carry a bounded scope label: global,
workspace or sandbox. The platform scope, whose workspace is empty,
counts as global. The histogram uses the same 1 ms to 15 s buckets as
the relay and peer histograms, and the timeout counter starts at 0 for
every scope, so rate() works on an idle replica. The timeout warning
now also logs the scope. A guard that fails for another reason, such as
a lock connection PostgreSQL did not open, is not counted as a timeout
and logs a "mutation lock acquisition failed" warning instead.
Lifecycle, driver-watch and reconcile paths take only process-local
locks and never time out, so they are not recorded.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* test(server): add PostgreSQL mutation lock integration tests
The scoped mutation locks were only unit-tested on SQLite, where the
database layer is a no-op. Add ignored postgres_* tests that run the
advisory-lock layer against a real PostgreSQL through mise run
test:rust:postgres, each in its own disposable schema (TestSchema).
persistence/mutation_lock_pg_tests.rs uses two stores on one database
as two replicas. It covers disjoint sandbox scopes holding at once, a
workspace key blocking only sandboxes in that workspace, the global key
blocking every scope, and exclusion in both directions against a raw
holder of the legacy global key, whose 55P03 stays a database error
outside a guard acquisition. It checks that timed-out and cancelled
acquisitions release the keys they already took, that a released lock
connection returns to the pool without residual locks, that a release
stalled on the network closes its session and the pool recovers, and
that the lock pool never grows past its size. It also checks that a
full lock pool times out as lock contention, while a lock connection
that PostgreSQL never answers fails as a database error.
compute/mutation_guard.rs runs the random scope mix across two
PostgreSQL runtimes and checks that a by-id guard waits for a workspace
writer on the other replica. The provider refresh lock test moves onto
the runner too, and checks that a refresh proceeds while every lock
connection is held.
A bench_postgres_* capacity check measures a paced burst of 500
reconnects 12 ms apart into one replica and fails when the p99 lock
wait reaches a fifth of the lock timeout. Machine load moves those
waits, so mise run test:rust:postgres skips it and mise run
test:rust:postgres:bench runs it alone, through a new
OPENSHELL_TEST_POSTGRES_PREFIX setting in the runner.
The tests add test-only helpers: the lock pool's size and idle count
and the backend pid of a held lock session. Nothing changes outside
tests and the test runner.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* ci: run the PostgreSQL-backed server tests
Branch Checks now runs the postgres_* tests through the same runner
developers use, on an x86_64 Linux runner with Docker. They cover lock
scope exclusion, interop with replicas still on the old global key,
cancellation, lock-pool bounds and connection failures, which no other
CI lane checks directly. The load-sensitive bench_postgres_* capacity
checks stay local.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* docs: document scoped mutation locks and lock-pool sizing
In the High Availability guide, size PostgreSQL for 14 connections per
gateway pod, 10 for data and 4 for the new lock pool, so three or more
replicas now exceed what the PostgreSQL default leaves, and keep
headroom for lock sessions that cancelled requests leave behind.
Explain which changes lock a sandbox, a workspace, or the whole fleet,
and that a lock wait over 10 seconds returns a retryable UNAVAILABLE
with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to
raise max_connections to the 14-per-pod size before upgrading, because
the rollout already runs new pods at that count, and warns that older
replicas keep serializing every mutation until the rollout finishes.
Monitor Capacity gains the lock metrics and their queries, and names
slow driver, credential, middleware, or profile source calls as another
cause of lock waits.
List the lock wait histogram and timeout counter, with their scope
label, on the Gateway Metrics page. The counter also counts background
work and leaves out lock connections that PostgreSQL does not open
although at least a second of the wait remained.
In the cluster debugging skill, replace the single advisory-lock
paragraph with the lock levels and pool, what each timeout detail
means, how a lock connection that PostgreSQL does not open shows up,
when terminating a lock backend is safe, a pg_locks query that shows
holders and waiters, and how to spot the global key. The per-pod greps
also read the timeout counter, and new rows cover lock timeouts and
lock connections that PostgreSQL does not open.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): check mutation lock ordering in debug builds
The mutation lock ordering rules were prose only, so a nested guard on
a different key passed every test and could only deadlock under
contention. Debug builds now register each held guard with the Tokio
task (or the thread, outside a task) that acquired it, and panic before
waiting when a local key, a blocking lifecycle gate or the SSH identity
lock would break rules 1, 4 or 5. Startup endpoint-status
reconciliation and the tests that hold a guard while driving guarded
code run under lock_order::branch(), which marks deliberate concurrency
within one task. Release builds compile the check out.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): do not hold gateway shutdown for the endpoint-status retry
When a replacement supervisor session failed its endpoint-status reset,
the spawned cleanup task kept the session lifetime guard through
retry_endpoint_status_after_supervisor_disconnect. That retry only ends
once the reset is durable, so a gateway shutdown during a database or
lock outage waited the full 10 s and reported ownership cleanup as
incomplete. The guard now covers only the lifecycle demotion, as in
finish_supervisor_session, and a SQLite test checks that shutdown
completes while the retry is still blocked.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* refactor(server): make the platform profile lock scope explicit
Platform-scope provider profile writers (import, update, delete) pass an
empty workspace on purpose and need X(global), because they validate
sandboxes in every workspace. MutationScope::profiles() now names that
escalation and picks Global for the platform scope, instead of relying
on MutationScope::Workspace("") to mean it. An empty workspace name now
trips a debug assertion, and release builds keep X(global) as a
fail-safe that over-locks rather than under-locks.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): amortize local mutation lock table sweeps
LocalMutationLocks swept its whole table on every key lookup, so a
burst of N concurrent acquisitions on one replica cost O(N^2) under a
blocking mutex. The table is now swept only when a new key grows it
past the larger of 64 and twice the live entries left by the last
sweep, which makes acquisition amortized O(1). Released entries,
including those of cancelled waiters, stay bounded at about twice the
peak live set. The lifecycle gate registry keeps its per-call sweep.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): make the database pool ceiling configurable
Both stores hardcoded their pool ceiling: 10 connections for Postgres, 5
for on-disk SQLite. One pool is shared by every database-backed RPC, so
that number is also the gateway's ceiling on concurrent database work.
Once every connection is checked out, callers queue on acquire and sqlx
logs "time to acquire exceeded slow threshold"; sandbox creates then time
out and are retried, which adds load rather than shedding it.
The right ceiling is deployment-specific — it depends on how many gateway
replicas share the database and what max_connections the server itself
allows — so it cannot be a single number baked into the binary.
Expose it on the same config surface as the rest of the gateway's
settings: --db-max-connections, OPENSHELL_DB_MAX_CONNECTIONS, and the
TOML key database_max_connections, resolved in that precedence order.
Helm deployments set the key through gatewayConfig. Omitting it keeps
each backend's previous value, so existing deployments do not move.
Values below 1 are rejected rather than silently replaced by the default:
a zero pool would block every acquire, and quietly ignoring a typo would
reproduce the ceiling the operator is trying to lift. An in-memory SQLite
database stays pinned to one connection, since the database lives inside
that connection.
Refs: #2561
Signed-off-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com>
(cherry picked from commit 33517b32bd)
[emacchi: dropped the server.dbMaxConnections Helm value; operators set
the key through gatewayConfig]
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): make the mutation lock pool size configurable
The mutation lock pool was fixed at 4 connections per replica. That
number rests on one reconnect-burst bench rather than scale testing, so
a gateway that serves many more sandboxes needs a way to raise it
without a rebuild. Add --db-lock-max-connections,
OPENSHELL_DB_LOCK_MAX_CONNECTIONS and the database_lock_max_connections
gateway.toml key next to the data pool knob, resolved in the same order;
Helm deployments set it through gatewayConfig. The default stays 4,
zero is rejected, SQLite ignores it with a warning, and the gateway logs
both pool sizes at startup. PostgreSQL now needs at least 2 data
connections, because the SSH identity lock keeps one while it queries
the pool. The connection sizing docs, the HA guide formula and the
debug skill now count data plus lock connections.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): export mutation lock errors and lock connections in use
A mutation lock acquisition that failed without timing out, such as a
lock connection PostgreSQL never opened or a failed lock statement,
only logged a warning, and readiness stays healthy because it pings the
data pool. Count those failures in
openshell_server_mutation_lock_errors_total{scope}, kept apart from
timeouts because they point at PostgreSQL capacity or connectivity
rather than contention. Export
openshell_server_mutation_lock_connections_in_use as a gauge slot held
by each checked-out lock connection, next to
openshell_server_mutation_lock_connections_capacity, which reads the
configured lock pool size; both exist only with PostgreSQL. The metrics
catalog, the HA guide's signals and PromQL examples, and the debug
skill cover the new series.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): time mutation lock holds and test a stalled holder
Nothing bounds how long a mutation guard is held, so a holder stuck in
a compute driver, credential backend or middleware call keeps its scope
locked while waiters time out. Each guard now records its hold in
openshell_server_mutation_lock_hold_seconds{scope} when it is released,
and logs a warning when the hold outlasted the lock wait timeout, so
operators can find the replica and scope behind those timeouts. A
PostgreSQL test with a stalled holder pins the current contract:
waiters on its sandbox, workspace and global scopes fail by their
deadline, other scopes proceed, the holder keeps its key, and a full
lock pool on one replica leaves the other working. The metrics catalog,
the HA guide and the debug skill describe the new signals, which
observe holds without bounding them.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
---------
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
Co-authored-by: divesh <dgude@nvidia.com>
Co-authored-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com>
* feat(kubernetes): add operator allowlist for sandbox pod annotations
Add allowed_pod_annotations to the Kubernetes driver config. Sandbox
template annotations must match an exact key or a <domain>/* entry, and
the default empty list rejects all of them. Keys in the openshell.ai and
agents.x-k8s.io domains are always rejected. Expose the setting through
the server.allowedPodAnnotations Helm value.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(kubernetes): cover pod annotation rejection on sandbox create
Assert the exact rejected annotation keys, exercise create_sandbox_inner
with a disallowed annotation, and point the rejection error at the Helm
gatewayConfig path.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(kubernetes): drop Helm path from pod annotation rejection error
The rejection error reaches API callers, not operators, so name only the
gateway config key. Spell the Helm path in the runtimes docs with the
quoted table key to avoid an invalid nested gatewayConfig rendering.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(kubernetes): condense pod annotation allowlist tests
Fold the allowlist matching and validation cases into table-driven tests,
reuse the json_struct helper for templates, and document why the
openshell.ai and agents.x-k8s.io domains are reserved.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* feat(kubernetes): add operator allowlist for sandbox pod labels
Sandbox template labels were copied onto workload pods unfiltered. Labels
select pods for NetworkPolicy rules and Services, so a caller could place a
sandbox under rules meant for other workloads or set OpenShell-owned labels.
Add allowed_pod_labels, sharing the allowed_pod_annotations entry format,
validation, and reserved openshell.ai / agents.x-k8s.io domains. Empty by
default, which rejects every template label.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(cli): backpressure sandbox uploads
Stream archive uploads over a bounded SSH subsystem whose credit follows extraction stdin progress. Preserve the existing input budget and tar destination semantics, validate completion, and clean up cancelled transfers. Add protocol, lifecycle, and large-upload regression coverage without changing russh.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(cli): preserve upload cancellation and errors
Keep cancellation active through upload setup and retry delays, and
terminate cancelled SSH probes. Preserve protocol failures through output
completion and report captured diagnostics on early extraction failure.
Remove redundant upload validation and extend the affected regressions.
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(cli): carry upload credit over SSH streams
Replace the framed upload protocol with SSH's own streams. The client
sends one JSON plan line, the raw archive and EOF. The supervisor's
stdout carries only cumulative totals of bytes written to extraction
stdin, extraction output goes to stderr, and the exit status reports
the result. Extraction starts inline through the ordinary exec path.
This removes the frame codec, SHA-256 digests, the CLI sha2 dependency
and the supervisor upload state machine while keeping the 1 MiB credit
window, the 4 MiB input guard, cancellation and error reporting. A
failed archive now stops SSH before closing stdin, so EOF never marks a
truncated archive complete.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(cli): isolate upload SSH connections and pin credit cadence
- Apply the forward multiplexing overrides to the upload ssh command, so
a user ControlMaster cannot carry an upload over another session's
master connection.
- Kill ssh before closing its stdin when a transfer fails locally, and
include ssh stderr in transport errors.
- Share ACK_INTERVAL_BYTES from core. Assert at compile time that CLI
chunks fit the credit left after an ack and that the window stays
below the supervisor's pending-input guard.
- Log the real reason when the supervisor rejects an upload.
- Test that a single-packet upload reports its final total.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(cli): assert upload ssh opens the subsystem without a TTY
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(prover): remove legacy unused reachability constraints
Remove can_write_to_endpoint() and the solver variables and encoding
methods that exist solely to support it. No active query calls this
method; confirmed zero callers across the monorepo. Crate is not
published externally, so no public API concern.
Removed as root cause:
- can_write_to_endpoint(): zero callers in the codebase
Removed as direct consumers of can_write_to_endpoint():
- binary_can_write: only read inside can_write_to_endpoint()
- credential_has_write: only read inside can_write_to_endpoint()
Removed as directly dead (no query ever read them):
- credential_has_destructive: flagged by compiler, silenced with
#[allow(dead_code)] rather than removed
- filesystem_readable: flagged by compiler, silenced with
#[allow(dead_code)] rather than removed
Removed as now-empty encoding methods:
- encode_credentials(): exclusively populated credential_has_write
and credential_has_destructive
- encode_filesystem(): exclusively populated filesystem_readable
Also removed two #[allow(dead_code)] allowances and the corresponding
HashMap::new() initialisers and build() call sites.
Active query behavior is unchanged. All existing prover tests pass
without modification and cover all four finding categories end-to-end.
No new tests added as they would duplicate existing coverage exactly.
Note: removing encode_credentials() leaves write_actions_for_scopes()
and destructive_actions_for_scopes() in credentials.rs with no callers.
Left as a follow-up outside the scope of this change.
Closes#2412
Signed-off-by: Evie Howard <evhoward@redhat.com>
* refactor(prover): remove dead write-capability helpers from binary registry
BinaryProtocol::can_write(), BinaryCapability::can_write(), and
write_mechanisms() lost their only callers when can_write_to_endpoint()
and its supporting model fields were removed. Delete the now-dead methods.
write_mechanisms() had no callers prior to this series; the two can_write()
methods were made dead by the preceding model.rs cleanup.
Signed-off-by: Evie Howard <evhoward@redhat.com>
* docs(prover): remove stale write-bypass mention from module doc comment
The module doc incorrectly listed write-bypass violation detection as a
prover capability. The prover only checks reachability queries; write-bypass
detection was never implemented.
Signed-off-by: Evie Howard <evhoward@redhat.com>
---------
Signed-off-by: Evie Howard <evhoward@redhat.com>
* fix(kubernetes): restart kube read timeout when each request is written (fixes#4451)
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(kubernetes): use a per-request tower timeout instead of a custom kube client
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(oci): share OCI image checks across Docker and Podman
Add an oci-image feature testsuite that runs against installed artifacts.
It covers custom WORKDIR placement, image content and ownership, workspace
writes from the main process and exec, file transfer including
directory-merge upload, OCI user identity, the managed /sandbox fallback,
and rejection of an unwritable WORKDIR.
CI runs the suite on the Docker rootful, Podman rootful, and Podman
rootless tmachine guests.
Replace the Docker-only custom_image e2e with the shared suite.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(oci): require WorkspaceValidationFailed and cover image USER fallback
The unwritable WORKDIR scenario now requires the WorkspaceValidationFailed
reason, so unrelated provisioning failures cannot pass it. The numeric
USER scenario supplies a policy without a process section, so the image
USER is the only source of the sandbox identity.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(podman): leave shared OCI identity checks to the oci-image suite
The oci-image feature suite covers the sandbox identity seen by the main
process and sandbox exec on Docker and Podman. The Podman e2e keeps the
checks only it can make: the pinned image ID and the isolated
workload/supervisor container pair.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(oci): accept existing workspace rejection diagnostics
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(oci): fix cleanup ordering and restore coverage parity
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(oci): remove migrated legacy workspace helpers and checks
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(drivers): report rejected OCI working directories as WorkspaceValidationFailed
The supervisor exits with the reserved status when the sandbox rejects the
image working directory. Docker maps that exit during readiness and
monitoring and records it before removing the supervisor container, and
Podman maps the supervisor companion's exit instead of the workload's.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(oci): require WorkspaceValidationFailed for unwritable WORKDIR
Unrelated provisioning failures can no longer pass the rejection scenario.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(oci): keep exit 78 reserved and report fixed workspace diagnostics
The supervisor no longer passes on a workload's own exit status 78, so
drivers cannot misreport it as a workspace rejection after a runtime
restart. The rejection is recognized once at agent start and carried as a
typed error. Docker reports the same fixed condition message as Podman and
logs the supervisor output instead, and its monitor removes the supervisor
container when a stop overlaps diagnostics collection. The OCI image suite
registers images for cleanup only after they build.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(oci): simplify workspace failure handling and OCI image tests
Report a recorded Docker workspace rejection once, where supervisor
readiness fails, and match its fixed message exactly. Read the Podman
supervisor exit status once per inspection. Choose the supervisor exit
status in one function and derive the workspace error type.
The OCI image suite shares step helpers, checks that a custom workspace
does not also create /sandbox, and its playbook waits for a connected
gateway and collects failure diagnostics through the shared role.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(oci): replace podman_oci_identity with shared and driver suite checks
The oci-image feature suite now checks, on Docker and Podman, that the
image user has no capabilities, has no_new_privs set, and cannot read
the sandbox control credentials, and that a custom WORKDIR is not a
mount. The checks that need container inspection (pinned image ID,
network=none, supervisor user and capabilities) move to a small test in
the Podman driver suite. Remove the legacy podman_oci_identity e2e test.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(oci): keep podman_oci_identity for Podman-only isolation checks
Restore the Podman e2e test removed in the previous commit and drop
only the checks the shared oci-image suite now runs on Docker and
Podman: workload capabilities, no_new_privs, credential readability,
and the workspace and /sandbox mount checks. The pinned image ID and
the workload/supervisor isolation checks stay, as does the Podman
driver suite unchanged.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(oci): simplify workspace failure reporting and OCI image suite
Carry the Docker control supervisor's failure reason in a typed error
instead of matching status text, build the fixed workspace message where
the wait result is classified, and drop the fake Docker server test.
Inline image cleanup ordering in the suite, read the container engine
once, share sandbox naming, and stop granting a literal /sandbox in the
image identity policy. Add e2e:oci-image mise tasks and use them from the
ARM64 Docker workflow and TESTING.md.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(oci): move shared container checks into the OCI image suite
Inspect the containers the driver created through the gateway's engine in
the shared suite: the workload runs the inspected image ID as the OCI
identity without a network or supervisor credential mounts, and the
supervisor is a separate non-root container. Keep only the Podman-specific
supervisor identity and capability checks in podman_oci_identity.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(oci): check the workspace rejection reason from sandbox JSON
Read the WorkspaceValidationFailed condition reason from sandbox get
--output json instead of matching CLI output, and restore the Docker
driver's original workspace failure helpers.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(oci): capture fixture image identity before sandbox creation
Signed-off-by: Evan Lezar <elezar@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
* fix(driver-mxc): pass gateway name to mocked backend
Refs #4191
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(windows): use native executable paths in network test fixtures
Signed-off-by: Philippe Martin <phmartin@redhat.com>
---------
Signed-off-by: Philippe Martin <phmartin@redhat.com>
The stop path wrote Stopping and Stopped with a single compare-and-swap.
In HA deployments every replica applies compute driver watch events for
the terminating runtime without the per-replica lifecycle gate, so a
watch write landing between the re-read and the CAS failed the stop RPC
with "update sandbox lifecycle failed due to concurrent modification"
and left the sandbox in Stopping.
Retry both writes on PersistenceError::Conflict up to
STOP_PHASE_CAS_RETRY_LIMIT, re-reading the sandbox on each attempt. The
Stopping write re-runs the stop preconditions and the name check; the
Stopped write proceeds only while the record is still Stopping, returns
an already Stopped record as-is, and aborts on any other phase. Gateway
startup recovery of retained Stopping sandboxes now shares the same
helper instead of writing against its pre-driver snapshot.
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* feat(providers): support stable credential placeholders
Add credentials[].stable_placeholder so a running client keeps using an externally rotated static credential through the reference it already holds. Squashed replay of the earlier four commits onto upstream main 71c3cd957; re-derives the frozen schema fingerprints and adapts the provider environment revision test to the added capability argument.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(e2e): apply stable-placeholder supervisor image override
Use ManagedGateway's per-process supervisor image override so the fixture
CA reaches the supervisor when runtime image environment variables take
precedence over gateway TOML. Restore the wrapper's original image after
each test, following the provider-readiness fixture.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(server): update public RPC schema fingerprint after merging main
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
Evan Lezar