mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-22 06:19:47 +08:00
windows
97
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew Newberry
|
df958f3613 |
fix(mxc): pass native Windows build and HTTPS qualification
Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
d08dce0e97 |
refactor(mxc): adopt main isolation backend contracts
Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
6e69360d0e |
refactor(runtime): extract portable driver and isolation foundations (#4106)
* refactor(runtime): decouple supervisor access and boundary audit validation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): keep supervisor process access boundary driven Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(cli): support environment values from caller variables Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): extract driver-independent MXC foundations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): defer UI policy support to MXC integration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): integrate current policy and SSH launch contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(runtime): reconcile fixtures with updated main Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): repair stacked branch CI contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(runtime): update boundary fixtures for current contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(runtime): clarify scoped guards for launch inputs and delete Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): validate foreground attachment intent before launch Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Shiju
|
b4baf23db3 |
fix(compute): allow stopping errored sandboxes for recovery (#4231)
* fix(compute): allow stopping errored sandboxes for recovery Signed-off-by: Shiju <shiju@nvidia.com> * test(compute): focus sandbox error recovery coverage Remove duplicate cancellation and idempotency checks covered by existing lifecycle tests. Keep recovery guidance focused on the stop/start commands and timeout exception. Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
|
e30c938bb1 |
feat(server): scope mutation locks and bound lock waits (#4147)
* test(server): add a local PostgreSQL test runner
The openshell-server tests that need a real PostgreSQL server are
ignored by default and had no shared way to run. The mutation-replay
test reads its own OPENSHELL_REPLAY_TEST_DATABASE_URL, so every
contributor had to provision a database by hand, and the advisory-lock
tests that follow in this series need the same setup.
Add mise run test:rust:postgres. It runs every ignored postgres_* test
in openshell-server against OPENSHELL_TEST_POSTGRES_URL, or starts a
disposable PostgreSQL container with Docker or Podman (CONTAINER_ENGINE
selects one) on the image pinned by the Kubernetes e2e fixture and
removes it on exit. Tests run one at a time because advisory locks are
database-wide. The runner always points the legacy replay variable at
the selected database, so an inherited URL cannot send that test to a
different server. test:postgres-runner checks that selection with a fake
cargo and needs no database or container engine. CI does not run the
PostgreSQL tests; TESTING.md documents the task.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): hold the mutation guard while deleting a provider
DeleteProvider checked that no sandbox referenced the provider and then
deleted the record without holding the sandbox mutation guard. Sandbox
create and provider attach take that guard while they write provider
references, so one of them could add a reference after the attached
sandbox check passed, and the delete then removed a provider that a
sandbox spec still named.
Take the guard before the attached sandbox check, as provider create and
update already do, so the check and the delete run against a stable
set of sandbox references, and reject an empty name after authorization
but before the guard. A new test holds the guard, attaches the provider
while the delete waits, and asserts that the delete fails with
FailedPrecondition and leaves the provider in place.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): release staged credentials when a refresh fails early
A provider credential refresh stages the minted values under new
credential handles before it commits the provider update, and deletes
those handles when validation or persistence fails. Two earlier returns
skipped that cleanup. The minted expiry was converted to a protobuf
timestamp only after staging, so an expiry outside the timestamp range
failed the refresh and left the staged values in the credential driver.
A failure to acquire the sandbox mutation guard returned the same way.
Convert the expiry before staging anything, so a bad value fails before
any handle exists, and delete the staged handles when the guard cannot
be acquired. A new test refreshes a stored credential with an expiry of
i64::MAX and asserts that the call fails, the provider keeps its
original handles, and the credential driver holds the same number of
values as before. The guard failure path has no test here, because the
SQLite guard used by unit tests cannot fail.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): acquire mutation locks on a dedicated PostgreSQL lock pool
The cross-object mutation guard took its advisory-lock session from the
10-connection data pool and closed it on every release, so each guarded
mutation opened a new database connection, and lock holders competed
with their own critical sections for data connections. A lock wait that
hit the 10 second lock_timeout surfaced as INTERNAL.
Add a lazy, dedicated 4-connection lock pool and a new
persistence::mutation_lock module for the lock key, mode and set types.
Store::acquire_distributed_mutation_guard now takes a lock set and a
deadline, and each lock statement sets lock_timeout to the time left
before that deadline, so a cancelled acquisition never waits past it on
the server. Returned connections are scrubbed with
pg_advisory_unlock_all() and reused, and each return is bounded by 5
seconds so a stalled session cannot keep a pool permit.
A lock statement that PostgreSQL ends with SQLSTATE 55P03 fails with a
new LockTimeout error, and so does a wait for a lock connection during
which every one was checked out at some point, or that started with less
than a second left, too little to open one. The API returns it as
UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT and a 1 second retry
delay, because the mutation lock was not acquired, so the request's
guarded writes did not run. Only the guard acquisition classifies 55P03;
any other 55P03 stays a database error. A lock connection that
PostgreSQL does not open by the deadline despite at least a second to do
so (refused, out of connection slots, starting up, or not answering, all
of which SQLx retries silently) is a database error, not a lock timeout.
Mutations still take the legacy global key exclusively, so old and new
replicas keep excluding each other during a rolling upgrade. Each
PostgreSQL-backed gateway pod can now open up to 14 database
connections (10 data plus 4 lock) instead of 10, and the chart's
autoscaling.maxReplicas comment says what its default of 4 needs. The
SSH host identity lock stays on a data connection, because sandbox
creation takes it while holding its mutation guard and a second
lock-pool connection there could exhaust the pool. The provider
credential refresh lock stays there too, so a refresh still progresses
while every lock connection is in use.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): partition mutation locks by workspace and sandbox
Every cross-object mutation, and every lifecycle and supervisor update,
took one process-wide mutex and, on PostgreSQL, one fleet-wide advisory
lock. Unrelated sandboxes and providers therefore serialized across all
replicas, and one slow critical section delayed every other mutation in
the fleet.
Replace that lock with hierarchical intention locks on a global key, one
key per workspace and one key per sandbox, each held shared or
exclusive. Global settings and policy writers and platform-scope profile
writers hold the global key exclusively. Provider and workspace-scoped
profile writers, including DeleteProvider and credential refresh, hold
their workspace key exclusively. Sandbox writers and supervisor reports
hold their sandbox key exclusively under shared global and workspace
keys. Lifecycle, driver-watch and reconcile paths take only
process-local keys and keep relying on compare-and-swap across replicas.
Workspace and sandbox keys are domain-separated SHA-256 prefixes. Each
guard takes its local keys and then its advisory locks in ascending
order under one 10 second deadline, and no path nests guards. The
settings mutex is removed, since the global and sandbox keys cover it.
Sandbox settings are keyed by sandbox name, which a sandbox key no
longer covers once another sandbox reuses the name, so a settings
update returns NOT_FOUND unless its sandbox still holds the name after
the read, and ABORTED if the settings row was recreated before the
write.
The global key keeps its legacy value, so older replicas, which hold it
exclusively for every mutation, still exclude new ones during a rolling
upgrade. Guards now also give up after 10 seconds on SQLite, and
provider refresh reports a guard timeout as UNAVAILABLE instead of
INTERNAL. A supervisor report for a missing sandbox returns NOT_FOUND.
Startup endpoint-status reconciliation still holds the global key for
its whole scan.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): reconcile endpoint status on startup one sandbox at a time
Startup endpoint-status reconciliation held the global mutation key
exclusively for its whole scan. On PostgreSQL that blocked every guarded
mutation on every replica until the starting gateway had walked all
sandboxes, and the scan first waited behind every in-flight sandbox
mutation. It also paged with an offset, so sandboxes deleted mid-scan
shifted rows between pages, and a write that raced the scan (a Conflict
from a lifecycle update on another replica, or a sandbox deleted after
it was listed) failed gateway startup.
Walk the sandboxes with keyset paging instead, and reset each candidate
under its own sandbox-scoped guard, up to four at a time to match the
lock pool. A candidate with a fresh shared owner is skipped before any
guard is taken. Under the guard the reconciliation re-checks the owner,
re-reads the sandbox, skips it when it is gone or no longer carries
endpoint status, and writes against the version it just read. A Conflict
or a database error at write time, which is how a row deleted after the
re-read surfaces, is retried with a fresh read, at most five attempts
per sandbox. A lock timeout on the guard is retried the same way, so a
holder of the sandbox, its workspace or the global key that releases
within the retry window cannot stop a replica from starting. No
lock-timeout retry starts once the scan has run for three lock timeouts,
so a scan stuck behind a long holder fails within about 40 seconds,
inside the chart's 60-second startup probe, and logs the sandbox it gave
up on. Store::list_all_messages loses its only caller and is removed.
Reconciliation stays fail-closed: a holder that outlasts the retry
window, an owner lookup failure, or a sandbox that keeps changing still
fails startup, so stale endpoint success is never served. User-visible
change: a restarting gateway no longer stalls mutations of unrelated
sandboxes and providers on its peers while it reconciles.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): skip the endpoint-status lock when a session was replaced
When a supervisor session ends, the gateway resets that sandbox's tool
server endpoint evidence unless a replacement session already owns
endpoint observation. The replacement check ran only after the reset
had read the sandbox and taken its sandbox mutation guard, so a reset
whose supervisor had already reconnected still queued behind other
mutations of that sandbox, and on PostgreSQL it used a lock-pool
connection, only to find it had nothing to do.
Check for a replacement first: a live local session, or a fresh owner
record on a peer. When one exists, return without reading the sandbox
or taking the guard. Otherwise take the guard and repeat the same check
under it, because a replacement's pre-acknowledgement reset runs under
the same sandbox key and resetting after it would wipe the
replacement's fresh evidence. Both checks share one helper, so they
cannot drift apart. When no replacement exists, the reset costs one
extra owner-row read.
This helps most when a replica notices a dead stream late, after the
supervisor has already reconnected elsewhere (a network partition or a
keepalive expiry).
Deferring to a replacement leaves the old session's evidence to the
replacement's pre-acknowledgement reset, which can time out on its
bounded mutation lock wait. That reset's failure path removed the
replacement and released its owner record but scheduled no
invalidation, so the old session's success evidence could outlive both
sessions. An old session that ended after its replacement registered
already deferred this way, and the unguarded check now makes every
disconnect that finds the replacement defer too. When the reset fails
while the replacement is still current, also schedule the retried
invalidation that follows a session end. That failure path keeps the
lifecycle demotion from abandon_session_setup, but runs it in the
spawned task before the retry and only while this session is still
current, so a timed-out reset returns promptly.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* test(server): cover staged credential cleanup on a lock timeout
A provider credential refresh deletes its staged handles when it cannot
acquire the provider mutation guard, but that path had no test: the
SQLite guard used by unit tests could not fail. Mutation locks now
acquire under a deadline that tests can shorten, so the timeout path
can be exercised directly.
The new test shortens the lock deadline to 50 ms, holds the workspace
mutation guard, and refreshes a provider that already stores an AWS
access key. The refresh returns UNAVAILABLE with the
MUTATION_LOCK_TIMEOUT reason, the provider keeps its original handle,
which still resolves to the old key, and the credential driver holds
the same number of values as before the refresh.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): export mutation lock wait and timeout metrics
A handler mutation guard can now wait up to 10 seconds for its keys and
then fail with UNAVAILABLE, but nothing showed operators how long guards
waited or how often they gave up, short of reading warning logs on each
replica.
Export openshell_server_mutation_lock_wait_seconds, a histogram of the
time one guard took to acquire all of its keys (local registry plus
PostgreSQL advisory locks), recorded on success, and
openshell_server_mutation_lock_timeouts_total, a counter of guard
acquisitions that timed out. Both carry a bounded scope label: global,
workspace or sandbox. The platform scope, whose workspace is empty,
counts as global. The histogram uses the same 1 ms to 15 s buckets as
the relay and peer histograms, and the timeout counter starts at 0 for
every scope, so rate() works on an idle replica. The timeout warning
now also logs the scope. A guard that fails for another reason, such as
a lock connection PostgreSQL did not open, is not counted as a timeout
and logs a "mutation lock acquisition failed" warning instead.
Lifecycle, driver-watch and reconcile paths take only process-local
locks and never time out, so they are not recorded.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* test(server): add PostgreSQL mutation lock integration tests
The scoped mutation locks were only unit-tested on SQLite, where the
database layer is a no-op. Add ignored postgres_* tests that run the
advisory-lock layer against a real PostgreSQL through mise run
test:rust:postgres, each in its own disposable schema (TestSchema).
persistence/mutation_lock_pg_tests.rs uses two stores on one database
as two replicas. It covers disjoint sandbox scopes holding at once, a
workspace key blocking only sandboxes in that workspace, the global key
blocking every scope, and exclusion in both directions against a raw
holder of the legacy global key, whose 55P03 stays a database error
outside a guard acquisition. It checks that timed-out and cancelled
acquisitions release the keys they already took, that a released lock
connection returns to the pool without residual locks, that a release
stalled on the network closes its session and the pool recovers, and
that the lock pool never grows past its size. It also checks that a
full lock pool times out as lock contention, while a lock connection
that PostgreSQL never answers fails as a database error.
compute/mutation_guard.rs runs the random scope mix across two
PostgreSQL runtimes and checks that a by-id guard waits for a workspace
writer on the other replica. The provider refresh lock test moves onto
the runner too, and checks that a refresh proceeds while every lock
connection is held.
A bench_postgres_* capacity check measures a paced burst of 500
reconnects 12 ms apart into one replica and fails when the p99 lock
wait reaches a fifth of the lock timeout. Machine load moves those
waits, so mise run test:rust:postgres skips it and mise run
test:rust:postgres:bench runs it alone, through a new
OPENSHELL_TEST_POSTGRES_PREFIX setting in the runner.
The tests add test-only helpers: the lock pool's size and idle count
and the backend pid of a held lock session. Nothing changes outside
tests and the test runner.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* ci: run the PostgreSQL-backed server tests
Branch Checks now runs the postgres_* tests through the same runner
developers use, on an x86_64 Linux runner with Docker. They cover lock
scope exclusion, interop with replicas still on the old global key,
cancellation, lock-pool bounds and connection failures, which no other
CI lane checks directly. The load-sensitive bench_postgres_* capacity
checks stay local.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* docs: document scoped mutation locks and lock-pool sizing
In the High Availability guide, size PostgreSQL for 14 connections per
gateway pod, 10 for data and 4 for the new lock pool, so three or more
replicas now exceed what the PostgreSQL default leaves, and keep
headroom for lock sessions that cancelled requests leave behind.
Explain which changes lock a sandbox, a workspace, or the whole fleet,
and that a lock wait over 10 seconds returns a retryable UNAVAILABLE
with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to
raise max_connections to the 14-per-pod size before upgrading, because
the rollout already runs new pods at that count, and warns that older
replicas keep serializing every mutation until the rollout finishes.
Monitor Capacity gains the lock metrics and their queries, and names
slow driver, credential, middleware, or profile source calls as another
cause of lock waits.
List the lock wait histogram and timeout counter, with their scope
label, on the Gateway Metrics page. The counter also counts background
work and leaves out lock connections that PostgreSQL does not open
although at least a second of the wait remained.
In the cluster debugging skill, replace the single advisory-lock
paragraph with the lock levels and pool, what each timeout detail
means, how a lock connection that PostgreSQL does not open shows up,
when terminating a lock backend is safe, a pg_locks query that shows
holders and waiters, and how to spot the global key. The per-pod greps
also read the timeout counter, and new rows cover lock timeouts and
lock connections that PostgreSQL does not open.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): check mutation lock ordering in debug builds
The mutation lock ordering rules were prose only, so a nested guard on
a different key passed every test and could only deadlock under
contention. Debug builds now register each held guard with the Tokio
task (or the thread, outside a task) that acquired it, and panic before
waiting when a local key, a blocking lifecycle gate or the SSH identity
lock would break rules 1, 4 or 5. Startup endpoint-status
reconciliation and the tests that hold a guard while driving guarded
code run under lock_order::branch(), which marks deliberate concurrency
within one task. Release builds compile the check out.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): do not hold gateway shutdown for the endpoint-status retry
When a replacement supervisor session failed its endpoint-status reset,
the spawned cleanup task kept the session lifetime guard through
retry_endpoint_status_after_supervisor_disconnect. That retry only ends
once the reset is durable, so a gateway shutdown during a database or
lock outage waited the full 10 s and reported ownership cleanup as
incomplete. The guard now covers only the lifecycle demotion, as in
finish_supervisor_session, and a SQLite test checks that shutdown
completes while the retry is still blocked.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* refactor(server): make the platform profile lock scope explicit
Platform-scope provider profile writers (import, update, delete) pass an
empty workspace on purpose and need X(global), because they validate
sandboxes in every workspace. MutationScope::profiles() now names that
escalation and picks Global for the platform scope, instead of relying
on MutationScope::Workspace("") to mean it. An empty workspace name now
trips a debug assertion, and release builds keep X(global) as a
fail-safe that over-locks rather than under-locks.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): amortize local mutation lock table sweeps
LocalMutationLocks swept its whole table on every key lookup, so a
burst of N concurrent acquisitions on one replica cost O(N^2) under a
blocking mutex. The table is now swept only when a new key grows it
past the larger of 64 and twice the live entries left by the last
sweep, which makes acquisition amortized O(1). Released entries,
including those of cancelled waiters, stay bounded at about twice the
peak live set. The lifecycle gate registry keeps its per-call sweep.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): make the database pool ceiling configurable
Both stores hardcoded their pool ceiling: 10 connections for Postgres, 5
for on-disk SQLite. One pool is shared by every database-backed RPC, so
that number is also the gateway's ceiling on concurrent database work.
Once every connection is checked out, callers queue on acquire and sqlx
logs "time to acquire exceeded slow threshold"; sandbox creates then time
out and are retried, which adds load rather than shedding it.
The right ceiling is deployment-specific — it depends on how many gateway
replicas share the database and what max_connections the server itself
allows — so it cannot be a single number baked into the binary.
Expose it on the same config surface as the rest of the gateway's
settings: --db-max-connections, OPENSHELL_DB_MAX_CONNECTIONS, and the
TOML key database_max_connections, resolved in that precedence order.
Helm deployments set the key through gatewayConfig. Omitting it keeps
each backend's previous value, so existing deployments do not move.
Values below 1 are rejected rather than silently replaced by the default:
a zero pool would block every acquire, and quietly ignoring a typo would
reproduce the ceiling the operator is trying to lift. An in-memory SQLite
database stays pinned to one connection, since the database lives inside
that connection.
Refs: #2561
Signed-off-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com>
(cherry picked from commit
|
||
|
John T. Myers
|
0b1f6f86d2 |
feat(providers): add CLI label creation and filtering (#4265)
* docs: document workspace and provider labels (fixes #4250) Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * feat(providers): support CLI labels and selector filtering Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(providers): update schema inventory and scoped test environment Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Shiju
|
6a6401b9c3 |
fix(cli): backpressure sandbox uploads (#4448)
* fix(cli): backpressure sandbox uploads Stream archive uploads over a bounded SSH subsystem whose credit follows extraction stdin progress. Preserve the existing input budget and tar destination semantics, validate completion, and clean up cancelled transfers. Add protocol, lifecycle, and large-upload regression coverage without changing russh. Signed-off-by: Shiju <shiju@nvidia.com> * fix(cli): preserve upload cancellation and errors Keep cancellation active through upload setup and retry delays, and terminate cancelled SSH probes. Preserve protocol failures through output completion and report captured diagnostics on early extraction failure. Remove redundant upload validation and extend the affected regressions. Signed-off-by: Shiju <shiju@nvidia.com> * refactor(cli): carry upload credit over SSH streams Replace the framed upload protocol with SSH's own streams. The client sends one JSON plan line, the raw archive and EOF. The supervisor's stdout carries only cumulative totals of bytes written to extraction stdin, extraction output goes to stderr, and the exit status reports the result. Extraction starts inline through the ordinary exec path. This removes the frame codec, SHA-256 digests, the CLI sha2 dependency and the supervisor upload state machine while keeping the 1 MiB credit window, the 4 MiB input guard, cancellation and error reporting. A failed archive now stops SSH before closing stdin, so EOF never marks a truncated archive complete. Signed-off-by: Shiju <shiju@nvidia.com> * fix(cli): isolate upload SSH connections and pin credit cadence - Apply the forward multiplexing overrides to the upload ssh command, so a user ControlMaster cannot carry an upload over another session's master connection. - Kill ssh before closing its stdin when a transfer fails locally, and include ssh stderr in transport errors. - Share ACK_INTERVAL_BYTES from core. Assert at compile time that CLI chunks fit the credit left after an ack and that the window stays below the supervisor's pending-input guard. - Log the real reason when the supervisor rejects an upload. - Test that a single-packet upload reports its final total. Signed-off-by: Shiju <shiju@nvidia.com> * test(cli): assert upload ssh opens the subsystem without a TTY Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
alangou
|
c608c13fbd |
fix(prover): distinguish sibling endpoints and fail closed on solver errors (#4326)
Signed-off-by: Adrien Langou <alangou@nvidia.com> |
||
|
Yuedong Wu
|
3495c8e92a |
fix(kubernetes): refresh sandbox runtime image on start (#4376)
Signed-off-by: Yuedong Wu <dwcn22@outlook.com> |
||
|
Oliver Calder
|
16c8ca5c43 |
fix(docs): use lowercase "snap" as it is not a proper noun (#4387)
Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Shiju
|
e585b49173 |
feat(providers): support stable credential placeholders (#3339)
* feat(providers): support stable credential placeholders Add credentials[].stable_placeholder so a running client keeps using an externally rotated static credential through the reference it already holds. Squashed replay of the earlier four commits onto upstream main 71c3cd957; re-derives the frozen schema fingerprints and adapts the provider environment revision test to the added capability argument. Signed-off-by: Shiju <shiju@nvidia.com> * fix(e2e): apply stable-placeholder supervisor image override Use ManagedGateway's per-process supervisor image override so the fixture CA reaches the supervisor when runtime image environment variables take precedence over gateway TOML. Restore the wrapper's original image after each test, following the provider-readiness fixture. Signed-off-by: Shiju <shiju@nvidia.com> * test(server): update public RPC schema fingerprint after merging main Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
Seth Jennings
|
4c1b16a4a1 |
feat(server): add operator-only provider credential retrieval (#4357)
* feat(server): add operator-only provider credential retrieval Authenticate operators exclusively through verified direct gateway mTLS. Export selected runtime credentials with freshness validation and all-or-nothing delivery. Coordinate refresh across callers and replicas, preserve rotated grants across RPC cancellation, and fence credential delivery against reconfiguration. Add generated bindings, security and regression tests, and operator documentation. Signed-off-by: Seth Jennings <sjenning@redhat.com> * feat(sdk): add curated operator credential retrieval Expose provider credential clients in Python and TypeScript with explicit workspace and key selection, freshness options, optional expiration, and preserved gateway errors. Reuse existing mTLS transports without introducing retries or secret caching. Add generated-wire and error-mapping tests, SDK package validation, and operator usage documentation. Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(server): honor recovery state for queued automatic refreshes Signed-off-by: Seth Jennings <sjenning@redhat.com> --------- Signed-off-by: Seth Jennings <sjenning@redhat.com> |
||
|
Oliver Calder
|
82bc756545 |
feat(snap): run gateway as a user service by default (#4099)
* feat(snap): run gateway as a user service by default Replace the existing system `gateway` service with separate `user-gateway` and `system-gateway` services. For new installs, only the `user-gateway` service is enabled. For existing installs, only the `system-gateway` service is enabled. Existing installs continue to have a one-time migration which removes legacy insecure configurations. It is up to `install.sh` or users to manually copy mTLS credentials from the root-owned `$SNAP_COMMON/tls` to the invoking user's OpenShell snap directory. This is explained in the snap description in `snapcraft.yaml`, visible in the Snap Store listing and via `snap info openshell`. Service enablement is now managed by a new snap configuration option named `gateway-mode`, so users can switch from one mode to another via e.g. `sudo snap set openshell gateway-mode=user`. The `install` and `post-refresh` hooks select which service to start by setting this mode. If the `gateway-mode` is already set, then we know the one-time migration has already taken place. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): simplify the snap gateway wrapper Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): use `snap start --user` when running as non-root Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): ensure install.sh waits for the correct docker daemon Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * feat(snap): document how to safely switch between gateway modes New installs run the gateway as a user service by default, while upgrades of existing installs from prior to the user gateway continue to use the existing system gateway. The user and system gateway are fundamentally different services and there is no way to migrate state between them. However, it may be desirable for a user to run the gateway as a system service after a fresh install. To support this, this commit adds documentation for how to safely transition from the user gateway mode to the system gateway mode, and vice versa. Also, clarifies when it is necessary to manually copy TLS credentials from the system gateway state directory to the user's home directory: when upgrading a pre-TLS snap installation, or when switching from user to system gateway mode. When upgrading an existing install with an HTTPS gateway running as a system service, copying credentials again is not necessary. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Daneyon Hansen
|
40e01424c1 |
fix(helm): improve GatewayClass overrides (#4345)
* fix(helm): improve GatewayClass overrides Give chart-created Gateways a distinct default name and allow GRPCRoutes to select a listener on an existing Gateway. Fixes: #2469 Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io> * fix(e2e): resolve rendered Gateway name Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io> * fix(helm): preserve Gateway name defaults Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io> --------- Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io> |
||
|
alangouandJohn Myers
|
b959bb6817 |
fix(policy): share endpoint path matching between Rust and Rego (#4336)
* fix(policy): share endpoint path matching between rust and rego Signed-off-by: Adrien Langou <alangou@nvidia.com> * fix(prover): align endpoint selector containment with runtime matching Signed-off-by: Adrien Langou <alangou@nvidia.com> * test(policy): register endpoint matcher in token grant fixtures Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Adrien Langou <alangou@nvidia.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Gaizka MenendezandTaylor Mutch
|
fe3942f2da |
feat(helm): migrate gateway configuration to gatewayConfig (#3384)
* feat(helm): add generic gateway TOML serializer Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(architecture): define Helm gateway configuration boundary Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): define default gateway configuration map Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): render gateway config from values map Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): cover generic gateway TOML rendering Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): protect gateway config secret boundary Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(helm): classify legacy gateway configuration values Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): derive dual-use resources from gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): migrate gateway config scenarios Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): derive credential resources from gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): migrate gateway config overlays Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): remove gateway configuration shadow values Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): align runtime config with resources Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(helm): add gateway config migration guide Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): validate rendered gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): cover config resource coherence Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): make Kubernetes E2E deployable Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(e2e): pass host aliases through gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(e2e): reference gateway config host alias Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): enforce gateway resource ownership Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(e2e): cover chart host gateway input Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): address gateway config review findings Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): repair RFC 0012 migration integration Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(e2e): align Kubernetes parity with RFC 0012 Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): complete gateway config migration Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * chore(helm): add SPDX header to TOML template * fix(helm): preserve legacy gateway configuration aliases Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): complete legacy gateway config compatibility Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): support gateway config arrays of tables * fix(helm): preserve sandbox identity aliases * fix(helm): address gateway config review findings Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): preserve gateway config migration settings Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * test(helm): validate extension arrays in CI values Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(helm): restore unconditional SPDX headers Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(helm): restore route SPDX header placement Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): simplify configuration and migration examples Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): remove redundant flag explanations Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): move chart configuration guidance to Kubernetes setup Signed-off-by: Taylor Mutch <taylormutch@gmail.com> --------- Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> Signed-off-by: Taylor Mutch <taylormutch@gmail.com> Co-authored-by: Taylor Mutch <taylormutch@gmail.com> |
||
|
alangou
|
81f712e64e |
fix(drivers): validate vendor-agnostic CDI device names (#4306)
Reject host paths and malformed CDI selectors for Docker and Podman using shared qualified-name validation. Preserve the generic GPU request checks for drivers using other identifier formats, including VM PCI addresses. Validate explicit and resolved CDI selectors, cover vendor-agnostic names and rejected paths, and update the related documentation. Signed-off-by: Adrien Langou <alangou@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
John T. Myers
|
67adcf1a3c |
feat(ocsf): emit full JSON records to supervisor stderr (#4323)
* feat(ocsf): emit full JSON records to supervisor stderr Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(ocsf): preserve console record boundaries Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Matthew Grossman
|
1fff8b97b9 |
feat(podman): honor OCI image working directories (#3982)
* fix(supervisor): report rejected OCI working directories as WorkspaceValidationFailed The RFC 0012 split dropped the supervisor exit status that drivers map to the WorkspaceValidationFailed condition. An image WORKDIR the sandbox identity cannot use then surfaced as ControlSupervisorStartFailed on Docker and as a signal kill on Podman. The supervisor now exits with the reserved status when the sandbox rejects the image working directory. Docker maps that supervisor exit during readiness and monitoring, and Podman maps the supervisor companion's exit instead of the workload's. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * feat(podman): honor OCI image working directories Podman sandboxes now use a custom OCI image WORKDIR as the workspace and as the working directory for agent commands, matching Docker. Empty, /, and /sandbox values keep the managed /sandbox workspace volume. A custom workspace stays in the image's container filesystem: no workspace volume, no archive upload, and no root setup step. Resolve the image ID, user, environment, and working directory from one pinned inspection, validate the workdir with the shared OCI rules, and reject Podman control-path overlaps and image volumes or driver mounts that cover it. The runtime starts from / and passes the resolved path to agent commands with --workdir. Add podman_oci_identity to the Podman CI test list. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(drivers): preserve custom workspace failures across supervisor cleanup Check the Podman custom workspace rather than the absent managed volume in the OCI identity E2E. Classify Bollard wait errors by exit code and record Docker workspace failures before removing the supervisor so readiness retains the specific failure reason. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(docker): use sandbox ID for recorded startup failures Keep the workload container ID for Docker inspection and use the sandbox ID to retrieve the monitor failure. Test with distinct identifiers so the lookup cannot accidentally pass with a container ID. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(podman): keep WORKDIR change focused on workspace behavior Remove the new cross-driver workspace failure classification path and its monitor/readiness workaround. Preserve Podman WORKDIR resolution, mount safety, and workspace rejection without requiring a distinct condition reason. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(podman): simplify inspected image metadata and test fixtures Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * code review Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
Divesh
|
834b79a8c2 |
feat(server): place supervisor sessions by consistent hash (#3661)
* feat(server): place supervisor sessions by consistent hash and hand them off on shutdown Signed-off-by: divesh <dgude@nvidia.com> * feat: route long-lived sandbox connections to the owning gateway replica Signed-off-by: divesh <dgude@nvidia.com> * feat(helm): add opt-in per-replica GRPCRoute routing Signed-off-by: divesh <dgude@nvidia.com> --------- Signed-off-by: divesh <dgude@nvidia.com> |
||
|
Eric Curtin
|
3082a9ad7d |
fix(cli): plan sandbox uploads before provisioning (#4193)
Reject bad uploads before create. Share the upload transfer helper. Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Mike Nguyen
|
9fd41e6bfd |
fix(ssh): persist sandbox host identities (#4094)
* fix(ssh): persist sandbox host identities Store each sandbox's Ed25519 host key in the gateway credential store and deliver it only to the supervisor. Preserve identity across restarts, delete owned credentials with the sandbox, and expose the public SHA256 fingerprint through sandbox and SSH-session APIs and client SDKs. Cover credential ownership, cancellation, deletion retries, client compatibility, and pinned SSH connections through lifecycle transitions. Closes #3835 Signed-off-by: Mike Nguyen <miken@nvidia.com> * fix(compute): clean up failed sandbox SSH identity creation Signed-off-by: Mike Nguyen <miken@nvidia.com> * test(ssh): wait for sandbox deletion before name reuse Signed-off-by: Mike Nguyen <miken@nvidia.com> --------- Signed-off-by: Mike Nguyen <miken@nvidia.com> |
||
|
alangou
|
33869a1df5 |
fix(images): update gateway base and allow supervisor base override (#4236)
Signed-off-by: Adrien Langou <alangou@nvidia.com> |
||
|
Emilien Macchianddivesh
|
8b3cc3fdc0 |
feat(server): add gateway capacity metrics and optional HPA (#3978)
* feat(server): add gateway capacity metrics and optional HPA Expose per-replica supervisor sessions, pending relay capacity, relay rejections and claim latency, and outbound peer request outcomes and latency. Use bounded labels and Prometheus histograms for the new latency metrics while preserving existing summary metrics. Add an optional Helm HPA with external-database and resource validation, conservative scale-down defaults, and support for custom metrics. Keep certificate hook pods outside gateway workload selectors. Document per-pod scraping, scaling limits, upgrade behavior, and PostgreSQL connection sizing. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): unify routing metrics and clarify replica capacity Combine local relay setup and outbound peer requests in one counter, labeled by operation, route, target, outcome, and gRPC status. Preserve peer latency metrics. Rename the rejection reason from global_capacity to replica_capacity to reflect the per-replica relay budget. Keep capacity limits unchanged. Update tests and documentation. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): align routed request metric labels and outcomes Count a local relay as successful only when its supervisor claims it, the same event that answers a peer relay on the owner, and rename the outcomes to local_error and remote_error so they say where an attempt failed. Label the peer latency histogram by operation, like the routed request counter, and rename the target label to relay_kind so it does not read as the Prometheus scrape target. Rename RelayCapacity.global to per_replica, drop the per-sandbox relay capacity gauge, which no per-sandbox series can pair with, describe the latency histogram as peer-only, and restore the note that unavailable spikes are expected during rollouts. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> --------- Signed-off-by: Emilien Macchi <emacchi@redhat.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> |
||
|
Shiju
|
1c123a4542 |
feat(gateway): check VM host tools during config preflight (#4037)
* feat(gateway): validate VM filesystem tools during preflight Check required local VM tools through config preflight and share executable resolution with VM image operations. Report selected paths and actionable errors without creating gateway or sandbox state. Bound probe output and execution time, and clean up probe descendants on interruption. Preserve pure static validation and skip local tool checks for remote driver endpoints and unrelated drivers. Fixes #3951 Related to #3955 Signed-off-by: Shiju <shiju@nvidia.com> * fix(gateway): stabilize filesystem preflight checks Combine identical filesystem-tool error arms and normalize rendered diagnostics in command tests so terminal wrapping preserves assertions. Describe driver TLS validation without depending on removed guest fields. Signed-off-by: Shiju <shiju@nvidia.com> * test(gateway): serialize preflight fixture paths as TOML Keep temporary paths quoted and escaped through the TOML serializer instead of relying on Rust Debug formatting. Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
Shiju
|
d676e036a4 |
fix(vm): reject conflicting workload identity selectors (#4036)
* fix(vm): enforce the configured workload identity Reject conflicting policy users and groups before VM image preparation and before guest attach or process startup changes state. Validate supervisor policy updates against the protected VM workload identity. Preserve the gateway CA transport and capability-free sandbox launcher. Signed-off-by: Shiju <shiju@nvidia.com> * test(sandbox): clarify VM identity rejection fixtures Name invalid user and group fixtures distinctly and move the final workload identity into its group mismatch test. Signed-off-by: Shiju <shiju@nvidia.com> * fix(supervisor): align VM identity startup with current APIs Pass the optional rejection-log key for VM identity failures and keep generic startup-write regressions free of VM identity constraints. Repair the call sites after the branch rebase so the identity and cleanup proposals compile against the current startup helpers. Signed-off-by: Shiju <shiju@nvidia.com> * fix(vm): restore inactive sandbox workload identity Recover the persisted overlay owner before publishing stopped and terminal sandboxes. Keep resources manageable when identity metadata is invalid. Clarify fixed MicroVM ownership in policy-generation guidance. Signed-off-by: Shiju <shiju@nvidia.com> * test(vm): flush identity fixture before restart Persist the canonical identity file before readiness and report the observed exec, canonical and file-owner identities before comparing them. Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
Philippe Martin
|
f7273e48f6 |
fix(providers): restore supervisor-backed GCP metadata discovery (#3973)
* fix(providers): restore supervisor-backed GCP metadata discovery Relay the reserved metadata endpoint to the supervisor and restore project, account, and placeholder token responses from live provider state. Cover Google SDK discovery and repeated refresh with provider E2E tests. Closes #3860 Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(providers): preserve default metadata account without an email Use the default account identifier when the optional service account email is missing or empty. Cover repeated SDK refresh for missing, empty, and configured email values with distinct providers for parallel E2E execution. Signed-off-by: Philippe Martin <phmartin@redhat.com> * test(sandbox): fix metadata relay lint and timeout Signed-off-by: Philippe Martin <phmartin@redhat.com> --------- Signed-off-by: Philippe Martin <phmartin@redhat.com> |
||
|
alangou
|
36819f476d |
fix(cli): stop uploads when Git filtering fails or selects no files (#3957)
Signed-off-by: Adrien Langou <alangou@nvidia.com> |
||
|
Matthew Grossman
|
5d6b3b8120 |
fix(kubernetes): serialize lifecycle cleanup with sandbox restart (#4078)
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
krishicks
|
8091f66877 |
feat(sandbox): write agent output to the container log (#4005)
Since #2726 the canonical main process's stdout and stderr are captured in pipes that feed only the in-memory replay buffer used by sandbox connect. Agent output therefore never reaches the container's own stdout and stderr, so it is missing from kubectl logs, docker logs, and podman logs and from anything that collects container logs. Before #2726 the entrypoint inherited the container's descriptors and its output appeared there. Copy the main process's output to the launcher's stdout and stderr in addition to the replay buffer, restoring the earlier behavior: - Output is copied byte for byte to the matching stream from a forwarder thread per stream, after it is published to the replay buffer. When the container runtime falls behind on a stream, that stream's reader waits instead of dropping output, so backpressure reaches the agent as it did with inherited descriptors, while the other stream and attachments keep receiving output. - Before the main process's exit is published, the output readers finish and queued output is drained to the container log, so an agent's final lines are not lost at shutdown. A 30 second deadline covers both; when it expires, readers waiting on the container log are released and drain the pipes into the replay buffer only, so a stalled container log cannot block exit reporting. - PTY-mode processes are not copied. The terminal stream carries escape sequences and echoed input, and terminal commands never reached the container log before #2726. - Exec, SSH, and SFTP sessions are not copied. Launcher log lines keep their existing format and remain in the container's stderr. They are written as whole lines, and a newline is inserted first when the agent left stderr mid-line, so launcher and agent lines do not merge. The Docker and VM drivers appended the tail of the workload's output to failure messages: Docker the workload container's log, and the VM driver the guest console, which carries the launcher's stdout and stderr. Those messages land in the sandbox's Ready condition and in platform events that the gateway republishes to the sandbox event stream. With agent output in that log, those messages would carry arbitrary agent output, including anything sensitive the agent prints, into gateway status and events. The supervisor starts its health endpoint only after the agent starts, so every Docker failure path could include agent output, and the VM driver reports one whenever the VM or host supervisor exits. Forward only the supervisor's log tail, matching the Podman driver, which reads the workload log solely to match fixed launcher markers and never forwards raw workload output. The workload's output remains available through docker logs and the VM's rootfs-console.log. Document where main process output appears in the logging docs and the cluster debugging skill. Closes #3928 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Drew Newberry
|
021400be8a |
refactor(auth): separate sandbox identity from TLS (#3110)
* refactor(auth): separate sandbox identity from TLS Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(auth): clarify gateway mTLS behavior Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(auth): include workspace scope in TLS authorization checks Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): bound service auth sandbox names for large PIDs Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Derek Carr
|
912a077bd6 |
feat(service): add bearer authorization passthrough (#3796)
* feat(service): add bearer authorization passthrough Signed-off-by: Derek Carr <decarr@redhat.com> * docs(sdk): add service authorization migration guide Signed-off-by: Derek Carr <decarr@redhat.com> * fix(server): remove stale version import Signed-off-by: Derek Carr <decarr@redhat.com> * docs(upgrade): remove service authorization SDK guide Signed-off-by: Derek Carr <decarr@redhat.com> * fix(e2e): relabel provider readiness TLS mount Signed-off-by: Derek Carr <decarr@redhat.com> * test(e2e): stabilize exposed service routing Signed-off-by: Derek Carr <decarr@redhat.com> * test(e2e): support HTTPS service routing Signed-off-by: Derek Carr <decarr@redhat.com> --------- Signed-off-by: Derek Carr <decarr@redhat.com> |
||
|
Eric CurtinandDrew Newberry
|
07a486d751 |
fix(cli): accept sandbox name before -- in exec (#3901)
* fix(cli): accept sandbox name before -- in exec Closes #3882 Signed-off-by: Eric Curtin <eric.curtin@docker.com> * fix(cli): define exec grammar in clap Signed-off-by: Eric Curtin <eric.curtin@docker.com> * docs(sandboxes): remove exec overview change from PR Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Shiju
|
0ea0d31020 |
fix(supervisor): restore canonical stdin after connection loss (#3852)
* fix(supervisor): restore canonical stdin after connection loss Probe idle SSH peers and enforce a receive deadline during transport I/O, including writes blocked by a stalled relay. Release the dead attachment's stdin lease through existing handler cleanup. Retry denied write intent on later ordinary input without displacing a healthy owner. Preserve explicit read-only, EOF and detach behavior, and discard control bytes retained while input ownership was denied. Cover half-open forwarding, blocked writes, healthy idle peers and competing reconnects through the production supervisor frame bridge and real SSH. Fixes #3648 Signed-off-by: Shiju <shiju@nvidia.com> * docs(skills): describe read-only reconnect input retry Explain what an openshell-cli user sees when automatic recovery reattaches before the supervisor closes the dead connection: the attachment reports read-only, later ordinary input retries stdin acquisition and prints `input enabled`, input typed while read-only is discarded, exit keys still detach, and an explicitly read-only viewer or a healthy owner is never affected. Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
krishicksandEvan Lezar
|
a875add234 |
feat(server): write gateway OCSF events to JSONL (#3264)
* feat(server): write gateway OCSF events to JSONL Previously, gateway security activity was available only in diagnostic output, and events not associated with a sandbox, such as TLS certificate reloads, had no independent structured record. Now, configuring `openshell.gateway.ocsf_log` writes every gateway-produced OCSF record to a bounded JSONL destination independently of `RUST_LOG`. The destination supports daily or disabled rotation, retention limits, queue bounds, and optional schema downgrade to OCSF 1.1 or 1.3. Additionally, existing gateway emitters (TLS reloads, service routing, and policy approval and auto-approval audits) emit structured events, so they reach the JSONL destination, console shorthand, and the affected sandbox's log stream. Records identify the gateway by its configured name in `device.uid` and `device.name`, shared across replicas, with `device.hostname` identifying the replica and `device.os` the gateway's operating system. Metrics and warnings expose known best-effort losses. Refs #2762 Signed-off-by: Kris Hicks <khicks@nvidia.com> * fix(mxc): attribute ETW events to gateway Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Kris Hicks <khicks@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> Co-authored-by: Evan Lezar <elezar@nvidia.com> |
||
|
Philippe MartinandJohn Myers
|
9cb72baa2e |
feat(docker): support corporate proxy CA bundles (#3549)
* feat(docker): support corporate proxy CA bundles Closes #3545 Validate and stage operator-owned proxy CA bundles for Docker supervisors, add corporate proxy E2E coverage, and document the trust contract. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(docker): validate proxy config on startup Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): use the E2E workload image for proxy tests Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): generate strict corporate proxy certificates Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): surface intercepted TLS fixture errors Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): drain buffered TLS proxy data Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): relay intercepted HTTP deterministically Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Philippe Martin <phmartin@redhat.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Drew Newberry
|
acbac9cb79 |
feat(sandbox): add main restart policy (#2798)
* feat(sandbox): add main restart policy Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): harden policy-driven restarts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): address restart review feedback Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> * fix(sandbox): port restart policy to current runtime Signed-off-by: Drew Newberry <anewberry@nvidia.com> * perf(sandbox): restart promptly after terminal delivery Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> |
||
|
Drew Newberry
|
e63cfa1182 |
fix(cli): keep SSH forwards owned by spawned process (#3759)
Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
36b0386c92 |
feat(cli): detach sandbox sessions with Ctrl-D (#3744)
Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
a67567e583 |
fix(snap): require mTLS for the snap gateway (#3726)
* fix(snap): require mTLS for the snap gateway Replace the installer opt-in with an authenticated snap gateway. The wrapper no longer forces plaintext, so the gateway serves TLS from the bundle it already generates in $SNAP_COMMON/tls. The install hook writes a config that enables mTLS user auth instead of unauthenticated access, and a new post-refresh hook migrates the exact legacy default on existing installs. install.sh waits for the gateway, detects whether it serves TLS, copies the client bundle into the target user's snap state directory, and registers the gateway over HTTPS. Older plaintext snap revisions still register over HTTP with a warning. The release canary asserts mTLS auth and HTTPS registration. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): pass config preflight and detect the mTLS gateway reliably An explicit [openshell.gateway.mtls_auth] table fails config preflight, which validates mTLS auth before the local TLS bundle supplies the client CA. Write a default that pins the Docker driver instead; with the wrapper's TLS bundle the gateway requires client certificates and enables mTLS user auth automatically, as the native packages do. The mTLS gateway rejects TLS handshakes without a client certificate, and it still answers plaintext loopback HTTP for sandbox service routing, so the installer could misdetect it as a legacy plaintext gateway. Probe HTTPS with the root-owned client bundle, and treat a gateway as legacy only when a plaintext gRPC Health call succeeds. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * chore(snap): simplify install hook comment Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): migrate insecure gateway configs on refresh Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(snap): simplify mTLS detection and config migration Detect the mTLS snap from the installed revision's post-refresh hook instead of probing plaintext gRPC, and drop the scheme global. Remove the installer's pre-hook config fallback, which is dead now that every channel ships the install hook and which wrote the insecure default. Give the install hook a single write path with a simple backup name, and shorten the manual client certificate steps. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): stop keeping a copy of replaced insecure configs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(install): make the snap an opt-in install method Stop selecting the OpenShell snap just because the snap command exists. Linux installs default to the Debian or RPM package; OPENSHELL_INSTALL_METHOD=snap (or deb, rpm) selects the package explicitly. Hosts that already have the OpenShell snap keep refreshing it rather than gaining a second gateway on the same port. The release canary and snap repro script opt in explicitly. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): let the gateway auto-detect its compute driver Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): restart the gateway after refresh Published revisions use refresh-mode: endure, and snapd honors the old revision's setting during a refresh, so the plaintext gateway kept running with the migrated config unused until a manual restart. Restart the gateway from the post-refresh hook so the mTLS config takes effect immediately. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(snap): drop refresh notes from the snap description Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(snap): trim snap refresh notes from installation docs Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
d4f5034d7f |
fix(gateway)!: make the WebSocket tunnel opt-in (#3727)
The /_ws_tunnel endpoint pipes a WebSocket into the full gRPC service. On a plaintext loopback gateway any web page could open it, since browsers do not apply CORS to WebSocket upgrades. Mount the tunnel only when enable_websocket_tunnel is set (config file, --enable-websocket-tunnel, or OPENSHELL_ENABLE_WEBSOCKET_TUNNEL; server.enableWebsocketTunnel in Helm). BREAKING CHANGE: gateways behind an authenticating edge proxy must set enable_websocket_tunnel = true for CLI edge-tunnel connections. Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Johnny Greco
|
854b2370b8 |
fix(policy): align quickstart, policy skills, and pypi profile with current behavior (#3695)
* fix(examples): add quickstart rule with policy update and show OCSF logs The quickstart applied policy.yaml with `openshell policy set`, which replaces the whole policy. The file omitted /bin from the restrictive default, so the live filesystem additivity check could reject it. Add the rule with `openshell policy update` instead, and keep policy.yaml as a complete policy for `sandbox create --policy` that covers the default read-only paths. The demo and README filtered logs with `--level warn`, but the server ranks OCSF events as INFO, which hid the policy decisions the demo shows. Query `--source sandbox` without a level filter and match the OCSF shorthand (DENIED/ALLOWED) instead of the retired key=value format. Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs(skills): align generate-sandbox-policy with proxy behavior Remove the `protocol: sql` validation check and the SQL `command` matcher, which the published policy docs no longer describe. Correct the private IP guidance: exact user-declared hostnames may reach private addresses without allowed_ips. Wildcard, hostless, and advisor-proposed endpoints still need allowed_ips, and loopback, link-local, unspecified, and cloud metadata addresses stay blocked. Stop describing an omitted protocol as pure L4 or uninspected. The proxy still terminates TLS, parses HTTP strictly, and enforces request authority; it only skips method and path rules. Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(providers): drop pip script paths from pypi profile binaries OpenShell identifies a process by /proc/<pid>/exe, so a pip script runs as its Python interpreter and the .venv/bin/pip entries could never match. The venv interpreters that run those scripts are already listed, so remove the script paths and explain in the header that users must list interpreters. Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs(skills): fix openshell-cli policy iteration steps Monitoring denials with `--level warn` hides them, because the server ranks OCSF policy events as INFO. Drop the level filter and describe the OCSF shorthand DENIED lines instead of the retired `action: deny` format. `policy get --full > file` produced input that `policy set` cannot parse: the output starts with revision details before the `---` separator, and --full adds provider-composed rules. Export `--base` and keep only the YAML after the separator. Also stop recommending full replacement for filesystem, Landlock, or process changes, which require recreating the sandbox, and drop the SQL mention that generate-sandbox-policy no longer covers. Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs(skills): qualify authority checks for omitted-protocol endpoints An endpoint without `protocol` only receives authority checks on HTTP requests the proxy parses after default TLS handling. Without an L7 route or required middleware, other CONNECT payloads such as HTTP/2 prior knowledge can use the raw relay, and `tls: skip` bypasses termination and parsing. Stop describing omitted-protocol endpoints as always authority-checked. Signed-off-by: Johnny Greco <jogreco@nvidia.com> --------- Signed-off-by: Johnny Greco <jogreco@nvidia.com> |
||
|
Drew Newberry
|
9244868056 |
docs: refresh architecture and agent guides (#3705)
* docs: refresh architecture and agent guides Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: describe updated security architecture neutrally Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: highlight new isolation primitives Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(sandboxes): clarify how to disconnect Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: align architecture and guides with current navigation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(extensibility): streamline extension authentication guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
c93fd94a5f |
feat(cli): import provider profiles from HTTP URLs (#3706)
* feat(cli): import provider profiles from HTTP URLs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(cli): initialize crypto provider for remote profiles Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
4688061882 |
fix(sandbox): deliver complete exec output before success (#3688)
* fix(sandbox): preserve exec output through channel close Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(exec): propagate output delivery failures before exit Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(cli): clarify exec output delivery failures Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
52cb8ecee7 |
fix(kubernetes): remove NetworkPolicy acknowledgement (#3677)
Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
grs
|
8369bc11a5 |
fix(podman): restore host gateway alias mediation (#3606)
* fix(podman): restore host gateway alias mediation Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(podman-e2e-tests): enable broader test podman e2e coverage Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(tests): make test more reliable Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(podman): fix macos linting error Signed-off-by: Gordon Sim <gsim@redhat.com> --------- Signed-off-by: Gordon Sim <gsim@redhat.com> |
||
|
Drew Newberry
|
0518bd4c83 |
fix(sandbox): preserve local sessions across host sleep (#3573)
* fix(cli): recover sandbox connect transport Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(auth): propagate non-expiring local sandbox sessions Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(cli): distinguish main exit from transport loss Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(cli): bound sandbox connect recovery Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
krishicks
|
0b351c4a9b |
fix(helm)!: reduce gateway Secret privileges (#3616)
* fix(driver-kubernetes-secrets)!: store provider credentials in one namespace The Kubernetes Secrets credential driver now stores every credential in its configured namespace in all workspace modes and rejects handles that reference any other namespace before contacting the Kubernetes API. The gateway reaches credential Secrets through the Role in that namespace; this allows removing the Secret rules from the ClusterRole. - Remove the workspace_mode, gateway_id, and allow_reference_namespace driver settings and stop rendering them from Helm. Configurations that set them fail at startup. Existing credential state is not migrated. - Add server.credentialDrivers.kubernetesSecrets.createNamespace to provision a dedicated credential namespace. The namespace is kept on uninstall, adopted by a reinstall of the same release, and left untouched when something else owns it. - Update the gateway config reference, Kubernetes setup docs, 0.1.0 upgrade guide, compute-runtime architecture, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> * fix(helm): reduce gateway Secret permissions Remove the gateway's Secret list permission in every workspace mode and grant source Secret reads through a Role in the sandbox namespace. Bootstrap Secret cleanup deletes Secrets by exact name instead of listing them. - Grant get on the copied client TLS and image-pull Secrets through a Role in the sandbox namespace. The ClusterRole keeps get and patch on those names for the ownership check and server-side apply into workspace namespaces. - Delete sandbox and supervisor bootstrap Secrets by exact name, derived from the runtime generation recorded on the Sandbox and, on restart, the target generation, tolerating 404. The generation annotation is cleared only after cleanup succeeds, and each bootstrap Secret has a Pod owner reference, so garbage collection removes any generation the driver does not name. - Drop Secret list from the ClusterRole and the shared-mode sandbox Role. - Extend the managed e2e RBAC checks to Secret list. - Update the Kubernetes setup and sandbox runtime docs, compute-runtime architecture, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> * fix(driver-kubernetes): stage workspace Secrets per runtime generation Every Secret the Kubernetes driver writes into a workspace namespace is now scoped to one sandbox runtime generation, immutable, and created with create only, so the gateway never reads, patches, or adopts an existing Secret there. This removes the gateway's cluster-wide get and patch on the copied client TLS and image-pull Secret names. - In managed mode, create an immutable copy of each configured image-pull Secret per generation, named os-pull-<id>-<generation>-<n> and owned by the generation's workload and supervisor Pods. Pods and the restarted Sandbox template reference those names, and generation cleanup deletes them by name. A Secret already holding a generation name fails the create. - Outside shared mode, stage the gateway client TLS material into the supervisor bootstrap Secret instead of copying the client TLS Secret into the workspace namespace. - Remove the fixed-name TLS and image-pull copies, the target ownership read, and the ClusterRole get and patch rule on the copied names. Source reads stay in the sandbox-namespace Role. - Update the managed e2e to expect generation image-pull Secrets and client TLS material in the supervisor bootstrap Secret, and to check that the gateway cannot read the copied names in workspace namespaces. - Update the gateway config and compute driver references, Kubernetes setup and sandbox runtime docs, compute-runtime architecture, driver README, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> * fix(helm)!: grant operator-mode Secret permissions through the workspace chart The operator-mode gateway ClusterRole grants no Secret permissions. The openshell-workspace chart Role, installed in each operator-managed namespace, grants the gateway create and delete on Secrets for sandbox runtime generations. Operator-managed namespaces require the workspace chart. - Fail the chart tests on any ClusterRole rule that includes Secrets in operator and shared modes. - Install the workspace chart when the operator e2e provisions a namespace, and assert that the gateway has no Secret permissions in a namespace without it. - Update the Kubernetes setup docs, 0.1.0 upgrade guide, compute-runtime architecture, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> --------- Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Piotr Mlocek
|
bed9e5eafc |
fix(supervisor): use better error message when sandbox connect is not available (#3572)
* fix(supervisor): explain unavailable main terminal attachments Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(supervisor): reset cursor after terminal attachment errors Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(supervisor): format read-only warnings for terminal clients Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs: drop terminal attachment documentation additions Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(supervisor): exit read-only viewers on Ctrl-C Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs: simplify read-only viewer Ctrl-C guidance Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> |