* fix(cli): backpressure sandbox uploads
Stream archive uploads over a bounded SSH subsystem whose credit follows extraction stdin progress. Preserve the existing input budget and tar destination semantics, validate completion, and clean up cancelled transfers. Add protocol, lifecycle, and large-upload regression coverage without changing russh.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(cli): preserve upload cancellation and errors
Keep cancellation active through upload setup and retry delays, and
terminate cancelled SSH probes. Preserve protocol failures through output
completion and report captured diagnostics on early extraction failure.
Remove redundant upload validation and extend the affected regressions.
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(cli): carry upload credit over SSH streams
Replace the framed upload protocol with SSH's own streams. The client
sends one JSON plan line, the raw archive and EOF. The supervisor's
stdout carries only cumulative totals of bytes written to extraction
stdin, extraction output goes to stderr, and the exit status reports
the result. Extraction starts inline through the ordinary exec path.
This removes the frame codec, SHA-256 digests, the CLI sha2 dependency
and the supervisor upload state machine while keeping the 1 MiB credit
window, the 4 MiB input guard, cancellation and error reporting. A
failed archive now stops SSH before closing stdin, so EOF never marks a
truncated archive complete.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(cli): isolate upload SSH connections and pin credit cadence
- Apply the forward multiplexing overrides to the upload ssh command, so
a user ControlMaster cannot carry an upload over another session's
master connection.
- Kill ssh before closing its stdin when a transfer fails locally, and
include ssh stderr in transport errors.
- Share ACK_INTERVAL_BYTES from core. Assert at compile time that CLI
chunks fit the credit left after an ack and that the window stays
below the supervisor's pending-input guard.
- Log the real reason when the supervisor rejects an upload.
- Test that a single-packet upload reports its final total.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(cli): assert upload ssh opens the subsystem without a TTY
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* feat(providers): support stable credential placeholders
Add credentials[].stable_placeholder so a running client keeps using an externally rotated static credential through the reference it already holds. Squashed replay of the earlier four commits onto upstream main 71c3cd957; re-derives the frozen schema fingerprints and adapts the provider environment revision test to the added capability argument.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(e2e): apply stable-placeholder supervisor image override
Use ManagedGateway's per-process supervisor image override so the fixture
CA reaches the supervisor when runtime image environment variables take
precedence over gateway TOML. Restore the wrapper's original image after
each test, following the provider-readiness fixture.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(server): update public RPC schema fingerprint after merging main
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(cli): preserve existing directories during single-file upload
Inspect remote destination types before choosing archive entry names.
Keep directory and source symlinks intact, recheck detected type changes,
and report the resulting path. Add real tar and live conformance coverage.
Closes#4175
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(cli): box upload future in sandbox_create
Boxing the upload future keeps the sandbox_create future under the clippy::large_futures threshold after sandbox_upload_planned began returning the uploaded path.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* feat(server): place supervisor sessions by consistent hash and hand them off on shutdown
Signed-off-by: divesh <dgude@nvidia.com>
* feat: route long-lived sandbox connections to the owning gateway replica
Signed-off-by: divesh <dgude@nvidia.com>
* feat(helm): add opt-in per-replica GRPCRoute routing
Signed-off-by: divesh <dgude@nvidia.com>
---------
Signed-off-by: divesh <dgude@nvidia.com>
`openshell forward service` minted an SSH session token before every
forwarded TCP connection and revoked it afterwards: two store commits per
connection. The token added nothing on that path. `ForwardTcp` already
authenticates the caller and authorizes it against the sandbox's workspace
on every stream before it looks at the token, the relay to the supervisor
is opened with the sandbox id and target only, and the token is never
forwarded, audited, or visible to the target service. The mechanism exists
for `openshell sandbox ssh`, where the process that opens the stream is an
ssh ProxyCommand holding nothing but the token. Reusing it per TCP
connection put a store write on the connect path and serialized concurrent
forwards on commit latency: #3494 measured the symptom, and #3543 made the
commits cheaper, but each one still holds SQLite's writer lock for an
fsync, so connection setup under a burst stayed linear in the number of
concurrent connections.
Let `target.tcp` streams omit `authorization_token`. The gateway admits
them on the already-authorized principal, counts them against the same
per-sandbox connection cap, and touches no store. `target.ssh` streams keep
requiring the token. A token supplied with a TCP target is still validated
and counted per token, so an older CLI against a new gateway is unchanged.
The CLI stops minting and revoking a session per forwarded connection;
against a gateway that predates this change it recognizes the
`authorization_token is required` rejection once and falls back to
per-connection tokens for the rest of that forward.
Tests cover token-less TCP admission and slot release, SSH targets still
rejected without a token, a supplied token still validated, and the
per-sandbox cap for token-less forwards. CLI integration tests run
`service_forward_tcp` against a mock gateway: token-less inits echo data
with no CreateSshSession or RevokeSshSession call, and a gateway that
rejects the empty token is detected once, after which every connection in
that forward mints and revokes its own token. Architecture and security
docs describe which targets carry a token, and the per-token connection
limit now reads 3, matching the gateway.
Signed-off-by: Jason T. Greene <jason.greene@redhat.com>
* fix(ssh): persist sandbox host identities
Store each sandbox's Ed25519 host key in the gateway credential store and
deliver it only to the supervisor. Preserve identity across restarts,
delete owned credentials with the sandbox, and expose the public SHA256
fingerprint through sandbox and SSH-session APIs and client SDKs.
Cover credential ownership, cancellation, deletion retries, client
compatibility, and pinned SSH connections through lifecycle transitions.
Closes#3835
Signed-off-by: Mike Nguyen <miken@nvidia.com>
* fix(compute): clean up failed sandbox SSH identity creation
Signed-off-by: Mike Nguyen <miken@nvidia.com>
* test(ssh): wait for sandbox deletion before name reuse
Signed-off-by: Mike Nguyen <miken@nvidia.com>
---------
Signed-off-by: Mike Nguyen <miken@nvidia.com>
* fix(sandbox): accept local connections natively on loopback-confined sockets
On kernels without SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV (RHEL 9 / RHCOS
5.14), the broker cannot safely write an accepted peer address into
workload memory, so accept/accept4 with a peer-address buffer failed with
EOPNOTSUPP. Static binaries and Go servers, which issue the raw syscall,
could not accept connections at all.
Move local acceptance into the kernel and replace per-accept inspection
with standing kernel confinement:
- Bind every broker-created TCP/UDP socket to the loopback device before
injection and verify the binding. Accepted sockets inherit it, so they
can neither receive routed ingress nor emit routed egress.
- Stop notifying accept/accept4 and remove the accept workers, the
SIGUSR2 accept-interrupt monitor, and the 64-accept ceiling.
- Continue getpeername natively for every descriptor except relayed
connections.
- Deny interface-selection socket options and MSG_FASTOPEN sends from
scalar syscall arguments, so confinement does not depend on the
capability state of the user namespace that owns the network namespace
and covers unregistered descriptors.
- Drop loopback-interface ingress on the non-loopback TCP control
listener before it listens, so a workload cannot reach it by
reconnecting a natively accepted socket.
- Mark descriptors above stdio close-on-exec in every workload pre_exec
path.
- Require an active confinement probe at qualification and carry it as
required authenticated audit evidence.
Document OpenShift 4.19 as the minimum release: RHCOS kernels for 4.16
through 4.18 are built without Landlock.
Closes#4058
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): close review gaps in native local accept confinement
- Reject a TCP boundary control listener on a loopback address,
including IPv4-mapped loopback. Production drivers bind the
unspecified address; a loopback listener would be reachable from
workload sockets the broker does not track.
- Extend the confinement probe to IPv6 and prove that an accepted
socket keeps its loopback binding after an AF_UNSPEC disconnect.
- Pin the accepted-peer contract with tests: a loopback-bound listener
admits only clients in the sandbox network namespace, and workload
sockets cannot bind a non-loopback source address.
- Correct the support matrix: accepted connections are bounded by
per-process descriptor limits, the runtime PID limit, and the sandbox
memory limit, not by a broker limit.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(sandbox): replace unsafe socket calls with socket2 and rustix
Socket confinement now uses socket2 for device binding and the ingress
filter, and rustix for interface lookup and AF_UNSPEC disconnect, so the
module contains no unsafe code. The control-listener filter is no longer
locked: no safe API exposes SO_LOCK_FILTER, and the listener descriptor
never leaves the trusted sandbox process, which marks every descriptor
above stdio close-on-exec before running workload code.
Tests added for native accept use rustix and socket2 instead of raw libc
calls. rustix issues accept4 and getpeername as raw syscalls, so the
direct-syscall test keeps its meaning. The close-on-exec sweep test runs
in a re-executed test process instead of a forked child.
The close_range(CLOSE_RANGE_CLOEXEC) syscall in the pre_exec hook remains
the only unsafe added by this branch; neither rustix nor nix wraps it.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): allowlist workload socket families
The workload seccomp filter denied only AF_PACKET, AF_BLUETOOTH, and
AF_VSOCK, and the broker continued socket() for every non-INET family.
Several protocol families, including AF_RXRPC, AF_SMC, and AF_KCM, carry
traffic over kernel-owned sockets that the broker never creates and that
are not bound to loopback.
Allow only AF_UNIX, AF_NETLINK (still limited to NETLINK_ROUTE), and the
brokered AF_INET/AF_INET6 families, and restrict socketpair(2) to
AF_UNIX. Both decisions use the scalar domain argument. The broker
independently refuses non-INET families other than AF_UNIX and
AF_NETLINK with EAFNOSUPPORT.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(sandbox): remove legacy read-only mode
After native local accept, only two broker paths still wrote into
workload memory: getpeername on relayed connections and the per-message
lengths of a first sendmmsg to the DNS relay. Legacy mode existed only to
refuse those writes on kernels without WAIT_KILLABLE_RECV, and the
getpeername refusal broke CPython TLS on RHEL 9.
The broker now never writes workload memory:
- getpeername is no longer mediated and reports the kernel peer on every
kernel; relayed connections report the loopback relay address.
- A first send to the DNS relay pins the broker's socket copy to the
relay and continues the syscall, so the kernel performs the send and
writes any per-message results.
Remove the listener mode, the task-memory write path and probe, and the
task_memory_write, cancellation, and task_memory_writes_disabled audit
evidence fields. WAIT_KILLABLE_RECV is still used when available and is
reported for diagnostics, but no mediation decision depends on it.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(sandbox): drop WAIT_KILLABLE_RECV and make handlers restart-safe
The broker no longer writes workload memory, so killable notification
waits only reduced how often a signal restarts a notified syscall. RHEL 9
kernels never had them, so the handlers must tolerate restarts anyway.
Install the listener without the flag on every kernel instead of
special-casing newer ones.
Handlers now check that the notification is still live immediately
before each side effect (bind, connect of the retained socket, listen,
and the first DNS send), and answer a restarted operation the broker
already completed as the kernel would: a repeated TCP connect returns
EISCONN, repeating the same UDP association succeeds, a repeat of a
completed bind succeeds, and a failed relay reports its errno.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): finish teardown only when every workload descendant has exited
Teardown waited only for registered process groups to disappear. A root
is unregistered once reaped, so a descendant that ignored SIGTERM could
outlive it while termination reported success and never sent SIGKILL,
violating the bounded-termination requirement.
The sandbox now becomes a child subreaper when it is not PID 1, so
orphaned descendants stay in its tree and are reaped. Termination waits
until no live descendant remains, and every scanned process is signalled
through a pidfd after confirming its start time, so a reused PID is never
signalled.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep a frozen workload from resuming itself
Freezing stops every workload process with SIGSTOP. tgkill and
rt_tgsigqueueinfo were not mediated, and mediated kill, rt_sigqueueinfo,
and tkill passed SIGCONT through, so a workload process that was not yet
stopped could resume the others while the supervisor recovered.
Mediate tgkill and rt_tgsigqueueinfo like tkill: refuse targets in the
sandbox thread group, report a thread outside the named group as
missing, and continue otherwise. The boundary marks the broker frozen
before stopping the workload and clears it after resuming; while frozen,
workload requests to send SIGCONT fail with EPERM.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): send mediated DNS datagrams from the broker socket
The native UDP send path re-ran the workload's own sendmsg, so
per-message ancillary data such as IP_PKTINFO rode along; only the
loopback destination contained a routing override.
The broker now reads the datagram and sends it from its retained,
relay-connected socket, which it builds without ancillary data, so a
per-message override cannot redirect the packet. It writes nothing back
into workload memory. A send carrying control data is refused with
EOPNOTSUPP rather than silently stripped.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep sandbox control variables out of the workload environment
The canonical process inherits the sandbox environment so the image's
own ENV reaches the workload, then removed only a denylist of credential
variables. Other variables in the reserved OPENSHELL_ namespace, such as
the serialized user environment and the log level, still reached the
workload.
Remove every inherited OPENSHELL_ variable before applying the declared
environment, and restore OPENSHELL_SANDBOX=1. The image's ordinary ENV
and declared variables are unaffected; declared variables cannot use the
reserved namespace.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep /proc read-only for GPU workloads
GPU mode granted read-write access to all of /proc so CUDA's cuInit could
write thread names to /proc/<pid>/task/<tid>/comm.
Keep /proc read-only. Open mediation now serves a writable open of the
caller's own thread comm file: the broker opens it and injects the
descriptor, with no syscall continued. The kernel accepts a comm write
only from the target's own thread group, so a substituted path or reused
thread ID cannot rename another process's thread. Any other /proc write
is left to Landlock, which denies it.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep the broker socket for connected UDP DNS sockets
Sending DNS datagrams from the broker's socket required the broker to
keep its copy, but the connect paths to the relay and to loopback still
released it. glibc connects the resolver socket and then sends A and
AAAA together with sendmmsg, which is mediated, so resolution failed.
Release the copy on those connect paths only for TCP. Add a regression
test for connect followed by sendmmsg that fails fast rather than
hanging.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep slow loopback connects from stalling mediation
The loopback connect path held the registry lock while polling a TCP
connect for up to five seconds on the single notification dispatcher.
A workload connecting to a busy local listener stalled every other
mediated syscall, including opens and signals.
Connect a duplicate of the retained socket without holding the lock. A
nonblocking socket gets the native EINPROGRESS and the kernel completes
the handshake on the shared socket; a blocking socket waits on a bounded
worker thread.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): contain network broker handler panics
A panic in a notification handler unwound the single broker thread while
the health flag still read healthy, so every later blocked workload
syscall hung until the sandbox was killed.
Run each dispatch under catch_unwind: a panicking handler fails only that
syscall with EIO and the broker keeps mediating. If the broker thread
ever exits, mark it unhealthy so dependent operations fail closed instead
of blocking.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): reject ancillary data on DNS sends instead of broker-sending
Sending mediated DNS datagrams from the broker's own socket required
keeping a broker handle for each DNS socket's whole life, which regressed
real name resolution through reclaim and resource accounting that the
mock-based unit tests did not exercise.
Revert to continuing the kernel send, but refuse a send that carries
ancillary control data (msg_controllen != 0) at read time, so a
per-message routing override such as IP_PKTINFO cannot ride a mediated
DNS send. A loopback destination contains an override that races the
check. Full broker-side UDP mediation is left to a separate change with
deployment e2e.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): address review findings across the hardening changes
Fixes:
- The blocking local-connect worker no longer toggles O_NONBLOCK on the
open file it shares with the workload; it waits with a plain connect.
- tgkill and rt_tgsigqueueinfo are notified only when they send SIGCONT,
so ordinary thread signals such as Go preemption stay in the kernel.
The frozen flag is re-read immediately before delivery.
- A shell redirect to the caller's own thread comm file works again;
O_CREAT and O_TRUNC are no-ops there and O_EXCL returns EEXIST.
- The teardown scan is a linear walk and fails closed when /proc cannot
be read.
Simplifications:
- Remove tests that need infrastructure outside the repo or prove
nothing: the topology harness tests, the static-server benchmark, the
direct-syscall accept test, the comm rename test without Landlock, a
serde-default test, and the test-only panic hook in setsockopt.
- Drop the unused Failed connect outcome, the redundant read-back after
binding to loopback, redundant OPENSHELL_SANDBOX settings, and stale or
duplicated comments; reuse the reserved environment prefix constant.
- Tighten the support matrix and OpenShift wording.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(cli): wait for input when piped exec stdin is nonblocking
Processes that inherit the same stdin share one open file description,
so any of them can make it nonblocking for all. `sandbox exec` then read
no input yet, got EAGAIN, and failed with "Resource temporarily
unavailable (os error 11)". Parallel e2e tests inherit the runner's
stdin, which made credential_gating fail intermittently.
The stdin reader now blocks in poll() until input or end of file arrives
instead of treating EAGAIN as fatal, and it leaves the shared descriptor's
flags alone. The e2e exec helper also stops inheriting the runner's
stdin, since its commands take no input.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): fix macOS lint and older-glibc test linking
Import HashSet only in the Linux-only process scan, and call gettid
through the raw syscall in a test, since the libc wrapper needs glibc
2.30 or newer.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(sandbox): clarify socket peer address reporting
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(server): separate image preparation and admission deadlines
Give sandbox image preparation its own deadline and start the admission
deadline after preparation finishes. Persist both phases across gateway
restarts and show recovery guidance for the phase that expired.
Preserve the current service authorization schema and regenerate the Go
bindings with the preparation timestamps.
Fixes#3952
Related to #3955
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(cli): simplify preparation timeout fallback selection
Use lazy Option fallbacks while preserving timeout messages and retained
sandbox behavior.
Signed-off-by: Shiju <shiju@nvidia.com>
* docs(server): clarify admission timer prerequisites
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(compute): enforce deadlines during initial sandbox create
Release stalled create operations after preparation expires and preserve
the timeout diagnosis across late driver results. Keep failed-create cleanup
bound to its original attempt so another replica can retry safely.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(server): satisfy deadline regression lints
Drop the create-error mutex guard before matching its cloned value and use
idiomatic iteration and timeout matching in the deadline fixtures.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(server): retain ownership of pending provisioning operations
Keep submitted create and start requests alive after caller cancellation,
monitor failure, or preparation timeout. Persist request ownership before
dispatch and retain staged uploads while the driver response is pending.
Require timeout cleanup to stop compute after driver settlement without
discarding an active cleanup claim. Fence late result handling against newer
operations, preserve failed-start recovery, and defer automatic restart while
another request owns the sandbox. Expose pending ownership in CLI JSON.
Add ordered multi-replica and cancellation regressions for the review findings.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(openshell): preserve tracing and accept ready create responses
Carry the request span into the detached provisioning worker so compute
driver calls remain attached to their parent trace after task handoff.
Update the compensation regression to require no backend DELETE when the
durable cleanup claim fails, matching the operation ownership requirement.
Accept the gateway's current Ready snapshot when a sandbox becomes ready
before CREATE returns. Do not require the client to observe an earlier
provisioning phase. Cover the Ready-only watch and command attachment.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(relay): close outbound relay stream when target closes first
Closes#3724
out_tx was cloned into the target-reading task, so the function's
own copy kept the outbound relay stream open until the client side
also ended. A target that closed a keep-alive connection never
reached the client as EOF, so reused connections hung forever.
Move out_tx into the target-reading task instead of cloning it, so
dropping it on target EOF ends the outbound stream right away.
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* fix(relay): keep client uploads after target half-close
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
---------
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* feat(cli): stream non-TTY exec input before EOF
Add --stream-stdin using the existing interactive exec RPC without a PTY. Preserve separate output streams and enforce the existing 4 MiB cumulative input cap while forwarding input.
Require explicit clean stdin EOF and drain the response through its final gRPC status. Cover held-open input, limits, cancellation, trailers, and default finite-input behavior with subprocess and live sandbox regressions.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(cli): distinguish exec cancellation from stdin EOF
Treat transport termination and response cancellation as separate test observations. Verify explicit stdin EOF through the shared frame writer and cover cancellation in the pinned Tonic decoder.
Signed-off-by: Shiju <shiju@nvidia.com>
* style(cli): use lazy optional stdin dispatch
Signed-off-by: Shiju <shiju@nvidia.com>
* test(cli): use imported duration in stdin EOF regression
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(policy): refresh pending proposals when the sandbox policy changes
Approving, removing, or undoing a rule, or updating the sandbox policy,
changes the inputs every other pending proposal was evaluated against.
Only proposals the new policy covered were reconciled; the rest kept
their old prover result and review token. The review surface
(GetDraftPolicy) therefore showed a stale evaluation, and the first
approval of the next proposal refreshed it and failed with
FAILED_PRECONDITION, so approving proposals one after another always
failed once.
Re-evaluate the remaining pending proposals at each policy change,
reusing the cached prover result unless the proposal's inputs changed.
Approval still rejects a review token that does not match the stored
evaluation, so a reviewer holding a pre-refresh evaluation must still
refetch it.
When a refresh does happen at approval time (inputs changed between
fetch and approve), the CLI now explains that the rule was re-evaluated
and how to review it, instead of printing the raw gRPC status.
Closes#3884
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
* fix(policy): make pending proposal refresh race-safe and bounded
Store refreshed evaluations with a compare-and-swap: the store re-reads
the proposal, refuses when its rule name, proposed rule, or review token
changed since the evaluation read it, copies only the evaluation fields
onto the stored record, and updates only if the payload is still the one
it read. A refresh can no longer revert a concurrent edit or observation,
and the edit path uses the same guard against a concurrent refresh.
Bound each refresh to the 32 newest pending proposals; the rest keep the
approval-time recheck, which still refuses a stale review token. Operator
decisions (approve, approve-all, remove, undo) refresh before responding.
UpdateConfig, which holds the gateway-wide sandbox sync guard, and
agent-driven auto-approval refresh in a background task instead, one per
sandbox with later changes coalesced into a single rerun.
Refs #3884
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
* docs(policy): describe proposal rechecks after approvals and approve-all
Explain that approving, removing, or undoing a rule rechecks the other
pending proposals so they can be approved one after another, when the
recheck is deferred or bounded, and what rule approve reports when a
proposal changed after it was listed. Show rule approve-all in Run Your
First Agent with its security-flag behavior.
Refs #3884
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
* fix(policy): refresh pending proposals after a full policy replacement
A full policy UpdateConfig (openshell policy set) re-reads the latest
revision after its atomic write, finds the revision it just committed,
and returns before reaching the pending-proposal refresh at the end of
the handler. Pending proposals kept their stale evaluation, so rule get
showed the old candidate and the next approval failed with the refresh
precondition. Schedule the background refresh right after the commit.
Refs #3884
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
---------
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
With a non-terminal stdin, sandbox exec read stdin to EOF before it sent
the exec request. A pipe that never closes (CI runners, supervisors, agent
harnesses) blocked the CLI forever in read(2) without the gateway ever
seeing the request, and a slow producer delayed the command until EOF.
Collect piped stdin on a detached reader thread for at most 200 ms. Input
that reaches EOF within that window still travels in the single request
that older gateways need. If the pipe is still open, start the command
through the streaming RPC and forward the collected prefix plus the rest of
stdin as it arrives, closing remote stdin at EOF. The 4 MiB cap covers the
prefix and the streamed remainder together.
Closes#3993
Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com>
* fix(mcp): explain revision-scoped policy and rejections
Explain the selected-revision method set in profile output and policy docs.
Distinguish protocol and policy rejection causes and give a next step while
preserving authorization, response statuses, error codes and YAML keys.
Cover CLI serialization, revision selection, exact extension rules, deny
precedence and rejection before forwarding with focused regressions.
Signed-off-by: Shiju <shiju@nvidia.com>
* docs(mcp): correct HTTP cancellation revision support
Limit notifications/cancelled to the three 2025 revisions in the core
method matrix. State that MCP 2026-07-28 HTTP cancellation closes the
response stream, matching the runtime rejection and sessionless docs.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
Reuse character-safe truncation for policy history errors so a multibyte character cannot panic the table renderer.
Distinguish unavailable provider-profile YAML from an absent profile, display a bounded diagnostic, and preserve strict serialization and redacted object navigation.
Cover the actual CLI renderer and TUI display/navigation paths, including invalid and absent profiles, Unicode input, and redacted errors.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(api): emit warning on WatchSandbox broadcast lag instead of terminating
Broadcast lag on the status, log, and platform receivers was converted to a RESOURCE_EXHAUSTED status that terminated the whole watch stream. Lag is recoverable: the receiver resumes at the oldest surviving message. Emit a SandboxStreamWarning and continue streaming instead; keep terminating on Closed. Add helpers and unit tests covering the warning payload and receiver recovery after lag.
Partially addresses #3055 (cursor/resume follow up separately).
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* refactor(server): group per-sandbox log bus state and stamp sequence numbers
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* feat(proto): add resume cursor fields to sandbox watch API
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* feat(server): stamp watch cursors from a shared per-sandbox sequence
Allocate cursors from a single SeqAllocator shared by the log and
platform event buses, so a sandbox's merged watch stream carries
unique, strictly increasing cursors. A single resume_after_cursor can
then unambiguously locate a client's position across both sources.
Rewrite both publish paths to allocate the sequence, stamp
event.cursor, send, and append to the tail under one lock. This
removes the previous get_mut().expect() TOCTOU race where a concurrent
remove() between the two lock sections could panic.
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* feat(server): serve WatchSandbox resume from cursor with gap detection
Add tail_after() to the log and platform event buses, returning every
buffered event newer than a client's resume cursor. Each PerSandbox now
tracks last_trimmed_seq (the highest seq it has evicted) so a resume is
reported as an unrecoverable ResumeGap only when this bus dropped an
event the client still needs.
Judging gaps by evictions, not by the tail's oldest seq, is required
under the shared cursor space: each bus's tail is non-contiguous in the
global sequence because the other bus owns the missing seqs, so
comparing against tail.front() would flag false gaps.
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* feat(server): resume WatchSandbox from cursor across log and platform buses
Wire resume_after_cursor into the watch producer. On a non-zero cursor,
replay events strictly after it from both the log and platform buses,
merge by shared cursor, and emit in order before entering the live loop.
A trimmed range on either bus is an unrecoverable gap and terminates the
stream with OUT_OF_RANGE carrying the requested and earliest-available
cursors, distinct from recoverable lag which warns and continues.
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* test(server): cover WatchSandbox cursor resume paths
Add handler-level tests for the resumable watch stream: replay strictly
after the client cursor, merge log and platform events in shared-cursor
order, suppress duplicates when resuming at the latest cursor, and
terminate with OUT_OF_RANGE when the requested cursor has been trimmed.
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* docs(api): document WatchSandbox loss-awareness and resume
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(server): deliver watch events once and harden cursor teardown
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* feat(sdk): add loss-aware resumable watch_logs to Rust SDK client
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(server): keep watch cursors monotonic across teardown and restart
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(server): merge live watch sources by cursor before emission
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(api): bind watch cursors to a cursor space and merge tail sources
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(server): revalidate the watch cursor space after collecting replay
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* test(server): update the public RPC schema fingerprint for the string cursor
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(server): hold watch events above the publication watermark and emit the watch lag warning before its batch
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* test(server): synchronize the watch live-order test with the end of initialization
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(sdk): use canonical sandbox name in watch_logs
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* test(sdk): guard canonical-name addressing in watch_logs
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(server): fix public rpc schema
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(api): reconcile watch resume rebase
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* fix(server): bound interactive relay cleanup
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
---------
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* feat(sandbox): default to official Alpine sandbox image
default_sandbox_image() now returns docker.io/library/alpine:3.22, a generic
version-qualified official image, so a fresh install no longer depends on the
community sandbox image catalog. All compute drivers (docker, podman,
kubernetes, vm) inherit this fallback.
Part of #3116.
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
* feat(deploy): default deployment configs to the official Alpine sandbox image
Update the shared gateway default_image, Helm chart values, the standalone
Kubernetes manifest, and the dev gateway task scripts to use
docker.io/library/alpine:3.22 instead of the community base image, consistent
with default_sandbox_image(). GPU e2e image-build base is left unchanged (CUDA
needs a glibc base).
Part of #3116.
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
* feat(driver): default to numeric non-root identity for USER-less images
With the default sandbox image now Alpine, images that declare no OCI USER
must start instead of being rejected. When the image declares no USER and
the policy requests none, the Podman and Docker drivers now supply a numeric
non-root identity (DEFAULT_SANDBOX_UID/GID = 1000) instead of rejecting,
matching the numeric-identity behavior of the Kubernetes and VM drivers. The
supervisor's resolved-identity path runs the sandbox as a synthesized
non-root account without the account existing in the image. Images that
declare a USER keep the OCI resolution path unchanged.
Part of #3116.
Signed-off-by: Akram <akram.benaissi@gmail.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(conformance): use Alpine workload image
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* refactor(policy): drop community image /app path from default policy
The restrictive default policy granted read-only access to /app, a directory
that only existed in the community base image. A generic Alpine default has no
/app, so remove it. Landlock best-effort already ignores absent paths; this
just stops advertising a community-specific layout in the default.
Part of #3116.
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
* docs(config): document Alpine default images
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(podman): report early sandbox termination
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(podman): initialize rootless workspace ownership
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(sandbox): qualify NVIDIA Ubuntu default
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(podman): initialize rootful default workspace
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(sftp): add native sandbox adapter
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sftp): gate runtime helper support to Linux
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sftp): support standard OpenSSH file operations
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sftp): harden rename and special file handling
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(runtime): remove community image dependencies
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* test(e2e): build provider readiness tool fixture
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(e2e): use a dedicated Noble fixture for Docker tests
Signed-off-by: Evan Lezar <elezar@nvidia.com>
---------
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
* feat(kubernetes): support HA gateway rebalancing
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* perf(server): cache peer connections, tokens, and owner lookups
Every forwarded relay rebuilt its setup from scratch: an owner lookup, a
blocking read of the peer token, a TLS connect to the owning replica, and
a TokenReview plus Pod GET on the receiving side. Sandbox service routing
does this per HTTP request, so the apiserver calls scaled with traffic.
Cache all of it on ServerState:
- peer channels pooled per endpoint, so relays multiplex over one
connection instead of redialing
- peer tokens keyed by SHA-256, expiring at min(ttl, token exp) so a hit
cannot accept an expired token
- owner records for 3s against a 45s ownership TTL, still freshness
checked before use
Entries are evicted when a relay fails. Also raise HTTP/2
max_concurrent_streams to 1024, since pooling funnels every relay between
two replicas onto one connection and hyper's default of 200 sits below
the 256 pending-relay budget.
Signed-off-by: divesh <dgude@nvidia.com>
* perf(server): pool upstream connections for sandbox services
Each HTTP request to a sandbox service opened its own supervisor relay,
paying a new TCP connection and HTTP/1 handshake every time. Worse, it
counted against the 32 in-flight relay cap, so a service handling more
than 32 concurrent requests failed outright.
Pool idle upstreams per endpoint and port, up to 8 each for 15s. Reuse is
safe because the pool only returns a connection hyper reports as ready,
and HTTP/1 cannot start a request until the previous body has drained.
Upgrades are never pooled since they take the connection over, and a
failed send evicts that endpoint. Pruning is bounded per key, with the
full sweep limited to once per 30s.
Signed-off-by: divesh <dgude@nvidia.com>
* fix(server): address HA gateway review findings (#3449)
- Let a gateway own supervisor sessions without a peer endpoint. Requiring
one whenever the store is PostgreSQL broke every single-instance
PostgreSQL deployment, because no sandbox supervisor could connect.
A cross-replica request to an owner that advertises no endpoint now fails
immediately naming the cause, instead of retrying until the wait timeout.
- Close a supervisor session on heartbeat only when another replica owns it,
or after renewals fail for the ownership TTL. A database error no longer
drops every session heartbeating during an outage.
- Clamp owner record ages at zero so a skewed or corrupt stored timestamp
cannot produce a negative age.
- Bound the cross-object advisory lock with a lock timeout, so a stuck holder
fails instead of blocking every mutation in the fleet.
- Refuse to start when a peer endpoint is configured on a multi-replica
backend but peer authentication is unavailable, and warn when a
multi-replica backend has no peer endpoint at all.
- Reject a plaintext peer endpoint when the gateway serves TLS.
- Skip the sandbox watch poller on single-replica backends, where the local
update bus already sees every write.
- Rate-limit the peer owner cache sweep so an insert no longer scans the
whole map under the lock.
- Retry GET and HEAD on a pooled upstream the sandbox closed, instead of
returning 502, and drop an emptied endpoint from the pool right away.
- Document the gateway peer environment variables and the post-rollout
ownership skew operators should expect.
Signed-off-by: divesh <dgude@nvidia.com>
* fix(server): harden HA supervisor ownership
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: divesh <dgude@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: divesh <dgude@nvidia.com>
Co-authored-by: Divesh Chowdary <47188680+FrostGod@users.noreply.github.com>
* feat(cli): promote profile commands to top level
Add profile discovery and management commands with shared handlers for
the existing provider entry points. List a flat catalog across scopes
and follow continuation tokens through full and short pages.
Describe metadata, credentials, endpoints, TLS inspection, and MCP access
settings while preserving complete JSON/YAML definitions. Cover parser
equivalence, scope forwarding, pagination, and inspection settings with
focused unit and compiled-CLI integration tests.
Update docs, public skills, examples, and E2E command invocations.
Refs #2588
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(cli): remove redundant workspace selector qualification
Use the imported WorkspaceSelector in the provider integration helper so
the target passes Clippy with warnings denied.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(policy): reject unknown endpoint security modes
Closes#3046
Validate TLS, enforcement, and access values across policy and provider profile ingress, and prevent runtime parsing from falling back to audit for unknown enforcement values.
Signed-off-by: Krzysztof Malczuk <kmalczuk@redhat.com>
* fix(policy)!: use enums for endpoint security modes
Replace the public TLS, enforcement, and access strings with protobuf enums and carry the typed values through policy composition, provider profiles, drivers, and runtime conversion.
Preserve the documented YAML spellings, reject unknown and invalid numeric enum values consistently, and update generated Go bindings, SDK conversions, tests, and policy documentation.
Signed-off-by: Krzysztof Malczuk <kmalczuk@redhat.com>
---------
Signed-off-by: Krzysztof Malczuk <kmalczuk@redhat.com>
Drew Newberry