mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-29 07:23:25 +08:00
dev
72
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew Newberry
|
d22394370d |
fix(helm): allow shared supervisor SCC-assigned IDs (#4785)
Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
32f026442e |
feat(sandbox): initiate supervisor connections across compute drivers (#4656)
* feat(sandbox): support sandbox-initiated supervisor connections Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): consolidate control transport setup Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(sandbox): remove prototype-specific unit tests Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(kubernetes): migrate legacy control flow on restart Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: remove prototype changes from published docs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): adapt reverse control to published backend protocol Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(sandbox): initiate supervisor connections across compute drivers Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): stage reverse control socket for the supervisor identity Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): keep runtime transports behind the backend Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): mount the runtime control volume read-only The sandbox runtime only dials the supervisor-owned control socket, so it does not need write access to the control volume. Mount it read-only in the runtime container and stop staging the control directory through the runtime bundle; the supervisor stager already initializes it through its own writable mount. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(kubernetes): separate legacy and runtime-initiated control services Keep os-boundary selecting the workload and add os-supervisor for runtime-initiated connections. Bind the mode marker to its runtime generation so a restart by an older gateway does not inherit stale mode metadata. Preserve the legacy Service across creation and restart, validate the active endpoint, and drop the Service patch permission. Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Oliver Calder
|
d4ad1bbd03 |
fix(snap): only enable gateway user service for one user (#4615)
* fix(snap): only enable gateway user service for one user Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * docs(snap): correct snap docs for the gateway user service Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): add documentation of global to per-user enablement migration Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Simon Scatton
|
b1cfc91720 |
ci(release): promote qualified prereleases to stable (#4611)
* ci(release): promote qualified prereleases to stable Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(release): preserve qualified images during stable publication Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(release): run shared release jobs with Nix Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(release): recheck latest before publishing shared channels Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
Piotr Mlocekanddivesh
|
604b9f66f6 |
feat(config): deliver gateway configuration over the supervisor session (#4320)
Supervisors poll the gateway for configuration every 10 seconds. This PR adds an opt-in push mode (config_delivery_mode = "push"): supervisors still start by polling, and once their ConnectSupervisor session is accepted, the gateway delivers every later change over it and supervisors apply it and report the result. poll stays the default, and older supervisors and gateways keep polling. --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> |
||
|
Drew Newberry
|
77929d9c14 |
feat(sandbox): publish and activate sandbox backend protocol (#3898)
* feat(isolation): publish backend protocol contracts independently of runtime Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(isolation): expose trusted launch and registry data contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): name the required capability sandbox Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): align protocol and compute capability names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): finalize public protocol contract names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(isolation)!: activate backend contracts in sandbox runtime and driver selection Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): simplify runtime and fix backend example Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(examples): organize WASI backend in Rust Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): defer protocol conformance fixtures Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(examples): add supervised V8 backend demo Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): stream V8 guest output with default policy Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): handle V8 rejections and roll back failed launches Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): align V8 crypto dependencies with CI AWS-LC Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(sandbox): represent Sandbox Protocol operations in protobuf Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): replace backend traits with RPC clients Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(proto): remove legacy isolation protocol reservations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(proto): clarify Sandbox Protocol names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(isolation): clarify authentication bootstrap and supervisor APIs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): simplify policy enforcement confirmation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): rotate launch generations and stabilize retry sessions Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): remove redundant backend adapters and TCP flags Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): repair Docker restarts and macOS lint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): simplify RPC contracts and backend registration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): deliver TCP denials before closing the RPC Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Emilien Macchi
|
b8526eb1d9 |
fix(server): install the metrics recorder before startup reconciliation (#4524)
The endpoint-status startup sweep ran before the gateway installed its Prometheus recorder, so the lock waits, timeouts, errors and hold times it recorded went to the no-op recorder and only its warnings reached the log. The sweep retries lock timeouts for up to three lock waits before startup fails, and a slow sweep that still succeeds is what operators want to see in the lock metrics. When metrics are enabled, the recorder is now installed before startup reconciliation, and the metrics listener still binds with the other listeners. A new pod's first scrape therefore already holds the sweep's samples, so they show in the raw counter and histogram count values, not in rate() or increase(). A sweep that fails still exits before any listener binds, so its timeouts remain log-only. The startup lock-timeout test now waits for the sweep's timeout counter instead of a fixed sleep and checks its timeout and wait samples, and the debug skill says which startup timeouts reach the metrics and how to read them. Part of #4307 Closes #4523 Signed-off-by: Emilien Macchi <emacchi@redhat.com> |
||
|
Florent BENOIT
|
a4c848adbd |
feat(vm): add host bind mount support via virtiofs (libkrun) (#3463)
* feat(vm): add host bind mount support via virtiofs (libkrun) Introduce host directory bind mounts for libkrun VM sandboxes using the virtiofs4 API with simplified permission semantics. The gateway accepts `enable_bind_mounts` in the `[openshell.drivers.vm]` TOML table and forwards it to the VM driver subprocess. Mounts are specified per-sandbox in driver_config.mounts[] with source, target, optional read_only (default: read-only), and an optional `type` that must be `bind`, so Docker and Podman bind entries (without `selinux_label`) parse unchanged. At sandbox validation the driver rejects mounts unless bind mounts are enabled and resource admission is disabled. It rejects GPU and other QEMU launches, sources with control characters, and more mounts than libkrun has interrupt lines for (32 on aarch64, 3 on x86_64), and checks targets against reserved VM-internal directories and each other (no nested targets). The QEMU check runs again once the launch backend is resolved. Host source directories are checked at provisioning, before image preparation, so a briefly missing source fails provisioning visibly instead of denying recovery of a persisted sandbox. Missing, inaccessible, symlinked, non-directory, and unreadable sources are rejected: libkrun does not follow a final symlink, and on macOS it panics during guest boot when it cannot open the share root. The driver generates virtiofs tags and always writes a guest-side manifest into the overlay disk, together with the init drop-ins, so an image-supplied manifest can never drive guest mounts. The launcher parses its tab-separated --vm-mount arguments from the right, so a source can never shift the tag or mode, and the driver logs each shared directory at launch. Guest init mounts the shares after every /sandbox ownership fixup and the init drop-ins, so recursive chowns never reach host files. It refuses targets that traverse symlinks in the image, hands newly created mount point parents under /sandbox to the sandbox user, and skips an empty manifest. krun_add_virtiofs4 is resolved optionally, so older libkrun runtimes still boot sandboxes without mounts. The kernel config lists CONFIG_FUSE_FS and CONFIG_VIRTIO_FS explicitly; the pinned libkrunfw base config already enables both. A vm_bind_mount e2e test checks a read-write share round-trips data with the host, accepts a new file, directory, and nested file that the host then sees, and that a read-only share rejects writes. libkrun on rootless Linux refuses file creation by a guest user whose UID/GID differ from its own, so the VM e2e gateway runs the sandbox as the gateway user there. It enables bind mounts like the Docker and Podman e2e gateways. The runtimes docs cover the mount limit, the symlink restriction, the guest-kernel exposure, and the rootless Linux UID/GID note. The debug-openshell-cluster skill lists VM bind mount checks. Signed-off-by: Florent Benoit <fbenoit@redhat.com> * fix(nix): refresh vm-runtime hashes to the libkrun v1.19.4 runtime The Nix vm-runtime package fetches the rolling vm-runtime release with fixed hashes pinned on 2026-09-16. The release was rebuilt on 2026-09-21 after the libkrun v1.19.4 / libkrunfw v5.6.1 bump (#3451), but the hashes were not updated, so CI kept substituting the 2026-09-15 runtime (libkrun v1.17.4) from the binary cache. That libkrun lacks krun_add_virtiofs4, which VM bind mounts need. Point the hashes at the current release assets. Signed-off-by: Florent Benoit <fbenoit@redhat.com> --------- Signed-off-by: Florent Benoit <fbenoit@redhat.com> |
||
|
Drew Newberry
|
b2069a267c |
feat(docker): support host and bridge supervisor networking (#4016)
* feat(docker): select supervisor networking for ECI workloads Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(docker): use automatic networking in e2e Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(docker): rely on existing e2e networking coverage Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): shorten supervisor networking configuration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): let local gateway discover supervisor endpoint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(docker): make supervisor networking operator-selected Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(docker): resolve host gateway in the driver Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): simplify host networking guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): trim configuration guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): discover host gateway once at driver startup Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Yuedong Wu
|
44c07dab2a |
feat(helm): make gateway PVC size and StorageClass configurable (#3854)
Signed-off-by: Yuedong Wu <dwcn22@outlook.com> |
||
|
Polite_realism
|
10241ab47b |
fix(podman): trust an operator-supplied additional CA for direct egress (#3996)
* fix(podman): trust an operator-supplied additional CA for direct egress Since v0.1.2 the Podman supervisor runs as its own container, separate from the workload image, so it can no longer inherit a private CA baked into the workload's system trust store. Policy-allowed, inspected HTTPS requests to destinations signed by that CA fail upstream TLS establishment, and the only workaround is rebuilding the supervisor image with the CA baked in. Add a Podman-only additional_ca_bundle setting, independent of the existing proxy_ca_bundle (which requires https_proxy), that bind-mounts an operator PEM into the supervisor container and folds it into the upstream trust store alongside the system CA bundle. Validated on the gateway host at sandbox-create time, matching the existing corporate proxy CA bundle's validation, so a misconfigured path or certificate-free bundle fails closed immediately rather than producing a failed sandbox once the supervisor container starts. Scoped to Podman only, reusing the existing additive-bundle mechanism without new gateway.toml schema, proto fields, or cross-driver plumbing -- a narrower alternative to the in-flight cross-driver implementation in #3292, which remains the intended long-term direction. Fixes #3781. Signed-off-by: politerealism <burdcat17@gmail.com> * fix(podman): keep additional CA mount on the supervisor build_isolation_specs() keeps only trusted_mount() destinations on the supervisor, and the additional CA path was not in that list. The bind mount stayed on the workload while the supervisor received --additional-ca-bundle for a file it did not have, so startup failed. Add the path to trusted_mount() and cover the final supervisor and workload specs with a test. Correct the docs and a code comment to say the bundle is validated when the driver starts, and add Podman troubleshooting guidance to the debug-openshell-cluster skill. Signed-off-by: politerealism <burdcat17@gmail.com> --------- Signed-off-by: politerealism <burdcat17@gmail.com> |
||
|
ShijuandJohn Myers
|
29a2b8e7d9 |
feat(providers): report bounded token-grant failures (#4427)
* feat(providers): report bounded token-grant failures Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): align policy metadata and TLS lockfile Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): fix trusted grant recovery fixture Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): enable denial API in recovery fixture Signed-off-by: Shiju <shiju@nvidia.com> * refactor(providers): simplify token-grant failure reporting Remove duplication and dead paths from the token-grant failure reporting without changing classification, the workload-visible 502 response, or the wire format. - Build the gRPC failure envelope with tonic-types, store the typed failure in EndpointFailure, and share one request-ID rule and one endpoint-status authority check. - Merge the gateway exchange client wrapper, drop forwarding validators, and name the request field when a reported cause is rejected. - Inject fallback credentials only from live provider state and delete the unread per-connection credential copy. Keep one 502 writer per path and restore the base token-cache handling. - Check the token endpoint after the cache lookup, still before any credential acquisition. - Document 429 and 5xx classification as implemented, and drop tests that repeat retained assertions through the same code path. Signed-off-by: Shiju <shiju@nvidia.com> * refactor(providers): trim token-grant failure tests and supervisor API Cut the token-grant failure tests to one test per distinct behavior across core, gateway, supervisor, CLI and Go, and replace the 794-line closure E2E with status assertions in the existing route-recovery E2E: the refused grant appears in sandbox status with the response's request ID, code, stage and upstream status, and the matching grant's later success clears it. - Pin the live provider snapshot with one helper used by inspected and fallback forwarding, and inject selector-matched grants through inject_for_selectors, which borrows the pinned context. Move the test-only wrapper into test_support. - Drop the 35-second wrappers around the gateway grant RPC. The shared channel bounds the connection attempt and the gateway bounds its token-service calls. - Log the token service's OAuth error and error_description, without control characters and limited to 256 characters each, with the request ID in the gateway or supervisor log. Status and the workload response still carry only the bounded cause. - Keep one check that an untrusted token-service certificate cannot assert a cause. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): restore test import and features lockfile The injection tests named EndpointObservationSender through the parent module, which no longer imports it. The features suite lockfile kept the tonic TLS dependency that the removed E2E fixture added. Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): fix body-bound fixture width and table type Pass the response-size width as a named argument; an implicitly captured constant panics at runtime. Name the mutation type in the validation table for Clippy. Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): pad exact-bound body without a format width MAX_TOKEN_RESPONSE_BYTES exceeds u16::MAX, the largest format width, so the padded fixture panicked. Pad with repeat_n as before. Signed-off-by: Shiju <shiju@nvidia.com> * test(server): update public RPC schema hash after rebase Main added the ListProvidersRequest label selector. The merged public schema hash comes from the Branch Checks storage_proto test output. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): keep token-service error text out of grant logs A token service can echo a submitted credential in its OAuth error or error_description. Log only the classified failure code, HTTP status and request ID, and cover the canary echo in a dedicated test binary. Signed-off-by: Shiju <shiju@nvidia.com> * docs(providers): tighten token grant failure documentation Signed-off-by: Shiju <shiju@nvidia.com> * docs(providers): condense token grant failure guidance Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
John T. Myers
|
909f336b62 |
fix(kubernetes): validate runtime control mount paths (#4522)
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
krishicks
|
07fd2386ce |
feat(server): export gateway OCSF events as OTLP log records (#4500)
The gateway only persisted its OCSF events to the JSONL file, so collectors had to tail that file. Adding `[openshell.gateway.otlp.ocsf]` now also exports each gateway OCSF event as an OTLP log record, using the same resource identity as gateway traces. `schema_version` optionally downgrades the exported JSON to OCSF 1.1 or 1.3. OTLP endpoints are now configured per signal, so OCSF events can go to a collector that feeds a SIEM while spans go elsewhere. The existing `[openshell.gateway.otlp]` `endpoint` is the default for every signal and on its own still enables trace export. `[otlp.traces]` overrides the trace endpoint, and `[otlp.ocsf]` takes its own `endpoint` or falls back to the default. Each record carries the OCSF JSON as its body, the event time as its timestamp, a severity mapped from `severity_id`, and `ocsf.*` and `openshell.sandbox.id` attributes for filtering without parsing the body. Records flow through a queue bounded by count and bytes to the SDK OTLP log exporter in batches bounded the same way. Failed exports retry with backoff; records lost to a full queue, an oversized body, or exhausted retries are counted in metrics and reported to the collector in an `openshell.ocsf.gap` record on the next successful export. The exporter lives in `openshell-otel` so other processes can share it. Helm deployments set the `traces` and `ocsf` tables through `gatewayConfig`, and the local development gateway tasks enable OCSF export. Closes #4499 Refs #2892 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Yuedong Wu
|
3f66c26af1 |
fix(helm): render Restricted-compatible security contexts (#3340)
Signed-off-by: Yuedong Wu <dwcn22@outlook.com> |
||
|
|
e30c938bb1 |
feat(server): scope mutation locks and bound lock waits (#4147)
* test(server): add a local PostgreSQL test runner
The openshell-server tests that need a real PostgreSQL server are
ignored by default and had no shared way to run. The mutation-replay
test reads its own OPENSHELL_REPLAY_TEST_DATABASE_URL, so every
contributor had to provision a database by hand, and the advisory-lock
tests that follow in this series need the same setup.
Add mise run test:rust:postgres. It runs every ignored postgres_* test
in openshell-server against OPENSHELL_TEST_POSTGRES_URL, or starts a
disposable PostgreSQL container with Docker or Podman (CONTAINER_ENGINE
selects one) on the image pinned by the Kubernetes e2e fixture and
removes it on exit. Tests run one at a time because advisory locks are
database-wide. The runner always points the legacy replay variable at
the selected database, so an inherited URL cannot send that test to a
different server. test:postgres-runner checks that selection with a fake
cargo and needs no database or container engine. CI does not run the
PostgreSQL tests; TESTING.md documents the task.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): hold the mutation guard while deleting a provider
DeleteProvider checked that no sandbox referenced the provider and then
deleted the record without holding the sandbox mutation guard. Sandbox
create and provider attach take that guard while they write provider
references, so one of them could add a reference after the attached
sandbox check passed, and the delete then removed a provider that a
sandbox spec still named.
Take the guard before the attached sandbox check, as provider create and
update already do, so the check and the delete run against a stable
set of sandbox references, and reject an empty name after authorization
but before the guard. A new test holds the guard, attaches the provider
while the delete waits, and asserts that the delete fails with
FailedPrecondition and leaves the provider in place.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): release staged credentials when a refresh fails early
A provider credential refresh stages the minted values under new
credential handles before it commits the provider update, and deletes
those handles when validation or persistence fails. Two earlier returns
skipped that cleanup. The minted expiry was converted to a protobuf
timestamp only after staging, so an expiry outside the timestamp range
failed the refresh and left the staged values in the credential driver.
A failure to acquire the sandbox mutation guard returned the same way.
Convert the expiry before staging anything, so a bad value fails before
any handle exists, and delete the staged handles when the guard cannot
be acquired. A new test refreshes a stored credential with an expiry of
i64::MAX and asserts that the call fails, the provider keeps its
original handles, and the credential driver holds the same number of
values as before. The guard failure path has no test here, because the
SQLite guard used by unit tests cannot fail.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): acquire mutation locks on a dedicated PostgreSQL lock pool
The cross-object mutation guard took its advisory-lock session from the
10-connection data pool and closed it on every release, so each guarded
mutation opened a new database connection, and lock holders competed
with their own critical sections for data connections. A lock wait that
hit the 10 second lock_timeout surfaced as INTERNAL.
Add a lazy, dedicated 4-connection lock pool and a new
persistence::mutation_lock module for the lock key, mode and set types.
Store::acquire_distributed_mutation_guard now takes a lock set and a
deadline, and each lock statement sets lock_timeout to the time left
before that deadline, so a cancelled acquisition never waits past it on
the server. Returned connections are scrubbed with
pg_advisory_unlock_all() and reused, and each return is bounded by 5
seconds so a stalled session cannot keep a pool permit.
A lock statement that PostgreSQL ends with SQLSTATE 55P03 fails with a
new LockTimeout error, and so does a wait for a lock connection during
which every one was checked out at some point, or that started with less
than a second left, too little to open one. The API returns it as
UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT and a 1 second retry
delay, because the mutation lock was not acquired, so the request's
guarded writes did not run. Only the guard acquisition classifies 55P03;
any other 55P03 stays a database error. A lock connection that
PostgreSQL does not open by the deadline despite at least a second to do
so (refused, out of connection slots, starting up, or not answering, all
of which SQLx retries silently) is a database error, not a lock timeout.
Mutations still take the legacy global key exclusively, so old and new
replicas keep excluding each other during a rolling upgrade. Each
PostgreSQL-backed gateway pod can now open up to 14 database
connections (10 data plus 4 lock) instead of 10, and the chart's
autoscaling.maxReplicas comment says what its default of 4 needs. The
SSH host identity lock stays on a data connection, because sandbox
creation takes it while holding its mutation guard and a second
lock-pool connection there could exhaust the pool. The provider
credential refresh lock stays there too, so a refresh still progresses
while every lock connection is in use.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): partition mutation locks by workspace and sandbox
Every cross-object mutation, and every lifecycle and supervisor update,
took one process-wide mutex and, on PostgreSQL, one fleet-wide advisory
lock. Unrelated sandboxes and providers therefore serialized across all
replicas, and one slow critical section delayed every other mutation in
the fleet.
Replace that lock with hierarchical intention locks on a global key, one
key per workspace and one key per sandbox, each held shared or
exclusive. Global settings and policy writers and platform-scope profile
writers hold the global key exclusively. Provider and workspace-scoped
profile writers, including DeleteProvider and credential refresh, hold
their workspace key exclusively. Sandbox writers and supervisor reports
hold their sandbox key exclusively under shared global and workspace
keys. Lifecycle, driver-watch and reconcile paths take only
process-local keys and keep relying on compare-and-swap across replicas.
Workspace and sandbox keys are domain-separated SHA-256 prefixes. Each
guard takes its local keys and then its advisory locks in ascending
order under one 10 second deadline, and no path nests guards. The
settings mutex is removed, since the global and sandbox keys cover it.
Sandbox settings are keyed by sandbox name, which a sandbox key no
longer covers once another sandbox reuses the name, so a settings
update returns NOT_FOUND unless its sandbox still holds the name after
the read, and ABORTED if the settings row was recreated before the
write.
The global key keeps its legacy value, so older replicas, which hold it
exclusively for every mutation, still exclude new ones during a rolling
upgrade. Guards now also give up after 10 seconds on SQLite, and
provider refresh reports a guard timeout as UNAVAILABLE instead of
INTERNAL. A supervisor report for a missing sandbox returns NOT_FOUND.
Startup endpoint-status reconciliation still holds the global key for
its whole scan.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): reconcile endpoint status on startup one sandbox at a time
Startup endpoint-status reconciliation held the global mutation key
exclusively for its whole scan. On PostgreSQL that blocked every guarded
mutation on every replica until the starting gateway had walked all
sandboxes, and the scan first waited behind every in-flight sandbox
mutation. It also paged with an offset, so sandboxes deleted mid-scan
shifted rows between pages, and a write that raced the scan (a Conflict
from a lifecycle update on another replica, or a sandbox deleted after
it was listed) failed gateway startup.
Walk the sandboxes with keyset paging instead, and reset each candidate
under its own sandbox-scoped guard, up to four at a time to match the
lock pool. A candidate with a fresh shared owner is skipped before any
guard is taken. Under the guard the reconciliation re-checks the owner,
re-reads the sandbox, skips it when it is gone or no longer carries
endpoint status, and writes against the version it just read. A Conflict
or a database error at write time, which is how a row deleted after the
re-read surfaces, is retried with a fresh read, at most five attempts
per sandbox. A lock timeout on the guard is retried the same way, so a
holder of the sandbox, its workspace or the global key that releases
within the retry window cannot stop a replica from starting. No
lock-timeout retry starts once the scan has run for three lock timeouts,
so a scan stuck behind a long holder fails within about 40 seconds,
inside the chart's 60-second startup probe, and logs the sandbox it gave
up on. Store::list_all_messages loses its only caller and is removed.
Reconciliation stays fail-closed: a holder that outlasts the retry
window, an owner lookup failure, or a sandbox that keeps changing still
fails startup, so stale endpoint success is never served. User-visible
change: a restarting gateway no longer stalls mutations of unrelated
sandboxes and providers on its peers while it reconciles.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): skip the endpoint-status lock when a session was replaced
When a supervisor session ends, the gateway resets that sandbox's tool
server endpoint evidence unless a replacement session already owns
endpoint observation. The replacement check ran only after the reset
had read the sandbox and taken its sandbox mutation guard, so a reset
whose supervisor had already reconnected still queued behind other
mutations of that sandbox, and on PostgreSQL it used a lock-pool
connection, only to find it had nothing to do.
Check for a replacement first: a live local session, or a fresh owner
record on a peer. When one exists, return without reading the sandbox
or taking the guard. Otherwise take the guard and repeat the same check
under it, because a replacement's pre-acknowledgement reset runs under
the same sandbox key and resetting after it would wipe the
replacement's fresh evidence. Both checks share one helper, so they
cannot drift apart. When no replacement exists, the reset costs one
extra owner-row read.
This helps most when a replica notices a dead stream late, after the
supervisor has already reconnected elsewhere (a network partition or a
keepalive expiry).
Deferring to a replacement leaves the old session's evidence to the
replacement's pre-acknowledgement reset, which can time out on its
bounded mutation lock wait. That reset's failure path removed the
replacement and released its owner record but scheduled no
invalidation, so the old session's success evidence could outlive both
sessions. An old session that ended after its replacement registered
already deferred this way, and the unguarded check now makes every
disconnect that finds the replacement defer too. When the reset fails
while the replacement is still current, also schedule the retried
invalidation that follows a session end. That failure path keeps the
lifecycle demotion from abandon_session_setup, but runs it in the
spawned task before the retry and only while this session is still
current, so a timed-out reset returns promptly.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* test(server): cover staged credential cleanup on a lock timeout
A provider credential refresh deletes its staged handles when it cannot
acquire the provider mutation guard, but that path had no test: the
SQLite guard used by unit tests could not fail. Mutation locks now
acquire under a deadline that tests can shorten, so the timeout path
can be exercised directly.
The new test shortens the lock deadline to 50 ms, holds the workspace
mutation guard, and refreshes a provider that already stores an AWS
access key. The refresh returns UNAVAILABLE with the
MUTATION_LOCK_TIMEOUT reason, the provider keeps its original handle,
which still resolves to the old key, and the credential driver holds
the same number of values as before the refresh.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): export mutation lock wait and timeout metrics
A handler mutation guard can now wait up to 10 seconds for its keys and
then fail with UNAVAILABLE, but nothing showed operators how long guards
waited or how often they gave up, short of reading warning logs on each
replica.
Export openshell_server_mutation_lock_wait_seconds, a histogram of the
time one guard took to acquire all of its keys (local registry plus
PostgreSQL advisory locks), recorded on success, and
openshell_server_mutation_lock_timeouts_total, a counter of guard
acquisitions that timed out. Both carry a bounded scope label: global,
workspace or sandbox. The platform scope, whose workspace is empty,
counts as global. The histogram uses the same 1 ms to 15 s buckets as
the relay and peer histograms, and the timeout counter starts at 0 for
every scope, so rate() works on an idle replica. The timeout warning
now also logs the scope. A guard that fails for another reason, such as
a lock connection PostgreSQL did not open, is not counted as a timeout
and logs a "mutation lock acquisition failed" warning instead.
Lifecycle, driver-watch and reconcile paths take only process-local
locks and never time out, so they are not recorded.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* test(server): add PostgreSQL mutation lock integration tests
The scoped mutation locks were only unit-tested on SQLite, where the
database layer is a no-op. Add ignored postgres_* tests that run the
advisory-lock layer against a real PostgreSQL through mise run
test:rust:postgres, each in its own disposable schema (TestSchema).
persistence/mutation_lock_pg_tests.rs uses two stores on one database
as two replicas. It covers disjoint sandbox scopes holding at once, a
workspace key blocking only sandboxes in that workspace, the global key
blocking every scope, and exclusion in both directions against a raw
holder of the legacy global key, whose 55P03 stays a database error
outside a guard acquisition. It checks that timed-out and cancelled
acquisitions release the keys they already took, that a released lock
connection returns to the pool without residual locks, that a release
stalled on the network closes its session and the pool recovers, and
that the lock pool never grows past its size. It also checks that a
full lock pool times out as lock contention, while a lock connection
that PostgreSQL never answers fails as a database error.
compute/mutation_guard.rs runs the random scope mix across two
PostgreSQL runtimes and checks that a by-id guard waits for a workspace
writer on the other replica. The provider refresh lock test moves onto
the runner too, and checks that a refresh proceeds while every lock
connection is held.
A bench_postgres_* capacity check measures a paced burst of 500
reconnects 12 ms apart into one replica and fails when the p99 lock
wait reaches a fifth of the lock timeout. Machine load moves those
waits, so mise run test:rust:postgres skips it and mise run
test:rust:postgres:bench runs it alone, through a new
OPENSHELL_TEST_POSTGRES_PREFIX setting in the runner.
The tests add test-only helpers: the lock pool's size and idle count
and the backend pid of a held lock session. Nothing changes outside
tests and the test runner.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* ci: run the PostgreSQL-backed server tests
Branch Checks now runs the postgres_* tests through the same runner
developers use, on an x86_64 Linux runner with Docker. They cover lock
scope exclusion, interop with replicas still on the old global key,
cancellation, lock-pool bounds and connection failures, which no other
CI lane checks directly. The load-sensitive bench_postgres_* capacity
checks stay local.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* docs: document scoped mutation locks and lock-pool sizing
In the High Availability guide, size PostgreSQL for 14 connections per
gateway pod, 10 for data and 4 for the new lock pool, so three or more
replicas now exceed what the PostgreSQL default leaves, and keep
headroom for lock sessions that cancelled requests leave behind.
Explain which changes lock a sandbox, a workspace, or the whole fleet,
and that a lock wait over 10 seconds returns a retryable UNAVAILABLE
with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to
raise max_connections to the 14-per-pod size before upgrading, because
the rollout already runs new pods at that count, and warns that older
replicas keep serializing every mutation until the rollout finishes.
Monitor Capacity gains the lock metrics and their queries, and names
slow driver, credential, middleware, or profile source calls as another
cause of lock waits.
List the lock wait histogram and timeout counter, with their scope
label, on the Gateway Metrics page. The counter also counts background
work and leaves out lock connections that PostgreSQL does not open
although at least a second of the wait remained.
In the cluster debugging skill, replace the single advisory-lock
paragraph with the lock levels and pool, what each timeout detail
means, how a lock connection that PostgreSQL does not open shows up,
when terminating a lock backend is safe, a pg_locks query that shows
holders and waiters, and how to spot the global key. The per-pod greps
also read the timeout counter, and new rows cover lock timeouts and
lock connections that PostgreSQL does not open.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): check mutation lock ordering in debug builds
The mutation lock ordering rules were prose only, so a nested guard on
a different key passed every test and could only deadlock under
contention. Debug builds now register each held guard with the Tokio
task (or the thread, outside a task) that acquired it, and panic before
waiting when a local key, a blocking lifecycle gate or the SSH identity
lock would break rules 1, 4 or 5. Startup endpoint-status
reconciliation and the tests that hold a guard while driving guarded
code run under lock_order::branch(), which marks deliberate concurrency
within one task. Release builds compile the check out.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): do not hold gateway shutdown for the endpoint-status retry
When a replacement supervisor session failed its endpoint-status reset,
the spawned cleanup task kept the session lifetime guard through
retry_endpoint_status_after_supervisor_disconnect. That retry only ends
once the reset is durable, so a gateway shutdown during a database or
lock outage waited the full 10 s and reported ownership cleanup as
incomplete. The guard now covers only the lifecycle demotion, as in
finish_supervisor_session, and a SQLite test checks that shutdown
completes while the retry is still blocked.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* refactor(server): make the platform profile lock scope explicit
Platform-scope provider profile writers (import, update, delete) pass an
empty workspace on purpose and need X(global), because they validate
sandboxes in every workspace. MutationScope::profiles() now names that
escalation and picks Global for the platform scope, instead of relying
on MutationScope::Workspace("") to mean it. An empty workspace name now
trips a debug assertion, and release builds keep X(global) as a
fail-safe that over-locks rather than under-locks.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): amortize local mutation lock table sweeps
LocalMutationLocks swept its whole table on every key lookup, so a
burst of N concurrent acquisitions on one replica cost O(N^2) under a
blocking mutex. The table is now swept only when a new key grows it
past the larger of 64 and twice the live entries left by the last
sweep, which makes acquisition amortized O(1). Released entries,
including those of cancelled waiters, stay bounded at about twice the
peak live set. The lifecycle gate registry keeps its per-call sweep.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): make the database pool ceiling configurable
Both stores hardcoded their pool ceiling: 10 connections for Postgres, 5
for on-disk SQLite. One pool is shared by every database-backed RPC, so
that number is also the gateway's ceiling on concurrent database work.
Once every connection is checked out, callers queue on acquire and sqlx
logs "time to acquire exceeded slow threshold"; sandbox creates then time
out and are retried, which adds load rather than shedding it.
The right ceiling is deployment-specific — it depends on how many gateway
replicas share the database and what max_connections the server itself
allows — so it cannot be a single number baked into the binary.
Expose it on the same config surface as the rest of the gateway's
settings: --db-max-connections, OPENSHELL_DB_MAX_CONNECTIONS, and the
TOML key database_max_connections, resolved in that precedence order.
Helm deployments set the key through gatewayConfig. Omitting it keeps
each backend's previous value, so existing deployments do not move.
Values below 1 are rejected rather than silently replaced by the default:
a zero pool would block every acquire, and quietly ignoring a typo would
reproduce the ceiling the operator is trying to lift. An in-memory SQLite
database stays pinned to one connection, since the database lives inside
that connection.
Refs: #2561
Signed-off-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com>
(cherry picked from commit
|
||
|
Yuedong Wu
|
3495c8e92a |
fix(kubernetes): refresh sandbox runtime image on start (#4376)
Signed-off-by: Yuedong Wu <dwcn22@outlook.com> |
||
|
Oliver Calder
|
16c8ca5c43 |
fix(docs): use lowercase "snap" as it is not a proper noun (#4387)
Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Seth Jennings
|
4c1b16a4a1 |
feat(server): add operator-only provider credential retrieval (#4357)
* feat(server): add operator-only provider credential retrieval Authenticate operators exclusively through verified direct gateway mTLS. Export selected runtime credentials with freshness validation and all-or-nothing delivery. Coordinate refresh across callers and replicas, preserve rotated grants across RPC cancellation, and fence credential delivery against reconfiguration. Add generated bindings, security and regression tests, and operator documentation. Signed-off-by: Seth Jennings <sjenning@redhat.com> * feat(sdk): add curated operator credential retrieval Expose provider credential clients in Python and TypeScript with explicit workspace and key selection, freshness options, optional expiration, and preserved gateway errors. Reuse existing mTLS transports without introducing retries or secret caching. Add generated-wire and error-mapping tests, SDK package validation, and operator usage documentation. Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(server): honor recovery state for queued automatic refreshes Signed-off-by: Seth Jennings <sjenning@redhat.com> --------- Signed-off-by: Seth Jennings <sjenning@redhat.com> |
||
|
Oliver Calder
|
82bc756545 |
feat(snap): run gateway as a user service by default (#4099)
* feat(snap): run gateway as a user service by default Replace the existing system `gateway` service with separate `user-gateway` and `system-gateway` services. For new installs, only the `user-gateway` service is enabled. For existing installs, only the `system-gateway` service is enabled. Existing installs continue to have a one-time migration which removes legacy insecure configurations. It is up to `install.sh` or users to manually copy mTLS credentials from the root-owned `$SNAP_COMMON/tls` to the invoking user's OpenShell snap directory. This is explained in the snap description in `snapcraft.yaml`, visible in the Snap Store listing and via `snap info openshell`. Service enablement is now managed by a new snap configuration option named `gateway-mode`, so users can switch from one mode to another via e.g. `sudo snap set openshell gateway-mode=user`. The `install` and `post-refresh` hooks select which service to start by setting this mode. If the `gateway-mode` is already set, then we know the one-time migration has already taken place. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): simplify the snap gateway wrapper Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): use `snap start --user` when running as non-root Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): ensure install.sh waits for the correct docker daemon Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * feat(snap): document how to safely switch between gateway modes New installs run the gateway as a user service by default, while upgrades of existing installs from prior to the user gateway continue to use the existing system gateway. The user and system gateway are fundamentally different services and there is no way to migrate state between them. However, it may be desirable for a user to run the gateway as a system service after a fresh install. To support this, this commit adds documentation for how to safely transition from the user gateway mode to the system gateway mode, and vice versa. Also, clarifies when it is necessary to manually copy TLS credentials from the system gateway state directory to the user's home directory: when upgrading a pre-TLS snap installation, or when switching from user to system gateway mode. When upgrading an existing install with an HTTPS gateway running as a system service, copying credentials again is not necessary. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Daneyon Hansen
|
40e01424c1 |
fix(helm): improve GatewayClass overrides (#4345)
* fix(helm): improve GatewayClass overrides Give chart-created Gateways a distinct default name and allow GRPCRoutes to select a listener on an existing Gateway. Fixes: #2469 Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io> * fix(e2e): resolve rendered Gateway name Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io> * fix(helm): preserve Gateway name defaults Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io> --------- Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io> |
||
|
Gaizka MenendezandTaylor Mutch
|
fe3942f2da |
feat(helm): migrate gateway configuration to gatewayConfig (#3384)
* feat(helm): add generic gateway TOML serializer Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(architecture): define Helm gateway configuration boundary Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): define default gateway configuration map Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): render gateway config from values map Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): cover generic gateway TOML rendering Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): protect gateway config secret boundary Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(helm): classify legacy gateway configuration values Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): derive dual-use resources from gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): migrate gateway config scenarios Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): derive credential resources from gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): migrate gateway config overlays Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): remove gateway configuration shadow values Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): align runtime config with resources Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(helm): add gateway config migration guide Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): validate rendered gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): cover config resource coherence Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): make Kubernetes E2E deployable Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(e2e): pass host aliases through gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(e2e): reference gateway config host alias Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): enforce gateway resource ownership Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(e2e): cover chart host gateway input Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): address gateway config review findings Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): repair RFC 0012 migration integration Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(e2e): align Kubernetes parity with RFC 0012 Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): complete gateway config migration Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * chore(helm): add SPDX header to TOML template * fix(helm): preserve legacy gateway configuration aliases Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): complete legacy gateway config compatibility Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): support gateway config arrays of tables * fix(helm): preserve sandbox identity aliases * fix(helm): address gateway config review findings Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): preserve gateway config migration settings Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * test(helm): validate extension arrays in CI values Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(helm): restore unconditional SPDX headers Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(helm): restore route SPDX header placement Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): simplify configuration and migration examples Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): remove redundant flag explanations Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): move chart configuration guidance to Kubernetes setup Signed-off-by: Taylor Mutch <taylormutch@gmail.com> --------- Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> Signed-off-by: Taylor Mutch <taylormutch@gmail.com> Co-authored-by: Taylor Mutch <taylormutch@gmail.com> |
||
|
alangou
|
81f712e64e |
fix(drivers): validate vendor-agnostic CDI device names (#4306)
Reject host paths and malformed CDI selectors for Docker and Podman using shared qualified-name validation. Preserve the generic GPU request checks for drivers using other identifier formats, including VM PCI addresses. Validate explicit and resolved CDI selectors, cover vendor-agnostic names and rejected paths, and update the related documentation. Signed-off-by: Adrien Langou <alangou@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
John T. Myers
|
67adcf1a3c |
feat(ocsf): emit full JSON records to supervisor stderr (#4323)
* feat(ocsf): emit full JSON records to supervisor stderr Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(ocsf): preserve console record boundaries Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Matthew Grossman
|
1fff8b97b9 |
feat(podman): honor OCI image working directories (#3982)
* fix(supervisor): report rejected OCI working directories as WorkspaceValidationFailed The RFC 0012 split dropped the supervisor exit status that drivers map to the WorkspaceValidationFailed condition. An image WORKDIR the sandbox identity cannot use then surfaced as ControlSupervisorStartFailed on Docker and as a signal kill on Podman. The supervisor now exits with the reserved status when the sandbox rejects the image working directory. Docker maps that supervisor exit during readiness and monitoring, and Podman maps the supervisor companion's exit instead of the workload's. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * feat(podman): honor OCI image working directories Podman sandboxes now use a custom OCI image WORKDIR as the workspace and as the working directory for agent commands, matching Docker. Empty, /, and /sandbox values keep the managed /sandbox workspace volume. A custom workspace stays in the image's container filesystem: no workspace volume, no archive upload, and no root setup step. Resolve the image ID, user, environment, and working directory from one pinned inspection, validate the workdir with the shared OCI rules, and reject Podman control-path overlaps and image volumes or driver mounts that cover it. The runtime starts from / and passes the resolved path to agent commands with --workdir. Add podman_oci_identity to the Podman CI test list. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(drivers): preserve custom workspace failures across supervisor cleanup Check the Podman custom workspace rather than the absent managed volume in the OCI identity E2E. Classify Bollard wait errors by exit code and record Docker workspace failures before removing the supervisor so readiness retains the specific failure reason. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(docker): use sandbox ID for recorded startup failures Keep the workload container ID for Docker inspection and use the sandbox ID to retrieve the monitor failure. Test with distinct identifiers so the lookup cannot accidentally pass with a container ID. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(podman): keep WORKDIR change focused on workspace behavior Remove the new cross-driver workspace failure classification path and its monitor/readiness workaround. Preserve Podman WORKDIR resolution, mount safety, and workspace rejection without requiring a distinct condition reason. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(podman): simplify inspected image metadata and test fixtures Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * code review Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
Divesh
|
834b79a8c2 |
feat(server): place supervisor sessions by consistent hash (#3661)
* feat(server): place supervisor sessions by consistent hash and hand them off on shutdown Signed-off-by: divesh <dgude@nvidia.com> * feat: route long-lived sandbox connections to the owning gateway replica Signed-off-by: divesh <dgude@nvidia.com> * feat(helm): add opt-in per-replica GRPCRoute routing Signed-off-by: divesh <dgude@nvidia.com> --------- Signed-off-by: divesh <dgude@nvidia.com> |
||
|
Mike Nguyen
|
9fd41e6bfd |
fix(ssh): persist sandbox host identities (#4094)
* fix(ssh): persist sandbox host identities Store each sandbox's Ed25519 host key in the gateway credential store and deliver it only to the supervisor. Preserve identity across restarts, delete owned credentials with the sandbox, and expose the public SHA256 fingerprint through sandbox and SSH-session APIs and client SDKs. Cover credential ownership, cancellation, deletion retries, client compatibility, and pinned SSH connections through lifecycle transitions. Closes #3835 Signed-off-by: Mike Nguyen <miken@nvidia.com> * fix(compute): clean up failed sandbox SSH identity creation Signed-off-by: Mike Nguyen <miken@nvidia.com> * test(ssh): wait for sandbox deletion before name reuse Signed-off-by: Mike Nguyen <miken@nvidia.com> --------- Signed-off-by: Mike Nguyen <miken@nvidia.com> |
||
|
alangou
|
33869a1df5 |
fix(images): update gateway base and allow supervisor base override (#4236)
Signed-off-by: Adrien Langou <alangou@nvidia.com> |
||
|
Emilien Macchianddivesh
|
8b3cc3fdc0 |
feat(server): add gateway capacity metrics and optional HPA (#3978)
* feat(server): add gateway capacity metrics and optional HPA Expose per-replica supervisor sessions, pending relay capacity, relay rejections and claim latency, and outbound peer request outcomes and latency. Use bounded labels and Prometheus histograms for the new latency metrics while preserving existing summary metrics. Add an optional Helm HPA with external-database and resource validation, conservative scale-down defaults, and support for custom metrics. Keep certificate hook pods outside gateway workload selectors. Document per-pod scraping, scaling limits, upgrade behavior, and PostgreSQL connection sizing. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): unify routing metrics and clarify replica capacity Combine local relay setup and outbound peer requests in one counter, labeled by operation, route, target, outcome, and gRPC status. Preserve peer latency metrics. Rename the rejection reason from global_capacity to replica_capacity to reflect the per-replica relay budget. Keep capacity limits unchanged. Update tests and documentation. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): align routed request metric labels and outcomes Count a local relay as successful only when its supervisor claims it, the same event that answers a peer relay on the owner, and rename the outcomes to local_error and remote_error so they say where an attempt failed. Label the peer latency histogram by operation, like the routed request counter, and rename the target label to relay_kind so it does not read as the Prometheus scrape target. Rename RelayCapacity.global to per_replica, drop the per-sandbox relay capacity gauge, which no per-sandbox series can pair with, describe the latency histogram as peer-only, and restore the note that unavailable spikes are expected during rollouts. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> --------- Signed-off-by: Emilien Macchi <emacchi@redhat.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> |
||
|
Shiju
|
1c123a4542 |
feat(gateway): check VM host tools during config preflight (#4037)
* feat(gateway): validate VM filesystem tools during preflight Check required local VM tools through config preflight and share executable resolution with VM image operations. Report selected paths and actionable errors without creating gateway or sandbox state. Bound probe output and execution time, and clean up probe descendants on interruption. Preserve pure static validation and skip local tool checks for remote driver endpoints and unrelated drivers. Fixes #3951 Related to #3955 Signed-off-by: Shiju <shiju@nvidia.com> * fix(gateway): stabilize filesystem preflight checks Combine identical filesystem-tool error arms and normalize rendered diagnostics in command tests so terminal wrapping preserves assertions. Describe driver TLS validation without depending on removed guest fields. Signed-off-by: Shiju <shiju@nvidia.com> * test(gateway): serialize preflight fixture paths as TOML Keep temporary paths quoted and escaped through the TOML serializer instead of relying on Rust Debug formatting. Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
Matthew Grossman
|
5d6b3b8120 |
fix(kubernetes): serialize lifecycle cleanup with sandbox restart (#4078)
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
krishicks
|
8091f66877 |
feat(sandbox): write agent output to the container log (#4005)
Since #2726 the canonical main process's stdout and stderr are captured in pipes that feed only the in-memory replay buffer used by sandbox connect. Agent output therefore never reaches the container's own stdout and stderr, so it is missing from kubectl logs, docker logs, and podman logs and from anything that collects container logs. Before #2726 the entrypoint inherited the container's descriptors and its output appeared there. Copy the main process's output to the launcher's stdout and stderr in addition to the replay buffer, restoring the earlier behavior: - Output is copied byte for byte to the matching stream from a forwarder thread per stream, after it is published to the replay buffer. When the container runtime falls behind on a stream, that stream's reader waits instead of dropping output, so backpressure reaches the agent as it did with inherited descriptors, while the other stream and attachments keep receiving output. - Before the main process's exit is published, the output readers finish and queued output is drained to the container log, so an agent's final lines are not lost at shutdown. A 30 second deadline covers both; when it expires, readers waiting on the container log are released and drain the pipes into the replay buffer only, so a stalled container log cannot block exit reporting. - PTY-mode processes are not copied. The terminal stream carries escape sequences and echoed input, and terminal commands never reached the container log before #2726. - Exec, SSH, and SFTP sessions are not copied. Launcher log lines keep their existing format and remain in the container's stderr. They are written as whole lines, and a newline is inserted first when the agent left stderr mid-line, so launcher and agent lines do not merge. The Docker and VM drivers appended the tail of the workload's output to failure messages: Docker the workload container's log, and the VM driver the guest console, which carries the launcher's stdout and stderr. Those messages land in the sandbox's Ready condition and in platform events that the gateway republishes to the sandbox event stream. With agent output in that log, those messages would carry arbitrary agent output, including anything sensitive the agent prints, into gateway status and events. The supervisor starts its health endpoint only after the agent starts, so every Docker failure path could include agent output, and the VM driver reports one whenever the VM or host supervisor exits. Forward only the supervisor's log tail, matching the Podman driver, which reads the workload log solely to match fixed launcher markers and never forwards raw workload output. The workload's output remains available through docker logs and the VM's rootfs-console.log. Document where main process output appears in the logging docs and the cluster debugging skill. Closes #3928 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Drew Newberry
|
021400be8a |
refactor(auth): separate sandbox identity from TLS (#3110)
* refactor(auth): separate sandbox identity from TLS Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(auth): clarify gateway mTLS behavior Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(auth): include workspace scope in TLS authorization checks Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): bound service auth sandbox names for large PIDs Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
krishicksandEvan Lezar
|
a875add234 |
feat(server): write gateway OCSF events to JSONL (#3264)
* feat(server): write gateway OCSF events to JSONL Previously, gateway security activity was available only in diagnostic output, and events not associated with a sandbox, such as TLS certificate reloads, had no independent structured record. Now, configuring `openshell.gateway.ocsf_log` writes every gateway-produced OCSF record to a bounded JSONL destination independently of `RUST_LOG`. The destination supports daily or disabled rotation, retention limits, queue bounds, and optional schema downgrade to OCSF 1.1 or 1.3. Additionally, existing gateway emitters (TLS reloads, service routing, and policy approval and auto-approval audits) emit structured events, so they reach the JSONL destination, console shorthand, and the affected sandbox's log stream. Records identify the gateway by its configured name in `device.uid` and `device.name`, shared across replicas, with `device.hostname` identifying the replica and `device.os` the gateway's operating system. Metrics and warnings expose known best-effort losses. Refs #2762 Signed-off-by: Kris Hicks <khicks@nvidia.com> * fix(mxc): attribute ETW events to gateway Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Kris Hicks <khicks@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> Co-authored-by: Evan Lezar <elezar@nvidia.com> |
||
|
Philippe MartinandJohn Myers
|
9cb72baa2e |
feat(docker): support corporate proxy CA bundles (#3549)
* feat(docker): support corporate proxy CA bundles Closes #3545 Validate and stage operator-owned proxy CA bundles for Docker supervisors, add corporate proxy E2E coverage, and document the trust contract. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(docker): validate proxy config on startup Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): use the E2E workload image for proxy tests Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): generate strict corporate proxy certificates Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): surface intercepted TLS fixture errors Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): drain buffered TLS proxy data Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): relay intercepted HTTP deterministically Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Philippe Martin <phmartin@redhat.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Drew Newberry
|
acbac9cb79 |
feat(sandbox): add main restart policy (#2798)
* feat(sandbox): add main restart policy Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): harden policy-driven restarts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): address restart review feedback Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> * fix(sandbox): port restart policy to current runtime Signed-off-by: Drew Newberry <anewberry@nvidia.com> * perf(sandbox): restart promptly after terminal delivery Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> |
||
|
Drew Newberry
|
a67567e583 |
fix(snap): require mTLS for the snap gateway (#3726)
* fix(snap): require mTLS for the snap gateway Replace the installer opt-in with an authenticated snap gateway. The wrapper no longer forces plaintext, so the gateway serves TLS from the bundle it already generates in $SNAP_COMMON/tls. The install hook writes a config that enables mTLS user auth instead of unauthenticated access, and a new post-refresh hook migrates the exact legacy default on existing installs. install.sh waits for the gateway, detects whether it serves TLS, copies the client bundle into the target user's snap state directory, and registers the gateway over HTTPS. Older plaintext snap revisions still register over HTTP with a warning. The release canary asserts mTLS auth and HTTPS registration. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): pass config preflight and detect the mTLS gateway reliably An explicit [openshell.gateway.mtls_auth] table fails config preflight, which validates mTLS auth before the local TLS bundle supplies the client CA. Write a default that pins the Docker driver instead; with the wrapper's TLS bundle the gateway requires client certificates and enables mTLS user auth automatically, as the native packages do. The mTLS gateway rejects TLS handshakes without a client certificate, and it still answers plaintext loopback HTTP for sandbox service routing, so the installer could misdetect it as a legacy plaintext gateway. Probe HTTPS with the root-owned client bundle, and treat a gateway as legacy only when a plaintext gRPC Health call succeeds. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * chore(snap): simplify install hook comment Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): migrate insecure gateway configs on refresh Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(snap): simplify mTLS detection and config migration Detect the mTLS snap from the installed revision's post-refresh hook instead of probing plaintext gRPC, and drop the scheme global. Remove the installer's pre-hook config fallback, which is dead now that every channel ships the install hook and which wrote the insecure default. Give the install hook a single write path with a simple backup name, and shorten the manual client certificate steps. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): stop keeping a copy of replaced insecure configs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(install): make the snap an opt-in install method Stop selecting the OpenShell snap just because the snap command exists. Linux installs default to the Debian or RPM package; OPENSHELL_INSTALL_METHOD=snap (or deb, rpm) selects the package explicitly. Hosts that already have the OpenShell snap keep refreshing it rather than gaining a second gateway on the same port. The release canary and snap repro script opt in explicitly. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): let the gateway auto-detect its compute driver Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): restart the gateway after refresh Published revisions use refresh-mode: endure, and snapd honors the old revision's setting during a refresh, so the plaintext gateway kept running with the migrated config unused until a manual restart. Restart the gateway from the post-refresh hook so the mTLS config takes effect immediately. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(snap): drop refresh notes from the snap description Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(snap): trim snap refresh notes from installation docs Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
d4f5034d7f |
fix(gateway)!: make the WebSocket tunnel opt-in (#3727)
The /_ws_tunnel endpoint pipes a WebSocket into the full gRPC service. On a plaintext loopback gateway any web page could open it, since browsers do not apply CORS to WebSocket upgrades. Mount the tunnel only when enable_websocket_tunnel is set (config file, --enable-websocket-tunnel, or OPENSHELL_ENABLE_WEBSOCKET_TUNNEL; server.enableWebsocketTunnel in Helm). BREAKING CHANGE: gateways behind an authenticating edge proxy must set enable_websocket_tunnel = true for CLI edge-tunnel connections. Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
9244868056 |
docs: refresh architecture and agent guides (#3705)
* docs: refresh architecture and agent guides Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: describe updated security architecture neutrally Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: highlight new isolation primitives Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(sandboxes): clarify how to disconnect Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: align architecture and guides with current navigation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(extensibility): streamline extension authentication guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
52cb8ecee7 |
fix(kubernetes): remove NetworkPolicy acknowledgement (#3677)
Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
grs
|
8369bc11a5 |
fix(podman): restore host gateway alias mediation (#3606)
* fix(podman): restore host gateway alias mediation Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(podman-e2e-tests): enable broader test podman e2e coverage Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(tests): make test more reliable Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(podman): fix macos linting error Signed-off-by: Gordon Sim <gsim@redhat.com> --------- Signed-off-by: Gordon Sim <gsim@redhat.com> |
||
|
krishicks
|
0b351c4a9b |
fix(helm)!: reduce gateway Secret privileges (#3616)
* fix(driver-kubernetes-secrets)!: store provider credentials in one namespace The Kubernetes Secrets credential driver now stores every credential in its configured namespace in all workspace modes and rejects handles that reference any other namespace before contacting the Kubernetes API. The gateway reaches credential Secrets through the Role in that namespace; this allows removing the Secret rules from the ClusterRole. - Remove the workspace_mode, gateway_id, and allow_reference_namespace driver settings and stop rendering them from Helm. Configurations that set them fail at startup. Existing credential state is not migrated. - Add server.credentialDrivers.kubernetesSecrets.createNamespace to provision a dedicated credential namespace. The namespace is kept on uninstall, adopted by a reinstall of the same release, and left untouched when something else owns it. - Update the gateway config reference, Kubernetes setup docs, 0.1.0 upgrade guide, compute-runtime architecture, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> * fix(helm): reduce gateway Secret permissions Remove the gateway's Secret list permission in every workspace mode and grant source Secret reads through a Role in the sandbox namespace. Bootstrap Secret cleanup deletes Secrets by exact name instead of listing them. - Grant get on the copied client TLS and image-pull Secrets through a Role in the sandbox namespace. The ClusterRole keeps get and patch on those names for the ownership check and server-side apply into workspace namespaces. - Delete sandbox and supervisor bootstrap Secrets by exact name, derived from the runtime generation recorded on the Sandbox and, on restart, the target generation, tolerating 404. The generation annotation is cleared only after cleanup succeeds, and each bootstrap Secret has a Pod owner reference, so garbage collection removes any generation the driver does not name. - Drop Secret list from the ClusterRole and the shared-mode sandbox Role. - Extend the managed e2e RBAC checks to Secret list. - Update the Kubernetes setup and sandbox runtime docs, compute-runtime architecture, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> * fix(driver-kubernetes): stage workspace Secrets per runtime generation Every Secret the Kubernetes driver writes into a workspace namespace is now scoped to one sandbox runtime generation, immutable, and created with create only, so the gateway never reads, patches, or adopts an existing Secret there. This removes the gateway's cluster-wide get and patch on the copied client TLS and image-pull Secret names. - In managed mode, create an immutable copy of each configured image-pull Secret per generation, named os-pull-<id>-<generation>-<n> and owned by the generation's workload and supervisor Pods. Pods and the restarted Sandbox template reference those names, and generation cleanup deletes them by name. A Secret already holding a generation name fails the create. - Outside shared mode, stage the gateway client TLS material into the supervisor bootstrap Secret instead of copying the client TLS Secret into the workspace namespace. - Remove the fixed-name TLS and image-pull copies, the target ownership read, and the ClusterRole get and patch rule on the copied names. Source reads stay in the sandbox-namespace Role. - Update the managed e2e to expect generation image-pull Secrets and client TLS material in the supervisor bootstrap Secret, and to check that the gateway cannot read the copied names in workspace namespaces. - Update the gateway config and compute driver references, Kubernetes setup and sandbox runtime docs, compute-runtime architecture, driver README, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> * fix(helm)!: grant operator-mode Secret permissions through the workspace chart The operator-mode gateway ClusterRole grants no Secret permissions. The openshell-workspace chart Role, installed in each operator-managed namespace, grants the gateway create and delete on Secrets for sandbox runtime generations. Operator-managed namespaces require the workspace chart. - Fail the chart tests on any ClusterRole rule that includes Secrets in operator and shared modes. - Install the workspace chart when the operator e2e provisions a namespace, and assert that the gateway has no Secret permissions in a namespace without it. - Update the Kubernetes setup docs, 0.1.0 upgrade guide, compute-runtime architecture, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> --------- Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Drew Newberry
|
069ae6bd96 |
fix(vm): scope GPU filesystem enrichment to assigned workloads (#3580)
* fix(vm): scope GPU filesystem enrichment to assigned workloads Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): clarify proxy baseline enrichment Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
b671171091 |
fix(sandbox): qualify task memory against workload child (#3574)
Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
84960e70a3 |
fix(kubernetes): scope resource admission RBAC (#3571)
* fix(kubernetes): scope resource admission RBAC Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(helm): gate PVC admission reads Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Evan Lezar
|
49df4d7ca7 |
fix(server): drain supervisor ownership cleanup on shutdown (#3547)
Closes #3546 Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Drew Newberry
|
1e34e8c576 |
fix(drivers): require admission labels for external resources (#3538)
* fix(drivers): require admission labels for external resources Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(drivers): address resource admission review findings Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(core): reserve driver-owned admission labels Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(core): clarify workspace admission label Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(drivers): clarify resource admission failures Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): configure resource admission fixtures Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(kubernetes): retry forbidden admission lookups Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): preserve external driver admission defaults Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Philippe Martin
|
35e0a68e4a |
feat(kubernetes): support corporate proxy CA bundle (#3447)
* feat(kubernetes): support corporate proxy CA bundle The Kubernetes driver had no way to supply a CA bundle for the corporate egress proxy, so an `https://` proxy with a private CA, or a TLS-intercepting proxy, could not be used. Podman and VM already expose `proxy_ca_bundle`. Add `proxy_ca_bundle` to `[openshell.drivers.kubernetes]` as a path the gateway Pod reads. The gateway stages the PEM into the existing per-generation supervisor bootstrap Secret and passes `--upstream-proxy-ca-bundle` on the supervisor argv. That Secret is already immutable, owner-referenced and garbage-collected, and its volume mounts every key at /.openshell/supervisor with no items filter, so this needs no new object kind, volume, mount, or RBAC verb, and works in shared, managed and operator workspace modes. The bundle is deliberately read from the gateway's filesystem rather than referenced as an object in the sandbox namespace. It becomes a trust anchor for every upstream the sandbox reaches, so it must stay in the gateway's trust domain; the immutable staging Secret also keeps the anchor from changing underneath a running sandbox. Bound the staged bundle at 256 KiB. The shared reader's limit is exactly the apiserver's own Secret limit and the bootstrap Secret carries four other keys, so a bundle between the two would pass gateway startup and then fail every sandbox create with an opaque `data: Too long`. Delegate the URL, no_proxy, connect_by_hostname and ca_bundle rules to the shared validate_upstream_proxy_settings, keeping the Secret-specific credential block local: this driver accepts an explicit `proxy_auth_allow_insecure = false` without credentials, which the shared rules reject. This also fixes the acknowledgement being demanded for an `https://` proxy, where the credential travels inside the verified TLS session. Add auth_setting_label so the inline-credential diagnostic names the Secret keys instead of proxy_auth_file, which this driver rejects as an unknown key. Document that the bundle should carry only the CA that signs the proxy's certificate, or that an intercepting proxy re-signs upstream certificates with. Public roots already reach the sandbox through the supervisor image and its TLS stack, and the bundle is concatenated with that system store into a single boundary control frame, so a full merged trust bundle spends the frame budget on duplicated roots. The frame, not the apiserver Secret limit, is the tighter of the two ceilings in practice; raising the staging bound requires checking it. Closes #3443 Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(helm): quote proxy CA ConfigMap references Signed-off-by: Philippe Martin <phmartin@redhat.com> --------- Signed-off-by: Philippe Martin <phmartin@redhat.com> |
||
|
John T. MyersandDrew Newberry
|
c8a4ff5f19 |
fix(kubernetes): bind bootstrap to runtime identity (#3531)
* fix(kubernetes): bind bootstrap to runtime identity Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(compute): compensate runtime binding failures Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(compute): clean up backend on store failure Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(compute): merge runtime binding after start Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(auth): bind restarted sandbox sessions Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * refactor(compute): fold runtime binding into authentication Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Emilien Macchi
|
99ed6a9df0 |
chore(build): remove stale static-supervisor leftovers (#3520)
The opt-in glibc-static supervisor variant from #2682 is gone: the Nix release builds (#2977) removed it from CI and the supervisor Dockerfile, and the RFC 0012 sandbox split (#2942) removed it from the staging script. The split also moved the binary that runs inside workload images to openshell-sandbox; the supervisor now runs from its own image and is dynamically linked. A few places still describe the old model: - skills/debug-openshell-cluster told operators to check that supervisor_image contains a static /openshell-supervisor. Ask instead for a supervisor from the matching release whose loader and shared libraries are available inside that image, and give a `--version` check that shows both. The static requirement for /openshell-sandbox is unchanged. - verify-static-binary.sh justified its check with the supervisor and the glibc-static variant. Describe the property it enforces instead, with the musl sandbox runtime as the main example. - stage-prebuilt-binaries.sh kept an unreachable gnu-static case in target_triple. No behavior change: resolve_component only ever selects gnu or musl. Signed-off-by: Emilien Macchi <emacchi@redhat.com> |