Legacy or never-initialized sandbox rows decode to the proto-default
Unspecified phase. The startup resume sweep skipped them, leaving the
backend unreconciled: never resumed, never errored. Unspecified was the
only non-terminal phase skipped, since unrecognized values already fall
through to Unknown and get resumed.
Treat Unspecified as running so the sweep reconciles it from live driver
state: resume when the backend resource exists, mark Error when it is gone.
Tests
- cargo test -p openshell-server \
resume_persisted_sandboxes_resumes_running_phases
Signed-off-by: Shiju <shiju@nvidia.com>
Docker BuildKit exposes TARGET* for the requested image platform. It exposes
BUILD* for the daemon running build stages. The bootstrap path did not pass an
explicit platform or seed those args, so Dockerfile behavior could depend on daemon defaults.
Set the build request platform from TARGETPLATFORM when supplied. Use the daemon
platform as the fallback. Query the daemon for BUILD*. Seed implicit args without
overwriting caller values. Do not inject empty variant args.
Tests
- mise run pre-commit
Signed-off-by: Shiju <shiju@nvidia.com>
Drop the vendored deploy/kube/manifests/agent-sandbox.yaml. CI and e2e
scripts now apply manifest.yaml directly from
github.com/kubernetes-sigs/agent-sandbox releases, pinned via
AGENT_SANDBOX_VERSION env (default v0.4.6, overridable) so internal runs
stay reproducible.
Public docs and the helm chart README continue to reference
/releases/latest/download/ — no OpenShell/agent-sandbox support matrix
exists yet (see #1649).
Adds an air-gap note in docs/kubernetes/setup.mdx enumerating the
manifest and image operators need to mirror to an internal registry.
Signed-off-by: Roshni Malani <rmalani@nvidia.com>
Closes#1347. Reported by @KodeDaemon in the NVIDIA Developer Discord after
upgrading NemoClaw 0.0.38 -> 0.0.39 left the inference provider stranded
with `provider.type exceeds maximum length (79 > 64)` on every credential
update.
`update_provider_record` rebuilt the merged Provider from `existing.r#type`
and ran `validate_provider_fields` over it. The CLI's `provider update`
sends `r#type: ""`, so the existing value is preserved without modification,
but the validator still measured it. Any record whose stored type predates
current MAX_PROVIDER_TYPE_LEN (or was written by a path that bypassed
validation, e.g. the TUI form which forwards r#type to the request without
going through normalize_provider_type) became unupdateable: the only escape
was `provider delete` + recreate, which loses any provider.config entries
the caller never sees.
Split the validator into create-time (validate_provider_fields, full check
including immutable name/type) and update-time (validate_provider_mutable_fields,
checks only credentials/config maps). The update path validates only what
the caller is mutating; immutable fields carried forward from existing are
trusted because they passed validation when stored.
Tests: new regression in `update_provider_allows_credential_rotation_on_legacy_oversized_type`
inserts a Provider with a 79-char type directly via `store.put_message`
(bypassing gRPC validation), updates a credential, asserts the update
succeeds and the original type is preserved. Existing
`update_provider_validates_merged_result` still covers the case where
incoming credential entries push the merged map past limits.
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
Co-authored-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
Allow operators to configure a default Kubernetes runtimeClassName that
is applied to sandbox pods when the CreateSandbox request does not
specify one. This avoids requiring every API caller to explicitly set the
runtime class for clusters that always need a specific RuntimeClass
(e.g. kata-containers, nvidia).
The fallback is applied in the Kubernetes driver only — per-request
values still take priority, and an empty default (the built-in) preserves
existing behavior (field omitted, cluster default applies).
* fix(helm): create sandbox JWT secret under cert-manager
The cert-manager install path (certManager.enabled=true,
pkiInitJob.enabled=false) left the gateway StatefulSet unable to start
because nothing created the openshell-jwt-keys Secret: cert-manager owns
TLS Secrets but does not mint the sandbox JWT signing key, and the
certgen hook only rendered when pkiInitJob.enabled was true.
Separate JWT signing-key provisioning from TLS PKI provisioning:
- certgen: add a --jwt-only mode that creates only the Opaque JWT
signing Secret, for use when another controller owns TLS Secrets.
- certgen.yaml: render the hook when pkiInitJob.enabled OR
certManager.enabled is true. cert-manager takes precedence and runs
the hook with --jwt-only even if pkiInitJob.enabled remains true.
Remove the mutual-exclusion failure between the two values.
- _helpers.tpl: add openshell.sandboxJwtSecretName, shared by the hook
and the StatefulSet mount.
- Update values, README, docs, architecture, and the
debug-openshell-cluster skill to reflect the new precedence; the
documented cert-manager install no longer needs pkiInitJob.enabled=false.
Closes#1691
* fix(helm): honor cert-manager precedence for client CA volume
The client CA volume logic treated pkiInitJob.enabled as proof that
built-in PKI owns the client CA. With cert-manager precedence now
allowing certManager.enabled=true alongside the default
pkiInitJob.enabled=true, that assumption mounts the server TLS cert
secret as the client CA and ignores
certManager.clientCaFromServerTlsSecret=false, which can break mTLS or
trust the wrong CA.
Gate the pkiInitJob.enabled term with (not certManager.enabled) in all
three client CA conditions (volume mount, volume definition, and secret
selection) so cert-manager owns TLS when enabled. Add a Helm test suite
covering built-in PKI, cert-manager shared CA, the regression config
(cert-manager + clientCaFromServerTlsSecret=false + default pkiInitJob),
and the no-client-CA case.
Remove three groups of copy-pasted code in openshell-server:
1. grpc/mod.rs had a private current_time_ms() wrapper identical to the
one already exported from persistence/mod.rs. Remove the duplicate
and update the three grpc sub-modules (policy, sandbox, service) to
import directly from crate::persistence.
2. test_store() was repeated verbatim in seven #[cfg(test)] blocks.
Promote a single canonical version to persistence/mod.rs (cfg-gated)
and replace all copies with crate::persistence::test_store() calls or
a thin Arc wrapper in supervisor_session.
3. grpc_client_mtls() and build_tls_root() were copy-pasted across
edge_tunnel_auth.rs and multiplex_tls_integration.rs. Move both into
the existing tests/common/mod.rs shared module and import from there.
The existing docs omitted or misstated several requirements when running
the gateway as a container with the Docker compute driver:
- OPENSHELL_GRPC_ENDPOINT is required; the Docker driver uses only the
scheme (http/https) — host and port are substituted automatically with
host.openshell.internal and the gateway's own bind port
- Supervisor binary must be extracted to a host path before starting the
gateway; bind-mount sources are resolved by the host Docker daemon so
the path must be identical inside and outside the gateway container
- Docker socket access requires adding the docker group (UID 1000 default)
- Port binding should remain 127.0.0.1; Docker driver adds a bridge
listener automatically
- add --server-san host.openshell.internal to generate-certs for mTLS
- Complete the mTLS docker run with all Docker driver requirements
- Add deploy/docker/gateway.toml — TOML config for the Docker driver
- Add deploy/docker/docker-compose.yml referencing the TOML
- Add docs/get-started/tutorials/docker-compose.mdx tutorial page
- Remote gateway registration instructions (--remote flag)
Address reviewer feedback:
- Move Docker Compose tutorials card to the bottom of the list
- Replace inline YAML snippet in Docker Compose section with a reference
to deploy/docker/ to avoid drift
- Clarify OPENSHELL_DB_URL is safe in compose.yml (plain SQLite path,
no credentials); the TOML block targets credential-bearing DSNs
- Note that ./ in source: resolves relative to the compose file directory
- Clarify that only the scheme from OPENSHELL_GRPC_ENDPOINT matters
- Add note that the tilde volume mount resolves to the same absolute
path on both host and container
The Ubuntu Snap canary downloads its artifact from a different workflow run
(the triggering Release Dev run) via run-id. Cross-run downloads require
authentication, so pass github.token to actions/download-artifact.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Auto-detection previously treated Podman as available only when the podman CLI
was visible on PATH. However, package manager services can run with a
restricted PATH, which lets Docker be selected even when a Podman API socket is
reachable. Additionally, podman may symlink /var/run/docker.sock to podman's
machine unix socket, which would be incorrectly detected as Docker. Worse
still: the podman machine may not even be running.
This replaces the Podman binary check with a functional HTTP probe against the
standard Podman socket paths. The probe requires /_ping to answer with a
Libpod-Api-Version header before treating the socket as Podman, which lets the
gateway select the embedded Podman driver only when the API is usable.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Add a desktop launcher for the OpenShell TUI so users can launch
"openshell term" from their desktop environment application menu.
The change adds three files:
- snap/local/term.desktop: desktop entry file for the application launcher
- snap/local/icon.png: application icon (copied from snap store data)
- snapcraft.yaml: new "term" app entry that runs "openshell term"
with home, network, ssh-keys, and system-observe plugs, plus install
rules to stage the desktop file and icon under meta/gui/
The desktop file references the icon via ${SNAP} which is resolved
at runtime to the snap installation directory. The term app reuses
the same connection plugs as the main openshell app.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Install the Snap built by the triggering Release Dev workflow by setting
merge-multiple: true on the artifact download. actions/download-artifact
otherwise extracts each artifact into its own subdirectory, leaving the
package at release/snap-linux-amd64/*.snap, so the install glob
./release/*.snap matched nothing. Merging flattens the artifact's contents
directly into release/ where the dangerous local snap install expects it.
Harden the Snap canary setup by enabling snapd.socket, waiting for snap
seeding (snap wait system seed.loaded), and running every step with strict
shell options (set -euo pipefail) so failures surface immediately.
Register the snapped gateway with the CLI as the documented local plaintext
snap-docker gateway, and print version and snap services, before running
openshell status so the canary verifies a configured and reachable gateway
instead of only the install.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
The RPM canary needs to exercise the install.sh user-service path, but a GitHub
Actions job container does not boot with systemd as PID 1. The Fedora RPM
canary needs to exercise the install.sh user-service path, but a GitHub Actions
job container does not boot with systemd as PID 1. This means the Fedora RPM
canary was incomplete as compared to the others.
With this change, we run Fedora as a nested privileged systemd container
instead, wait for systemd to become reachable, then start the root user manager
so systemctl --user works for the RPM gateway unit, achieving parity with the
other canary tests.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(providers): add Google Vertex AI provider
Adds Vertex AI provider profiles, routing, credential refresh plumbing, CLI support, docs, and regression coverage. Keeps the related NETLINK_ROUTE seccomp allowance needed by Vertex client tooling that calls getifaddrs.
* docs: add Vertex AI sandbox usage for Claude Code and OpenCode
Cover the full end-to-end setup for running Claude Code and OpenCode
inside an OpenShell sandbox via inference.local with a Vertex AI backend:
- google-vertex-ai.mdx: add 'Use from a Sandbox' section with tabbed
examples for Claude Code (--bare flag, no /v1 suffix) and OpenCode
(/v1 suffix required). Add providers_v2_enabled prerequisite and
--no-verify note for global region. Document policy proposals table
covering metadata.google.internal (always blocked), downloads.claude.ai,
and storage.googleapis.com.
- inference-routing.mdx: expand 'Use the Local Endpoint' section with
tabbed examples for Claude Code, OpenCode, Python OpenAI SDK, and
Python Anthropic SDK. Add notes explaining the /v1 path suffix
difference between clients.
- supported-agents.mdx: update Claude Code and OpenCode rows to mention
inference.local support and correct base URL requirements.
* fix: address vertex review findings
* test(sandbox): retry on spurious Ok in fork-exec ambiguity test
On arm64 under heavy CI load, the /proc fd scan in
find_socket_inode_owners can transiently miss the parent process's
socket fd entry, returning only the child as an owner. This causes
resolve_process_identity to return Ok (single owner, no ambiguity
check fires) instead of the expected ambiguous-ownership Err.
Extend the retry loop to also handle unexpected Ok results, mirroring
the existing retry for transient Err results. 10 retries at 50ms gives
a 500ms settling window, which is sufficient for procfs to stabilize
on loaded arm64 runners.
* fix: address vertex review regressions
* docs(router): clarify stream_response semantics for Vertex rawPredict routing
Document the three call sites of prepare_backend_request and their
stream_response values in a caller table:
- send_backend_request: false → :rawPredict (unary endpoint)
- send_backend_request_streaming: true → :streamRawPredict
- verify_backend_endpoint: explicitly false to probe the unary endpoint
Cross-reference the table from build_provider_url and
is_vertex_anthropic_rawpredict_route so the stream_response=true guard
in the suffix upgrade branch is understood in full context.
Also note that is_vertex_anthropic_rawpredict_route is a structural
predicate (model_in_path + anthropic_messages + :rawPredict suffix),
not a named-provider check, so any future provider with the same route
shape inherits the transforms automatically.
* fix(config): reject unknown fields in nested gateway config tables
The gateway TOML loader silently ignored keys placed under the wrong
table header. PR #1661 fixed one instance of this (supervisor_image
under [openshell.gateway.gateway_jwt]) but the root cause remained: the
nested gateway config tables did not deny unknown fields, so a misplaced
key was accepted and dropped instead of erroring.
Concretely, tasks/scripts/gateway.sh emitted `sandbox_namespace` right
after the [openshell.gateway.gateway_jwt] heredoc, so it landed inside
the gateway_jwt table rather than [openshell.gateway]. The k8s driver
already receives the namespace via [openshell.drivers.kubernetes], so
the stray line was dead config that parsed without complaint.
Changes:
- Add #[serde(deny_unknown_fields)] to the nested gateway config tables
that are part of the config-file parse tree: TlsConfig, OidcConfig,
MtlsAuthConfig, GatewayAuthConfig, GatewayJwtConfig.
- Remove the misplaced sandbox_namespace line from gateway.sh.
- Drop the unused Serialize/Deserialize derives from Config and
ServiceRoutingConfig (see below).
- Add a regression test asserting a key under the wrong nested table is
rejected.
Move three duplicated definitions into openshell-core so every
consumer has a single canonical source:
- format_bytes: identical 14-line function existed in docker,
kubernetes, and vm drivers. Moved to openshell-core::progress
where all three already imported from.
- DEFAULT_SANDBOX_PIDS_LIMIT: i64 constant (2048) duplicated in
docker driver and podman config. Moved to openshell-core::config
alongside other shared defaults. Podman re-exports it for
internal call-site compatibility.
- current_time_ms: secrets.rs in openshell-sandbox reimplemented
the same logic as openshell-core::time::now_ms(). Remove the
local copy and call now_ms() directly via the existing dep.
The gateway.sh script appended supervisor_image after the
[openshell.gateway.gateway_jwt] table header, so TOML parsed it as a
gateway_jwt field. The Podman driver never saw the override and fell
back to the default ghcr.io/nvidia/openshell/supervisor:latest.
Move supervisor_image into [openshell.drivers.podman] where the driver
config deserializer expects it.
* ci(kubernetes): pin mise in e2e workflow
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* ci(kubernetes): mirror postgres image for ha e2e
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* ci(kubernetes): reuse e2e workflow for ha
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
---------
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
release-helm and tag-ghcr-release now depend on the release job.
This is to prevent a GHCR image or helm chart from being published when some
other aspect of the release fails.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Add PageUp/PageDown key support to the policy, logs, and draft/rules
views. All three panes now scroll by one viewport height per keypress.
Also fix scroll_policy() clamping to stop at the last viewport of
content instead of the last line, preventing a blank-screen overshoot
on G and PageDown.
Signed-off-by: Major Hayden <major@redhat.com>
* fix(gateway): align package TLS bootstrap path
Closes#1593
Default package-managed gateway services to a stable local TLS directory and use that same value for certificate generation and runtime startup.
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* test(packaging): validate package asset paths exist
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* ci(e2e): pin mise in kubernetes job
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
---------
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Closes#1307
Default the Podman host gateway alias override to gvproxy's host-loopback IP on macOS while preserving host-gateway resolution on Linux. Wire the setting through Podman config, gateway TOML inheritance, and the standalone driver, and document the platform behavior.
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* docs(providers): note that ANTHROPIC_API_KEY requires an API account, not a subscription
Anthropic subscription users authenticate via OAuth, not an API key,
causing a silent failure when creating the provider. Adds a Note callout
in the provider type table and quickstart guide directing subscription
users to generate an API key from console.anthropic.com.
Closes#620
* docs(providers): fix Note placement and remove subscription brand names
Move the Note callout in manage-providers.mdx to after the complete
provider type table so it does not break table rendering. Remove
subscription brand names from both Note callouts.
Rootless Podman sandbox containers reach the host through pasta's local
connection bypass, which translates L2 frames to L4 host sockets. The
dev gateway script binds to 127.0.0.1 by default, which is not routable
through pasta. Auto-detect rootless mode and bind to 0.0.0.0 so sandbox
containers can connect to the gateway.
- Auto-detect rootless Podman in gateway.sh and export
OPENSHELL_BIND_ADDRESS=0.0.0.0 when not explicitly set
- Add e2e:podman:rootless mise task and CI matrix entry to validate
rootless Podman networking end-to-end
- CI creates a non-root user inside the privileged container to trigger
Podman's rootless code paths (pasta, user namespace isolation)
Signed-off-by: Naveen Malik <nmalik@redhat.com>
* feat(python-sdk): support OIDC Bearer auth on SandboxClient
PR #1596 hardened the gateway side of the OIDC story; the Python SDK
was the remaining gap — it only supported plaintext or mTLS, with no
Bearer metadata anywhere. Deployments with OIDC enabled (the
recommended posture since PR #935 / PR #1404) were unreachable from
the SDK.
Adds:
- `bearer_token: str | Callable[[], str] | None` kwarg on
`SandboxClient`. Static strings or zero-arg callables (the latter
is invoked per RPC, so callers can drop in a refresh loop or
token-file watcher without reconstructing the client). Composes
with `tls` for OIDC-over-mTLS deployments.
- `_BearerAuthInterceptor` implementing all four
`grpc.{Unary,Stream}{Unary,Stream}ClientInterceptor` types.
Appends `authorization: Bearer <token>` to outgoing metadata.
Implemented as an interceptor (not call credentials) so it works
on both plaintext (`disableTls=true` dev) and TLS channels without
`grpc.composite_channel_credentials`.
- `TlsConfig` ergonomics: all three fields (`ca_path`, `cert_path`,
`key_path`) are now optional with `cert_path` / `key_path`
required-together-or-not-at-all (enforced in `__post_init__`). This
unlocks three transport profiles from one dataclass:
* full mTLS (all three)
* CA-only trust (`ca_path` only)
* system roots (`TlsConfig()` — for OIDC gateways behind a
public CA)
- `from_active_cluster` mirrors `crates/openshell-tui/src/lib.rs`
`build_oidc_channel`:
* For any `https://` gateway, always build a secure channel.
Pick the strongest TLS profile available in `mtls/` (full
mTLS → CA-only → system roots). No more `insecure_channel`
fallback for HTTPS.
* Gate OIDC bearer attachment on
`metadata.json["auth_mode"] == "oidc"`. Matches
`crates/openshell-cli/src/main.rs:132` and the TUI; a stale
`oidc_token.json` next to a non-OIDC gateway no longer causes
the SDK to attach a bearer.
- `_OidcRefresher` — thread-safe, in-process native OAuth2 refresh
modeled on `google.oauth2.credentials.Credentials` and
`botocore.tokens.SSOTokenProvider`. Lazily checks expiry on every
RPC; when stale, re-reads disk first (the CLI may have rotated
the bundle), and only then exchanges the refresh_token against
the IdP's token endpoint discovered via OIDC discovery
(`/.well-known/openid-configuration`, cached after first call).
Concurrent RPCs share a single refresh via `threading.Lock` (no
IdP stampede). Honors refresh-token rotation. Surfaces IdP
failures as `SandboxError` with the RFC 6749 error body included
for diagnostics.
Mirrors the Rust CLI's HTTP-policy posture from
`crates/openshell-cli/src/oidc_auth.rs`:
* `follow_redirects=False` so a 3xx during discovery can't
steer us to an attacker-controlled token endpoint.
* Discovery `issuer` is validated against the configured
issuer; a discovery document claiming a different issuer is
rejected, preventing the SDK from POSTing the refresh_token
to a malicious endpoint.
* `insecure: bool` flag plumbed through to httpx's
`verify=` so self-signed-cert deployments work the same way
they do in the Rust CLI.
Built on `httpx` (chosen over `urllib` specifically for
follow_redirects + verify control as kwargs). The OAuth2
refresh-token grant itself (RFC 6749 §6) is one form-encoded
POST — handled inline rather than via a dedicated OAuth library;
tried `authlib`'s `OAuth2Client` first but it auto-injects an
Authorization header on every request, which breaks the
unauthenticated discovery GET.
- `_make_cluster_bearer_provider(..., auto_refresh=True,
write_back=True, insecure=False)` factory. Defaults to the
refresher path with write-back enabled; `auto_refresh=False`
falls back to the read-only fail-closed behavior for callers that
don't want the SDK to make outbound HTTP calls to the IdP.
`write_back=True` is the default (changed from the first round of
review): IdPs with refresh-token rotation (Keycloak with
rotation, Entra in strict mode) invalidate the old refresh_token
on each refresh, so an in-memory-only refresh would leave the
on-disk bundle pointing at an invalidated value — any second
process starting from disk would `invalid_grant`. With write-back
enabled by default, the SDK keeps the shared cache consistent
with the IdP.
- `from_active_cluster` exposes `auto_refresh`, `write_back`, and
`insecure` kwargs (defaults: True / True / False). The
high-level `Sandbox` context manager surfaces the same three
kwargs and forwards them through, so callers using the wrapper
have parity with `SandboxClient` for OIDC-protected gateways.
- `SandboxClient.close()` chains to a `_bearer_close` hook so the
`_OidcRefresher`'s underlying `httpx.Client` is released
deterministically instead of leaking sockets/FDs until GC runs
`__del__`. Idempotent.
- `_OidcRefresher._write_to_disk` uses `tempfile.mkstemp` (PID +
random suffix) instead of a fixed `.oidc_token.json.tmp` path,
so two writers racing on the same gateway directory don't
trample each other's tmp content. Success path atomically
replaces; failure path unlinks the orphan.
OAuth2 refresh policy and write-back semantics deliberately mirror
what the major Python SDKs do — see
github.com/googleapis/google-auth-library-python (`Credentials`)
and github.com/boto/botocore (`SSOTokenProvider`):
| Library | Native refresh | Writes back |
|-------------------------------|----------------|-------------|
| google-auth Credentials | yes | no |
| botocore SSOTokenProvider | yes | yes |
| openshell SandboxClient (here)| yes (opt-out) | yes (opt-out)|
OpenShell sits between the two; chose write-back-by-default because
the rotation invariant matters more for our deployments than the
"CLI is the only writer" assumption that fits google-auth.
Adds `httpx>=0.27` as a runtime dependency. No new OAuth2 library —
the refresh grant is a single POST.
Tested:
- 42 sandbox_test.py tests pass (5 pre-existing + 37 new across
the bearer interceptor, fail-closed provider, refresher
behavior, TlsConfig validation, from_active_cluster auth ladder,
security-review regressions, Sandbox-wrapper kwarg forwarding,
and lifecycle / concurrency probes).
`mise run test:python` → 47 passed total across the python
suite.
- `mise run python:lint` (ruff) clean.
- End-to-end against a Keycloak-protected gateway on OpenShift:
* unauthenticated `Health` bypass works
* admin + `openshell:all` reaches user-callable methods
* reader (`sandbox:read`) denied on `CreateSandbox` by scope
* admin + `openshell:all` denied on PR #1596 sandbox-only
methods at the router (the new gate is honored from the SDK)
* full provider CRUD lifecycle via the SDK
* callable token provider rotates per RPC as expected
- Regression-probed against three pre-review security findings:
* **Discovery issuer validation** — a discovery document
claiming a different `issuer` than the configured one is
rejected with a clear `SandboxError` before any refresh POST
can reach the attacker-controlled endpoint.
* **Redirect during discovery** — `follow_redirects=False` on
the underlying httpx client means a 3xx during discovery
surfaces as a SandboxError rather than silently chasing the
redirect.
* **Cross-process rotation** — a two-process simulation shows
process B starting from disk and successfully refreshing
with the rotated refresh_token, because process A's
write-back updated the shared cache.
- Refresher unit tests cover: cached-fresh fast path, disk-rotated
re-read before refresh, OAuth2 exchange against the discovered
token endpoint, refresh-token rotation, atomic write-back at
0600 mode (default), default-on write_back proven by test,
concurrent N-thread coordination (one refresh shared across 8
threads), IdP failure surfaced with the error body, the
client_credentials / no-refresh_token error path, issuer-
mismatch rejection, redirect-during-discovery rejection,
insecure flag plumbing.
- Lifecycle / concurrency regression tests added: `close()`
invokes the `_bearer_close` hook (idempotent), the refresher's
`httpx.Client` is marked closed after `SandboxClient.close()`,
and 16 concurrent writers don't leave orphan tmp files behind
while producing a valid final bundle. The `Sandbox` wrapper has
direct forwarding tests proving `auto_refresh`, `write_back`,
and `insecure` reach `from_active_cluster` (both explicit
values and defaults).
- End-to-end against a real OpenShift + Keycloak cluster from
inside a pod: real OIDC discovery against
`keycloak.keycloak.svc.cluster.local:8080`, refresh-token grant
POST, atomic write-back of the rotated bundle at 0600, and a
follow-up RPC reusing the freshly-rotated in-memory token —
full round-trip in ~170ms.
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
* fix(python-sdk): adopt newer on-disk OIDC bundle before refreshing
_OidcRefresher.current_access_token() only adopted the on-disk
oidc_token.json when its access token was still fresh; otherwise it
refreshed using the in-memory bundle. With refresh-token rotation
enabled (Keycloak with rotation, Entra strict mode), this let a process
keep using an invalidated refresh_token:
1. Process A holds a stale in-memory bundle with refresh_token=r1.
2. Process B refreshes first and writes a rotated (r2) but now
near-expiry bundle to disk.
3. Process A re-reads disk, sees the access token is not fresh, ignores
the disk bundle, and POSTs the stale r1 — which the IdP has already
invalidated, yielding invalid_grant.
Fix: when the cached bundle is stale, adopt the on-disk bundle if it was
refreshed more recently than ours, even when its access token is also
stale. "More recently" is decided by expires_at — a refresh mints a new
access token with a forward expiry alongside the rotated refresh_token,
so the later expiry carries the newest refresh_token. Comparing by
expiry (rather than unconditionally preferring disk) preserves the
write_back=False case, where the in-memory bundle has already rotated
past the on-disk copy and must not be clobbered. When the adopted
bundle's issuer differs, the cached token endpoint is reset so the
refresh re-discovers against the new issuer.
Adds regression tests for the cross-process rotation race and the
issuer-change re-discovery path.
* fix(python-sdk): recover from invalid_grant on lost rotation race
The expiry-based disk re-read narrows but does not fully close the
cross-process refresh-token rotation race: two processes sharing a
gateway directory can both enter their refresh window, both POST their
copy of the refresh_token, and with rotation enabled the IdP invalidates
the loser's token (invalid_grant). Neither google-auth nor botocore
close this window without an OS file lock; a Python-only flock would not
coordinate with the Rust CLI/TUI that also write oidc_token.json, so
locking is not worth its cost here.
Recover instead of prevent: distinguish an OAuth2 invalid_grant (the
refresh_token was rejected) from transport/5xx failures via a private
_InvalidGrantError, and on invalid_grant re-read oidc_token.json once. If
a peer wrote a different refresh_token (it won the race), adopt and retry
with it — returning early if it is already fresh — so the loser succeeds
transparently instead of forcing a re-authenticate. If disk offers no new
token, the rejection is genuine and surfaces the re-authenticate hint as
before. The retry is single-shot; a second invalid_grant propagates.
Adds tests for the peer-rotation recovery and the genuine-rejection
(no-retry) paths.
---------
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
* refactor(proto): move phase and current_policy_version into SandboxStatus
Move phase and current_policy_version from SandboxSpec into
SandboxStatus to correctly model mutable runtime state. Update all
callers in the gateway server, TUI, and Python SDK to read and write
these fields through SandboxStatus accessors.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(server): preserve sandbox status on statusless driver updates
When a driver update arrives without a status payload (e.g. before
Kubernetes populates the status subresource), preserve the stored
phase, conditions, and current policy version instead of resetting
them. Adds a regression test covering the edge case.
Signed-off-by: Derek Carr <decarr@redhat.com>
---------
Signed-off-by: Derek Carr <decarr@redhat.com>
* feat(build): add simple nix flake with formatter for nix code
* feat(flake): setup rust toolchain, able to build and run unit tests
* feat(flake): add support for arm linux and macos
* feat(toolchain): add rust-src and rust-analyzer to the toolchain
Shiju