This changes the DNS resolver test fixture to retry until TCP and UDP can share
an ephemeral port.
It fixes an issue seen in CI where a port that was successfully assigned for
one protocol failed for the other.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Ctrl-C closes gateway listeners while supervisor sessions are still current,
causing benign HTTP/2 broken pipes to be logged as warnings. Track gateway
shutdown so recognized transport closes are treated as expected.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(helm): refresh kubeconfig for existing k3d clusters
Docker can recreate the k3d load balancer on a new API port. Start existing
clusters and prefer fresh k3d entries so create does not retain a stale
endpoint.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(dev): conditionally enable local OTLP export
Probe port 4317 before adding OTLP configuration for the VM, Docker,
and Podman gateway tasks. Document the startup behavior and troubleshooting
for local collector availability.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
---------
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(gateway): add installation name configuration
Add a first-class operator-assigned gateway name with TOML, CLI, environment,
and Helm configuration surfaces. Local gateways default to openshell, while
Helm defaults to the chart fullname; operators sharing a collector across
namespaces or clusters can set a globally distinct name.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(gateway): identify gateways in exported traces
Attach the configured gateway installation name and compute driver to the
gateway OpenTelemetry resource so operators can filter traces from multiple
installations that share a collector. Forward the gateway name and OTLP
endpoint to managed external drivers so their distinct service resources carry
the same installation identity.
Keep service.name stable per process type, omit blank resource values, and
leave per-span operation names and request attributes unchanged.
Refs #2507
Signed-off-by: Kris Hicks <khicks@nvidia.com>
---------
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(driver): add MXC compute driver for Windows isolation sessions
Introduces the openshell-driver-mxc crate implementing ComputeDriver
backed by Microsoft MXC isolation sessions (Windows only). Wires the
new driver into the server's build_compute_runtime dispatch and adds
the Mxc variant to ComputeDriverKind.
Also adds a local protobuf-src stub (tools/protobuf-src-local) to
unblock Windows builds that lack MSYS2/MinGW, and pins the zig
Windows x64 toolchain in mise.lock.
(cherry picked from commit 4f7012224efb18fbfeb47aa87e0cfd3f036f32f0)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* wip(mxc): checkpoint hung-agent work (recon, policy_map embed, A1 wiring, demo artifacts)
Safety checkpoint of uncommitted work from the background agent run that stalled mid-Step-7. Includes: mxc-driver-recon.md (Step 0.5), policy_map.rs (~876L embedded mapper), A1 policy-threading edits across driver.rs/policy.rs/mxc.rs/compute/mod.rs, and examples/ (demo.yaml + mxc-gateway.toml). Not yet verified to compile end-to-end; to be reorganized into the skill's Step 11 commit sequence.
(cherry picked from commit 38e42c03870be3d10e984a54f17a3b61122ff510)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* test(mxc): fix lifecycle and policy unit-test compile drift
- Bring futures::StreamExt into scope for the watch-stream `.next()` call in
driver::lifecycle_tests so the negative policy proof test compiles.
- Bind a local `mapper` and drop the unused/deprecated NetworkBinary in the
embedded-mapper network-policy rejection test.
Signed-off-by: Jamie King <jamiek@nvidia.com>
(cherry picked from commit 039b0baf98735ca672dae52be8d3af2417dc0c1a)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* fix(mxc): downgrade missing sandbox_token to debug log
The gateway mints `sandbox_token` only when a sandbox-JWT issuer is
configured. There is no in-sandbox supervisor on MXC (supervisor-removal
design — D1/D4), so no component ever consumes the token; requiring it
on the driver side blocks the demo's `--disable-tls` smoke gateway with a
spurious `invalid_argument`. Log the absence and proceed instead.
Signed-off-by: Jamie King <jamiek@nvidia.com>
(cherry picked from commit cea209797d0edcb1d152251e748900b0a63cca62)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* fix(mxc): keep sandbox Ready after a successful one-shot agent exec
monitor_exec demoted Ready->Error on exit 0 (reason ExecCompleted), so the positive demo (write hello.txt + exit) landed in Error phase. Keep Ready=True (reason AgentCompleted) on success; only non-zero exits go to ExecFailed. Tighten the positive lifecycle test to assert the terminal condition stays Ready=True/AgentCompleted. Verified live via gateway mock round-trip: phase now Provisioning->Ready with no demotion.
(cherry picked from commit 54ab030f03ca0f83d0050d8dd843b633651684ad)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* feat(mxc): add processContainer backend for default-deny enforcement
Add a backend selector to the MXC driver (isolation_session default | process_container). process_container drives a one-shot AppContainer that is genuinely default-deny: a write to any ungranted path is denied by the OS, unlike isolation_session which is grant-only and cannot deny. The lifecycle forks on the flag - isolation_session keeps provision/start/exec, process_container runs a single ephemeral container via run_oneshot.
Also: run-demo.ps1 gains -Backend and hardens the CLI register/create calls; docs corrected to state isolation_session does NOT deny out-of-policy writes and that the negative proof requires process_container.
Verified end-to-end on a real demo box (gateway -> CLI -> driver -> MXC): in-policy write succeeds, out-of-policy write denied (PermissionDenied), OVERALL: PASS.
(cherry picked from commit c6cde3860bbe1b8edb3147d3e840f6bf0ece32d8)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* refactor(driver-mxc): embed policy mapper as a module; remove standalone crate
Adopt the proto-based mapper (map_to_mxc) as the single source of truth,
embedded in openshell-driver-mxc as a Windows-gated `policy_map` module.
Rewire EmbeddedPolicyMapper to call it directly on the typed SandboxPolicy,
deleting the serde_yaml proto->YAML bridge. Move the CLI to a windows-gated
example and the parity tests into the crate; delete openshell-policy-mapper.
- gate policy_map + seam Windows-only (MXC is Windows-only)
- drop serde_yaml; add dev-deps openshell-policy, clap, anyhow
- normalize mapped paths to Windows form in the seam, in one place
- docs: add driver-mxc to AGENTS.md table; correct design doc section 17 test lane
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
(cherry picked from commit f22f9c7a25b9a651c5c5cc73f62fb01c4d6c1a8d)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* feat(driver-mxc): implement lossless split_policy for proxy-delegated egress
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
(cherry picked from commit 96d6afa0e2dc6a1d54edd12c34a0ceb0a30dadd0)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* feat(driver-mxc): implement Pattern-C governed-egress split through the policy seam
- split_policy: SocketAddr proxy_redirect (replaces bare port), processcontainer
containment guard naming MXC M1, version preserved in the trimmed proxy_policy,
delegation reported as an info loss item
- seam: MappedConfig carries trimmed_policy + proxy_addr; MapCtx.egress selects
the split path; coarse path unchanged when egress is disabled
- driver: [openshell.drivers.mxc] egress_proxy / egress_proxy_addr config,
validated at create (isolation_session rejected until M1); lifecycle threads
the redirect into provision and stores the trimmed policy per sandbox,
emitting an EgressRedirect platform event
- mxc: optional MxcNetwork block (defaultPolicy=block + proxy) in provision and
one-shot configs; mock records configs for test assertions
- tests: lossless-invariant suite over all example policies (validate +
serialize round-trip), split lifecycle proof, M1 rejection; example gains
--split --proxy-addr writing mxc-config.json / trimmed-policy.yaml /
loss-report.json
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
(cherry picked from commit 34d54ad9f25dc6034c3ba15668555ff0d22cddd8)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* fix(driver-mxc): emit MXC network.proxy as {localhost: port}
Verified against the real wxc-exec 0.6.0-alpha via --dry-run: MXC accepts
only the {localhost: N} proxy shape (the form the design doc specifies)
and rejects {host, port} with a parse error. Schema 0.6.0-alpha can
express only a loopback port, so non-127.0.0.1 redirect addresses are now
rejected: split_policy emits an error loss (no proxy block) and the driver
refuses egress_proxy_addr values off 127.0.0.1. Per-sandbox attribution
must use per-sandbox ports until the schema widens.
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
(cherry picked from commit edde8d5434571fd5398409204fcf6862672c0793)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* fix(driver-mxc): serialize isolation_session stop/deprovision as unit variants
Empirical contract finding from the real test lane (build 26300.8553,
wxc-exec 2026-06-10): the stop and deprovision experimental blocks are
unit variants in the wxc-exec schema and must serialize as null; sending
{} is rejected with malformed_request (invalid type: map, expected unit),
while provision/start accept maps. The production invoker, the real-lane
test, the probe script, and the e2e runner all sent {} - the driver could
provision and run an agent but never stop or delete an isolation-session
sandbox against this build. Pinned by a unit test.
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
(cherry picked from commit 0df39ca0b22ebc21eb965b2b567a5b2cac26af32)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* test(driver-mxc): add Tier-0 mapper coverage matrix with schema drift guard
Three-quadrant, table-driven matrix (38 tests): mappable fields assert
exact MXC output; every OpenShell field MXC cannot express asserts a loss
item with the expected severity (and seam rejection on error); an empty
policy asserts the restrictive default-deny posture for every MXC knob
OpenShell does not control. The handled_fields_inventory drift guard
serializes a fully-populated policy and compares its YAML keys against
the mapper-handled field lists, so a new openshell-policy field fails the
suite until consciously mapped, delegated, or reported as loss.
Re-exports the policy seam types for integration tests; adds serde_yml,
base64, serde_json as dev-dependencies.
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
(cherry picked from commit 91807f984a3b16846e35d6ca0d5ec41057aafa3a)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* feat(driver-mxc): inject agent_env into sandbox process.env
Add MxcComputeConfig.agent_env: each entry is either KEY=VALUE (verbatim) or a bare KEY resolved from the gateway host environment at launch, keeping secrets (e.g. inference API keys) out of the config file. Wire it into the agent process so gateway-launched agents can authenticate to cloud endpoints (process.env was previously hardcoded empty). Unit-tested via resolve_agent_env_passthrough_and_host_lookup.
Also add a gateway-driven cloud-inference (T1) test harness: mxc-inference.toml (agent_env + curl agent), inference.yaml policy, and run-inference-test.ps1 which starts the gateway, creates an isolation_session sandbox, runs an authenticated Nemotron call, and bundles redacted results. Documented agent_env in mxc-gateway.toml. Validated end-to-end on the test box (chat HTTP 200 + completion via the gateway).
(cherry picked from commit 94d9e827b8b77e0af9dc654943ea8e7d8400cc9f)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* test(driver-mxc): avoid unsafe env mutation in resolve_agent_env test
Replace std::env::{set,remove}_var (unsafe + racy under parallel test
execution in edition 2024) with a read-only PATH lookup. Preserves all
three behaviors under test and drops the #[allow(unsafe_code)].
(cherry picked from commit ac5766eeab2db4e8cc6fcd8d8a97809edaf3df30)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* fix(mxc): adapt MXC driver to current GitHub OpenShell API
The MXC driver crate was authored on GitLab against an earlier proto/core
API. Adapt it to the API on GitHub main:
- build_capabilities_response no longer takes supports_interactive_session
- DriverSandboxSpec.gpu (bool) is now resource_requirements; detect GPU via
effective_driver_gpu_count(driver_gpu_requirements(..))
- DriverSandbox gained a `workspace` field
- SandboxPolicy gained `network_middlewares`: pass it through the proxy split,
emit a loss item on the coarse MXC path, and account for it in the mapper
drift-guard test
Verified: cargo check + 75 mock-based tests pass (lib 27, examples 10,
policy_mapper_matrix 38).
Signed-off-by: Jamie King <jamiek@nvidia.com>
* feat(server): wire the MXC compute driver into the gateway on Windows
Register openshell-driver-mxc as the Windows-only in-process compute
backend so compute_driver = "mxc" resolves to a working runtime:
- ComputeRuntime::new_mxc, adapted to the current 11-arg from_driver
- mxc_policy_sink A1 side channel, staged in create_sandbox before dispatch
- mxc_config_from_context loader and the Mxc dispatch arm (Windows
constructs; other targets return an explicit "Windows-only" error)
- Windows-gated openshell-driver-mxc dependency
- Mxc arms for the telemetry, config-file required-fields, and CLI
reserved-builtin matches to keep them exhaustive/correct
Verified with cargo check --workspace --features openshell-prover/bundled-z3
on x86_64-pc-windows-msvc, stacked on PR #2496.
Signed-off-by: Jamie King <jamiek@nvidia.com>
* fix(driver-mxc): implement GetGatewayListenerRequirements for #2496 base
Signed-off-by: Jamie King <jamiek@nvidia.com>
* test(driver-mxc): add probe-gated real wxc-exec test lane (no mocks)
- tests/wxc_exec_real.rs: ignored-by-default integration tests against a
real wxc-exec. Six --dry-run contract tests run wherever the binary
exists (they caught the network.proxy shape mismatch); enforcement
tests (processcontainer default-deny positive/negative, isolation
session lifecycle round trip with a deprovision drop-guard) probe the
backend and SKIP with a recorded reason where it is not live.
- examples/probe-mxc-host.ps1: classifies a host (OS build, --probe,
per-backend trial) and emits a JSON capability verdict.
- examples/run-mxc-e2e.ps1 + e2e-policies/: scenario runner generalizing
run-demo.ps1 (fs-rw, fs-readonly, fs-default-deny-empty,
network-policy-rejected) with PASS/FAIL/SKIP gating and a stale
OPENSHELL_MXC_MOCK_WXC guard in real mode.
- tasks/windows.toml: windows:test:mxc-real:x64, windows:e2e:mxc,
windows:e2e:mxc:mock.
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
(cherry picked from commit 49afafe892caded59c4df50a9b652011ad97f41c)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* fix(driver-mxc): address PR review feedback
Signed-off-by: Shailendra Singh <shailendras@nvidia.com>
* docs: defer public MXC documentation
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
* fix(driver-mxc): build Windows capabilities response
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
* fix(server): gate in-tree tracing on Windows
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
* test(windows): fix cross-platform test assumptions
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
* test(windows): verify process and PEM portably
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
* test(windows): keep lifecycle command in policy
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
---------
Signed-off-by: Jamie King <jamiek@nvidia.com>
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
Signed-off-by: Shailendra Singh <shailendras@nvidia.com>
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
Co-authored-by: Prashant K <pkhodade@nvidia.com>
Co-authored-by: Giedrius Burachas <gburachas@nvidia.com>
Co-authored-by: Shailendra Singh <shailendras@nvidia.com>
Co-authored-by: Drew Newberry <385+drew@users.noreply.github.com>
Requests denied by relay_with_route_selection because no L7 endpoint path
matched returned a 403 without emitting an OCSF event or recording policy
activity, leaving the deny invisible to operators and telemetry. Every
other early-return deny in that loop already logs.
Emit an HTTP Activity deny event and an l7_policy activity record before
writing the denial. Split event construction out of emit_l7_request_log
so the event shape is directly testable, and narrow both helpers to take
the request action rather than the full L7RequestInfo, which they only
used for that field.
Signed-off-by: William Trépanier <wtrep@icloud.com>
Mirror the VM, Podman, and Docker driver tracing setup for Kubernetes.
Export standalone driver spans through OTLP/gRPC as the distinct
openshell-driver-kubernetes service, preserve gateway trace context, record
lifecycle operations and gRPC failures, and flush spans on shutdown.
Kubernetes currently runs in-process when selected as a built-in gateway
driver. Use the temporary server-boundary shim shared with Podman and Docker
so traces retain the shape they will have when Kubernetes moves to a
separate process. Move the common ComputeDriver RPC tracing layer into
openshell-otel to keep all drivers aligned.
Propagate the active W3C context through the controller-reserved Sandbox
annotation and enable Agent Sandbox OTLP export in the local k3s workflow.
This connects asynchronous controller reconciliation spans to the originating
OpenShell create trace.
Expose gateway OTLP configuration through Helm and add an Aspire collector
to the local k3s workflow. Extend helm:k3s:forward with OTLP ingest and trace
UI forwarding for Kubernetes and local container gateway development.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Agents will often give a ton of feedback, but it can be hard to quantify the
impact of the issue the feedback is about.
This change updates the PR review feedback template to be more human-readable,
rooting any concerns in user-visible behavior when appropriate, and comparing
the new behavior to old behavior so that PR authors can make a determination of
whether they want to accept or reject the feedback.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Make the local k3s gateway workflow match the Docker and Podman flows by
registering and selecting successful plaintext Skaffold deployments with the
OpenShell CLI. Derive the registration name from the worktree-specific k3d
cluster name so parallel worktrees retain independent gateway metadata.
Add helm:k3s:forward as the standard way to expose the Kubernetes gateway on
localhost:8090, and update the development and debugging guidance to use the
active registered gateway instead of one-off endpoint flags.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* ci(branch-checks): run Rust checks in Nix shells
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(branch-checks): run Rust tests with nextest
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(branch-checks): cache Rust workspace artifacts
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(branch-checks): run cargo-deny in Nix shell
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
---------
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Apply the same two-layer defense from the proxy accept loop (#2369/#2370)
to the SSH accept loop: classify transient vs terminal accept errors with
exponential backoff on EMFILE/resource-exhaustion, and notify the sandbox
when the accept loop exits so the container terminates instead of running
without SSH access.
- Add SshAcceptAction enum and classify_ssh_accept_error in ssh.rs,
mirroring the proxy pattern (EMFILE/ENFILE/ENOBUFS → Retry with
backoff, unknown errors → Terminal after 10 consecutive failures)
- Replace the bare accept().await in run_ssh_server with a classify-and-
retry loop; resets consecutive-error counter on each successful accept
- Thread ssh_exit_tx: Option<oneshot::Sender<()>> through run_process;
hold it as a drop-guard inside the SSH spawn so the receiver fires when
the task ends for any reason
- Wire ssh_exited future in lib.rs (created only when ssh_socket_path is
Some) and select! on it in both process_enabled paths, returning an
error so the sandbox container restarts
Closes#2372
Signed-off-by: politerealism <burdcat17@gmail.com>
* feat(tui): show persisted guidance on rejected policy chunks
The reviewer's rejection note is stored in PolicyChunk.rejection_reason and
already reaches the TUI in GetDraftPolicyResponse.chunks, but openshell-tui
never read the field. The note was dropped at the last step, so a reviewer
had no way to recall why a chunk had been rejected.
Render it in two places, following the truncate-in-list / full-in-popup
convention in the TUI development guide: a truncated, dimmed suffix on the
list row, and a "Guidance:" line in the detail popup.
Gate the accessor on status == "rejected" rather than on the field alone.
Approving a chunk passes None for the reason and the gateway writes the
field only when Some, so a chunk that was rejected and later approved still
carries the old note. Reading the field unconditionally would surface a
stale rejection on an approved rule.
Part of #1098, Definition of Done item "Rejected chunks show persisted
guidance". The other items on that issue are untouched.
Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
* fix(tui): scroll the draft detail popup so long guidance stays reachable
A rejection reason has no server-side length cap, so a paragraph-length one
overflowed the fixed 22-row detail popup: the tail was clipped and the later
fields and both action hints were pushed off screen. The same overflow already
affected the unbounded rationale and security_notes fields, so fix the popup
rather than special-case the guidance line.
Split the popup's inner area into a scrolling body and a pinned hint row,
following the pattern in ui/create_provider.rs, and drive it with j/k, the
arrow keys, PageUp, PageDown, g and G. The approve and close controls now stay
on screen at every scroll position, and the bottom border carries the scroll
position.
Wrap free-form values explicitly instead of relying on Paragraph's Wrap, so the
rendered row count is exactly lines.len() and the scroll clamp cannot under-run
the content. Paragraph::line_count would answer the same question but sits
behind ratatui's unstable-rendered-line-info feature.
Add deterministic TestBackend coverage at 80x24 with a 2,000-character reason,
covering the head and tail, the pinned hints, over-scroll clamping, and the
pre-existing long-rationale case. Document the reviewer-facing guidance in the
policy advisor page.
Part of #1098.
Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
* fix(tui): wrap the denial row and hint footer on a narrow popup
Making the scroll clamp exact meant dropping Paragraph's Wrap, which also
removed wrapping from the lines that do not go through push_wrapped. At 70
columns the denial row clipped mid-value and lost the last-seen timestamp, and
at 60 a pending chunk with scrollable content pushed [Esc] Close past the right
edge of the single hint row. The key still worked, but that is the same
controls-not-visible failure one axis over.
Keep the denial row on one line while it fits and wrap it onto the label indent
when it does not, so the common 80-column layout is unchanged. Pack the footer
hints into as many rows as they need without splitting a hint, and derive the
footer height from that: adding a hint row shrinks the body and can itself
change whether the content scrolls, so the two settle together.
The earlier tests were all 80x24, which is why they missed this. Cover the
denial timestamps and the approve, reject and close hints at 60, 70 and 80
columns, plus the packing helper directly.
Part of #1098.
Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
* fix(tui): measure popup wrapping in display columns, not chars
wrap_value packed rows by chars().count(), and a CJK glyph is one char but two
terminal columns. A double-width rejection reason therefore produced rows about
twice the popup's width: the right half of every row was clipped, and unlike
the vertical case there is no horizontal scroll to recover it.
Measure width with Span::width(), the same measurement ratatui applies when it
lays cells out, so the wrap agrees with the renderer by construction and no new
dependency is needed. This covers word packing, the hard-break path for a word
wider than the line, the label indent, the footer hint packing, and the denial
row's fits-on-one-line check.
The list row's shortened copy had the same defect through truncate_str, which
counts chars. Leave truncate_str alone for its two existing callers and add
truncate_display for the guidance row.
Cover it with an 80x24 render regression that interleaves markers through the
CJK text: a trailing marker lands on its own short row in both the broken and
fixed layouts, so it would not detect this. The two unit tests measure with an
independent column oracle rather than the helper under test, which would make
them tautological.
Part of #1098.
Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
---------
Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
* fix(sandbox): terminate sandbox when proxy accept loop exits unexpectedly
Add a oneshot notification channel to ProxyHandle that fires when the
proxy task exits for any reason (panic, abort, or loop break). The
sandbox main loop now races this signal alongside the entrypoint process
and shutdown signals, terminating with a clear OCSF error if the proxy
dies. This prevents the sandbox from continuing to report Ready with a
dead proxy.
Follows the existing sidecar control channel pattern. All five wait
paths (process+sidecar, process-only, network+sidecar, network-only
on Linux, and non-Linux) now monitor proxy liveness.
Refs #2337
Signed-off-by: politerealism <burdcat17@gmail.com>
* test(proxy): add unit tests for exit detection drop-guard and take_exit_receiver
Verify the oneshot drop-guard pattern fires on normal task exit and
abort, and document the take_exit_receiver contract including the
double-call hazard.
Signed-off-by: politerealism <burdcat17@gmail.com>
* fix(proxy): classify terminal vs transient accept errors and exit on unrecoverable failures
Extract accept-loop error handling into classify_accept_error() with
AcceptAction enum so the decision logic is independently testable.
Terminal errors (EBADF/EINVAL/ENOTSOCK) now exit the loop immediately,
allowing the supervisor to detect proxy death. Unknown errors exit
after 10 consecutive failures. FD exhaustion retries with exponential
backoff capped at 5s (fix unreachable cap from min(6) to min(7)).
Adds 9 unit tests covering classification, counter interactions,
backoff progression, and bounded exit behavior.
Signed-off-by: politerealism <burdcat17@gmail.com>
* refactor(proxy): use libc constants and Option-based exit receiver
Replace raw errno numbers with libc:: constants behind #[cfg(unix)]
gates in classify_accept_error and its tests. Refactor
ProxyHandle::exited_rx to Option<Receiver<()>> so take_exit_receiver
uses .take() instead of allocating a dummy sender. Add a
ProxyHandle-level test that verifies the full drop → abort → receiver
fires path.
Signed-off-by: politerealism <burdcat17@gmail.com>
* fix(proxy): restore full transient errno classification from main
The refactored classifier only recognized EMFILE/ENFILE as retryable,
routing all other documented transient accept errors (ENOBUFS, ENOMEM,
ECONNABORTED, ECONNRESET, EINTR, ENETDOWN, etc.) through the unknown-
error budget which terminates the proxy after 10 consecutive failures.
Restore the complete transient errno table from main so these errors
retry indefinitely with appropriate backoff.
Signed-off-by: politerealism <burdcat17@gmail.com>
* fix(sandbox): resolve Clippy warnings in proxy-exit detection code
Replace redundant closure |p| p.take_exit_receiver() with the method
reference ProxyHandle::take_exit_receiver, and change all five
select! arms from _ = &mut proxy_exited to () = &mut proxy_exited
to satisfy clippy::ignored_unit_patterns. Also run cargo fmt on the
new proxy test in proxy.rs.
Signed-off-by: politerealism <burdcat17@gmail.com>
* ci: trigger required check re-run
Signed-off-by: politerealism <burdcat17@gmail.com>
* fix(sandbox): remove unnecessary std::pin qualification
Signed-off-by: politerealism <burdcat17@gmail.com>
---------
Signed-off-by: politerealism <burdcat17@gmail.com>
On Agent Sandbox v1beta1, a `sandbox stop` then `sandbox start` left the
sandbox stuck at `Starting` and `start` timed out after 300s, even though
the workload pod was `Running 1/1` and the CR reported `Ready=True`.
The v1beta1 controller (v0.5.0) sets `Suspended=True (PodTerminated)` on
stop and does not clear it on resume, so a resumed CR carries both
`Ready=True (DependenciesReady)` and a stale `Suspended=True`. The gateway's
`derive_phase` treated any `Suspended=True` condition as `Stopped` before it
consulted `Ready`, so a resumed-but-Ready sandbox was read as not-running and
the `Starting` phase never advanced to `Ready`.
`Ready=True` means the sandbox is usable through this gateway and must win
over a `Suspended=True` condition. Only resolve `Stopped` from `Suspended`
when `Ready` is not `True`; a genuine stop leaves `Ready` unset or `False`,
so it still resolves to `Stopped`. v1alpha1 (which encodes desired state as
`spec.replicas` and emits no lingering `Suspended`) is unaffected.
The underlying stale condition is an upstream bug in the agent-sandbox
controller, fixed by kubernetes-sigs/agent-sandbox#1150 (in v0.5.6). This
gateway change keeps OpenShell resilient regardless of controller version.
Fixes#2932
Signed-off-by: Russell Bryant <rbryant@redhat.com>
The Maturin-based wheel packaging was a historical remnant from when the local gateway launch path and OpenShell CLI were coupled in one binary. The gateway and CLI now ship as standalone artifacts, so the Python distribution should contain only the SDK.
Build a single platform-independent setuptools wheel, verify that it cannot contain native code or an openshell entry point, and simplify the release jobs and documentation for SDK-only PyPI installs.
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* fix(core): fall back to podman CLI when no API socket responds
Auto-detection only checked well-known Podman socket paths, so a Podman
machine exposing its API socket at a non-standard location went
undetected. The symlink at a well-known path is not always present; it
varies by Podman version, machine provider, and platform.
Extend detect_podman_socket() to fall back to podman CLI discovery when
no well-known candidate responds: podman info --format json determines
whether the service is local or remote, and podman machine inspect
resolves the host-side forwarded socket for VM-backed machines. All
existing callers (driver auto-detection, the Podman driver, and the VM
driver's container-engine fallback) pick this up without change.
Select the machine backing the active Podman connection instead of the
first entry from podman machine inspect, honoring podman's connection
precedence: CONTAINER_CONNECTION, then CONTAINER_HOST (mapped to a
connection by URI), then the containers.conf default. An explicit
endpoint that maps to no known machine is left unresolved rather than
guessing an unrelated machine. When CONTAINER_HOST is an explicit unix://
socket, use that path directly since podman info connects through it.
Update the gateway config reference and the Podman driver README, which
described probe-only detection.
Signed-off-by: Russell Bryant <rbryant@redhat.com>
* fix(core): inspect podman machine by name and bound discovery probes
Address two startup defects in Podman socket auto-detection:
- Remote discovery ran `podman machine inspect` with no arguments, which
inspects only `podman-machine-default`. On a host whose default
connection is a different machine, selection fell back to the first
(wrong) entry, so the gateway could operate against the wrong backend.
Resolve the active machine first and inspect it by name, trying the
`-root`-stripped machine for rootful connections and returning None
when no machine can be mapped instead of substituting another.
- `podman info`, `podman machine inspect`, and `podman system connection
list` used unbounded `Command::output()`, so a stalled machine, SSH
connection, or helper could hang gateway startup indefinitely. Route
all three through a bounded runner that kills and reaps the child on a
documented deadline and returns None so detection can continue.
Signed-off-by: Russell Bryant <rbryant@redhat.com>
* fix(core): bound podman probe stdout drainage and kill descendants
The previous timeout only bounded waiting for the direct child to exit.
Draining stdout still called read_to_end, which waits for pipe EOF — a
Podman probe can leave a daemonized descendant (SSH multiplexer, gvproxy)
that inherited the stdout pipe, so drainage could wait indefinitely even
after the direct child exited, defeating the deadline.
Run each probe as its own process-group leader, cap stdout drainage by
the same deadline via a channel, and on expiry kill the whole process
group so descendants holding the pipe are terminated and EOF is reached.
Add a regression test where the direct child exits but a backgrounded
descendant keeps stdout open.
Signed-off-by: Russell Bryant <rbryant@redhat.com>
* fix(core): make podman probe deadline absolute against escaped descendants
The prior drainage timeout still ended in an untimed rx.recv() after
killing the process group. A descendant that escaped the probe's process
group (e.g. via setsid) while holding the stdout pipe would not be killed,
so read_to_end never saw EOF, the reader never sent, and that recv() could
block gateway startup indefinitely.
On drainage timeout, best-effort kill the process group and return None
immediately, abandoning the reader thread instead of waiting on it again.
The deadline now bounds the whole call regardless of what descendants do.
Add a regression test whose stdout-holding descendant escapes the process
group (`set -m`) and assert the call still returns promptly.
Signed-off-by: Russell Bryant <rbryant@redhat.com>
---------
Signed-off-by: Russell Bryant <rbryant@redhat.com>
The podman_gateway_start e2e test shells out to `podman` directly (unlike
every other Podman e2e test, which routes through the gateway). The harness
`e2e/with-podman-gateway.sh` clobbers `XDG_CONFIG_HOME` with an empty dir to
isolate CLI/SDK gateway metadata. On macOS the podman client resolves its VM
connection config from `XDG_CONFIG_HOME`, so a bare `podman ps` falls back to a
nonexistent native rootless socket and fails with "unable to connect to Podman
socket". On Linux the socket resolves via `XDG_RUNTIME_DIR`, so the test passes
there and this is a macOS-only false failure.
Target the same API socket the gateway uses by passing `--url unix://$SOCKET`
when the harness-exported `OPENSHELL_PODMAN_SOCKET` is set, mirroring the
shell's `podman_cmd` helper. When the var is unset (running the test outside the
harness), fall back to plain `podman`, so Linux behavior is unchanged.
Signed-off-by: Russell Bryant <rbryant@redhat.com>
* feat(sandbox,podman): trust corporate CA for https:// proxies and intercepted TLS
The corporate proxy chaining only accepted plain http:// proxy URLs, so
operators whose forward proxy terminates TLS with a private corporate CA
had no way to reach it, and TLS-intercepting proxies (mitmproxy, squid
ssl-bump) that re-sign tunneled server certificates broke every upstream
handshake after CONNECT.
The supervisor now accepts https:// proxy URLs: it wraps the connection to
the proxy in TLS before the CONNECT handshake, verifying the proxy
certificate against the built-in Mozilla roots, the system CA bundle, and an
optional operator corporate CA bundle. The upstream dial returns a
Plain/Tls stream enum consumed generically by the relay paths.
The corporate CA is delivered as a driver-supplied command-line argument
(--upstream-proxy-ca-bundle), never an environment variable, matching the
hardened proxy-config model where a sandbox image cannot influence the
operator's egress boundary. It is folded into the sandbox combined trust
bundle (write_ca_files) and the L7 upstream verification store
(build_upstream_client_config) at startup, so intercepted upstream
handshakes succeed and sandbox workloads trust the re-signed certificates.
Configuration is fail-closed: a CA bundle set without a proxy, or an
unreadable or certificate-free file, is fatal rather than silently
weakening the trust boundary.
The shared parse_upstream_proxy_url validator accepts https:// (recording
the scheme so the driver and supervisor agree), keeping the explicit-port
requirement. The Podman driver gains a proxy_ca_bundle operator setting
(TOML, --sandbox-proxy-ca-bundle, OPENSHELL_SANDBOX_PROXY_CA_BUNDLE) that
bind-mounts the host PEM read-only into the sandbox (a CA certificate is not
secret) and points --upstream-proxy-ca-bundle at it, with a create-time
readability check.
The standalone dev gateway task passes OPENSHELL_SANDBOX_PROXY_CA_BUNDLE
through to the generated podman config, so a local gateway can be pointed at
a TLS-intercepting proxy without hand-editing the regenerated TOML.
Refs #1792
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(sandbox): reject CA bundles with valid PEM framing but invalid X.509 DER
The proxy CA bundle validation counted PEM blocks that base64-decoded
successfully, but did not verify the decoded bytes were accepted as
trust anchors by RootCertStore. A bundle with syntactically valid PEM
framing but invalid DER would pass the startup check while contributing
zero usable anchors, causing opaque TLS failures at runtime instead of
a fail-closed startup error.
Validate decoded certificates through RootCertStore::add_parsable_certificates
and reject the bundle unless at least one is accepted.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(sandbox,podman): allow proxy auth without insecure acknowledgement for https:// proxies
For an https:// proxy the Proxy-Authorization credential travels inside
the verified TLS session, so the proxy_auth_allow_insecure
acknowledgement is unnecessary. Previously both http:// and https://
proxies required it, producing a misleading cleartext-risk diagnostic
for a path that is already encrypted.
Skip the requirement when the proxy URL uses https://; the
acknowledgement is still tolerated if set. Updated in both the
supervisor and Podman driver validation paths, with docs and tests.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix: format
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(kubernetes): use truly unsupported scheme in proxy validation test
https:// is now a supported proxy scheme after a13c4dce, so the
unsupported-scheme test must use a genuinely unsupported scheme.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(e2e): sign the https proxy fixture listener cert with a CA
The corporate-proxy E2E fixture served a single `openssl req -x509`
certificate as its TLS listener identity. OpenSSL marks that certificate
`basicConstraints: critical, CA:TRUE`, and rustls refuses a CA
certificate presented as an end-entity certificate (CaUsedAsEndEntity).
The supervisor's TLS handshake with the proxy therefore failed, the
upstream dial errored, and the workload's CONNECT was dropped without a
response, so podman_corporate_proxy_trusts_ca_bundle_for_https_proxy
failed on the approved destination while policy denial still worked.
Generate a corporate CA and a separate listener leaf signed by it, serve
the leaf chain, and publish only the CA as the bundle the supervisor
trusts. This is what an intercepting proxy actually presents, and it
exercises the corporate-CA trust path rather than pinning the listener
certificate itself.
Refs #1792
Signed-off-by: Philippe Martin <phmartin@redhat.com>
---------
Signed-off-by: Philippe Martin <phmartin@redhat.com>
Mirror the VM and Podman driver tracing setup for Docker. Export Docker
driver spans through OTLP/gRPC as the distinct openshell-driver-docker
service, preserve gateway trace context, record lifecycle and asynchronous
provisioning operations, and report gRPC failures.
Docker currently runs in-process when selected as a built-in gateway driver.
Add the same temporary server-boundary shim used by Podman so traces retain
the shape they will have when Docker moves to a separate process. Generalize
the gateway provider selection for both in-process drivers and share the OTLP
collector fixture across Docker, Podman, and VM tracing tests.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(nix): add glibc 2.28 development shell
* feat(nix): add musl development shell
* feat(build): use mold in musl development shell
* feat(flake): add nix remote cache
* feat(build): use mold in default development shell
* feat(provider): add ability to request token exchange instead of client credentials as OAuth grant_type
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(proxy): add further tests for token exchange
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(provider): add runnable example for token exchange
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(e2e): cover Podman token exchange grants
Signed-off-by: Gordon Sim <gsim@redhat.com>
* refactor(oauth): extract duplicated functionality from server and supervisor
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(provider): evict nearest-to-expiry entry from intermediate token cache
Signed-off-by: Gordon Sim <gsim@redhat.com>
* doc(supervisor): add podman example for token exchange
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(provider): withhold token-exchange subject credentials
Signed-off-by: Gordon Sim <gsim@redhat.com>
---------
Signed-off-by: Gordon Sim <gsim@redhat.com>
Evan Lezar