* feat(ci): detect breaking protobuf changes
Compare the proto module against the PR or merge-group base and report Buf violations in Branch Checks. Add local reproduction and fixture coverage.
Closes#3794
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
* fix(ci): pin protobuf check container image
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
* fix(ci): qualify protobuf compatibility by release train
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* refactor(ci): reuse protobuf compatibility action
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* refactor(ci): run protobuf checks as a Nix app with one ref
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
---------
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Co-authored-by: Mrunal Patel <mrunalp@gmail.com>
* test(tmachine): add K3s conformance scenario
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* refactor(tmachine): use Helm values file for K3s installer
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(tmachine): run K3s conformance in integration jobs
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(tmachine): verify installer scripts and document version baseline
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
---------
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Remove architecture/. It was a constant source of merge conflicts, became
an effectively append-only log of the project, and was of dubious value.
Design records live in rfc/, crate details in crate READMEs, and user
documentation in docs/.
Move the git-ignored plans directory from architecture/plans to plans/,
keeping the old .gitignore entry. Remove the arch-doc-writer agents and
update AGENTS.md, CONTRIBUTING.md, skills, the feature request template,
and links in proto/, rfc/, and examples/ that pointed into architecture/.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* docs: align page file names and nav labels with published URLs
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* docs: redirect moved dev pages and fix agent guide redirects
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* ci(docs): check that page URLs match their file paths
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* fix(docs): align navigation checks with repo conventions
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* docs: omit historical redirect aliases
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* fix(docs): preserve published overview and TypeScript URLs
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* fix(docs): redirect unversioned overview and TypeScript URLs
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
---------
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* fix(snap): require mTLS for the snap gateway
Replace the installer opt-in with an authenticated snap gateway. The wrapper
no longer forces plaintext, so the gateway serves TLS from the bundle it
already generates in $SNAP_COMMON/tls. The install hook writes a config that
enables mTLS user auth instead of unauthenticated access, and a new
post-refresh hook migrates the exact legacy default on existing installs.
install.sh waits for the gateway, detects whether it serves TLS, copies the
client bundle into the target user's snap state directory, and registers the
gateway over HTTPS. Older plaintext snap revisions still register over HTTP
with a warning. The release canary asserts mTLS auth and HTTPS registration.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): pass config preflight and detect the mTLS gateway reliably
An explicit [openshell.gateway.mtls_auth] table fails config preflight,
which validates mTLS auth before the local TLS bundle supplies the client
CA. Write a default that pins the Docker driver instead; with the wrapper's
TLS bundle the gateway requires client certificates and enables mTLS user
auth automatically, as the native packages do.
The mTLS gateway rejects TLS handshakes without a client certificate, and
it still answers plaintext loopback HTTP for sandbox service routing, so the
installer could misdetect it as a legacy plaintext gateway. Probe HTTPS with
the root-owned client bundle, and treat a gateway as legacy only when a
plaintext gRPC Health call succeeds.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* chore(snap): simplify install hook comment
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): migrate insecure gateway configs on refresh
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(snap): simplify mTLS detection and config migration
Detect the mTLS snap from the installed revision's post-refresh hook instead
of probing plaintext gRPC, and drop the scheme global. Remove the installer's
pre-hook config fallback, which is dead now that every channel ships the
install hook and which wrote the insecure default. Give the install hook a
single write path with a simple backup name, and shorten the manual client
certificate steps.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): stop keeping a copy of replaced insecure configs
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(install): make the snap an opt-in install method
Stop selecting the OpenShell snap just because the snap command exists. Linux
installs default to the Debian or RPM package; OPENSHELL_INSTALL_METHOD=snap
(or deb, rpm) selects the package explicitly. Hosts that already have the
OpenShell snap keep refreshing it rather than gaining a second gateway on the
same port. The release canary and snap repro script opt in explicitly.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): let the gateway auto-detect its compute driver
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): restart the gateway after refresh
Published revisions use refresh-mode: endure, and snapd honors the old
revision's setting during a refresh, so the plaintext gateway kept running
with the migrated config unused until a manual restart. Restart the gateway
from the post-refresh hook so the mTLS config takes effect immediately.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(snap): drop refresh notes from the snap description
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(snap): trim snap refresh notes from installation docs
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(fern): publish and order versioned release docs
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(fern): remove version availability badges
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(fern): keep version badges optional
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
---------
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* test(podman): run driver-podman userns suite against rootful Podman too
The driver-specific-integration job only ran the driver-podman testsuite
(default/auto/keep-id/private userns reference checks) against
fedora-podman-rootless, leaving rootful behavior for this scenario
unverified even though the compute driver auto-detects and explicitly
supports rootful Podman.
The default-userns-baseline and userns-profile playbooks hard-asserted a
rootless tmachine gateway user, so pointing them at a rootful environment
would have failed that assertion immediately rather than exercising
anything. They now detect rootful vs. rootless via the existing
tmachine_container_runtime role and branch the reference-capture user
accordingly, while keeping the captured reference file itself owned by
tmachine, since the archived test binary that reads it back always runs
unprivileged as tmachine regardless of daemon mode.
Signed-off-by: politerealism <burdcat17@gmail.com>
* test(podman): add real-daemon coverage for resource limits and daemon failure
Neither the Podman driver's resource-limit enforcement nor its behavior
when the Podman daemon is unreachable had any test coverage against a
real daemon; both were only exercised through unit tests against a
mocked Podman client.
podman_resource_limits.rs creates a sandbox with --cpu/--memory flags and
reads /sys/fs/cgroup/memory.max and cpu.max from inside the sandbox
itself, verifying the limit is actually enforced rather than just echoed
back by the template API. Expected values are cross-checked against the
driver's own parse_cpu_to_microseconds/parse_memory_to_bytes and against
a real local `podman run --cpus/--memory` container.
podman_preflight.rs spawns the standalone openshell-driver-podman binary
against a guaranteed-nonexistent Podman socket and asserts it exits
non-zero within its bounded retry window with an actionable error naming
the socket path, rather than hanging or failing silently.
Signed-off-by: politerealism <burdcat17@gmail.com>
* test(podman): make rootful userns and cgroup checks pass
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* test(podman): match lifecycle containers by isolation role label
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* test(podman): accept non-expiring bootstrap tokens
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: politerealism <burdcat17@gmail.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
* fix(install): avoid installing incompatible docker snap
The work to land RFC-0012 added new restrictions when interacting with
Docker by setting `NoNewPrivs`. This prevents the `docker` snap from
transitioning its AppArmor profile from `snap.docker.dockerd` to
`docker-default` when it tries to launch a container. Thus, the `docker`
snap is currently incompatible with OpenShell.
This commit prevents `install.sh` from installing the `docker` snap
before installing the `openshell` snap, and instead requires the user to
install a non-snap Docker daemon before proceeding with installing the
snap. Systems without the `snap` command are unaffected, since they
install native packages without checking for the presence of Docker or
other compute providers.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): only snapd 2.76 for openshell snap since store installs work
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fixup! fix(install): avoid installing incompatible docker snap
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fixup! fix(snap): only snapd 2.76 for openshell snap since store installs work
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
---------
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(podman): restore host gateway alias mediation
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(podman-e2e-tests): enable broader test podman e2e coverage
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(tests): make test more reliable
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(podman): fix macos linting error
Signed-off-by: Gordon Sim <gsim@redhat.com>
---------
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(snap): update stale snap docs and tests
Previously, the `openshell` snap required the `docker` snap. Now, it
works with any Docker daemon running on the system. Furthermore, the
`snap-declaration` assertion on the `openshell` snap when installed from
the Snap Store causes the `openshell` snap to always connect to the
system `:docker` slot, rather than a slot provided by the `docker` snap.
This commit updates the documentation, including the `description` field in
`snapcraft.yaml`, to ensure that all information is correct and
up-to-date.
Additionally, some tests connected the `openshell:docker` plug to the
`docker` snap's `docker:docker-daemon` slot, which is inconsistent with
how the `openshell` snap operates when installed from the store. Update
those tests to connect to the system `:docker` slot as well.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): require snapd 2.76 for openshell snap
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(nix): align snap gateway reproducer timeout with release-canary
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): require snapd 2.77 for openshell snap
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fixup! fix(snap): require snapd 2.77 for openshell snap
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* feat(snap): install openshell snap via install.sh when snap available
Change `install.sh` to install the `openshell` snap by default when
snapd is installed on the host. This installs the snap from the
`latest/stable` channel, which should match the most up-to-date release
tag on github.
If `OPENSHELL_VERSION=dev` is set for `install.sh`, then it will install
the `openshell` snap from the `latest/edge` channel, which matches the
latest dev release available on github.
The `openshell` snap currently requires Docker in order to function. If
a Docker daemon is already installed on the system, it will be used by
the `openshell` snap. Otherwise, `install.sh` will install the `docker`
snap first, wait for the Docker daemon to be ready, and then install the
`openshell` snap.
Also, update the `release-canary.yml` to split the `ubuntu-snap` job
into `ubuntu-snap-system-docker` and `ubuntu-snap-provisions-docker`,
which test the two aforementioned scenarios. Previously, `ubuntu-snap`
manually installed a given snap artifact as built from CI, but with
these new jobs, it instead uses `install.sh` to install the published
`openshell` snap from the `latest/edge` track, thus matching the
behavior of the other release canary jobs.
Make corresponding changes to the `nix` guest reproducer, documentation,
and tests.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): configure local gateway authentication
The `openshell` snap runs the gateway as a systemd system service, which
runs as root. Thus, the mTLS certs are generated by root and stored in a
root-owned directory to which non-root users do not have access. For
this reason, the snap's `openshell-gateway-wrapper` script sets
`OPENSHELL_DISABLE_TLS=true`.
This commit ensures that the `openshell` snap's gateway allows
unauthenticated local access by writing a default `gateway.toml`
configuration file during the install hook, which runs after the snap is
first installed but before services are started. The config file
contains sets `allow_unauthenticated_users = true`.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(install): configure snap gateway authentication
Recently, a new install hook was added which writes a default config
file for the `openshell` snap to allow unauthenticated local access to
the gateway. This is because the gateway service runs as root and the
mTLS certificates are not accessible to non-root users.
However, the `install.sh` script installs the `openshell` snap from the
snap store, and the published version may not yet have that new install
hook. Or, the user may already have the snap installed, in which case
the install hook does not run. In either case, we need `install.sh` to
ensure that the config file is written to set the gateway auth to
`allow_unauthenticated_users = true`, and then restart the openshell
gateway service.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fixup! feat(snap): install openshell snap via install.sh when snap available
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
---------
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
PRs labeled test:e2e-kubernetes now run the Kubernetes HA suite and the
Kubernetes credential-driver suite (Kubernetes Secrets and Vault). Both stay
optional and off in merge groups.
- Read the test:e2e-kubernetes label for the HA and credential-driver
lanes in Branch E2E Checks.
- Point gator at test:e2e for Helm and Kubernetes coverage and at
test:e2e-kubernetes for gateway high availability and credential
driver storage.
Refs #3481
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(sandbox): default to official Alpine sandbox image
default_sandbox_image() now returns docker.io/library/alpine:3.22, a generic
version-qualified official image, so a fresh install no longer depends on the
community sandbox image catalog. All compute drivers (docker, podman,
kubernetes, vm) inherit this fallback.
Part of #3116.
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
* feat(deploy): default deployment configs to the official Alpine sandbox image
Update the shared gateway default_image, Helm chart values, the standalone
Kubernetes manifest, and the dev gateway task scripts to use
docker.io/library/alpine:3.22 instead of the community base image, consistent
with default_sandbox_image(). GPU e2e image-build base is left unchanged (CUDA
needs a glibc base).
Part of #3116.
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
* feat(driver): default to numeric non-root identity for USER-less images
With the default sandbox image now Alpine, images that declare no OCI USER
must start instead of being rejected. When the image declares no USER and
the policy requests none, the Podman and Docker drivers now supply a numeric
non-root identity (DEFAULT_SANDBOX_UID/GID = 1000) instead of rejecting,
matching the numeric-identity behavior of the Kubernetes and VM drivers. The
supervisor's resolved-identity path runs the sandbox as a synthesized
non-root account without the account existing in the image. Images that
declare a USER keep the OCI resolution path unchanged.
Part of #3116.
Signed-off-by: Akram <akram.benaissi@gmail.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(conformance): use Alpine workload image
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* refactor(policy): drop community image /app path from default policy
The restrictive default policy granted read-only access to /app, a directory
that only existed in the community base image. A generic Alpine default has no
/app, so remove it. Landlock best-effort already ignores absent paths; this
just stops advertising a community-specific layout in the default.
Part of #3116.
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
* docs(config): document Alpine default images
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(podman): report early sandbox termination
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(podman): initialize rootless workspace ownership
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(sandbox): qualify NVIDIA Ubuntu default
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(podman): initialize rootful default workspace
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(sftp): add native sandbox adapter
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sftp): gate runtime helper support to Linux
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sftp): support standard OpenSSH file operations
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sftp): harden rename and special file handling
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(runtime): remove community image dependencies
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* test(e2e): build provider readiness tool fixture
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(e2e): use a dedicated Noble fixture for Docker tests
Signed-off-by: Evan Lezar <elezar@nvidia.com>
---------
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
* feat(kubernetes): support HA gateway rebalancing
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* perf(server): cache peer connections, tokens, and owner lookups
Every forwarded relay rebuilt its setup from scratch: an owner lookup, a
blocking read of the peer token, a TLS connect to the owning replica, and
a TokenReview plus Pod GET on the receiving side. Sandbox service routing
does this per HTTP request, so the apiserver calls scaled with traffic.
Cache all of it on ServerState:
- peer channels pooled per endpoint, so relays multiplex over one
connection instead of redialing
- peer tokens keyed by SHA-256, expiring at min(ttl, token exp) so a hit
cannot accept an expired token
- owner records for 3s against a 45s ownership TTL, still freshness
checked before use
Entries are evicted when a relay fails. Also raise HTTP/2
max_concurrent_streams to 1024, since pooling funnels every relay between
two replicas onto one connection and hyper's default of 200 sits below
the 256 pending-relay budget.
Signed-off-by: divesh <dgude@nvidia.com>
* perf(server): pool upstream connections for sandbox services
Each HTTP request to a sandbox service opened its own supervisor relay,
paying a new TCP connection and HTTP/1 handshake every time. Worse, it
counted against the 32 in-flight relay cap, so a service handling more
than 32 concurrent requests failed outright.
Pool idle upstreams per endpoint and port, up to 8 each for 15s. Reuse is
safe because the pool only returns a connection hyper reports as ready,
and HTTP/1 cannot start a request until the previous body has drained.
Upgrades are never pooled since they take the connection over, and a
failed send evicts that endpoint. Pruning is bounded per key, with the
full sweep limited to once per 30s.
Signed-off-by: divesh <dgude@nvidia.com>
* fix(server): address HA gateway review findings (#3449)
- Let a gateway own supervisor sessions without a peer endpoint. Requiring
one whenever the store is PostgreSQL broke every single-instance
PostgreSQL deployment, because no sandbox supervisor could connect.
A cross-replica request to an owner that advertises no endpoint now fails
immediately naming the cause, instead of retrying until the wait timeout.
- Close a supervisor session on heartbeat only when another replica owns it,
or after renewals fail for the ownership TTL. A database error no longer
drops every session heartbeating during an outage.
- Clamp owner record ages at zero so a skewed or corrupt stored timestamp
cannot produce a negative age.
- Bound the cross-object advisory lock with a lock timeout, so a stuck holder
fails instead of blocking every mutation in the fleet.
- Refuse to start when a peer endpoint is configured on a multi-replica
backend but peer authentication is unavailable, and warn when a
multi-replica backend has no peer endpoint at all.
- Reject a plaintext peer endpoint when the gateway serves TLS.
- Skip the sandbox watch poller on single-replica backends, where the local
update bus already sees every write.
- Rate-limit the peer owner cache sweep so an insert no longer scans the
whole map under the lock.
- Retry GET and HEAD on a pooled upstream the sandbox closed, instead of
returning 502, and drop an emptied endpoint from the pool right away.
- Document the gateway peer environment variables and the post-rollout
ownership skew operators should expect.
Signed-off-by: divesh <dgude@nvidia.com>
* fix(server): harden HA supervisor ownership
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: divesh <dgude@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: divesh <dgude@nvidia.com>
Co-authored-by: Divesh Chowdary <47188680+FrostGod@users.noreply.github.com>
* ci(release): advance seeded prereleases daily at Zurich time
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(release): schedule prereleases on weekdays
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
---------
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* Implement Windows host proxy integration and update dependencies for OpenShell
* Update README and gateway config to clarify egress proxy address handling and allocation
* Refactor ProxyIdentityMode to return Result for static_binary and add tests for binary path and SHA256 hash
* Enhance platform_hosts_path for Windows to use SystemRoot and improve error handling for hosts file reading
* Refactor FileFingerprint to use Option for mtime and ctime, simplifying metadata handling
* Add conditional compilation for Windows host module
* add unit tests for OPA policy evaluation and identity handling
* remove openshell-supervisor-network from unsupported driver package test exclusion list
* feat(mxc): enable host proxy TLS state generation
Generate per-sandbox TLS state for the MXC host proxy so HTTPS L7 enforcement can use the same MITM path as Linux. Grant generated CA material to the MXC process and inject standard trust env vars, while matching Linux behavior by disabling TLS termination on CA setup failure and relying on proxy fail-closed handling.
* fix(docs): remove outdated notes on governed egress from docs
* fix(tests): update TLS environment variable paths to use temporary directory
* fix(examples): make run-mxc-e2e harness correct and orphan-free
The MXC e2e harness never actually exercised the fs scenarios: it started
the gateway once and patched agent_command per scenario AFTERWARDS, so the
running gateway kept launching the default demo agent (not shipped in the
kit) and every fs scenario failed with CreateProcessW error:2. It also
scored on the `sandbox create` exit code (non-zero due to the harmless
interactive attach), wrote sandbox records to the persistent gateway DB
(leaving orphans that collided on later runs), and its deny scenarios never
proved denial.
Changes:
- Start a FRESH gateway per scenario so each scenario's agent_command is
actually loaded (root cause of CreateProcessW error:2).
- Score by on-disk artifact / expected outcome, not `sandbox create` exit.
- Real deny assertions: a control write to a granted path must succeed
(proves the agent ran) while the denied write must be absent. fs-empty
probes an ungranted out-of-share path (share_dir is mapped rw by design).
- Run the gateway on an ephemeral in-memory DB (sqlite::memory:) so the
harness never writes to the persistent store and cannot leave orphan
sandbox records; also use unique per-run sandbox names + pre-delete.
- Fix the process_container probe: use a real cwd + absolute cmd.exe
(canonical wxc-exec does not expand %TEMP% -> 0x8007010B).
- Fix summary counts (@() so a single FAIL is counted and exit is non-zero).
Verified PASS=4 FAIL=0 on 7F203-MXC-003 (no BaseContainer velocity keys)
using a canonical wxc-exec build (AppContainer fallback).
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(e2e): probe timeout is milliseconds (10ms->30000ms)
MXC process.timeout is wall-clock ms (wire.rs). The 10 value meant 10ms,
which the base-container tier (7F203-MXC-001/.181) enforced strictly and
timed the probe out. AppContainer path (.18/-003) happened to slip under
it. Bump to 30000ms so the process_container preflight probe is reliable
across both tiers.
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(mxc): use native paths in real runtime probes
Signed-off-by: Shailendra Singh <shailendras@nvidia.com>
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(mxc): make processcontainer work with mxc-latest-released wxc-exec
Three fixes to support the release wxc-exec binary (BaseContainer dispatcher)
in addition to mxc-fixes-env-vars:
1. Seed process env from host (driver.rs)
ProcessContainer starts with a completely blank environment -- no PATH,
SystemRoot, or anything. Seed the process env from the gateway host
environment so the agent binary can locate DLLs and run. Skip internal
Windows drive-letter variables (keys starting with '=') which cause
CreateProcessW to return ERROR_ENVVAR_NOT_FOUND. User agent_env entries
and TLS CA vars are applied as overrides on top of the host env.
2. Remove TLS readonly_paths grant (driver.rs)
The release wxc-exec (BaseContainer dispatcher) requires write-DAC
permission on every path in readonly_paths to set up AppContainer ACLs.
Adding the proxy's temp TLS directory caused a DACL error and exit -1.
The CA cert paths remain available to the agent via TLS env vars.
3. Remove allowedHosts from network JSON (mxc.rs)
The release wxc-exec rejects network.allowedHosts / network.blockedHosts
on Windows with "not yet supported". Removed the loopback exemption
attempt (127.0.0.1, ::1, localhost) from the network section.
Intra-container loopback works natively in the release binary without
it -- the spawner can connect to the server at 127.0.0.1:22000 directly.
Additional changes:
- mxc-ws-agent.rs: add relay-debug.txt error capture and relay-ready.txt
marker for reliable timing of host client connections.
- mxc-ws-gateway.toml: debug = true for JSON config dump during diagnosis.
- run-ws-agent-test.ps1: default port changed to 17670 (gateway default);
relay-ready.txt polling before ws-echo to avoid connecting before the
spawner has established the proxy bridge.
Signed-off-by: Prashant Khodade <pkhodade@nvidia.com>
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(e2e): address CodeRabbit review on run-mxc-e2e.ps1 (MR !46)
Four robustness/correctness fixes from CodeRabbit:
1. Start-Gw: kill the spawned gateway before the "did not start within 30s"
throw. If the process is alive but never binds the port, $gw is not yet
assigned in the caller, so the finally block cannot reap it -> orphan
gateway holding the port for the next run.
2. create-fail scoring: a non-zero `sandbox create` exit alone is not proof
of a policy rejection (gateway-registration/transport/fixture errors also
exit non-zero and would false-pass). PASS now requires a genuine
rejection signal (network / invalid_argument / network_policies) AND that
it is not an infrastructure failure; other non-zero exits go to FAIL with
output captured.
3. deny scenarios (ControlTarget path): snapshot the deny target AFTER
Wait-File lands the control artifact, so a late denied write (enforcement
regression racing the control write) can no longer be recorded as PASS.
4. -KeepRunning: break out of the scenario loop after the first scenario so
a later scenario does not start a second gateway on the same port
(previously a reliable port collision instead of a usable debug mode).
Re-verified PASS=4 FAIL=0 on both boxes (7F203-MXC-001 base-container and
7F203-MXC-003 AppContainer fallback); network-policy-rejected correctly
scores as "policy rejection".
Signed-off-by: Akber Raza <akberr@nvidia.com>
* feat(mxc-e2e): collect run-mxc-e2e output into a results bundle
Mirror the sibling run-*.ps1 scripts by collecting every run's logs into a
timestamped results-e2e-<stamp>\ folder and zipping it. The bundle contains the
console transcript, per-scenario gateway stdout/stderr, the exact TOML rendered
for each scenario, the policy fixture used, and a summary.txt with the verdict
table.
Per-scenario gateway logs now land in gateway.<scenario>.log/.err.log inside the
bundle instead of a single fixed gateway.e2e.log in the script directory.
Wrap pre-flight, mode setup, scenario definitions, and the scenario loop in a
single try/catch/finally so the finally always writes the summary, stops the
transcript, and zips the bundle -- even on a pre-flight failure. The existing
per-scenario gateway-cleanup try/finally stays nested inside. All scenario
logic, scoring rules, and comments are preserved.
Signed-off-by: Prashant S Khodade <pkhodade@nvidia.com>
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(mxc-e2e): address CodeRabbit review on run-mxc-e2e.ps1
- Require -Scenario when -KeepRunning: the loop breaks after the first
scenario, so a full-suite run would execute only one scenario yet still
report the suite as PASS. Fail fast so a partial run can't be mislabeled
complete.
- Start-Transcript now runs inside the guarded try block with a
$transcriptStarted flag; Stop-Transcript is only called when it actually
started, so a Start-Transcript failure still yields the results bundle.
- Wrap the -Scenario filter in @() so a single exact match stays an array
(reliable .Count and a proper array for the scenario loop on PS 5.1).
Signed-off-by: Prashant S Khodade <pkhodade@nvidia.com>
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(examples): pass gateway config via OPENSHELL_GATEWAY_CONFIG for spaced paths
Start-Process -ArgumentList does not quote array elements, so launching the
gateway with a bare --config <path> token split on any space in the install
path (e.g. C:\Users\First Last\...), and clap rejected the fragment with
'unrecognized subcommand'. Every MXC example launcher that started the gateway
hit this when the kit was unzipped under a path containing a space.
Pass the config path through the OPENSHELL_GATEWAY_CONFIG env var (which the
gateway already reads via clap) and drop the --config token. Env vars carry
spaces safely.
Affected: run-ocsf-audit, run-mxc-e2e, run-demo, run-inference-test,
run-ollama-test. run-mtls-test was not affected (its launch passes no config
path). Root-caused and fix-verified on 7F203-MXC-003 from a spaced path.
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(run-mxc-e2e): improve scoring logic and enhance command execution handling
* fix(mxc): reconcile proxy support after rebase
Restore the proxy-enabled OCSF audit example removed by 13185f6e now that the host CONNECT proxy is present. Adapt the proxy lifecycle test to the target branch's DriverSandboxSpec policy delivery contract.
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(mxc): grant sandbox access to proxy CA
- Share the per-sandbox public CA bundle with the AppContainer
- Add real wxc-exec HTTPS proxy coverage and document trust isolation
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(mxc): reject unsupported network middleware
- Reject middleware-bearing MXC policies before sandbox lifecycle begins
- Guard host proxy startup and document the unsupported registry path
- Add mapper, lifecycle, and host proxy regression coverage
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(mxc): reconcile host proxy with main
- remove obsolete inference routing from the host proxy adapter
- use the workspace AWS-LC provider in host-proxy tests
- adapt the forward-proxy test to ProxyIdentityMode
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(build): switch to bundled Z3 for Windows MSVC builds
* fix(mxc): reconcile host proxy after rebase
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Akber Raza <akberr@nvidia.com>
Signed-off-by: Shailendra Singh <shailendras@nvidia.com>
Signed-off-by: Prashant Khodade <pkhodade@nvidia.com>
Signed-off-by: Prashant S Khodade <pkhodade@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Jamie King <jamiek@nvidia.com>
Co-authored-by: Shailendra Singh <shailendras@nvidia.com>
Co-authored-by: Prashant Khodade <pkhodade@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
Evan Lezar