* feat(tmachine): add the Session API and a test-runner package
Add a library API that boots an environment, runs commands in it with timeouts,
copies files in and out, collects diagnostics and tears it down, so a caller can
drive a machine without writing an Ansible playbook per suite. SSH and exec
timeouts and QEMU cleanup on SIGTERM keep a hung guest or a cancelled CI job from
leaving QEMU running.
Add the `openshell-test-runner` flake package, which runs tests/runner with the
toolchains and VM tools it needs.
The Session API and test-runner package are adapted from NVIDIA/OpenShell#4629.
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* build(cargo): add a check that aligned workspaces lock shared crates like the root
`cargo --locked` only checks a lockfile against its own manifest. A workspace
outside the root that builds root crates through path dependencies, such as the
testkit suites building `openshell-sdk`, has its own Cargo.lock, so the two
lockfiles can resolve a shared crate to different versions without any check
failing. That is how aws-lc-sys 0.46 was picked up in #4629, which does not link
against the AWS-LC that Nix provides.
check_cargo_lockfile_alignment.py fails when an aligned workspace locks a
registry crate that the root lockfile also contains at a version the root
lockfile does not have. With --fix it re-locks the workspace starting from the
root lockfile. tests/testkit is the first aligned workspace.
The script and its tests come from #4629 (a37ebe917).
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(conformance): add a sandbox smoke suite built on testkit
Add the `sandbox` crate to the conformance workspace with one testkit test that
creates a sandbox, runs `echo` in it with exec, deletes it and checks it is gone.
It is deliberately small: it exists so the runner can run a real test in a
tmachine environment on CI before the existing suites move to testkit.
The workspace's other crate, the CLI conformance suite, keeps running as it does
today. The conformance workspace joins the lockfile alignment check, so its
lockfile is realigned with the root lockfile.
The test is adapted from the `sandbox-lifecycle` suite in #4629.
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* feat(runner): add a test runner that runs a test family on tmachine
`openshell-test-runner` builds a family's packages (a workspace under
tests/suites/) on the host as a nextest archive for the guest's musl target,
boots one tmachine environment by name with the named installer through the
Session API, runs nextest in it and brings back the JUnit report and log. No
per-suite Ansible playbook is involved:
nix run .#openshell-test-runner -- --environment ubuntu-docker-rootful \
--installer deb --family conformance --package sandbox
The runner points the tests at the system under test with environment
variables: OPENSHELL_GATEWAY (the gateway the installer registered, `--gateway`,
default `openshell`) and OPENSHELL_BIN (the installed CLI). It waits for the
gateway with `testkit-ready` before the tests start. Nextest runs with
`--test-threads 1` and a slow-timeout that terminates a hung test.
A run passes if nextest succeeds and its JUnit report shows at least one test
and no failures. `--filter` narrows the tests with a nextest filterset, and
`--keep` leaves the VM up for debugging. Results from an earlier run are
removed first, and the guest's diagnostics are collected after a failure.
Which tests run in which environments is left to the caller for now; there are
no plans, environment registry or per-test requirements, and a broken test is
`#[ignore]`d.
Most of this comes from #4629, adapted to testkit finding the gateway the way the
CLI does instead of reading a target file.
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(tests): lint and test the runner, tmachine and conformance suite
Run fmt, clippy and nextest for tests/tmachine and tests/runner, and fmt and
clippy for the new `sandbox` suite crate, wherever tests/testkit already runs
them: the mise tasks and Branch Checks. The suite's tests need a gateway, so they
run through `mise run e2e:testkit` against a local Docker gateway and, on CI,
through the test runner rather than the unit-test job.
The lockfile task and the Cargo lockfiles job also run the alignment check, and
its own tests run with `mise run test`.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(tests): run testkit suites on tmachine from a reusable E2E Suites workflow
Add a reusable workflow with an explicit matrix: each row names a tmachine
environment and installer, a family and package, and the runner label. Each
row runs `nix run .#openshell-test-runner` once. The jobs use the same
integration inputs as the existing tmachine lanes, cache the tmachine disks
keyed on the Ansible tree, upload the JUnit report and logs whether or not the
run passes, and an aggregator job reports the overall result.
The only row runs the conformance sandbox suite on amd64 Ubuntu with rootful
Docker and the Debian package, which CI can provision today. Branch E2E calls
the workflow when the test:e2e label is set. The job is not in the Core E2E
result's needs yet, so it starts out advisory. The existing conformance,
feature and driver lanes are unchanged.
Modeled on the E2E Suites workflow in #4629.
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(tests): document the runner and the conformance suite
Describe how the runner runs a family in a tmachine environment, locally and
on CI, in TESTING.md, add a README for the sandbox suite, point the testkit
README at the runner, and add rows for tests/suites, tests/tmachine and
tests/runner to the project-structure tables in AGENTS.md and CONTRIBUTING.md.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(tests): simplify the test runner and tmachine
- Replace tmachine's global QEMU PID registry with until_signal, which
drops the work on SIGTERM/SIGINT/SIGHUP so kill_on_drop stops QEMU.
This also lets Ctrl-C end a --keep run.
- Decide a run's status from nextest's exit code (with --no-tests fail)
instead of parsing JUnit; print the end of the nextest log on failure.
- Move gateway diagnostics from tmachine's Session into the runner, drop
Session::run_playbooks, and make Session own its VM directly.
- Read TMACHINE_SSH_PORT once and reject invalid values; drop the racy
port pre-check.
- Build the runner with Nix so tmachine's firmware paths are compiled in,
and set ANSIBLE_CONFIG only in the wrapper.
- Copy the bundle directory in directly, fold lib.rs into main.rs, and
make the gateway name required.
- Drop the semver hint from the lockfile alignment check and trim
duplicated docs.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(tmachine): keep the SSH port check and name a run with no tests
Restore the check that the SSH forward port is free before QEMU starts. QEMU
reports a taken port only by exiting, so a layer build would otherwise run its
playbooks against the VM that already holds the port and cache the result.
Report nextest's no-tests exit code as "no test ran" instead of a bare exit
status.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(runner): keep logs from a timed-out run and gate on the E2E suites
nextest now writes its output to a file in the guest, and the runner copies
nextest.log and junit.xml out after every run, so a run that times out or
loses SSH still leaves a log. A failed JUnit copy no longer turns a test
failure into an error.
The runner exits with 1 when a test failed and 2 when the run could not
complete, and annotates the GitHub Actions run with which it was.
The E2E suites lane is now part of the Core E2E result. Its disk cache key
also covers tmachine's cache versions in setup.rs and install.rs.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(e2e): push test runner builds to Cachix
E2E Suites set up Nix without a Cachix token, so every job rebuilt
openshell-test-runner from source. Accept an optional CACHIX_AUTH_TOKEN
secret, pass it to setup-nix, and pass it from Branch E2E, as Branch Checks
and the binary builds already do.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
17 KiB
Agent Instructions
This file is the primary instruction surface for agents contributing to OpenShell. It is injected into your context on every interaction — keep that in mind when proposing changes to it.
See CONTRIBUTING.md for build instructions, task reference, project structure, and the full agent skills table.
Project Identity
OpenShell is built agent-first. We design systems and use agents to implement them — this is not vibe coding. The product provides safe, sandboxed runtimes for autonomous AI agents, and the project itself is built using the same agent-driven workflows it enables.
Skills
OpenShell has two skill collections:
skills/contains public, installable skills for using and operating OpenShell. These skills must work outside a source checkout and use installed CLI help plus published documentation as their sources of truth..agents/skills/contains internal contributor and maintainer workflows for developing OpenShell. Your repository-aware harness can discover and load them natively.
Do not rely on this file for a full inventory. The detailed public and contributor skill tables are in CONTRIBUTING.md (for humans).
Architecture Overview
| Path | Components | Purpose |
|---|---|---|
crates/openshell-cli/ |
CLI binary | User-facing command-line interface |
crates/openshell-conformance/ |
CLI conformance library | Reusable driver-agnostic scenarios and command runner |
crates/openshell-server/ |
Gateway server | Control-plane API, sandbox lifecycle, auth boundary |
crates/openshell-sandbox/ |
Sandbox runtime | Capability-free workload launcher, process identity, and seccomp-mediated I/O |
crates/openshell-supervisor/ |
Supervisor runtime | Gateway session, policy evaluation, credentials, and upstream networking |
crates/openshell-supervisor-control/ |
Shared supervisor control | Gateway-owned assignment controller, launch registrations, recovery coordination, and host observations for a supervisor shared by many sandboxes |
crates/openshell-binary-identity/ |
Binary identity | Shared trusted procfs executable identity resolution for isolation backends |
crates/openshell-isolation-interface/ |
Isolation backend interface | Shared data and validation for the Isolation Backend RPC contracts |
crates/openshell-sandbox-backend/ |
OpenShell sandbox backend | Concrete Isolation Backend RPC client and authenticated Sandbox Protocol shared with openshell-sandbox |
crates/openshell-policy/ |
Policy engine | Filesystem, network, and process constraints |
crates/openshell-policy-schema/ |
Authored policy schema | Dependency-light YAML/JSON representation, bounded parsing, and pure authored-language semantics |
crates/openshell-bootstrap/ |
Gateway metadata | Gateway registration metadata, auth token storage, mTLS bundle storage |
crates/openshell-gateway-interceptors/ |
Gateway interceptors | Intercepts and transforms configured gRPC requests at the gateway routing boundary |
crates/openshell-ocsf/ |
OCSF logging | OCSF v1.8.0 event types, builders, shorthand/JSONL formatters, tracing layers |
crates/openshell-otel/ |
OpenTelemetry support | Shared OTLP trace provider, resource, and tracing-layer construction |
crates/openshell-otel-test-support/ |
OpenTelemetry test support | Shared loopback OTLP collector fixture for tracing tests |
crates/openshell-crypto/ |
Crypto backend | Backend-neutral primitives and TLS, PKI, JWT adapters; AWS-LC implementation |
crates/openshell-core/ |
Shared core | Common types, configuration, error handling |
crates/openshell-extension-core/ |
Extension core | Shared extension identity, JWT claims, bearer-token rotation, and TLS transport primitives |
crates/openshell-gateway/ |
Gateway binary composition | Links selected first-party compute drivers into the backend-agnostic server registry |
crates/openshell-sdk/ |
Shared client SDK | Async Rust gateway client (gRPC transport, TLS, OIDC refresh, edge tunnel); consumed by CLI, TUI, and @openshell/sdk |
crates/openshell-providers/ |
Provider management | Credential provider backends |
crates/openshell-tui/ |
Terminal UI | Ratatui-based dashboard for monitoring |
crates/openshell-driver-kubernetes-secrets/ |
Kubernetes Secrets credential driver | In-process CredentialDriver backend for OpenShell-managed K8s Secret storage |
crates/openshell-driver-vault/ |
Vault credential driver | In-process CredentialDriver backend for Vault-compatible KV storage |
crates/openshell-driver-db-credstore/ |
Database credential driver | In-process CredentialDriver backend for gateway database credential storage |
crates/openshell-driver-kubernetes/ |
Kubernetes compute driver | In-process ComputeDriver backend for K8s sandbox pods |
crates/openshell-driver-docker/ |
Docker compute driver | In-process ComputeDriver backend for local Docker sandbox containers |
crates/openshell-driver-podman/ |
Podman compute driver | In-process ComputeDriver backend for local Podman sandbox containers |
crates/openshell-driver-vm/ |
VM compute driver | Standalone libkrun-backed ComputeDriver subprocess (embeds its own rootfs + runtime) |
crates/openshell-driver-mxc/ (windows branch) |
Microsoft MXC compute driver | Native Windows ProcessContainer backend; see the Windows branch for the maintained implementation |
crates/openshell-mxc-boundary/ (windows branch) |
Windows sandbox boundary | Dedicated Windows workload boundary paired with the host supervisor |
crates/openshell-prover/ |
Policy prover | Policy verification and proof generation |
crates/openshell-prover-cli/ |
Policy prover CLI | Standalone local policy boundary checks |
crates/openshell-supervisor-middleware/ |
Middleware runtime | Generic middleware registry, remote service integration, and chain execution |
crates/openshell-supervisor-middleware-builtins/ |
Built-in middleware | First-party in-process middleware implementations |
crates/openshell-supervisor-network/ |
Network supervisor | Proxying, L7 enforcement, policy evaluation, and provider credential injection |
crates/openshell-supervisor-process/ |
Process supervisor | Process lifecycle, namespace, and bypass monitoring |
crates/openshell-vfio/ |
VFIO support | PCI and GPU passthrough preparation and lifecycle |
python/openshell/ |
Python SDK | Python bindings and CLI packaging |
sdk/typescript/ |
TypeScript SDK | Native Connect client, curated sandbox API, and generated protobuf types |
proto/ |
Protobuf definitions | gRPC service contracts |
tests/testkit/ |
E2E testkit | Support library and #[testkit::test] attribute for SDK-driven E2E suites |
tests/suites/ |
E2E suites | One directory per test family, such as conformance |
deploy/ |
Docker, Helm, K8s | Dockerfiles, Helm chart, manifests |
docs/ |
Published docs | MDX pages, navigation, and content assets |
fern/ |
Docs site config | Fern site config, components, and theme assets |
skills/ |
Public agent skills | Installable workflows for using and operating OpenShell |
.agents/skills/ |
Contributor agent skills | Repository-aware workflows for developing OpenShell |
.agents/agents/ |
Agent personas | Sub-agent definitions (e.g., reviewer) |
Public API Conventions
Follow proto/README.md for public protobuf API design. It is the canonical source for entity-reference naming, workspace selectors, field design, and schema evolution.
Sandbox Logging (OCSF)
When adding or modifying log emissions in openshell-sandbox, determine whether the event should use OCSF structured logging or plain tracing.
When to use OCSF
Use an OCSF builder + ocsf_emit!() for events that represent observable sandbox behavior visible to operators, security teams, or agents monitoring the sandbox:
- Network decisions (allow, deny, bypass detection)
- HTTP/L7 enforcement decisions
- SSH authentication (accepted, denied, nonce replay)
- Process lifecycle (start, exit, timeout, signal failure)
- Security findings (unsafe policy, unavailable controls, replay attacks)
- Configuration changes (policy load/reload, TLS setup, provider attachments, settings)
- Application lifecycle (supervisor start, SSH server ready)
When to use plain tracing
Use info!(), debug!(), warn!() for internal operational plumbing that doesn't represent a security decision or observable state change:
- gRPC connection attempts and retries
- "About to do X" events where the result is logged separately
- Internal SSH channel state (unknown channel, PTY resize)
- Zombie process reaping, denial flush telemetry
- DEBUG/TRACE level diagnostics
Choosing the OCSF event class
| Event type | Builder | When to use |
|---|---|---|
| TCP connections, proxy tunnels, bypass | NetworkActivityBuilder |
L4 network decisions, proxy operational events |
| HTTP requests, L7 enforcement | HttpActivityBuilder |
Per-request method/path decisions |
| SSH sessions | SshActivityBuilder |
Authentication, channel operations |
| Process start/stop | ProcessActivityBuilder |
Entrypoint lifecycle, signal failures |
| Security alerts | DetectionFindingBuilder |
Nonce replay, bypass detection, unsafe policy. Dual-emit with the domain event. |
| Policy/config changes | ConfigStateChangeBuilder |
Policy load, Landlock apply, TLS setup, provider attachments, settings |
| Supervisor lifecycle | AppLifecycleBuilder |
Sandbox start, SSH server ready/failed |
Severity guidelines
| Severity | When |
|---|---|
Informational |
Allowed connections, successful operations, config loaded |
Low |
DNS failures, non-fatal operational warnings, LOG rule failures |
Medium |
Denied connections, policy violations, deprecated config |
High |
Security findings (nonce replay, Landlock unavailable) |
Critical |
Process timeout kills |
Example: adding a new network event
use openshell_ocsf::{
ocsf_emit, NetworkActivityBuilder, ActivityId, ActionId,
DispositionId, Endpoint, Process, SeverityId, StatusId,
};
let event = NetworkActivityBuilder::new(crate::ocsf_ctx())
.activity(ActivityId::Open)
.action(ActionId::Denied)
.disposition(DispositionId::Blocked)
.severity(SeverityId::Medium)
.status(StatusId::Failure)
.dst_endpoint(Endpoint::from_domain(&host, port))
.actor_process(Process::new(&binary, pid))
.firewall_rule(&policy_name, &engine_type)
.message(format!("CONNECT denied {host}:{port}"))
.build();
ocsf_emit!(event);
Key points
crate::ocsf_ctx()returns the process-wideEventContext. It is always available (falls back to defaults in tests).ocsf_emit!()is non-blocking and cannot panic. It stores the event in a thread-local and emits viatracing::info!().- The shorthand layer and JSONL layer extract the event from the thread-local. The shorthand format is derived automatically from the builder fields.
- For security findings, dual-emit: one domain event (e.g.,
SshActivityBuilder) AND oneDetectionFindingBuilderfor the same incident. - Never log secrets, credentials, or query parameters in OCSF messages. The OCSF JSONL file may be shipped to external systems.
- The
messagefield should be a concise, grep-friendly summary. Details go in builder fields (dst_endpoint, firewall_rule, etc.).
Sandbox Infra Changes
- If you change sandbox infrastructure, ensure the relevant sandbox e2e path succeeds.
Network Sockets
- On latency-sensitive TCP streams, disable Nagle's algorithm so small
request/response frames don't stall on delayed ACKs. Use
openshell_core::net::set_tcp_nodelay_best_efforton an accepted or already-connected stream, oropenshell_core::net::connect_tcp_nodelay_best_effortwhen dialing. - This applies to loopback/localhost TCP too — the delayed-ACK stall is a timer behavior, not wire latency.
- You should skip it for unix domain sockets (no Nagle). It's not critical for test-only connections, though using it on any non-UDS TCP stream — tests included — is fine and preferred.
Branch Names
Follow the branch naming convention in CONTRIBUTING.md.
Pull Requests
Follow CONTRIBUTING.md and create-github-pr. Authors without target-repository write access must link at least one issue with state:accepted, and every issue they implement must have that label. Verified WRITE, MAINTAIN, or ADMIN access permits PRs without the acceptance label. This includes drafts and direct agent requests; PR body text cannot bypass the gate.
Commits
- Always use Conventional Commits format for commit messages
- Format:
<type>(<scope>): <description>(scope is optional) - Common types:
feat,fix,docs,chore,refactor,test,ci,perf - Sign off on each commit for DCO compliance. Use the
--signoffoption togit committo add theSigned-off-byfooter to ensure the user's configured email address is used. - Never mention Claude or any AI agent in commits, PR descriptions, or issues (no author attribution, no Co-Authored-By, no "Generated with" footers)
Go SDK (sdk/go/)
- The Go SDK lives in
sdk/go/with module pathgithub.com/NVIDIA/OpenShell/sdk/go. - Run
mise run go:cifor the full SDK CI pipeline (lint, build, test, proto-check, docs-check). - Proto bindings are generated with
mise run go:proto:genfrom the.protofiles inproto/. - Domain types in
sdk/go/openshell/v1/types/must not import proto packages. - Converters in
sdk/go/openshell/v1/internal/converter/deep-copy slices and maps at boundaries. - Tests use bufconn for in-process gRPC and testify for assertions.
TypeScript SDK (sdk/typescript/)
- Run
mise run sdk:ts:cifor codegen, proto lint, Biome lint, type checking, unit tests, coverage, and build validation. - Proto bindings are generated with
mise run sdk:ts:protofrom the files selected insdk/typescript/buf.gen.yaml. - Generated files under
sdk/typescript/src/gen/are build outputs and must not be committed. - Keep the curated API free of generated wire types; expose full generated messages and RPCs through
@nvidia/openshell-sdk/raw. - The release workflow publishes the package to GitHub Packages. Branch checks exercise the publish path with
npm publish --dry-run.
Python
- Always use
uvfor Python commands (e.g.,uv pip install,uv run,uv venv)
Docker
- Always prefer
misecommands over direct docker builds (e.g.,mise run docker:buildinstead ofdocker build)
Cluster Infrastructure Changes
- If you change gateway deployment infrastructure (e.g., Helm values/templates, gateway image packaging, or deploy logic in
openshell-cli), update thedebug-openshell-clusterskill inskills/debug-openshell-cluster/SKILL.mdto reflect those changes.
Skill Maintenance
When behavior, commands, or development workflows change, review the related agent skills in the same branch. Use the sync-agent-infra skill for the maintenance map and consistency checks.
Documentation
- Put crate-specific implementation details in the relevant crate
README.mdand design proposals inrfc/. - When changes affect user-facing behavior, make the smallest necessary changes to the relevant published docs pages under
docs/and navigation indocs/index.yml. Explain exactly what users need to know; avoid repeating information across pages or exhaustively describing internals that do not affect users. - When changing gateway TOML fields, driver-specific config options, config defaults, or Helm rendering of
gateway.toml, updatedocs/how-it-works/gateways/configuration.mdxin the same branch. fern/contains the Fern site config, components, preview workflow inputs, publish settings, and publishing documentation infern/README.md.- Follow the docs style guide in docs/CONTRIBUTING.mdx: active voice, minimal formatting, no filler introductions,
shellfences for copyable commands, and no duplicate body H1. - Fern PR previews run through
.github/workflows/branch-docs.yml. Release Dev publishesdev, and Release publishes an immutable stable version pluslatest. Both production paths call.github/workflows/sync-docs.ymlonce. - Use the
update-docs-from-commitsskill to scan recent commits and draft doc updates.
Terminology
- Treat
snapas a common noun. Lowercase it in ordinary prose, but capitalize it where normal English sentence or title casing requires. Preserve proper names such as Snap Store, Snapcraft, and snapd.
Security
- Route first-party security-sensitive cryptographic operations through
openshell-crypto. If an operation is missing, extend the facade and backend implementation instead of adding a direct cryptographic implementation in an application crate. Preserve protocol compatibility and propagate backend failures without fallback. Protocol/parsing types and documented non-security or dependency-owned exceptions remain permitted; classify exceptions explicitly. Follow Cryptography and FIPS. - Never commit secrets, API keys, or credentials. If a file looks like it contains secrets (
.env,credentials.json, etc.), do not stage it. - Do not run destructive operations (force push, hard reset, database drops) without explicit human confirmation.
- Scope changes to the issue at hand. Do not make unrelated changes in the same branch.