Files
OpenShell/AGENTS.md
T
Matthew GrossmanandEvan Lezar 33138516ad feat(tests): add the tmachine Session API and run a conformance smoke test on tmachine (#4773)
* feat(tmachine): add the Session API and a test-runner package

Add a library API that boots an environment, runs commands in it with timeouts,
copies files in and out, collects diagnostics and tears it down, so a caller can
drive a machine without writing an Ansible playbook per suite. SSH and exec
timeouts and QEMU cleanup on SIGTERM keep a hung guest or a cancelled CI job from
leaving QEMU running.

Add the `openshell-test-runner` flake package, which runs tests/runner with the
toolchains and VM tools it needs.

The Session API and test-runner package are adapted from NVIDIA/OpenShell#4629.

Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* build(cargo): add a check that aligned workspaces lock shared crates like the root

`cargo --locked` only checks a lockfile against its own manifest. A workspace
outside the root that builds root crates through path dependencies, such as the
testkit suites building `openshell-sdk`, has its own Cargo.lock, so the two
lockfiles can resolve a shared crate to different versions without any check
failing. That is how aws-lc-sys 0.46 was picked up in #4629, which does not link
against the AWS-LC that Nix provides.

check_cargo_lockfile_alignment.py fails when an aligned workspace locks a
registry crate that the root lockfile also contains at a version the root
lockfile does not have. With --fix it re-locks the workspace starting from the
root lockfile. tests/testkit is the first aligned workspace.

The script and its tests come from #4629 (a37ebe917).

Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(conformance): add a sandbox smoke suite built on testkit

Add the `sandbox` crate to the conformance workspace with one testkit test that
creates a sandbox, runs `echo` in it with exec, deletes it and checks it is gone.
It is deliberately small: it exists so the runner can run a real test in a
tmachine environment on CI before the existing suites move to testkit.

The workspace's other crate, the CLI conformance suite, keeps running as it does
today. The conformance workspace joins the lockfile alignment check, so its
lockfile is realigned with the root lockfile.

The test is adapted from the `sandbox-lifecycle` suite in #4629.

Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* feat(runner): add a test runner that runs a test family on tmachine

`openshell-test-runner` builds a family's packages (a workspace under
tests/suites/) on the host as a nextest archive for the guest's musl target,
boots one tmachine environment by name with the named installer through the
Session API, runs nextest in it and brings back the JUnit report and log. No
per-suite Ansible playbook is involved:

  nix run .#openshell-test-runner -- --environment ubuntu-docker-rootful \
    --installer deb --family conformance --package sandbox

The runner points the tests at the system under test with environment
variables: OPENSHELL_GATEWAY (the gateway the installer registered, `--gateway`,
default `openshell`) and OPENSHELL_BIN (the installed CLI). It waits for the
gateway with `testkit-ready` before the tests start. Nextest runs with
`--test-threads 1` and a slow-timeout that terminates a hung test.

A run passes if nextest succeeds and its JUnit report shows at least one test
and no failures. `--filter` narrows the tests with a nextest filterset, and
`--keep` leaves the VM up for debugging. Results from an earlier run are
removed first, and the guest's diagnostics are collected after a failure.

Which tests run in which environments is left to the caller for now; there are
no plans, environment registry or per-test requirements, and a broken test is
`#[ignore]`d.

Most of this comes from #4629, adapted to testkit finding the gateway the way the
CLI does instead of reading a target file.

Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* ci(tests): lint and test the runner, tmachine and conformance suite

Run fmt, clippy and nextest for tests/tmachine and tests/runner, and fmt and
clippy for the new `sandbox` suite crate, wherever tests/testkit already runs
them: the mise tasks and Branch Checks. The suite's tests need a gateway, so they
run through `mise run e2e:testkit` against a local Docker gateway and, on CI,
through the test runner rather than the unit-test job.

The lockfile task and the Cargo lockfiles job also run the alignment check, and
its own tests run with `mise run test`.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* ci(tests): run testkit suites on tmachine from a reusable E2E Suites workflow

Add a reusable workflow with an explicit matrix: each row names a tmachine
environment and installer, a family and package, and the runner label. Each
row runs `nix run .#openshell-test-runner` once. The jobs use the same
integration inputs as the existing tmachine lanes, cache the tmachine disks
keyed on the Ansible tree, upload the JUnit report and logs whether or not the
run passes, and an aggregator job reports the overall result.

The only row runs the conformance sandbox suite on amd64 Ubuntu with rootful
Docker and the Debian package, which CI can provision today. Branch E2E calls
the workflow when the test:e2e label is set. The job is not in the Core E2E
result's needs yet, so it starts out advisory. The existing conformance,
feature and driver lanes are unchanged.

Modeled on the E2E Suites workflow in #4629.

Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(tests): document the runner and the conformance suite

Describe how the runner runs a family in a tmachine environment, locally and
on CI, in TESTING.md, add a README for the sandbox suite, point the testkit
README at the runner, and add rows for tests/suites, tests/tmachine and
tests/runner to the project-structure tables in AGENTS.md and CONTRIBUTING.md.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(tests): simplify the test runner and tmachine

- Replace tmachine's global QEMU PID registry with until_signal, which
  drops the work on SIGTERM/SIGINT/SIGHUP so kill_on_drop stops QEMU.
  This also lets Ctrl-C end a --keep run.
- Decide a run's status from nextest's exit code (with --no-tests fail)
  instead of parsing JUnit; print the end of the nextest log on failure.
- Move gateway diagnostics from tmachine's Session into the runner, drop
  Session::run_playbooks, and make Session own its VM directly.
- Read TMACHINE_SSH_PORT once and reject invalid values; drop the racy
  port pre-check.
- Build the runner with Nix so tmachine's firmware paths are compiled in,
  and set ANSIBLE_CONFIG only in the wrapper.
- Copy the bundle directory in directly, fold lib.rs into main.rs, and
  make the gateway name required.
- Drop the semver hint from the lockfile alignment check and trim
  duplicated docs.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(tmachine): keep the SSH port check and name a run with no tests

Restore the check that the SSH forward port is free before QEMU starts. QEMU
reports a taken port only by exiting, so a layer build would otherwise run its
playbooks against the VM that already holds the port and cache the result.

Report nextest's no-tests exit code as "no test ran" instead of a bare exit
status.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(runner): keep logs from a timed-out run and gate on the E2E suites

nextest now writes its output to a file in the guest, and the runner copies
nextest.log and junit.xml out after every run, so a run that times out or
loses SSH still leaves a log. A failed JUnit copy no longer turns a test
failure into an error.

The runner exits with 1 when a test failed and 2 when the run could not
complete, and annotates the GitHub Actions run with which it was.

The E2E suites lane is now part of the Core E2E result. Its disk cache key
also covers tmachine's cache versions in setup.rs and install.rs.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* ci(e2e): push test runner builds to Cachix

E2E Suites set up Nix without a Cachix token, so every job rebuilt
openshell-test-runner from source. Accept an optional CACHIX_AUTH_TOKEN
secret, pass it to setup-nix, and pass it from Branch E2E, as Branch Checks
and the binary builds already do.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
2026-10-28 18:29:25 +00:00

17 KiB

Agent Instructions

This file is the primary instruction surface for agents contributing to OpenShell. It is injected into your context on every interaction — keep that in mind when proposing changes to it.

See CONTRIBUTING.md for build instructions, task reference, project structure, and the full agent skills table.

Project Identity

OpenShell is built agent-first. We design systems and use agents to implement them — this is not vibe coding. The product provides safe, sandboxed runtimes for autonomous AI agents, and the project itself is built using the same agent-driven workflows it enables.

Skills

OpenShell has two skill collections:

  • skills/ contains public, installable skills for using and operating OpenShell. These skills must work outside a source checkout and use installed CLI help plus published documentation as their sources of truth.
  • .agents/skills/ contains internal contributor and maintainer workflows for developing OpenShell. Your repository-aware harness can discover and load them natively.

Do not rely on this file for a full inventory. The detailed public and contributor skill tables are in CONTRIBUTING.md (for humans).

Architecture Overview

Path Components Purpose
crates/openshell-cli/ CLI binary User-facing command-line interface
crates/openshell-conformance/ CLI conformance library Reusable driver-agnostic scenarios and command runner
crates/openshell-server/ Gateway server Control-plane API, sandbox lifecycle, auth boundary
crates/openshell-sandbox/ Sandbox runtime Capability-free workload launcher, process identity, and seccomp-mediated I/O
crates/openshell-supervisor/ Supervisor runtime Gateway session, policy evaluation, credentials, and upstream networking
crates/openshell-supervisor-control/ Shared supervisor control Gateway-owned assignment controller, launch registrations, recovery coordination, and host observations for a supervisor shared by many sandboxes
crates/openshell-binary-identity/ Binary identity Shared trusted procfs executable identity resolution for isolation backends
crates/openshell-isolation-interface/ Isolation backend interface Shared data and validation for the Isolation Backend RPC contracts
crates/openshell-sandbox-backend/ OpenShell sandbox backend Concrete Isolation Backend RPC client and authenticated Sandbox Protocol shared with openshell-sandbox
crates/openshell-policy/ Policy engine Filesystem, network, and process constraints
crates/openshell-policy-schema/ Authored policy schema Dependency-light YAML/JSON representation, bounded parsing, and pure authored-language semantics
crates/openshell-bootstrap/ Gateway metadata Gateway registration metadata, auth token storage, mTLS bundle storage
crates/openshell-gateway-interceptors/ Gateway interceptors Intercepts and transforms configured gRPC requests at the gateway routing boundary
crates/openshell-ocsf/ OCSF logging OCSF v1.8.0 event types, builders, shorthand/JSONL formatters, tracing layers
crates/openshell-otel/ OpenTelemetry support Shared OTLP trace provider, resource, and tracing-layer construction
crates/openshell-otel-test-support/ OpenTelemetry test support Shared loopback OTLP collector fixture for tracing tests
crates/openshell-crypto/ Crypto backend Backend-neutral primitives and TLS, PKI, JWT adapters; AWS-LC implementation
crates/openshell-core/ Shared core Common types, configuration, error handling
crates/openshell-extension-core/ Extension core Shared extension identity, JWT claims, bearer-token rotation, and TLS transport primitives
crates/openshell-gateway/ Gateway binary composition Links selected first-party compute drivers into the backend-agnostic server registry
crates/openshell-sdk/ Shared client SDK Async Rust gateway client (gRPC transport, TLS, OIDC refresh, edge tunnel); consumed by CLI, TUI, and @openshell/sdk
crates/openshell-providers/ Provider management Credential provider backends
crates/openshell-tui/ Terminal UI Ratatui-based dashboard for monitoring
crates/openshell-driver-kubernetes-secrets/ Kubernetes Secrets credential driver In-process CredentialDriver backend for OpenShell-managed K8s Secret storage
crates/openshell-driver-vault/ Vault credential driver In-process CredentialDriver backend for Vault-compatible KV storage
crates/openshell-driver-db-credstore/ Database credential driver In-process CredentialDriver backend for gateway database credential storage
crates/openshell-driver-kubernetes/ Kubernetes compute driver In-process ComputeDriver backend for K8s sandbox pods
crates/openshell-driver-docker/ Docker compute driver In-process ComputeDriver backend for local Docker sandbox containers
crates/openshell-driver-podman/ Podman compute driver In-process ComputeDriver backend for local Podman sandbox containers
crates/openshell-driver-vm/ VM compute driver Standalone libkrun-backed ComputeDriver subprocess (embeds its own rootfs + runtime)
crates/openshell-driver-mxc/ (windows branch) Microsoft MXC compute driver Native Windows ProcessContainer backend; see the Windows branch for the maintained implementation
crates/openshell-mxc-boundary/ (windows branch) Windows sandbox boundary Dedicated Windows workload boundary paired with the host supervisor
crates/openshell-prover/ Policy prover Policy verification and proof generation
crates/openshell-prover-cli/ Policy prover CLI Standalone local policy boundary checks
crates/openshell-supervisor-middleware/ Middleware runtime Generic middleware registry, remote service integration, and chain execution
crates/openshell-supervisor-middleware-builtins/ Built-in middleware First-party in-process middleware implementations
crates/openshell-supervisor-network/ Network supervisor Proxying, L7 enforcement, policy evaluation, and provider credential injection
crates/openshell-supervisor-process/ Process supervisor Process lifecycle, namespace, and bypass monitoring
crates/openshell-vfio/ VFIO support PCI and GPU passthrough preparation and lifecycle
python/openshell/ Python SDK Python bindings and CLI packaging
sdk/typescript/ TypeScript SDK Native Connect client, curated sandbox API, and generated protobuf types
proto/ Protobuf definitions gRPC service contracts
tests/testkit/ E2E testkit Support library and #[testkit::test] attribute for SDK-driven E2E suites
tests/suites/ E2E suites One directory per test family, such as conformance
deploy/ Docker, Helm, K8s Dockerfiles, Helm chart, manifests
docs/ Published docs MDX pages, navigation, and content assets
fern/ Docs site config Fern site config, components, and theme assets
skills/ Public agent skills Installable workflows for using and operating OpenShell
.agents/skills/ Contributor agent skills Repository-aware workflows for developing OpenShell
.agents/agents/ Agent personas Sub-agent definitions (e.g., reviewer)

Public API Conventions

Follow proto/README.md for public protobuf API design. It is the canonical source for entity-reference naming, workspace selectors, field design, and schema evolution.

Sandbox Logging (OCSF)

When adding or modifying log emissions in openshell-sandbox, determine whether the event should use OCSF structured logging or plain tracing.

When to use OCSF

Use an OCSF builder + ocsf_emit!() for events that represent observable sandbox behavior visible to operators, security teams, or agents monitoring the sandbox:

  • Network decisions (allow, deny, bypass detection)
  • HTTP/L7 enforcement decisions
  • SSH authentication (accepted, denied, nonce replay)
  • Process lifecycle (start, exit, timeout, signal failure)
  • Security findings (unsafe policy, unavailable controls, replay attacks)
  • Configuration changes (policy load/reload, TLS setup, provider attachments, settings)
  • Application lifecycle (supervisor start, SSH server ready)

When to use plain tracing

Use info!(), debug!(), warn!() for internal operational plumbing that doesn't represent a security decision or observable state change:

  • gRPC connection attempts and retries
  • "About to do X" events where the result is logged separately
  • Internal SSH channel state (unknown channel, PTY resize)
  • Zombie process reaping, denial flush telemetry
  • DEBUG/TRACE level diagnostics

Choosing the OCSF event class

Event type Builder When to use
TCP connections, proxy tunnels, bypass NetworkActivityBuilder L4 network decisions, proxy operational events
HTTP requests, L7 enforcement HttpActivityBuilder Per-request method/path decisions
SSH sessions SshActivityBuilder Authentication, channel operations
Process start/stop ProcessActivityBuilder Entrypoint lifecycle, signal failures
Security alerts DetectionFindingBuilder Nonce replay, bypass detection, unsafe policy. Dual-emit with the domain event.
Policy/config changes ConfigStateChangeBuilder Policy load, Landlock apply, TLS setup, provider attachments, settings
Supervisor lifecycle AppLifecycleBuilder Sandbox start, SSH server ready/failed

Severity guidelines

Severity When
Informational Allowed connections, successful operations, config loaded
Low DNS failures, non-fatal operational warnings, LOG rule failures
Medium Denied connections, policy violations, deprecated config
High Security findings (nonce replay, Landlock unavailable)
Critical Process timeout kills

Example: adding a new network event

use openshell_ocsf::{
    ocsf_emit, NetworkActivityBuilder, ActivityId, ActionId,
    DispositionId, Endpoint, Process, SeverityId, StatusId,
};

let event = NetworkActivityBuilder::new(crate::ocsf_ctx())
    .activity(ActivityId::Open)
    .action(ActionId::Denied)
    .disposition(DispositionId::Blocked)
    .severity(SeverityId::Medium)
    .status(StatusId::Failure)
    .dst_endpoint(Endpoint::from_domain(&host, port))
    .actor_process(Process::new(&binary, pid))
    .firewall_rule(&policy_name, &engine_type)
    .message(format!("CONNECT denied {host}:{port}"))
    .build();
ocsf_emit!(event);

Key points

  • crate::ocsf_ctx() returns the process-wide EventContext. It is always available (falls back to defaults in tests).
  • ocsf_emit!() is non-blocking and cannot panic. It stores the event in a thread-local and emits via tracing::info!().
  • The shorthand layer and JSONL layer extract the event from the thread-local. The shorthand format is derived automatically from the builder fields.
  • For security findings, dual-emit: one domain event (e.g., SshActivityBuilder) AND one DetectionFindingBuilder for the same incident.
  • Never log secrets, credentials, or query parameters in OCSF messages. The OCSF JSONL file may be shipped to external systems.
  • The message field should be a concise, grep-friendly summary. Details go in builder fields (dst_endpoint, firewall_rule, etc.).

Sandbox Infra Changes

  • If you change sandbox infrastructure, ensure the relevant sandbox e2e path succeeds.

Network Sockets

  • On latency-sensitive TCP streams, disable Nagle's algorithm so small request/response frames don't stall on delayed ACKs. Use openshell_core::net::set_tcp_nodelay_best_effort on an accepted or already-connected stream, or openshell_core::net::connect_tcp_nodelay_best_effort when dialing.
  • This applies to loopback/localhost TCP too — the delayed-ACK stall is a timer behavior, not wire latency.
  • You should skip it for unix domain sockets (no Nagle). It's not critical for test-only connections, though using it on any non-UDS TCP stream — tests included — is fine and preferred.

Branch Names

Follow the branch naming convention in CONTRIBUTING.md.

Pull Requests

Follow CONTRIBUTING.md and create-github-pr. Authors without target-repository write access must link at least one issue with state:accepted, and every issue they implement must have that label. Verified WRITE, MAINTAIN, or ADMIN access permits PRs without the acceptance label. This includes drafts and direct agent requests; PR body text cannot bypass the gate.

Commits

  • Always use Conventional Commits format for commit messages
  • Format: <type>(<scope>): <description> (scope is optional)
  • Common types: feat, fix, docs, chore, refactor, test, ci, perf
  • Sign off on each commit for DCO compliance. Use the --signoff option to git commit to add the Signed-off-by footer to ensure the user's configured email address is used.
  • Never mention Claude or any AI agent in commits, PR descriptions, or issues (no author attribution, no Co-Authored-By, no "Generated with" footers)

Go SDK (sdk/go/)

  • The Go SDK lives in sdk/go/ with module path github.com/NVIDIA/OpenShell/sdk/go.
  • Run mise run go:ci for the full SDK CI pipeline (lint, build, test, proto-check, docs-check).
  • Proto bindings are generated with mise run go:proto:gen from the .proto files in proto/.
  • Domain types in sdk/go/openshell/v1/types/ must not import proto packages.
  • Converters in sdk/go/openshell/v1/internal/converter/ deep-copy slices and maps at boundaries.
  • Tests use bufconn for in-process gRPC and testify for assertions.

TypeScript SDK (sdk/typescript/)

  • Run mise run sdk:ts:ci for codegen, proto lint, Biome lint, type checking, unit tests, coverage, and build validation.
  • Proto bindings are generated with mise run sdk:ts:proto from the files selected in sdk/typescript/buf.gen.yaml.
  • Generated files under sdk/typescript/src/gen/ are build outputs and must not be committed.
  • Keep the curated API free of generated wire types; expose full generated messages and RPCs through @nvidia/openshell-sdk/raw.
  • The release workflow publishes the package to GitHub Packages. Branch checks exercise the publish path with npm publish --dry-run.

Python

  • Always use uv for Python commands (e.g., uv pip install, uv run, uv venv)

Docker

  • Always prefer mise commands over direct docker builds (e.g., mise run docker:build instead of docker build)

Cluster Infrastructure Changes

  • If you change gateway deployment infrastructure (e.g., Helm values/templates, gateway image packaging, or deploy logic in openshell-cli), update the debug-openshell-cluster skill in skills/debug-openshell-cluster/SKILL.md to reflect those changes.

Skill Maintenance

When behavior, commands, or development workflows change, review the related agent skills in the same branch. Use the sync-agent-infra skill for the maintenance map and consistency checks.

Documentation

  • Put crate-specific implementation details in the relevant crate README.md and design proposals in rfc/.
  • When changes affect user-facing behavior, make the smallest necessary changes to the relevant published docs pages under docs/ and navigation in docs/index.yml. Explain exactly what users need to know; avoid repeating information across pages or exhaustively describing internals that do not affect users.
  • When changing gateway TOML fields, driver-specific config options, config defaults, or Helm rendering of gateway.toml, update docs/how-it-works/gateways/configuration.mdx in the same branch.
  • fern/ contains the Fern site config, components, preview workflow inputs, publish settings, and publishing documentation in fern/README.md.
  • Follow the docs style guide in docs/CONTRIBUTING.mdx: active voice, minimal formatting, no filler introductions, shell fences for copyable commands, and no duplicate body H1.
  • Fern PR previews run through .github/workflows/branch-docs.yml. Release Dev publishes dev, and Release publishes an immutable stable version plus latest. Both production paths call .github/workflows/sync-docs.yml once.
  • Use the update-docs-from-commits skill to scan recent commits and draft doc updates.

Terminology

  • Treat snap as a common noun. Lowercase it in ordinary prose, but capitalize it where normal English sentence or title casing requires. Preserve proper names such as Snap Store, Snapcraft, and snapd.

Security

  • Route first-party security-sensitive cryptographic operations through openshell-crypto. If an operation is missing, extend the facade and backend implementation instead of adding a direct cryptographic implementation in an application crate. Preserve protocol compatibility and propagate backend failures without fallback. Protocol/parsing types and documented non-security or dependency-owned exceptions remain permitted; classify exceptions explicitly. Follow Cryptography and FIPS.
  • Never commit secrets, API keys, or credentials. If a file looks like it contains secrets (.env, credentials.json, etc.), do not stage it.
  • Do not run destructive operations (force push, hard reset, database drops) without explicit human confirmation.
  • Scope changes to the issue at hand. Do not make unrelated changes in the same branch.