OpenShell/TESTING.md

Testing

Running Tests

mise run test          # Rust + Python unit tests
mise run e2e           # Docker-backed end-to-end tests
mise run e2e:docker    # Same as e2e, explicit Docker-backed suite
mise run e2e:k3s:gpu   # GPU E2E against the OpenShell k3s-cluster gateway
mise run ci            # Everything: lint, compile checks, and tests

Test Layout

crates/*/src/          # Inline #[cfg(test)] modules
crates/*/tests/        # Rust integration tests
python/openshell/      # Python unit tests (*_test.py suffix)
e2e/python/            # Python E2E tests (test_*.py prefix)
e2e/rust/              # Rust CLI E2E tests

Rust Tests

Unit tests live inline with #[cfg(test)] mod tests blocks. Integration tests go in crates/*/tests/ and are named *_integration.rs.

Use #[tokio::test] for anything async:

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn store_round_trip() {
        let store = Store::connect("sqlite::memory:").await.unwrap();
        store.put("sandbox", "abc", "my-sandbox", b"payload").await.unwrap();
        let record = store.get("sandbox", "abc").await.unwrap().unwrap();
        assert_eq!(record.payload, b"payload");
    }
}

Run Rust tests only:

mise run test:rust     # cargo test --workspace

Python Unit Tests

Python unit tests use the *_test.py suffix convention (not test_* prefix) and live alongside the source in python/openshell/. They use mock-based patterns with fake gRPC stubs:

def test_exec_python_serializes_callable_payload() -> None:
    stub = _FakeStub()
    client = _client_with_fake_stub(stub)

    def add(a: int, b: int) -> int:
        return a + b

    result = client.exec_python("sandbox-1", add, args=(2, 3))
    assert result.exit_code == 0

Run Python unit tests only:

mise run test:python   # uv run pytest python/

E2E Tests

E2E tests run against a live gateway. By default, mise run e2e starts an ephemeral standalone gateway with the Docker compute driver, runs the suite, and cleans it up afterward. To run the suite against an existing plaintext gateway, set OPENSHELL_GATEWAY_ENDPOINT:

OPENSHELL_GATEWAY_ENDPOINT=http://127.0.0.1:18080 mise run e2e

Raw endpoint mode is HTTP-only. Use a named gateway config when a gateway requires mTLS.

Python E2E (e2e/python/)

Tests use the sandbox fixture from conftest.py to create real sandboxes:

def test_exec_returns_stdout(sandbox):
    with sandbox(delete_on_exit=True) as sb:
        result = sb.exec(["echo", "hello"])
        assert result.exit_code == 0
        assert "hello" in result.stdout

Run Docker-backed Python e2e tests:

mise run e2e:docker:python

Sandbox.exec_python

exec_python serializes a Python callable with cloudpickle, sends it to the sandbox, and returns the result. Because cloudpickle serializes module-level functions by reference (which fails inside the sandbox), use one of these patterns:

Closures from factory functions:

def _make_adder():
    def add(a, b):
        return a + b
    return add

def test_addition(sandbox):
    with sandbox(delete_on_exit=True) as sb:
        result = sb.exec_python(_make_adder(), args=(2, 3))
        assert result.stdout.strip() == "5"

Bound methods on local classes:

def test_multiply(sandbox):
    class Calculator:
        def multiply(self, a, b):
            return a * b

    with sandbox(delete_on_exit=True) as sb:
        result = sb.exec_python(Calculator().multiply, args=(6, 7))
        assert result.stdout.strip() == "42"

Shared Fixtures (e2e/python/conftest.py)

Fixture	Scope	Purpose
sandbox_client	session	gRPC client connected to the active gateway
sandbox	function	Factory returning a Sandbox context manager
inference_client	session	Client for managing inference routes
mock_inference_route	session	Creates a mock OpenAI-protocol route for tests

Rust CLI E2E (e2e/rust/)

Rust-based e2e tests that exercise the openshell CLI binary as a subprocess. They live in the openshell-e2e crate and use a shared harness for sandbox lifecycle management, output parsing, and cleanup.

Tests:

• tests/custom_image.rs — custom Docker image build and sandbox run
• tests/sync.rs — bidirectional file sync round-trip (including large files)
• tests/port_forward.rs — TCP port forwarding through a sandbox

Run all CLI e2e tests:

mise run e2e:docker:rust

Run a single test directly with cargo:

cargo test --manifest-path e2e/rust/Cargo.toml --features e2e --test sync

Run a single Rust e2e test target with the Docker-backed gateway wrapper:

OPENSHELL_E2E_RUST_TEST=smoke mise run e2e:docker:rust
OPENSHELL_E2E_RUST_TEST=sync mise run e2e:docker:rust

Run only the Docker-backed smoke test:

mise run e2e:docker:smoke

Run GPU E2E against the OpenShell k3s-cluster gateway:

mise run e2e:k3s:gpu

The harness (e2e/rust/src/harness/) provides:

Module	Purpose
binary	Builds and resolves the openshell binary from the workspace
sandbox	SandboxGuard RAII type — creates sandboxes and deletes them on drop
output	ANSI stripping and field extraction from CLI output
port	wait_for_port() and find_free_port() for TCP testing

Environment Variables

Variable	Purpose
OPENSHELL_GATEWAY	Override active gateway name for E2E tests
OPENSHELL_GATEWAY_ENDPOINT	Run E2E tests against an existing plaintext HTTP gateway endpoint
OPENSHELL_E2E_RUST_TEST	Run one Rust E2E integration test target, for example smoke or sync