Files
OpenShell/proto/compute_driver.proto
Derek Carr 59479f492a feat(k8s): add namespace-per-workspace support (RFC 0011 Phase 3) (#2656)
* feat(k8s): add namespace-per-workspace support (RFC 0011 Phase 3)

Implement three workspace namespace modes for the Kubernetes compute
driver: shared (default, preserves current single-namespace behavior),
managed (auto-creates/deletes namespaces per workspace), and operator
(pre-provisioned namespaces with dynamic discovery via label selector
or drop-in allowlist file).

Key changes:
- WorkspaceMode enum and namespace resolution in driver config
- Managed namespace lifecycle with ServiceAccount and OpenShift SCC
  annotation propagation
- Cluster-wide sandbox CR watchers for managed/operator modes
- NamespaceValidator (Exact/Prefix/Allowlist) for SA token auth
- Workspace-aware credential secret storage
- Helm ClusterRole for multi-namespace RBAC
- Gateway config, architecture, and reference docs

Signed-off-by: Derek Carr <decarr@redhat.com>

* test(k8s): add e2e tests for workspace namespace modes

Add end-to-end tests for managed and operator workspace modes
introduced in RFC 0011 Phase 3. The managed mode tests verify
namespace creation with correct labels, ServiceAccount provisioning,
sandbox CR placement, and namespace survival with remaining sandboxes.
The operator mode tests verify rejection of unlabeled and nonexistent
namespaces. The positive operator path (sandbox in labeled namespace)
is known to fail due to an RBAC gap and will be addressed separately.

Also fixes Helm 4 compatibility: move SPDX license headers inside
conditional guards in 8 chart templates to prevent empty comment-only
documents, and fix a trailing whitespace trimmer in clusterrole.yaml
that concatenated the license header with apiVersion.

Adds cleanup sweep in with-kube-gateway.sh to remove managed and
operator namespaces before Helm uninstall, and mise tasks for running
each mode independently.

Signed-off-by: Derek Carr <decarr@redhat.com>

* feat(k8s): add operator namespace label watcher

Spawn a background kube::runtime::watcher in the K8s driver that
watches namespaces matching the configured label selector and populates
the OperatorNamespaceAllowlist at runtime. The driver owns the
allowlist and exposes its Arc so the server can share the same set with
the SA token authenticator.

create_sandbox now gates pod creation on the allowlist in operator
mode — workspaces whose namespace is not yet labeled are rejected at
resource render time rather than silently proceeding. Workspace
lifecycle itself is unaffected; only sandbox (resource) creation is
gated.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(k8s): harden operator mode and address review findings

Close the fail-open gap in operator mode when only
operator_namespace_file is configured: the allowlist is now created
unconditionally in operator mode (fail-closed from startup).

Implement the namespace file watcher using the notify crate, following
the TLS hot-reload pattern (parent-directory watch, 1s debounce,
ConfigMap symlink-swap safe). The file format is a JSON array of
namespace name strings.

Additional fixes from the 10-reviewer audit:
- Change allowlist rejection from InvalidArgument to FailedPrecondition
  so callers know the request may succeed later once the namespace is
  provisioned.
- NamespaceValidator::Allowlist now holds the OperatorNamespaceAllowlist
  newtype instead of a raw Arc<RwLock<BTreeSet>>, eliminating silent
  denial on RwLock poison.
- Verify LABEL_MANAGED_BY and LABEL_GATEWAY_ID ownership before
  deleting a managed namespace.
- Replace fixed 5s sleep in operator e2e test with a 30s poll loop.
- Add Helm validation for workspaceMode values.
- Fix Helm README type column and description for operator fields.
- Add insert/remove methods to OperatorNamespaceAllowlist; label
  watcher now uses them instead of reaching through shared().
- Reject configs with both operator_namespace_label and
  operator_namespace_file set.

Signed-off-by: Derek Carr <decarr@redhat.com>

* feat(k8s): add workspace-level compute driver RPCs and harden RBAC

Decouple namespace lifecycle from sandbox lifecycle by adding
EnsureWorkspace/DeleteWorkspace RPCs to the ComputeDriver service.
Namespace creation now happens before credential storage and namespace
deletion happens on workspace delete, fixing credential storage in
managed workspace mode.

- Add EnsureWorkspace and DeleteWorkspace proto RPCs with
  implementations across all compute drivers (K8s managed delegates to
  ensure_namespace/delete_namespace_if_empty; others no-op)
- Wire ensure_workspace into provider create/update/refresh paths so
  the namespace exists before the credential driver writes secrets
- Wire delete_workspace into workspace deletion for cleanup
- Remove delete_namespace_if_empty from sandbox deletion path
- Scope ClusterRole secrets access to non-shared workspace modes
- Add TODO for TLS cert hot-reload in sandbox gRPC client
- Harden e2e tests with control-plane sandbox resolution assertions
- Fix docker image save --platform flag for OCI index manifests

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(k8s): address re-review findings and add test coverage

- Use server-side apply for TLS secret sync (fixes second sandbox
  creation failure when TLS is enabled)
- Scope gateway-ID label selector unconditionally across all workspace
  modes (fixes operator reads/watches/deletes seeing foreign sandboxes)
- Validate operator allowlist in EnsureWorkspace and DeleteWorkspace
  RPCs (prevents credential writes to namespaces outside the allowlist)
- Extend ClusterRole secrets patch+delete to all non-shared modes with
  credential driver enabled (fixes operator credential storage RBAC)
- Validate namespace ownership on 409 conflict in ensure_namespace
  (prevents adopting unowned namespaces in managed mode)
- Replace delete_namespace_if_empty with unconditional delete_namespace
  letting Kubernetes cascade cleanup (fixes stuck terminating CRs)
- Strengthen NetworkPolicy TODO to cover both managed and operator modes
- Extract selector and ownership logic into testable free functions
- Add unit tests for gateway-ID selectors and namespace ownership
- Add Helm ClusterRole RBAC tests for operator credential driver

Signed-off-by: Derek Carr <decarr@redhat.com>

* ci(k8s): add workspace managed and operator mode e2e to CI

Wire the existing e2e:kubernetes:workspace-managed and
e2e:kubernetes:workspace-operator mise tasks into the branch-e2e
workflow so they run alongside the other core Kubernetes e2e suites.
Both are gated by run_core_e2e and included in the Core E2E result
gate.

Signed-off-by: Derek Carr <decarr@redhat.com>

* test(k8s): add e2e tests for workspace namespace modes

Add 7 new e2e tests covering workspace namespace lifecycle, TLS secret
copying, ownership conflict detection, DNS-1123 validation, operator
namespace preservation, and dynamic label watcher behavior. Fix async
sandbox deletion race condition in existing tests by polling sandbox
list instead of asserting immediately after delete.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(k8s): grant secrets/patch unconditionally and backfill gateway-id labels

Address two review findings:

1. RBAC: server-side apply (PATCH) is used for TLS secret sync in
   multi-namespace modes, but the ClusterRole only granted patch when
   the kubernetes-secrets credential driver was enabled. Grant patch
   unconditionally for non-shared modes since TLS sync always needs it;
   keep delete gated on the credential driver.

2. Upgrade safety: the new gateway-id label selector would orphan
   legacy Sandbox CRs that predate its introduction. Add a startup
   backfill in shared mode that patches any managed Sandbox CR missing
   the gateway-id label before the driver begins serving requests.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(k8s): address workspace namespace review findings

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(k8s): address follow-up review findings

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(k8s): preserve workspace lookup after rebase

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(helm): allow managed secret creation

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(k8s): stop pods in workspace namespace

Signed-off-by: Derek Carr <decarr@redhat.com>

* test(k8s): scope pod deletion check to v1alpha1

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(k8s): address workspace namespace review findings

Signed-off-by: Derek Carr <decarr@redhat.com>

---------

Signed-off-by: Derek Carr <decarr@redhat.com>
2026-08-14 21:22:41 +00:00

363 lines
13 KiB
Protocol Buffer

// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
syntax = "proto3";
package openshell.compute.v1;
import "google/protobuf/struct.proto";
import "options.proto";
// Internal compute-driver contract used by the gateway.
//
// Conventions:
// - This file owns driver-native request, response, and observation types.
// - Compute drivers must not import or return the public `openshell.v1.Sandbox`
// resource model.
// - The gateway translates between these internal driver-native messages and
// the public OpenShell API resource model.
service ComputeDriver {
// Report driver capabilities and defaults.
rpc GetCapabilities(GetCapabilitiesRequest) returns (GetCapabilitiesResponse);
// Report additional gateway listeners required by this driver instance.
//
// A requirement is not authorization to expose the gateway. The gateway
// owns validation, authorization, and the authoritative bind.
rpc GetGatewayListenerRequirements(GetGatewayListenerRequirementsRequest)
returns (GetGatewayListenerRequirementsResponse);
// Validate a sandbox before create-time provisioning.
rpc ValidateSandboxCreate(ValidateSandboxCreateRequest)
returns (ValidateSandboxCreateResponse);
// Fetch the platform-observed sandbox state for one sandbox.
rpc GetSandbox(GetSandboxRequest) returns (GetSandboxResponse);
// List platform-observed sandbox state for all sandboxes.
rpc ListSandboxes(ListSandboxesRequest) returns (ListSandboxesResponse);
// Provision platform resources for a sandbox.
rpc CreateSandbox(CreateSandboxRequest) returns (CreateSandboxResponse);
// Idempotently stop platform resources without deleting persistent state.
rpc StopSandbox(StopSandboxRequest) returns (StopSandboxResponse);
// Idempotently start platform resources for a stopped sandbox.
rpc StartSandbox(StartSandboxRequest) returns (StartSandboxResponse);
// Tear down platform resources for a sandbox.
rpc DeleteSandbox(DeleteSandboxRequest) returns (DeleteSandboxResponse);
// Stream sandbox observations from the platform.
rpc WatchSandboxes(WatchSandboxesRequest) returns (stream WatchSandboxesEvent);
// Ensure platform resources for a workspace exist (e.g. namespace).
// Idempotent: succeeds if resources already exist.
rpc EnsureWorkspace(EnsureWorkspaceRequest) returns (EnsureWorkspaceResponse);
// Tear down platform resources for a workspace.
rpc DeleteWorkspace(DeleteWorkspaceRequest) returns (DeleteWorkspaceResponse);
}
message GetCapabilitiesRequest {}
message GetCapabilitiesResponse {
reserved 4, 5;
reserved "supports_gpu", "gpu_count";
// Human-readable driver name.
string driver_name = 1;
// Driver implementation version string.
string driver_version = 2;
// Default sandbox image recommended by the driver.
string default_image = 3;
}
message GetGatewayListenerRequirementsRequest {}
message GatewayListenerRequirement {
// Untrusted human-readable driver rationale for diagnostics.
string reason = 1;
oneof selector {
// Concrete IP:port address requested by the driver. The port must match
// the gateway's configured primary listener port.
string exact_bind_address = 2;
// Ask the gateway to bind the IPv4 address selected by its default route.
// This matches rootless pasta's default upstream-interface selection.
GatewayDefaultRouteInterfaceRequirement default_route_interface = 3;
// Ask the gateway to ensure an IPv4 loopback listener is present. This
// covers runtimes whose host forwarder terminates on gateway loopback.
GatewayLoopbackInterfaceRequirement loopback_interface = 4;
}
}
message GatewayDefaultRouteInterfaceRequirement {}
message GatewayLoopbackInterfaceRequirement {}
message GetGatewayListenerRequirementsResponse {
repeated GatewayListenerRequirement requirements = 1;
}
// Driver-owned sandbox model used for create requests and platform observations.
//
// This intentionally omits gateway-owned lifecycle fields such as the public
// `openshell.v1.SandboxPhase` and persisted metadata. The gateway derives and
// stores those fields after translating driver observations.
message DriverSandbox {
// Stable sandbox ID assigned by the gateway.
string id = 1;
// Compute-runtime sandbox name.
string name = 2;
// Compute-platform namespace or equivalent tenancy boundary.
string namespace = 3;
// Provisioning input supplied by the gateway. Drivers may omit this in
// observed snapshots returned by Get/List/Watch.
DriverSandboxSpec spec = 4;
// Raw platform-observed status.
DriverSandboxStatus status = 5;
// Workspace the sandbox belongs to. Used by drivers to construct
// collision-safe resource names and labels in shared-namespace mode.
string workspace = 6;
}
// Driver-owned provisioning inputs required to create a sandbox.
message DriverSandboxSpec {
// Log level exposed to processes running inside the sandbox.
string log_level = 1;
// Environment variables injected into the sandbox runtime.
map<string, string> environment = 5;
// Runtime template consumed by the driver during provisioning.
DriverSandboxTemplate template = 6;
// Portable resource requirements used by the gateway for driver selection
// and by drivers for provisioning.
ResourceRequirements resource_requirements = 9;
reserved 10;
reserved "gpu_device";
// Gateway-minted JWT identifying this sandbox to the gateway. Set by
// the gateway on create; the driver materialises it via its native
// secret mechanism (Docker/Podman/VM bind-mount a per-sandbox file;
// the Kubernetes driver ignores this field and relies on its projected
// ServiceAccount token bootstrap instead). Never echoed to the public
// Sandbox proto.
string sandbox_token = 11 [(openshell.options.v1.secret) = true];
}
message ResourceRequirements {
// GPU requirements for the sandbox. Presence indicates a GPU request.
GpuResourceRequirements gpu = 1;
}
// Driver GPU resource requirements.
message GpuResourceRequirements {
// Optional number of GPUs requested. When omitted, the request is for one
// GPU using the selected driver's default assignment behavior.
optional uint32 count = 1;
}
// Driver-owned runtime template consumed by the compute platform.
//
// This message describes the sandbox workload in backend-neutral terms.
// Platform-specific knobs (Kubernetes runtimeClassName, annotations,
// volumeClaimTemplates, etc.) belong in `platform_config`.
message DriverSandboxTemplate {
// Fully-qualified OCI image reference used to boot the sandbox.
string image = 1;
// Socket path inside the sandbox where the agent service listens.
string agent_socket_path = 3;
// Metadata labels applied to compute-platform resources.
// Drivers map these to the platform's native tagging mechanism
// (Kubernetes labels, cloud instance tags, etc.).
map<string, string> labels = 4;
// Additional environment variables injected into the sandbox runtime.
map<string, string> environment = 6;
// Typed compute-resource requirements for the sandbox workload.
DriverResourceRequirements resources = 10;
// Opaque, platform-specific configuration passed through to the driver.
// The gateway does not inspect this; each driver defines its own schema.
// For the Kubernetes driver this carries fields such as runtimeClassName,
// annotations, and volumeClaimTemplates.
google.protobuf.Struct platform_config = 11;
// Caller-provided config for the selected driver only.
// This is the inner block selected from public SandboxTemplate.driver_config.
// The selected driver owns nested schema validation.
google.protobuf.Struct driver_config = 12;
}
// Typed compute-resource requirements.
//
// Values use Kubernetes-style quantity strings (e.g. "500m", "2", "4Gi")
// because they are a well-known, widely-adopted notation. Drivers for
// non-Kubernetes platforms must parse these strings into their native units.
message DriverResourceRequirements {
// Minimum CPU cores requested (e.g. "500m", "2").
string cpu_request = 1;
// Maximum CPU cores allowed (e.g. "500m", "4").
string cpu_limit = 2;
// Minimum memory requested (e.g. "256Mi", "4Gi").
string memory_request = 3;
// Maximum memory allowed (e.g. "512Mi", "8Gi").
string memory_limit = 4;
}
// Raw status observed directly from the compute platform.
//
// The gateway derives the public `openshell.v1.SandboxPhase` from these
// conditions plus `deleting`.
message DriverSandboxStatus {
// Compute-platform sandbox object name.
string sandbox_name = 1;
// Platform-assigned instance identifier for the compute unit running the
// sandbox agent (e.g. Kubernetes pod name, VM instance ID, hostname).
// The gateway uses this to correlate incoming connections back to a sandbox.
string instance_id = 2;
// File descriptor or address for reaching the agent service inside the
// sandbox, when available.
string agent_fd = 3;
// File descriptor or address for reaching the sandbox supervisor service,
// when available.
string sandbox_fd = 4;
// Raw readiness and lifecycle conditions reported by the platform.
repeated DriverCondition conditions = 5;
// True when the compute platform has begun deleting this sandbox.
bool deleting = 6;
}
// Raw compute-platform condition.
message DriverCondition {
// Condition class reported by the compute platform.
string type = 1;
// Condition status value such as `True`, `False`, or `Unknown`.
string status = 2;
// Short machine-readable reason associated with the condition.
string reason = 3;
// Human-readable condition message.
string message = 4;
// Timestamp reported by the platform for the last transition.
string last_transition_time = 5;
}
// Raw compute-platform event correlated to a sandbox.
message DriverPlatformEvent {
// Event timestamp in milliseconds since epoch.
int64 timestamp_ms = 1;
// Event source (for example `kubernetes`).
string source = 2;
// Event type or severity (for example `Normal` or `Warning`).
string type = 3;
// Short machine-readable reason code.
string reason = 4;
// Human-readable event message.
string message = 5;
// Optional platform-specific metadata attached to the event.
map<string, string> metadata = 6;
}
message ValidateSandboxCreateRequest {
// Proposed sandbox configuration to validate before provisioning.
DriverSandbox sandbox = 1;
}
message ValidateSandboxCreateResponse {}
message GetSandboxRequest {
// Stable sandbox ID stored by the gateway.
string sandbox_id = 1;
// Compute-runtime name used by the driver.
string sandbox_name = 2;
}
message GetSandboxResponse {
// Platform-observed sandbox snapshot returned by the driver.
DriverSandbox sandbox = 1;
}
message ListSandboxesRequest {}
message ListSandboxesResponse {
// Platform-observed sandbox snapshots returned by the driver.
repeated DriverSandbox sandboxes = 1;
}
message CreateSandboxRequest {
// Sandbox configuration to provision on the compute platform.
DriverSandbox sandbox = 1;
}
message CreateSandboxResponse {}
message StopSandboxRequest {
// Stable sandbox ID stored by the gateway.
string sandbox_id = 1;
// Compute-runtime name used by the driver.
string sandbox_name = 2;
}
message StopSandboxResponse {}
message StartSandboxRequest {
// Stable sandbox ID stored by the gateway.
string sandbox_id = 1;
// Compute-runtime name used by the driver.
string sandbox_name = 2;
}
message StartSandboxResponse {}
message DeleteSandboxRequest {
// Stable sandbox ID stored by the gateway.
string sandbox_id = 1;
// Compute-runtime name used by the driver.
string sandbox_name = 2;
}
message DeleteSandboxResponse {
// True when a platform resource was deleted by this request.
bool deleted = 1;
}
message WatchSandboxesRequest {}
message WatchSandboxesSandboxEvent {
// Updated driver-native snapshot for one sandbox.
DriverSandbox sandbox = 1;
}
message WatchSandboxesDeletedEvent {
// Sandbox ID removed from the compute platform.
string sandbox_id = 1;
}
message WatchSandboxesPlatformEvent {
// Sandbox ID correlated to the platform event.
string sandbox_id = 1;
// Raw platform event emitted for the sandbox.
DriverPlatformEvent event = 2;
}
message WatchSandboxesEvent {
oneof payload {
// Updated or newly observed sandbox snapshot.
WatchSandboxesSandboxEvent sandbox = 1;
// Sandbox deletion observation.
WatchSandboxesDeletedEvent deleted = 2;
// Raw platform event correlated to a sandbox.
WatchSandboxesPlatformEvent platform_event = 3;
}
}
message EnsureWorkspaceRequest {
// Workspace identifier used by the gateway.
string workspace = 1;
}
message EnsureWorkspaceResponse {}
message DeleteWorkspaceRequest {
// Workspace identifier used by the gateway.
string workspace = 1;
}
message DeleteWorkspaceResponse {}