When no drivers are explicitly configured, the server now automatically
detects the appropriate compute driver by checking the runtime environment:
- Kubernetes: detected via KUBERNETES_SERVICE_HOST env var (set inside pods)
- Podman: detected by checking if podman binary is available on PATH
- Docker: detected by checking if docker binary is available on PATH
Priority order: Kubernetes → Podman → Docker. VM is never auto-detected
and must be selected explicitly via --drivers vm.
The Auto variant is internal-only and does not serialize to config files.
The default --drivers value is now empty, triggering auto-detection.
* feat(auth): add OIDC/Keycloak authentication with RBAC
Add OAuth2/OIDC authentication to the gateway server with role-based
access control, CLI login flows, and full deployment plumbing.
Server: JWT validation against configurable OIDC issuer (oidc.rs),
JWKS key caching with TTL and rotation handling, method classification
(unauthenticated/sandbox-secret/dual-auth/bearer), identity extraction
with provider-agnostic Identity type, and RBAC enforcement via
AuthzPolicy with configurable admin/user roles and auth-only mode.
CLI: browser-based Authorization Code + PKCE flow, Client Credentials
flow for CI/automation, token storage with refresh, gateway add/login/
logout commands, OIDC bearer token injection over mTLS transport,
discovery endpoint for auto-configuration.
Security: sandbox-secret scope restriction on UpdateConfig (policy
sync only), anti-spoofing header stripping, dual-auth fallthrough
from sandbox-secret to Bearer token.
Deployment: OIDC config wired through DeployOptions, Docker env vars,
Helm values/templates, HelmChart manifest, cluster-entrypoint.sh, and
bootstrap scripts. Keycloak dev server script with pre-configured
realm (test users, roles, PKCE client, CI client).
Tested with Keycloak. The roles claim path and role names are
configurable to support other OIDC providers.
* feat(auth): add OAuth2 scope-based fine-grained permissions
Add opt-in scope enforcement on top of existing OIDC role-based access
control. When --oidc-scopes-claim is set, the server extracts scopes
from the JWT and checks them per-method against an exhaustive scope map.
Scopes: sandbox:read, sandbox:write, provider:read, provider:write,
config:read, config:write, inference:read, inference:write, and
openshell:all (wildcard). Methods not in the scope map require
openshell:all. Scopes layer on top of roles and cannot escalate
privilege. Auth-only mode (empty role names) still enforces scopes
when enabled.
Server: scopes_claim in OidcConfig, scope extraction from JWT
(space-delimited and JSON array formats), standard OIDC scope
filtering, scope check in AuthzPolicy after role check.
CLI: --oidc-scopes on gateway add/start stored in metadata and
consumed by gateway login, --oidc-scopes-claim on gateway start
forwarded to server, scopes parameter in browser and client
credentials OAuth2 flows with openid deduplication.
Deployment: oidc_scopes_claim wired through DeployOptions, docker.rs,
Helm, bootstrap scripts, and cluster entrypoint.
Keycloak: realm config updated with built-in OIDC scopes and 9
OpenShell client scopes as optional on openshell-cli and openshell:all
as default on openshell-ci.
* fix(auth): address branch review findings
Add GetInferenceBundle to sandbox-secret methods so sandbox inference
route refresh works under OIDC. Make GetSandboxConfig dual-auth so CLI
users can read sandbox settings with Bearer tokens.
Preserve OIDC gateway metadata on restart — a bare gateway start
without --oidc-* flags no longer erases the stored OIDC registration.
Document CI client ID requirement (openshell-ci vs openshell-cli) in
the testing guide. Add security note about auth-only mode blast radius
for GitHub Actions.
* fix(auth): complete review findings for OIDC auth boundary
Move OpenShell/GetSandboxConfig from sandbox-secret-only to dual-auth
so CLI users can read sandbox settings with Bearer tokens while sandbox
supervisors continue using the shared secret.
Add sandbox secret interceptor to the inference bundle fetch path so
GetInferenceBundle works under OIDC-enabled gateways. Extract shared
interceptor constructor to avoid duplication.
Add GetSandboxConfig to the config:read scope map so scope enforcement
applies consistently when scopes are enabled.
Refactor OIDC metadata preservation into apply_oidc_gateway_metadata()
with explicit resume semantics — only preserve existing OIDC metadata
on real resume paths, not on fresh deployments.
Update architecture docs and testing guide to reflect the corrected
method classifications and add new test coverage for interceptor
injection, scope requirements, metadata preservation, and dual-auth
classification.
* refactor(auth): use oauth2 crate for CLI OIDC flows
Replace hand-written PKCE generation, authorization URL construction,
token exchange, client credentials, and token refresh with the oauth2
crate's typed API.
Eliminates sha2, hex, and getrandom dependencies from the CLI. The
custom urlencoded() helper and manual form POST logic are replaced by
BasicClient methods with proper type-state safety.
Discovery and the callback server remain custom since the oauth2 crate
does not provide OIDC discovery or a localhost redirect listener.
* refactor(auth): move server auth modules into auth/ directory
Group oidc.rs, authz.rs, identity.rs, and the auth HTTP endpoints
under src/auth/ module directory. No behavioral changes.
auth/mod.rs — module root, re-exports HTTP router
auth/oidc.rs — JWT validation, JWKS caching, method classification
auth/authz.rs — role and scope authorization policy
auth/identity.rs — provider-agnostic Identity type
auth/http.rs — /auth/connect and /auth/oidc-config endpoints
* fix(auth): use RequestBody auth type for client credentials flow
The oauth2 crate defaults to BasicAuth (HTTP Basic header) but Keycloak
and most OIDC providers expect client_secret_post (credentials in the
request body). Set AuthType::RequestBody explicitly to match the
pre-refactor behavior.
Also re-export Identity, IdentityProvider, and JwksCache from the auth
module so ServerState's public API remains nameable by external consumers.
* fix(auth): forward OPENSHELL_OIDC_SCOPES through cluster bootstrap
Pass --oidc-scopes to gateway start so the metadata includes requested
scopes after cluster bootstrap. Without this, users had to manually
edit metadata.json to set scopes for gateway login.
Usage: OPENSHELL_OIDC_SCOPES="openshell:all" mise run cluster
* test(auth): add OIDC e2e tests for RBAC, scopes, and client credentials
Add 10 end-to-end tests covering OIDC authentication against a live
K3s cluster with Keycloak:
RBAC (5 tests): admin can create providers, user cannot, user can list
sandboxes, unauthenticated requests rejected, health probe works
without auth.
Scopes (4 tests): sandbox-scoped token can list sandboxes but not
providers, openshell:all grants full access, no-scopes token denied.
Client credentials (1 test): CI token via client_credentials grant.
Tests are opt-in via OPENSHELL_E2E_OIDC=1 and OPENSHELL_E2E_OIDC_SCOPES=1
env vars. They derive the Keycloak URL from gateway metadata to match
the server's configured issuer.
Run with:
OPENSHELL_E2E_OIDC=1 OPENSHELL_E2E_OIDC_SCOPES=1 \
PYTHONPATH=python uv run pytest e2e/python/oidc/ -v
* fix(docs): fix markdown lint errors in OIDC architecture docs
Add blank lines before lists and fenced code blocks to satisfy
markdownlint MD031 and MD032 rules.
* Adding qemu vm driver support with GPU pass-through
Signed-off-by: Vincent Caux-Brisebois <vcauxbrisebo@nvidia.com>
* Add GPU rootfs variant, harden VFIO binding, and fix networking and supervisor reliability issues discovered during GPU VM bring-up.
Signed-off-by: Vincent Caux-Brisebois <vcauxbrisebo@nvidia.com>
---------
Signed-off-by: Vincent Caux-Brisebois <vcauxbrisebo@nvidia.com>
* feat(podman): add Podman compute driver for rootless sandbox management
Adds openshell-driver-podman, a new compute driver that manages OpenShell
sandboxes as rootless Podman containers via the Podman REST API over a
Unix socket. Enables local workstation sandboxes without Kubernetes.
Driver features:
- Bridge networking with ephemeral host-port mapping for rootless SSH reachability
- Named volumes for workspace storage, Podman native health checks, GPU via CDI
- Supervisor binary sideloaded via image volume mount (BYOC-compatible)
- SSH handshake secret injected via Podman secrets API (not plaintext env)
- Typed ContainerSpec structs, input validation, and path-traversal guards
- Cgroups v2 required; fails fast on v1 hosts
- Bounded event stream buffer; watch stream reconnection handled by server watch_loop
- Graceful shutdown and standalone driver binary with gRPC bridge
Rootless-specific fixes:
- Skip drop_privileges when user namespace lacks SETUID/SETGID/DAC_READ_SEARCH caps
- Add /run/netns tmpfs mount for ip netns in rootless containers
- Use secret_env map (not secrets array) for env-var injection in libpod API
- Resolve SSH endpoint to 127.0.0.1:<host_port> instead of unreachable bridge IP
Server/sandbox hardening:
- Split loopback and link-local SSRF gates; Podman/VM drivers allow loopback
- Close SSRF bypass in SSH tunnel Host path by resolving DNS before connecting
- Prevent OPENSHELL_* env var override by user-supplied spec environment maps
- Disable SQLite pool idle_timeout/max_lifetime for in-memory databases
- Emit deleted_event on 404-during-inspect instead of regressing sandbox phase
- Key delete cleanup by stable sandbox_id to survive container label drift
CLI fixes:
- Restore --name as a named flag on sandbox create (not positional)
- Fix exec command arg parsing to not consume sandboxed-command flags
- Propagate SSH verbosity via OPENSHELL_SSH_LOG_LEVEL
Build tooling:
- Add tasks/scripts/container-engine.sh: auto-detects Podman or Docker, exposes
unified ce_* helpers; all build/cluster/VM scripts updated to use it
- Add docker:build:supervisor mise task for standalone supervisor image
- Add openshell-driver-podman to Dockerfile.images pre-fetch/build stages
- Add e2e/rust/e2e-podman.sh and e2e:podman mise task for full lifecycle testing
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(driver-podman): derive grpc endpoint from server bind port
When a user starts the gateway on a non-default port (e.g. --port 8081),
sandbox containers were receiving OPENSHELL_ENDPOINT pointing at the
default port 8080. The driver's auto-detection fallback read
OPENSHELL_BIND_ADDRESS from the environment, which was stale or unset,
and fell back to DEFAULT_SERVER_PORT.
Add gateway_port to PodmanComputeConfig and thread config.bind_address.port()
from the server into the driver so the fallback uses the actual listening
port. Remove the OPENSHELL_BIND_ADDRESS env var read and the
extract_port_from_bind_address helper which are no longer needed.
Add --gateway-port / OPENSHELL_GATEWAY_PORT to the standalone driver
binary for parity when the driver is run outside the embedded server path.
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(driver-podman): address PR feedback on env test safety and cluster DNS docs
Replace hand-rolled unsafe TempEnvVar RAII guard with temp_env::with_vars
and a static ENV_LOCK mutex, fixing a data race in parallel test execution.
The prior safety comment incorrectly claimed Cargo runs tests single-threaded.
Update debug-openshell-cluster skill to accurately document the DNS proxy
strategy (setup_dns_proxy + public DNS fallback) and clarify the separation
between cluster DNS and sandbox agent DNS enforcement.
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(e2e): resolve CI failures in auth timeout, test harness, and formatting
- Short-circuit browser_auth_flow when OPENSHELL_NO_BROWSER=1 instead
of waiting the full 120s AUTH_TIMEOUT for a callback that never arrives
- Add timeout to SandboxGuard::create() and create_with_upload() to
prevent indefinite hangs (matches create_keep() which already had one)
- Add missing '--' separator in no_proxy test before command args
- Add #![cfg(feature = "e2e")] gate to sandbox_lifecycle.rs
- Run cargo fmt on openshell-driver-podman
- Refine cluster DNS docs for Podman in debug-openshell-cluster skill
Signed-off-by: Adam Miller <admiller@redhat.com>
* refactor(server): remove allows_loopback_endpoints from ComputeRuntime
SSRF protection is now handled at the network and proxy layers
(openshell-core net.rs, openshell-sandbox proxy.rs) rather than
requiring per-driver flags on ComputeRuntime. Update architecture
docs to reflect supervisor relay SSH transport and add rootless
networking deep-dive.
Signed-off-by: Adam Miller <admiller@redhat.com>
---------
Signed-off-by: Adam Miller <admiller@redhat.com>
* feat(server): add Prometheus metrics infrastructure and gRPC/HTTP request metrics
Add metrics exposition via a dedicated server port (--metrics-port,
default disabled) following the same optional-listener pattern as the
health port. The metrics crate facade records counters and histograms
in the MultiplexedService hot path, and a PrometheusHandle renders them
at GET /metrics on the dedicated port.
Metrics added:
- openshell_server_grpc_requests_total (counter: method, code)
- openshell_server_grpc_request_duration_seconds (histogram: method, code)
- openshell_server_http_requests_total (counter: path, status)
- openshell_server_http_request_duration_seconds (histogram: path, status)
* feat(helm): add metrics port to openshell-server helm chart
Expose the Prometheus metrics endpoint in the helm chart by adding
service.metricsPort (default 9090) to values, the --metrics-port arg
and container port to the statefulset, and a metrics service port.
Set metricsPort to 0 to disable.
Move /health, /healthz, and /readyz to a dedicated plaintext HTTP port
(default 8081) so Kubernetes probes work without mTLS client certificates.
- Add health_bind_address to Config with --health-port CLI arg
- Spawn standalone axum::serve for health_router on the health port
- Remove health routes from the main multiplexed HTTP router
- Update Helm statefulset probes from tcpSocket to httpGet on health port
- Fix cluster-healthcheck.sh to open/close TCP without sending data,
avoiding InvalidContentType TLS errors in the gateway log
Wrap gRPC and HTTP services with TraceLayer to log method, path,
status, and latency for every request. Health/readiness probe
endpoints are logged at debug level to reduce noise.
Policies with allowed_ips entries targeting loopback, link-local, or
unspecified ranges now fail at connection time instead of being silently
blocked at runtime. The shorthand log format for DENIED events includes
a [reason:...] suffix so operators can distinguish 'allowlist miss' from
'structurally un-allowable'. The mechanistic mapper skips proposals for
always-blocked destinations, preventing the infinite TUI notification
loop. The gateway validates proposed rules on approval as defense-in-depth.
- Extract shared IP helpers (is_always_blocked_ip, is_always_blocked_net,
is_internal_ip) to openshell_core::net
- Reject always-blocked entries in parse_allowed_ips with hard error
- Skip implicit allowed_ips synthesis for always-blocked literal IP hosts
- Add status_detail to HttpActivityBuilder for denial reason propagation
- Enrich NET and HTTP shorthand with [reason:...] for DENIED events
- Add engine: tag to HTTP shorthand (consistency with NET shorthand)
- Filter always-blocked proposals in mechanistic mapper generate_proposals
- Add validate_rule_not_always_blocked server-side defense-in-depth
- Update architecture docs, published docs, and E2E test assertions
TLD wildcards like *.com were accepted by policy validation but silently
failed at the proxy layer — 0 bytes returned, no denial logged. Now
rejected at submission time with a clear error message.
Closes#787
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* fix(install): restrict tar extraction to expected binary member
Prevents CWE-22 path traversal by extracting only the expected APP_NAME
member instead of the full archive contents. Adds --no-same-owner and
--no-same-permissions for defense-in-depth.
OS-20
* fix(deploy): quote registry credentials in YAML heredocs
Wraps username/password values with a yaml_quote helper to prevent YAML
injection from special characters in registry credentials (CWE-94).
Applied to all three heredoc blocks that emit registries.yaml auth.
OS-23
* fix(server): redact session token in SSH tunnel rate-limit log
Logs only the last 4 characters of bearer tokens to prevent credential
exposure in log aggregation systems (CWE-532).
OS-18
* fix(server): escape gateway_display in auth connect page
Applies html_escape() to the Host/X-Forwarded-Host header value before
rendering it into the HTML template, preventing HTML injection (CWE-79).
OS-17
* fix(server): prevent XSS via code param with validation and proper JS escaping
Adds server-side validation rejecting confirmation codes that do not
match the CLI-generated format, replaces manual JS string escaping with
serde_json serialization (handling U+2028/U+2029 line terminators), and
adds a Content-Security-Policy header with nonce-based script-src.
OS-16
* fix(sandbox): add byte cap and idle timeout to streaming inference relay
Prevents resource exhaustion from upstream inference endpoints that stream
indefinitely or hold connections open. Adds a 32 MiB total body limit
and 30-second per-chunk idle timeout (CWE-400).
OS-21
* fix(policy): narrow port field from u32 to u16 to reject invalid values
Prevents meaningless port values >65535 from being accepted in policy
YAML definitions. The proto field remains uint32 (protobuf has no u16)
with validation at the conversion boundary.
OS-22
* fix(deps): migrate from archived serde_yaml to serde_yml
Replaces serde_yaml 0.9 (archived, RUSTSEC-2024-0320) with serde_yml
0.0.12, a maintained API-compatible fork. All import sites updated
across openshell-policy, openshell-sandbox, and openshell-router.
OS-19
* fix(server): re-validate sandbox-submitted security_notes and cap hit_count
The gateway now re-runs security heuristics on proposed policy chunks
instead of trusting sandbox-provided security_notes, validates host
wildcards, caps hit_count at 100, and clamps confidence to [0,1]. The
TUI approve-all path is updated to use ApproveAllDraftChunks RPC which
respects the security_notes filtering gate (CWE-284, confused deputy).
OS-15
* chore: apply cargo fmt and update Cargo.lock for serde_yml
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
Check for existing sandbox name before persisting, matching the
provider-creation pattern. The CLI now surfaces a clear hint instead
of a raw UNIQUE constraint error.
Closes#691
* feat(sandbox): switch device plugin to CDI injection mode
Configure the NVIDIA device plugin to use deviceListStrategy=cdi-cri so
that GPU devices are injected via direct CDI device requests in the CRI.
Sandbox pods now only require the nvidia.com/gpu resource request —
runtimeClassName is no longer set on GPU pods.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(router): use max_completion_tokens for OpenAI GPT-5+ validation probe
OpenAI GPT-5 models reject the legacy max_tokens parameter and require
max_completion_tokens. The inference validation probe now sends
max_completion_tokens as the primary parameter, with an automatic
fallback to max_tokens when the backend returns HTTP 400 (for
legacy/self-hosted backends that only support the older parameter).
Closes#517
Signed-off-by: Maxime Grenu <maxime.grenu@gmail.com>
* style(router): fix cargo fmt import order and line length
---------
Signed-off-by: Maxime Grenu <maxime.grenu@gmail.com>
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* fix(sandbox,server): fix chunk merge duplicates and OPA variable collision with overlapping policies
Two related bugs triggered when a draft rule approval creates a second
policy entry for the same host:port:
1. merge_chunk_into_policy looked up existing rules by chunk.rule_name
(auto-generated as allow_{host}_{port}), which never matched the
user's original rule name. Now scans all network_policies entries
for a host:port endpoint match before falling back to insertion,
and merges allowed_ips into the existing endpoint.
2. The Rego allow_request rule and _matching_endpoint_configs
comprehension used 'some ep; ep := policy.endpoints[_]' which
caused regorus to error with 'duplicated definition of local
variable ep' when multiple policies covered the same host:port.
Refactored to isolate endpoint iteration inside helper functions
(_policy_allows_l7, _policy_endpoint_configs) so variables are
scoped per-policy evaluation.
Refs: #567
* test(e2e): add overlapping policy tests and update FWD-2 for implicit allowed_ips
- Update FWD-2 (test_forward_proxy_denied_without_allowed_ips ->
test_forward_proxy_allows_private_ip_host_without_allowed_ips):
literal IP host no longer requires explicit allowed_ips, expects 200.
- Add OVL-1: overlapping L4 policies for same host:port must not crash
OPA and should allow forward proxy connections.
- Add OVL-2: overlapping L7 policies for same host:port must not crash
OPA and should allow CONNECT tunnel establishment.
Refs: #567
* style: apply cargo fmt formatting
* test(e2e): update SSRF-3 and SSRF-6 for implicit allowed_ips behavior
SSRF-6: Private IP with literal IP host now gets implicit allowed_ips
from PR #570, so CONNECT returns 200 instead of 403.
SSRF-3: Loopback is still blocked but via the always-blocked path
(implicit allowed_ips is synthesized, then resolve_and_check_allowed_ips
catches it). Log message says 'always-blocked' instead of 'internal
address'.
* fix(e2e): use negative assertion for SSRF-6 when nothing listens on target port
When the SSRF check passes but nothing listens on the target port,
recv() returns empty bytes. Use 'assert 403 not in' (matching SSRF-4
pattern) instead of 'assert 200 in'.
* fix(e2e): update provider tests for redacted credential values
PR #569 changed credential redaction from clearing the map to
replacing values with 'REDACTED'. Update e2e assertions to expect
credential keys with REDACTED values instead of an empty map.
redact_provider_credentials used HashMap::clear() which removed both
keys and values, causing provider list/get to report 0 credential keys.
Replace with value-only redaction so key names and counts are preserved
while secret values remain hidden.
Refs: #350
* fix(l7): reject ambiguous HTTP framing in REST proxy (SEC-009)
Harden the L7 REST proxy HTTP parser against request smuggling:
- Reject requests containing both Content-Length and Transfer-Encoding
headers per RFC 7230 Section 3.3.3 (CL/TE ambiguity)
- Replace String::from_utf8_lossy with strict UTF-8 validation to
prevent interpretation gaps with upstream servers
- Reject bare LF line endings (require CRLF per HTTP spec)
- Validate HTTP version string (HTTP/1.0 or HTTP/1.1 only)
* fix(server): harden shell_escape and command construction (SEC-002)
Harden the gRPC exec handler against command injection via the
structured-to-shell-string conversion:
- Reject null bytes and newlines/carriage returns in shell_escape()
- Add input validation in exec_sandbox: reject control characters in
command args, env values, and workdir
- Enforce size limits: max 1024 args, 32 KiB per arg/value, 4 KiB
workdir, 256 KiB total assembled command string
- Change shell_escape and build_remote_exec_command to return Result
so callers must handle validation failures
* fix(server): add command validation at SSH transport boundary (SEC-003)
Add defense-in-depth validation in run_exec_with_russh before sending
the command to the sandbox SSH server:
- Reject null bytes in command string at transport boundary
- Enforce max command length (256 KiB) at transport boundary
- Enhance stream_exec_over_ssh logging with command length, stdin
length, and truncated command preview for audit trail
* fix(sandbox): validate port range and extend loopback check in SSH (SEC-007)
Harden the SSH direct-tcpip channel handler:
- Validate port_to_connect <= 65535 before u32-to-u16 cast to prevent
port truncation (e.g., 65537 becoming port 1)
- Replace string-literal loopback check with is_loopback_host() that
covers the full 127.0.0.0/8 range, IPv4-mapped IPv6 (::ffff:127.x),
bracketed IPv6, and case-insensitive localhost
- Remove #[allow(clippy::cast_possible_truncation)] since the cast is
now proven safe by the preceding range check
* fix(sandbox): sanitize inference error messages returned to sandbox (SEC-008)
Replace verbatim internal error strings in router_error_to_http with
generic messages to prevent information leakage to sandboxed code.
Upstream URLs, internal hostnames, TLS details, and file paths are no
longer exposed. Full error context is still logged server-side at warn
level by the caller for debugging.
* fix(server): block internal IPs in SSH proxy target validation (SEC-006)
Add IP validation in start_single_use_ssh_proxy to prevent SSRF if a
sandbox status record were poisoned:
- Resolve DNS before connecting and validate the resolved IP
- Block loopback (127.0.0.0/8) and link-local (169.254.0.0/16, covers
cloud metadata endpoint) addresses
- Block IPv4-mapped IPv6 variants of the same ranges
- Connect to the validated SocketAddr directly to prevent TOCTOU
- Add debug logging of resolved target IP for audit
* fix(sandbox): add resource limits to chunked body parser (SEC-010)
Harden parse_chunked_body in the inference interception path:
- Replace all unchecked +2 additions with checked_add for consistent
overflow safety across all target architectures
- Add MAX_CHUNKED_BODY (10 MiB) to cap decoded body size
- Add MAX_CHUNK_COUNT (4096) to prevent CPU exhaustion via tiny chunks
- Early-reject chunk sizes larger than remaining buffer space
* fix(cli): double-escape command for SSH path, validate host and name (SEC-004)
Harden doctor_exec against command injection in the SSH remote path:
- Apply shell_escape to inner_cmd in the SSH path so it survives the
double shell interpretation (SSH remote shell + sh -lc). This also
fixes a correctness bug where multi-word commands were silently
broken in the SSH path.
- Add validate_gateway_name to reject shell metacharacters in gateway
names before use in container_name
- Add validate_ssh_host to reject metacharacters in remote_host loaded
from metadata.json
* fix(sandbox): add CIDR breadth warning and control-plane port blocklist (SEC-005)
Defense-in-depth for the allowed_ips feature:
- Log a warning when a CIDR entry has a prefix length < /16, as overly
broad ranges may unintentionally expose control-plane services
- Block K8s API (6443), etcd (2379/2380), and kubelet (10250/10255)
ports unconditionally in resolve_and_check_allowed_ips, even when
the resolved IP matches an allowed_ips entry
* test(e2e): update assertion for sanitized inference error message (SEC-008)
The SEC-008 fix changed the error message from 'no compatible route
for source protocol ...' to 'no compatible inference route available'.
Update the E2E assertion substring to match.
* fix(sandbox): prevent overread request smuggling in L7 REST parser
The HTTP/1.1 parser used a 1024-byte read buffer that could capture
bytes from a pipelined second request. Those overflow bytes were
forwarded upstream as body overflow without L7 policy evaluation,
enabling request smuggling that bypasses per-request method/path
enforcement.
Replace multi-byte read with byte-at-a-time read_u8 that stops exactly
at the CRLFCRLF header terminator. Add regression test proving two
pipelined requests are parsed independently.
Refs: #350
* fix(server): harden sandbox TLS secret volume permissions to 0400
Kubernetes secret volumes default to 0644, allowing the unprivileged
sandbox user to read the mTLS client private key via the Landlock
baseline /etc read set. A compromised sandbox could use the key to
impersonate the control-plane client.
Set defaultMode to 256 (octal 0400, owner-read only) on both the
default and custom pod template paths. The supervisor reads TLS
materials as root before forking, so this does not affect normal
operation.
Refs: #350
* fix(sandbox): stop using cmdline paths for binary policy matching
/proc/pid/cmdline is fully attacker-controlled (argv[0] can be set to
any string via execve) and had no integrity verification, unlike
exec.path and ancestors which are kernel-managed and get TOFU/SHA256
checks. The Rego policy used cmdline_paths as a grant-access signal,
allowing any sandboxed process to claim the identity of an allowed
binary and bypass network restrictions.
Remove the cmdline exact-match rule and exclude cmdline_paths from the
glob-match rule. Only exec.path and exec.ancestors (from /proc/pid/exe)
are now used for binary identity. cmdline_paths remain in the OPA input
for deny-reason diagnostics only.
Refs: #350
* fix(sandbox): reject symlink and non-dir read_write paths before chown
The supervisor runs as root and calls chown on each read_write path.
Since chown follows symlinks, a malicious container image could place a
symlink (e.g. /sandbox -> /etc/shadow) to trick the supervisor into
transferring ownership of arbitrary files to the sandbox user.
Add symlink_metadata (lstat) check before chown to reject symlinks and
non-directory entries. The TOCTOU window is not exploitable because no
untrusted child process has been forked yet at this point.
Refs: #350
* fix(server): redact provider credentials in gRPC CRUD responses
Provider CRUD RPCs (create, get, list, update) returned full Provider
objects including plaintext credentials (API keys, secrets). Any
authenticated client -- including sandbox workloads running untrusted
code -- could read credentials for all providers.
Add redact_provider_credentials helper that clears the credentials map
before returning. Internal server paths (inference routing, sandbox env
injection) read from the store directly and are unaffected. Update
tests to verify redaction and assert persistence via direct store reads.
Refs: #350
* fix(sandbox): enforce non-root fallback when process user unset
drop_privileges silently returned Ok(()) when both run_as_user and
run_as_group were None, even when running as root. In local/dev mode
policies are loaded from disk without passing through the server-side
ensure_sandbox_process_identity normalization, so child processes could
retain root and all capabilities (SYS_ADMIN, NET_ADMIN).
When running as root with no process identity configured, fall back to
sandbox:sandbox instead of no-oping. Non-root runtimes are unaffected.
Refs: #350
* fix(sandbox): deny forward proxy for L7-configured endpoints
The forward proxy path only performed L4 (endpoint) and allowed_ips
checks. If an endpoint had L7 rules (method/path restrictions), a
sandboxed process could bypass them by using HTTP_PROXY with plain
http:// requests instead of CONNECT tunneling, since L7 inspection
only runs in the CONNECT path.
Add a guard in handle_forward_proxy that queries the endpoint's L7
config and returns 403 if any L7 rules are present, forcing traffic
through the CONNECT path where per-request inspection happens.
Refs: #350
* test(e2e): add regression test for forward proxy L7 bypass
Verifies that the forward proxy path (plain http:// via HTTP_PROXY)
returns 403 for endpoints with L7 rules configured, preventing
sandboxed processes from bypassing per-request method/path enforcement
by avoiding the CONNECT tunnel.
Refs: #350
* fix: update tests for cmdline path and TLS volume changes
Update OPA tests to verify cmdline_paths no longer grant access
(regression tests for the security fix). Fix formatting in TLS
volume test.
Refs: #350
* test(e2e): update provider e2e tests for credential redaction
Provider CRUD gRPC responses no longer include credential values.
Update three e2e tests to assert credentials are empty in responses.
Provider functionality is verified by existing e2e tests that check
env var injection into sandboxes (which read from the store directly).
Refs: #350
* chore: reformat for Rust 1.94 assert! macro style
* fix(sandbox): make drop_privileges tests root-aware for CI
CI runs as root but has no 'sandbox' user. The security fix correctly
errors when running as root with no process identity and no fallback
user available -- this is the intended behavior (refuse to run as root).
Update tests to expect that error in root-without-sandbox-user
environments instead of unconditionally asserting Ok.
Refs: #350
* fix(sandbox): allow non-directory read_write entries like /dev/null
The symlink guard incorrectly rejected all non-directory entries.
Character devices like /dev/null are legitimate read_write paths
used in sandbox policies. Only reject symlinks, which are the
actual attack vector for the chown privilege escalation.
Refs: #350
* feat(sandbox): log connection attempts that bypass proxy path
Add iptables LOG + REJECT rules inside the sandbox network namespace to
detect and diagnose direct connection attempts that bypass the HTTP
CONNECT proxy. This provides two improvements:
1. Fast-fail UX: applications get immediate ECONNREFUSED instead of a
30-second timeout when they bypass the proxy
2. Diagnostics: a /dev/kmsg monitor emits structured BYPASS_DETECT
tracing events with destination, protocol, process identity, and
actionable hints
Both TCP and UDP bypass attempts are covered (UDP catches DNS bypass).
The feature degrades gracefully if iptables or /dev/kmsg are unavailable.
Closes#268
* chore: track .python-version to pin Python 3.13.12 for uv
The sandbox base image runs Python 3.13. A stale venv on 3.12 causes
all exec_python E2E tests to fail because cloudpickle bytecode is not
compatible across minor versions.
* fix(cluster): preserve hostGatewayIP across fast deploys
The fast deploy's helm upgrade was missing the hostGatewayIP value that
the bootstrap entrypoint injects into the HelmChart CR. This caused
host.openshell.internal hostAliases to be lost from the gateway pod
and sandbox pods after any fast deploy, breaking host gateway routing.
Read the IP from the HelmChart CR and pass it through to helm upgrade.
* wip: fix iptables path resolution, use dmesg for kmsg, add CAP_SYSLOG
* fix(sandbox): restore NetworkNamespace Drop impl, remove dead kmsg code
The Drop impl for NetworkNamespace was accidentally deleted during the
bypass detection refactor, which would cause network namespaces and
veth interfaces to leak on every sandbox shutdown.
Also removes dead kmsg volume/mount code (bypass monitor uses dmesg
instead of direct /dev/kmsg access) and removes an accidentally
committed session transcript file.
Closes#273
Verify inference endpoints synchronously on the server during set/update, expose a --no-verify escape hatch in the CLI and Python helper, and return actionable failures when validation does not pass.
* feat(sandbox): inject host gateway hostAliases into sandbox pods
Sandbox pods running in the k3s cluster cannot resolve host.docker.internal
by default, preventing them from reaching services on the Docker host.
Detect the host gateway IP (default route) in the cluster entrypoint,
thread it through the Helm chart to the gateway server, and inject
hostAliases entries (host.docker.internal, host.openshell.internal)
into every sandbox pod spec. The injection is conditional -- when the
IP is empty (non-Docker deployments), no hostAliases are added.
---------
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
Compute version strings from git describe in openshell-core's build.rs
using the guess-next-dev scheme (e.g. 0.0.4-dev.6+g2bf9969). All binary
crates and the TUI splash screen now use the shared openshell_core::VERSION
constant instead of CARGO_PKG_VERSION.
In Docker/CI builds where .git is absent, falls back to CARGO_PKG_VERSION
which is already set correctly by the sed-patch pipeline. Also adds
OPENSHELL_CARGO_VERSION support to the cluster image and fast-deploy
supervisor builds for parity with the gateway.
* feat(sandbox): add gpu sandbox scheduling support
Allow sandbox creation to request GPU resources explicitly or infer them from GPU image names. This wires GPU intent through bootstrap, validates gateway support, and adds dedicated GPU E2E coverage for follow-up cluster testing.
Drew Newberry