mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-30 07:33:59 +08:00
v0.0.59
121
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Russell Bryant
|
7274a6bea3 |
feat(cli): add --env flag to sandbox create/exec and fix env var passthrough (#1730)
* feat(cli): add --env flag to sandbox create and exec
Add `--env KEY=VALUE` (repeatable) to `openshell sandbox create` and
`openshell sandbox exec`, wiring user-specified environment variables
into the existing `SandboxSpec.environment` and
`ExecSandboxRequest.environment` protobuf fields which were previously
only accessible via the SDK/gRPC API.
Fix a bug where user-specified environment variables set via
`SandboxSpec.environment` were silently dropped inside sandbox SSH
sessions. The SSH handler's `apply_child_env()` calls `env_clear()` to
strip supervisor-internal secrets before spawning user shells, but this
also wiped container-level env vars including user-specified ones. This
was introduced in
|
||
|
Eric Curtin
|
1399f371e0 |
refactor: deduplicate OCSF builder setters and persistence helpers (#1800)
Eliminate three categories of repeated code:
1. OCSF builder setter methods
Add six macros to builders/mod.rs (impl_activity_setter!,
impl_action_disposition_setters!, impl_actor_process_setter!,
impl_dst_endpoint_setter!, impl_src_endpoint_addr_setter!,
impl_firewall_rule_setter!) that generate the identical one-liner
setters duplicated across network, http, ssh, process, finding,
and lifecycle builders. Removes ~55 hand-written method bodies.
2. Persistence protobuf decode-and-hydrate block
Extract a private decode_record<T>() helper that decodes an
ObjectRecord into a protobuf message and hydrates resource_version.
The identical three-line block was repeated in get_message,
get_message_by_name, list_messages, and list_messages_with_selector.
Each method now delegates to decode_record via .map().transpose()
or .into_iter().map().collect().
3. Store enum dispatch arms
Add a store_dispatch! macro that expands
match self { Postgres(s) => s.method(...).await, Sqlite(s) => ... }.
Applied to all 13 dispatch-only methods on Store, removing ~80
lines of mechanical match arms with zero logic.
|
||
|
Shiju
|
25abc9e3c7 |
feat(inference): allow local embeddings route (#1774)
* feat(inference): allow local embeddings route Route OpenAI-compatible embeddings through the local inference proxy so sandboxed vector workloads reach a configured provider via the same route classification and auth path that chat, completion, and model discovery already use. - Add openai_embeddings to the OpenAI-compatible protocol set so providers (openai, nvidia) advertise embeddings routing. - Classify POST /v1/embeddings as the openai_embeddings protocol in the sandbox L7 patterns. - Serve embeddings buffered with an accurate Content-Length, since the response is a single JSON object rather than an SSE token stream. The streaming path appends an SSE error frame on a size-cap or idle-timeout truncation, which would corrupt a one-object body the client parses whole. protocol_returns_buffered_body() selects the path. - Probe an embeddings-only backend against /v1/embeddings during validation, after the chat and completion protocols so a multi-protocol route still prefers those. - Extract two shared helpers. http_status_text() backs both response formatters and adds 401/422/429/503 for embeddings passthrough and router error mapping; write_inference_router_error() backs the streaming and buffered routing paths. - Return an OpenAI-shaped embeddings body from the mock route. Tests cover profile lookup, L7 pattern detection, the mock body, and buffered Content-Length framing with no chunked transfer-encoding and no SSE error frame. Signed-off-by: Shiju <shiju@nvidia.com> * fix(inference): cap buffered inference response body The buffered proxy path read the whole upstream response into memory with no size bound. The route timeout bounds elapsed time but not memory, so a misbehaving or oversized upstream could force unbounded allocation in the sandbox proxy. The streaming path already caps each response at 32 MiB; the buffered path did not. Cap the buffered read at the same 32 MiB. An advertised over-cap body is rejected from its Content-Length before any bytes are read, and chunks accumulate under the same bound so a chunked or mislabeled body cannot slip past. An over-cap response fails as an upstream protocol error, surfaced as HTTP 502 at the proxy boundary, and is never partially returned. Tests - cargo test -p openshell-router \ proxy_to_backend_rejects_over_cap_response_body Signed-off-by: Shiju <shiju@nvidia.com> * fix(inference): validate embeddings models against all advertised protocols A managed route resolves to its provider profile's full protocol set, so an embeddings model such as text-embedding-3-small lists openai_chat_completions alongside openai_embeddings. Route verification probed only the first writable protocol and stopped on its failure. It sent a chat probe with the embedding model, the provider rejected it as wrong-shape, and the route failed validation before the embeddings probe ran. Embeddings-only configs could not be verified. Try the advertised protocols in preference order. A request-shape rejection (HTTP 400, 404, 405, 422) falls through to the next protocol, so an embeddings model validates against /v1/embeddings even when the chat probe rejects it. Credential, rate-limit, connectivity, and upstream-health failures stay terminal and stop validation at the first probe, so a bad key or a down backend is reported as itself rather than masked by a later probe. validation_probe becomes validation_probes, which returns the ordered list, and the per-probe fallback retry (max_completion_tokens versus max_tokens) moves into a shared helper. Tests - cargo test -p openshell-router \ verify_embeddings_model_falls_through_chat_probe - cargo test -p openshell-router verify_stops_on_credentials_failure Signed-off-by: Shiju <shiju@nvidia.com> * fix(inference): serve model discovery responses buffered GET /v1/models returns a single JSON model list, the same response shape as embeddings. The sandbox inference proxy was routing it through the SSE streaming path. A streaming size-cap or idle-timeout truncation appends an SSE error frame to the body, which corrupts a payload the client parses as one JSON object. Make response framing a property of the protocol. A new ResponseFraming field on InferenceApiPattern is set once per pattern in default_patterns. model_discovery and openai_embeddings are now Buffered, while chat completions, completions, responses, and Anthropic messages stay Streaming. The proxy dispatch gates on pattern.is_buffered(), which replaces the stringly-typed protocol_returns_buffered_body predicate so the streaming-versus-buffered decision lives in one place and cannot drift across the sites that read it. Model discovery now flows through the same buffered path as embeddings, framed with an accurate Content-Length and bounded by the buffered-read size cap that path already enforces. Tests - cargo test -p openshell-sandbox protocol_framing_classification - cargo test -p openshell-sandbox \ inference_model_discovery_served_buffered_with_content_length Signed-off-by: Shiju <shiju@nvidia.com> * docs(inference): document embeddings route in supported patterns Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
Calum Murray
|
b392b2eefe |
feat(providersv2): add path auth_style (#1622)
Signed-off-by: Calum Murray <cmurray@redhat.com> |
||
|
Evan Lezar
|
c3964a6513 |
feat(kubernetes): support driver config passthrough (#1744)
* test(docker): default sandbox template fixtures Signed-off-by: Evan Lezar <elezar@nvidia.com> * feat(cli): add driver config json plumbing Signed-off-by: Evan Lezar <elezar@nvidia.com> * feat(kubernetes): tolerate invalid driver config Signed-off-by: Evan Lezar <elezar@nvidia.com> * feat(kubernetes): prototype driver config specifics Signed-off-by: Evan Lezar <elezar@nvidia.com> * docs(kubernetes): note driver-local config schema boundary Signed-off-by: Evan Lezar <elezar@nvidia.com> * feat(kubernetes): support runtime class driver config Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Shiju
|
97986d9059 |
fix(server): resume unspecified sandbox phase (#1765)
Legacy or never-initialized sandbox rows decode to the proto-default Unspecified phase. The startup resume sweep skipped them, leaving the backend unreconciled: never resumed, never errored. Unspecified was the only non-terminal phase skipped, since unrecognized values already fall through to Unknown and get resumed. Treat Unspecified as running so the sweep reconciles it from live driver state: resume when the backend resource exists, mark Error when it is gone. Tests - cargo test -p openshell-server \ resume_persisted_sandboxes_resumes_running_phases Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
LateNightHackathonandlatenighthackathon
|
c26d4e8e1f |
fix(grpc): allow credential rotation when legacy provider.type exceeds current limit (#1350)
Closes #1347. Reported by @KodeDaemon in the NVIDIA Developer Discord after upgrading NemoClaw 0.0.38 -> 0.0.39 left the inference provider stranded with `provider.type exceeds maximum length (79 > 64)` on every credential update. `update_provider_record` rebuilt the merged Provider from `existing.r#type` and ran `validate_provider_fields` over it. The CLI's `provider update` sends `r#type: ""`, so the existing value is preserved without modification, but the validator still measured it. Any record whose stored type predates current MAX_PROVIDER_TYPE_LEN (or was written by a path that bypassed validation, e.g. the TUI form which forwards r#type to the request without going through normalize_provider_type) became unupdateable: the only escape was `provider delete` + recreate, which loses any provider.config entries the caller never sees. Split the validator into create-time (validate_provider_fields, full check including immutable name/type) and update-time (validate_provider_mutable_fields, checks only credentials/config maps). The update path validates only what the caller is mutating; immutable fields carried forward from existing are trusted because they passed validation when stored. Tests: new regression in `update_provider_allows_credential_rotation_on_legacy_oversized_type` inserts a Provider with a 79-char type directly via `store.put_message` (bypassing gRPC validation), updates a credential, asserts the update succeeds and the original type is preserved. Existing `update_provider_validates_merged_result` still covers the case where incoming credential entries push the merged map past limits. Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com> Co-authored-by: latenighthackathon <latenighthackathon@users.noreply.github.com> |
||
|
Eric Curtin
|
69764d8a91 | refactor: deduplicate shared driver and TUI helpers (#1741) | ||
|
Taylor Mutch
|
5f58cb018f |
fix(helm): create sandbox JWT secret when cert-manager is enabled (#1700)
* fix(helm): create sandbox JWT secret under cert-manager The cert-manager install path (certManager.enabled=true, pkiInitJob.enabled=false) left the gateway StatefulSet unable to start because nothing created the openshell-jwt-keys Secret: cert-manager owns TLS Secrets but does not mint the sandbox JWT signing key, and the certgen hook only rendered when pkiInitJob.enabled was true. Separate JWT signing-key provisioning from TLS PKI provisioning: - certgen: add a --jwt-only mode that creates only the Opaque JWT signing Secret, for use when another controller owns TLS Secrets. - certgen.yaml: render the hook when pkiInitJob.enabled OR certManager.enabled is true. cert-manager takes precedence and runs the hook with --jwt-only even if pkiInitJob.enabled remains true. Remove the mutual-exclusion failure between the two values. - _helpers.tpl: add openshell.sandboxJwtSecretName, shared by the hook and the StatefulSet mount. - Update values, README, docs, architecture, and the debug-openshell-cluster skill to reflect the new precedence; the documented cert-manager install no longer needs pkiInitJob.enabled=false. Closes #1691 * fix(helm): honor cert-manager precedence for client CA volume The client CA volume logic treated pkiInitJob.enabled as proof that built-in PKI owns the client CA. With cert-manager precedence now allowing certManager.enabled=true alongside the default pkiInitJob.enabled=true, that assumption mounts the server TLS cert secret as the client CA and ignores certManager.clientCaFromServerTlsSecret=false, which can break mTLS or trust the wrong CA. Gate the pkiInitJob.enabled term with (not certManager.enabled) in all three client CA conditions (volume mount, volume definition, and secret selection) so cert-manager owns TLS when enabled. Add a Helm test suite covering built-in PKI, cert-manager shared CA, the regression config (cert-manager + clientCaFromServerTlsSecret=false + default pkiInitJob), and the no-client-CA case. |
||
|
Taylor Mutch
|
e4bcfdfaaa | fix(gateway): allow local sandbox jwt to not expire (#1721) | ||
|
Eric Curtin
|
d5b79e5ba1 |
refactor(server): deduplicate test helpers and grpc utilities (#1708)
Remove three groups of copy-pasted code in openshell-server: 1. grpc/mod.rs had a private current_time_ms() wrapper identical to the one already exported from persistence/mod.rs. Remove the duplicate and update the three grpc sub-modules (policy, sandbox, service) to import directly from crate::persistence. 2. test_store() was repeated verbatim in seven #[cfg(test)] blocks. Promote a single canonical version to persistence/mod.rs (cfg-gated) and replace all copies with crate::persistence::test_store() calls or a thin Arc wrapper in supervisor_session. 3. grpc_client_mtls() and build_tls_root() were copy-pasted across edge_tunnel_auth.rs and multiplex_tls_integration.rs. Move both into the existing tests/common/mod.rs shared module and import from there. |
||
|
John T. Myers
|
62c421b21e |
feat(providers): add profile-backed policy visibility (#1640)
* chore: wip providers v2 tui and codex profile * chore: wip effective policy get and codex profile * chore: wip provider profiles and tui detail views * feat(tui): annotate policy proposal review status |
||
|
Adam Miller
|
f061b1d923 |
feat(providers): add Google Vertex AI inference provider (#1568)
* feat(providers): add Google Vertex AI provider Adds Vertex AI provider profiles, routing, credential refresh plumbing, CLI support, docs, and regression coverage. Keeps the related NETLINK_ROUTE seccomp allowance needed by Vertex client tooling that calls getifaddrs. * docs: add Vertex AI sandbox usage for Claude Code and OpenCode Cover the full end-to-end setup for running Claude Code and OpenCode inside an OpenShell sandbox via inference.local with a Vertex AI backend: - google-vertex-ai.mdx: add 'Use from a Sandbox' section with tabbed examples for Claude Code (--bare flag, no /v1 suffix) and OpenCode (/v1 suffix required). Add providers_v2_enabled prerequisite and --no-verify note for global region. Document policy proposals table covering metadata.google.internal (always blocked), downloads.claude.ai, and storage.googleapis.com. - inference-routing.mdx: expand 'Use the Local Endpoint' section with tabbed examples for Claude Code, OpenCode, Python OpenAI SDK, and Python Anthropic SDK. Add notes explaining the /v1 path suffix difference between clients. - supported-agents.mdx: update Claude Code and OpenCode rows to mention inference.local support and correct base URL requirements. * fix: address vertex review findings * test(sandbox): retry on spurious Ok in fork-exec ambiguity test On arm64 under heavy CI load, the /proc fd scan in find_socket_inode_owners can transiently miss the parent process's socket fd entry, returning only the child as an owner. This causes resolve_process_identity to return Ok (single owner, no ambiguity check fires) instead of the expected ambiguous-ownership Err. Extend the retry loop to also handle unexpected Ok results, mirroring the existing retry for transient Err results. 10 retries at 50ms gives a 500ms settling window, which is sufficient for procfs to stabilize on loaded arm64 runners. * fix: address vertex review regressions * docs(router): clarify stream_response semantics for Vertex rawPredict routing Document the three call sites of prepare_backend_request and their stream_response values in a caller table: - send_backend_request: false → :rawPredict (unary endpoint) - send_backend_request_streaming: true → :streamRawPredict - verify_backend_endpoint: explicitly false to probe the unary endpoint Cross-reference the table from build_provider_url and is_vertex_anthropic_rawpredict_route so the stream_response=true guard in the suffix upgrade branch is understood in full context. Also note that is_vertex_anthropic_rawpredict_route is a structural predicate (model_in_path + anthropic_messages + :rawPredict suffix), not a named-provider check, so any future provider with the same route shape inherits the transforms automatically. |
||
|
Evan Lezar
|
79aa355dd0 |
refactor(driver): trim compute capability response (#1402)
Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Piotr Mlocek
|
3d441e73ac |
fix(config): reject unknown fields in nested gateway config tables (#1666)
* fix(config): reject unknown fields in nested gateway config tables The gateway TOML loader silently ignored keys placed under the wrong table header. PR #1661 fixed one instance of this (supervisor_image under [openshell.gateway.gateway_jwt]) but the root cause remained: the nested gateway config tables did not deny unknown fields, so a misplaced key was accepted and dropped instead of erroring. Concretely, tasks/scripts/gateway.sh emitted `sandbox_namespace` right after the [openshell.gateway.gateway_jwt] heredoc, so it landed inside the gateway_jwt table rather than [openshell.gateway]. The k8s driver already receives the namespace via [openshell.drivers.kubernetes], so the stray line was dead config that parsed without complaint. Changes: - Add #[serde(deny_unknown_fields)] to the nested gateway config tables that are part of the config-file parse tree: TlsConfig, OidcConfig, MtlsAuthConfig, GatewayAuthConfig, GatewayJwtConfig. - Remove the misplaced sandbox_namespace line from gateway.sh. - Drop the unused Serialize/Deserialize derives from Config and ServiceRoutingConfig (see below). - Add a regression test asserting a key under the wrong nested table is rejected. |
||
|
Kirit ThadakaandJohn Myers
|
c63ac76d06 |
feat(telemetry): add anonymous opt-out OpenShell usage telemetry (#1433)
* feat(telemetry): add anonymous opt-out usage telemetry Signed-off-by: Kirit93 <kthadaka@nvidia.com> * Removed enums from schema Signed-off-by: Kirit Thadaka <kthadaka@nvidia.com> * Updated telemetry URL Signed-off-by: Kirit93 <kthadaka@nvidia.com> * ci(kubernetes): pin mise installer for e2e --------- Signed-off-by: Kirit93 <kthadaka@nvidia.com> Signed-off-by: Kirit Thadaka <kthadaka@nvidia.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Alexander Watson
|
e98ea3ee93 | feat(policy): add agentic approval loop (#1528) | ||
|
Taylor Mutch
|
0f73d117a5 |
fix(podman): avoid host-gateway on macOS machines (#1637)
Closes #1307 Default the Podman host gateway alias override to gvproxy's host-loopback IP on macOS while preserving host-gateway resolution on Linux. Wire the setting through Podman config, gateway TOML inheritance, and the standalone driver, and document the platform behavior. Signed-off-by: Taylor Mutch <taylormutch@gmail.com> |
||
|
Derek Carr
|
d01d10650b |
refactor(proto): move phase and current_policy_version into status (#1565)
* refactor(proto): move phase and current_policy_version into SandboxStatus Move phase and current_policy_version from SandboxSpec into SandboxStatus to correctly model mutable runtime state. Update all callers in the gateway server, TUI, and Python SDK to read and write these fields through SandboxStatus accessors. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(server): preserve sandbox status on statusless driver updates When a driver update arrives without a status payload (e.g. before Kubernetes populates the status subresource), preserve the stored phase, conditions, and current policy version instead of resetting them. Adds a regression test covering the edge case. Signed-off-by: Derek Carr <decarr@redhat.com> --------- Signed-off-by: Derek Carr <decarr@redhat.com> |
||
|
Mrunal Patel
|
3f520dd4e2 |
feat(server): declare gRPC auth (mode + scope + role) at the handler, enforce at the router (#1596)
* feat(server): per-handler gRPC auth annotations Move scope, role, and auth-mode metadata to the handler definition site via #[rpc_authz] + #[rpc_auth] proc macros. The previously hand-maintained SCOPED_METHODS, ADMIN_METHODS, UNAUTHENTICATED_METHODS, and ALLOWED_SANDBOX_METHODS tables are now generated from per-method annotations on the tonic service impls, with canonical gRPC paths derived from the service name and method name. Adds a new openshell-server-macros proc-macro crate, an aggregator in auth/method_authz.rs, and an exhaustiveness test that decodes the protobuf FileDescriptorSet (now emitted by openshell-core/build.rs) and verifies every RPC has an annotation. Signed-off-by: Mrunal Patel <mrunalp@gmail.com> * refactor(server): rename `sandbox-secret` auth mode to `sandbox` PR #1404 replaced the shared sandbox secret with per-sandbox gateway-minted JWTs. A handler marked `sandbox` now authenticates as a specific `Principal::Sandbox`, not as a holder of a shared credential. Rename `auth = "sandbox-secret"` to `auth = "sandbox"` and `AuthMode::SandboxSecret` to `AuthMode::Sandbox` so the name matches the post-#1404 identity model. Signed-off-by: Mrunal Patel <mrunalp@gmail.com> * fix(server): enforce per-handler AuthMode at the router Addresses review feedback on the per-handler auth-annotation work. - Router-level enforcement of #[rpc_auth] auth mode (HIGH). The previous router only checked is_sandbox_callable() for Principal::Sandbox; user principals still flowed into AuthzPolicy::check() and bypassed the per-handler declaration. A user with `openshell:all` could therefore reach `sandbox`-only handlers like GetSandboxProviderEnvironment, ReportPolicyStatus, PushSandboxLogs, and SubmitPolicyAnalysis even though their annotations said sandbox-only. Adds an is_user_callable() predicate and rejects User principals at the router for `sandbox` / `unauthenticated` methods. - Proc macro now errors on duplicate keys in #[rpc_auth(...)] (LOW). A second `auth`, `scope`, or `role` previously silently overwrote the first value; now it fails to compile. - Regression tests: a unit test for is_user_callable() and a router test that proves a user with admin role + openshell:all cannot reach the nine sandbox-only handlers. Signed-off-by: Mrunal Patel <mrunalp@gmail.com> * docs(server): finish renaming sandbox-secret to sandbox in method_authz doc comments Signed-off-by: Mrunal Patel <mrunalp@gmail.com> * refactor(server-macros): drop standalone `rpc_auth` stub The stub was a safety net that fired only when a method had `#[rpc_auth(...)]` without an enclosing `#[rpc_authz]`. Triggering it required `rpc_auth` to be imported, which is why both call sites carried `#[allow(unused_imports)] use openshell_server_macros::{rpc_auth, rpc_authz};`. Drop the stub and the unused-import workaround. A missing `#[rpc_authz]` now surfaces as rustc's standard "cannot find attribute `rpc_auth` in this scope" — clear enough, and one fewer import + lint exception. Addresses review comment on PR #1596. Signed-off-by: Mrunal Patel <mrunalp@gmail.com> * refactor(server-macros): emit fixed `AUTH_METADATA` const per service The previous trait-derived const name turned `OpenShell` into `OPEN_SHELL_AUTH_METADATA`, splitting the project name across an underscore. Each impl already lives in its own module (`crate::grpc::`, `crate::inference::`), so the module path is enough to disambiguate between services — a fixed `AUTH_METADATA` name reads more naturally. Aggregator in `auth/method_authz.rs` now references `crate::grpc::AUTH_METADATA` and `crate::inference::AUTH_METADATA` directly. Addresses review comment on PR #1596. Signed-off-by: Mrunal Patel <mrunalp@gmail.com> * docs(server-macros): fix typo in AUTH_METADATA_CONST doc comment OpenShell is one word; reference name in the doc should be OPENSHELL_AUTH_METADATA, not OPEN_SHELL_AUTH_METADATA. Addresses review nit on PR #1596. Signed-off-by: Mrunal Patel <mrunalp@gmail.com> --------- Signed-off-by: Mrunal Patel <mrunalp@gmail.com> |
||
|
alangou
|
2bdc968ed9 |
fix(gateway): make readiness health checks dependency-aware (#1328)
* feat(gateway): add readiness probe metrics and test-only store close Emit Prometheus readiness metrics for database probes (healthy gauge and outcome-labeled latency histogram) with coverage in health HTTP tests. Restrict Store::close behind test support cfg to prevent accidental runtime pool shutdown under live traffic. Signed-off-by: Adrien Langou <alangou@nvidia.com> * test(e2e): add simple e2e test with kubernetes to test /readyz Signed-off-by: Adrien Langou <alangou@nvidia.com> --------- Signed-off-by: Adrien Langou <alangou@nvidia.com> |
||
|
Taylor Mutch
|
7d38aa8b8e |
fix(homebrew): repair local driver bootstrap state (#1527)
* fix(homebrew): repair local driver bootstrap state * fix(bootstrap): satisfy default SAN doc lint |
||
|
John T. Myers
|
0cef26521a |
feat(providers): derive discovery from profiles (#1503)
* feat(providers): derive discovery from profiles Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(providers): keep v2 discovery profile-only Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * docs(providers): update providers v2 behavior Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(providers): make github profile read-only Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Taylor Mutch
|
a3b16c18ab | feat(auth): per-sandbox authentication to gateway (#1404) | ||
|
Colin Walters
|
e7f965a988 |
refactor(sandbox,driver-vm): Start moving to rustix (esp over libc unsafe) (#1505)
In the Rust ecosystem there's largely three ways to do system calls: - raw libc - nix - rustix Of the three, libc is almost all `unsafe` and really 95% of use cases should be either nix or rustix. nix is the original one, but after having looked at the code of both, I think rustix is just better designed and organized. It's also reached 1.0, whereas nix is still making semver-breaking changes (in fact we're behind here in this project). Now in practice, we have both *transitively* in the depchain already, and that's true for quite a lot of projects. But I think rustix is better, so let's add rustix as a workspace dependency (process feature) and migrate a few use cases to it - it's especially better than the raw libc which is suprisingly widespread. If we agree to do this, then many other calls can be ported. Signed-off-by: Colin Walters <walters@verbum.org> |
||
|
Russell Bryant
|
af753748d3 |
fix(server): respect OPENSHELL_PODMAN_SOCKET env var in embedded driver (#1483)
The env var was only wired up via clap in the standalone openshell-driver-podman binary. When the Podman driver runs embedded in the gateway, config came exclusively from TOML deserialization and the env var was never consulted. Apply it as a post-deserialization override, matching the existing OPENSHELL_K8S_WORKSPACE_DEFAULT_STORAGE_SIZE pattern. Closes #1446 |
||
|
Eric Curtin
|
5620c8b229 |
refactor: deduplicate repeated patterns across crates (#1499)
Remove ~280 lines of duplicated code across 30 files in 5 areas: - centered_rect: consolidate 5 identical TUI layout helpers into a single pub fn in openshell-tui/src/ui/mod.rs - server test helpers: replace ~100 inline Store::connect() calls with local test_store() helpers; deduplicate test_server_state() in grpc/service.rs to use the shared test_support version - rogue PKI: extract 20-line rogue CA+client cert generation block (duplicated in two integration tests) into generate_rogue_pki() in tests/common/mod.rs - provider tests: replace 8 identical 28-line test modules with a single macro_rules! test_discovers_env_credential! invocation - label constants: centralize openshell.ai/ container label keys in openshell-core::driver_utils; update Docker and Kubernetes drivers to import from there instead of redefining them locally |
||
|
mjamivandJohn Myers
|
528fb29147 |
fix(sandbox): allow first-label L7 host wildcards (#1304)
* fix(sandbox): allow first-label L7 host wildcards * docs(sandbox): document L7 host wildcard contract + add OPA runtime tests - Add Host Wildcards section to architecture/security-policy.md describing accepted (first-label *, **, intra-label *-X) and rejected (bare, TLD, non-first-label, recursive-in-label) forms, and noting that wildcards never cross '.' boundaries. - Expand the policy-schema.mdx 'host' field description to reflect the same contract instead of only mentioning '*.example.com'. - Add OPA runtime tests asserting '*-aiplatform.googleapis.com' matches 'us-central1-aiplatform.googleapis.com' and does not match 'us-central1.aiplatform.googleapis.com' (cross-dot boundary). Locks validator/runtime alignment for intra-label wildcards. * chore: update mise lockfile * test(server): tolerate serialized inference upserts --------- Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Eric Curtin
|
e3f009fb9a | refactor(server): extract shared relay-await and sandbox-scan helpers (#1495) | ||
|
Drew NewberryandTaylor Mutch
|
77e6c7a12a |
test(server): cover service endpoint plaintext security (#1352)
* test(server): cover service endpoint plaintext security * test(server): align tls test with from_files Option<&Path> signature TlsAcceptor::from_files now accepts the client CA path as Option<&Path> (per the require_client_auth refactor on main). Wrap the helper's CA path in Some(...) so the new plaintext-service-http tests compile after rebasing onto current main. --------- Co-authored-by: Taylor Mutch <taylormutch@gmail.com> |
||
|
Adel Zaalouk
|
bdaa08fbb0 | fix(server): add ConnectSupervisor and RelayStream to SANDBOX_METHODS (#1475) | ||
|
Eric Curtin
|
b332ffdf7d |
refactor: deduplicate shared driver and provider constants (#1474)
Move default_sandbox_image() and SUPERVISOR_IMAGE_BINARY_PATH into openshell-core so all compute drivers share a single authoritative definition. Eliminates four identical copies of default_sandbox_image() and two identical copies of SUPERVISOR_IMAGE_BINARY_PATH. Implement ProviderPlugin for ProviderDiscoverySpec to remove the repeated three-method delegation boilerplate from eight simple provider modules. Each now declares only a SPEC constant; the blanket impl handles id(), discover_existing(), and credential_env_vars(). |
||
|
Piotr Mlocek
|
3b53184971 | test(persistence): make CAS conflict test deterministic (#1464) | ||
|
Eric Curtin
|
10af3e609f |
refactor: deduplicate shared test helpers (#1399)
Extract four categories of copy-pasted test code into single canonical locations: - POLICY_OBJECT_TYPE / DRAFT_CHUNK_OBJECT_TYPE constants that were defined identically in both persistence/sqlite.rs and persistence/postgres.rs are now owned by persistence/mod.rs. - test_server_state() was duplicated across grpc/provider.rs, grpc/policy.rs, and grpc/sandbox.rs. A shared grpc::test_support module in grpc/mod.rs now owns the single implementation; each submodule imports it. - The TestOpenShell stub (full OpenShell trait impl), install_rustls_provider(), PkiBundle, generate_pki(), and start_test_server() were copied across up to five openshell-server integration test files. A new tests/common/mod.rs module owns them; each test file uses mod common. - StubResponse, unique_socket_path(), and spawn_podman_stub() were duplicated between openshell-driver-podman/src/driver.rs and openshell-driver-podman/src/grpc.rs. A new src/test_utils.rs (cfg(test)-gated) owns the shared helpers. |
||
|
Seth Jennings
|
c527341d8a |
feat(k8s): make default workspace PVC storage size configurable (#1436)
The Kubernetes driver hardcoded the workspace PVC size to 2Gi. Add a workspace_default_storage_size field to KubernetesComputeConfig so operators can tune it via TOML config, Helm values, or the OPENSHELL_K8S_WORKSPACE_DEFAULT_STORAGE_SIZE environment variable. |
||
|
John T. Myers
|
d255cdd9c9 |
feat(providers): add credential refresh foundation (#1349)
* feat(providers): add credential refresh foundation * feat(providers): mint oauth refresh token credentials * fix(providers): allow delegated refresh bootstrap * fix(providers): guard refresh credential modes * fix(providers): refine refresh lifecycle UX * fix(cli): restore provider test gateway response import * fix(server): clean auth endpoint test qualifications * fix(providers): tighten refresh authorization and collisions * chore(providers): trim bundled v2 profiles * fix(providers): resolve refresh rebase fallout * fix(cli): accept rfc3339 credential expiry * test(providers): update inferred claude provider type * test(providers): avoid removed outlook default profile * test(providers): isolate attach limit fixtures |
||
|
Adam Miller
|
436c59a2a1 |
fix(rpm): restore 0.0.0.0 bind address for Podman via default gateway.toml (#1438)
* fix(rpm): restore 0.0.0.0 bind address for Podman via default gateway.toml The gateway binary default changed to 127.0.0.1 in recent commits ( |
||
|
Drew Newberry
|
f257ed0193 |
refactor(packaging): rely on gateway runtime defaults (#1415)
* fix(packaging): use gateway TOML config in packages * refactor(packaging): rely on gateway runtime defaults |
||
|
Derek Carr
|
7f16d60ef1 |
feat(persistence): implement optimistic concurrency control with CAS (#1292)
Add resource_version-based optimistic concurrency control to the persistence layer. Every write now requires an explicit WriteCondition (MustCreate or MatchResourceVersion), enforced at compile time by gating unconditional put/put_message behind #[cfg(test)]. - Add WriteCondition enum and put_if for conditional writes - Add update_message_cas for atomic read-modify-write operations - Add list_messages/list_messages_with_selector helpers that hydrate resource_version from authoritative DB rows - Convert all production write paths to CAS-aware methods - Gate put/put_message behind #[cfg(test)] to prevent non-CAS writes - Use structured PersistenceError::UniqueViolation matching instead of string matching for duplicate detection - Hydrate resource_version from WriteResult directly on creates, eliminating unnecessary read-after-write round trips Signed-off-by: Derek Carr <decarr@redhat.com> |
||
|
Adam MillerandTaylor Mutch
|
71209e6ac6 |
feat(rpm): replace init-pki.sh with openshell-gateway generate-certs (#1426)
* feat(rpm): replace init-pki.sh with openshell-gateway generate-certs
Cuts the RPM gateway over to the unified Rust certgen path. The systemd
user unit's first ExecStartPre now invokes:
/usr/bin/openshell-gateway generate-certs --output-dir %S/openshell/tls
producing the same six-PEM layout init-pki.sh built (ca.{crt,key},
server/tls.{crt,key}, client/tls.{crt,key}) and the same CLI mTLS copy
under $XDG_CONFIG_HOME/openshell/gateways/openshell/mtls/. None of the
OPENSHELL_TLS_* / OPENSHELL_PODMAN_TLS_* paths in the unit change.
Adds host.containers.internal to the gateway's built-in SAN list so
podman containers reaching their host validate cleanly with no
per-deployment --server-san flag. Docker (host.docker.internal) and
Kubernetes (cluster.local DNS) were already covered.
Drops 197 lines of openssl shell, the install/file lines for the script
itself, and updates the docs (man page, RPM CONFIGURATION.md, env-file
generator comment) to point at the new entrypoint. The %S state dir,
unit security hardening, and consumer paths are untouched.
* docs(certgen): remove stale init-pki.sh references in comments
---------
Co-authored-by: Taylor Mutch <taylormutch@gmail.com>
|
||
|
Drew Newberry
|
910d3f09f2 |
feat(vm): boot sandboxes from ext4 root disks (#1263)
* feat(vm): boot sandboxes from ext4 root disks * feat(vm): boot sandboxes with writable overlay disk * fix(sandbox): improve sandbox provisioning behavior * fix(vm): repair sandbox image provisioning * fix(vm): clean up rebase fallout * perf(vm): cache empty overlay images * perf(vm): skip repeated sandbox ownership repair * fix(vm): address lifecycle review issues * fix(vm): repair gateway vm startup * wip |
||
|
Taylor MutchandDrew Newberry
|
b61a98dbad |
feat(gateway): add TOML configuration file (RFC 0003) (#1317)
* feat(gateway): add TOML configuration file (RFC 0003)
Introduces an opt-in --config / OPENSHELL_GATEWAY_CONFIG flag that loads a
TOML file with gateway-wide settings and per-driver tables. Source
precedence is CLI > env > file > built-in default, implemented via clap's
ValueSource so existing flags and env vars keep their priority.
Driver crates (kubernetes, docker, podman, vm) now derive Deserialize on
their config structs. SupervisorSideloadMethod gains Deserialize with
kebab-case rename. A per-driver inheritance allowlist on the loader side
overlays [openshell.gateway] shared defaults (default_image,
supervisor_image, image_pull_policy, guest_tls_*, ssh_handshake_skew_secs,
client_tls_secret_name, host_gateway_ip, enable_user_namespaces) onto
each [openshell.drivers.<name>] table before deserialization.
The Helm chart renders a new gateway-config ConfigMap and mounts it at
/etc/openshell/gateway.toml. The migrated OPENSHELL_* env entries are
dropped from the StatefulSet — only the Secret-backed
OPENSHELL_SSH_HANDSHAKE_SECRET remains. database_url stays on --db-url.
Adds examples/gateway/gateway.example.toml and updates architecture/gateway.md
with the source precedence and inheritance rules.
* docs(gateway): drop ssh_handshake_skew_secs and ssh_handshake_secret from examples
Both fields are scheduled for removal. Remove the example values and the
env-only note so the gateway.toml example and the architecture doc stop
recommending settings that will not exist much longer.
* docs(rfc): correct OPENSHELL_CONFIG to OPENSHELL_GATEWAY_CONFIG in RFC 0003
* docs(gateway): add per-driver TOML example configurations
Adds focused single-driver examples next to the comprehensive
gateway.example.toml: kubernetes, docker, podman, and microvm. Each one
demonstrates the realistic settings for that driver plus how shared
[openshell.gateway] defaults inherit into the driver table.
A new unit test (`checked_in_examples_parse`) loads every example through
the config_file loader so schema drift fails CI rather than silently
shipping a broken example.
* refactor(gateway): drop image_pull_policy from shared inheritance
Kubernetes and Podman use mutually-incompatible vocabularies for the same
TOML key:
- Kubernetes: `Always | IfNotPresent | Never` (free-form string passed
verbatim to the K8s API).
- Podman: `always | missing | never | newer` (strict lowercase enum
deserialised into `ImagePullPolicy`).
No value means the same thing in both drivers. Sharing the key at
`[openshell.gateway]` scope and inheriting it into every active driver's
table meant any value safe for one driver was either wrong or silently
dropped for the other (`IfNotPresent` → `ImagePullPolicy::Missing` after
`.unwrap_or_default()`). Operators run one driver per gateway, so the
"shared default" never pays for itself.
Make `image_pull_policy` driver-local:
- Remove the field from `GatewayFileSection` and from
`inheritable_keys()` for both Kubernetes and Podman.
- Drop the file→`RunArgs` merge for the gateway-scope key.
- Stop unconditionally clobbering the driver value with
`config.sandbox_image_pull_policy` in the runtime wiring — only apply
the CLI/env override when it was set (and, for Podman, only when it
parses into the lowercase enum).
- Move the key under `[openshell.drivers.kubernetes]` and
`[openshell.drivers.podman]` in every example, the RFC, the
architecture doc, and the Helm-rendered gateway ConfigMap.
The supervisor pull policy follows the same shape: it is K8s-only and
moves into `[openshell.drivers.kubernetes]` alongside `image_pull_policy`
in the Helm template.
* fix(gateway): address review feedback on TOML configuration
Resolves the P1 and P2 issues raised in PR #1317:
- Helm gateway ConfigMap moves `grpc_endpoint` under
`[openshell.drivers.kubernetes]` so the default install no longer fails
the gateway's `deny_unknown_fields` schema check.
- `kubernetes_config_from_file` and `podman_config_from_file` only let
the gateway-wide CLI/env `grpc_endpoint` overwrite the driver-table
value when it was actually supplied, preserving file-only configs.
- Kubernetes driver default `image_pull_policy` is now empty (was Podman
vocabulary "missing"), so default deployments let the Kubernetes API
apply its own policy instead of being rejected.
- New `disable_tls` gateway field plumbs `.Values.server.disableTls`
through the TOML ConfigMap instead of relying on env vars dropped from
the StatefulSet.
- StatefulSet pod template now carries a `checksum/gateway-config`
annotation so `helm upgrade` rolls pods when the ConfigMap changes.
- Auxiliary listener resolution preserves the full `SocketAddr` from
`health_bind_address` / `metrics_bind_address`, so a loopback-pinned
health port is not silently relocated onto the public bind address.
- `ssh_session_ttl_secs` from the file is now applied to `Config` (it
was previously accepted by the loader but never read).
New regression coverage: cli-level merge tests for the new fields plus
helm-unittest assertions for the ConfigMap shape, checksum annotation,
and `disable_tls` rendering.
* docs(gateway): consolidate gateway TOML examples into docs reference
Replaces the per-driver example files under examples/gateway/ with a
single published reference page at docs/reference/gateway-config.mdx
covering source precedence, layout, the full example, and the four
per-driver examples (Kubernetes, Docker, Podman, microVM). Drew flagged
during PR #1317 review that the examples belong with the user-facing
docs rather than in a sibling examples/ directory.
The cross-references in architecture/gateway.md and RFC 0003 are updated
to point at the new docs page; the round-trip test in config_file.rs is
removed (schema coverage stays on the inline parses_full_example test
and per-field merge tests — doc-snippet drift belongs in a separate
docs-lint, not in a cross-tree Rust unit test).
* refactor(core): move DEFAULT_K8S_NAMESPACE into K8s driver
The constant is Kubernetes-specific (used only by KubernetesComputeConfig's
Default impl) and does not belong in openshell-core. Relocate it to the
driver crate that owns the K8s vocabulary; openshell-core retains only
truly cross-cutting defaults.
* refactor(core): move Podman bridge default into Podman driver
DEFAULT_NETWORK_NAME is Podman vocabulary, consumed only by the Podman
driver. Also drops the unused DEFAULT_IMAGE_PULL_POLICY constant.
* docs(auth): scrub remaining SSH handshake secret references
Sweeps the trailing mentions left after the rebase: the gateway
config-file module doc, the Helm gateway-config ConfigMap header,
the gateway-config.mdx env-only note, and the RPM systemd unit
comment for init-gateway-env.sh.
* docs(gateway): clarify OPENSHELL_GRPC_ENDPOINT applies to all drivers
The previous comment implied the callback endpoint was Kubernetes-only,
but the value is propagated to every compute driver (Kubernetes, Docker,
Podman, VM) and must be reachable from wherever the sandbox runs.
* refactor(gateway): move driver options into config (#1394)
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(e2e): regenerate gateway config via TOML for docker + podman harnesses
The gateway CLI flags moved into TOML config tables in 560550d2 (#1394),
which made every existing e2e/with-{docker,podman}-gateway.sh invocation
fail with "unexpected argument '--sandbox-namespace'" (and a long tail
of similar driver-specific options) before the gateway could even bind.
Replace the obsolete CLI flags with a synthesized
`[openshell.drivers.<driver>]` table written to `${STATE_DIR}/gateway.toml`
and passed via the new `--config` flag. Only the gateway-wide flags
that survived 560550d2 (bind-address, port, drivers, db-url, tls-*,
disable-tls, log-level, health-port) stay on the command line.
Both scripts get a small `toml_string` helper to properly TOML-quote
the values (the previous `%q` printf format produced bash-escape, not
TOML-escape). The Docker harness also corrects two field names that
diverged from the driver schema: `docker_network_name` →
`network_name`, and the supervisor binary/image plumbing now reads
through to `supervisor_bin` / `supervisor_image` in the same table.
The Podman harness drops `--ssh-gateway-port` (deleted in 560550d2 —
gRPC + SSH are multiplexed on the same port now) and substitutes
`network_name` + `gateway_port` for the obsolete `--sandbox-namespace`
(which the Podman driver never had as a typed field).
* fix(core): swap bind-only 0.0.0.0 SSH gateway host for cluster URL host
CLI's resolve_ssh_gateway treated 0.0.0.0 as a loopback "keep as-is"
when the cluster URL was also loopback, so the SSH proxy connected to
0.0.0.0:port. The unspecified address is never a valid connect target
and is not present in any TLS cert SAN, which produced BadCertificate
TLS handshake failures during `openshell sandbox create -- ...` in
docker/podman e2e (e.g. bypass_detection).
Resolution: when the server returns 0.0.0.0 or :: as the gateway host
and both endpoints are loopback, fall back to the cluster URL's host
(which the CLI is already using to reach the gateway, so it must
resolve and match the cert).
* fix(e2e): repair podman harness on macOS
Podman 5.x with the applehv/libkrun provider no longer creates the legacy
~/.local/share/containers/podman/machine/podman.sock symlink, and
`podman system service` is a Linux-only subcommand — the macOS client
delegates the API service to the VM. Both assumptions in the harness
were stale, so the script tried to start a temporary service that podman
rejected with "unknown flag: --time".
- Discover the macOS socket via `podman machine inspect` instead of the
hardcoded path.
- On Darwin, fail fast with a "start podman machine" message rather than
attempting the Linux-only `podman system service` fallback.
- Write socket_path into [openshell.drivers.podman] so the in-process
driver picks up the discovered socket; the driver reads TOML only
after the config refactor (560550d2), so OPENSHELL_PODMAN_SOCKET alone
was no longer enough.
* fix(server): use clone_from for TLS client CA assignment
clippy 1.95.0 rejects assigning the result of `Clone::clone()` to an
existing variable under `-D warnings` (`assigning_clones`). Switch to
`clone_from(&...)` to satisfy the lint and avoid the redundant
allocation.
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
|
||
|
Florent BENOIT
|
442b0b6b92 |
feat(exec): add bidirectional streaming for interactive TTY sessions (#1331)
* feat(exec): add bidirectional streaming for interactive TTY sessions The existing ExecSandbox RPC sends stdin upfront in the request body, making interactive programs (bash, top, vim) unusable with --tty. This adds an ExecSandboxInteractive bidirectional streaming RPC that forwards live keystrokes, terminal resize events, and EOF to the sandbox. - Add ExecSandboxInteractive RPC, ExecSandboxInput and ExecSandboxWindowResize messages to the proto definition - Implement server handler with relay bridging, PTY allocation via russh channel.split(), and timeout support - Implement CLI client with raw mode, spawn_blocking stdin reader, SIGWINCH resize forwarding, and process::exit for clean shutdown - Route to interactive path only when --tty is explicitly passed Signed-off-by: Florent Benoit <fbenoit@redhat.com> * fix(exec): address interactive TTY streaming review feedback Server-side (sandbox.rs): - Close SSH channel after EOF so PTY-backed programs (bash, vim, top) terminate on client disconnect instead of leaking the session - Break output loop when tx.send() fails (client gone) to tear down the streaming pipeline promptly - Default zero cols/rows to 80x24 before request_pty, matching the proto "0 = use default" contract and preventing 1x1 terminals for non-CLI clients Client-side (run.rs): - Replace spawn_blocking stdin reader with a detached std::thread so the tokio runtime shutdown does not hang on a thread blocked in stdin.read(). This removes the need for std::process::exit(), allowing save_last_sandbox() in the caller to execute normally - Switch from futures::channel::mpsc with try_send() to tokio::sync::mpsc with blocking_send() for proper backpressure — try_send silently stopped forwarding all input when the channel filled during paste or slow network - Add TaskGuard (abort-on-drop) for the resize task to ensure cleanup on all return paths including early errors --------- Signed-off-by: Florent Benoit <fbenoit@redhat.com> |
||
|
Adam Miller
|
63bdcd1379 |
fix(sandbox): exempt host gateway from SSRF block for rootless Podman (#1279)
* fix(sandbox): exempt host gateway from SSRF block for rootless Podman pasta assigns a link-local IP (169.254.x.x) as the host gateway in rootless Podman containers, which was always blocked by SSRF guards. Read the trusted gateway IP from /etc/hosts at proxy startup (injected by the driver via host.openshell.internal / host.containers.internal aliases) and exempt only that specific IP from the link-local block, while keeping 169.254.169.254 (cloud metadata) hard-blocked regardless. * fix(net): add is_link_local_ip() helper and harden trusted-gateway SSRF Extract a public is_link_local_ip(IpAddr) helper in openshell-core::net covering IPv4 169.254.0.0/16, IPv6 fe80::/10, and IPv4-mapped variants. Refactor the inline IPv6 fe80::/10 bitmask in is_always_blocked_ip() and is_internal_ip() to use the new helper, eliminating two copies of the same pattern. Harden the trusted-gateway SSRF exemption in proxy.rs: both detect_trusted_host_gateway() and resolve_and_check_trusted_gateway() now reject always-blocked non-link-local IPs (loopback, unspecified) with a warning. The exemption was intended only for link-local addresses used by rootless Podman with pasta; a /etc/hosts entry mapping host.openshell.internal to 127.0.0.1 or 0.0.0.0 previously bypassed the loopback/unspecified SSRF invariant. Add 169.254. prefix to the generate_security_notes() host checks in mechanistic_mapper and policy server, closing a gap where link-local literal hosts were not flagged as internal/private in advisory notes. * fix(sandbox): warn on ambiguous /etc/hosts gateway and add real function tests When detect_trusted_host_gateway() finds more than one distinct IP for host.openshell.internal, emit a structured warning so operators can diagnose unexpected /etc/hosts state. The first entry still wins (and any runtime mismatch is caught by resolve_and_check_trusted_gateway), but the ambiguity is now observable. Replace inline guard-logic tests with real calls to resolve_and_check_trusted_gateway() using IP literals, which bypass DNS and give deterministic results. This exercises the actual function wiring rather than mirroring the predicate logic manually. Add parse_hosts_file_for_host tests covering: single entry, same-IP deduplication across lines, multiple distinct IPs (documents first-wins ordering), multi-alias lines, missing alias, and comment handling. * fix(sandbox): narrow trusted-gateway exemption to link-local IPs only Two security fixes addressing PR review feedback: 1. Replace the compound is_always_blocked_ip && !is_link_local_ip guard in detect_trusted_host_gateway() with a plain !is_link_local_ip check. Private RFC 1918 addresses (e.g. Docker bridge 172.17.0.1, K8s node 192.168.x.x) are not always-blocked, so they slipped through the old guard and received the SSRF exemption, bypassing resolve_and_reject_internal(). The exemption is only valid for the rootless Podman/pasta link-local case. 2. Normalize IPv4-mapped IPv6 in is_cloud_metadata_ip() so that ::ffff:169.254.169.254 is correctly identified as cloud metadata. is_link_local_ip() already recognises this form as link-local; without the normalization it would pass the metadata guard and receive the trusted-gateway exemption, violating the 'metadata is never exempted' invariant. Apply the same !is_link_local_ip tightening to resolve_and_check_trusted_gateway() as defense-in-depth, and update/add tests to cover both guards including IPv4-mapped metadata, private IP rejection, and the Docker bridge scenario. |
||
|
Seth Jennings
|
c94cddbfb8 |
feat(server): separate HTTPS from mTLS authentication (#1351)
Make --tls-client-ca optional and make client certificates always optional when a CA is configured. This decouples HTTPS encryption from mTLS authentication, allowing mTLS and OIDC bearer tokens to coexist as parallel authentication mechanisms. When --tls-client-ca is provided, client certificates are validated against the CA when presented but never required. Clients may connect with or without a certificate — authentication is handled at the application layer (e.g. OIDC). Two TLS modes are now supported: - HTTPS with optional mTLS (--tls-client-ca provided) - HTTPS-only (--tls-client-ca omitted) The --disable-gateway-auth flag is preserved for backward compatibility but is now a no-op. The allow_unauthenticated field has been removed from TlsConfig. The Helm chart conditionally includes the client-ca volume and env var based on whether clientCaSecretName is configured. |
||
|
Taylor Mutch
|
7a0c444445 |
refactor!(auth): drop SSH handshake secret (#1274)
* refactor!(auth): drop SSH handshake secret in favor of mTLS The OPENSHELL_SSH_HANDSHAKE_SECRET / x-sandbox-secret mechanism was misnamed: it does not authenticate SSH (which flows over the RelayStream gRPC RPC and is gated by mTLS plus supervisor Unix-socket permissions). It only gated a small set of sandbox-to-gateway control-plane RPCs, and production deployments already enforce mTLS on that channel — so the shared secret was redundant. Replace the secret check with an mTLS-presence marker. Sandbox-class methods (ReportPolicyStatus, PushSandboxLogs, GetSandboxProviderEnvironment, SubmitPolicyAnalysis, GetSandboxConfig, GetInferenceBundle) accept callers without a Bearer token; the gRPC mTLS handshake is the trust boundary. Dual-auth methods treat Bearer-present as full-scope CLI access and Bearer-absent as sandbox-restricted scope via validate_sandbox_caller_update. Drops the secret from all drivers (K8s, Podman, VM), the sandbox gRPC interceptor, the Helm chart (values + pre-install hook + StatefulSet env), the RPM bootstrap script, the man pages, and the debug-openshell-cluster skill. Also removes the never-read ssh_handshake_skew_secs flag and config field. BREAKING CHANGE: --ssh-handshake-secret / OPENSHELL_SSH_HANDSHAKE_SECRET and --ssh-handshake-skew-secs / OPENSHELL_SSH_HANDSHAKE_SKEW_SECS are removed from the gateway, sandbox, and all driver binaries. The openshell-ssh-handshake K8s Secret is no longer managed by the chart; operators may delete the orphan. Deployments using --disable-gateway-auth must enforce caller authentication at the fronting proxy, since the gateway no longer validates a per-request secret on sandbox-class methods. Refs OS-174. * docs(auth): scrub residual SSH handshake secret references Sweep across docs, e2e scripts, the Podman driver README/NETWORKING notes, the RPM/Helm/setup guides, the gateway man page, and RFC 0003 to remove instructions and examples that still referenced OPENSHELL_SSH_HANDSHAKE_SECRET / --ssh-handshake-secret / ssh_handshake_skew_secs. The mechanism is gone; nothing should still suggest setting it. Negative-assertion regression tests are kept so the env var cannot silently be re-introduced. * test(drivers): drop SSH handshake secret negative-assertion tests The supporting code, env vars, CLI flags, and config plumbing are gone — these tests asserted absence of strings that no longer have any path to being set. Remove the guards from the Docker, Podman, Kubernetes, and VM driver test modules. |
||
|
Russell Bryant
|
94025d834a |
fix(server): downgrade expected connection teardown errors to debug (#1369)
h2 connection errors during sandbox deletion are expected — the supervisor gRPC stream is torn down when the sandbox is removed. Downgrade these from ERROR/WARN to DEBUG so they don't pollute gateway logs during normal operations. Signed-off-by: Russell Bryant <russell.bryant@gmail.com> |
||
|
Eric Curtin
|
6deb1f005a |
refactor(core): eliminate duplicate utilities across crates (#1381)
Consolidate shared constants and helpers into openshell-core to avoid duplicate definitions scattered across crates. Adds OPENSHELL_SANDBOX to the sandbox_env module so all drivers and the sandbox supervisor reference it via a typed constant rather than string literals, matching the existing pattern for SANDBOX_ID, ENDPOINT, and the other env vars. |
||
|
Drew Newberry
|
0471c6d2ab | fix(gateway): keep vm driver opt-in (#1375) | ||
|
Alexander Watson
|
ea2fddbe2d |
feat(policy): agent-driven policy management — the agent half (#1323)
* feat(policy): plumb chunk_ids and rejection_reason through proposal pipeline Prereq plumbing for the agent revise-and-resubmit loop. Two narrow additive proto changes unblock the upcoming /wait endpoint (#1092), prover validation badge (#1097), and reject --guidance surfaces (#1098). - SubmitPolicyAnalysisResponse: add accepted_chunk_ids so the in-sandbox agent gets handles to watch its proposals. Surfaced through the typed grpc_client wrapper and policy.local's POST /v1/proposals 202 body. Closes #1094. - PolicyChunk + StoredDraftChunk + DraftChunkPayload: add validation_result (gateway prover verdict, populated by #1097) and rejection_reason (operator free-form text). Both plain strings; no enums, no parsing on the read path. Closes #1096. - RejectDraftChunk now persists the existing reason field into the chunk's rejection_reason so it round-trips back to the agent via GetDraftPolicy. UndoDraftChunk clears it on the way back to pending so consumers cannot read a stale guidance string from a prior reject -> re-approve -> undo cycle. Whole surface stays gated behind agent_policy_proposals_enabled. Two focused tests cover the round-trip and the undo-clears guarantee. Signed-off-by: Alexander Watson <zredlined@gmail.com> * feat(sandbox): add /v1/proposals/{id} and /wait long-poll to policy.local The agent feedback channel back from policy.local. Two new routes let the in-sandbox agent learn its proposal's outcome on a single blocking HTTP call — zero LLM tokens during the wait. - GET /v1/proposals/{chunk_id} returns the chunk's current state in one gateway call. - GET /v1/proposals/{chunk_id}/wait?timeout=<s> blocks until the chunk transitions out of pending. Default 60s, clamped [1, 300]. Agent re-issues on timeout to extend. Response carries the chunk's status plus the two feedback fields shipped in the prereq commit: rejection_reason (free-form reviewer text) and validation_result (gateway prover verdict, empty until #1097). On timeout: same shape with timed_out: true so the agent can disambiguate without parsing. Wait handler short-polls GetDraftPolicy every 1s inside the request with a tokio::time::Instant deadline. One gateway connection is opened per request and reused across all polls, so a 60s wait does one TLS handshake instead of sixty. A future commit can swap the loop body for a tokio::sync::broadcast driven by a watcher task — the agent-visible contract (URL, query, response shape) is independent of the polling implementation. All routes stay behind agent_policy_proposals_enabled. Closes #1092. Signed-off-by: Alexander Watson <zredlined@gmail.com> * docs(sandbox): teach policy_advisor skill the wait + redraft loop The agent-facing instructions for the feedback loop. The endpoints exist; this is the doc that makes them usable. policy_advisor.md gains: - API entries for GET /v1/proposals/{chunk_id} and /wait?timeout=<s>, including the field semantics (status, rejection_reason, validation_result, timed_out). - A note on the submit response's accepted_chunk_ids / rejection_reasons split so the agent handles partial acceptance. - Step 6 saves the chunk_ids and addresses any submit-time rejections before waiting. - Step 7 walks the four wait outcomes: approved (retry, with the honest "may still fail" caveat), rejected (read rejection_reason AND validation_result; address whichever has content), still-pending with timed_out (re-call), non-2xx (surface, do not retry). skills.rs gains two assertions on the skill content so a future edit cannot drop the wait endpoint or the rejection_reason directive silently. Closes #1095. Signed-off-by: Alexander Watson <zredlined@gmail.com> * test(policy-advisor): add end-to-end smoke for the agent feedback loop A focused smoke that exercises the new policy.local /wait endpoint on a live gateway + sandbox, separate from the existing no-LLM regression harness (which still drives the OLD retry-with-bash-loop recovery pattern). Two flows: - Flow A — approve-and-retry: agent submits, /wait blocks, host runs `openshell rule approve`, /wait returns status=approved. Confirms the happy path round-trip latency. - Flow B — reject-with-guidance: agent submits, /wait blocks, host runs `openshell rule reject --reason "..."`, /wait returns status=rejected with the exact reviewer text in rejection_reason. Confirms the free-form guidance contract round-trips through the agent feedback channel. No GitHub credentials needed — proposals are synthetic and never trigger outbound traffic. Both flows expect agent_policy_proposals_enabled=true and a running gateway. Adds three cases to sandbox-runner.sh: submit-test-proposal (no GH deps), proposal-status, proposal-wait. The existing put-file and submit-proposal cases used by test.sh are untouched. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(policy-advisor): surface real CLI errors from wait-smoke preflight The preflight piped openshell's stderr to /dev/null and relied on jq to default the missing setting key to "<unset>", but under `set -euo pipefail` a non-zero exit from openshell makes the whole pipeline fail and the command substitution exits the script silently before the intended fail() message can print. Capture stderr explicitly, check the CLI exit code, and surface the real error plus the expected fix (port-forward + gateway add + select) when the CLI cannot reach the gateway. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(policy-advisor): pass --json to settings get in wait-smoke preflight `openshell settings get --global` defaults to a human-readable table; jq cannot parse it and the preflight died with a numeric-literal error. Pass --json so jq gets actual JSON. Also touched up the suggested recovery commands in the preflight error to match the real CLI shape (`gateway add <endpoint> --name <name>` and the env-var override warning). Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(policy): dedup draft chunks only in mechanistic mode; return effective id The smoke harness for the agent feedback loop caught a real bug in the gateway: SubmitPolicyAnalysis's response carried a chunk_id that was never persisted whenever the SQL ON CONFLICT path fired. Two failure modes, both load-bearing: - Agent-authored proposals targeting the same host/port/binary (e.g. the redraft-after-rejection loop) silently folded into one row and any RejectDraftChunk by the new chunk_id failed with "chunk not found." Latent since #1151, surfaced by #1094 returning chunk_ids. - Mechanistic mode had the same class of bug — the dedup fold-in is the intended behavior there, but the response still advertised the newly-generated UUID instead of the existing row's id. Less visible because no current caller reads mechanistic chunk_ids back, but the proto contract was violated either way. Fix in three parts: - put_draft_chunk now takes Option<&str> dedup_key explicitly and returns the effective row id (via RETURNING). None binds NULL to the dedup_key column, which bypasses the partial-index ON CONFLICT path entirely. Caller-decides semantics replace store-side magic. - handle_submit_policy_analysis picks dedup_key per chunk using an allowlist (only "mechanistic" dedups) and pushes the returned effective_id to accepted_chunk_ids. New modes default to no-dedup so a misconfigured caller cannot silently lose proposals. - The two-copy draft_chunk_dedup_key helper consolidated to one observation_dedup_key in policy_store.rs with a doc comment. Tests: - agent_authored_submits_for_same_endpoint_do_not_dedup pins the redraft-loop contract: two intentional submissions with the same host/port/binary get distinct chunk_ids, both findable via GetDraftPolicy, both rejectable by id. - mechanistic_submits_for_same_endpoint_dedup_into_one_chunk locks in the observation-mode dedup AND asserts both submits return the same effective_id — would have caught the deeper bug. Proto: SubmitPolicyAnalysisRequest.analysis_mode doc updated to describe the actual semantics (mechanistic dedups, agent_authored and unknown modes do not). Signed-off-by: Alexander Watson <zredlined@gmail.com> * docs(examples): retarget policy-management demo at the /wait endpoint The narrated demo (examples/agent-driven-policy-management) has been the public face of this feature since #1151. Its agent prompt told Codex to retry the original PUT every few seconds for up to 120 seconds — a polling workaround for the missing /wait endpoint that this branch shipped. Update the demo to exercise /wait so the canonical reading of the feature reflects the actual UX win. - agent-task.md: step 4 is now "call /wait, branch on status" with the three outcomes spelled out (approved → retry once; rejected → read rejection_reason and revise or stop; pending+timed_out → re-issue /wait once, do NOT busy-loop or shorten the timeout). Also makes explicit that the demo submits one rule per proposal so accepted_chunk_ids[0] is the safe single id to wait on. - demo.sh: header docstring rewritten as a six-step loop that mirrors the README. narrate_sandbox_workflow drops its parallel numbering and uses bullets (the runtime narration is the agent's sub-actions, not a separate decomposition of the loop). Approve step header and success message now reference /wait waking the agent, not "policy hot-reload retry." - README.md: top-of-file flow expanded from 5 to 6 steps to include the /wait call and chunk_id capture; "Going further" section now describes both regression scripts and the boundary between them (real-GitHub retry vs. synthetic /wait wire test). Slow-path qualifier corrected from "image pull on first run" to "sandbox cold-start (SSH bring-up plus Codex install)". - wait-smoke.sh header rewritten to make it unambiguous this is a regression, NOT a tutorial, with explicit prereq commands instead of prereq descriptions, and a pointer at demo.sh for the narrated story. No code paths change; this is the readability pass. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(examples): pass --yes on demo.sh's global setting writes Global setting updates require explicit confirmation in non-interactive mode; demo.sh's enable_agent_proposals and the cleanup restore path were missing --yes and hard-failed the preflight. Pre-existing issue that surfaced now that more of the demo runs through this path. No other behavior change. Signed-off-by: Alexander Watson <zredlined@gmail.com> * feat(sandbox): emit OCSF audit events for policy proposal lifecycle The demo's policy decision trace previously showed only the proxy enforcement story (HTTP:PUT DENIED, CONFIG:LOADED, HTTP:PUT ALLOWED). It was silent about who proposed what or who decided what — the audit-trail receipts for the agent feedback loop were missing. policy.local now emits sandbox-side OCSF events at the observation moments, into the same stream as the existing CONFIG:LOADED: - CONFIG:PROPOSED on submit_proposal acceptance. Per accepted chunk: the message names the chunk_id, target endpoint, L7 method/path, and binary so the trace correlates against the inbox card via chunk_id. - CONFIG:APPROVED on /wait observation of approved status. - CONFIG:REJECTED on /wait observation of rejected status. Carries the reviewer's free-form rejection_reason in the message AND as an unmapped field, both sanitized (control chars stripped, capped at 200 chars with an ellipsis marker). The agent still reads the raw text via GET /v1/proposals/{id}; sanitization is audit-side only, per AGENTS.md's no-secrets-in-OCSF rule. The submit path defends the audit_summaries / accepted_chunk_ids index pairing against a future gateway change that compresses past rejected chunks (the proto doesn't promise 1:1 ordering with the request). Today client-side validation makes the lengths always match; if they don't, the pairing falls back to a generic per-id event rather than mis-attribute. The wait handler's emit site fires once per terminal-status observation. Multiple concurrent waiters on the same chunk would each emit one event; acceptable for single-waiter-per-chunk demos and the right place to dedup is the SIEM. demo.sh's trace filter now surfaces the four CONFIG: events alongside HTTP:PUT, so the trace at the end of every run tells the full story from deny to allow via propose -> approve. wait-smoke.sh's prereq notes recommend redirecting kubectl port-forward output so its "Handling connection for 8090" lines don't bleed into demo narration. Three new unit tests on the sandbox-side helpers — summary builder happy path, fallback, and the rejection_reason sanitizer. Signed-off-by: Alexander Watson <zredlined@gmail.com> * feat(policy): /wait awaits local policy reload; demo auto-approves redrafts Three things in one commit, all surfaced by running the demo end-to-end against a real gateway and finding the agent had to draft a broader second proposal. 1. /wait race fix. Previously /wait returned `approved` the moment it observed the gateway's chunk status flip, but the local supervisor reloads policy on its own poll cycle (~10s in practice). The agent's retry would race the reload and hit the still-old policy, getting denied. Codex then drafted a broader rule and re-submitted — sound agent behavior, but not what /wait should provoke. Now /wait captures the local policy version at start, and after observed-approved waits for the supervisor to load a strictly-newer version before returning. Bounded by the caller's deadline; best-effort return if the deadline elapses without the version bumping. Two new unit tests pin the happy path and the deadline-clamped fallback. 2. demo.sh auto-approve loop. Replaces approve_when_pending + wait_for_agent with one approve_pending_until_agent_exits function that keeps watching for pending chunks and approving them until the agent process exits (or the configured timeout). Defense in depth against future redraft scenarios for any reason; today (post-fix #1) the agent should only submit one proposal per task, but we don't want to hang silently if it does submit more. 3. UX. Step headers now carry "[t+1.2s]" relative timestamps so reading the run output makes latency visible (the demo's whole point is the wait is cheap — surface that). A spin_wait helper renders an ASCII spinner during the watch loop so the demo never looks frozen on a TTY. Falls back to plain sleep on non-TTY contexts. Closes the race condition diagnosed from the trace timing where the gateway approved at t+0, sandbox observed at t+0.3s, but the supervisor didn't load v2 until t+9.4s — well after the agent had already retried and been denied. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(sandbox): /wait detects policy reload by content, not the schema version The previous attempt at the /wait-after-approve race fix compared `SandboxPolicy.version` between /wait start and the policy reload — but that field is the *schema* version (constant 1), not a revision counter. Every comparison was `current(1) > baseline(1) == false`, so the wait blocked until the agent's 300s timeout regardless of whether the supervisor had actually reloaded. The demo SSH connection then timed out around the 240s mark. Diagnosed from a live run's OCSF trace: supervisor pulled v2 at +8.5s after approval (CONFIG:LOADED), but the sandbox-side CONFIG:APPROVED that my /wait emits didn't fire until +304s — exactly at the 300s deadline. Fix: compare the whole policy via prost's derived PartialEq. Any field change (network_policies map being the only one that actually mutates today) flips equality. A clone-per-200ms-tick on a few-KB proto is cheap inside the bounded wait window. Tests rewritten to match the new contract: the supervisor-reload fixture now keeps `version: 1` constant and changes `network_policies` contents, mirroring the exact failure mode from the live run. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(examples): redact tokens with python literal-string replace, not sed The sed-based redact_log in demo.sh broke when one of the auth tokens contained a character that conflicted with sed's pattern parser ("unterminated substitute pattern" on the Codex JWT). The whole log tail then blanks on failure, hiding the very failure context we're trying to surface. Switch to a python subprocess that takes the tokens via argv and does literal str.replace. No regex, no delimiter games, no truncation. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(sandbox): scope /wait reload check to the approved rule Reviewer (John Myers) flagged two failure modes in the prior whole-policy fingerprint approach used by policy.local /wait: - False sleep: when the supervisor reloads between two /wait calls (the skill tells the agent to re-issue on timed_out), the new call snapshots the already-updated policy as baseline and burns the full timeout waiting for a change that never comes. - False wakeup: any unrelated reload (other agent's approval, settings change) flips the diff, but the chunk's actual rule may not be loaded yet — the agent retries and hits policy_denied for no real signal. Replace the diff with rule-coverage. New public helper openshell_policy::policy_covers_rule reuses endpoints_overlap (so it matches add_rule's merge semantics, including the fold-into-existing-key case) plus an L7 allow check on method/path (so an existing endpoint that doesn't yet contain the proposed method doesn't signal coverage). Add policy_reloaded: true|false to the /wait response on approve, with a 500ms floor on the reload-wait phase so approvals arriving near the deadline still get a fair shot at reloaded=true. Update the policy_advisor skill to branch on it: reloaded=true → retry; reloaded=false → re-issue /wait once with timeout=30, then surface to user. Don't loop tightly. Tests: - 9 new unit tests in openshell-policy pinning coverage semantics (L4-only, L7 method gap, fold-into-existing-key, empty binaries). - 4 new tokio tests in policy_local mirroring John's exact scenarios. - wait-smoke.sh asserts policy_reloaded=true on Flow A. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(server): make GetDraftPolicy dual-auth so /wait works under OIDC policy.local calls GetDraftPolicy from inside the sandbox supervisor via the sandbox gRPC client, which authenticates with the shared x-sandbox-secret. GetDraftPolicy was listed only in the Bearer-auth scope table (config:read) and was not in SANDBOX_SECRET_METHODS or DUAL_AUTH_METHODS, so OIDC-enabled gateways rejected those calls and the /wait long-poll surfaced gateway_lookup_failed. Local/no-OIDC setups happened to work because the auth check is short-circuited. Add GetDraftPolicy to DUAL_AUTH_METHODS, matching the existing GetSandboxConfig pattern (called by both CLI reviewer surfaces with Bearer and the sandbox supervisor with x-sandbox-secret). Dual-auth short-circuits the scope check for sandbox-secret callers, so the config:read entry in authz.rs continues to gate Bearer-only flows. Mirror the openshell_get_sandbox_config_is_dual_auth assertion for GetDraftPolicy. Note: ssh_handshake_secret is server-wide, not per-sandbox, so a sandbox-secret caller can today name any sandbox in a SubmitPolicyAnalysis request — and now in a GetDraftPolicy request. The exposure is symmetric with the existing SANDBOX_SECRET_METHODS pattern. Filed as a follow-up: per-sandbox secret binding, tracked separately. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(ci): address rebased check failures Signed-off-by: Alexander Watson <zredlined@gmail.com> --------- Signed-off-by: Alexander Watson <zredlined@gmail.com> |