krishicks
a24340beba
When the gateway refuses a caller that has already authenticated, it now emits an OCSF API Activity event. The caller gets exactly the same error as before. The event records: - Who: the principal (user, mTLS operator, sandbox, or gateway peer). - What: the gRPC method, plus the sandbox or workspace that was refused when there is one. - Why: a reason such as cross_sandbox, missing_role or not_workspace_member. - Correlation: the caller's address and request ID, the same fields the authentication events carry. That covers the denials made centrally by the router and the shared guards, plus the checks inside handlers that have no shared guard: a sandbox naming another sandbox (in either sandbox-name resolver), a non-user caller refused by mutation admission, a sandbox sending a config update it isn't allowed to, and a non-admin granting the workspace admin role. Denials raised in the mutation admission worker, which runs in a spawned task, keep the request's context. When a sandbox credential is refused for calling a method it may not call, touching another sandbox, or acting across workspaces, the gateway also emits a High-severity Detection Finding. It carries the same correlation_uid as the denial event. A real supervisor never makes these requests, so a finding means the credential is probably being misused. Two cases are recorded without a finding: a method the gateway doesn't recognize (unknown_method), so a newer supervisor doesn't look like a compromised one, and a supervisor whose own sandbox was deleted mid-request, which is refused but isn't misuse. The shorthand line names the refused sandbox or workspace, for example sandbox:<id>, so operators reading console logs can see the target without JSONL or OTLP export. To support this, the OCSF crate's Actor can now hold a user instead of a process, and the API Activity builder gained the fields gateway events need. Supervisor inference events come out unchanged. Gateway events declare the container profile only when they concern a sandbox, as MXC findings do. The OCSF 1.8.0 resource_details schema is vendored so the refused sandbox or workspace goes in the standard resources field. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com>
OpenShell gateway server
The server owns control-plane RPCs, authentication, provider credential delivery,
and sandbox lifecycle orchestration. The openshell-gateway crate composes the
server with selected compute drivers.
Sandbox launch identity
A new start, including an automatic restart, commits a fresh runtime generation and an incremented authorization epoch before invoking the compute driver. The launch session ID is derived from the durable sandbox ID and generation. Retries reuse that generation and session, including after gateway restart. Credential renewal retains both; credentials for an older launch cannot authorize the new one.
Operator credential export
GetProviderCredentials exports selected profile-declared runtime environment
keys from one provider in an explicitly named workspace. It requires the distinct
Principal::Operator, established only from a verified TLS leaf certificate with
OU=operator when mtls_auth.operator_enabled is enabled. The TLS listener supplies
its certificate fingerprint separately from user identity; the ordinary
identity-only serving path cannot establish an operator.
Operators explicitly inherit user/platform-admin permissions, but not sandbox or peer identities. Descriptor authorization and a handler guard both protect the export RPC. Operator mutation admission uses a separate caller namespace from ordinary mTLS users with the same certificate subject. Credential responses never enter durable mutation replay or generic interceptor routing.
The export resolver validates the entire selection before minting, resolves only selected credential handles, and fences provider revisions, refresh epochs, and profile declarations before delivery. Driver expiration and provider expiration combine using the earlier timestamp. OAuth minting inputs, broker-only keys, and supervisor-bound dynamic grants are not exportable. A failed response can still have committed refresh effects. Missing refresh-owned stored outputs trigger refresh-on-read; missing static outputs and other credential-backend errors fail closed without falling back to inline values.
Refresh coordination
Explicit rotation, background refresh, and operator refresh-on-read share the same refresh coordination:
- A process-local weak keyed mutex coalesces waiters without retaining deleted providers indefinitely.
- A bounded admission queue and executor pool limit detached refresh work.
- PostgreSQL deployments also take a dedicated per-provider/per-credential advisory lock; SQLite uses the local mutex. Worker secret cleanup takes the same locks and reloads state before changing its resource version.
- A durable
refresh_in_progressmarker precedes upstream minting. Arefresh_committingmarker protects the interval between grant persistence and provider-handle persistence. Only a completed provider commit publishesrefreshedstate, with a unique completed-mint identity in metadata annotations. Waiters coalesce against that identity, not bookkeeping changes to the resource version. - Automatic callers recheck live recovery and retry deadlines under both locks. A failed mint blocks queued automatic exchanges without disabling scheduled worker retries or a separately requested manual rotation.
- Detached refresh tasks complete despite request cancellation, preserving replacement refresh tokens returned by the issuer.
- Existing generation/CAS fencing remains active for deletion and reconfiguration. An ambiguous rotating-token exchange is parked for recovery rather than automatically consuming the grant again.
A crash or lost database-lock session can leave a parked marker. Reconfigure or reauthorize the refresh grant after investigating its issuer state. Changing a marker directly is not a supported recovery workflow.
Focused verification
cargo test -p openshell-server --features test-support,prebuilt-z3
cargo clippy -p openshell-server --all-targets --features test-support,prebuilt-z3 -- -D warnings
The ignored postgres_refresh_locks_coordinate_independent_replica_sessions test
runs under mise run test:rust:postgres (see TESTING.md) and still accepts
OPENSHELL_REFRESH_TEST_DATABASE_URL pointing to a disposable PostgreSQL database.
It exercises independent sessions without the local mutex. Do not point this
fixture at an existing deployment database.