* feat(network): support additional destination CAs Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): harden additional CA propagation Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): narrow additional CA review scope Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): harden additional CA rotation Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * feat(network): complete additional CA delivery Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(docker): remove stale runtime fixture field Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): adapt additional CA series to upstream main Resolve rebase fallout against current upstream main after the series was rebased 214 commits forward: - follow upstream #3110: supervisor→gateway TLS is CA-only, so configure_gateway_tls and its isolation tests drop the client identity - follow upstream tls_secret_names reduction to [String; 1] in the Podman driver while keeping the network trust bundle delivery - port [openshell.supervisor.network] rendering into the openshell.gatewayConfigToml serializer in _toml.tpl - adapt tests to upstream struct/function changes (DriverStartupContext, SandboxRunConfig, IsolationSpecInput, supervisor_pod, VM bootstrap image) - drop conformance-artifact-prefix plumbing removed upstream Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(network): adopt upstream additional CA bundle surface Unify the additional destination CA series on the mechanism merged to main in #3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(ci): make gateway startup tests wrap-tolerant and refresh example lockfile The network trust startup tests asserted on raw substrings of gateway diagnostics, but the CLI renderer wraps output at a fixed width with continuation markers and hard-breaks long paths. Compare assertions on whitespace- and continuation-squashed text so they hold regardless of wrap position. Format the e2e workspace (separate cargo workspace, not covered by the root cargo fmt run) and refresh examples/isolation-backend-v8/Cargo.lock for the libc dependency added to the network trust crate. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(podman): reuse TLS CA secret during stopped-pair trust replacement The additional CA e2e never ran in CI before the first maintainer test dispatch, and its local container inspection was broken in two ways: it located the supervisor container through e2e network membership (the supervisor is host-networked or runtime-bridged, so the filter matched the CI job container instead), and it required the gateway mTLS CA as a bind mount (Podman delivers it as a container secret when user namespaces remap UIDs). Select the supervisor by managed labels and accept the secret delivery form. The lifecycle leg then exposed a driver bug: stopped-pair trust replacement re-created the per-sandbox TLS CA secret, colliding with the secret still owned by the renamed stopped pair. Mirror the token and resolver secret handling: require and reuse the existing secret during replacement instead of creating it. Validated end-to-end with the Podman additional CA e2e lane. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(server): restore upstream recovery semantics for start-request failures Recovery no longer marks sandboxes Error/InvalidPersistedSandbox when driver_start_request fails during the startup or lifecycle sweeps. Gateway configuration problems (for example a missing launch signer) must not permanently fail unrelated sandbox records; the sweep logs a warning and skips them, matching the behavior before the additional CA work. Sandbox-record failures surface through the normal start path. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(podman): refresh stale supervisors without replacing workloads Replace the stopped-pair trust reconciliation with supervisor-only recreation. The destination-trust generation label moves from the workload to the supervisor container, and start_sandbox recreates only the supervisor when its generation is missing or stale. The workload container, its volumes, its secrets, and its workspace data are never replaced: the additional CA bundle is mounted only into the supervisor. Running sandboxes are never gated on trust labels, and recovery without fresh launch credentials keeps the existing supervisor. The intentional -removal fence machinery in the watcher existed only for workload replacement and is removed with it. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(docker): drop workload replacement for destination-trust changes Delete the stopped-workload trust reconciliation path and all of its machinery: the trust-generation label on workload containers, the generation match check in the start fence, the stopped-sandbox replacement, the bounded workspace archive copy with its 512 MiB cap and regular-file-only validation, and the boundary-state rollback helpers. No replacement machinery is needed: the trusted supervisor container is already re-created from the gateway's current configuration on every start via ensure_control_process_for_container, so stop/start picks up the current additional CA bundle while the workload container, its volumes, and its workspace data are never replaced. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * docs(configuration): drop reconciliation wording from additional CA lifecycle The stop/start lifecycle description already matches the supervisor-only refresh model; remove the two remaining references to restart reconciliation of workload containers. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): port additional CA test fixtures to the crypto interface Route the gateway TLS isolation test PKI, the destination CA source fixtures, and the gateway startup trust tests through openshell_crypto::pki now that rcgen access is centralized there: generate_keypair/self_signed/signed_by replace the direct rcgen calls and CertifiedKey field access follows the wrapper's shape. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(e2e): repair additional CA lane for Docker and non-container builds The first maintainer test dispatch revealed two latent issues in the additional CA e2e that no local run had exercised: the Podman gateway CA secret helper referenced ContainerEngine without the container feature gate, breaking Kubernetes and VM builds of the test, and the Docker branch asserted a client ca.crt bind mount that the Docker driver never creates. Gate the helper like its callers, and assert Docker's actual delivery path instead: the gateway mTLS CA staged as tls/ca.pem inside the read-only supervisor state volume, selected through the OPENSHELL_TLS_CA environment pointer. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(helm): restore alphabetical README ordering for supervisor keys The helm:docs check regenerates the chart README with helm-docs, which orders rows alphabetically; a conflict resolution had placed supervisor.network.additionalCaConfigMapName after the supervisor.podSecurityContext rows. Regenerate with mise run helm:docs. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * chore: refresh testkit lockfile for openshell-core libc dependency Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(driver-podman): stage operator CA bundle as driver-owned artifact Standalone Podman drivers accepting --sandbox-additional-ca-bundle previously kept the operator's source path as the trust artifact path. The immutable-artifact verification performed on every create/start requires exactly mode 0444, which an operator-owned source path cannot guarantee, so a readable-but-writable source failed reconciliation. Read the operator bundle with the same bounded, no-symlink validation, then stage the normalized material into a driver-owned artifact inside a private tempdir held for the driver process lifetime. The artifact is created 0600, synced, and only then restricted to 0444 so no observer ever sees a writable artifact. The operator source and its permissions are never referenced after startup. Addresses GATOR-291f43b2-02. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(driver-podman): pin retained workload image during supervisor refresh Supervisor-only trust reconciliation re-entered create_sandbox_with_existing_token, which re-pulled and re-resolved the workload image tag even though the workload container is retained. If the tag moved to a different image between stop and start, identity resolution and child-environment construction would reflect the new image while the retained workload still runs the original one, leaving mode-0600 bootstrap files the retained user cannot read. During a refresh with a retained workload, inspect the stopped container and pin to the immutable image ID it was created from (Config.Image); never pull or resolve the request tag. The requested reference is still used for child metadata so the retained workload's environment is byte-identical to its original launch. Addresses GATOR-291f43b2-01. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * test(e2e): verify host-only VM trust delivery and drop curl dependency The additional-CA lane's VM assertions read the destination CA out of the guest overlay and a guest-side supervisor-argv marker — artifacts from the guest-supervisor design that no longer exist now that the VM driver stages the CA host-only and runs the supervisor on the host. Inspect the driver-staged host artifact (digest-compared against the source), assert the guest overlay contains no CA material at the old mount paths, and keep the serial-console no-disclosure check. The workload validation scripts also ran curl, which exists only in the container-engine fixture image; the Kubernetes and VM lanes use the uv-based workload image and died instantly with "canonical main process already finished". Rewrite the main-process, exec, and lifecycle validation scripts in python3 (present in every lane's workload image). The network-policy binary identity is the resolved /proc/<pid>/exe target, so the versioned python3.12 binaries are allowlisted rather than the unversioned symlinks. The hostname mismatch check keeps its semantics: same reachable fixture, mismatched TLS SNI, and the failure must specifically be hostname verification. Addresses GATOR-291f43b2-04. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(kubernetes): allow admission-added labels on destination-trust ConfigMaps The destination-trust ConfigMap validation compared the whole label map for equality. A mutating admission webhook that attaches an organizational label to the created ConfigMap made the create response fail validation, and the retained ConfigMap then failed every subsequent create/start through the existing-object path, blocking sandbox launches despite the ownership labels, immutable flag, digest, and certificate bytes remaining correct. Validate the gateway-owned label pairs individually and allow unrelated labels; the exact trust-material checks are unchanged. Covered by a regression test with an admission-added label and a missing gateway-owned label. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(podman): close supervisor-refresh rollback gaps The supervisor-only trust refresh renamed the previous supervisor to a random rollback name and then performed the token-secret lookup. A transient lookup failure returned with the supervisor still renamed, where a normal delete could never find it, and the rollback rename and final rollback removal ignored their own failures, leaving orphans with attached control and channel resources. Complete every fallible preparation before the rename. On a failed replacement, restore the canonical name; if the restore itself fails, return an error naming the rollback container so the orphan stays identifiable — a later retry still succeeds because the canonical name is absent and the replacement path recreates a fresh supervisor. Treat not-found during the final rollback removal as success and name the orphan in both the log and the error on other removal failures; a retry observes the fresh canonical supervisor and resumes the normal start path. Covered by failure-injection tests for each transaction boundary: preparation failure before any rename, replacement failure restoring the canonical name, and restore failure naming the orphaned rollback container. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * docs: address destination-trust review corrections The gateway-configuration RFC documented an additional_ca_cert_paths list that the strict configuration parser never accepted; document the additional_ca_bundle single-path setting the parser actually enforces. The debug-openshell-cluster skill named the wrong generation annotation (openshell.ai/network-supervisor-trust-generation) and described digest argv and network-init canonicalization checks the implementation does not perform. Correct it to openshell.ai/additional-ca-generation and describe the actual delivery: the supervisor Pod mounts the immutable ConfigMap read-only and receives --additional-ca-bundle for the mount path, with the driver validating the digest annotation and exact bytes per creation. Classify the destination-trust generation hashes as deterministic identity derivation rather than authenticity proofs, following the openshell-crypto exclusions table: trust decisions come from the gateway's normalization, immutable staging, exact-byte validation, and TLS verification. Record the classification in the crypto README and at each hashing site. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): reconcile additional CA tests with suspend/resume and auth changes Upstream's suspend/resume lifecycle took StartSandboxRequest fields 7 and 8, so the durable-snapshot field moves from 7 to 9; update the wire-format test to assert the field-9 key (0x4a) and initialize the new resume controls. Repair the cli test module after the auth opt-in test merged alongside the environment-guard helpers, and initialize the gateway driver-startup context's network-trust fields in the shared supervision test. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(server): box start-recovery futures for clippy large-futures The durable-snapshot field grows StartSandboxRequest enough that the start-recovery match and the recovery-expiry test exceed the large-futures limit under -D warnings. Box::pin both futures, matching the pattern the recovery path used before the suspend/resume rework. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> --------- Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
OpenShell crypto backend
See Cryptography and FIPS for the architecture, contributor rules, and production OpenSSL/FIPS follow-up requirements.
openshell-crypto owns first-party crypto backend selection. AWS-LC is the only
production implementation. This change preserves TLS algorithms, Ed25519 gateway
JWTs, P-256 certificate keys, native trust roots, and credential envelope formats.
The interface is intended to support a system-OpenSSL backend for regulated
deployments, related to #900.
This stage implements AWS-LC only. An OpenSSL proof of concept will validate the
provider integration separately; this is not a FIPS build or compliance claim.
Boundaries
CryptoBackend defines randomness, incremental SHA-256, and AES-256-GCM without
crypto-library types. ProtocolBackend adds Rustls, rcgen, and jsonwebtoken
adapters. These adapters retain their protocol libraries' validation and encoding;
they do not expose AWS-LC types. A later native OpenSSL TLS integration may need
an additional transport adapter; this interface does not claim to provide one.
Digest creation, updates, and finalization return Result so backends can report
provider and operation failures. Callers discard a digest after an update error;
the one-shot SHA-256 helper propagates the first failure without fallback.
Callers use aead, pki, tls, and jwt, or an explicit CryptoContext.
Certificate policy and JWT claim validation remain at their existing callers.
pki::KeyPair owns a backend SigningKey and implements rcgen 0.14's
PublicKeyData and SigningKey traits directly.
The backend generates/imports keys, signs messages, and exports PKCS#8 DER.
PEM wrapping and certificate encoding remain in the facade. Public keys expose
algorithm-specific bytes and SPKI; private exports are fallible, including for
non-exportable keys. Certificate encoding never requires private-key export.
pki::Certificate retains public issuance parameters separately from the encoded
certificate, allowing generated certificates to be used as issuers without
retaining private keys. Persisted proxy CAs use pki::issuer_from_der and keep
their original certificate bytes in issued chains.
Persisted CA keys are imported through the selected backend. Provider resources
must remain alive for the full key lifetime.
Default certificate serial numbers and key identifiers are derived through the
selected digest backend, preserving the existing SHA-256 derivation without
depending on rcgen's compiled crypto implementation.
Build selection
The crate's default aws-lc feature selects the current implementation. The root
workspace declares that choice once in its openshell-crypto dependency;
application crates inherit it rather than choosing a backend independently.
Standalone examples select their own default through their path dependency.
Disabling crate defaults removes its AWS-LC implementation and requires an
explicit context before use; use without initialization fails closed with an
initialization panic. This mode is for embedders and backend contract tests, not
an independently runnable OpenShell product configuration.
Integration features (tls-tonic, tls-kube, tls-sqlx) activate
dependencies. The aws-lc feature supplies their AWS-LC choices conditionally.
These libraries still own their crypto selection internally; they do not consult
CryptoContext. Cargo feature unification does not prevent direct library use.
A second backend must adapt these integrations and the application startup path.
Disabling this crate's feature alone does not remove dependency-owned crypto
from the entire product graph.
The later OpenSSL build must use system shared libraries without vendoring, respect system provider configuration, and verify the resulting linked artifacts. Linking and module qualification are build/deployment concerns, separate from the Rust key and primitive contracts. The proof of concept will test those concerns and the Rustls provider integration before a production OpenSSL backend is added.
Lifecycle and posture
default_context() lazily selects AWS-LC when that feature is enabled. An embedder can call
install_default_context(context) before first use. Reinstalling a clone of the
same context succeeds; installing a different context fails even if names match.
There is no live backend switching. Independent contexts can be used without
changing process state, including in contract tests.
tls::provider() constructs the selected context's provider.
tls::ensure_default_provider() retains an existing Rustls process default and
returns it; it does not attest ownership. Directly constructed OpenShell TLS
configurations use the facade provider, including the CLI health-check connector.
The dependency-owned clients listed below retain their own selection paths.
If a context was explicitly selected before facade use, these configurations use
that context even if another library already initialized Rustls. Otherwise they
preserve an embedder's installed provider. Key loading, custom verification
algorithms, and the gateway client-certificate verifier use the same
configuration-provider policy.
Explicit protocol-version constraints remain unchanged. A provider incompatible
with the requested versions fails during construction rather than falling back.
This rule controls first-party configs; it cannot override dependency-owned TLS.
JWT initialization likewise preserves an existing process provider.
install_jwt_provider() reports whether this call installed it; jsonwebtoken 10
has no public getter for verifying a previously installed provider. JWT adapters
retain that library's algorithm/key binding and claim-validation checks. Select a
custom context before JWT initialization, including indirect initialization by
dependencies. Neither global provider can be replaced safely.
CryptoContext::verify_posture(false) reports the context's capabilities, not
ownership of process globals. A strict request is always rejected in this stage.
The module version is explicitly unknown because the backend wrapper does not
provide that information through the API used here. Backend identity, enabled
algorithms, and a successful operation are not deployment FIPS evidence.
Compatibility and exclusions
AEAD stores a random 12-byte nonce and appends a 16-byte tag to ciphertext. AAD construction, envelope version, key IDs, and encoding remain in the credential driver. Random nonces do not guarantee uniqueness; existing key-volume and rotation constraints still apply. Authentication failures expose no plaintext or distinction between wrong keys, AAD, nonce, ciphertext, or tags.
Coverage excludes dependency-owned SSH/russh primitives, AWS SigV4 signing, SPIFFE RustCrypto verification, and non-Rust client runtimes. AWS SDK HTTPS also retains its dependency-owned selection. Non-security content hashes and ordinary scheduling randomness are not migrated. No SSH or PQC capability is asserted by the primitive capability report. OpenSSL, strict policy, module version discovery, and deployment qualification belong in follow-up work.
Operator certificate fingerprints, runtime refresh replay hashes and derived token IDs, bearer-cache fingerprints, Sandbox Protocol request digests and request-ID entropy, Kubernetes/VM runtime fence entropy, VM socket-directory entropy, and rootfs staging-token entropy use the selected backend. These operations retain SHA-256, UUID v4 encoding where applicable, and existing hex encodings. Backend failures propagate; callers must not substitute another implementation or continue with a placeholder identity.
The remaining first-party uses require explicit classification; the facade is not a whole-application crypto coverage claim:
| Remaining use | Current scope and follow-up |
|---|---|
| Reqwest clients (OIDC/JWKS, token exchange, provider refresh, Vault, token grants, telemetry) | These use Rustls's process-default provider after ensuring one exists; they do not necessarily honor an explicitly selected CryptoContext. The OpenSSL follow-up must supply compatible preconfigured TLS or another supported transport integration and test actual provider use. |
Kubernetes client TLS (kube-client 0.99) |
The AWS-LC-enabled client attempts to install AWS-LC as the Rustls process default. It does not consult CryptoContext; the OpenSSL follow-up must adapt this transport before claiming selected-backend coverage. |
Outer-fence evidence digest (openshell-isolation-interface::contract) |
The digest binds generation and native enforcement evidence using sha2 through Sha256Digest::compute. This security-relevant hash remains outside facade dispatch and must be migrated with failure propagation before claiming exclusive backend ownership. |
VM image/archive digest verification and cached supervisor validation (openshell-driver-vm) |
Security-relevant artifact integrity remains outside this runtime credential migration. Moving it requires covering the streaming readers, embedded-artifact cache, and build-time digest generation together; it must be addressed before claiming exclusive backend ownership. |
| UUID generation outside Sandbox Protocol request envelopes | Includes runtime identities, authentication nonces, and session identifiers as well as ordinary correlation IDs. These still use the UUID library's entropy source. They are not attested by context posture; a follow-up must classify each use and propagate entropy failure through currently infallible constructors. |
DNS policy contract fingerprints (openshell-supervisor-network::policy_dns) |
These bind cached DNS state to policy and are security-relevant. They remain outside this migration; migrating the cache identity API and its failure handling is required for exclusive backend ownership. |
Credential-storage names (credentials, Kubernetes Secrets, Vault) |
Deterministic namespace derivation, not encryption or proof of authorization. These retain their current hashing implementation and stored names; backend coverage is not asserted for them. |
| Provider-refresh advisory-lock keys | Deterministic database lock selection for coordinating refreshes, not authentication or encryption. These remain outside the selected-backend contract. |
| Endpoint IDs and pagination fingerprints | Deterministic identification/request matching, not authentication. These remain outside the selected-backend contract. |
| Debug token fingerprints | Diagnostic display only; direct hashing remains outside the contract even though the input is a credential. |
Destination-trust generation labels (openshell-server network trust, Podman operator staging, Kubernetes ConfigMap naming, VM staged-snapshot checks) |
Deterministic identity derivation for naming and generation equality, not encryption or proof of authorization. Trust decisions use the gateway's normalization, immutable staging, exact-byte validation, and TLS verification rather than these digests. These retain sha2 directly. |
| Schema compatibility tests, content-cache identities, retry jitter, test fixtures | Non-authentication uses remain unchanged. |
Cargo feature selection does not prevent a crate from adding another crypto dependency. Review first-party security-sensitive operations against this scope; no source-scanning boundary lint is installed.
Durable proxy CA loading uses backend key import and pki::issuer_from_der.
The helper uses x509-parser without verification features to read the subject,
key usage, and subject key identifier. Missing identifiers use the selected
backend's SHA-256 over SPKI, truncated to 20 bytes. It rejects trailing bytes,
malformed metadata, and subject names that rcgen cannot represent without loss
(including repeated attribute OIDs and multi-valued RDNs).
Parsing does not establish trust or verify signatures; the proxy separately
checks certificate/key matching through the selected TLS backend.
rcgen's x509-parser feature stays disabled: in 0.14.10 it also compiles CSR
verification that requires AWS-LC or Ring. No fork or fallback backend is needed
for CA import, and the parser's former Ring exception is removed from deny.toml.
The standalone facade also excludes AWS-LC when default features are disabled.
The build-time Z3 archive downloader retains the narrowly allowed Rustls/Ring
wrappers in deny.toml; context capabilities do not attest build tools.
Extending and checking
Implement both traits without importing application crates. Retain provider/key resources for the full operation lifetime, propagate entropy and authentication failures, and never fall back after a failure. Document unsupported algorithms and avoid secret-bearing diagnostics. Add protocol adapters rather than moving certificate or authorization policy into primitive implementations.
Run cargo test -p openshell-crypto for known-answer crypto, tampering, JWT
validation, key import/export, encryption compatibility, context substitution,
and an actual TLS handshake after process-default preemption.
cargo test -p openshell-crypto --no-default-features --test independent_backend
exercises independent test-only key and AEAD implementations without AWS-LC or
Ring. Inspect that boundary with
cargo tree -p openshell-crypto --no-default-features --edges normal,build,dev.
Branch Checks runs this test separately from workspace tests and rejects Ring or
AWS-LC in the dependency tree, including build and test dependencies.
Existing bootstrap, credential-store, gateway TLS/OIDC, and proxy tests exercise the
migrated consumers. The cargo-deny Ring restrictions and their explicit wrapper
exceptions remain in force.