When pulling container images from ghcr.io, credentials are always
sent. If the token lacks access to a specific repo (e.g., a public
repo the caller's PAT cannot read), the pull fails with a 401/403
even though an unauthenticated pull would succeed.
Add fallback logic across all three pull paths:
- ensure_image() (local Docker daemon): detect auth failure from the
pull stream and retry with no credentials.
- pull_remote_image() (remote SSH daemon): same pattern using the new
consume_pull_stream() helper.
- cluster-entrypoint.sh (k3s/containerd): validate credentials via
the GHCR token endpoint before writing auth config to
registries.yaml; skip auth block when credentials are rejected so
containerd uses anonymous pulls.
Also adds is_auth_failure() helper that matches 401/403 status codes
and common Docker error messages (pull access denied, unauthorized,
denied: access forbidden), curl to the cluster runtime image for the
credential test, and updated error diagnosis messaging.
* feat(sandbox): inject host gateway hostAliases into sandbox pods
Sandbox pods running in the k3s cluster cannot resolve host.docker.internal
by default, preventing them from reaching services on the Docker host.
Detect the host gateway IP (default route) in the cluster entrypoint,
thread it through the Helm chart to the gateway server, and inject
hostAliases entries (host.docker.internal, host.openshell.internal)
into every sandbox pod spec. The injection is conditional -- when the
IP is empty (non-Docker deployments), no hostAliases are added.
---------
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
Add --gpu flag to 'openshell gateway start' that enables NVIDIA GPU
access inside the cluster container. When enabled, the Docker container
receives GPU device requests via bollard, k3s auto-detects the NVIDIA
runtime, and the NVIDIA k8s-device-plugin is deployed via HelmChart CR
so Kubernetes workloads can request nvidia.com/gpu resources.
- Add GPU device request to Docker container creation (bollard DeviceRequest)
- Set GPU_ENABLED=true env var to gate GPU manifest deployment
- Multi-stage Dockerfile build to bundle NVIDIA Container Toolkit binaries
- Conditional GPU manifest copy in cluster entrypoint
- HelmChart CR for nvidia-device-plugin v0.18.2 with GFD and NFD enabled
- Thread gpu bool through DeployOptions, CLI flag, and run dispatch
---------
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
* refactor(docker): rename server image to gateway
Rename Dockerfile.server to Dockerfile.gateway and update all image
references from openshell/server to openshell/gateway across Helm
charts, Kubernetes manifests, mise tasks, build/deploy scripts, CI
workflows, and documentation.
The underlying Rust binary (navigator-server) is unchanged -- this
rename only affects the Docker image name and Dockerfile.
* fix: catch remaining server->gateway references in docs and comments
* fix(security): harden sandbox SSH with mandatory HMAC secret, NetworkPolicy, and nonce replay detection
Closes#25
- Make NEMOCLAW_SSH_HANDSHAKE_SECRET mandatory: server and sandbox both
refuse to start if the secret is empty/unset. Cluster deployments
auto-generate it via openssl rand in the entrypoint script.
- Add Kubernetes NetworkPolicy restricting sandbox port 2222 ingress to
the gateway pod only, preventing lateral movement from other cluster
workloads.
- Add NSSH1 nonce replay detection with a TTL-bounded cache, rejecting
replayed handshakes within the timestamp validity window.
- Add unit tests for verify_preface (valid, replay, expired, bad HMAC,
malformed) and env injection.
* fix(deploy): pass sshHandshakeSecret in fast deploy helm upgrade
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
Closes#48, #52
## Summary
- Replace the envoy-gateway-based TLS setup with inline PKI generation during cluster bootstrap, generating CA, server, and client certificates directly in the `navigator-bootstrap` crate
- Remove all envoy gateway Helm templates (`gateway.yaml`, `gatewayclass.yaml`, `grpcroute.yaml`, PKI job, traffic policies) and the `Dockerfile.pki-job`
- Add native mTLS support to the navigator server with `tokio-rustls`, mounting client TLS certs as volumes into sandbox pods
- Update cluster entrypoint, healthcheck, and deploy scripts to work with the new direct-TLS architecture
- Add TLS security e2e test and fix formatting/clippy warnings
## Test Plan
- All unit tests pass (`cargo test --workspace`)
- Clippy clean (`cargo clippy --workspace --all-targets`)
- Format clean (`cargo fmt --all -- --check`)
- Python tests pass (`uv run pytest python/`)
- Full `mise run pre-commit` passes
Closes#13
## Summary
- add Python sandbox execution APIs for command and callable workflows
- consolidate sandbox policy fixtures and expand e2e test coverage for policy and Python exec paths
- update CI/build config and images for sandbox e2e execution dependencies
## Test Plan
- mise run pre-commit
## Summary
- Add a "push" path for local development that exports Docker images from the host daemon and imports them directly into the k3s containerd runtime, eliminating the need for a local registry or remote image pulls during `mise run cluster`
- Add `mise run cluster:deploy` task for fast iterative rebuilds (~25s) against a running cluster without recreating it
- Update `cluster-entrypoint.sh` to patch HelmChart manifest with local image tags and `IfNotPresent` pull policy when push mode is active
## Details
Previously, after the local registry (k3d + skaffold) was removed, `mise run cluster` only pulled component images from the remote CloudFront registry. This meant local code changes required a remote publish cycle to test in-cluster.
This MR adds a push path that sits alongside the existing pull path:
- **`push.rs`** — Uses bollard to `export_images()` all 3 component images into a single tar (with shared layer dedup), uploads via Docker `put_archive` API into the cluster container, then runs `ctr images import` to load them into containerd
- **`docker.rs`** — Passes `IMAGE_TAG` and `IMAGE_PULL_POLICY` env vars into the cluster container when `NAVIGATOR_PUSH_IMAGES` is set
- **`lib.rs`** — Wires push into the deploy flow between `clean_stale_nodes()` and `wait_for_cluster_ready()`, guarded by `remote_opts.is_none() && NAVIGATOR_PUSH_IMAGES`
- **`cluster-entrypoint.sh`** — Patches the baked-in HelmChart manifest with sed to use the local image tag and pull policy before k3s starts
- **`cluster.toml`** — Updated `[cluster]` task to tag and push images; new `[cluster:deploy]` task for iterative deploys via `docker save | ctr import` + `helm upgrade`
The remote deploy path (`nav cluster admin deploy --remote`) is completely unaffected — the push logic only activates for local deploys with the env var set.
## Test Plan
- `mise run cluster` — end-to-end success, cluster healthy with local images (~168s)
- `mise run cluster:deploy` — iterative deploy success, helm upgrade + rollout (~43s)
- `mise run pre-commit` — all checks pass (fmt, clippy, 82 tests, python lint)
- Verified images present in containerd: `ctr -a /run/k3s/containerd/containerd.sock images ls | grep navigator/`
- Remote path verified safe via code analysis (guards prevent push logic from running)
The previous approach wrote the host.docker.internal IP to the k3s
resolv.conf, assuming it would serve DNS. On Docker Desktop, this IP
is reachable but doesn't run a DNS server, causing CoreDNS to fail
resolving external names and breaking the PKI job and Helm install.
The new approach discovers Docker's embedded DNS listener ports from
the DOCKER_OUTPUT iptables chain, then sets up DNAT rules to forward
DNS traffic from k3s pods (via PREROUTING) to Docker's actual DNS
resolver on 127.0.0.11. The container's eth0 IP is used as the
nameserver in the k3s resolv.conf since it's routable from all pods.
Drew Newberry