diff --git a/.agents/skills/debug-navigator-cluster/SKILL.md b/.agents/skills/debug-navigator-cluster/SKILL.md index 8491894e9..36abf4759 100644 --- a/.agents/skills/debug-navigator-cluster/SKILL.md +++ b/.agents/skills/debug-navigator-cluster/SKILL.md @@ -18,13 +18,12 @@ Diagnose why a openshell cluster failed to start after `openshell gateway start` 5. Wait for k3s to generate kubeconfig (up to 60s) 6. **Clean stale nodes**: Remove any `NotReady` k3s nodes left over from previous container instances that reused the same persistent volume 7. **Prepare local images** (if `OPENSHELL_PUSH_IMAGES` is set): In `internal` registry mode, bootstrap waits for the in-cluster registry and pushes tagged images there. In `external` mode, bootstrap uses legacy `ctr -n k8s.io images import` push-mode behavior. -7. **Reconcile TLS PKI**: Load existing TLS secrets from the cluster; if missing, incomplete, or malformed, generate fresh PKI (CA + server + client certs). Apply secrets to cluster. If rotation happened and the OpenShell workload is already running, rollout restart and wait for completion (failed rollout aborts deploy). -8. **Store CLI mTLS credentials**: Persist client cert/key/CA locally for CLI authentication. -9. Wait for cluster health checks to pass (up to 6 min): - - k3s API server readiness (`/readyz`) - - `navigator` statefulset ready in `navigator` namespace - - `navigator-gateway` Gateway programmed in `navigator` namespace - - TLS secrets `navigator-server-tls` and `navigator-client-tls` exist +8. **Reconcile TLS PKI**: Load existing TLS secrets from the cluster; if missing, incomplete, or malformed, generate fresh PKI (CA + server + client certs). Apply secrets to cluster. If rotation happened and the OpenShell workload is already running, rollout restart and wait for completion (failed rollout aborts deploy). +9. **Store CLI mTLS credentials**: Persist client cert/key/CA locally for CLI authentication. +10. Wait for cluster health checks to pass (up to 6 min): + - k3s API server readiness (`/readyz`) + - `openshell` statefulset ready in `openshell` namespace + - TLS secrets `openshell-server-tls` and `openshell-client-tls` exist in `openshell` namespace For local deploys, metadata endpoint selection now depends on Docker connectivity: @@ -53,24 +52,42 @@ Before running commands, establish: 1. **Cluster name**: Default is `openshell`, giving container name `openshell-cluster-openshell` 2. **Remote or local**: If the user deployed with `--remote `, all Docker commands must target that host -3. **Config directory**: `~/.config/openshell/clusters/{name}/` +3. **Config directory**: `~/.config/openshell/gateways/{name}/` -For remote clusters, prefix Docker commands with SSH: +The CLI provides a shortcut for running commands inside the gateway container: ```bash -# Remote docker commands -ssh docker +# Run kubectl inside the gateway container (auto-resolves local/remote) +openshell doctor exec -- kubectl -# Remote kubectl inside the container +# Interactive k9s session +openshell doctor exec -- k9s + +# Interactive shell +openshell doctor exec -- sh +``` + +For direct Docker access (when the CLI is unavailable), use Docker commands directly for local clusters or prefix with SSH for remote: + +```bash +# Local +docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl ' + +# Remote ssh docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl ' ``` -For local clusters, run Docker commands directly: +### Step 0: Quick Connectivity Check -```bash -docker -docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl ' -``` +Run `openshell status` first. This immediately reveals: +- Which gateway and endpoint the CLI is targeting +- Whether the CLI can reach the server (mTLS handshake success/failure) +- The server version if connected + +Common errors at this stage: +- **`tls handshake eof`**: The server isn't running or mTLS credentials are missing/mismatched +- **`connection refused`**: The container isn't running or port mapping is broken +- **`No gateway configured`**: No gateway has been deployed yet ### Step 1: Check Docker Container State @@ -110,7 +127,7 @@ Look for: - DNS resolution failures in the entrypoint script - k3s startup errors (certificate issues, port binding failures) -- Manifest copy errors from `/opt/navigator/manifests/` +- Manifest copy errors from `/opt/openshell/manifests/` - `iptables` or `cgroup` errors (privilege/capability issues) ### Step 3: Check k3s Cluster Health @@ -149,49 +166,48 @@ If any pressure condition is `True`, pods will be evicted and new ones rejected. ### Step 4: Check OpenShell Server StatefulSet -The OpenShell server is deployed via a HelmChart CR as a StatefulSet with persistent storage. Check its status: +The OpenShell server is deployed via a HelmChart CR as a StatefulSet named `openshell` in the `openshell` namespace. Check its status: ```bash # StatefulSet status -docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n navigator get statefulset/navigator -o wide' +docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n openshell get statefulset/openshell -o wide' # OpenShell pod logs -docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n navigator logs statefulset/navigator --tail=100' +docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n openshell logs statefulset/openshell --tail=100' # Describe statefulset for events -docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n navigator describe statefulset/navigator' +docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n openshell describe statefulset/openshell' # Helm install job logs (the job that installs the OpenShell chart) -docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n kube-system logs -l job-name=helm-install-navigator --tail=200' +docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n kube-system logs -l job-name=helm-install-openshell --tail=200' ``` Common issues: +- **Replicas 0/0**: The StatefulSet has been scaled to zero — no pods are running. This can happen after a failed deploy, manual scale-down, or Helm values misconfiguration. Fix: `kubectl -n openshell scale statefulset openshell --replicas=1` - **ImagePullBackOff**: The component image failed to pull. In `internal` mode, verify internal registry readiness and pushed image tags (Step 6). In `external` mode, check `/etc/rancher/k3s/registries.yaml` credentials/endpoints and DNS (Step 8). Default external registry is `ghcr.io/nvidia/openshell/`. Ensure a valid `--registry-token` (or `OPENSHELL_REGISTRY_TOKEN`) was provided during deploy. - **CrashLoopBackOff**: The server is crashing. Check pod logs for the actual error. - **Pending**: Insufficient resources or scheduling constraints. -### Step 5: Check Gateway and Networking +### Step 5: Check Networking -The Envoy Gateway provides HTTP/gRPC ingress: +The OpenShell server is exposed via a NodePort service on port `30051`: ```bash -# Gateway status -docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n navigator get gateway/navigator-gateway' +# Service status +docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n openshell get service/openshell' # Check port bindings on the host docker port openshell-cluster- ``` -Expected ports: `6443/tcp`, `30051/tcp` (mapped to configurable host port, default 8080; set via `--port` on deploy). -Only one local cluster can run on a Docker host at a time because `6443` is fixed. -`mise run cluster` handles this by removing conflicting local `openshell-cluster-*` containers first. +Expected port: `30051/tcp` (mapped to configurable host port, default 8080; set via `--port` on deploy). If ports are missing or conflicting, another process may be using them. Check with: ```bash # On the host (or remote host) -ss -tlnp | grep -E ':(6443|8080)\s' +ss -tlnp | grep -E ':8080\s' ``` If using Docker-in-Docker (`DOCKER_HOST=tcp://docker:2375`), verify metadata points at `https://docker` (not `https://127.0.0.1`). @@ -204,7 +220,7 @@ Component images (server, sandbox, pki-job) can reach kubelet via two paths: ```bash # Verify image refs currently used by openshell deployment -docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n navigator get deploy navigator -o jsonpath="{.spec.template.spec.containers[*].image}"' +docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec.containers[*].image}"' # Verify registry mirror/auth endpoint configuration docker exec openshell-cluster- cat /etc/rancher/k3s/registries.yaml @@ -214,7 +230,7 @@ docker exec openshell-cluster- cat /etc/rancher/k3s/registries.yaml ```bash # Check if images were imported into containerd (k3s default namespace is k8s.io) -docker exec openshell-cluster- ctr -a /run/k3s/containerd/containerd.sock images ls | grep navigator +docker exec openshell-cluster- ctr -a /run/k3s/containerd/containerd.sock images ls | grep openshell ``` If images are missing, re-import with: @@ -241,21 +257,31 @@ TLS certificates are generated by the `navigator-bootstrap` crate (using `rcgen` ```bash # Check if the three TLS secrets exist -docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n navigator get secret navigator-server-tls navigator-server-client-ca navigator-client-tls' +docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n openshell get secret openshell-server-tls openshell-server-client-ca openshell-client-tls' # Inspect server cert expiry (if openssl is available in the container) -docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n navigator get secret navigator-server-tls -o jsonpath="{.data.tls\.crt}" | base64 -d | openssl x509 -noout -dates 2>/dev/null || echo "openssl not available"' +docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n openshell get secret openshell-server-tls -o jsonpath="{.data.tls\.crt}" | base64 -d | openssl x509 -noout -dates 2>/dev/null || echo "openssl not available"' # Check if CLI-side mTLS files exist locally -ls -la ~/.config/openshell/clusters//mtls/ +ls -la ~/.config/openshell/gateways//mtls/ ``` On redeploy, bootstrap reuses existing secrets if they are valid PEM. If secrets are missing or malformed, fresh PKI is generated and the OpenShell workload is automatically restarted. If the rollout restart fails after rotation, the deploy aborts and CLI-side certs are not updated. Certificates use rcgen defaults (effectively never expire). +If the local mTLS files are missing but the secrets exist in the cluster, you can extract them manually: + +```bash +mkdir -p ~/.config/openshell/gateways//mtls +docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n openshell get secret openshell-client-tls -o jsonpath="{.data.ca\.crt}"' | base64 -d > ~/.config/openshell/gateways//mtls/ca.crt +docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n openshell get secret openshell-client-tls -o jsonpath="{.data.tls\.crt}"' | base64 -d > ~/.config/openshell/gateways//mtls/tls.crt +docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n openshell get secret openshell-client-tls -o jsonpath="{.data.tls\.key}"' | base64 -d > ~/.config/openshell/gateways//mtls/tls.key +``` + Common mTLS issues: -- **Secrets missing**: The `navigator` namespace may not have been created yet (Helm controller race). Bootstrap waits up to 2 minutes for the namespace. +- **Secrets missing**: The `openshell` namespace may not have been created yet (Helm controller race). Bootstrap waits up to 2 minutes for the namespace. - **mTLS mismatch after manual secret deletion**: Delete all three secrets and redeploy — bootstrap will regenerate and restart the workload. -- **CLI can't connect after redeploy**: Check that `~/.config/openshell/clusters//mtls/` contains `ca.crt`, `tls.crt`, `tls.key` and that they were updated at deploy time. +- **CLI can't connect after redeploy**: Check that `~/.config/openshell/gateways//mtls/` contains `ca.crt`, `tls.crt`, `tls.key` and that they were updated at deploy time. +- **Local mTLS files missing**: The gateway was deployed but CLI credentials weren't persisted (e.g., interrupted deploy). Extract from the cluster secret as shown above. ### Step 8: Check Kubernetes Events @@ -295,36 +321,38 @@ The entrypoint script selects DNS resolvers in this priority: 3. Host gateway IP (Docker Desktop only, `192.168.*`) 4. Fallback to `8.8.8.8` / `8.8.4.4` -If DNS is broken, all image pulls from the distribution registry will fail, as will pods that need external network access (PKI job, cert-manager). +If DNS is broken, all image pulls from the distribution registry will fail, as will pods that need external network access. ## Common Failure Patterns | Symptom | Likely Cause | Fix | |---------|-------------|-----| +| `tls handshake eof` from `openshell status` | Server not running or mTLS credentials missing/mismatched | Check StatefulSet replicas (Step 4) and mTLS files (Step 7) | +| StatefulSet `0/0` replicas | StatefulSet scaled to zero (failed deploy, manual scale-down, or Helm misconfiguration) | `kubectl -n openshell scale statefulset openshell --replicas=1` | +| Local mTLS files missing | Deploy was interrupted before credentials were persisted | Extract from cluster secret `openshell-client-tls` (Step 7) | | Container not found | Image not built | `mise run docker:build:cluster` (local) or re-deploy (remote) | | Container exited, OOMKilled | Insufficient memory | Increase host memory or reduce workload | | Container exited, non-zero exit | k3s crash, port conflict, privilege issue | Check `docker logs` and `docker inspect` for details | | `/readyz` fails | k3s still starting or crashed | Wait longer or check container logs for k3s errors | -| OpenShell pods `Pending` | Insufficient CPU/memory for scheduling, or PVC not bound | Check `kubectl describe pod` for scheduling failures and `kubectl get pvc -n navigator` for volume status | -| OpenShell pods `CrashLoopBackOff` | Server application error | Check `kubectl logs` on the crashing pod | +| OpenShell pods `Pending` | Insufficient CPU/memory for scheduling, or PVC not bound | Check `kubectl describe pod` for scheduling failures and `kubectl get pvc -n openshell` for volume status | +| OpenShell pods `CrashLoopBackOff` | Server application error | Check `kubectl -n openshell logs statefulset/openshell` | | OpenShell pods `ImagePullBackOff` (push mode) | Images not imported or wrong containerd namespace | Check `k3s ctr -n k8s.io images ls` for component images (Step 6) | | OpenShell pods `ImagePullBackOff` (pull mode) | Registry auth or DNS issue | Check `/etc/rancher/k3s/registries.yaml` credentials and DNS (Step 8) | | Image import fails (`k3s ctr` exit code != 0) | Corrupt tar stream or containerd not ready | Retry after k3s is fully started; check container logs | | Push mode images not found by kubelet | Imported into wrong containerd namespace | Must use `k3s ctr -n k8s.io images import`, not `k3s ctr images import` | -| Gateway not `Programmed` | Envoy Gateway not ready | Check `envoy-gateway-system` pods and Helm install logs | -| mTLS secrets missing | Bootstrap couldn't apply secrets (namespace not ready, kubectl exec failure) | Check deploy logs and verify `navigator` namespace exists (Step 7) | -| mTLS mismatch after redeploy | PKI rotated but workload not restarted, or rollout failed | Check that all three TLS secrets exist and that the navigator pod restarted after cert rotation (Step 7) | -| Helm install job failed | Chart values error or dependency issue | Check `helm-install-navigator` job logs in `kube-system` | +| mTLS secrets missing | Bootstrap couldn't apply secrets (namespace not ready, kubectl exec failure) | Check deploy logs and verify `openshell` namespace exists (Step 7) | +| mTLS mismatch after redeploy | PKI rotated but workload not restarted, or rollout failed | Check that all three TLS secrets exist and that the openshell pod restarted after cert rotation (Step 7) | +| Helm install job failed | Chart values error or dependency issue | Check `helm-install-openshell` job logs in `kube-system` | | Architecture mismatch (remote) | Built on arm64, deploying to amd64 | Cross-build the image for the target architecture | | SSH connection failed (remote) | SSH key/host/Docker issues | Test `ssh docker ps` manually | -| Port conflict | Another service on 6443 or the configured gateway host port (default 8080) | Stop conflicting service or use `--port` on `openshell gateway start` to pick a different host port | +| Port conflict | Another service on the configured gateway host port (default 8080) | Stop conflicting service or use `--port` on `openshell gateway start` to pick a different host port | | gRPC connect refused to `127.0.0.1:443` in CI | Docker daemon is remote (`DOCKER_HOST=tcp://...`) but metadata still points to loopback | Verify metadata endpoint host matches `DOCKER_HOST` and includes non-loopback host | | DNS failures inside container | Entrypoint DNS detection failed | Check `/etc/rancher/k3s/resolv.conf` and container startup logs | | Node DiskPressure / MemoryPressure / PIDPressure | Insufficient disk, memory, or PIDs on host | Free disk (`docker system prune -a --volumes`), increase memory, or expand host resources. Bootstrap auto-detects via `HEALTHCHECK_NODE_PRESSURE` marker | | Pods evicted with "The node had condition: [DiskPressure]" | Host disk full, kubelet evicting pods | Free disk space on host, then `openshell gateway destroy && openshell gateway start` | | `metrics-server` errors in logs | Normal k3s noise, not the root cause | These errors are benign — look for the actual failing health check component | | Stale NotReady nodes from previous deploys | Volume reused across container recreations | The deploy flow now auto-cleans stale nodes; if it still fails, manually delete NotReady nodes (see Step 3) or choose "Recreate" when prompted | -| gRPC `UNIMPLEMENTED` for newer RPCs in push mode | Helm values still point at older pulled images instead of the pushed refs | Verify rendered `navigator-helmchart.yaml` uses the expected push refs (`server`, `sandbox`, `pki-job`) and not `:latest` | +| gRPC `UNIMPLEMENTED` for newer RPCs in push mode | Helm values still point at older pulled images instead of the pushed refs | Verify rendered `openshell-helmchart.yaml` uses the expected push refs (`server`, `sandbox`, `pki-job`) and not `:latest` | ## Remote Cluster Debugging @@ -345,13 +373,10 @@ docker -H ssh:// ps -a docker -H ssh:// logs openshell-cluster- ``` -**Setting up kubectl access** (requires tunnel): +**Viewing gateway logs**: ```bash -openshell gateway tunnel --name --remote -# Then in another terminal: -export KUBECONFIG=~/.config/openshell/clusters//kubeconfig -kubectl get pods -A +openshell doctor logs --name --remote ``` ## Full Diagnostic Dump @@ -367,6 +392,9 @@ KCFG="KUBECONFIG=/etc/rancher/k3s/k3s.yaml" # Helper: run docker command locally or remotely run() { if [ -n "$HOST" ]; then ssh "$HOST" "$@"; else "$@"; fi; } +echo "=== Connectivity Check ===" +openshell status + echo "=== Container State ===" run docker ps -a --filter "name=${CONTAINER}" --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Ports}}' run docker inspect "${CONTAINER}" --format '{{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} error={{.State.Error}}' 2>/dev/null @@ -393,19 +421,19 @@ echo "=== Failing Pods ===" run docker exec "${CONTAINER}" sh -lc "${KCFG} kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded" 2>&1 echo "=== OpenShell StatefulSet ===" -run docker exec "${CONTAINER}" sh -lc "${KCFG} kubectl -n navigator get statefulset/navigator -o wide" 2>&1 +run docker exec "${CONTAINER}" sh -lc "${KCFG} kubectl -n openshell get statefulset/openshell -o wide" 2>&1 -echo "=== OpenShell Gateway ===" -run docker exec "${CONTAINER}" sh -lc "${KCFG} kubectl -n navigator get gateway/navigator-gateway" 2>&1 +echo "=== OpenShell Service ===" +run docker exec "${CONTAINER}" sh -lc "${KCFG} kubectl -n openshell get service/openshell" 2>&1 + +echo "=== TLS Secrets ===" +run docker exec "${CONTAINER}" sh -lc "${KCFG} kubectl -n openshell get secret openshell-server-tls openshell-server-client-ca openshell-client-tls" 2>&1 echo "=== Recent Events ===" run docker exec "${CONTAINER}" sh -lc "${KCFG} kubectl get events -A --sort-by=.lastTimestamp" 2>&1 | tail -n 50 -echo "=== PKI Job Logs ===" -run docker exec "${CONTAINER}" sh -lc "${KCFG} kubectl -n navigator logs -l job-name=navigator-gateway-pki --tail=100" 2>&1 - echo "=== Helm Install OpenShell Logs ===" -run docker exec "${CONTAINER}" sh -lc "${KCFG} kubectl -n kube-system logs -l job-name=helm-install-navigator --tail=100" 2>&1 +run docker exec "${CONTAINER}" sh -lc "${KCFG} kubectl -n kube-system logs -l job-name=helm-install-openshell --tail=100" 2>&1 echo "=== Registry Configuration ===" run docker exec "${CONTAINER}" cat /etc/rancher/k3s/registries.yaml 2>&1 @@ -415,4 +443,7 @@ run docker exec "${CONTAINER}" cat /etc/rancher/k3s/resolv.conf 2>&1 echo "=== Port Bindings ===" run docker port "${CONTAINER}" 2>&1 + +echo "=== Local mTLS Credentials ===" +ls -la ~/.config/openshell/gateways/${NAME}/mtls/ 2>/dev/null || echo "mTLS directory not found" ``` diff --git a/.agents/skills/openshell-cli/SKILL.md b/.agents/skills/openshell-cli/SKILL.md index 02e383ae2..cfa50c1dc 100644 --- a/.agents/skills/openshell-cli/SKILL.md +++ b/.agents/skills/openshell-cli/SKILL.md @@ -497,8 +497,11 @@ openshell gateway destroy # Destroy permanently # Deploy to remote host openshell gateway start --remote user@host --ssh-key ~/.ssh/id_rsa --name remote-cluster -# Set up kubectl access -openshell gateway tunnel --name remote-cluster +# View gateway container logs +openshell doctor logs --name remote-cluster + +# Run kubectl inside the remote gateway container +openshell doctor exec --name remote-cluster -- kubectl get pods -A # Get cluster info openshell gateway info --name remote-cluster diff --git a/.agents/skills/openshell-cli/cli-reference.md b/.agents/skills/openshell-cli/cli-reference.md index 82a261139..c0e3f742c 100644 --- a/.agents/skills/openshell-cli/cli-reference.md +++ b/.agents/skills/openshell-cli/cli-reference.md @@ -29,7 +29,6 @@ openshell │ ├── stop [opts] │ ├── destroy [opts] │ ├── info [--name] -│ ├── tunnel [opts] │ └── select [name] ├── status ├── inference @@ -62,6 +61,9 @@ openshell │ ├── list [opts] │ ├── update --type [opts] │ └── delete ... +├── doctor +│ ├── logs [--name] [-n] [--tail] [--remote] [--ssh-key] +│ └── exec [--name] [--remote] [--ssh-key] -- ├── term ├── completions └── ssh-proxy [opts] @@ -82,9 +84,6 @@ Provision or start a cluster (local or remote). | `--ssh-key ` | none | SSH private key for remote deployment | | `--port ` | 8080 | Host port mapped to gateway | | `--gateway-host ` | none | Override gateway host in metadata | -| `--kube-port [PORT]` | none | Expose K8s control plane on host port | -| `--update-kube-config` | false | Write kubeconfig into `~/.kube/config` | -| `--get-kubeconfig` | false | Print kubeconfig to stdout | | `--recreate` | false | Destroy and recreate from scratch if a gateway already exists (skips interactive prompt) | ### `openshell gateway stop` @@ -103,29 +102,50 @@ Destroy a cluster and all its state. Same flags as `stop`. ### `openshell gateway info` -Show deployment details: endpoint, kubeconfig path, kube port, remote host. +Show deployment details: endpoint and remote host. | Flag | Description | |------|-------------| | `--name ` | Cluster name (defaults to active) | -### `openshell gateway tunnel` - -Print or start an SSH tunnel for kubectl access to a remote cluster. - -| Flag | Description | -|------|-------------| -| `--name ` | Cluster name (defaults to active) | -| `--remote ` | SSH destination | -| `--ssh-key ` | SSH private key | -| `--print-command` | Only print the SSH command, don't execute | - ### `openshell gateway select [name]` Set the active gateway. Writes to `~/.config/openshell/active_gateway`. When called without arguments, lists all provisioned gateways with the active one marked with `*`. --- +## Doctor Commands + +### `openshell doctor logs` + +Fetch logs from the gateway Docker container. + +| Flag | Default | Description | +|------|---------|-------------| +| `--name ` | active gateway | Gateway name | +| `-n, --lines ` | all | Number of log lines to return | +| `--tail` | false | Stream live logs (follow mode) | +| `--remote ` | auto-resolved | SSH destination for remote gateways | +| `--ssh-key ` | none | SSH private key for remote gateways | + +### `openshell doctor exec -- ` + +Run a command inside the gateway container with KUBECONFIG pre-configured. +Launches an interactive `docker exec` session (tunnelled over SSH for remote gateways). + +| Flag | Default | Description | +|------|---------|-------------| +| `--name ` | active gateway | Gateway name | +| `--remote ` | auto-resolved | SSH destination for remote gateways | +| `--ssh-key ` | none | SSH private key for remote gateways | + +Examples: +- `openshell doctor exec -- kubectl get pods -A` +- `openshell doctor exec -- k9s` +- `openshell doctor exec -- sh` (interactive shell) + +--- + ## Status Command ### `openshell status` diff --git a/.claude/agent-memory/arch-doc-writer/MEMORY.md b/.claude/agent-memory/arch-doc-writer/MEMORY.md index dd37ffd0f..645eb216f 100644 --- a/.claude/agent-memory/arch-doc-writer/MEMORY.md +++ b/.claude/agent-memory/arch-doc-writer/MEMORY.md @@ -34,8 +34,8 @@ ## Bootstrap Crate Details - `docker.rs`: `ensure_container()` sets ~12 env vars (REGISTRY_*, IMAGE_*, PUSH_IMAGE_REFS, etc.) -- `runtime.rs`: Polling params: kubeconfig 30x2s, health 180x2s, mTLS 90x2s -- `metadata.rs`: Metadata at `clusters/{name}_metadata.json` (flat), kubeconfig/mTLS at `clusters/{name}/` (nested) +- `runtime.rs`: Polling params: health 180x2s, mTLS 90x2s +- `metadata.rs`: Metadata at `gateways/{name}/metadata.json` (nested), mTLS at `gateways/{name}/mtls/` (nested) - `push.rs`: Uses `ctr` (not `k3s ctr`) with k3s containerd socket, `k8s.io` namespace - IMPORTANT: `ClusterHandle::destroy()` does NOT remove metadata; only CLI `cluster_admin_destroy()` in run.rs does - `ensure_image()`: Local-only refs (no `/`) get error with build instructions, not a Docker Hub pull attempt diff --git a/.env.example b/.env.example index 3b3678c63..61cdf4e3c 100644 --- a/.env.example +++ b/.env.example @@ -20,9 +20,3 @@ # Host port mapped to the k3s NodePort (30051) where the OpenShell gateway # listens. The CLI connects here. Must be unique per cluster. #GATEWAY_PORT=8080 - -# Expose the Kubernetes control plane on this host port (maps to 6443 -# inside the container). Leave unset to skip the binding — this is the -# default and allows multiple clusters to coexist without port conflicts. -# Set this only when you need direct kubectl access to the cluster. -#KUBE_PORT=6443 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2c8dc141c..8da8cd909 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -51,6 +51,24 @@ openshell --help openshell sandbox create -- codex ``` +### Cluster debugging helpers + +Two additional scripts in `scripts/bin/` provide gateway-aware wrappers for cluster debugging: + +| Script | What it does | +|--------|-------------| +| `kubectl` | Runs `kubectl` inside the active gateway's k3s container via `openshell doctor exec` | +| `k9s` | Runs `k9s` inside the active gateway's k3s container via `openshell doctor exec` | + +These work for both local and remote gateways (SSH is handled automatically). Examples: + +```bash +kubectl get pods -A +kubectl logs -n navigator statefulset/navigator +k9s +k9s -n navigator +``` + ## Main Tasks These are the primary `mise` tasks for day-to-day development: diff --git a/Cargo.lock b/Cargo.lock index 7d19ba15e..a8d57e5e4 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2691,7 +2691,6 @@ dependencies = [ "rcgen", "serde", "serde_json", - "serde_yaml", "tar", "tempfile", "tokio", @@ -2728,7 +2727,6 @@ dependencies = [ "rustls-pemfile", "serde", "serde_json", - "serde_yaml", "tar", "temp-env", "tempfile", diff --git a/architecture/README.md b/architecture/README.md index 6b9977e6e..3a554a74a 100644 --- a/architecture/README.md +++ b/architecture/README.md @@ -232,7 +232,7 @@ For more detail, see [Policy Language](security-policy.md). The CLI is the primary way users interact with the platform. It provides commands organized into four groups: -- **Gateway management** (`openshell gateway`): Deploy, stop, destroy, and inspect clusters. Supports both local and remote (SSH) targets. Includes a tunnel command for accessing the Kubernetes API on remote clusters. +- **Gateway management** (`openshell gateway`): Deploy, stop, destroy, and inspect clusters. Supports both local and remote (SSH) targets. - **Sandbox management** (`openshell sandbox`): Create sandboxes (with optional file upload and provider auto-discovery), connect to sandboxes via SSH, and delete sandboxes. - **Top-level commands**: `openshell status` (cluster health), `openshell logs` (sandbox logs), `openshell forward` (port forwarding), `openshell policy` (sandbox policy management). - **Provider management** (`openshell provider`): Create, update, list, and delete external service credentials. diff --git a/architecture/build-containers.md b/architecture/build-containers.md index 63b45d373..c4f9a6e25 100644 --- a/architecture/build-containers.md +++ b/architecture/build-containers.md @@ -359,7 +359,7 @@ After building, the script: 1. Resolves the local registry address (defaults to `127.0.0.1:5000/navigator`). In CI, uses `$CI_REGISTRY_IMAGE`. 2. Ensures a local Docker registry container (`navigator-local-registry`) is running on port 5000 (creates one if needed). 3. Pushes prebuilt local component images (server, sandbox) to the local registry via `cluster-push-component.sh`. -4. Runs `nav gateway start --name --update-kube-config` to create or update the cluster container. +4. Runs `nav gateway start --name ` to create or update the cluster container. ### Environment Variables diff --git a/architecture/gateway-deploy-connect.md b/architecture/gateway-deploy-connect.md index 99fffb0fb..6c8836645 100644 --- a/architecture/gateway-deploy-connect.md +++ b/architecture/gateway-deploy-connect.md @@ -97,7 +97,6 @@ openshell/ clusters/ _metadata.json # ClusterMetadata JSON / - kubeconfig # rewritten kubeconfig for kubectl mtls/ # mTLS bundle (when TLS enabled) ca.crt # cluster CA certificate tls.crt # client certificate diff --git a/architecture/gateway-security.md b/architecture/gateway-security.md index 3f6ec428a..be341dd1c 100644 --- a/architecture/gateway-security.md +++ b/architecture/gateway-security.md @@ -133,7 +133,7 @@ Files are written atomically using a temp-dir -> validate -> rename strategy wit PKI provisioning occurs during `deploy_cluster_with_logs()` (`crates/navigator-bootstrap/src/lib.rs:284`). The full sequence: 1. **Cluster container launched** -- a k3s container is created via Docker with a persistent volume. -2. **Kubeconfig extracted** -- the bootstrap waits for k3s readiness and retrieves the kubeconfig. +2. **k3s readiness** -- the bootstrap waits for k3s to become ready inside the container. 3. **Extra SANs computed** -- for remote deployments, the SSH destination hostname and its resolved IP are added to the server certificate's SANs. For local deployments, the detected gateway host (if any) is added. 4. **`reconcile_pki()` called** (`crates/navigator-bootstrap/src/lib.rs:515`): 1. Wait for the `navigator` namespace to exist (created by the Helm controller). @@ -150,7 +150,6 @@ sequenceDiagram participant K8s as k3s / K8s API CLI->>Docker: Create container, wait for k3s - CLI->>Docker: Extract kubeconfig CLI->>K8s: Wait for navigator namespace CLI->>K8s: Read existing TLS secrets alt Secrets valid diff --git a/architecture/gateway-single-node.md b/architecture/gateway-single-node.md index cf999e677..1243482fb 100644 --- a/architecture/gateway-single-node.md +++ b/architecture/gateway-single-node.md @@ -8,7 +8,7 @@ This document describes how OpenShell bootstraps a single-node k3s gateway insid - Keep Docker as the only runtime dependency for provisioning and lifecycle operations. - Package the OpenShell gateway as one container image, transferred to the target host via registry pull. - Support idempotent `deploy` behavior (safe to re-run). -- Persist gateway access artifacts (kubeconfig, metadata, mTLS certs) in the local XDG config directory. +- Persist gateway access artifacts (metadata, mTLS certs) in the local XDG config directory. - Track the active gateway so most CLI commands resolve their target automatically. Out of scope: @@ -18,13 +18,12 @@ Out of scope: ## Components - `crates/navigator-cli/src/main.rs`: CLI entry point; `clap`-based command parsing. -- `crates/navigator-cli/src/run.rs`: CLI command implementations (`gateway_start`, `gateway_stop`, `gateway_destroy`, `gateway_info`, `gateway_tunnel`). +- `crates/navigator-cli/src/run.rs`: CLI command implementations (`gateway_start`, `gateway_stop`, `gateway_destroy`, `gateway_info`, `doctor_logs`). - `crates/navigator-cli/src/bootstrap.rs`: Auto-bootstrap helpers for `sandbox create` (offers to deploy a gateway when one is unreachable). - `crates/navigator-bootstrap/src/lib.rs`: Gateway lifecycle orchestration (`deploy_gateway`, `deploy_gateway_with_logs`, `gateway_handle`, `check_existing_deployment`). - `crates/navigator-bootstrap/src/docker.rs`: Docker API wrappers (network, volume, container, image operations). - `crates/navigator-bootstrap/src/image.rs`: Remote image registry pull with XOR-obfuscated distribution credentials. -- `crates/navigator-bootstrap/src/runtime.rs`: In-container operations via `docker exec` (kubeconfig readiness, health polling, stale node cleanup, deployment restart). -- `crates/navigator-bootstrap/src/kubeconfig.rs`: Kubeconfig rewriting, storage, and merge into local `~/.kube/config`. +- `crates/navigator-bootstrap/src/runtime.rs`: In-container operations via `docker exec` (health polling, stale node cleanup, deployment restart). - `crates/navigator-bootstrap/src/metadata.rs`: Gateway metadata creation, storage, and active gateway tracking. - `crates/navigator-bootstrap/src/mtls.rs`: Gateway TLS detection and CLI mTLS bundle extraction. - `crates/navigator-bootstrap/src/push.rs`: Local development image push into k3s containerd. @@ -45,10 +44,11 @@ All gateway lifecycle commands live under `openshell gateway`: |---|---| | `openshell gateway start [--name NAME] [--remote user@host] [--ssh-key PATH]` | Provision or update a gateway | | `openshell gateway stop [--name NAME] [--remote user@host]` | Stop the container (preserves state) | -| `openshell gateway destroy [--name NAME] [--remote user@host]` | Destroy container, attached volumes, kubeconfig directory, metadata, and network | -| `openshell gateway info [--name NAME]` | Show deployment details (endpoint, kubeconfig path, SSH host) | -| `openshell gateway tunnel [--name NAME] [--remote user@host] [--print-command]` | Start or print SSH tunnel for kubectl access | +| `openshell gateway destroy [--name NAME] [--remote user@host]` | Destroy container, attached volumes, metadata, and network | +| `openshell gateway info [--name NAME]` | Show deployment details (endpoint, SSH host) | | `openshell status` | Show gateway health via gRPC/HTTP | +| `openshell doctor logs [--name NAME] [--remote user@host] [--tail N]` | Fetch gateway container logs | +| `openshell doctor exec [--name NAME] [--remote user@host] -- ` | Run a command inside the gateway container | | `openshell gateway select ` | Set the active gateway | | `openshell gateway select` | Open an interactive chooser on a TTY, or list all gateways in non-interactive mode | @@ -95,8 +95,6 @@ sequenceDiagram B->>R: ensure_volume B->>R: ensure_container (privileged, k3s server) B->>R: start_container - B->>R: wait_for_kubeconfig (30 attempts, 2s apart) - B->>B: rewrite and store kubeconfig B->>R: clean_stale_nodes (kubectl delete node) B->>R: wait_for_gateway_ready (180 attempts, 2s apart) B->>R: poll for secret navigator-cli-client (90 attempts, 2s apart) @@ -104,7 +102,7 @@ sequenceDiagram B->>B: atomically store mTLS bundle B->>B: create and persist gateway metadata JSON B->>B: save_active_gateway - B-->>C: GatewayHandle (kubeconfig_path, metadata, docker client) + B-->>C: GatewayHandle (metadata, docker client) ``` ## End-State Connectivity Diagram @@ -113,29 +111,20 @@ sequenceDiagram flowchart LR subgraph WS[User workstation] NAV[navigator-cli] - KUBECTL[kubectl] - KC[stored kubeconfig ~/.config/openshell/gateways/NAME/kubeconfig] MTLS[mTLS bundle ca.crt, tls.crt, tls.key] - TUN[ssh -L 6443:127.0.0.1:6443 user@host] end subgraph HOST[Target host] DOCKER[Docker daemon] K3S[openshell-cluster-NAME single k3s container] - KAPI[Kubernetes API :6443] G8080[Gateway :port -> :30051 mTLS default 8080] SBX[Sandbox runtime] end - KC --> KUBECTL - KUBECTL --> KAPI - TUN -. remote mode .-> KAPI - NAV --> G8080 MTLS -. client cert auth .-> G8080 DOCKER --> K3S - K3S --> KAPI K3S --> G8080 K3S --> G80 K3S --> G443 @@ -183,10 +172,9 @@ For the target daemon (local or remote): - Extra host: `host.docker.internal:host-gateway`. - Port mappings: - | Container Port | Host Port | Purpose | - |---|---|---| - | 6443/tcp | 6443 | Kubernetes API | - | 30051/tcp | configurable (default 8080) | OpenShell service NodePort (mTLS) | + | Container Port | Host Port | Purpose | + |---|---|---| + | 30051/tcp | configurable (default 8080) | OpenShell service NodePort (mTLS) | - Container environment variables (see [Container Environment Variables](#container-environment-variables) below). - If the container exists with a different image ID (compared by inspecting the content-addressable ID), it is stopped, force-removed, and recreated. If the image matches, the existing container is reused. @@ -196,12 +184,9 @@ For the target daemon (local or remote): After the container starts: -1. **Poll kubeconfig**: `wait_for_kubeconfig()` runs `cat /etc/rancher/k3s/k3s.yaml` via `docker exec` up to 30 times, 2 seconds apart (60s total). Each attempt first checks that the container is still running. Validates the output contains `apiVersion:` and `clusters:`. -2. **Rewrite kubeconfig**: Replace the server address with `https://127.0.0.1:6443`. Rename `default` entries (cluster, context, user) to the gateway name. Remote mode appends `-remote` suffix (e.g., `{name}-remote`) to avoid collisions with local contexts. -3. **Store kubeconfig** at `~/.config/openshell/gateways/{name}/kubeconfig` (or `$XDG_CONFIG_HOME/openshell/gateways/{name}/kubeconfig`). -4. **Clean stale nodes**: `clean_stale_nodes()` finds `NotReady` nodes via `kubectl get nodes` and deletes them. This is needed when a container is recreated but reuses the persistent volume -- k3s registers a new node (using the container ID as hostname) while old node entries persist in etcd. Non-fatal on error; returns the count of removed nodes. -5. **Push local images** (optional, local deploy only): If `OPENSHELL_PUSH_IMAGES` is set, the comma-separated image refs are exported from the local Docker daemon as a single tar, uploaded into the container via `docker put_archive`, and imported into containerd via `ctr images import` in the `k8s.io` namespace. After import, `kubectl rollout restart deployment/navigator -n navigator` is run, followed by `kubectl rollout status --timeout=180s` to wait for completion. See `crates/navigator-bootstrap/src/push.rs`. -6. **Wait for gateway health**: `wait_for_gateway_ready()` polls the Docker HEALTHCHECK status up to 180 times, 2 seconds apart (6 min total). A background task streams container logs during this wait. Failure modes: +1. **Clean stale nodes**: `clean_stale_nodes()` finds `NotReady` nodes via `kubectl get nodes` and deletes them. This is needed when a container is recreated but reuses the persistent volume -- k3s registers a new node (using the container ID as hostname) while old node entries persist in etcd. Non-fatal on error; returns the count of removed nodes. +2. **Push local images** (optional, local deploy only): If `OPENSHELL_PUSH_IMAGES` is set, the comma-separated image refs are exported from the local Docker daemon as a single tar, uploaded into the container via `docker put_archive`, and imported into containerd via `ctr images import` in the `k8s.io` namespace. After import, `kubectl rollout restart deployment/navigator -n navigator` is run, followed by `kubectl rollout status --timeout=180s` to wait for completion. See `crates/navigator-bootstrap/src/push.rs`. +3. **Wait for gateway health**: `wait_for_gateway_ready()` polls the Docker HEALTHCHECK status up to 180 times, 2 seconds apart (6 min total). A background task streams container logs during this wait. Failure modes: - Container exits during polling: error includes recent log lines. - Container has no HEALTHCHECK instruction: fails immediately. - HEALTHCHECK reports unhealthy on final attempt: error includes recent logs. @@ -234,7 +219,7 @@ Metadata fields: Metadata location: `~/.config/openshell/gateways/{name}_metadata.json` -Note: metadata is stored at the `gateways/` level (not nested inside `{name}/` like kubeconfig and mTLS). +Note: metadata is stored at the `gateways/` level (not nested inside `{name}/` like mTLS). After deploy, the CLI calls `save_active_gateway(name)`, writing the gateway name to `~/.config/openshell/active_gateway`. Subsequent commands that don't specify `--gateway` or `OPENSHELL_GATEWAY` resolve to this active gateway. @@ -344,23 +329,6 @@ flowchart LR ## Access Model -### Kubernetes API access - -- Kubeconfig always targets `https://127.0.0.1:6443`. -- For remote gateways, the user must open an SSH local-forward tunnel: - -```bash -ssh -L 6443:127.0.0.1:6443 -N user@host -``` - -CLI helper: - -```bash -openshell gateway tunnel --name -``` - -The `--remote` flag is optional; the CLI resolves the SSH destination from stored gateway metadata. Pass `--print-command` to print the SSH command without executing it. - ### Gateway endpoint exposure - Local: `https://127.0.0.1:{port}` (or `https://{docker_host}:{port}` when `DOCKER_HOST` is a non-loopback TCP endpoint). Default port is 8080. @@ -380,8 +348,7 @@ The `--remote` flag is optional; the CLI resolves the SSH destination from store 1. Stop the container. 2. Remove the container (`force=true`). Tolerates 404. 3. Remove the volume (`force=true`). Tolerates 404. -4. Remove the stored kubeconfig file. -5. Remove the network if no containers remain attached (`cleanup_network_if_unused()`). +4. Remove the network if no containers remain attached (`cleanup_network_if_unused()`). **CLI layer** (`gateway_destroy()` in `run.rs` additionally): @@ -398,9 +365,8 @@ The `--remote` flag is optional; the CLI resolves the SSH destination from store - Error handling surfaces: - Docker API failures from inspect/create/start/remove. - SSH connection failures when creating the remote Docker client. - - Kubeconfig readiness timeout (60s) with recent container logs in the error message. - Health check timeout (6 min) with recent container logs. - - Container exit during any polling phase (kubeconfig, health, mTLS) with diagnostic information (exit code, OOM status, recent logs). + - Container exit during any polling phase (health, mTLS) with diagnostic information (exit code, OOM status, recent logs). - mTLS secret polling timeout (3 min). - Local image ref without registry prefix: clear error with build instructions rather than a failed Docker Hub pull. @@ -442,7 +408,6 @@ Environment variables that affect bootstrap behavior when set on the host: | `IMAGE_TAG` | Sets image tag (default: `"dev"`) when `OPENSHELL_CLUSTER_IMAGE` is not set | | `NAV_GATEWAY_TLS_ENABLED` | Overrides HelmChart manifest for TLS enabled check (`true`/`1`/`yes`/`false`/`0`/`no`) | | `XDG_CONFIG_HOME` | Base config directory (default: `$HOME/.config`) | -| `KUBECONFIG` | Target kubeconfig path for merge (first colon-separated path; default: `$HOME/.kube/config`) | | `DOCKER_HOST` | When `tcp://` and non-loopback, the host is added as a TLS SAN and used as the gateway endpoint | | `OPENSHELL_PUSH_IMAGES` | Comma-separated image refs to push into the gateway's containerd (local deploy only) | | `OPENSHELL_REGISTRY_HOST` | Override the distribution registry host | @@ -464,7 +429,6 @@ openshell/ gateways/ {name}_metadata.json # GatewayMetadata JSON {name}/ - kubeconfig # rewritten kubeconfig mtls/ # mTLS bundle (when TLS enabled) ca.crt tls.crt @@ -477,7 +441,6 @@ openshell/ - `crates/navigator-bootstrap/src/docker.rs` -- Docker API wrappers - `crates/navigator-bootstrap/src/image.rs` -- registry pull, XOR credentials - `crates/navigator-bootstrap/src/runtime.rs` -- exec, health polling, stale node cleanup -- `crates/navigator-bootstrap/src/kubeconfig.rs` -- kubeconfig rewrite and merge - `crates/navigator-bootstrap/src/metadata.rs` -- metadata CRUD, active gateway, SSH resolution - `crates/navigator-bootstrap/src/mtls.rs` -- TLS detection, secret extraction, atomic write - `crates/navigator-bootstrap/src/push.rs` -- local image push into k3s containerd diff --git a/architecture/system-architecture.md b/architecture/system-architecture.md index 0712ab541..fbd23fa8c 100644 --- a/architecture/system-architecture.md +++ b/architecture/system-architecture.md @@ -95,7 +95,7 @@ graph TB TUI -- "gRPC polling (mTLS)
every 2s" --> Gateway SDK -- "gRPC over HTTPS (mTLS)" --> Gateway CLI -- "HTTP CONNECT upgrade
/connect/ssh (mTLS)" --> Gateway - CLI -. "reads certs &
kubeconfig" .-> LocalConfig + CLI -. "reads mTLS certs" .-> LocalConfig %% ============================================================ %% CONNECTIONS: Gateway internals diff --git a/crates/navigator-bootstrap/Cargo.toml b/crates/navigator-bootstrap/Cargo.toml index c410fa788..d752d3cb5 100644 --- a/crates/navigator-bootstrap/Cargo.toml +++ b/crates/navigator-bootstrap/Cargo.toml @@ -18,7 +18,6 @@ miette = { workspace = true } rcgen = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } -serde_yaml = { workspace = true } tar = "0.4" tokio = { workspace = true } tracing = { workspace = true } diff --git a/crates/navigator-bootstrap/src/constants.rs b/crates/navigator-bootstrap/src/constants.rs index e174e516e..76f5d2f3f 100644 --- a/crates/navigator-bootstrap/src/constants.rs +++ b/crates/navigator-bootstrap/src/constants.rs @@ -2,6 +2,9 @@ // SPDX-License-Identifier: Apache-2.0 pub const NETWORK_NAME: &str = "openshell-cluster"; + +/// Path to the kubeconfig inside the k3s container. +/// Used by in-container kubectl operations (node cleanup, PKI reconciliation, etc.). pub const KUBECONFIG_PATH: &str = "/etc/rancher/k3s/k3s.yaml"; /// K8s secret holding the server's TLS certificate and private key. diff --git a/crates/navigator-bootstrap/src/docker.rs b/crates/navigator-bootstrap/src/docker.rs index 1cfc4c021..92d889671 100644 --- a/crates/navigator-bootstrap/src/docker.rs +++ b/crates/navigator-bootstrap/src/docker.rs @@ -21,7 +21,6 @@ use bollard::query_parameters::{ use futures::StreamExt; use miette::{IntoDiagnostic, Result, WrapErr}; use std::collections::HashMap; -use std::path::Path; const REGISTRY_NAMESPACE_DEFAULT: &str = "openshell"; @@ -230,7 +229,6 @@ pub async fn ensure_image( Ok(()) } -#[allow(clippy::too_many_arguments)] pub async fn ensure_container( docker: &Docker, name: &str, @@ -238,7 +236,6 @@ pub async fn ensure_container( extra_sans: &[String], ssh_gateway_host: Option<&str>, gateway_port: u16, - kube_port: Option, disable_tls: bool, disable_gateway_auth: bool, registry_token: Option<&str>, @@ -301,15 +298,6 @@ pub async fn ensure_container( } let mut port_bindings = HashMap::new(); - if let Some(kp) = kube_port { - port_bindings.insert( - "6443/tcp".to_string(), - Some(vec![PortBinding { - host_ip: Some("0.0.0.0".to_string()), - host_port: Some(kp.to_string()), - }]), - ); - } port_bindings.insert( "30051/tcp".to_string(), Some(vec![PortBinding { @@ -317,10 +305,7 @@ pub async fn ensure_container( host_port: Some(gateway_port.to_string()), }]), ); - let mut exposed_ports = vec!["30051/tcp".to_string()]; - if kube_port.is_some() { - exposed_ports.push("6443/tcp".to_string()); - } + let exposed_ports = vec!["30051/tcp".to_string()]; let mut host_config = HostConfig { privileged: Some(true), @@ -524,10 +509,9 @@ pub async fn check_port_conflicts( docker: &Docker, name: &str, gateway_port: u16, - kube_port: Option, ) -> Result> { let our_container = container_name(name); - let needed_ports: Vec = std::iter::once(gateway_port).chain(kube_port).collect(); + let needed_ports: Vec = vec![gateway_port]; let containers = docker .list_containers(Some( @@ -616,11 +600,7 @@ pub async fn stop_container(docker: &Docker, container_name: &str) -> Result<()> } } -pub async fn destroy_gateway_resources( - docker: &Docker, - name: &str, - kubeconfig_path: &Path, -) -> Result<()> { +pub async fn destroy_gateway_resources(docker: &Docker, name: &str) -> Result<()> { let container_name = container_name(name); let volume_name = volume_name(name); @@ -718,8 +698,6 @@ pub async fn destroy_gateway_resources( return Err(err).into_diagnostic(); } - let _ = std::fs::remove_file(kubeconfig_path); - // Force-remove the network during a full destroy. First disconnect any // stale endpoints that Docker may still report (race between container // removal and network bookkeeping), then remove the network itself. diff --git a/crates/navigator-bootstrap/src/kubeconfig.rs b/crates/navigator-bootstrap/src/kubeconfig.rs deleted file mode 100644 index 01db49bf6..000000000 --- a/crates/navigator-bootstrap/src/kubeconfig.rs +++ /dev/null @@ -1,322 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -use crate::paths::xdg_config_dir; -use miette::{IntoDiagnostic, Result, WrapErr}; -use serde::{Deserialize, Serialize}; -use std::collections::BTreeMap; -use std::path::{Path, PathBuf}; - -pub fn stored_kubeconfig_path(name: &str) -> Result { - let base = xdg_config_dir()?; - Ok(base - .join("openshell") - .join("gateways") - .join(name) - .join("kubeconfig")) -} - -pub fn print_kubeconfig(name: &str) -> Result<()> { - let path = stored_kubeconfig_path(name)?; - let contents = std::fs::read_to_string(&path) - .into_diagnostic() - .wrap_err_with(|| format!("failed to read kubeconfig at {}", path.display()))?; - print!("{contents}"); - Ok(()) -} - -pub fn update_local_kubeconfig(name: &str, target_path: &Path) -> Result<()> { - let stored_path = stored_kubeconfig_path(name)?; - let stored_contents = std::fs::read_to_string(&stored_path) - .into_diagnostic() - .wrap_err_with(|| format!("failed to read kubeconfig at {}", stored_path.display()))?; - let stored_config: Kubeconfig = serde_yaml::from_str(&stored_contents) - .into_diagnostic() - .wrap_err("failed to parse stored kubeconfig")?; - - let mut target_config = if target_path.exists() { - let contents = std::fs::read_to_string(target_path) - .into_diagnostic() - .wrap_err_with(|| format!("failed to read kubeconfig at {}", target_path.display()))?; - serde_yaml::from_str(&contents) - .into_diagnostic() - .wrap_err("failed to parse target kubeconfig")? - } else { - Kubeconfig::default() - }; - - merge_kubeconfig(&mut target_config, stored_config); - - if target_config.api_version.is_empty() { - target_config.api_version = "v1".to_string(); - } - if target_config.kind.is_empty() { - target_config.kind = "Config".to_string(); - } - - if let Some(parent) = target_path.parent() { - std::fs::create_dir_all(parent) - .into_diagnostic() - .wrap_err_with(|| format!("failed to create {}", parent.display()))?; - } - - let rendered = serde_yaml::to_string(&target_config) - .into_diagnostic() - .wrap_err("failed to serialize kubeconfig")?; - std::fs::write(target_path, rendered) - .into_diagnostic() - .wrap_err_with(|| format!("failed to write kubeconfig to {}", target_path.display()))?; - Ok(()) -} - -pub fn default_local_kubeconfig_path() -> Result { - if let Ok(paths) = std::env::var("KUBECONFIG") - && let Some(first) = paths.split(':').next() - && !first.is_empty() - { - return Ok(PathBuf::from(first)); - } - - let home = std::env::var("HOME") - .into_diagnostic() - .wrap_err("HOME is not set")?; - Ok(PathBuf::from(home).join(".kube").join("config")) -} - -pub fn store_kubeconfig(path: &Path, contents: &str) -> Result<()> { - if let Some(parent) = path.parent() { - std::fs::create_dir_all(parent) - .into_diagnostic() - .wrap_err_with(|| format!("failed to create {}", parent.display()))?; - } - std::fs::write(path, contents) - .into_diagnostic() - .wrap_err_with(|| format!("failed to write kubeconfig to {}", path.display()))?; - Ok(()) -} - -/// Rewrite kubeconfig for remote deployment. -/// -/// The kubeconfig points to `127.0.0.1:`, which works with an SSH tunnel: -/// `ssh -L :127.0.0.1:6443 user@host` -/// -/// The gateway name includes "-remote" suffix to distinguish from local gateways. -pub fn rewrite_kubeconfig_remote( - contents: &str, - gateway_name: &str, - _destination: &str, - kube_port: Option, -) -> String { - let remote_name = format!("{gateway_name}-remote"); - rewrite_kubeconfig(contents, &remote_name, kube_port) -} - -/// Rewrite the raw k3s kubeconfig for use on the host. -/// -/// When `kube_port` is `Some`, the server URL is rewritten to -/// `https://127.0.0.1:`. When `None`, the original server URL -/// is left intact (the control plane is not exposed on the host). -pub fn rewrite_kubeconfig(contents: &str, gateway_name: &str, kube_port: Option) -> String { - let mut replaced = Vec::new(); - for line in contents.lines() { - let trimmed = line.trim_start(); - if let Some(kp) = kube_port - && trimmed.starts_with("server:") - { - let indent_len = line.len() - trimmed.len(); - let indent = &line[..indent_len]; - replaced.push(format!("{indent}server: https://127.0.0.1:{kp}")); - continue; - } - // Rename default cluster/context/user to the gateway name - // Handle both "name: default" and "- name: default" (YAML list item) - if trimmed == "name: default" || trimmed == "- name: default" { - let indent_len = line.len() - trimmed.len(); - let indent = &line[..indent_len]; - let prefix = if trimmed.starts_with("- ") { "- " } else { "" }; - replaced.push(format!("{indent}{prefix}name: {gateway_name}")); - continue; - } - if trimmed == "cluster: default" || trimmed == "user: default" { - let indent_len = line.len() - trimmed.len(); - let indent = &line[..indent_len]; - let key = trimmed.split(':').next().unwrap_or("cluster"); - replaced.push(format!("{indent}{key}: {gateway_name}")); - continue; - } - if trimmed == "current-context: default" { - replaced.push(format!("current-context: {gateway_name}")); - continue; - } - replaced.push(line.to_string()); - } - - let mut output = replaced.join("\n"); - if contents.ends_with('\n') { - output.push('\n'); - } - output -} - -fn merge_kubeconfig(target: &mut Kubeconfig, incoming: Kubeconfig) { - merge_named(&mut target.clusters, incoming.clusters); - merge_named(&mut target.contexts, incoming.contexts); - merge_named(&mut target.users, incoming.users); - - if incoming.current_context.is_some() { - target.current_context = incoming.current_context; - } - if incoming.preferences.is_some() { - target.preferences = incoming.preferences; - } - - target - .extra - .extend(incoming.extra.into_iter().filter(|(k, _)| !k.is_empty())); -} - -fn merge_named(target: &mut Vec, incoming: Vec) { - for entry in incoming { - if let Some(existing) = target.iter_mut().find(|item| item.name() == entry.name()) { - *existing = entry; - } else { - target.push(entry); - } - } -} - -trait NamedEntry { - fn name(&self) -> &str; -} - -#[derive(Debug, Default, Serialize, Deserialize)] -struct Kubeconfig { - #[serde(rename = "apiVersion", default)] - api_version: String, - #[serde(default)] - kind: String, - #[serde(default)] - clusters: Vec, - #[serde(default)] - contexts: Vec, - #[serde(default)] - users: Vec, - #[serde(rename = "current-context", default)] - current_context: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - preferences: Option, - #[serde(flatten, default)] - extra: BTreeMap, -} - -#[derive(Debug, Serialize, Deserialize, Clone)] -struct NamedCluster { - name: String, - cluster: serde_yaml::Value, -} - -impl NamedEntry for NamedCluster { - fn name(&self) -> &str { - &self.name - } -} - -#[derive(Debug, Serialize, Deserialize, Clone)] -struct NamedContext { - name: String, - context: serde_yaml::Value, -} - -impl NamedEntry for NamedContext { - fn name(&self) -> &str { - &self.name - } -} - -#[derive(Debug, Serialize, Deserialize, Clone)] -struct NamedUser { - name: String, - user: serde_yaml::Value, -} - -impl NamedEntry for NamedUser { - fn name(&self) -> &str { - &self.name - } -} - -#[cfg(test)] -mod tests { - use super::rewrite_kubeconfig; - - #[test] - fn rewrite_updates_server_address_with_kube_port() { - let input = "apiVersion: v1\nclusters:\n- cluster:\n server: https://10.0.0.1:6443\n"; - let output = rewrite_kubeconfig(input, "test-cluster", Some(6443)); - assert!(output.contains("server: https://127.0.0.1:6443")); - } - - #[test] - fn rewrite_updates_server_address_custom_port() { - let input = "apiVersion: v1\nclusters:\n- cluster:\n server: https://10.0.0.1:6443\n"; - let output = rewrite_kubeconfig(input, "test-cluster", Some(7443)); - assert!(output.contains("server: https://127.0.0.1:7443")); - assert!(!output.contains("server: https://127.0.0.1:6443")); - } - - #[test] - fn rewrite_preserves_server_when_no_kube_port() { - let input = "apiVersion: v1\nclusters:\n- cluster:\n server: https://10.0.0.1:6443\n"; - let output = rewrite_kubeconfig(input, "test-cluster", None); - assert!( - output.contains("server: https://10.0.0.1:6443"), - "server address should be preserved when kube_port is None" - ); - } - - #[test] - fn rewrite_preserves_trailing_newline() { - let input = "apiVersion: v1\nserver: https://10.0.0.1\n"; - let output = rewrite_kubeconfig(input, "test-cluster", Some(6443)); - assert!(output.ends_with('\n')); - } - - #[test] - fn rewrite_renames_default_entries() { - let input = "apiVersion: v1 - clusters: - - name: default - cluster: - server: https://10.0.0.1:6443 - contexts: - - name: default - context: - cluster: default - user: default - users: - - name: default - current-context: default - "; - let output = rewrite_kubeconfig(input, "my-cluster", Some(6443)); - assert!( - output.contains("name: my-cluster"), - "should contain 'name: my-cluster'" - ); - assert!( - output.contains("cluster: my-cluster"), - "should contain 'cluster: my-cluster'" - ); - assert!( - output.contains("user: my-cluster"), - "should contain 'user: my-cluster'" - ); - assert!( - output.contains("current-context: my-cluster"), - "should contain 'current-context: my-cluster'" - ); - assert!( - !output.contains("name: default"), - "should not contain 'name: default'" - ); - } -} diff --git a/crates/navigator-bootstrap/src/lib.rs b/crates/navigator-bootstrap/src/lib.rs index 19bcf4092..f19ea9540 100644 --- a/crates/navigator-bootstrap/src/lib.rs +++ b/crates/navigator-bootstrap/src/lib.rs @@ -8,7 +8,6 @@ pub mod image; mod constants; mod docker; -mod kubeconfig; mod metadata; mod mtls; mod paths; @@ -24,19 +23,15 @@ pub(crate) static XDG_TEST_LOCK: Mutex<()> = Mutex::new(()); use bollard::Docker; use miette::{IntoDiagnostic, Result}; -use std::path::{Path, PathBuf}; use std::sync::{Arc, Mutex}; use crate::constants::{ - CLIENT_TLS_SECRET_NAME, SERVER_CLIENT_CA_SECRET_NAME, SERVER_TLS_SECRET_NAME, container_name, - volume_name, + CLIENT_TLS_SECRET_NAME, SERVER_CLIENT_CA_SECRET_NAME, SERVER_TLS_SECRET_NAME, volume_name, }; use crate::docker::{ - check_existing_gateway, check_port_conflicts, create_ssh_docker_client, - destroy_gateway_resources, ensure_container, ensure_image, ensure_network, ensure_volume, - start_container, stop_container, + check_existing_gateway, check_port_conflicts, destroy_gateway_resources, ensure_container, + ensure_image, ensure_network, ensure_volume, start_container, stop_container, }; -use crate::kubeconfig::{rewrite_kubeconfig, rewrite_kubeconfig_remote, store_kubeconfig}; use crate::metadata::{ create_gateway_metadata, create_gateway_metadata_with_host, extract_host_from_ssh_destination, local_gateway_host, resolve_ssh_hostname, @@ -45,14 +40,11 @@ use crate::mtls::store_pki_bundle; use crate::pki::generate_pki; use crate::runtime::{ clean_stale_nodes, exec_capture_with_exit, fetch_recent_logs, openshell_workload_exists, - restart_openshell_deployment, wait_for_gateway_ready, wait_for_kubeconfig, + restart_openshell_deployment, wait_for_gateway_ready, }; -pub use crate::docker::ExistingGatewayInfo; -pub use crate::kubeconfig::{ - default_local_kubeconfig_path, print_kubeconfig, stored_kubeconfig_path, - update_local_kubeconfig, -}; +pub use crate::constants::container_name; +pub use crate::docker::{ExistingGatewayInfo, create_ssh_docker_client}; pub use crate::metadata::{ GatewayMetadata, clear_active_gateway, get_gateway_metadata, list_gateways, load_active_gateway, load_gateway_metadata, load_last_sandbox, remove_gateway_metadata, @@ -111,10 +103,6 @@ pub struct DeployOptions { /// Useful in CI where `127.0.0.1` is not reachable from the test runner /// (e.g., `host.docker.internal`). pub gateway_host: Option, - /// Host port to expose the k3s Kubernetes control plane on. - /// When `None`, the control plane port (6443) is not exposed on the host, - /// allowing multiple clusters to run simultaneously without port conflicts. - pub kube_port: Option, /// Disable TLS entirely — the server listens on plaintext HTTP. pub disable_tls: bool, /// Disable gateway authentication (mTLS client certificate requirement). @@ -138,7 +126,6 @@ impl DeployOptions { remote: None, port: DEFAULT_GATEWAY_PORT, gateway_host: None, - kube_port: None, disable_tls: false, disable_gateway_auth: false, registry_token: None, @@ -167,14 +154,6 @@ impl DeployOptions { self } - /// Set the host port for the k3s Kubernetes control plane. - /// When set, the control plane is accessible via `kubectl` at this port. - #[must_use] - pub fn with_kube_port(mut self, kube_port: u16) -> Self { - self.kube_port = Some(kube_port); - self - } - /// Disable TLS entirely — the server listens on plaintext HTTP. #[must_use] pub fn with_disable_tls(mut self, disable: bool) -> Self { @@ -207,16 +186,11 @@ impl DeployOptions { #[derive(Debug, Clone)] pub struct GatewayHandle { name: String, - kubeconfig_path: PathBuf, metadata: GatewayMetadata, docker: Docker, } impl GatewayHandle { - pub fn kubeconfig_path(&self) -> &Path { - &self.kubeconfig_path - } - /// Get the gateway metadata. pub fn metadata(&self) -> &GatewayMetadata { &self.metadata @@ -232,7 +206,7 @@ impl GatewayHandle { } pub async fn destroy(&self) -> Result<()> { - destroy_gateway_resources(&self.docker, &self.name, &self.kubeconfig_path).await + destroy_gateway_resources(&self.docker, &self.name).await } } @@ -263,7 +237,6 @@ where let image_ref = options.image_ref.unwrap_or_else(default_gateway_image_ref); let port = options.port; let gateway_host = options.gateway_host; - let kube_port = options.kube_port; let disable_tls = options.disable_tls; let disable_gateway_auth = options.disable_gateway_auth; let registry_token = options.registry_token; @@ -359,7 +332,7 @@ where // bound by another container, leaving the new container with only loopback // and no default route. Detecting this up-front avoids a confusing 30s // timeout followed by a misleading "Docker networking issue" diagnostic. - let conflicts = check_port_conflicts(&target_docker, &name, port, kube_port).await?; + let conflicts = check_port_conflicts(&target_docker, &name, port).await?; if !conflicts.is_empty() { let details: Vec = conflicts .iter() @@ -389,7 +362,6 @@ where &extra_sans, ssh_gateway_host.as_deref(), port, - kube_port, disable_tls, disable_gateway_auth, registry_token.as_deref(), @@ -397,14 +369,7 @@ where ) .await?; start_container(&target_docker, &name).await?; - let raw_kubeconfig = wait_for_kubeconfig(&target_docker, &name).await?; - // Rewrite kubeconfig based on deployment mode - let rewritten = remote_opts.as_ref().map_or_else( - || rewrite_kubeconfig(&raw_kubeconfig, &name, kube_port), - |opts| rewrite_kubeconfig_remote(&raw_kubeconfig, &name, &opts.destination, kube_port), - ); - store_kubeconfig(&kubeconfig_path, &rewritten)?; // Clean up stale k3s nodes left over from previous container instances that // used the same persistent volume. Without this, pods remain scheduled on // NotReady ghost nodes and the health check will time out. @@ -420,8 +385,8 @@ where // valid; only generate fresh PKI when secrets are missing, incomplete, // malformed, or expiring within MIN_REMAINING_VALIDITY_DAYS. // - // Ordering is: kubeconfig ready -> reconcile secrets -> (if rotated and - // workload exists: rollout restart and wait) -> persist CLI-side bundle. + // Ordering is: reconcile secrets -> (if rotated and workload exists: + // rollout restart and wait) -> persist CLI-side bundle. // // We check workload presence before reconciliation. On a fresh/recreated // cluster, secrets are always newly generated and a restart is unnecessary. @@ -492,7 +457,6 @@ where &name, remote_opts.as_ref(), port, - kube_port, ssh_gateway_host.as_deref(), disable_tls, ); @@ -500,7 +464,6 @@ where Ok(GatewayHandle { name, - kubeconfig_path, metadata, docker: target_docker, }) @@ -515,14 +478,12 @@ pub async fn gateway_handle(name: &str, remote: Option<&RemoteOptions>) -> Resul Some(remote_opts) => create_ssh_docker_client(remote_opts).await?, None => Docker::connect_with_local_defaults().into_diagnostic()?, }; - let kubeconfig_path = stored_kubeconfig_path(name)?; // Try to load existing metadata, fall back to creating new metadata // with the default ports (the actual ports are only known at deploy time). let metadata = load_gateway_metadata(name) - .unwrap_or_else(|_| create_gateway_metadata(name, remote, DEFAULT_GATEWAY_PORT, None)); + .unwrap_or_else(|_| create_gateway_metadata(name, remote, DEFAULT_GATEWAY_PORT)); Ok(GatewayHandle { name: name.to_string(), - kubeconfig_path, metadata, docker, }) @@ -535,6 +496,74 @@ pub async fn ensure_gateway_image(version: &str, registry_token: Option<&str>) - Ok(image_ref) } +/// Fetch logs from the gateway Docker container. +/// +/// Connects to Docker (local or remote), retrieves logs from +/// `openshell-cluster-{name}`, and writes them to the provided writer. +/// +/// When `follow` is true, streams logs in real-time (blocks until cancelled). +/// When `lines` is `Some(n)`, returns the last `n` lines; when `None`, +/// returns all available logs. +pub async fn gateway_container_logs( + remote: Option<&RemoteOptions>, + name: &str, + lines: Option, + follow: bool, + mut writer: W, +) -> Result<()> { + use bollard::container::LogOutput; + use bollard::query_parameters::LogsOptionsBuilder; + use futures::StreamExt; + use miette::WrapErr; + + let docker = match remote { + Some(remote_opts) => create_ssh_docker_client(remote_opts).await?, + None => Docker::connect_with_local_defaults().into_diagnostic()?, + }; + + let container = container_name(name); + + let tail_value = match (follow, lines) { + (true, _) => "0".to_string(), + (false, Some(n)) => n.to_string(), + (false, None) => "all".to_string(), + }; + + let options = LogsOptionsBuilder::new() + .follow(follow) + .stdout(true) + .stderr(true) + .tail(&tail_value) + .timestamps(true) + .build(); + + let mut stream = docker.logs(&container, Some(options)); + + while let Some(item) = stream.next().await { + match item { + Ok(log) => { + let text = match log { + LogOutput::StdOut { message } + | LogOutput::StdErr { message } + | LogOutput::Console { message } => { + String::from_utf8_lossy(&message).to_string() + } + LogOutput::StdIn { .. } => continue, + }; + writer + .write_all(text.as_bytes()) + .into_diagnostic() + .wrap_err("failed to write log output")?; + } + Err(err) => { + return Err(miette::miette!("error reading container logs: {err}")); + } + } + } + + Ok(()) +} + fn default_gateway_image_ref() -> String { if let Ok(image) = std::env::var("OPENSHELL_CLUSTER_IMAGE") && !image.trim().is_empty() diff --git a/crates/navigator-bootstrap/src/metadata.rs b/crates/navigator-bootstrap/src/metadata.rs index 6d0979a62..12e1b98a1 100644 --- a/crates/navigator-bootstrap/src/metadata.rs +++ b/crates/navigator-bootstrap/src/metadata.rs @@ -7,7 +7,7 @@ use miette::{IntoDiagnostic, Result, WrapErr}; use serde::{Deserialize, Serialize}; use std::path::PathBuf; -/// Gateway metadata stored alongside the kubeconfig. +/// Gateway metadata stored alongside deployment info. #[derive(Debug, Clone, Serialize, Deserialize)] pub struct GatewayMetadata { /// The gateway name. @@ -18,11 +18,6 @@ pub struct GatewayMetadata { pub is_remote: bool, /// Host port mapped to the gateway `NodePort`. pub gateway_port: u16, - /// Host port mapped to the k3s Kubernetes control plane (6443 inside the container). - /// `None` means the control plane is not exposed on the host. - /// Old metadata files without this field are deserialized as `None`. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub kube_port: Option, /// For remote gateways, the SSH destination (e.g., `user@hostname`). #[serde(skip_serializing_if = "Option::is_none")] pub remote_host: Option, @@ -55,9 +50,8 @@ pub fn create_gateway_metadata( name: &str, remote: Option<&RemoteOptions>, port: u16, - kube_port: Option, ) -> GatewayMetadata { - create_gateway_metadata_with_host(name, remote, port, kube_port, None, false) + create_gateway_metadata_with_host(name, remote, port, None, false) } /// Create gateway metadata, optionally overriding the gateway host. @@ -73,7 +67,6 @@ pub fn create_gateway_metadata_with_host( name: &str, remote: Option<&RemoteOptions>, port: u16, - kube_port: Option, gateway_host: Option<&str>, disable_tls: bool, ) -> GatewayMetadata { @@ -108,7 +101,6 @@ pub fn create_gateway_metadata_with_host( gateway_endpoint, is_remote, gateway_port: port, - kube_port, remote_host, resolved_host, auth_mode: None, @@ -379,11 +371,10 @@ mod tests { #[test] fn local_gateway_metadata() { - let meta = create_gateway_metadata("test", None, 8080, None); + let meta = create_gateway_metadata("test", None, 8080); assert_eq!(meta.name, "test"); assert_eq!(meta.gateway_endpoint, "https://127.0.0.1:8080"); assert_eq!(meta.gateway_port, 8080); - assert!(meta.kube_port.is_none()); assert!(!meta.is_remote); assert!(meta.remote_host.is_none()); assert!(meta.resolved_host.is_none()); @@ -391,23 +382,11 @@ mod tests { #[test] fn local_gateway_metadata_custom_port() { - let meta = create_gateway_metadata("test", None, 9090, None); + let meta = create_gateway_metadata("test", None, 9090); assert_eq!(meta.gateway_endpoint, "https://127.0.0.1:9090"); assert_eq!(meta.gateway_port, 9090); } - #[test] - fn local_gateway_metadata_with_kube_port() { - let meta = create_gateway_metadata("test", None, 8080, Some(7443)); - assert_eq!(meta.kube_port, Some(7443)); - } - - #[test] - fn local_gateway_metadata_without_kube_port() { - let meta = create_gateway_metadata("test", None, 8080, None); - assert!(meta.kube_port.is_none()); - } - #[test] fn local_gateway_host_from_docker_host_tcp_service_name() { let host = local_gateway_host_from_docker_host("tcp://docker:2375"); @@ -429,7 +408,7 @@ mod tests { #[test] fn remote_gateway_metadata_has_resolved_host() { let opts = RemoteOptions::new("user@10.0.0.5"); - let meta = create_gateway_metadata("test", Some(&opts), 8080, Some(6443)); + let meta = create_gateway_metadata("test", Some(&opts), 8080); assert!(meta.is_remote); assert_eq!(meta.remote_host.as_deref(), Some("user@10.0.0.5")); // When the host is a plain IP, ssh -G should resolve it to itself @@ -439,17 +418,15 @@ mod tests { format!("https://{}:8080", meta.resolved_host.as_ref().unwrap()) ); assert_eq!(meta.gateway_port, 8080); - assert_eq!(meta.kube_port, Some(6443)); } #[test] - fn metadata_roundtrip_with_kube_port() { + fn metadata_roundtrip() { let meta = GatewayMetadata { name: "test".to_string(), gateway_endpoint: "https://10.0.0.5:8080".to_string(), is_remote: true, gateway_port: 8080, - kube_port: Some(7443), remote_host: Some("user@openshell-dev".to_string()), resolved_host: Some("10.0.0.5".to_string()), auth_mode: None, @@ -461,35 +438,11 @@ mod tests { assert_eq!(parsed.resolved_host.as_deref(), Some("10.0.0.5")); assert_eq!(parsed.gateway_endpoint, "https://10.0.0.5:8080"); assert_eq!(parsed.gateway_port, 8080); - assert_eq!(parsed.kube_port, Some(7443)); - } - - #[test] - fn metadata_roundtrip_without_kube_port() { - let meta = GatewayMetadata { - name: "test".to_string(), - gateway_endpoint: "https://10.0.0.5:8080".to_string(), - is_remote: true, - gateway_port: 8080, - kube_port: None, - remote_host: Some("user@openshell-dev".to_string()), - resolved_host: Some("10.0.0.5".to_string()), - auth_mode: None, - edge_team_domain: None, - edge_auth_url: None, - }; - let json = serde_json::to_string(&meta).unwrap(); - assert!( - !json.contains("kube_port"), - "None should be omitted from JSON" - ); - let parsed: GatewayMetadata = serde_json::from_str(&json).unwrap(); - assert!(parsed.kube_port.is_none()); } #[test] fn metadata_deserialize_without_resolved_host() { - // Existing metadata files won't have the resolved_host or kube_port fields. + // Existing metadata files won't have the resolved_host field. // Ensure backwards compatibility via serde(default). let json = r#"{ "name": "test", @@ -500,8 +453,6 @@ mod tests { }"#; let parsed: GatewayMetadata = serde_json::from_str(json).unwrap(); assert!(parsed.resolved_host.is_none()); - // kube_port should default to None when not present in JSON - assert!(parsed.kube_port.is_none()); } #[test] @@ -510,7 +461,6 @@ mod tests { "test", None, 8080, - None, Some("host.docker.internal"), false, ); @@ -525,13 +475,13 @@ mod tests { #[test] fn local_gateway_metadata_with_no_gateway_host_override() { // When gateway_host is None, behaviour matches create_gateway_metadata. - let meta = create_gateway_metadata_with_host("test", None, 8080, None, None, false); + let meta = create_gateway_metadata_with_host("test", None, 8080, None, false); assert_eq!(meta.gateway_endpoint, "https://127.0.0.1:8080"); } #[test] fn local_gateway_metadata_with_tls_disabled() { - let meta = create_gateway_metadata_with_host("test", None, 8080, None, None, true); + let meta = create_gateway_metadata_with_host("test", None, 8080, None, true); assert_eq!(meta.gateway_endpoint, "http://127.0.0.1:8080"); } @@ -541,7 +491,6 @@ mod tests { "test", None, 8080, - None, Some("host.docker.internal"), true, ); @@ -551,8 +500,7 @@ mod tests { #[test] fn remote_gateway_metadata_with_tls_disabled() { let opts = RemoteOptions::new("user@10.0.0.5"); - let meta = - create_gateway_metadata_with_host("test", Some(&opts), 8080, Some(6443), None, true); + let meta = create_gateway_metadata_with_host("test", Some(&opts), 8080, None, true); assert!(meta.is_remote); assert!(meta.gateway_endpoint.starts_with("http://")); assert!(!meta.gateway_endpoint.starts_with("https://")); diff --git a/crates/navigator-bootstrap/src/runtime.rs b/crates/navigator-bootstrap/src/runtime.rs index 54b427430..0bc893d84 100644 --- a/crates/navigator-bootstrap/src/runtime.rs +++ b/crates/navigator-bootstrap/src/runtime.rs @@ -13,44 +13,6 @@ use std::collections::VecDeque; use std::time::Duration; use tokio::sync::mpsc::{UnboundedReceiver, UnboundedSender, unbounded_channel}; -pub async fn wait_for_kubeconfig(docker: &Docker, name: &str) -> Result { - let container_name = container_name(name); - let attempts = 30; - for attempt in 0..attempts { - // Check if the container is still running before trying to exec into it - if let Err(status_err) = - crate::docker::check_container_running(docker, &container_name).await - { - let logs = fetch_recent_logs(docker, &container_name, 20).await; - return Err(miette::miette!( - "gateway container is not running while waiting for kubeconfig: {status_err}\n{logs}" - )); - } - - match exec_capture( - docker, - &container_name, - vec!["cat".to_string(), KUBECONFIG_PATH.to_string()], - ) - .await - { - Ok(output) if is_valid_kubeconfig(&output) => return Ok(output), - Ok(_) => {} - Err(err) if attempt + 1 < attempts => { - let _ = err; - } - Err(err) => { - let logs = fetch_recent_logs(docker, &container_name, 20).await; - return Err(err.wrap_err(format!("failed waiting for kubeconfig\n{logs}"))); - } - } - tokio::time::sleep(Duration::from_secs(2)).await; - } - - let logs = fetch_recent_logs(docker, &container_name, 20).await; - Err(miette::miette!("timed out waiting for kubeconfig\n{logs}")) -} - /// Log markers emitted by the entrypoint and health-check scripts when DNS /// resolution fails inside the container. Detecting these early lets us /// short-circuit the 6-minute polling loop and surface a clear diagnosis. @@ -542,19 +504,6 @@ async fn detect_openshell_workload_kind(docker: &Docker, container_name: &str) - )) } -fn is_valid_kubeconfig(output: &str) -> bool { - output.contains("apiVersion:") && output.contains("clusters:") -} - -pub async fn exec_capture( - docker: &Docker, - container_name: &str, - cmd: Vec, -) -> Result { - let (output, _status) = exec_capture_with_exit(docker, container_name, cmd).await?; - Ok(output) -} - pub async fn exec_capture_with_exit( docker: &Docker, container_name: &str, diff --git a/crates/navigator-cli/Cargo.toml b/crates/navigator-cli/Cargo.toml index c28676105..98362cba4 100644 --- a/crates/navigator-cli/Cargo.toml +++ b/crates/navigator-cli/Cargo.toml @@ -22,7 +22,6 @@ navigator-providers = { path = "../navigator-providers" } navigator-tui = { path = "../navigator-tui" } serde = { workspace = true } serde_json = { workspace = true } -serde_yaml = { workspace = true } prost-types = { workspace = true } # Async runtime diff --git a/crates/navigator-cli/src/doctor_llm_prompt.md b/crates/navigator-cli/src/doctor_llm_prompt.md new file mode 100644 index 000000000..2135bb70d --- /dev/null +++ b/crates/navigator-cli/src/doctor_llm_prompt.md @@ -0,0 +1,441 @@ + + + +# Debug OpenShell Gateway + +You are diagnosing an OpenShell gateway cluster. Use **only** `openshell` CLI commands (`openshell status`, `openshell doctor logs`, `openshell doctor exec`) to inspect and fix the cluster. Do **not** use raw `docker` or `kubectl` commands directly — always go through the `openshell doctor` interface. Run diagnostics automatically through the steps below in order. Stop and report findings as soon as a root cause is identified. + +## Tools Available + +```bash +# Quick connectivity check (run this first) +openshell status + +# Fetch container logs (latest 100 lines) +openshell doctor logs --lines 100 + +# Stream live logs +openshell doctor logs --tail + +# Run any command inside the gateway container (KUBECONFIG is pre-configured) +openshell doctor exec -- kubectl get pods -A +openshell doctor exec -- kubectl -n openshell logs statefulset/openshell --tail=100 +openshell doctor exec -- sh + +# Target a specific gateway by name +openshell doctor logs --name +openshell doctor exec --name -- kubectl get pods -A + +# Target a remote gateway +openshell doctor logs --name --remote +openshell doctor exec --name --remote -- kubectl get pods -A +``` + +For operations that `doctor exec` cannot perform (container inspection, host-level Docker commands), use Docker directly: + +```bash +# Local +docker ps -a --filter name=openshell-cluster- +docker inspect openshell-cluster- +docker port openshell-cluster- + +# Remote +ssh docker ps -a --filter name=openshell-cluster- +``` + +## Overview + +`openshell gateway start` creates a Docker container running k3s with the OpenShell server and Envoy Gateway deployed via Helm. The deployment stages, in order, are: + +1. **Pre-deploy check**: `openshell gateway start` in interactive mode prompts to **reuse** (keep volume, clean stale nodes) or **recreate** (destroy everything, fresh start). `mise run cluster` always recreates before deploy. +2. Ensure cluster image is available (local build or remote pull) +3. Create Docker network (`openshell-cluster`) and volume (`openshell-cluster-{name}`) +4. Create and start a privileged Docker container (`openshell-cluster-{name}`) +5. Wait for k3s to generate kubeconfig (up to 60s) +6. **Clean stale nodes**: Remove any `NotReady` k3s nodes left over from previous container instances that reused the same persistent volume +7. **Prepare local images** (if `OPENSHELL_PUSH_IMAGES` is set): In `internal` registry mode, bootstrap waits for the in-cluster registry and pushes tagged images there. In `external` mode, bootstrap uses legacy `ctr -n k8s.io images import` push-mode behavior. +8. **Reconcile TLS PKI**: Load existing TLS secrets from the cluster; if missing, incomplete, or malformed, generate fresh PKI (CA + server + client certs). Apply secrets to cluster. If rotation happened and the OpenShell workload is already running, rollout restart and wait for completion (failed rollout aborts deploy). +9. **Store CLI mTLS credentials**: Persist client cert/key/CA locally for CLI authentication. +10. Wait for cluster health checks to pass (up to 6 min): + - k3s API server readiness (`/readyz`) + - `openshell` statefulset ready in `openshell` namespace + - TLS secrets `openshell-server-tls` and `openshell-client-tls` exist in `openshell` namespace + +For local deploys, metadata endpoint selection depends on Docker connectivity: + +- default local Docker socket (`unix:///var/run/docker.sock`): `https://127.0.0.1:{port}` (default port 8080) +- TCP Docker daemon (`DOCKER_HOST=tcp://:`): `https://:{port}` for non-loopback hosts + +The host port is configurable via `--port` on `openshell gateway start` (default 8080) and is stored in `ClusterMetadata.gateway_port`. + +The TCP host is also added as an extra gateway TLS SAN so mTLS hostname validation succeeds. + +The default cluster name is `openshell`. The container is `openshell-cluster-{name}`. + +## Workflow + +### Determine Context + +Before running commands, establish: + +1. **Cluster name**: Default is `openshell`, giving container name `openshell-cluster-openshell` +2. **Remote or local**: If the user deployed with `--remote `, pass `--remote ` to all `openshell doctor` commands +3. **Config directory**: `~/.config/openshell/gateways/{name}/` + +### Step 0: Quick Connectivity Check + +Run `openshell status` first. This immediately reveals: +- Which gateway and endpoint the CLI is targeting +- Whether the CLI can reach the server (mTLS handshake success/failure) +- The server version if connected + +Common errors at this stage: +- **`tls handshake eof`**: The server isn't running or mTLS credentials are missing/mismatched +- **`connection refused`**: The container isn't running or port mapping is broken +- **`No gateway configured`**: No gateway has been deployed yet + +### Step 1: Check Docker Container State + +First, determine if the container exists and its state: + +```bash +docker ps -a --filter name=openshell-cluster- --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Ports}}' +``` + +If the container does not exist: + +```bash +# Check if the image is available +docker images 'openshell/cluster*' --format 'table {{.Repository}}\t{{.Tag}}\t{{.Size}}' +``` + +If the image is missing, re-deploy so bootstrap can pull the published cluster image (or set `OPENSHELL_CLUSTER_IMAGE` explicitly). + +If the container exists but is not running, inspect it: + +```bash +docker inspect openshell-cluster- --format '{{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} error={{.State.Error}}' +``` + +- **OOMKilled=true**: The host doesn't have enough memory. +- **ExitCode != 0**: k3s crashed. Proceed to Step 2 for logs. + +### Step 2: Check Container Logs + +Get recent container logs to identify startup failures: + +```bash +openshell doctor logs --lines 100 +``` + +Look for: + +- DNS resolution failures in the entrypoint script +- k3s startup errors (certificate issues, port binding failures) +- Manifest copy errors from `/opt/openshell/manifests/` +- `iptables` or `cgroup` errors (privilege/capability issues) + +### Step 3: Check k3s Cluster Health + +Verify k3s itself is functional: + +```bash +# API server readiness +openshell doctor exec -- kubectl get --raw="/readyz" + +# Node status +openshell doctor exec -- kubectl get nodes -o wide + +# All pods +openshell doctor exec -- kubectl get pods -A -o wide +``` + +If `/readyz` fails, k3s is still starting or has crashed. Check container logs (Step 2). + +If pods are in `CrashLoopBackOff`, `ImagePullBackOff`, or `Pending`, investigate those pods specifically. + +Also check for node pressure conditions that cause the kubelet to evict pods and reject scheduling: + +```bash +# Check node conditions (DiskPressure, MemoryPressure, PIDPressure) +openshell doctor exec -- kubectl get nodes -o jsonpath="{range .items[*]}{.metadata.name}{range .status.conditions[*]} {.type}={.status}{end}{\"\n\"}{end}" + +# Check disk usage inside the container +openshell doctor exec -- df -h / + +# Check memory usage +openshell doctor exec -- free -h +``` + +If any pressure condition is `True`, pods will be evicted and new ones rejected. The bootstrap detects `HEALTHCHECK_NODE_PRESSURE` markers from the health-check script and aborts early with a clear diagnosis. To fix: free disk/memory on the host, then recreate the gateway. + +### Step 4: Check OpenShell Server StatefulSet + +The OpenShell server is deployed via a HelmChart CR as a StatefulSet named `openshell` in the `openshell` namespace. Check its status: + +```bash +# StatefulSet status +openshell doctor exec -- kubectl -n openshell get statefulset/openshell -o wide + +# OpenShell pod logs +openshell doctor exec -- kubectl -n openshell logs statefulset/openshell --tail=100 + +# Describe statefulset for events +openshell doctor exec -- kubectl -n openshell describe statefulset/openshell + +# Helm install job logs (the job that installs the OpenShell chart) +openshell doctor exec -- kubectl -n kube-system logs -l job-name=helm-install-openshell --tail=200 +``` + +Common issues: + +- **Replicas 0/0**: The StatefulSet has been scaled to zero — no pods are running. This can happen after a failed deploy, manual scale-down, or Helm values misconfiguration. Fix: `openshell doctor exec -- kubectl -n openshell scale statefulset openshell --replicas=1` +- **ImagePullBackOff**: The component image failed to pull. In `internal` mode, verify internal registry readiness and pushed image tags (Step 6). In `external` mode, check `/etc/rancher/k3s/registries.yaml` credentials/endpoints and DNS (Step 8). Default external registry is `ghcr.io/nvidia/openshell/`. Ensure a valid `--registry-token` (or `OPENSHELL_REGISTRY_TOKEN`) was provided during deploy. +- **CrashLoopBackOff**: The server is crashing. Check pod logs for the actual error. +- **Pending**: Insufficient resources or scheduling constraints. + +### Step 5: Check Networking + +The OpenShell server is exposed via a NodePort service on port `30051`: + +```bash +# Service status +openshell doctor exec -- kubectl -n openshell get service/openshell + +# Check port bindings on the host +docker port openshell-cluster- +``` + +Expected port: `30051/tcp` (mapped to configurable host port, default 8080; set via `--port` on deploy). + +If ports are missing or conflicting, another process may be using them. Check with: + +```bash +# On the host (or remote host) +ss -tlnp | grep -E ':8080\s' +``` + +### Step 6: Check Image Availability + +Component images (server, sandbox, pki-job) can reach kubelet via two paths: + +**Local/external pull mode** (default local via `mise run cluster`): Local images are tagged to the configured local registry base (default `127.0.0.1:5000/openshell/*`), pushed to that registry, and pulled by k3s via `registries.yaml` mirror endpoint (typically `host.docker.internal:5000`). The `cluster` task pushes prebuilt local tags (`openshell/*:dev`, falling back to `localhost:5000/openshell/*:dev` or `127.0.0.1:5000/openshell/*:dev`). + +```bash +# Verify image refs currently used by openshell deployment +openshell doctor exec -- kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec.containers[*].image}" + +# Verify registry mirror/auth endpoint configuration +openshell doctor exec -- cat /etc/rancher/k3s/registries.yaml +``` + +**Legacy push mode**: Images are imported into the k3s containerd `k8s.io` namespace. + +```bash +# Check if images were imported into containerd (k3s default namespace is k8s.io) +openshell doctor exec -- ctr -a /run/k3s/containerd/containerd.sock images ls | grep openshell +``` + +If images are missing, re-import with: + +```bash +docker save | docker exec -i openshell-cluster- ctr -a /run/k3s/containerd/containerd.sock images import - +``` + +**External pull mode** (remote deploy, or local with `OPENSHELL_REGISTRY_HOST`/`IMAGE_REPO_BASE` pointing at a non-local registry): Images are pulled from an external registry at runtime. The entrypoint generates `/etc/rancher/k3s/registries.yaml`. + +```bash +# Verify registries.yaml exists and has credentials +openshell doctor exec -- cat /etc/rancher/k3s/registries.yaml + +# Test pulling an image manually from inside the cluster +openshell doctor exec -- crictl pull ghcr.io/nvidia/openshell/gateway:latest +``` + +If `registries.yaml` is missing or has wrong values, verify env wiring (`OPENSHELL_REGISTRY_HOST`, `OPENSHELL_REGISTRY_INSECURE`, username/password for authenticated registries). + +### Step 7: Check mTLS / PKI + +TLS certificates are generated by the `navigator-bootstrap` crate (using `rcgen`) and stored as K8s secrets before the Helm release installs. There is no PKI job or cert-manager — certificates are applied directly via `kubectl apply`. + +```bash +# Check if the three TLS secrets exist +openshell doctor exec -- kubectl -n openshell get secret openshell-server-tls openshell-server-client-ca openshell-client-tls + +# Inspect server cert expiry (if openssl is available in the container) +openshell doctor exec -- sh -c 'kubectl -n openshell get secret openshell-server-tls -o jsonpath="{.data.tls\.crt}" | base64 -d | openssl x509 -noout -dates 2>/dev/null || echo "openssl not available"' + +# Check if CLI-side mTLS files exist locally +ls -la ~/.config/openshell/gateways//mtls/ +``` + +On redeploy, bootstrap reuses existing secrets if they are valid PEM. If secrets are missing or malformed, fresh PKI is generated and the OpenShell workload is automatically restarted. If the rollout restart fails after rotation, the deploy aborts and CLI-side certs are not updated. Certificates use rcgen defaults (effectively never expire). + +If the local mTLS files are missing but the secrets exist in the cluster, you can extract them manually: + +```bash +mkdir -p ~/.config/openshell/gateways//mtls +openshell doctor exec -- kubectl -n openshell get secret openshell-client-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > ~/.config/openshell/gateways//mtls/ca.crt +openshell doctor exec -- kubectl -n openshell get secret openshell-client-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways//mtls/tls.crt +openshell doctor exec -- kubectl -n openshell get secret openshell-client-tls -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways//mtls/tls.key +``` + +Common mTLS issues: +- **Secrets missing**: The `openshell` namespace may not have been created yet (Helm controller race). Bootstrap waits up to 2 minutes for the namespace. +- **mTLS mismatch after manual secret deletion**: Delete all three secrets and redeploy — bootstrap will regenerate and restart the workload. +- **CLI can't connect after redeploy**: Check that `~/.config/openshell/gateways//mtls/` contains `ca.crt`, `tls.crt`, `tls.key` and that they were updated at deploy time. +- **Local mTLS files missing**: The gateway was deployed but CLI credentials weren't persisted (e.g., interrupted deploy). Extract from the cluster secret as shown above. + +### Step 8: Check Kubernetes Events + +Events catch scheduling failures, image pull errors, and resource issues: + +```bash +openshell doctor exec -- kubectl get events -A --sort-by=.lastTimestamp | tail -n 50 +``` + +Look for: + +- `FailedScheduling` — resource constraints +- `ImagePullBackOff` / `ErrImagePull` — registry auth failure or DNS issue (check `/etc/rancher/k3s/registries.yaml`) +- `CrashLoopBackOff` — application crashes +- `OOMKilled` — memory limits too low +- `FailedMount` — volume issues + +### Step 9: Check DNS Resolution + +DNS misconfiguration is a common root cause, especially on remote/Linux hosts: + +```bash +# Check the resolv.conf k3s is using +openshell doctor exec -- cat /etc/rancher/k3s/resolv.conf + +# Test DNS resolution from inside the container +openshell doctor exec -- sh -c 'nslookup google.com || wget -q -O /dev/null http://google.com && echo "network ok" || echo "network unreachable"' +``` + +Check the entrypoint's DNS decision in the container logs: + +```bash +openshell doctor logs --lines 20 +``` + +The entrypoint script selects DNS resolvers in this priority: + +1. Viable nameservers from `/etc/resolv.conf` (not loopback/link-local) +2. Docker `ExtServers` from `/etc/resolv.conf` comments +3. Host gateway IP (Docker Desktop only, `192.168.*`) +4. Fallback to `8.8.8.8` / `8.8.4.4` + +If DNS is broken, all image pulls from the distribution registry will fail, as will pods that need external network access. + +## Common Failure Patterns + +| Symptom | Likely Cause | Fix | +|---------|-------------|-----| +| `tls handshake eof` from `openshell status` | Server not running or mTLS credentials missing/mismatched | Check StatefulSet replicas (Step 4) and mTLS files (Step 7) | +| StatefulSet `0/0` replicas | StatefulSet scaled to zero (failed deploy, manual scale-down, or Helm misconfiguration) | `openshell doctor exec -- kubectl -n openshell scale statefulset openshell --replicas=1` | +| Local mTLS files missing | Deploy was interrupted before credentials were persisted | Extract from cluster secret `openshell-client-tls` (Step 7) | +| Container not found | Image not built | `mise run docker:build:cluster` (local) or re-deploy (remote) | +| Container exited, OOMKilled | Insufficient memory | Increase host memory or reduce workload | +| Container exited, non-zero exit | k3s crash, port conflict, privilege issue | Check `openshell doctor logs` and `docker inspect` for details | +| `/readyz` fails | k3s still starting or crashed | Wait longer or check container logs for k3s errors | +| OpenShell pods `Pending` | Insufficient CPU/memory for scheduling, or PVC not bound | Check `openshell doctor exec -- kubectl describe pod` and `kubectl get pvc -n openshell` | +| OpenShell pods `CrashLoopBackOff` | Server application error | Check `openshell doctor exec -- kubectl -n openshell logs statefulset/openshell` | +| OpenShell pods `ImagePullBackOff` (push mode) | Images not imported or wrong containerd namespace | Check `k3s ctr -n k8s.io images ls` for component images (Step 6) | +| OpenShell pods `ImagePullBackOff` (pull mode) | Registry auth or DNS issue | Check `/etc/rancher/k3s/registries.yaml` credentials and DNS (Step 8) | +| Image import fails | Corrupt tar stream or containerd not ready | Retry after k3s is fully started; check container logs | +| Push mode images not found by kubelet | Imported into wrong containerd namespace | Must use `k3s ctr -n k8s.io images import`, not `k3s ctr images import` | +| mTLS secrets missing | Bootstrap couldn't apply secrets | Check deploy logs and verify `openshell` namespace exists (Step 7) | +| mTLS mismatch after redeploy | PKI rotated but workload not restarted | Check TLS secrets exist and openshell pod restarted after cert rotation (Step 7) | +| Helm install job failed | Chart values error or dependency issue | Check `helm-install-openshell` job logs in `kube-system` | +| Architecture mismatch (remote) | Built on arm64, deploying to amd64 | Cross-build the image for the target architecture | +| SSH connection failed (remote) | SSH key/host/Docker issues | Test `ssh docker ps` manually | +| Port conflict | Another service on gateway host port (default 8080) | Stop conflicting service or use `--port` on `openshell gateway start` | +| gRPC connect refused to `127.0.0.1:443` in CI | Docker daemon is remote but metadata points to loopback | Verify metadata endpoint host matches `DOCKER_HOST` | +| DNS failures inside container | Entrypoint DNS detection failed | Check `/etc/rancher/k3s/resolv.conf` and container startup logs | +| Node DiskPressure / MemoryPressure / PIDPressure | Insufficient disk, memory, or PIDs on host | Free disk (`docker system prune -a --volumes`), increase memory, or expand host resources | +| Pods evicted with "The node had condition: [DiskPressure]" | Host disk full, kubelet evicting pods | Free disk space on host, then `openshell gateway destroy && openshell gateway start` | +| `metrics-server` errors in logs | Normal k3s noise, not the root cause | These errors are benign — look for the actual failing health check component | +| Stale NotReady nodes from previous deploys | Volume reused across container recreations | Deploy flow auto-cleans stale nodes; if it still fails, manually delete NotReady nodes or choose "Recreate" when prompted | +| gRPC `UNIMPLEMENTED` for newer RPCs in push mode | Helm values still point at older pulled images | Verify rendered `openshell-helmchart.yaml` uses expected push refs | + +## Remote Cluster Debugging + +For clusters deployed with `--remote `, pass `--remote ` to all `openshell doctor` commands: + +```bash +openshell doctor logs --name --remote +openshell doctor exec --name --remote -- kubectl get pods -A +``` + +For direct Docker access on the remote host: + +```bash +ssh docker ps -a +ssh docker logs openshell-cluster- +ssh docker exec openshell-cluster- sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get pods -A' +``` + +## Full Diagnostic Dump + +Run all diagnostics at once for a comprehensive report: + +```bash +NAME="openshell" # cluster name, adjust if needed + +echo "=== Connectivity Check ===" +openshell status + +echo "=== Container State ===" +docker ps -a --filter "name=openshell-cluster-${NAME}" --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Ports}}' +docker inspect "openshell-cluster-${NAME}" --format '{{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} error={{.State.Error}}' 2>/dev/null + +echo "=== Container Logs (last 50 lines) ===" +openshell doctor logs --name "$NAME" --lines 50 + +echo "=== k3s Readiness ===" +openshell doctor exec --name "$NAME" -- kubectl get --raw='/readyz' + +echo "=== Nodes ===" +openshell doctor exec --name "$NAME" -- kubectl get nodes -o wide + +echo "=== Node Conditions ===" +openshell doctor exec --name "$NAME" -- kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{range .status.conditions[*]} {.type}={.status}{end}{"\n"}{end}' + +echo "=== Disk Usage ===" +openshell doctor exec --name "$NAME" -- df -h / + +echo "=== All Pods ===" +openshell doctor exec --name "$NAME" -- kubectl get pods -A -o wide + +echo "=== Failing Pods ===" +openshell doctor exec --name "$NAME" -- kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded + +echo "=== OpenShell StatefulSet ===" +openshell doctor exec --name "$NAME" -- kubectl -n openshell get statefulset/openshell -o wide + +echo "=== OpenShell Service ===" +openshell doctor exec --name "$NAME" -- kubectl -n openshell get service/openshell + +echo "=== TLS Secrets ===" +openshell doctor exec --name "$NAME" -- kubectl -n openshell get secret openshell-server-tls openshell-server-client-ca openshell-client-tls + +echo "=== Recent Events ===" +openshell doctor exec --name "$NAME" -- kubectl get events -A --sort-by=.lastTimestamp | tail -n 50 + +echo "=== Helm Install OpenShell Logs ===" +openshell doctor exec --name "$NAME" -- kubectl -n kube-system logs -l job-name=helm-install-openshell --tail=100 + +echo "=== Registry Configuration ===" +openshell doctor exec --name "$NAME" -- cat /etc/rancher/k3s/registries.yaml + +echo "=== DNS Configuration ===" +openshell doctor exec --name "$NAME" -- cat /etc/rancher/k3s/resolv.conf + +echo "=== Port Bindings ===" +docker port "openshell-cluster-${NAME}" + +echo "=== Local mTLS Credentials ===" +ls -la ~/.config/openshell/gateways/${NAME}/mtls/ 2>/dev/null || echo "mTLS directory not found" +``` diff --git a/crates/navigator-cli/src/main.rs b/crates/navigator-cli/src/main.rs index 1b1b6dc62..d7647929c 100644 --- a/crates/navigator-cli/src/main.rs +++ b/crates/navigator-cli/src/main.rs @@ -170,6 +170,7 @@ const HELP_TEMPLATE: &str = "\ gateway: Manage the gateway lifecycle status: Show gateway status and information inference: Manage inference configuration + doctor: Diagnose gateway issues \x1b[1mADDITIONAL COMMANDS\x1b[0m term: Launch the OpenShell interactive TUI @@ -276,6 +277,24 @@ const INFERENCE_EXAMPLES: &str = "\x1b[1mEXAMPLES\x1b[0m $ openshell inference update --model gpt-4-turbo "; +const DOCTOR_HELP: &str = "\x1b[1mALIAS\x1b[0m + dr + +\x1b[1mEXAMPLES\x1b[0m + $ openshell doctor logs --lines 100 + $ openshell doctor exec -- kubectl get pods -A + $ openshell doctor llm-help + +\x1b[1mAI AGENT USAGE\x1b[0m + If you are a coding agent (LLM) diagnosing a gateway issue, run: + + openshell doctor llm-help + + This prints a detailed diagnostic prompt with step-by-step instructions + for debugging gateway clusters using `openshell doctor logs` and + `openshell doctor exec`. +"; + /// `OpenShell` CLI - agent execution and management. #[derive(Parser, Debug)] #[command(name = "openshell")] @@ -405,6 +424,20 @@ enum Commands { command: Option, }, + // =================================================================== + // DIAGNOSTIC COMMANDS + // =================================================================== + /// Diagnose gateway issues. + /// + /// Inspect logs, run commands inside the gateway container, and get + /// AI-assisted debugging guidance. If you are a coding agent, run + /// `openshell doctor llm-help` for a full diagnostic prompt. + #[command(visible_alias = "dr", hide = true, after_help = DOCTOR_HELP, help_template = SUBCOMMAND_HELP_TEMPLATE)] + Doctor { + #[command(subcommand)] + command: Option, + }, + // =================================================================== // ADDITIONAL COMMANDS // =================================================================== @@ -673,14 +706,6 @@ enum GatewayCommands { #[arg(long, default_value = "openshell", env = "OPENSHELL_GATEWAY")] name: String, - /// Write stored kubeconfig into local kubeconfig. - #[arg(long)] - update_kube_config: bool, - - /// Print stored kubeconfig to stdout. - #[arg(long)] - get_kubeconfig: bool, - /// SSH destination for remote deployment (e.g., user@hostname). #[arg(long)] remote: Option, @@ -702,13 +727,6 @@ enum GatewayCommands { #[arg(long)] gateway_host: Option, - /// Expose the Kubernetes control plane on a host port for kubectl access. - /// Pass without a value to auto-select a free port, or pass a specific - /// port number. When omitted entirely, the control plane is not exposed, - /// allowing multiple clusters to coexist without port conflicts. - #[arg(long, num_args = 0..=1, default_missing_value = "0")] - kube_port: Option, - /// Destroy and recreate the gateway from scratch if one already exists. /// /// Without this flag, an interactive prompt asks what to do; in @@ -905,6 +923,78 @@ enum InferenceCommands { }, } +// ----------------------------------------------------------------------- +// Doctor (diagnostic) commands +// ----------------------------------------------------------------------- + +#[derive(Subcommand, Debug)] +enum DoctorCommands { + /// Fetch logs from the gateway container. + #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] + Logs { + /// Gateway name (defaults to active gateway). + #[arg(long, env = "OPENSHELL_GATEWAY")] + name: Option, + + /// Number of log lines to return (default: all). + #[arg(short, long)] + lines: Option, + + /// Stream live logs (follow mode). + #[arg(long)] + tail: bool, + + /// Override SSH destination for remote gateways. + #[arg(long)] + remote: Option, + + /// Path to SSH private key for remote gateways. + #[arg(long, value_hint = ValueHint::FilePath)] + ssh_key: Option, + }, + + /// Run a command inside the gateway container. + /// + /// Launches an interactive `docker exec` session in the gateway's k3s + /// container with KUBECONFIG pre-configured. When the gateway is remote, + /// the session is tunnelled over SSH automatically. + /// + /// Examples: + /// openshell doctor exec -- kubectl get pods -A + /// openshell doctor exec -- k9s + /// openshell doctor exec -- sh + #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] + Exec { + /// Gateway name (defaults to active gateway). + #[arg(long, env = "OPENSHELL_GATEWAY")] + name: Option, + + /// Override SSH destination for remote gateways. + #[arg(long)] + remote: Option, + + /// Path to SSH private key for remote gateways. + #[arg(long, value_hint = ValueHint::FilePath)] + ssh_key: Option, + + /// Command and arguments to run inside the container. + #[arg(trailing_var_arg = true, required = true)] + command: Vec, + }, + + /// Print a diagnostic prompt for AI-assisted gateway debugging. + /// + /// Outputs a system prompt that a coding agent can use to autonomously + /// diagnose gateway issues using `openshell doctor logs` and + /// `openshell doctor exec`. + /// + /// Examples: + /// openshell doctor llm-help + /// openshell doctor llm-help | pbcopy + #[command(help_template = LEAF_HELP_TEMPLATE)] + LlmHelp, +} + #[derive(Subcommand, Debug)] enum SandboxCommands { /// Create a sandbox. @@ -1238,13 +1328,10 @@ async fn main() -> Result<()> { }) => match command { GatewayCommands::Start { name, - update_kube_config, - get_kubeconfig, remote, ssh_key, port, gateway_host, - kube_port, recreate, plaintext, disable_gateway_auth, @@ -1253,13 +1340,10 @@ async fn main() -> Result<()> { } => { run::gateway_admin_deploy( &name, - update_kube_config, - get_kubeconfig, remote.as_deref(), ssh_key.as_deref(), port, gateway_host.as_deref(), - kube_port, recreate, plaintext, disable_gateway_auth, @@ -1316,23 +1400,46 @@ async fn main() -> Result<()> { .unwrap_or_else(|| "openshell".to_string()); run::gateway_admin_info(&name)?; } - GatewayCommands::Tunnel { + }, + + // ----------------------------------------------------------- + // Doctor (diagnostic) commands + // ----------------------------------------------------------- + Some(Commands::Doctor { command: Some(command) }) => match command { + DoctorCommands::Logs { name, + lines, + tail, remote, ssh_key, - print_command, } => { let name = name .or_else(|| resolve_gateway_name(&cli.gateway)) .unwrap_or_else(|| "openshell".to_string()); - run::gateway_admin_tunnel( - &name, - remote.as_deref(), - ssh_key.as_deref(), - print_command, - )?; + run::doctor_logs(&name, lines, tail, remote.as_deref(), ssh_key.as_deref()).await?; + } + DoctorCommands::Exec { + name, + remote, + ssh_key, + command, + } => { + let name = name + .or_else(|| resolve_gateway_name(&cli.gateway)) + .unwrap_or_else(|| "openshell".to_string()); + run::doctor_exec(&name, remote.as_deref(), ssh_key.as_deref(), &command)?; + } + DoctorCommands::LlmHelp => { + run::doctor_llm()?; } }, + Some(Commands::Doctor { command: None }) => { + Cli::command() + .find_subcommand_mut("doctor") + .expect("doctor subcommand exists") + .print_help() + .expect("Failed to print help"); + } // ----------------------------------------------------------- // Top-level status @@ -1988,7 +2095,6 @@ mod tests { gateway_endpoint: endpoint.to_string(), is_remote: true, gateway_port: 0, - kube_port: None, remote_host: None, resolved_host: None, auth_mode: Some("cloudflare_jwt".to_string()), diff --git a/crates/navigator-cli/src/run.rs b/crates/navigator-cli/src/run.rs index 6f72a861e..a9a57ae2a 100644 --- a/crates/navigator-cli/src/run.rs +++ b/crates/navigator-cli/src/run.rs @@ -16,10 +16,9 @@ use hyper_util::{client::legacy::Client, rt::TokioExecutor}; use indicatif::{MultiProgress, ProgressBar, ProgressStyle}; use miette::{IntoDiagnostic, Result, WrapErr}; use navigator_bootstrap::{ - DeployOptions, GatewayMetadata, RemoteOptions, clear_active_gateway, - default_local_kubeconfig_path, get_gateway_metadata, list_gateways, load_active_gateway, - print_kubeconfig, remove_gateway_metadata, save_active_gateway, save_last_sandbox, - store_gateway_metadata, update_local_kubeconfig, + DeployOptions, GatewayMetadata, RemoteOptions, clear_active_gateway, container_name, + get_gateway_metadata, list_gateways, load_active_gateway, remove_gateway_metadata, + save_active_gateway, save_last_sandbox, store_gateway_metadata, }; use navigator_core::proto::{ CreateProviderRequest, CreateSandboxRequest, DeleteProviderRequest, DeleteSandboxRequest, @@ -865,7 +864,6 @@ pub async fn gateway_add(endpoint: &str, name: Option<&str>, no_auth: bool) -> R gateway_endpoint: endpoint.clone(), is_remote: true, gateway_port: 0, - kube_port: None, remote_host: None, resolved_host: None, auth_mode: Some("cloudflare_jwt".to_string()), @@ -1169,13 +1167,10 @@ fn print_failure_diagnosis(diagnosis: &navigator_bootstrap::errors::GatewayFailu /// Provision or start a gateway (local or remote). pub async fn gateway_admin_deploy( name: &str, - update_kube_config: bool, - get_kubeconfig: bool, remote: Option<&str>, ssh_key: Option<&str>, port: u16, gateway_host: Option<&str>, - kube_port: Option, recreate: bool, disable_tls: bool, disable_gateway_auth: bool, @@ -1248,20 +1243,6 @@ pub async fn gateway_admin_deploy( let handle = deploy_gateway_with_panel(options, name, location).await?; - if update_kube_config { - let target_path = default_local_kubeconfig_path()?; - update_local_kubeconfig(name, &target_path)?; - eprintln!( - "{} Updated kubeconfig at {}", - "✓".green().bold(), - target_path.display() - ); - } - - if get_kubeconfig { - print_kubeconfig(name)?; - } - print_deploy_summary(name, &handle); // Auto-activate: set this gateway as the active gateway. @@ -1354,13 +1335,6 @@ fn cleanup_gateway_metadata(name: &str) { } } -fn resolve_remote(name: &str, remote_override: Option<&str>) -> Option { - match resolve_gateway_control_target(name, remote_override) { - GatewayControlTarget::Remote(dest) => Some(dest), - GatewayControlTarget::Local | GatewayControlTarget::ExternalRegistration => None, - } -} - /// Stop a gateway. pub async fn gateway_admin_stop( name: &str, @@ -1416,8 +1390,6 @@ pub fn gateway_admin_info(name: &str) -> Result<()> { ) })?; - let kubeconfig_path = navigator_bootstrap::stored_kubeconfig_path(name)?; - println!("{}", "Gateway Info".cyan().bold()); println!(); println!(" {} {}", "Gateway:".dimmed(), metadata.name); @@ -1426,15 +1398,6 @@ pub fn gateway_admin_info(name: &str) -> Result<()> { "Gateway endpoint:".dimmed(), metadata.gateway_endpoint ); - println!( - " {} {}", - "Stored kubeconfig:".dimmed(), - kubeconfig_path.display() - ); - - if let Some(kube_port) = metadata.kube_port { - println!(" {} {kube_port}", "Kube port:".dimmed()); - } if metadata.is_remote { if let Some(ref host) = metadata.remote_host { @@ -1445,70 +1408,158 @@ pub fn gateway_admin_info(name: &str) -> Result<()> { if let Some(ref resolved) = metadata.resolved_host { println!(" {} {resolved}", "Resolved host:".dimmed()); } - - if let (Some(host), Some(kube_port)) = (&metadata.remote_host, metadata.kube_port) { - println!(); - println!("{}", "SSH tunnel for kubectl access:".dimmed()); - println!(" openshell gateway tunnel --name {name}"); - println!("Or manually:"); - println!(" ssh -L {kube_port}:127.0.0.1:6443 {host}"); - } } Ok(()) } -/// Print or start an SSH tunnel for kubectl access to a remote gateway. -pub fn gateway_admin_tunnel( +/// Fetch logs from the gateway Docker container. +/// +/// Connects to the Docker daemon (local or remote via SSH) and retrieves +/// logs from the `openshell-cluster-{name}` container. +pub async fn doctor_logs( name: &str, - remote_override: Option<&str>, + lines: Option, + tail: bool, + remote: Option<&str>, ssh_key: Option<&str>, - print_command: bool, ) -> Result<()> { - let remote = resolve_remote(name, remote_override).ok_or_else(|| { - miette::miette!( - "Gateway '{name}' is not a remote gateway (no SSH destination found).\n\ - SSH tunnels are only needed for remote gateways." - ) - })?; + // Build remote options: explicit --remote flag, or auto-resolve from metadata + let remote_opts = if let Some(dest) = remote { + let mut opts = RemoteOptions::new(dest); + if let Some(key) = ssh_key { + opts = opts.with_ssh_key(key); + } + Some(opts) + } else if let Some(metadata) = get_gateway_metadata(name) + && metadata.is_remote + && let Some(ref host) = metadata.remote_host + { + let mut opts = RemoteOptions::new(host.clone()); + if let Some(key) = ssh_key { + opts = opts.with_ssh_key(key); + } + Some(opts) + } else { + None + }; - let kube_port = get_gateway_metadata(name) - .and_then(|m| m.kube_port) - .ok_or_else(|| { - miette::miette!( - "Gateway '{name}' was deployed without --kube-port.\n\ - Redeploy with --kube-port to enable kubectl access via SSH tunnel." - ) - })?; + let stdout = std::io::stdout().lock(); + navigator_bootstrap::gateway_container_logs(remote_opts.as_ref(), name, lines, tail, stdout) + .await +} - let ssh_cmd = ssh_key.map_or_else( - || format!("ssh -L {kube_port}:127.0.0.1:6443 -N {remote}"), - |key| format!("ssh -i {key} -L {kube_port}:127.0.0.1:6443 -N {remote}"), - ); +/// Run a command inside the gateway Docker container. +/// +/// Spawns `docker exec` (or `ssh docker exec` for remote gateways) +/// as a child process with the user's terminal attached, so interactive +/// tools like `k9s` and `kubectl` work natively. +pub fn doctor_exec( + name: &str, + remote: Option<&str>, + ssh_key: Option<&str>, + command: &[String], +) -> Result<()> { + let container = container_name(name); + let is_tty = std::io::stdin().is_terminal(); - if print_command { - println!("{ssh_cmd}"); - return Ok(()); - } + // Wrap the user command with KUBECONFIG set + let inner_cmd = if command.is_empty() { + "KUBECONFIG=/etc/rancher/k3s/k3s.yaml sh".to_string() + } else { + let escaped: Vec = command.iter().map(|a| shell_escape(a)).collect(); + format!("KUBECONFIG=/etc/rancher/k3s/k3s.yaml {}", escaped.join(" ")) + }; - eprintln!("Starting SSH tunnel to {remote}..."); - eprintln!("Press Ctrl+C to stop the tunnel."); - eprintln!(); + // Resolve remote destination: explicit --remote flag, or auto-resolve from metadata + let remote_host = if let Some(dest) = remote { + Some(dest.to_string()) + } else if let Some(metadata) = get_gateway_metadata(name) + && metadata.is_remote + { + metadata.remote_host.clone() + } else { + None + }; - let status = Command::new("sh") - .arg("-c") - .arg(&ssh_cmd) + let mut cmd = if let Some(ref host) = remote_host { + // Remote: ssh docker exec [-it] sh -lc '' + let mut c = Command::new("ssh"); + if let Some(key) = ssh_key { + c.args(["-i", key]); + } + // -t forces TTY allocation over SSH when we have a local TTY + if is_tty { + c.arg("-t"); + } + c.arg(host); + c.arg("docker"); + c.arg("exec"); + if is_tty { + c.args(["-it"]); + } else { + c.arg("-i"); + } + c.args([&container, "sh", "-lc", &inner_cmd]); + c + } else { + // Local: docker exec [-it] sh -lc '' + let mut c = Command::new("docker"); + c.arg("exec"); + if is_tty { + c.args(["-it"]); + } else { + c.arg("-i"); + } + c.args([&container, "sh", "-lc", &inner_cmd]); + c + }; + + let status = cmd .status() .into_diagnostic() - .wrap_err("failed to start SSH tunnel")?; + .wrap_err("failed to execute docker exec")?; if !status.success() { - return Err(miette::miette!("SSH tunnel exited with status: {}", status)); + let code = status.code().unwrap_or(1); + std::process::exit(code); } Ok(()) } +/// Print the LLM diagnostic prompt to stdout. +/// +/// Outputs a system prompt that a coding agent can use to autonomously +/// diagnose gateway issues using `openshell doctor logs` and +/// `openshell doctor exec`. +pub fn doctor_llm() -> Result<()> { + use std::io::Write; + let stdout = std::io::stdout(); + let mut handle = stdout.lock(); + handle + .write_all(include_bytes!("doctor_llm_prompt.md")) + .into_diagnostic() + .wrap_err("failed to write LLM prompt to stdout")?; + Ok(()) +} + +/// Shell-escape a single argument for safe inclusion in a `sh -c` string. +fn shell_escape(s: &str) -> String { + if s.is_empty() { + return "''".to_string(); + } + // If the string is clean (alphanumeric, hyphens, underscores, dots, slashes, colons, equals), + // no quoting needed. + if s.chars() + .all(|c| c.is_ascii_alphanumeric() || "-_./,:=@".contains(c)) + { + return s.to_string(); + } + // Otherwise, single-quote it (escaping embedded single quotes) + format!("'{}'", s.replace('\'', "'\\''")) +} + /// Create a sandbox when no gateway is configured. /// /// Offers to bootstrap a new gateway first, then delegates to [`sandbox_create`]. @@ -3759,7 +3810,6 @@ mod tests { gateway_endpoint: endpoint.to_string(), is_remote: true, gateway_port: 0, - kube_port: None, remote_host: None, resolved_host: None, auth_mode: Some("cloudflare_jwt".to_string()), diff --git a/deploy/docker/Dockerfile.cluster b/deploy/docker/Dockerfile.cluster index f34aa239a..e0acb0a6e 100644 --- a/deploy/docker/Dockerfile.cluster +++ b/deploy/docker/Dockerfile.cluster @@ -8,6 +8,7 @@ # # This image includes: # - k3s binary and all supporting binaries (containerd-shim, runc, CNI, etc.) +# - k9s for interactive cluster debugging (via `openshell doctor exec -- k9s`) # - Packaged OpenShell Helm chart # - HelmChart CR for auto-deploying OpenShell # - Custom entrypoint for DNS configuration in Docker environments @@ -82,6 +83,9 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ # ip, ipset, conntrack, nsenter, pigz, etc.) COPY --from=k3s /bin/ /bin/ +# Copy k9s binary for interactive cluster debugging via `openshell doctor exec -- k9s` +COPY --from=k9s /tmp/k9s /usr/local/bin/k9s + # Copy iptables/nftables tooling (xtables-nft-multi, iptables-detect.sh, etc.) # These are in /bin/aux/ in the k3s image and must be on PATH. # Note: the Ubuntu iptables package provides /usr/sbin/iptables, but k3s diff --git a/docs/reference/cli.md b/docs/reference/cli.md index aa8bc509e..11694c009 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -47,7 +47,6 @@ openshell │ ├── destroy │ ├── info │ ├── add -│ ├── tunnel │ └── select [name] ├── sandbox │ ├── create @@ -72,6 +71,9 @@ openshell │ ├── set │ ├── update │ └── get +├── doctor +│ ├── logs +│ └── exec └── completions ``` @@ -102,10 +104,18 @@ Manage the OpenShell runtime cluster. | `openshell gateway destroy` | Permanently remove the cluster and all its data. | | `openshell gateway info` | Show detailed information about the cluster. | | `openshell gateway add ` | Register an existing remote gateway by URL. | -| `openshell gateway tunnel` | Set up a kubectl tunnel to a remote cluster. | | `openshell gateway select ` | Set the active cluster. All subsequent commands target this cluster. | | `openshell gateway select` | List all registered clusters (when called without a name). | +## Diagnostic Commands + +Troubleshoot gateway issues. + +| Command | Description | +|---|---| +| `openshell doctor logs` | Fetch logs from the gateway Docker container. Use `--tail` for streaming, `-n` to limit line count. | +| `openshell doctor exec -- ` | Run a command inside the gateway container (e.g., `kubectl get pods -A`, `k9s`, `sh`). | + ## Sandbox Commands Create and manage isolated agent execution environments. diff --git a/examples/gateway-deploy-connect.md b/examples/gateway-deploy-connect.md index 1f084c9f1..0dd3cd4e5 100644 --- a/examples/gateway-deploy-connect.md +++ b/examples/gateway-deploy-connect.md @@ -72,17 +72,12 @@ openshell sandbox create --name remote-test -- echo "running on remote host" openshell sandbox connect remote-test ``` -### 3. Access the Kubernetes API (optional) +### 3. View gateway logs (optional) -The Kubernetes API on a remote cluster is only reachable through an -SSH tunnel: +To inspect the gateway container logs: ```bash -# Start a tunnel in the background -openshell gateway tunnel - -# In another terminal -kubectl get pods -n navigator +openshell doctor logs ``` ### 4. Clean up diff --git a/scripts/bin/k9s b/scripts/bin/k9s new file mode 100755 index 000000000..efcfd19d0 --- /dev/null +++ b/scripts/bin/k9s @@ -0,0 +1,15 @@ +#!/usr/bin/env bash + +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Gateway-aware k9s wrapper. +# +# Runs k9s inside the active gateway's k3s container via +# `openshell doctor exec`. All arguments are forwarded to k9s. +# +# Usage: +# k9s +# k9s -n navigator + +exec openshell doctor exec -- k9s "$@" diff --git a/scripts/bin/kubectl b/scripts/bin/kubectl new file mode 100755 index 000000000..5a0fd4e75 --- /dev/null +++ b/scripts/bin/kubectl @@ -0,0 +1,15 @@ +#!/usr/bin/env bash + +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Gateway-aware kubectl wrapper. +# +# Runs kubectl inside the active gateway's k3s container via +# `openshell doctor exec`. All arguments are forwarded to kubectl. +# +# Usage: +# kubectl get pods -A +# kubectl logs -n navigator statefulset/navigator + +exec openshell doctor exec -- kubectl "$@" diff --git a/scripts/remote-deploy.sh b/scripts/remote-deploy.sh index edb49e398..8812bc9bd 100755 --- a/scripts/remote-deploy.sh +++ b/scripts/remote-deploy.sh @@ -20,7 +20,6 @@ Options: --remote-dir DIR Remote checkout directory (default: openshell) --name NAME Cluster name (default: openshell) --port PORT Gateway port (default: 8080) - --kube-port [PORT] Expose kube-apiserver on a host port; omit value to auto-pick --ssh-key PATH SSH private key for ssh/rsync --skip-sync Skip rsync and use the existing remote checkout --recreate Destroy and recreate the gateway from scratch @@ -54,7 +53,6 @@ REMOTE_HOST="" REMOTE_DIR=${REMOTE_DIR:-openshell} CLUSTER_NAME=${CLUSTER_NAME:-openshell} GATEWAY_PORT=${GATEWAY_PORT:-8080} -KUBE_PORT="${KUBE_PORT:-}" SSH_KEY="${SSH_KEY:-}" IMAGE_TAG=${IMAGE_TAG:-dev} CARGO_VERSION=${OPENSHELL_CARGO_VERSION:-0.0.0-dev} @@ -80,15 +78,6 @@ while [[ $# -gt 0 ]]; do GATEWAY_PORT="$2" shift 2 ;; - --kube-port) - if [[ $# -gt 1 && ! "$2" =~ ^-- ]]; then - KUBE_PORT="$2" - shift 2 - else - KUBE_PORT="auto" - shift - fi - ;; --ssh-key) require_value "$1" "${2-}" SSH_KEY="$2" @@ -202,8 +191,7 @@ ssh -t "${SSH_ARGS[@]}" "${REMOTE_HOST}" \ "${CARGO_VERSION}" \ "${RECREATE}" \ "${PLAINTEXT}" \ - "${DISABLE_GATEWAY_AUTH}" \ - "${KUBE_PORT}" <<'REMOTE_EOF' + "${DISABLE_GATEWAY_AUTH}" <<'REMOTE_EOF' set -euo pipefail REMOTE_DIR="$1" @@ -214,7 +202,6 @@ CARGO_VERSION="$5" RECREATE="$6" PLAINTEXT="$7" DISABLE_GATEWAY_AUTH="$8" -KUBE_PORT="${9:-}" cd "${REMOTE_DIR}" @@ -271,13 +258,6 @@ fi if [[ "${DISABLE_GATEWAY_AUTH}" == "true" ]]; then start_args+=(--disable-gateway-auth) fi -if [[ -n "${KUBE_PORT}" ]]; then - if [[ "${KUBE_PORT}" == "auto" ]]; then - start_args+=(--kube-port) - else - start_args+=(--kube-port "${KUBE_PORT}") - fi -fi echo "==> Starting gateway..." mise exec -- openshell "${start_args[@]}" diff --git a/tasks/scripts/cluster-bootstrap.sh b/tasks/scripts/cluster-bootstrap.sh index 86b39924c..69ad78cf0 100755 --- a/tasks/scripts/cluster-bootstrap.sh +++ b/tasks/scripts/cluster-bootstrap.sh @@ -243,7 +243,7 @@ if [ -z "${OPENSHELL_CLUSTER_IMAGE:-}" ]; then export OPENSHELL_CLUSTER_IMAGE="openshell/cluster:${IMAGE_TAG}" fi -DEPLOY_CMD=(openshell gateway start --name "${CLUSTER_NAME}" --port "${GATEWAY_PORT}" --update-kube-config) +DEPLOY_CMD=(openshell gateway start --name "${CLUSTER_NAME}" --port "${GATEWAY_PORT}") if [ -n "${GATEWAY_HOST:-}" ]; then DEPLOY_CMD+=(--gateway-host "${GATEWAY_HOST}") @@ -261,14 +261,6 @@ if [ -n "${GATEWAY_HOST:-}" ]; then fi fi -if [ -n "${KUBE_PORT:-}" ]; then - # Explicit port requested — pass it through. - DEPLOY_CMD+=(--kube-port "${KUBE_PORT}") -else - # Auto-select a free port so kubectl works during local development. - DEPLOY_CMD+=(--kube-port) -fi - "${DEPLOY_CMD[@]}" # Clear the fast-deploy state file so the next incremental deploy @@ -280,4 +272,3 @@ rm -f "${DEPLOY_FAST_STATE_FILE}" echo "" echo "Cluster '${CLUSTER_NAME}' is ready." -echo "KUBECONFIG has been updated."