The strip_headers list in send_backend_request included default_headers
names (e.g. anthropic-version), which removed the client's value. The
already_sent check then saw the original input still contained the header
and skipped injecting the route default. Neither value reached upstream.
Remove default_headers names from strip_headers so client values pass
through. The existing already_sent guard still injects defaults only when
the client omits them.
* feat(sandbox): inject host gateway hostAliases into sandbox pods
Sandbox pods running in the k3s cluster cannot resolve host.docker.internal
by default, preventing them from reaching services on the Docker host.
Detect the host gateway IP (default route) in the cluster entrypoint,
thread it through the Helm chart to the gateway server, and inject
hostAliases entries (host.docker.internal, host.openshell.internal)
into every sandbox pod spec. The injection is conditional -- when the
IP is empty (non-Docker deployments), no hostAliases are added.
---------
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
Compute version strings from git describe in openshell-core's build.rs
using the guess-next-dev scheme (e.g. 0.0.4-dev.6+g2bf9969). All binary
crates and the TUI splash screen now use the shared openshell_core::VERSION
constant instead of CARGO_PKG_VERSION.
In Docker/CI builds where .git is absent, falls back to CARGO_PKG_VERSION
which is already set correctly by the sed-patch pipeline. Also adds
OPENSHELL_CARGO_VERSION support to the cluster image and fast-deploy
supervisor builds for parity with the gateway.
PR #281 removed the shared openshell-cluster Docker network in favor of
the default bridge. This restores custom bridge networking but makes each
gateway use its own isolated network named openshell-cluster-{name},
matching the existing container/volume naming convention.
Changes:
- Add network_name() to constants.rs for per-gateway network naming
- Add ensure_network() with retry/backoff and force_remove_network()
parameterized by network name instead of a global constant
- Attach containers to their per-gateway network via network_mode
- Disconnect and remove the network during gateway destroy
- Wire ensure_network() into the deploy flow before ensure_volume()
- Update architecture docs to reflect per-gateway network isolation
* feat(sandbox): add gpu sandbox scheduling support
Allow sandbox creation to request GPU resources explicitly or infer them from GPU image names. This wires GPU intent through bootstrap, validates gateway support, and adds dedicated GPU E2E coverage for follow-up cluster testing.
* feat(tui): add log copy and visual selection mode
Add clipboard support for the TUI log viewer using OSC 52 escape
sequences. Users can copy individual log lines (y), the visible
viewport (Y), or enter visual selection mode (v) to select a range
of lines with j/k and yank with y.
- Add clipboard module with OSC 52 base64 encoding
- Add format_log_line_plain() for plain-text log rendering
- Add visual selection mode with highlight styling and status bar
- Context-switch nav bar hints between normal and visual modes
- Add 7 unit tests for plain-text log formatting
Closes#274
* fix(tui): write OSC 52 to /dev/tty instead of stdout
ratatui owns stdout via the alternate screen buffer, so OSC 52
escape sequences written to stdout are swallowed by the backend.
Write directly to /dev/tty to bypass the buffer and reach the
terminal emulator.
* feat(tui): support light terminal backgrounds with adaptive theme
Replace the hardcoded dark-only color constants in theme.rs with a
runtime Theme struct that has dark() and light() factory constructors.
The active theme is stored on App and threaded through all draw
functions, enabling the TUI to render correctly on both dark and light
terminal backgrounds.
- Add Theme struct with 16 semantic style fields and ThemeMode enum
- Detect terminal background via COLORFGBG env var at startup
- Add --theme dark|light|auto CLI flag and OPENSHELL_THEME env var
- Migrate all 499 styles:: references across 11 UI files to app.theme
- Clean up 9 inline Style constructions that bypassed the theme system
- Add unit tests verifying dark/light palettes and legacy regression
Closes#264
* fix(tui): use OSC 11 query for reliable light/dark terminal detection
Replace COLORFGBG env var heuristic with terminal-colorsaurus crate,
which sends an OSC 11 query to read the actual background color. This
fixes auto-detection in iTerm2 and other terminals that don't set
COLORFGBG.
* fix(proxy): stream inference responses instead of buffering entire body
The inference.local proxy path called response.bytes().await which
buffered the entire upstream response before sending anything to the
client. For streaming SSE responses this inflated TTFB from sub-second
to the full generation time, causing clients with TTFB timeouts to abort.
Add a streaming proxy variant that returns response headers immediately
and forwards body chunks incrementally using HTTP chunked transfer
encoding. Non-streaming responses and mock routes continue to work
through the existing buffered path.
Closes#260
* docs: update architecture docs and example for inference streaming
* test(example): add direct NVIDIA endpoint tests via L7 TLS intercept
Expand inference example to 4 test cases: inference.local and direct
endpoint, each streaming and non-streaming. The direct path exercises
the L7 REST relay (relay_chunked) to verify it already streams
correctly. NVIDIA_API_KEY is picked up from the sandbox env when
started with --provider nvidia.
* feat(policy): add policy recommendation plumbing — denial aggregation, transport, approval pipeline, and mechanistic recommendations
Implement the infrastructure layer for automated policy recommendations (#204):
- Proto: 9 new RPCs and messages for draft policy lifecycle (submit, get, approve, reject, approve-all, edit, undo, clear, history)
- Persistence: SQLite/Postgres migrations and store methods for draft_policy_chunks and denial_summaries tables
- Server: Full gRPC handler implementations with mechanistic mapper that auto-generates NetworkPolicyRule proposals from denial summaries
- Sandbox: DenialAggregator with MPSC channel, deduplication, periodic flush to gateway via SubmitPolicyAnalysis
- CLI: 'openshell draft' subcommand with get/approve/reject/approve-all/undo/clear/history operations
- TUI: Draft recommendations panel accessible from sandbox policy view
- Docs: Architecture documentation in architecture/policy-advisor.md
* feat(policy): add L7-aware mechanistic mapper and policy advisor CTF example
Add L7 rule generation to mechanistic mapper (build_l7_rules,
generalise_path, looks_like_id) with 3 new unit tests. Add
examples/policy-advisor/ with a 7-gate CTF script, restrictive
sandbox policy, and walkthrough README.
* fix(policy): use sandbox name for denial flush and add TUI draft badges
Fix denial aggregator passing sandbox UUID instead of name to
SubmitPolicyAnalysis, which caused 'sandbox not found' errors on
flush. Add notification badges to the TUI sandbox list and detail
header showing pending draft recommendation counts.
* fix(policy): deduplicate draft chunks and tolerate overlapping OPA rules
Skip draft chunk creation when a pending/approved chunk already covers
the same host:port endpoint, preventing duplicate rules across denial
aggregator flush cycles.
Rewrite three OPA complete rules (network_policy_for_request,
matched_network_policy, matched_endpoint_config) to tolerate multiple
matching policies without triggering a "complete rule conflict" error.
network_policy_for_request becomes a boolean, matched_network_policy
uses a set comprehension with min(), and matched_endpoint_config uses
an array comprehension with index-0 selection.
* feat(tui): interactive draft actions, highlight bar, and detail popup
Rework the draft recommendations panel to match the logs UX:
- Highlight bar (green accent + background) instead of arrow marker
- Viewport-aware j/k scrolling with g/G for top/bottom
- Enter opens a full-screen detail popup showing endpoints, binaries,
rationale, security notes, and action hints
Add approve/reject/approve-all draft actions:
- [a] approve selected chunk, [x] reject, [A] approve all pending
- Actions work from both the list view and the detail popup
- gRPC calls run async; result updates status bar and refreshes data
- Nav bar shows all available keybindings
Fix draft count refresh: sandbox_draft_counts now refreshes on every
tick (not just Dashboard), so the detail header badge updates in
real time.
Improve badge labels: show 'N pending' instead of a bare number in
both the dashboard sandbox list and sandbox detail header.
* refactor(policy): DB-level draft chunk dedup with hit counter and timestamps
Replace the in-memory HashSet dedup in SubmitPolicyAnalysis with a
database-level upsert. New denormalized columns on draft_policy_chunks:
- host, port: extracted from proposed_rule at insert time
- hit_count: incremented on conflict (same sandbox + host + port)
- first_seen_ms, last_seen_ms: track when the endpoint was first and
most recently proposed
A partial unique index (WHERE status IN ('pending','approved')) ensures
only one active chunk per endpoint per sandbox; rejected/superseded
chunks don't block new proposals.
Surface hit_count and first/last_seen in:
- CLI: 'openshell draft get' shows 'Hits: N (first ..., last ...)'
- TUI: detail popup shows hits row; list view shows 'Nx' suffix
* fix(policy): optimistic retry on policy version conflicts + structured logging
merge_chunk_into_policy and remove_chunk_from_policy now retry up to 5
times on UNIQUE constraint violations (version conflicts from concurrent
approvals). Each attempt re-reads the latest policy, re-merges the rule,
and increments the version. This eliminates the race condition where
rapid successive approvals would fail with a DB error.
Add structured tracing to all draft action handlers:
- ApproveDraftChunk: logs rule_name, host, port, hit_count before merge
and version + policy_hash after success
- RejectDraftChunk: logs rule_name, host, port, reason
- ApproveAllDraftChunks: logs pending_count at start, per-chunk merge
progress, and final summary with chunks_approved/skipped
- UndoDraftChunk: logs before/after with rule_name and version
- Retry attempts log as warnings with attempt number and conflicting
version
* wip: forward proxy fix, mapper allowed_ips, TUI polish, CTF rewrite
* fix(tui): use correct --gateway flag for ssh-proxy ProxyCommand
* chore: add Docker cleanup script for stale images, volumes, and build cache
* feat(tui): approve-all confirmation modal and CTF cleanup
Add [A] confirmation popup that snapshots pending chunks, shows a
scrollable list, and approves each chunk individually on confirm.
This prevents approving chunks that arrived after the modal opened.
Remove transient issue #205 reference from CTF victory banner.
* fix(tui): correct import ordering for rustfmt
* wip: stateful toggle model, rename to network rules
Draft chunks now follow a toggle state machine:
pending -> approved | rejected (initial decision)
approved <-> rejected (toggle)
One row per (sandbox_id, host, port) via expanded unique index.
Rejecting an approved rule removes it from the active policy.
Re-approving a rejected rule merges it back.
Rename CLI from 'draft' to 'rule', TUI from 'Draft Recommendations'
to 'Network Rules'. State-aware keybindings: approved shows [x] Revoke,
rejected shows [a] Approve. Fix sandbox detail hiding delete confirmation
behind pending message.
* refactor(policy): move mapper sandbox-side, slim schema, per-binary granularity
Move mechanistic mapper from gateway to sandbox so all analysis runs
sandbox-side (N sandboxes = N independent pipelines). Gateway is now a
thin validate + persist + approval layer.
Architectural changes:
- Move mechanistic_mapper.rs from navigator-server to navigator-sandbox
- Sandbox flush flow: aggregator drains -> mapper runs -> proposals sent
- Gateway SubmitPolicyAnalysis: validate + persist only, no mapper
- Drop denial_summaries table (write-only, zero readers)
- Consolidate migrations 003+004+005 into single 003
Schema slimming:
- Drop 5 unused columns from draft_policy_chunks (stage, denial_refs,
supersedes_chunk_id, analysis_mode, decided_by)
- Add per-binary granularity: binary column, widen unique index to
(sandbox_id, host, port, binary)
- Mapper groups by (host, port, binary), one proposal per triple
- Merge appends binary to existing rule; revoke removes just that binary
CTF & UX:
- 7-gate CTF: add Gate 3 (curl -> ifconfig.me:80) for per-binary demo
- TUI shows binary short name in list, full path in detail popup
- CLI output shows binary field
- Idempotent rule names, hit_count accumulates real denial counts
- Rationale text no longer bakes in stale denial count
Add OPENSHELL_SANDBOX_IMAGE_PULL_POLICY env/CLI flag that propagates
through Config -> SandboxClient -> pod spec generation. When set, the
policy is applied to both inline and pod-template sandbox containers.
Helm chart exposes server.sandboxImagePullPolicy (default empty = K8s
default behavior). cluster-deploy-fast sets it to Always so dev
clusters always pull fresh sandbox images.
* fix(cli): improve completion coverage and gateway selection
Closes#232
Add typed completion metadata for gateway, sandbox, and path arguments, make gateway selection interactive on TTYs, and expose root subcommands in completions while keeping non-interactive behavior stable.
Add --gpu flag to 'openshell gateway start' that enables NVIDIA GPU
access inside the cluster container. When enabled, the Docker container
receives GPU device requests via bollard, k3s auto-detects the NVIDIA
runtime, and the NVIDIA k8s-device-plugin is deployed via HelmChart CR
so Kubernetes workloads can request nvidia.com/gpu resources.
- Add GPU device request to Docker container creation (bollard DeviceRequest)
- Set GPU_ENABLED=true env var to gate GPU manifest deployment
- Multi-stage Dockerfile build to bundle NVIDIA Container Toolkit binaries
- Conditional GPU manifest copy in cluster entrypoint
- HelmChart CR for nvidia-device-plugin v0.18.2 with GFD and NFD enabled
- Thread gpu bool through DeployOptions, CLI flag, and run dispatch
---------
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
Replace the clear_process_identity + maybe_infer_sandbox_user workaround
with strict enforcement: run_as_user and run_as_group must always be
'sandbox'. If omitted from YAML, they are defaulted server-side via
ensure_sandbox_process_identity(). Any other value is rejected.
This fixes 'policy set' failing with 'process policy cannot be changed
on a live sandbox' for custom-image sandboxes, where the CLI was
clearing identity fields but the YAML had 'sandbox' explicitly set.
- Remove clear_process_identity() from CLI and TUI
- Add ensure_sandbox_process_identity() to default missing fields
- Rename RootProcessIdentity -> InvalidProcessIdentity
- Replace maybe_infer_sandbox_user with validate_sandbox_user (fail-fast)
- Update OPA test policies to use run_as_user: sandbox
* feat(inference): add sandbox-system inference route for platform-level inference
Add a separate system-level inference route that the sandbox supervisor
can use in-process for platform functions (e.g., embedded agent harness
for policy analysis), distinct from the user-facing inference.local
endpoint. The system route is accessed via an in-process API on the
supervisor, ensuring userland code in the sandbox netns cannot reach it.
- Extend proto with route_name fields on Set/Get inference messages
- Add ResolvedRoute.name field to the router for route segregation
- Server resolves both user and sandbox-system routes in bundles
- Sandbox partitions routes into user/system caches on refresh
- Expose InferenceContext::system_inference() in-process API
- CLI --sandbox flag targets the system route on set/get/update
- Integration tests using mock:// routes for the full in-process path
Closes#207
* refactor(cli): rename --sandbox flag to --system for inference commands
The --sandbox flag could be misread as targeting a user-level sandbox
operation. Rename to --system to clearly indicate it configures the
platform-level system inference route.
* style(cli): collapse short if-else per rustfmt 1.94
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
The previous rename commit missed hardcoded namespace and workload
references in the bootstrap code. The Helm chart deploys to the
'openshell' namespace with 'openshell' workload names, but the
bootstrap code was still looking for 'navigator' namespace and
workloads, causing cluster startup to hang waiting for TLS secrets
that were never found.
Changes:
- lib.rs: Update secret fetch to use -n openshell
- runtime.rs: Rename functions and update all kubectl commands
to reference openshell namespace and workloads
UpdateProvider RPC was doing full replacement of credentials and config
maps, silently destroying data on partial updates. This changes the
semantics so empty maps preserve existing values, non-empty maps merge
(upsert), and empty-string values delete individual keys. Also makes
provider type immutable after creation and adds field validation on
update.
Closes#199
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* feat(cli): reorganize help output like gh CLI
- Add custom help template with grouped command sections (SANDBOX, GATEWAY, ADDITIONAL)
- Add bold headers using ANSI escape codes
- Add USAGE section showing command/subcommand pattern
- Add consistent help template for subcommands with COMMANDS section
- Add examples for all subcommands (sandbox, gateway, forward, logs, policy, provider, inference)
- Show help when running a command without a subcommand
- Add ALIAS section to subcommand help for aliased commands
- Remove less useful aliases (prov, inf, st, t, comp), keep core ones (sb, gw, lg, fwd, pol)
- Add RequestingCompute provisioning step with elapsed time display
* fix(server): replay platform events for late WatchSandbox subscribers
The CLI was stuck on 'Requesting compute...' even when the pod was
already pulling the container image. This happened because Kubernetes
events (Scheduled, Pulling, etc.) that fired before the CLI subscribed
to WatchSandbox were lost.
Add a tail buffer to PlatformEventBus (mirroring TracingLogBus) so
platform events are buffered and replayed to late subscribers. This
ensures the CLI shows accurate provisioning status regardless of when
it connects.
Closes#196
Piggyback on the existing 2-second sandbox list refresh to detect policy
version changes. When the user is on the sandbox screen and the listed
current_policy_version differs from the displayed version, fetch the
updated policy via GetSandboxPolicy. Scroll position is preserved across
auto-refreshes so the user is not disrupted.
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
The ProxyCommand in handle_shell_connect and handle_exec_command was
using --gateway for the URL, but after the #175 rename --gateway means
cluster name and --gateway-endpoint is the URL flag. The ssh-proxy
subprocess could not match token mode (no --gateway-endpoint) or name
mode (no --name), so it exited immediately — causing a screen flash
and eventual terminal corruption from rapid suspend/resume cycles.
Use --gateway-endpoint for the URL and --gateway for the cluster name,
matching the format already used by start_port_forwards and the CLI.
Fixes#188
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
When --provider <name> is passed and no provider with that name exists on
the server, the CLI now checks if the name is a recognized provider type
(e.g. claude, nvidia, github). If it is, the provider is auto-created
from local credentials using the same discovery flow as inferred providers.
If the name is not a recognized type, the CLI returns a clear error
instead of deferring validation to the server.
This eliminates the FailedPrecondition error that occurred when passing
--provider nvidia on a fresh cluster where no providers had been created
yet.
John T. Myers