- Log when Tegra platform is detected and host-files bind mount is added,
including the kernel version from the Docker daemon.
- Extract CDI GID snapshot logic into `snapshot_cdi_gids()` function that
only activates when GPU devices are present (/dev/nvidiactl exists).
- Log preserved CDI-injected GIDs when they are restored after initgroups.
- Fix cargo fmt formatting issue in docker.rs.
- Bump nvidia-container-toolkit from 1.18.2 to 1.19.0 to support the
-host-cuda-version flag used by newer CDI spec generation.
- Replace local filesystem check for host-files-for-container.d with
Docker API kernel version detection (contains "tegra"). This fixes
remote SSH deploys where the CLI machine may not have the directory.
- Only perform the Tegra check when GPU devices are requested.
Detect 101 Switching Protocols in relay_response() and switch to raw
bidirectional TCP relay instead of re-entering the HTTP parsing loop.
Previously, is_bodiless_response() treated 101 as a generic 1xx
informational response, forwarding only the headers and returning to
the HTTP parsing loop. After a 101, subsequent bytes are upgraded
protocol frames (e.g. WebSocket), not HTTP — causing the relay to
block or silently drop all post-upgrade traffic.
Changes:
- Add RelayOutcome enum (Reusable/Consumed/Upgraded) replacing bool
return type across L7Provider::relay trait and all relay functions
- Detect 101 before generic 1xx handler in relay_response(), capture
overflow bytes, return RelayOutcome::Upgraded
- Validate client sent Upgrade + Connection: Upgrade headers before
accepting 101 (rejects unsolicited upgrades from non-compliant
upstream servers)
- Extract shared handle_upgrade() helper used by both relay_rest()
and relay_passthrough_with_credentials()
- Add l7_decision=allow_upgrade audit log annotation for upgrades
- Add unit tests for 101 overflow capture, unsolicited 101 rejection,
and client_requested_upgrade header validation
- Add integration test: WebSocket echo through L7Provider::relay
Fixes: #652
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* fix(sandbox): add GPU device nodes and nvidia-persistenced to landlock baseline
Landlock READ_FILE/WRITE_FILE restricts open(2) on character device files
even when DAC permissions would otherwise allow it. GPU sandboxes need
/dev/nvidiactl, /dev/nvidia-uvm, /dev/nvidia-uvm-tools, /dev/nvidia-modeset,
and per-GPU /dev/nvidiaX nodes in the policy to allow NVML initialization.
Additionally, CDI bind-mounts /run/nvidia-persistenced/socket into the
container. NVML tries to connect to this socket at init time; if the
directory is not in the landlock policy, it receives EACCES (not
ECONNREFUSED), which causes NVML to abort with NVML_ERROR_INSUFFICIENT_PERMISSIONS
even though nvidia-persistenced is optional.
Both classes of paths are auto-added to the baseline when /dev/nvidiactl is
present. Per-GPU device nodes are enumerated at runtime to handle multi-GPU
configurations.
PR #476 added Copilot as a supported provider but missed adding the
Copilot variant to the CliProviderType enum, causing --provider copilot
to fail at CLI argument parsing. Also adds a parity test to prevent
future drift between CliProviderType and ProviderRegistry.
Closes#707
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
Replace manual Header::set_path() + append() with
builder.append_path_with_name() which emits GNU LongName extensions
for paths exceeding the 100-byte POSIX tar name field limit.
Fixes#705
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
Remove Box::leak usage that permanently leaked one String allocation per
forward proxy request. Write the rewritten request line directly to the
output buffer during iteration instead of mutating a Vec<&str> element.
Closes#709
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
The image push pipeline buffered the entire Docker image tar 3x in
memory (export, tar wrap, Bytes copy), causing OOM kills for images
over ~1-2 GB. Replace the in-memory pipeline with a temp-file +
streaming upload: export to a NamedTempFile, then stream the outer
tar (header, 8 MiB file chunks, footer) directly into
upload_to_container via body_try_stream. Peak memory drops from ~3x
image size to ~8 MiB constant.
Also adds incremental export progress reporting every 100 MiB.
Check for existing sandbox name before persisting, matching the
provider-creation pattern. The CLI now surfaces a clear hint instead
of a raw UNIQUE constraint error.
Closes#691
* feat(sandbox): switch device plugin to CDI injection mode
Configure the NVIDIA device plugin to use deviceListStrategy=cdi-cri so
that GPU devices are injected via direct CDI device requests in the CRI.
Sandbox pods now only require the nvidia.com/gpu resource request —
runtimeClassName is no longer set on GPU pods.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* feat(bootstrap): switch GPU injection to CDI where supported
Use an explicit CDI device request (driver="cdi", device_ids=["nvidia.com/gpu=all"])
when the Docker daemon reports CDI spec directories via GET /info (SystemInfo.CDISpecDirs).
This makes device injection declarative and decouples spec generation from consumption.
When the daemon reports no CDI spec directories, fall back to the legacy NVIDIA device
request (driver="nvidia", count=-1) which relies on the NVIDIA Container Runtime hook.
Failure modes for both paths are equivalent: a missing or stale NVIDIA Container Toolkit
installation will cause container start to fail.
CDI spec generation is out of scope for this change; specs are expected to be
pre-generated out-of-band, for example by the NVIDIA Container Toolkit.
---------
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
Replace flat pty_master/input_sender/pty_request fields in SshHandler
with a HashMap<ChannelId, ChannelState> so each channel tracks its own
PTY resources independently. This fixes window_change_request resizing
the wrong PTY when multiple channels are open simultaneously.
Also fixes ioctl UB in set_winsize (pass &winsize not winsize by value)
and adds warn! logging for unknown channels across all handlers.
Resolves#543
* perf(sandbox): streaming SHA256, spawn_blocking for identity resolution
Key changes:
- Replace full file read + SHA256 with streaming 64KB-buffered hash
(saves 124MB allocation for node binary)
- Wrap evaluate_opa_tcp in spawn_blocking to prevent blocking tokio
runtime during heavy /proc I/O and SHA256 computation
- Add file-based perf logging for profiling proxy latency phases
Profiling data (node binary, 124MB):
- Cold TOFU: ~890ms (read+hash), warm: 0ms (cache hit)
- evaluate_opa_tcp: cold=1002ms, warm=11ms
- OPA evaluation: 1ms
- DNS+TCP connect: 166-437ms
Made-with: Cursor
Signed-off-by: Rafael Koike <koike.rafael@gmail.com>
* refactor(sandbox): replace perf_log with tracing::debug
Replace the custom file-based perf_log() helper with standard
tracing::debug!() macros as requested in PR review. This removes
the custom log file writes to /var/log/openshell-perf.log and
routes all performance timing through the tracing framework at
DEBUG level, consistent with the rest of the codebase.
Made-with: Cursor
* refactor(sandbox): reduce tracing to 6 key diagnostic logs
Address PR review feedback:
1. Remove ~20 inner-phase timing logs, keeping only the 6 that tell
the full diagnostic story:
- evaluate_opa_tcp TOTAL (proxy.rs)
- dns_resolve_and_tcp_connect (proxy.rs)
- file_sha256 (procfs.rs)
- verify_or_cache CACHE HIT / CACHE MISS / TOTAL cold (identity.rs)
2. Restore intent-describing comments that were replaced by timing logs:
- "TOFU verify the immediate binary" (proxy.rs)
- "Walk the process tree upward to collect ancestor binaries" (proxy.rs)
- "Collect cmdline paths for script-based binary detection." (proxy.rs)
- "First: scan descendants of the entrypoint process" (procfs.rs)
- "Fallback: scan all of /proc in case the process isn't in the tree" (procfs.rs)
- "Skip PIDs we already checked" (procfs.rs)
3. Preserve file path in file_sha256 read errors instead of losing
context via into_diagnostic().
4. Tests: 293 passed, 1 pre-existing failure (drop_privileges), 1 ignored.
Made-with: Cursor
* style(sandbox): apply rustfmt formatting to debug macros
---------
Signed-off-by: Rafael Koike <koike.rafael@gmail.com>
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* feat(sandbox): add L7 query parameter matchers
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* fix(sandbox): decode + as space in query params and validate glob syntax
Three improvements from PR #617 review:
1. Decode + as space in query string values per the
application/x-www-form-urlencoded convention. This matches Python's
urllib.parse, JavaScript's URLSearchParams, Go's url.ParseQuery, and
most HTTP frameworks. Literal + should be sent as %2B.
2. Add glob pattern syntax validation (warnings) for query matchers.
Checks for unclosed brackets and braces in glob/any patterns. These
are warnings (not errors) because OPA's glob.match is forgiving,
but they surface likely typos during policy loading.
3. Add missing test cases: empty query values, keys without values,
unicode after percent-decoding, empty query strings, and literal +
via %2B encoding.
* fix(sandbox): add missing query_params field in forward proxy L7 request info
* style(sandbox): fix formatting in proxy L7 query param parsing
---------
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* fix(sandbox): handle per-path Landlock errors instead of abandoning entire ruleset
A single missing path (e.g., /app in containers without that directory)
caused PathFd::new() to propagate an error out of the entire Landlock
setup closure. Under BestEffort mode, this silently disabled all
filesystem restrictions for the sandbox.
Changes:
- Extract try_open_path() and classify_path_error() helpers that handle
PathFd failures per-path instead of per-ruleset
- BestEffort mode: skip inaccessible paths with a warning, apply
remaining rules
- HardRequirement mode: fail immediately on any inaccessible path
- Add zero-rule safety check to prevent applying an empty ruleset that
would block all filesystem access
- Pre-filter system-injected baseline paths (e.g., /app) in enrichment
functions so missing paths never reach Landlock
- Add unit tests for try_open_path, classify_path_error, and error
classification for ENOENT, EACCES, ELOOP, ENAMETOOLONG, ENOTDIR
- Update user-facing docs and architecture docs with Landlock behavior
tables, baseline path filtering, and compatibility mode semantics
- Fix stale ABI::V1 references in docs (code uses ABI::V2)
Closes#664
* fix(sandbox): use debug log for NotFound in Landlock best-effort mode
NotFound errors for stale baseline paths (e.g. /app persisted in the
server-stored policy but absent in this container) are expected in
best-effort mode. Downgrade from warn! to debug! so the message does
not leak into SSH exec stdout (the pre_exec hook inherits the tracing
subscriber whose writer targets fd 1).
Genuine errors (permission denied, symlink loops, etc.) remain at warn!
for operator visibility.
Also move custom_image e2e marker from /opt to /etc (a Landlock baseline
read-only path) since the security fix now properly enforces filesystem
restrictions.
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
The CL/TE desynchronisation guard added in #663 for the REST path was
not applied to the inference request parser. A request containing both
Content-Length and Transfer-Encoding headers could be interpreted
differently by the proxy and the upstream server, enabling HTTP request
smuggling (CWE-444, RFC 7230 Section 3.3.3).
Add the same rejection check and tests mirroring the REST parser
coverage, including TE substring validation.
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
Co-authored-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
The base sandbox image installs Python via uv at
/sandbox/.uv/python/*/bin/python*, but the proxy resolves binary
identity via /proc/PID/exe (the real path, not the symlink). The test
policy only listed /usr/bin/python* and /usr/local/bin/python*, so OPA
denied the connection at L4 before L7 evaluation could run.
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
Fork PRs receive a read-only GITHUB_TOKEN that cannot push to gh-pages,
causing the deploy step to fail with a 403. Skip the deploy for fork PRs
while still running the docs build to validate changes compile cleanly.
* refactor(l7): export evaluate_l7_request for cross-module use
Make evaluate_l7_request() public so the forward proxy path can
evaluate individual requests against L7 policy without going
through the full relay_with_inspection() loop.
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
* fix(proxy): add L7 inspection to forward proxy path
The forward proxy previously rejected all requests to endpoints with
L7 rules (blanket 403), forcing clients through the CONNECT tunnel.
This meant policies like read-only (allow GET, block POST) had no
effect on plain http:// requests through the forward proxy.
Replace the blanket rejection with actual L7 evaluation:
- Query L7 config for the endpoint (same as before)
- Clone the OPA engine and evaluate the request method/path
- Allow if L7 policy permits, deny with 403 if enforcement is enforce
- Audit mode: log but allow (matching CONNECT path behavior)
- Fail-closed: deny on evaluation errors
The forward proxy uses Connection: close (one request per connection),
so a single evaluation suffices — no relay loop needed.
Update e2e tests to validate the new behavior:
- GET /allowed → 200 (L7 policy allows)
- POST /allowed → 403 (L7 policy denies, enforcement: enforce)
Update security-policy.md to reflect the new forward proxy L7 behavior.
Closes#643
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
* style(proxy): apply cargo fmt formatting
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
---------
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
Co-authored-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
* fix(l7): reject duplicate Content-Length headers to prevent request smuggling
Both parse_body_length() in rest.rs and try_parse_http_request() in
inference.rs silently accepted multiple Content-Length headers,
overwriting with the last value seen. Per RFC 7230 Section 3.3.3,
a message with multiple Content-Length headers with differing values
must be rejected to prevent HTTP request smuggling (CWE-444).
An attacker could send conflicting Content-Length values causing the
proxy and downstream server to disagree on message boundaries.
Fix:
- rest.rs: detect duplicate CL headers with differing values and
return an error before forwarding
- inference.rs: add ParseResult::Invalid variant; detect duplicate
CL headers and return Invalid with a descriptive reason
- proxy.rs: handle ParseResult::Invalid by sending HTTP 400 and
denying the connection
Closes#637
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
* fix(l7): address review feedback on Content-Length smuggling defense
- inference.rs: reject unparseable Content-Length values instead of
silently defaulting to 0 via unwrap_or(0)
- rest.rs: reject unparseable Content-Length values so a valid+invalid
duplicate pair cannot bypass the differing-values check
- rest.rs: fix Transfer-Encoding substring match (.contains("chunked")
→ split/trim exact match) to align with inference.rs and prevent
false positives on values like "chunkedx"
- proxy.rs: log parsing details server-side via tracing::warn and
return generic "Bad Request" body instead of leaking internal
parsing reasons to sandboxed code
- Add tests for all new rejection paths in inference.rs and rest.rs
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
* style(l7): apply cargo fmt formatting
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
---------
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
Co-authored-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
Three issues prevent k3s from starting on kernels where the nf_tables
xt extension bridge (nft_compat) is unavailable:
1. kube-router's network policy controller uses the xt_comment iptables
extension and panics on startup with "Extension comment revision 0
not supported, missing kernel module?" Pass --disable-network-policy
to k3s so the controller never runs. The NSSH1 HMAC handshake remains
the primary sandbox SSH isolation boundary, so this does not weaken
the effective security posture.
2. flannel and kube-proxy also fail to insert rules via the nf_tables
iptables backend on the same kernels. Add an xt_comment probe at
cluster-entrypoint startup; if the probe fails, switch to
iptables-legacy via update-alternatives before any other netfilter
work so that flannel, kube-proxy, and the DNS proxy all use a
consistent backend.
3. The br_netfilter kernel module must be loaded on the host for
iptables rules to apply to pod bridge traffic. Without it, ClusterIP
DNAT (including kube-dns at 10.43.0.10) is never applied to pod
packets, causing silent DNS timeouts deep in the health-check loop.
Add an early check that fails fast with an actionable error message
if the module is not present, instructing the user to run
`sudo modprobe br_netfilter` on the host.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
On Jetson/Tegra platforms nvidia-smi is installed at /usr/sbin/nvidia-smi
rather than /usr/bin/nvidia-smi and may not be on PATH inside the sandbox.
Fall back to the full path when the bare command is not found.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
initgroups(3) replaces all supplemental groups with the user's entries
from /etc/group, discarding GIDs injected by the container runtime via
CDI (e.g. GID 44/video needed for /dev/nvmap on Tegra). Snapshot the
container-level GIDs before initgroups runs and merge them back
afterwards, excluding GID 0 (root) to avoid privilege retention.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Use ghcr.io/nvidia/k8s-device-plugin:2ab68c16 which includes support for
mounting /etc/nvidia-container-runtime/host-files-for-container.d into the
device plugin pod, required for correct CDI spec generation on Tegra-based
systems.
Also included is an nvcdi API bump that ensures that additional GIDs are
included in the generated CDI spec.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Bind-mount /etc/nvidia-container-runtime/host-files-for-container.d
(read-only) into the gateway container when it exists, so the nvidia
runtime running inside k3s can apply the same host-file injection
config as on the host — required for Jetson/Tegra platforms.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(sandbox): emit warning when Landlock filesystem sandbox degrades silently
BestEffort Landlock previously swallowed failures at debug level, making
sandbox bypass invisible to operators at default log levels. Upgrade the
degradation log to warn with an actionable message pointing to the
hard_requirement setting. Add info-level startup log showing the
requested ABI and path counts so operators always know what Landlock
protections are active.
Closes#584
* fix(sandbox): revert unintended ABI bump from V2 to V5
Signed-off-by: John Myers <johntmyers@users.noreply.github.com>
---------
Signed-off-by: John Myers <johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* fix(router): use max_completion_tokens for OpenAI GPT-5+ validation probe
OpenAI GPT-5 models reject the legacy max_tokens parameter and require
max_completion_tokens. The inference validation probe now sends
max_completion_tokens as the primary parameter, with an automatic
fallback to max_tokens when the backend returns HTTP 400 (for
legacy/self-hosted backends that only support the older parameter).
Closes#517
Signed-off-by: Maxime Grenu <maxime.grenu@gmail.com>
* style(router): fix cargo fmt import order and line length
---------
Signed-off-by: Maxime Grenu <maxime.grenu@gmail.com>
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
relay_passthrough_with_credentials called relay_http_request_with_resolver
(which internally relays the upstream response back to the client) and then
immediately called relay_response_to_client a second time. The second call
blocked forever waiting for a response that would never arrive, deadlocking
every CONNECT tunnel after its first request/response pair.
This caused npm install (and any HTTP/1.1 keep-alive client) to hang
indefinitely when routed through the sandbox proxy without L7 rules.
The L7-inspection path (relay_rest) was not affected — it correctly makes
a single call to relay_http_request_with_resolver.
* fix(sandbox,server): fix chunk merge duplicates and OPA variable collision with overlapping policies
Two related bugs triggered when a draft rule approval creates a second
policy entry for the same host:port:
1. merge_chunk_into_policy looked up existing rules by chunk.rule_name
(auto-generated as allow_{host}_{port}), which never matched the
user's original rule name. Now scans all network_policies entries
for a host:port endpoint match before falling back to insertion,
and merges allowed_ips into the existing endpoint.
2. The Rego allow_request rule and _matching_endpoint_configs
comprehension used 'some ep; ep := policy.endpoints[_]' which
caused regorus to error with 'duplicated definition of local
variable ep' when multiple policies covered the same host:port.
Refactored to isolate endpoint iteration inside helper functions
(_policy_allows_l7, _policy_endpoint_configs) so variables are
scoped per-policy evaluation.
Refs: #567
* test(e2e): add overlapping policy tests and update FWD-2 for implicit allowed_ips
- Update FWD-2 (test_forward_proxy_denied_without_allowed_ips ->
test_forward_proxy_allows_private_ip_host_without_allowed_ips):
literal IP host no longer requires explicit allowed_ips, expects 200.
- Add OVL-1: overlapping L4 policies for same host:port must not crash
OPA and should allow forward proxy connections.
- Add OVL-2: overlapping L7 policies for same host:port must not crash
OPA and should allow CONNECT tunnel establishment.
Refs: #567
* style: apply cargo fmt formatting
* test(e2e): update SSRF-3 and SSRF-6 for implicit allowed_ips behavior
SSRF-6: Private IP with literal IP host now gets implicit allowed_ips
from PR #570, so CONNECT returns 200 instead of 403.
SSRF-3: Loopback is still blocked but via the always-blocked path
(implicit allowed_ips is synthesized, then resolve_and_check_allowed_ips
catches it). Log message says 'always-blocked' instead of 'internal
address'.
* fix(e2e): use negative assertion for SSRF-6 when nothing listens on target port
When the SSRF check passes but nothing listens on the target port,
recv() returns empty bytes. Use 'assert 403 not in' (matching SSRF-4
pattern) instead of 'assert 200 in'.
* fix(e2e): update provider tests for redacted credential values
PR #569 changed credential redaction from clearing the map to
replacing values with 'REDACTED'. Update e2e assertions to expect
credential keys with REDACTED values instead of an empty map.
When a policy endpoint specifies a literal IP address as the host
(e.g. host: 192.168.86.157), the user has explicitly declared intent
to allow that destination. The SSRF guard requiring allowed_ips was
redundant for this case and forced users to duplicate the IP.
Synthesize an implicit allowed_ips entry when the host parses as an IP
address, so the existing allowlist-validation path is used instead of
the blanket internal-IP rejection. Loopback and link-local addresses
remain blocked by resolve_and_check_allowed_ips.
Applies to both the CONNECT and FORWARD proxy paths.
Refs: #567
redact_provider_credentials used HashMap::clear() which removed both
keys and values, causing provider list/get to report 0 credential keys.
Replace with value-only redaction so key names and counts are preserved
while secret values remain hidden.
Refs: #350
Pass CLUSTER_GPU=1 inline in e2e:python:gpu's depends so that the
cluster is bootstrapped with --gpu when GPU e2e tests are run.
Add --gpu flag handling to cluster-bootstrap.sh and default
OPENSHELL_E2E_GPU_IMAGE to an empty string so the server resolves
the default sandbox image when no override is provided.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Piotr Mlocek