* feat(podman): add Podman compute driver for rootless sandbox management
Adds openshell-driver-podman, a new compute driver that manages OpenShell
sandboxes as rootless Podman containers via the Podman REST API over a
Unix socket. Enables local workstation sandboxes without Kubernetes.
Driver features:
- Bridge networking with ephemeral host-port mapping for rootless SSH reachability
- Named volumes for workspace storage, Podman native health checks, GPU via CDI
- Supervisor binary sideloaded via image volume mount (BYOC-compatible)
- SSH handshake secret injected via Podman secrets API (not plaintext env)
- Typed ContainerSpec structs, input validation, and path-traversal guards
- Cgroups v2 required; fails fast on v1 hosts
- Bounded event stream buffer; watch stream reconnection handled by server watch_loop
- Graceful shutdown and standalone driver binary with gRPC bridge
Rootless-specific fixes:
- Skip drop_privileges when user namespace lacks SETUID/SETGID/DAC_READ_SEARCH caps
- Add /run/netns tmpfs mount for ip netns in rootless containers
- Use secret_env map (not secrets array) for env-var injection in libpod API
- Resolve SSH endpoint to 127.0.0.1:<host_port> instead of unreachable bridge IP
Server/sandbox hardening:
- Split loopback and link-local SSRF gates; Podman/VM drivers allow loopback
- Close SSRF bypass in SSH tunnel Host path by resolving DNS before connecting
- Prevent OPENSHELL_* env var override by user-supplied spec environment maps
- Disable SQLite pool idle_timeout/max_lifetime for in-memory databases
- Emit deleted_event on 404-during-inspect instead of regressing sandbox phase
- Key delete cleanup by stable sandbox_id to survive container label drift
CLI fixes:
- Restore --name as a named flag on sandbox create (not positional)
- Fix exec command arg parsing to not consume sandboxed-command flags
- Propagate SSH verbosity via OPENSHELL_SSH_LOG_LEVEL
Build tooling:
- Add tasks/scripts/container-engine.sh: auto-detects Podman or Docker, exposes
unified ce_* helpers; all build/cluster/VM scripts updated to use it
- Add docker:build:supervisor mise task for standalone supervisor image
- Add openshell-driver-podman to Dockerfile.images pre-fetch/build stages
- Add e2e/rust/e2e-podman.sh and e2e:podman mise task for full lifecycle testing
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(driver-podman): derive grpc endpoint from server bind port
When a user starts the gateway on a non-default port (e.g. --port 8081),
sandbox containers were receiving OPENSHELL_ENDPOINT pointing at the
default port 8080. The driver's auto-detection fallback read
OPENSHELL_BIND_ADDRESS from the environment, which was stale or unset,
and fell back to DEFAULT_SERVER_PORT.
Add gateway_port to PodmanComputeConfig and thread config.bind_address.port()
from the server into the driver so the fallback uses the actual listening
port. Remove the OPENSHELL_BIND_ADDRESS env var read and the
extract_port_from_bind_address helper which are no longer needed.
Add --gateway-port / OPENSHELL_GATEWAY_PORT to the standalone driver
binary for parity when the driver is run outside the embedded server path.
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(driver-podman): address PR feedback on env test safety and cluster DNS docs
Replace hand-rolled unsafe TempEnvVar RAII guard with temp_env::with_vars
and a static ENV_LOCK mutex, fixing a data race in parallel test execution.
The prior safety comment incorrectly claimed Cargo runs tests single-threaded.
Update debug-openshell-cluster skill to accurately document the DNS proxy
strategy (setup_dns_proxy + public DNS fallback) and clarify the separation
between cluster DNS and sandbox agent DNS enforcement.
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(e2e): resolve CI failures in auth timeout, test harness, and formatting
- Short-circuit browser_auth_flow when OPENSHELL_NO_BROWSER=1 instead
of waiting the full 120s AUTH_TIMEOUT for a callback that never arrives
- Add timeout to SandboxGuard::create() and create_with_upload() to
prevent indefinite hangs (matches create_keep() which already had one)
- Add missing '--' separator in no_proxy test before command args
- Add #![cfg(feature = "e2e")] gate to sandbox_lifecycle.rs
- Run cargo fmt on openshell-driver-podman
- Refine cluster DNS docs for Podman in debug-openshell-cluster skill
Signed-off-by: Adam Miller <admiller@redhat.com>
* refactor(server): remove allows_loopback_endpoints from ComputeRuntime
SSRF protection is now handled at the network and proxy layers
(openshell-core net.rs, openshell-sandbox proxy.rs) rather than
requiring per-driver flags on ComputeRuntime. Update architecture
docs to reflect supervisor relay SSH transport and add rootless
networking deep-dive.
Signed-off-by: Adam Miller <admiller@redhat.com>
---------
Signed-off-by: Adam Miller <admiller@redhat.com>
Ubuntu Noble's git links against libcurl-gnutls, which does not read
SSL_CERT_FILE. Without GIT_SSL_CAINFO, `git clone` over HTTPS fails
inside every community sandbox image with "server certificate
verification failed".
Add GIT_SSL_CAINFO to tls_env_vars pointing at the same combined CA
bundle already used by CURL_CA_BUNDLE / REQUESTS_CA_BUNDLE.
Fixes#790FixesNVIDIA/NemoClaw#2270FixesNVIDIA/NemoClaw#1828
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
Closes#879
Resolve policy hostnames against the sandbox's /etc/hosts before DNS so Kubernetes hostAliases are visible to the proxy while keeping the existing allowed_ips SSRF enforcement semantics intact.
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Closes#905
Skip proxy baseline read-write enrichment when a path is already explicitly configured as read-only, add regression coverage for the proto and local policy code paths, and document the precedence rule in the sandbox architecture docs.
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* fix(sandbox): canonicalize HTTP request-targets before L7 policy evaluation
The L7 REST proxy evaluated OPA path rules against the raw request-target
and forwarded the raw header block byte-for-byte to the upstream. The
upstream normalized `..`, `%2e%2e`, and `//` before dispatching, so a
compromised agent inside the sandbox could bypass any path-based allow
rule by prefixing the blocked path with an allowed one (e.g.
`GET /public/../secret` matches `/public/**` at the proxy but the
upstream serves `/secret`). The inference-routing detection had the same
normalization gap via `normalize_inference_path`, which only stripped
scheme+authority and preserved dot-segments verbatim.
- Add `crates/openshell-sandbox/src/l7/path.rs` with
`canonicalize_request_target`. Percent-decodes (with uppercase hex
re-emission), resolves `.` / `..` segments per RFC 3986 5.2.4,
collapses doubled slashes, strips trailing `;params`, rejects
fragments / raw non-ASCII / control bytes / encoded slashes (unless
explicitly enabled per-endpoint), and returns a `rewritten` flag so
callers know when to rewrite the outbound request line.
- Wire the canonicalizer into `rest.rs::parse_http_request`. The
canonical path is the `target` on `L7Request` that OPA sees, and
when the canonical form differs from the raw input the request line
is rebuilt in `raw_header` so the upstream dispatches on the exact
bytes the policy evaluated.
- Canonicalize the forward (plain HTTP proxy) path at
`proxy.rs`'s second L7 evaluation site. A non-canonical request-target
is rejected with 400 Bad Request and an OCSF `NetworkActivity` Fail
event rather than silently passed through.
- Replace the old `normalize_inference_path` body with a call to the
canonicalizer. On canonicalization error the raw path is returned
(so inference detection simply misses and the normal forward path
handles and rejects).
- Document the invariant in `sandbox-policy.rego`: `input.request.path`
is pre-canonicalized so rules must not attempt defensive matching
against `..` or `%2e%2e`.
- Architecture doc (`architecture/sandbox.md`) lists the new module.
- 24 new unit tests cover dot segments, percent-encoded dot segments,
double-slash collapse, encoded-slash reject/opt-in, null/control byte
rejection, legitimate percent-encoded bytes round-tripping, mixed-case
percent normalization, fragment rejection, absolute-form stripping,
empty / length-bounded / non-ASCII handling, and regression payloads
for the specific bypasses above.
Tracks OS-99.
* fix(sandbox): close forward-proxy parser-differential and wire allow_encoded_slash
Addresses two review findings on the L7 path canonicalization change.
1. `handle_forward_proxy` previously evaluated OPA on the canonical path
but wrote the raw path to the upstream via `rewrite_forward_request`,
leaving the parser-differential open for plain-HTTP forward-proxy
traffic even though it was closed for the L7 tunnel path. `path` is
now reassigned to the canonical form inside the L7 block before the
outbound write, so the bytes dispatched to the upstream match what
OPA evaluated. Covered by a new regression test against
`rewrite_forward_request`.
2. `allow_encoded_slash` is now wired through the YAML policy schema,
the proto `NetworkEndpoint` message, the Rego endpoint config, and
the `L7EndpointConfig` → `RestProvider` canonicalization options.
Operators can opt in per-endpoint for upstreams like GitLab that
embed `%2F` in namespaced resource paths; the default remains
strict. The `RestProvider` gains a constructor
`RestProvider::with_options` so `relay_rest` constructs a provider
with per-endpoint options while `relay_passthrough_with_credentials`
retains strict defaults.
Integration tests:
- `parse_http_request_canonicalizes_target_and_rewrites_raw_header`
drives a non-canonical request through the parser and asserts both
`L7Request.target` (what OPA evaluates) and `raw_header` (what the
upstream receives) are canonical.
- `parse_http_request_canonicalization_preserves_query_string`,
`parse_http_request_leaves_canonical_input_byte_for_byte`,
`parse_http_request_rejects_traversal_above_root`,
`parse_http_request_preserves_http_10_version_on_rewrite`.
- `parse_http_request_accepts_encoded_slash_when_endpoint_opts_in` /
`_rejects_encoded_slash_by_default` verify the
`allow_encoded_slash` gate.
- `test_rewrite_forward_request_uses_canonical_path_on_the_wire`
asserts the fix to the forward-proxy flow.
- `parse_l7_config_allow_encoded_slash_{defaults_false,opt_in}` verify
the YAML/Rego wiring.
* fix(sandbox): strip " (deleted)" suffix from unlinked /proc/<pid>/exe paths
When a running binary is unlinked from its filesystem path — the common
case is a `docker cp` hot-swap of `/opt/openshell/bin/openshell-sandbox`
during the `cluster-deploy-fast` dev upgrade workflow — the Linux kernel
appends the literal string ` (deleted)` to the `/proc/<pid>/exe` readlink
target. The tainted `PathBuf` then flows into `collect_ancestor_binaries`
and on into `BinaryIdentityCache::verify_or_cache`, which tries to
`std::fs::metadata` the path. `stat()` fails with `ENOENT` because the
literal suffix isn't a real filesystem path, and the CONNECT proxy denies
every outbound request with:
ancestor integrity check failed for \
/opt/openshell/bin/openshell-sandbox (deleted): \
Failed to stat ...: No such file or directory (os error 2)
Reproduced in production 2026-04-15: a cluster-deploy-fast-style hot-swap
of the supervisor binary caused every pod whose PID 1 held the now-deleted
inode to deny ALL outbound CONNECTs (slack.com, registry.npmjs.org,
169.254.169.254, etc.), breaking Slack REST delivery, npm installs, and
IMDS probes simultaneously. Existing pre-hot-swap TCP tunnels (e.g. Slack
Socket Mode WSS) kept working because they never re-evaluate the proxy.
Strip the suffix in `binary_path()` so downstream callers see the clean,
grep-friendly path. This aligns the cache key and log messages with the
original on-disk location.
Note: stripping the suffix does NOT by itself make the identity cache
tolerant of a legitimate binary replacement — `verify_or_cache` will now
`stat` and hash whatever currently lives at the stripped path, which is
the NEW binary, and surface a clearer `Binary integrity violation` error.
Fully unblocking the cluster-deploy-fast hot-swap workflow needs a
follow-up that either (a) reads running-binary content from
`/proc/<pid>/exe` directly via `File::open` (procfs resolves this to the
live in-memory executable even when the original inode has been unlinked),
or (b) keys the identity cache by exec dev+inode instead of path. Happy to
send that as a separate PR once the approach is decided — filing this
narrow fix first because it stands on its own: it fixes a concrete
misleading error and unblocks the obvious next step.
Added `binary_path_strips_deleted_suffix` test that copies `/bin/sleep`
to a temp path, spawns a child from it, unlinks the temp binary, verifies
the raw readlink contains the ` (deleted)` suffix, then asserts the public
API returns the stripped path.
Signed-off-by: mjamiv <michael.commack@gmail.com>
* fix(sandbox): narrow the " (deleted)" suffix strip and exercise it end-to-end
Address review feedback from @johntmyers on the initial version of this PR:
procfs::binary_path
- Only strip the kernel's " (deleted)" suffix when stat() on the raw
readlink target reports NotFound. A live executable whose basename
literally ends with " (deleted)" is now returned unchanged instead of
being silently truncated, which matters because identity.rs hashes
whatever this function returns as a trust anchor.
- Operate on raw bytes via OsStrExt, so filenames that are not valid
UTF-8 still get exactly one suffix stripped. The previous
strip_suffix-on-&str path skipped non-UTF-8 entirely and fell through
to returning the tainted path.
- Expand the doc comment to describe both guardrails.
procfs tests
- binary_path_preserves_live_deleted_basename: copy /bin/sleep to a
live file literally named "sleepy (deleted)", spawn it, and assert
that the returned path still ends with " (deleted)".
- binary_path_strips_suffix_for_non_utf8_filename: exec a binary whose
basename contains a 0xFF byte, unlink it, and assert that
binary_path returns the stripped non-UTF-8 path. Writes the bytes
with OpenOptions + sync_all + explicit drop so the write fd is fully
released before exec() to avoid ETXTBSY under concurrent tests.
proxy: extract resolve_process_identity helper
- Pull the peer-resolution + TOFU verify + ancestor walk + cmdline
collection block out of evaluate_opa_tcp into resolve_process_identity.
- Introduce IdentityError which carries the deny reason along with
whatever partial identity data was resolved before the failure so
evaluate_opa_tcp can thread that into ConnectDecision unchanged.
- evaluate_opa_tcp now calls the helper and proceeds straight to the
OPA evaluate step; the surface visible to OPA and OCSF is unchanged.
proxy: end-to-end regression test for the hot-swap contract
- resolve_process_identity_surfaces_binary_integrity_violation_on_hot_swap
stands up a real TcpListener, copies /bin/bash to a temp path,
primes BinaryIdentityCache with that binary, spawns bash with a
/dev/tcp one-liner that opens a real connection to the listener,
and captures the peer's ephemeral port from accept().
- Simulates `docker cp` correctly: unlink the running binary (which
persists via the child's exec mapping) and create a fresh file
with different bytes at the same path. Writing in place is
rejected by the kernel with ETXTBSY, so the old single-inode
approach did not actually model the production failure mode.
- Asserts the error returned by resolve_process_identity contains
"Binary integrity violation" (from BinaryIdentityCache) and does
NOT contain "Failed to stat" or "(deleted)" — the pre-PR-#844
failure mode. The binary field on the error is populated and is
free of the tainted suffix.
- Skips cleanly if /bin/bash is not installed. Child process is
always reaped before the assertion block so a failure does not
leak a sleeping process.
---------
Signed-off-by: mjamiv <michael.commack@gmail.com>
* fix(sandbox): block AF_NETLINK in seccomp unconditionally
Move AF_NETLINK to the unconditional socket-domain block list alongside
AF_PACKET, AF_BLUETOOTH, and AF_VSOCK. Previously it was only blocked in
NetworkMode::Block, leaving it accessible in Proxy mode where network
namespace isolation already scopes netlink to the sandbox's own veth —
making this a defense-in-depth hardening rather than a live exposure.
Closes OS-94
* fix(sandbox): scope inference.local interception to port 443
The pre-OPA interception for inference.local matched on hostname alone,
allowing any port to bypass OPA policy evaluation — including under
deny-all (network_policies: {}). Add a port check so only port 443
takes the interception path; all other ports on inference.local now
fall through to OPA and are subject to normal policy evaluation.
Closes OS-95
* fix(sandbox): enforce RLIMIT_NPROC to prevent fork bomb DoS
Set a hard limit of 512 processes per UID in harden_child_process(),
applied before privilege drop so the sandbox user cannot raise it.
Prevents unrestricted fork() from exhausting the process table — most
relevant for local dev mode where K8s pod cgroup pids.max is absent.
Closes OS-96
* chore(sandbox): fix rustfmt formatting for seccomp blocked_domains
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
The proxy's upstream TLS client only trusted Mozilla root CAs
(webpki-roots), which prevented TLS termination from working with
internal/corporate hosts using private CA certificates.
Load system CA certificates from the container's trust store
(e.g. /etc/ssl/certs/ca-certificates.crt) in addition to
webpki-roots. This allows custom sandbox images to include
corporate CAs via update-ca-certificates.
Signed-off-by: Matthias Osswald <mat.osswald@sap.com>
format_sse_error only escaped `\` and `"`, leaving two problems:
1. Control characters (`\n`, `\r`, `\t`, and all `\u0000-\u001F`) in
`reason` produce output that fails `serde_json::from_str` — defeating
#834's goal of giving clients a parseable SSE truncation signal.
2. An unescaped `\n\n` inside `reason` splits the single error event
into two SSE frames, letting a misbehaving upstream inject a forged
frame (e.g. a fake tool_calls delta) into the client's stream. Latent
today since all in-tree callers pass static strings, but a footgun
for any future caller passing upstream error text, and the function's
docstring already invites dynamic reasons.
Replace the manual escape with `serde_json::to_writer` (already a
workspace dep of `openshell-sandbox`). Add unit tests for control
character escaping and SSE event-boundary injection.
Closes#840
Signed-off-by: mjamiv <michael.commack@gmail.com>
* feat(policy): add deny rules to network policy schema
Closes#565
Add L7 deny rules that block specific requests even when allowed by
access presets or explicit allow rules. Deny rules mirror the full
capability set of allow rules (method, path, query params, SQL command)
and take precedence -- if a request matches any deny rule, it is blocked
regardless of allow rules.
This enables the "allow everything except these specific operations"
pattern without enumerating every allowed endpoint. For example, granting
read-write access to GitHub while blocking PR approvals, branch
protection changes, and ruleset modifications.
* fix(policy): deny query matching fails closed, mirror allow-side validation
Addresses PR review findings P1 and P3:
P1: Deny-side query matching now uses fail-closed semantics. If ANY
value for a query key matches the deny matcher, the deny fires. The
previous implementation reused allow-side "all values must match"
logic which allowed ?force=true&force=false to bypass a deny on
force=true.
P3: Deny-side query validation now mirrors the full allow-side checks:
empty any lists, non-string matcher values, glob+any mutual exclusion,
glob type checks, and glob syntax warnings are all validated.
Closes#783
Only chown read_write paths that the supervisor creates at startup, and leave pre-existing image paths with their original ownership. Add sandbox tests for creation, symlink rejection, and existing-path ownership preservation, and update architecture docs to match the new behavior.
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* fix(inference): prevent silent truncation of large streaming responses
The L7 inference proxy silently dropped tool_calls from large streaming
responses due to an aggressive 30s per-chunk idle timeout and a reqwest
total-request timeout that capped the entire body stream. Reasoning models
that pause during "thinking" phases triggered these timeouts, producing
valid-looking but truncated HTTP responses with no client-visible error.
- Extract prepare_backend_request() helper and create a streaming variant
that omits the total request timeout; body stream liveness is now
enforced solely by the per-chunk idle timeout
- Add 30s connect_timeout to the reqwest Client builder
- Increase CHUNK_IDLE_TIMEOUT from 30s to 120s for reasoning models
- Inject SSE error events (proxy_stream_error) before the HTTP chunked
terminator on all truncation paths so clients can detect data loss
- Wrap the streaming relay in BufWriter to reduce per-chunk TLS flush
overhead
- Bump OCSF severity for streaming truncation from Low to Medium
Closes#829
* fix(inference): drop BufWriter to preserve SSE incremental delivery
The BufWriter introduced in the previous commit buffered SSE frames until
the 16KB capacity filled or the stream ended, defeating incremental token
delivery and potentially reintroducing client-visible timeouts on healthy
streams. Revert to per-chunk write_all+flush but keep the single
format_chunk() call that coalesces framing into one write.
Also fix the streaming integration test: add a 3s mock delay that exceeds
the 1s route timeout so the test actually validates that the streaming
path omits the total request timeout. Previously the mock responded
immediately, passing regardless of timeout behavior.
* fix(sandbox): harden seccomp denylist, SSRF protection, and inference policy enforcement
- Remove seccomp skip in NetworkMode::Allow so baseline syscall
restrictions apply regardless of network mode
- Block cross-process manipulation syscalls (process_vm_writev,
pidfd_open, pidfd_getfd, pidfd_send_signal) symmetric with existing
ptrace and process_vm_readv blocks
- Block clone/clone3 with CLONE_NEWUSER flag, new mount API syscalls
(fsopen, fsconfig, fsmount, fspick, move_mount, open_tree), and
namespace manipulation (setns, umount2, pivot_root)
- Block userfaultfd and perf_event_open consistent with Docker default
seccomp profile
- Deny and close keep-alive inference connections after a non-inference
request instead of silently continuing the loop
- Add CGNAT (100.64.0.0/10), benchmarking (198.18.0.0/15), and other
special-use IP ranges to SSRF protection in both proxy and
mechanistic mapper
* fix(sandbox): install clone3 seccomp filter before main filter
* test(ocsf): fix shorthand firewall engine expectation
L4 CONNECT and forward-proxy denial responses (403) and upstream
failure responses (502) were returned with empty bodies, making them
indistinguishable from each other when using curl -s. Add a
build_json_error_response() helper that produces consistent JSON
bodies with error and detail fields, matching the format already
used by the L7 deny path.
Also fix Url::to_display_string() in openshell-ocsf to include
non-default ports in the shorthand log output. Requests to e.g.
http://host:9876/path were logged as http://host/path, dropping the
port number.
Closes#807, closes#808
* fix(sandbox): add parent-side Landlock availability probe logging
Landlock status was only logged from inside the pre_exec child process
where the tracing/OCSF pipeline is non-functional after fork. This made
Landlock failures completely invisible in sandbox logs.
Add a probe_availability() function that issues the raw
landlock_create_ruleset syscall to check kernel support, and call it
from the parent process before fork in all three spawn paths (entrypoint,
SSH PTY, SSH pipe). Uses std::sync::Once to emit exactly once per
sandbox lifetime.
WIP - addresses logging gap from #803.
* test(e2e): add Landlock filesystem enforcement tests
Verify Landlock availability logging and enforcement in e2e:
- OCSF probe event appears in sandbox logs
- Read-only paths block writes, allow reads
- Read-write paths allow both
- Paths outside policy are denied entirely
- User-owned paths outside policy are still blocked (proves
Landlock enforces independently of Unix DAC permissions)
Requires Linux host with Landlock support (GitHub Actions runners,
Docker Desktop linuxkit). Related to #803.
* test(e2e): add xfail tests proving #803 privilege ordering bug
Two strict xfail tests that demonstrate the root cause of #803:
- PathFd::new() runs as uid 998 after drop_privileges, so root-only
paths (mode 700) silently fail and Landlock degrades
- When ALL paths fail, best_effort silently drops Landlock entirely
These tests will pass after the two-phase Landlock fix (open PathFds
as root before drop_privileges, restrict_self after).
* fix(test): fix Landlock e2e tests based on test run results
- Remove log-reading tests: the Landlock probe logs to the
supervisor's container stdout, not the in-sandbox file appender
at /var/log/openshell*.log*
- Replace broken xfail test (checked for child-process log messages
that never reach the file appender) with a stat()-based test that
verifies /root is actually in the Landlock allowlist
- Keep enforcement tests (all passing) and the root-only policy
xfail test (correctly proves #803 bug)
* fix(test): remove mixed-policy xfail test
Landlock doesn't restrict stat(), so the test passed unexpectedly.
In the mixed policy case where /root is silently skipped, Landlock
still applies (using other paths) and blocks /root even harder
(not in allowlist = denied). The observable security degradation
only occurs when ALL paths fail, which the existing xfail test
already covers.
* fix(sandbox): two-phase Landlock to fix privilege ordering (#803)
Split Landlock apply into prepare() and enforce():
- prepare() runs as root before drop_privileges: opens PathFds,
creates ruleset, adds rules. Root-only paths (mode 700) now
succeed instead of silently failing as uid 998.
- enforce() runs after drop_privileges: calls restrict_self()
which does not require root.
This fixes the root cause of #803 where drop_privileges() ran
before sandbox::apply(), causing PathFd::new() to fail on
root-only paths. In best_effort mode this silently dropped all
Landlock restrictions.
The fix applies to all three spawn paths: entrypoint (process.rs),
SSH PTY (ssh.rs), and SSH pipe exec (ssh.rs).
Removes xfail marker from e2e test that now passes.
* fix(sandbox): address PR review feedback
- enforce() now respects best_effort: if restrict_self() fails and
policy is best_effort, log and degrade instead of aborting startup
- log_sandbox_readiness distinguishes best_effort (degraded) from
hard_requirement (will fail) in OCSF messages
* fix(sandbox): resolve symlinked binary paths in network policy matching
Policy binary paths specified as symlinks (e.g., /usr/bin/python3) were
silently denied because the kernel reports the canonical path via
/proc/<pid>/exe (e.g., /usr/bin/python3.11). The strict string equality
in Rego never matched.
Expand policy binary paths by resolving symlinks through the container
filesystem (/proc/<pid>/root/) after the entrypoint starts. The OPA data
now contains both the original and resolved paths, so Rego's existing
strict equality check naturally matches either.
- Add resolve_binary_in_container() helper for Linux symlink resolution
- Add from_proto_with_pid() and reload_from_proto_with_pid() to OpaEngine
- Trigger one-shot OPA rebuild after entrypoint_pid is stored
- Thread entrypoint_pid through run_policy_poll_loop for hot-reloads
- Improve deny reason with symlink debugging hint
- Add 18 new tests including hot-reload and Linux symlink e2e tests
Closes#770
* fix(sandbox): skip procfs-dependent tests when /proc/<pid>/root/ is inaccessible
The Linux-specific symlink resolution tests depend on /proc/<pid>/root/
being readable, which requires CAP_SYS_PTRACE or permissive ptrace
scope. This is unavailable in CI containers, rootless containers, and
hardened hosts. Add a procfs_root_accessible() guard that skips these
tests gracefully instead of failing.
* fix(sandbox): add explicit logging when symlink resolution fails and improve deny messages
When /proc/<pid>/root/ is inaccessible (restricted ptrace, rootless
containers, hardened hosts), resolve_binary_in_container now logs a
per-binary warning with the specific error, the path it tried, and
actionable guidance (use canonical path or grant CAP_SYS_PTRACE).
Previously this was completely silent.
The Rego deny reason for binary mismatches now leads with 'SYMLINK HINT'
and includes a concrete fix command ('readlink -f' inside the sandbox)
plus what to look for in logs if automatic resolution isn't working.
* fix(sandbox): use read_link instead of canonicalize for symlink resolution
std::fs::canonicalize resolves /proc/<pid>/root itself (a kernel
pseudo-symlink to /) which strips the prefix needed for path extraction.
This caused resolution to silently fail in all environments, not just CI.
Replace with an iterative read_link loop that walks the symlink chain
within the container namespace without resolving the /proc mount point.
Add normalize_path helper for relative symlink targets containing ..
components. Update procfs_root_accessible test guard to actually probe
the full resolution path instead of just checking path existence.
* fix(sandbox): defer symlink resolution until container filesystem is ready
The one-shot resolve ran immediately after ProcessHandle::spawn, before
the child's mount namespace and /proc/<pid>/root/ were populated. This
caused symlink_metadata to fail with ENOENT on every binary, and the
poll loop never retried because it only reloads when the policy hash
changes on the server.
Replace the synchronous resolve with an async task that probes
/proc/<pid>/root/ with retries (10 attempts, 500ms apart, 5s total).
The child's mount namespace is typically ready within a few hundred ms.
Also inline error values into warning message strings so they appear in
default log output (not just as structured tracing fields that may be
elided), and add debug-level logs before each symlink_metadata call to
aid diagnosis.
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
Policies with allowed_ips entries targeting loopback, link-local, or
unspecified ranges now fail at connection time instead of being silently
blocked at runtime. The shorthand log format for DENIED events includes
a [reason:...] suffix so operators can distinguish 'allowlist miss' from
'structurally un-allowable'. The mechanistic mapper skips proposals for
always-blocked destinations, preventing the infinite TUI notification
loop. The gateway validates proposed rules on approval as defense-in-depth.
- Extract shared IP helpers (is_always_blocked_ip, is_always_blocked_net,
is_internal_ip) to openshell_core::net
- Reject always-blocked entries in parse_allowed_ips with hard error
- Skip implicit allowed_ips synthesis for always-blocked literal IP hosts
- Add status_detail to HttpActivityBuilder for denial reason propagation
- Enrich NET and HTTP shorthand with [reason:...] for DENIED events
- Add engine: tag to HTTP shorthand (consistency with NET shorthand)
- Filter always-blocked proposals in mechanistic mapper generate_proposals
- Add validate_rule_not_always_blocked server-side defense-in-depth
- Update architecture docs, published docs, and E2E test assertions
The drop_privileges_succeeds_for_current_user test failed on non-root
systems because initgroups(3) requires CAP_SETGID even when switching
to the current user. Add a new drop_privileges_succeeds_for_current_group
test that omits run_as_user, bypassing initgroups while still exercising
setgid() and the GID post-condition verification. Mark the original
full-path test with #[ignore] so it can be run explicitly as root.
Fixes#622
TLD wildcards like *.com were accepted by policy validation but silently
failed at the proxy layer — 0 bytes returned, no denial logged. Now
rejected at submission time with a clear error message.
Closes#787
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* feat(sandbox): integrate OCSF structured logging for all sandbox events
WIP: Replace ad-hoc tracing calls with OCSF event builders across all
sandbox subsystems (network, SSH, process, filesystem, config, lifecycle).
- Register ocsf_logging_enabled setting (defaults false)
- Replace stdout/file fmt layers with OcsfShorthandLayer
- Add conditional OcsfJsonlLayer for /var/log/openshell-ocsf.log
- Update LogPushLayer to extract OCSF shorthand for gRPC push
- Migrate ~106 log sites to OCSF builders (NetworkActivity, HttpActivity,
SshActivity, ProcessActivity, DetectionFinding, ConfigStateChange,
AppLifecycle)
- Add openshell-ocsf to all Docker build contexts
* fix(scripts): attach provider to all smoke test phases to avoid rate limits
GitHub's unauthenticated API rate limit (60/hour) causes flaky 403s for
Phases 1, 2, and 4. Fix by attaching the provider to all sandboxes and
upgrading the Phase 1 policy to L7 so credential injection works.
Phase 4 (tls:skip) cannot inject credentials by design, so relax the
assertion to accept either 200 or 403 from upstream -- both prove the
proxy forwarded the request.
* fix(ocsf): remove timestamp from shorthand format to avoid double-timestamp
The display layer (gateway logs, TUI, sandbox logs CLI) already prepends
a timestamp. Having one in the shorthand output too produces redundant
double-timestamps like:
15:49:11 sandbox INFO 15:49:11.649 I NET:OPEN ALLOWED ...
Now the shorthand is just the severity + structured content:
15:49:11 sandbox INFO I NET:OPEN ALLOWED ...
* refactor(ocsf): replace single-char severity with bracketed labels
Replace cryptic single-character severity codes (I/L/M/H/C/F) with
readable bracketed labels: [LOW], [MED], [HIGH], [CRIT], [FATAL].
Informational severity (the happy-path default) is omitted entirely to
keep normal log output clean and avoid redundancy with the tracing-level
INFO that the display layer already provides.
Before: sandbox INFO I NET:OPEN ALLOWED ...
After: sandbox INFO NET:OPEN ALLOWED ...
Before: sandbox INFO M NET:OPEN DENIED ...
After: sandbox INFO [MED] NET:OPEN DENIED ...
* feat(sandbox): use OCSF level label for structured events in log push
Set the level field to 'OCSF' instead of 'INFO' for OCSF events in the
gRPC log push. This visually distinguishes structured OCSF events from
plain tracing output in the TUI and CLI sandbox logs:
sandbox OCSF NET:OPEN [INFO] ALLOWED python3(42) -> api.example.com:443
sandbox OCSF NET:OPEN [MED] DENIED python3(42) -> blocked.com:443
sandbox INFO Fetching sandbox policy via gRPC
* fix(sandbox): convert new Landlock path-skip warning to OCSF
PR #677 added a warn!() for inaccessible Landlock paths in best-effort
mode. Convert to ConfigStateChangeBuilder with degraded state so it
flows through the OCSF shorthand format consistently.
* fix(sandbox): use rolling appender for OCSF JSONL file
Match the main openshell.log rotation mechanics (daily, 3 files max)
instead of a single unbounded append-only file. Prevents disk exhaustion
when ocsf_logging_enabled is left on in long-running sandboxes.
* fix(sandbox): address reviewer warnings for OCSF integration
W1: Remove redundant 'OCSF' prefix from shorthand file layer — the
class name (NET:OPEN, HTTP:GET) already identifies structured events
and the LogPushLayer separately sets the level field.
W2: Log a debug message when OCSF_CTX.set() is called a second time
instead of silently discarding via let _.
W3: Document the boundary between OCSF-migrated events and intentionally
plain tracing calls (DEBUG/TRACE, transient, internal plumbing).
W4: Migrate remaining iptables LOG rule failure warnings in netns.rs
(IPv4 TCP/UDP, IPv6 TCP/UDP) to ConfigStateChangeBuilder for
consistency with the IPv4 bypass rule failure already migrated.
W5: Migrate malformed inference request warn to NetworkActivity with
ActivityId::Refuse and SeverityId::Medium.
W6: Use Medium severity for L7 deny decisions (both CONNECT tunnel and
FORWARD proxy paths) to match the CONNECT deny severity pattern.
Allows and audits remain Informational.
* refactor(sandbox): rename ocsf_logging_enabled to ocsf_json_enabled
The shorthand logs are already OCSF-structured events. The setting
specifically controls the JSONL file export, so the name should reflect
that: ocsf_json_enabled.
* fix(ocsf): add timestamps to shorthand file layer output
The OcsfShorthandLayer writes directly to the log file with no outer
display layer to supply timestamps. Add a UTC timestamp prefix to every
line so the file output matches what tracing::fmt used to provide.
Before: CONFIG:VALIDATED [INFO] Validated 'sandbox' user exists in image
After: 2026-04-01T15:49:11.649Z CONFIG:VALIDATED [INFO] Validated ...
* fix(docker): touch openshell-ocsf source to invalidate cargo cache
The supervisor-workspace stage touches sandbox and core sources to force
recompilation over the rust-deps dummy stubs, but openshell-ocsf was
missing. This caused the Docker cargo cache to use stale ocsf objects
from the deps stage, preventing changes to the ocsf crate (like the
timestamp fix) from appearing in the final binary.
Also adds a shorthand layer test verifying timestamp output, and drafts
the observability docs section.
* fix(ocsf): add OCSF level prefix to file layer shorthand output
Without a level prefix, OCSF events in the log file have no visual
anchor at the position where standard tracing lines show INFO/WARN.
This makes scanning the file harder since the eye has nothing consistent
to lock onto after the timestamp.
Before: 2026-04-01T04:04:13.065Z CONFIG:DISCOVERY [INFO] ...
After: 2026-04-01T04:04:13.065Z OCSF CONFIG:DISCOVERY [INFO] ...
* fix(ocsf): clean up shorthand formatting for listen and SSH events
- Fix double space in NET:LISTEN, SSH:LISTEN, and other events where
action is empty (e.g., 'NET:LISTEN [INFO] 10.200.0.1' -> 'NET:LISTEN [INFO] 10.200.0.1')
- Add listen address to SSH:LISTEN event (was empty)
- Downgrade SSH handshake intermediate steps (reading preface, verifying)
from OCSF events to debug!() traces. Only the final verdict
(accepted/denied) is an OCSF event now, reducing noise from 3 events
to 1 per SSH connection.
- Apply same spacing fix to HTTP shorthand for consistency.
* docs(observability): update examples with OCSF prefix and formatting fixes
Align doc examples with the deployed output:
- Add OCSF level prefix to all shorthand examples in the log file
- Show mixed OCSF + standard tracing in the file format section
- Update listen events (no double space, SSH includes address)
- Show one SSH:OPEN per connection instead of three
- Update grep patterns to use 'OCSF NET:' etc.
* docs(agents): add OCSF logging guidance to AGENTS.md
Add a Sandbox Logging (OCSF) section to AGENTS.md so agents have
in-context guidance for deciding whether new log emissions should use
OCSF structured logging or plain tracing. Covers event class selection,
severity guidelines, builder API usage, dual-emit pattern for security
findings, and the no-secrets rule.
Also adds openshell-ocsf to the Architecture Overview table.
* fix: remove workflow files accidentally included during rebase
These files were already merged to main in separate PRs. They got
pulled into our branch during rebase conflict resolution for the
deleted docs-preview-pr.yml file.
* docs(observability): use sandbox connect instead of raw SSH
Users access sandboxes via 'openshell sandbox connect', not direct SSH.
* fix(docs): correct settings CLI syntax in OCSF JSON export page
The settings CLI requires --key and --value named flags, not positional
arguments. Also fix the per-sandbox form: the sandbox name is a
positional argument, not a --sandbox flag.
* fix(e2e): update log assertions for OCSF shorthand format
The E2E tests asserted on the old tracing::fmt key=value format
(action=allow, l7_decision=audit, FORWARD, L7_REQUEST, always-blocked).
Update to match the new OCSF shorthand (ALLOWED/DENIED, HTTP:, NET:,
engine:ssrf, policy:).
* feat(sandbox): convert WebSocket upgrade log calls to OCSF
PR #718 added two log calls for WebSocket upgrade handling:
- 101 Switching Protocols info → NetworkActivity with Upgrade activity.
This is a significant state change (L7 enforcement drops to raw relay).
- Unsolicited 101 without client Upgrade header → DetectionFinding with
High severity. A non-compliant upstream sending 101 without a client
Upgrade request could be attempting to bypass L7 inspection.
* fix(install): restrict tar extraction to expected binary member
Prevents CWE-22 path traversal by extracting only the expected APP_NAME
member instead of the full archive contents. Adds --no-same-owner and
--no-same-permissions for defense-in-depth.
OS-20
* fix(deploy): quote registry credentials in YAML heredocs
Wraps username/password values with a yaml_quote helper to prevent YAML
injection from special characters in registry credentials (CWE-94).
Applied to all three heredoc blocks that emit registries.yaml auth.
OS-23
* fix(server): redact session token in SSH tunnel rate-limit log
Logs only the last 4 characters of bearer tokens to prevent credential
exposure in log aggregation systems (CWE-532).
OS-18
* fix(server): escape gateway_display in auth connect page
Applies html_escape() to the Host/X-Forwarded-Host header value before
rendering it into the HTML template, preventing HTML injection (CWE-79).
OS-17
* fix(server): prevent XSS via code param with validation and proper JS escaping
Adds server-side validation rejecting confirmation codes that do not
match the CLI-generated format, replaces manual JS string escaping with
serde_json serialization (handling U+2028/U+2029 line terminators), and
adds a Content-Security-Policy header with nonce-based script-src.
OS-16
* fix(sandbox): add byte cap and idle timeout to streaming inference relay
Prevents resource exhaustion from upstream inference endpoints that stream
indefinitely or hold connections open. Adds a 32 MiB total body limit
and 30-second per-chunk idle timeout (CWE-400).
OS-21
* fix(policy): narrow port field from u32 to u16 to reject invalid values
Prevents meaningless port values >65535 from being accepted in policy
YAML definitions. The proto field remains uint32 (protobuf has no u16)
with validation at the conversion boundary.
OS-22
* fix(deps): migrate from archived serde_yaml to serde_yml
Replaces serde_yaml 0.9 (archived, RUSTSEC-2024-0320) with serde_yml
0.0.12, a maintained API-compatible fork. All import sites updated
across openshell-policy, openshell-sandbox, and openshell-router.
OS-19
* fix(server): re-validate sandbox-submitted security_notes and cap hit_count
The gateway now re-runs security heuristics on proposed policy chunks
instead of trusting sandbox-provided security_notes, validates host
wildcards, caps hit_count at 100, and clamps confidence to [0,1]. The
TUI approve-all path is updated to use ApproveAllDraftChunks RPC which
respects the security_notes filtering gate (CWE-284, confused deputy).
OS-15
* chore: apply cargo fmt and update Cargo.lock for serde_yml
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
Detect 101 Switching Protocols in relay_response() and switch to raw
bidirectional TCP relay instead of re-entering the HTTP parsing loop.
Previously, is_bodiless_response() treated 101 as a generic 1xx
informational response, forwarding only the headers and returning to
the HTTP parsing loop. After a 101, subsequent bytes are upgraded
protocol frames (e.g. WebSocket), not HTTP — causing the relay to
block or silently drop all post-upgrade traffic.
Changes:
- Add RelayOutcome enum (Reusable/Consumed/Upgraded) replacing bool
return type across L7Provider::relay trait and all relay functions
- Detect 101 before generic 1xx handler in relay_response(), capture
overflow bytes, return RelayOutcome::Upgraded
- Validate client sent Upgrade + Connection: Upgrade headers before
accepting 101 (rejects unsolicited upgrades from non-compliant
upstream servers)
- Extract shared handle_upgrade() helper used by both relay_rest()
and relay_passthrough_with_credentials()
- Add l7_decision=allow_upgrade audit log annotation for upgrades
- Add unit tests for 101 overflow capture, unsolicited 101 rejection,
and client_requested_upgrade header validation
- Add integration test: WebSocket echo through L7Provider::relay
Fixes: #652
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* fix(sandbox): add GPU device nodes and nvidia-persistenced to landlock baseline
Landlock READ_FILE/WRITE_FILE restricts open(2) on character device files
even when DAC permissions would otherwise allow it. GPU sandboxes need
/dev/nvidiactl, /dev/nvidia-uvm, /dev/nvidia-uvm-tools, /dev/nvidia-modeset,
and per-GPU /dev/nvidiaX nodes in the policy to allow NVML initialization.
Additionally, CDI bind-mounts /run/nvidia-persistenced/socket into the
container. NVML tries to connect to this socket at init time; if the
directory is not in the landlock policy, it receives EACCES (not
ECONNREFUSED), which causes NVML to abort with NVML_ERROR_INSUFFICIENT_PERMISSIONS
even though nvidia-persistenced is optional.
Both classes of paths are auto-added to the baseline when /dev/nvidiactl is
present. Per-GPU device nodes are enumerated at runtime to handle multi-GPU
configurations.
Remove Box::leak usage that permanently leaked one String allocation per
forward proxy request. Write the rewritten request line directly to the
output buffer during iteration instead of mutating a Vec<&str> element.
Closes#709
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
Replace flat pty_master/input_sender/pty_request fields in SshHandler
with a HashMap<ChannelId, ChannelState> so each channel tracks its own
PTY resources independently. This fixes window_change_request resizing
the wrong PTY when multiple channels are open simultaneously.
Also fixes ioctl UB in set_winsize (pass &winsize not winsize by value)
and adds warn! logging for unknown channels across all handlers.
Resolves#543
* perf(sandbox): streaming SHA256, spawn_blocking for identity resolution
Key changes:
- Replace full file read + SHA256 with streaming 64KB-buffered hash
(saves 124MB allocation for node binary)
- Wrap evaluate_opa_tcp in spawn_blocking to prevent blocking tokio
runtime during heavy /proc I/O and SHA256 computation
- Add file-based perf logging for profiling proxy latency phases
Profiling data (node binary, 124MB):
- Cold TOFU: ~890ms (read+hash), warm: 0ms (cache hit)
- evaluate_opa_tcp: cold=1002ms, warm=11ms
- OPA evaluation: 1ms
- DNS+TCP connect: 166-437ms
Made-with: Cursor
Signed-off-by: Rafael Koike <koike.rafael@gmail.com>
* refactor(sandbox): replace perf_log with tracing::debug
Replace the custom file-based perf_log() helper with standard
tracing::debug!() macros as requested in PR review. This removes
the custom log file writes to /var/log/openshell-perf.log and
routes all performance timing through the tracing framework at
DEBUG level, consistent with the rest of the codebase.
Made-with: Cursor
* refactor(sandbox): reduce tracing to 6 key diagnostic logs
Address PR review feedback:
1. Remove ~20 inner-phase timing logs, keeping only the 6 that tell
the full diagnostic story:
- evaluate_opa_tcp TOTAL (proxy.rs)
- dns_resolve_and_tcp_connect (proxy.rs)
- file_sha256 (procfs.rs)
- verify_or_cache CACHE HIT / CACHE MISS / TOTAL cold (identity.rs)
2. Restore intent-describing comments that were replaced by timing logs:
- "TOFU verify the immediate binary" (proxy.rs)
- "Walk the process tree upward to collect ancestor binaries" (proxy.rs)
- "Collect cmdline paths for script-based binary detection." (proxy.rs)
- "First: scan descendants of the entrypoint process" (procfs.rs)
- "Fallback: scan all of /proc in case the process isn't in the tree" (procfs.rs)
- "Skip PIDs we already checked" (procfs.rs)
3. Preserve file path in file_sha256 read errors instead of losing
context via into_diagnostic().
4. Tests: 293 passed, 1 pre-existing failure (drop_privileges), 1 ignored.
Made-with: Cursor
* style(sandbox): apply rustfmt formatting to debug macros
---------
Signed-off-by: Rafael Koike <koike.rafael@gmail.com>
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* feat(sandbox): add L7 query parameter matchers
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* fix(sandbox): decode + as space in query params and validate glob syntax
Three improvements from PR #617 review:
1. Decode + as space in query string values per the
application/x-www-form-urlencoded convention. This matches Python's
urllib.parse, JavaScript's URLSearchParams, Go's url.ParseQuery, and
most HTTP frameworks. Literal + should be sent as %2B.
2. Add glob pattern syntax validation (warnings) for query matchers.
Checks for unclosed brackets and braces in glob/any patterns. These
are warnings (not errors) because OPA's glob.match is forgiving,
but they surface likely typos during policy loading.
3. Add missing test cases: empty query values, keys without values,
unicode after percent-decoding, empty query strings, and literal +
via %2B encoding.
* fix(sandbox): add missing query_params field in forward proxy L7 request info
* style(sandbox): fix formatting in proxy L7 query param parsing
---------
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* fix(sandbox): handle per-path Landlock errors instead of abandoning entire ruleset
A single missing path (e.g., /app in containers without that directory)
caused PathFd::new() to propagate an error out of the entire Landlock
setup closure. Under BestEffort mode, this silently disabled all
filesystem restrictions for the sandbox.
Changes:
- Extract try_open_path() and classify_path_error() helpers that handle
PathFd failures per-path instead of per-ruleset
- BestEffort mode: skip inaccessible paths with a warning, apply
remaining rules
- HardRequirement mode: fail immediately on any inaccessible path
- Add zero-rule safety check to prevent applying an empty ruleset that
would block all filesystem access
- Pre-filter system-injected baseline paths (e.g., /app) in enrichment
functions so missing paths never reach Landlock
- Add unit tests for try_open_path, classify_path_error, and error
classification for ENOENT, EACCES, ELOOP, ENAMETOOLONG, ENOTDIR
- Update user-facing docs and architecture docs with Landlock behavior
tables, baseline path filtering, and compatibility mode semantics
- Fix stale ABI::V1 references in docs (code uses ABI::V2)
Closes#664
* fix(sandbox): use debug log for NotFound in Landlock best-effort mode
NotFound errors for stale baseline paths (e.g. /app persisted in the
server-stored policy but absent in this container) are expected in
best-effort mode. Downgrade from warn! to debug! so the message does
not leak into SSH exec stdout (the pre_exec hook inherits the tracing
subscriber whose writer targets fd 1).
Genuine errors (permission denied, symlink loops, etc.) remain at warn!
for operator visibility.
Also move custom_image e2e marker from /opt to /etc (a Landlock baseline
read-only path) since the security fix now properly enforces filesystem
restrictions.
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
The CL/TE desynchronisation guard added in #663 for the REST path was
not applied to the inference request parser. A request containing both
Content-Length and Transfer-Encoding headers could be interpreted
differently by the proxy and the upstream server, enabling HTTP request
smuggling (CWE-444, RFC 7230 Section 3.3.3).
Add the same rejection check and tests mirroring the REST parser
coverage, including TE substring validation.
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
Co-authored-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
* refactor(l7): export evaluate_l7_request for cross-module use
Make evaluate_l7_request() public so the forward proxy path can
evaluate individual requests against L7 policy without going
through the full relay_with_inspection() loop.
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
* fix(proxy): add L7 inspection to forward proxy path
The forward proxy previously rejected all requests to endpoints with
L7 rules (blanket 403), forcing clients through the CONNECT tunnel.
This meant policies like read-only (allow GET, block POST) had no
effect on plain http:// requests through the forward proxy.
Replace the blanket rejection with actual L7 evaluation:
- Query L7 config for the endpoint (same as before)
- Clone the OPA engine and evaluate the request method/path
- Allow if L7 policy permits, deny with 403 if enforcement is enforce
- Audit mode: log but allow (matching CONNECT path behavior)
- Fail-closed: deny on evaluation errors
The forward proxy uses Connection: close (one request per connection),
so a single evaluation suffices — no relay loop needed.
Update e2e tests to validate the new behavior:
- GET /allowed → 200 (L7 policy allows)
- POST /allowed → 403 (L7 policy denies, enforcement: enforce)
Update security-policy.md to reflect the new forward proxy L7 behavior.
Closes#643
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
* style(proxy): apply cargo fmt formatting
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
---------
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
Co-authored-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
* fix(l7): reject duplicate Content-Length headers to prevent request smuggling
Both parse_body_length() in rest.rs and try_parse_http_request() in
inference.rs silently accepted multiple Content-Length headers,
overwriting with the last value seen. Per RFC 7230 Section 3.3.3,
a message with multiple Content-Length headers with differing values
must be rejected to prevent HTTP request smuggling (CWE-444).
An attacker could send conflicting Content-Length values causing the
proxy and downstream server to disagree on message boundaries.
Fix:
- rest.rs: detect duplicate CL headers with differing values and
return an error before forwarding
- inference.rs: add ParseResult::Invalid variant; detect duplicate
CL headers and return Invalid with a descriptive reason
- proxy.rs: handle ParseResult::Invalid by sending HTTP 400 and
denying the connection
Closes#637
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
* fix(l7): address review feedback on Content-Length smuggling defense
- inference.rs: reject unparseable Content-Length values instead of
silently defaulting to 0 via unwrap_or(0)
- rest.rs: reject unparseable Content-Length values so a valid+invalid
duplicate pair cannot bypass the differing-values check
- rest.rs: fix Transfer-Encoding substring match (.contains("chunked")
→ split/trim exact match) to align with inference.rs and prevent
false positives on values like "chunkedx"
- proxy.rs: log parsing details server-side via tracing::warn and
return generic "Bad Request" body instead of leaking internal
parsing reasons to sandboxed code
- Add tests for all new rejection paths in inference.rs and rest.rs
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
* style(l7): apply cargo fmt formatting
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
---------
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
Co-authored-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
Three issues prevent k3s from starting on kernels where the nf_tables
xt extension bridge (nft_compat) is unavailable:
1. kube-router's network policy controller uses the xt_comment iptables
extension and panics on startup with "Extension comment revision 0
not supported, missing kernel module?" Pass --disable-network-policy
to k3s so the controller never runs. The NSSH1 HMAC handshake remains
the primary sandbox SSH isolation boundary, so this does not weaken
the effective security posture.
2. flannel and kube-proxy also fail to insert rules via the nf_tables
iptables backend on the same kernels. Add an xt_comment probe at
cluster-entrypoint startup; if the probe fails, switch to
iptables-legacy via update-alternatives before any other netfilter
work so that flannel, kube-proxy, and the DNS proxy all use a
consistent backend.
3. The br_netfilter kernel module must be loaded on the host for
iptables rules to apply to pod bridge traffic. Without it, ClusterIP
DNAT (including kube-dns at 10.43.0.10) is never applied to pod
packets, causing silent DNS timeouts deep in the health-check loop.
Add an early check that fails fast with an actionable error message
if the module is not present, instructing the user to run
`sudo modprobe br_netfilter` on the host.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(sandbox): emit warning when Landlock filesystem sandbox degrades silently
BestEffort Landlock previously swallowed failures at debug level, making
sandbox bypass invisible to operators at default log levels. Upgrade the
degradation log to warn with an actionable message pointing to the
hard_requirement setting. Add info-level startup log showing the
requested ABI and path counts so operators always know what Landlock
protections are active.
Closes#584
* fix(sandbox): revert unintended ABI bump from V2 to V5
Signed-off-by: John Myers <johntmyers@users.noreply.github.com>
---------
Signed-off-by: John Myers <johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
relay_passthrough_with_credentials called relay_http_request_with_resolver
(which internally relays the upstream response back to the client) and then
immediately called relay_response_to_client a second time. The second call
blocked forever waiting for a response that would never arrive, deadlocking
every CONNECT tunnel after its first request/response pair.
This caused npm install (and any HTTP/1.1 keep-alive client) to hang
indefinitely when routed through the sandbox proxy without L7 rules.
The L7-inspection path (relay_rest) was not affected — it correctly makes
a single call to relay_http_request_with_resolver.
* fix(sandbox,server): fix chunk merge duplicates and OPA variable collision with overlapping policies
Two related bugs triggered when a draft rule approval creates a second
policy entry for the same host:port:
1. merge_chunk_into_policy looked up existing rules by chunk.rule_name
(auto-generated as allow_{host}_{port}), which never matched the
user's original rule name. Now scans all network_policies entries
for a host:port endpoint match before falling back to insertion,
and merges allowed_ips into the existing endpoint.
2. The Rego allow_request rule and _matching_endpoint_configs
comprehension used 'some ep; ep := policy.endpoints[_]' which
caused regorus to error with 'duplicated definition of local
variable ep' when multiple policies covered the same host:port.
Refactored to isolate endpoint iteration inside helper functions
(_policy_allows_l7, _policy_endpoint_configs) so variables are
scoped per-policy evaluation.
Refs: #567
* test(e2e): add overlapping policy tests and update FWD-2 for implicit allowed_ips
- Update FWD-2 (test_forward_proxy_denied_without_allowed_ips ->
test_forward_proxy_allows_private_ip_host_without_allowed_ips):
literal IP host no longer requires explicit allowed_ips, expects 200.
- Add OVL-1: overlapping L4 policies for same host:port must not crash
OPA and should allow forward proxy connections.
- Add OVL-2: overlapping L7 policies for same host:port must not crash
OPA and should allow CONNECT tunnel establishment.
Refs: #567
* style: apply cargo fmt formatting
* test(e2e): update SSRF-3 and SSRF-6 for implicit allowed_ips behavior
SSRF-6: Private IP with literal IP host now gets implicit allowed_ips
from PR #570, so CONNECT returns 200 instead of 403.
SSRF-3: Loopback is still blocked but via the always-blocked path
(implicit allowed_ips is synthesized, then resolve_and_check_allowed_ips
catches it). Log message says 'always-blocked' instead of 'internal
address'.
* fix(e2e): use negative assertion for SSRF-6 when nothing listens on target port
When the SSRF check passes but nothing listens on the target port,
recv() returns empty bytes. Use 'assert 403 not in' (matching SSRF-4
pattern) instead of 'assert 200 in'.
* fix(e2e): update provider tests for redacted credential values
PR #569 changed credential redaction from clearing the map to
replacing values with 'REDACTED'. Update e2e assertions to expect
credential keys with REDACTED values instead of an empty map.
When a policy endpoint specifies a literal IP address as the host
(e.g. host: 192.168.86.157), the user has explicitly declared intent
to allow that destination. The SSRF guard requiring allowed_ips was
redundant for this case and forced users to duplicate the IP.
Synthesize an implicit allowed_ips entry when the host parses as an IP
address, so the existing allowlist-validation path is used instead of
the blanket internal-IP rejection. Loopback and link-local addresses
remain blocked by resolve_and_check_allowed_ips.
Applies to both the CONNECT and FORWARD proxy paths.
Refs: #567
* feat(sandbox): auto-detect TLS and terminate unconditionally for credential injection
Closes#533
The proxy now auto-detects TLS by peeking the first bytes of each
connection. When TLS is detected, it terminates unconditionally —
enabling credential injection and optional L7 inspection without
requiring explicit 'tls: terminate' in the policy.
