The default socket path assumed Linux conventions ($XDG_RUNTIME_DIR or
/run/user/{uid}), which do not exist on macOS. Use the podman machine
socket at $HOME/.local/share/containers/podman/machine/podman.sock
on macOS and show a targeted "podman machine start" hint when missing.
Signed-off-by: Florent Benoit <fbenoit@redhat.com>
* feat(podman): add Podman compute driver for rootless sandbox management
Adds openshell-driver-podman, a new compute driver that manages OpenShell
sandboxes as rootless Podman containers via the Podman REST API over a
Unix socket. Enables local workstation sandboxes without Kubernetes.
Driver features:
- Bridge networking with ephemeral host-port mapping for rootless SSH reachability
- Named volumes for workspace storage, Podman native health checks, GPU via CDI
- Supervisor binary sideloaded via image volume mount (BYOC-compatible)
- SSH handshake secret injected via Podman secrets API (not plaintext env)
- Typed ContainerSpec structs, input validation, and path-traversal guards
- Cgroups v2 required; fails fast on v1 hosts
- Bounded event stream buffer; watch stream reconnection handled by server watch_loop
- Graceful shutdown and standalone driver binary with gRPC bridge
Rootless-specific fixes:
- Skip drop_privileges when user namespace lacks SETUID/SETGID/DAC_READ_SEARCH caps
- Add /run/netns tmpfs mount for ip netns in rootless containers
- Use secret_env map (not secrets array) for env-var injection in libpod API
- Resolve SSH endpoint to 127.0.0.1:<host_port> instead of unreachable bridge IP
Server/sandbox hardening:
- Split loopback and link-local SSRF gates; Podman/VM drivers allow loopback
- Close SSRF bypass in SSH tunnel Host path by resolving DNS before connecting
- Prevent OPENSHELL_* env var override by user-supplied spec environment maps
- Disable SQLite pool idle_timeout/max_lifetime for in-memory databases
- Emit deleted_event on 404-during-inspect instead of regressing sandbox phase
- Key delete cleanup by stable sandbox_id to survive container label drift
CLI fixes:
- Restore --name as a named flag on sandbox create (not positional)
- Fix exec command arg parsing to not consume sandboxed-command flags
- Propagate SSH verbosity via OPENSHELL_SSH_LOG_LEVEL
Build tooling:
- Add tasks/scripts/container-engine.sh: auto-detects Podman or Docker, exposes
unified ce_* helpers; all build/cluster/VM scripts updated to use it
- Add docker:build:supervisor mise task for standalone supervisor image
- Add openshell-driver-podman to Dockerfile.images pre-fetch/build stages
- Add e2e/rust/e2e-podman.sh and e2e:podman mise task for full lifecycle testing
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(driver-podman): derive grpc endpoint from server bind port
When a user starts the gateway on a non-default port (e.g. --port 8081),
sandbox containers were receiving OPENSHELL_ENDPOINT pointing at the
default port 8080. The driver's auto-detection fallback read
OPENSHELL_BIND_ADDRESS from the environment, which was stale or unset,
and fell back to DEFAULT_SERVER_PORT.
Add gateway_port to PodmanComputeConfig and thread config.bind_address.port()
from the server into the driver so the fallback uses the actual listening
port. Remove the OPENSHELL_BIND_ADDRESS env var read and the
extract_port_from_bind_address helper which are no longer needed.
Add --gateway-port / OPENSHELL_GATEWAY_PORT to the standalone driver
binary for parity when the driver is run outside the embedded server path.
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(driver-podman): address PR feedback on env test safety and cluster DNS docs
Replace hand-rolled unsafe TempEnvVar RAII guard with temp_env::with_vars
and a static ENV_LOCK mutex, fixing a data race in parallel test execution.
The prior safety comment incorrectly claimed Cargo runs tests single-threaded.
Update debug-openshell-cluster skill to accurately document the DNS proxy
strategy (setup_dns_proxy + public DNS fallback) and clarify the separation
between cluster DNS and sandbox agent DNS enforcement.
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(e2e): resolve CI failures in auth timeout, test harness, and formatting
- Short-circuit browser_auth_flow when OPENSHELL_NO_BROWSER=1 instead
of waiting the full 120s AUTH_TIMEOUT for a callback that never arrives
- Add timeout to SandboxGuard::create() and create_with_upload() to
prevent indefinite hangs (matches create_keep() which already had one)
- Add missing '--' separator in no_proxy test before command args
- Add #![cfg(feature = "e2e")] gate to sandbox_lifecycle.rs
- Run cargo fmt on openshell-driver-podman
- Refine cluster DNS docs for Podman in debug-openshell-cluster skill
Signed-off-by: Adam Miller <admiller@redhat.com>
* refactor(server): remove allows_loopback_endpoints from ComputeRuntime
SSRF protection is now handled at the network and proxy layers
(openshell-core net.rs, openshell-sandbox proxy.rs) rather than
requiring per-driver flags on ComputeRuntime. Update architecture
docs to reflect supervisor relay SSH transport and add rootless
networking deep-dive.
Signed-off-by: Adam Miller <admiller@redhat.com>
---------
Signed-off-by: Adam Miller <admiller@redhat.com>
Previously `openshell sandbox upload <local-dir> <remote-dir>` flattened
the local directory's contents into the remote destination, dropping the
source directory's basename. This diverged from the standard `scp -r` /
`cp -r` semantics that operators expect from any *nix-style transfer
tool, and could silently shadow files at the destination.
Now an upload of `foo` to `/sandbox/dest/` lands at `/sandbox/dest/foo/`,
matching `scp -r foo remote:/sandbox/dest/` and `cp -r foo /sandbox/dest/`.
Paths without a meaningful basename (`.`, `/`) preserve the legacy flat
extraction.
Closes#885
* feat(server): add Prometheus metrics infrastructure and gRPC/HTTP request metrics
Add metrics exposition via a dedicated server port (--metrics-port,
default disabled) following the same optional-listener pattern as the
health port. The metrics crate facade records counters and histograms
in the MultiplexedService hot path, and a PrometheusHandle renders them
at GET /metrics on the dedicated port.
Metrics added:
- openshell_server_grpc_requests_total (counter: method, code)
- openshell_server_grpc_request_duration_seconds (histogram: method, code)
- openshell_server_http_requests_total (counter: path, status)
- openshell_server_http_request_duration_seconds (histogram: path, status)
* feat(helm): add metrics port to openshell-server helm chart
Expose the Prometheus metrics endpoint in the helm chart by adding
service.metricsPort (default 9090) to values, the --metrics-port arg
and container port to the statefulset, and a metrics service port.
Set metricsPort to 0 to disable.
Ubuntu Noble's git links against libcurl-gnutls, which does not read
SSL_CERT_FILE. Without GIT_SSL_CAINFO, `git clone` over HTTPS fails
inside every community sandbox image with "server certificate
verification failed".
Add GIT_SSL_CAINFO to tls_env_vars pointing at the same combined CA
bundle already used by CURL_CA_BUNDLE / REQUESTS_CA_BUNDLE.
Fixes#790FixesNVIDIA/NemoClaw#2270FixesNVIDIA/NemoClaw#1828
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
Closes#879
Resolve policy hostnames against the sandbox's /etc/hosts before DNS so Kubernetes hostAliases are visible to the proxy while keeping the existing allowed_ips SSRF enforcement semantics intact.
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Closes#905
Skip proxy baseline read-write enrichment when a path is already explicitly configured as read-only, add regression coverage for the proto and local policy code paths, and document the precedence rule in the sandbox architecture docs.
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
the transfer of very large sandbox images to containerd can
timeout depending on the size of the image and the speed of
the local host.
Made-with: Cursor
The 30s read_timeout on the shared kube client was killing the
long-lived watch streams during idle periods, causing a reconnect
cycle every 30 seconds. Use a separate client with no read_timeout
for watch_sandboxes so the streams stay open indefinitely.
Move /health, /healthz, and /readyz to a dedicated plaintext HTTP port
(default 8081) so Kubernetes probes work without mTLS client certificates.
- Add health_bind_address to Config with --health-port CLI arg
- Spawn standalone axum::serve for health_router on the health port
- Remove health routes from the main multiplexed HTTP router
- Update Helm statefulset probes from tcpSocket to httpGet on health port
- Fix cluster-healthcheck.sh to open/close TCP without sending data,
avoiding InvalidContentType TLS errors in the gateway log
Wrap gRPC and HTTP services with TraceLayer to log method, path,
status, and latency for every request. Health/readiness probe
endpoints are logged at debug level to reduce noise.
* fix(sandbox): canonicalize HTTP request-targets before L7 policy evaluation
The L7 REST proxy evaluated OPA path rules against the raw request-target
and forwarded the raw header block byte-for-byte to the upstream. The
upstream normalized `..`, `%2e%2e`, and `//` before dispatching, so a
compromised agent inside the sandbox could bypass any path-based allow
rule by prefixing the blocked path with an allowed one (e.g.
`GET /public/../secret` matches `/public/**` at the proxy but the
upstream serves `/secret`). The inference-routing detection had the same
normalization gap via `normalize_inference_path`, which only stripped
scheme+authority and preserved dot-segments verbatim.
- Add `crates/openshell-sandbox/src/l7/path.rs` with
`canonicalize_request_target`. Percent-decodes (with uppercase hex
re-emission), resolves `.` / `..` segments per RFC 3986 5.2.4,
collapses doubled slashes, strips trailing `;params`, rejects
fragments / raw non-ASCII / control bytes / encoded slashes (unless
explicitly enabled per-endpoint), and returns a `rewritten` flag so
callers know when to rewrite the outbound request line.
- Wire the canonicalizer into `rest.rs::parse_http_request`. The
canonical path is the `target` on `L7Request` that OPA sees, and
when the canonical form differs from the raw input the request line
is rebuilt in `raw_header` so the upstream dispatches on the exact
bytes the policy evaluated.
- Canonicalize the forward (plain HTTP proxy) path at
`proxy.rs`'s second L7 evaluation site. A non-canonical request-target
is rejected with 400 Bad Request and an OCSF `NetworkActivity` Fail
event rather than silently passed through.
- Replace the old `normalize_inference_path` body with a call to the
canonicalizer. On canonicalization error the raw path is returned
(so inference detection simply misses and the normal forward path
handles and rejects).
- Document the invariant in `sandbox-policy.rego`: `input.request.path`
is pre-canonicalized so rules must not attempt defensive matching
against `..` or `%2e%2e`.
- Architecture doc (`architecture/sandbox.md`) lists the new module.
- 24 new unit tests cover dot segments, percent-encoded dot segments,
double-slash collapse, encoded-slash reject/opt-in, null/control byte
rejection, legitimate percent-encoded bytes round-tripping, mixed-case
percent normalization, fragment rejection, absolute-form stripping,
empty / length-bounded / non-ASCII handling, and regression payloads
for the specific bypasses above.
Tracks OS-99.
* fix(sandbox): close forward-proxy parser-differential and wire allow_encoded_slash
Addresses two review findings on the L7 path canonicalization change.
1. `handle_forward_proxy` previously evaluated OPA on the canonical path
but wrote the raw path to the upstream via `rewrite_forward_request`,
leaving the parser-differential open for plain-HTTP forward-proxy
traffic even though it was closed for the L7 tunnel path. `path` is
now reassigned to the canonical form inside the L7 block before the
outbound write, so the bytes dispatched to the upstream match what
OPA evaluated. Covered by a new regression test against
`rewrite_forward_request`.
2. `allow_encoded_slash` is now wired through the YAML policy schema,
the proto `NetworkEndpoint` message, the Rego endpoint config, and
the `L7EndpointConfig` → `RestProvider` canonicalization options.
Operators can opt in per-endpoint for upstreams like GitLab that
embed `%2F` in namespaced resource paths; the default remains
strict. The `RestProvider` gains a constructor
`RestProvider::with_options` so `relay_rest` constructs a provider
with per-endpoint options while `relay_passthrough_with_credentials`
retains strict defaults.
Integration tests:
- `parse_http_request_canonicalizes_target_and_rewrites_raw_header`
drives a non-canonical request through the parser and asserts both
`L7Request.target` (what OPA evaluates) and `raw_header` (what the
upstream receives) are canonical.
- `parse_http_request_canonicalization_preserves_query_string`,
`parse_http_request_leaves_canonical_input_byte_for_byte`,
`parse_http_request_rejects_traversal_above_root`,
`parse_http_request_preserves_http_10_version_on_rewrite`.
- `parse_http_request_accepts_encoded_slash_when_endpoint_opts_in` /
`_rejects_encoded_slash_by_default` verify the
`allow_encoded_slash` gate.
- `test_rewrite_forward_request_uses_canonical_path_on_the_wire`
asserts the fix to the forward-proxy flow.
- `parse_l7_config_allow_encoded_slash_{defaults_false,opt_in}` verify
the YAML/Rego wiring.
* feat(cli): Allow printing only the policy when retrieving sandbox info
* feat(cli): Indicate policy source and revision when retrieving sandboxes
* cargo fmt
The CLI and TUI build an SSH ProxyCommand string by interpolating fields
from CreateSshSessionResponse into a shell command. Three fields -
sandbox_id, token, and the assembled gateway_url (composed from
gateway_scheme, gateway_host, gateway_port, connect_path) - were not
passed through shell_escape, even though exe_command and gateway_name
were. The resulting string is handed to `ssh -o ProxyCommand=...` which
OpenSSH routes through `/bin/sh -c`, so a hostile or compromised gateway
could inject shell metacharacters and execute arbitrary commands under
the developer's workstation account.
- Add build_proxy_command in openshell-core::forward that wraps every
interpolated value in shell_escape. Use it at all four ProxyCommand
sites: openshell-cli/src/ssh.rs (ssh_session_config) and the three
TUI sites in openshell-tui/src/lib.rs (shell connect, sandbox exec,
port-forward reconciliation).
- Add validate_ssh_session_response in openshell-core::forward, called
at each site immediately after response.into_inner(). Enforces a
conservative charset per field (sandbox_id, token, gateway_host,
gateway_scheme, gateway_port, connect_path, host_key_fingerprint) so
malformed responses fail loudly at the gRPC trust boundary before
shell escaping is attempted. Belt-and-suspenders with the escape.
- Harden render_ssh_config so `gateway` and `name` are shell-escaped
even though validate_gateway_name currently gates `gateway`.
- Document the charset contract on CreateSshSessionResponse in
proto/openshell.proto: servers must uphold these rules and clients
reject responses that violate them.
- Add regression unit tests covering adversarial values in every field
and a build_proxy_command test asserting no shell metacharacter
remains outside single-quoted regions.
Tracks OS-100.
* fix(sandbox): strip " (deleted)" suffix from unlinked /proc/<pid>/exe paths
When a running binary is unlinked from its filesystem path — the common
case is a `docker cp` hot-swap of `/opt/openshell/bin/openshell-sandbox`
during the `cluster-deploy-fast` dev upgrade workflow — the Linux kernel
appends the literal string ` (deleted)` to the `/proc/<pid>/exe` readlink
target. The tainted `PathBuf` then flows into `collect_ancestor_binaries`
and on into `BinaryIdentityCache::verify_or_cache`, which tries to
`std::fs::metadata` the path. `stat()` fails with `ENOENT` because the
literal suffix isn't a real filesystem path, and the CONNECT proxy denies
every outbound request with:
ancestor integrity check failed for \
/opt/openshell/bin/openshell-sandbox (deleted): \
Failed to stat ...: No such file or directory (os error 2)
Reproduced in production 2026-04-15: a cluster-deploy-fast-style hot-swap
of the supervisor binary caused every pod whose PID 1 held the now-deleted
inode to deny ALL outbound CONNECTs (slack.com, registry.npmjs.org,
169.254.169.254, etc.), breaking Slack REST delivery, npm installs, and
IMDS probes simultaneously. Existing pre-hot-swap TCP tunnels (e.g. Slack
Socket Mode WSS) kept working because they never re-evaluate the proxy.
Strip the suffix in `binary_path()` so downstream callers see the clean,
grep-friendly path. This aligns the cache key and log messages with the
original on-disk location.
Note: stripping the suffix does NOT by itself make the identity cache
tolerant of a legitimate binary replacement — `verify_or_cache` will now
`stat` and hash whatever currently lives at the stripped path, which is
the NEW binary, and surface a clearer `Binary integrity violation` error.
Fully unblocking the cluster-deploy-fast hot-swap workflow needs a
follow-up that either (a) reads running-binary content from
`/proc/<pid>/exe` directly via `File::open` (procfs resolves this to the
live in-memory executable even when the original inode has been unlinked),
or (b) keys the identity cache by exec dev+inode instead of path. Happy to
send that as a separate PR once the approach is decided — filing this
narrow fix first because it stands on its own: it fixes a concrete
misleading error and unblocks the obvious next step.
Added `binary_path_strips_deleted_suffix` test that copies `/bin/sleep`
to a temp path, spawns a child from it, unlinks the temp binary, verifies
the raw readlink contains the ` (deleted)` suffix, then asserts the public
API returns the stripped path.
Signed-off-by: mjamiv <michael.commack@gmail.com>
* fix(sandbox): narrow the " (deleted)" suffix strip and exercise it end-to-end
Address review feedback from @johntmyers on the initial version of this PR:
procfs::binary_path
- Only strip the kernel's " (deleted)" suffix when stat() on the raw
readlink target reports NotFound. A live executable whose basename
literally ends with " (deleted)" is now returned unchanged instead of
being silently truncated, which matters because identity.rs hashes
whatever this function returns as a trust anchor.
- Operate on raw bytes via OsStrExt, so filenames that are not valid
UTF-8 still get exactly one suffix stripped. The previous
strip_suffix-on-&str path skipped non-UTF-8 entirely and fell through
to returning the tainted path.
- Expand the doc comment to describe both guardrails.
procfs tests
- binary_path_preserves_live_deleted_basename: copy /bin/sleep to a
live file literally named "sleepy (deleted)", spawn it, and assert
that the returned path still ends with " (deleted)".
- binary_path_strips_suffix_for_non_utf8_filename: exec a binary whose
basename contains a 0xFF byte, unlink it, and assert that
binary_path returns the stripped non-UTF-8 path. Writes the bytes
with OpenOptions + sync_all + explicit drop so the write fd is fully
released before exec() to avoid ETXTBSY under concurrent tests.
proxy: extract resolve_process_identity helper
- Pull the peer-resolution + TOFU verify + ancestor walk + cmdline
collection block out of evaluate_opa_tcp into resolve_process_identity.
- Introduce IdentityError which carries the deny reason along with
whatever partial identity data was resolved before the failure so
evaluate_opa_tcp can thread that into ConnectDecision unchanged.
- evaluate_opa_tcp now calls the helper and proceeds straight to the
OPA evaluate step; the surface visible to OPA and OCSF is unchanged.
proxy: end-to-end regression test for the hot-swap contract
- resolve_process_identity_surfaces_binary_integrity_violation_on_hot_swap
stands up a real TcpListener, copies /bin/bash to a temp path,
primes BinaryIdentityCache with that binary, spawns bash with a
/dev/tcp one-liner that opens a real connection to the listener,
and captures the peer's ephemeral port from accept().
- Simulates `docker cp` correctly: unlink the running binary (which
persists via the child's exec mapping) and create a fresh file
with different bytes at the same path. Writing in place is
rejected by the kernel with ETXTBSY, so the old single-inode
approach did not actually model the production failure mode.
- Asserts the error returned by resolve_process_identity contains
"Binary integrity violation" (from BinaryIdentityCache) and does
NOT contain "Failed to stat" or "(deleted)" — the pre-PR-#844
failure mode. The binary field on the error is populated and is
free of the tainted suffix.
- Skips cleanly if /bin/bash is not installed. Child process is
always reaped before the assertion block so a failure does not
leak a sleeping process.
---------
Signed-off-by: mjamiv <michael.commack@gmail.com>
* fix(sandbox): block AF_NETLINK in seccomp unconditionally
Move AF_NETLINK to the unconditional socket-domain block list alongside
AF_PACKET, AF_BLUETOOTH, and AF_VSOCK. Previously it was only blocked in
NetworkMode::Block, leaving it accessible in Proxy mode where network
namespace isolation already scopes netlink to the sandbox's own veth —
making this a defense-in-depth hardening rather than a live exposure.
Closes OS-94
* fix(sandbox): scope inference.local interception to port 443
The pre-OPA interception for inference.local matched on hostname alone,
allowing any port to bypass OPA policy evaluation — including under
deny-all (network_policies: {}). Add a port check so only port 443
takes the interception path; all other ports on inference.local now
fall through to OPA and are subject to normal policy evaluation.
Closes OS-95
* fix(sandbox): enforce RLIMIT_NPROC to prevent fork bomb DoS
Set a hard limit of 512 processes per UID in harden_child_process(),
applied before privilege drop so the sandbox user cannot raise it.
Prevents unrestricted fork() from exhausting the process table — most
relevant for local dev mode where K8s pod cgroup pids.max is absent.
Closes OS-96
* chore(sandbox): fix rustfmt formatting for seccomp blocked_domains
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
The proxy's upstream TLS client only trusted Mozilla root CAs
(webpki-roots), which prevented TLS termination from working with
internal/corporate hosts using private CA certificates.
Load system CA certificates from the container's trust store
(e.g. /etc/ssl/certs/ca-certificates.crt) in addition to
webpki-roots. This allows custom sandbox images to include
corporate CAs via update-ca-certificates.
Signed-off-by: Matthias Osswald <mat.osswald@sap.com>
`git describe --tags --long --match "v*"` matches the `vm-dev` tag
alongside release tags. When `vm-dev` sits on or past the latest
release tag in the commit graph (currently the case on main), git
picks it, and the resulting `vm-dev-N-gSHA` string strips the leading
`v` down to `m-dev-N-gSHA`. With `commits == 0` that's returned
verbatim, producing a binary that reports itself as `m-dev`:
$ openshell --version
openshell m-dev
Confirmed on v0.0.28 (#832) and reproduced on v0.0.29 as well.
Restrict the glob to numeric release tags (`v[0-9]*`). All release
tags follow `v\d+\.\d+\.\d+`, so this loses no valid version — it
only filters out `vm-dev`, `vm-prod`, and any similar non-release
tags that share the `v` prefix.
Verified locally on this repo's current HEAD:
# before
$ git describe --tags --long --match 'v*'
vm-dev-0-g355d845
# after
$ git describe --tags --long --match 'v[0-9]*'
v0.0.29-2-g355d845
Closes#832
Signed-off-by: mjamiv <michael.commack@gmail.com>
format_sse_error only escaped `\` and `"`, leaving two problems:
1. Control characters (`\n`, `\r`, `\t`, and all `\u0000-\u001F`) in
`reason` produce output that fails `serde_json::from_str` — defeating
#834's goal of giving clients a parseable SSE truncation signal.
2. An unescaped `\n\n` inside `reason` splits the single error event
into two SSE frames, letting a misbehaving upstream inject a forged
frame (e.g. a fake tool_calls delta) into the client's stream. Latent
today since all in-tree callers pass static strings, but a footgun
for any future caller passing upstream error text, and the function's
docstring already invites dynamic reasons.
Replace the manual escape with `serde_json::to_writer` (already a
workspace dep of `openshell-sandbox`). Add unit tests for control
character escaping and SSE event-boundary injection.
Closes#840
Signed-off-by: mjamiv <michael.commack@gmail.com>
* feat(policy): add deny rules to network policy schema
Closes#565
Add L7 deny rules that block specific requests even when allowed by
access presets or explicit allow rules. Deny rules mirror the full
capability set of allow rules (method, path, query params, SQL command)
and take precedence -- if a request matches any deny rule, it is blocked
regardless of allow rules.
This enables the "allow everything except these specific operations"
pattern without enumerating every allowed endpoint. For example, granting
read-write access to GitHub while blocking PR approvals, branch
protection changes, and ruleset modifications.
* fix(policy): deny query matching fails closed, mirror allow-side validation
Addresses PR review findings P1 and P3:
P1: Deny-side query matching now uses fail-closed semantics. If ANY
value for a query key matches the deny matcher, the deny fires. The
previous implementation reused allow-side "all values must match"
logic which allowed ?force=true&force=false to bypass a deny on
force=true.
P3: Deny-side query validation now mirrors the full allow-side checks:
empty any lists, non-string matcher values, glob+any mutual exclusion,
glob type checks, and glob syntax warnings are all validated.
Closes#783
Only chown read_write paths that the supervisor creates at startup, and leave pre-existing image paths with their original ownership. Add sandbox tests for creation, symlink rejection, and existing-path ownership preservation, and update architecture docs to match the new behavior.
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* fix(inference): prevent silent truncation of large streaming responses
The L7 inference proxy silently dropped tool_calls from large streaming
responses due to an aggressive 30s per-chunk idle timeout and a reqwest
total-request timeout that capped the entire body stream. Reasoning models
that pause during "thinking" phases triggered these timeouts, producing
valid-looking but truncated HTTP responses with no client-visible error.
- Extract prepare_backend_request() helper and create a streaming variant
that omits the total request timeout; body stream liveness is now
enforced solely by the per-chunk idle timeout
- Add 30s connect_timeout to the reqwest Client builder
- Increase CHUNK_IDLE_TIMEOUT from 30s to 120s for reasoning models
- Inject SSE error events (proxy_stream_error) before the HTTP chunked
terminator on all truncation paths so clients can detect data loss
- Wrap the streaming relay in BufWriter to reduce per-chunk TLS flush
overhead
- Bump OCSF severity for streaming truncation from Low to Medium
Closes#829
* fix(inference): drop BufWriter to preserve SSE incremental delivery
The BufWriter introduced in the previous commit buffered SSE frames until
the 16KB capacity filled or the stream ended, defeating incremental token
delivery and potentially reintroducing client-visible timeouts on healthy
streams. Revert to per-chunk write_all+flush but keep the single
format_chunk() call that coalesces framing into one write.
Also fix the streaming integration test: add a 3s mock delay that exceeds
the 1s route timeout so the test actually validates that the streaming
path omits the total request timeout. Previously the mock responded
immediately, passing regardless of timeout behavior.
* fix(sandbox): harden seccomp denylist, SSRF protection, and inference policy enforcement
- Remove seccomp skip in NetworkMode::Allow so baseline syscall
restrictions apply regardless of network mode
- Block cross-process manipulation syscalls (process_vm_writev,
pidfd_open, pidfd_getfd, pidfd_send_signal) symmetric with existing
ptrace and process_vm_readv blocks
- Block clone/clone3 with CLONE_NEWUSER flag, new mount API syscalls
(fsopen, fsconfig, fsmount, fspick, move_mount, open_tree), and
namespace manipulation (setns, umount2, pivot_root)
- Block userfaultfd and perf_event_open consistent with Docker default
seccomp profile
- Deny and close keep-alive inference connections after a non-inference
request instead of silently continuing the loop
- Add CGNAT (100.64.0.0/10), benchmarking (198.18.0.0/15), and other
special-use IP ranges to SSRF protection in both proxy and
mechanistic mapper
* fix(sandbox): install clone3 seccomp filter before main filter
* test(ocsf): fix shorthand firewall engine expectation
L4 CONNECT and forward-proxy denial responses (403) and upstream
failure responses (502) were returned with empty bodies, making them
indistinguishable from each other when using curl -s. Add a
build_json_error_response() helper that produces consistent JSON
bodies with error and detail fields, matching the format already
used by the L7 deny path.
Also fix Url::to_display_string() in openshell-ocsf to include
non-default ports in the shorthand log output. Requests to e.g.
http://host:9876/path were logged as http://host/path, dropping the
port number.
Closes#807, closes#808
* fix(sandbox): add parent-side Landlock availability probe logging
Landlock status was only logged from inside the pre_exec child process
where the tracing/OCSF pipeline is non-functional after fork. This made
Landlock failures completely invisible in sandbox logs.
Add a probe_availability() function that issues the raw
landlock_create_ruleset syscall to check kernel support, and call it
from the parent process before fork in all three spawn paths (entrypoint,
SSH PTY, SSH pipe). Uses std::sync::Once to emit exactly once per
sandbox lifetime.
WIP - addresses logging gap from #803.
* test(e2e): add Landlock filesystem enforcement tests
Verify Landlock availability logging and enforcement in e2e:
- OCSF probe event appears in sandbox logs
- Read-only paths block writes, allow reads
- Read-write paths allow both
- Paths outside policy are denied entirely
- User-owned paths outside policy are still blocked (proves
Landlock enforces independently of Unix DAC permissions)
Requires Linux host with Landlock support (GitHub Actions runners,
Docker Desktop linuxkit). Related to #803.
* test(e2e): add xfail tests proving #803 privilege ordering bug
Two strict xfail tests that demonstrate the root cause of #803:
- PathFd::new() runs as uid 998 after drop_privileges, so root-only
paths (mode 700) silently fail and Landlock degrades
- When ALL paths fail, best_effort silently drops Landlock entirely
These tests will pass after the two-phase Landlock fix (open PathFds
as root before drop_privileges, restrict_self after).
* fix(test): fix Landlock e2e tests based on test run results
- Remove log-reading tests: the Landlock probe logs to the
supervisor's container stdout, not the in-sandbox file appender
at /var/log/openshell*.log*
- Replace broken xfail test (checked for child-process log messages
that never reach the file appender) with a stat()-based test that
verifies /root is actually in the Landlock allowlist
- Keep enforcement tests (all passing) and the root-only policy
xfail test (correctly proves #803 bug)
* fix(test): remove mixed-policy xfail test
Landlock doesn't restrict stat(), so the test passed unexpectedly.
In the mixed policy case where /root is silently skipped, Landlock
still applies (using other paths) and blocks /root even harder
(not in allowlist = denied). The observable security degradation
only occurs when ALL paths fail, which the existing xfail test
already covers.
* fix(sandbox): two-phase Landlock to fix privilege ordering (#803)
Split Landlock apply into prepare() and enforce():
- prepare() runs as root before drop_privileges: opens PathFds,
creates ruleset, adds rules. Root-only paths (mode 700) now
succeed instead of silently failing as uid 998.
- enforce() runs after drop_privileges: calls restrict_self()
which does not require root.
This fixes the root cause of #803 where drop_privileges() ran
before sandbox::apply(), causing PathFd::new() to fail on
root-only paths. In best_effort mode this silently dropped all
Landlock restrictions.
The fix applies to all three spawn paths: entrypoint (process.rs),
SSH PTY (ssh.rs), and SSH pipe exec (ssh.rs).
Removes xfail marker from e2e test that now passes.
* fix(sandbox): address PR review feedback
- enforce() now respects best_effort: if restrict_self() fails and
policy is best_effort, log and degrade instead of aborting startup
- log_sandbox_readiness distinguishes best_effort (degraded) from
hard_requirement (will fail) in OCSF messages
* fix(sandbox): resolve symlinked binary paths in network policy matching
Policy binary paths specified as symlinks (e.g., /usr/bin/python3) were
silently denied because the kernel reports the canonical path via
/proc/<pid>/exe (e.g., /usr/bin/python3.11). The strict string equality
in Rego never matched.
Expand policy binary paths by resolving symlinks through the container
filesystem (/proc/<pid>/root/) after the entrypoint starts. The OPA data
now contains both the original and resolved paths, so Rego's existing
strict equality check naturally matches either.
- Add resolve_binary_in_container() helper for Linux symlink resolution
- Add from_proto_with_pid() and reload_from_proto_with_pid() to OpaEngine
- Trigger one-shot OPA rebuild after entrypoint_pid is stored
- Thread entrypoint_pid through run_policy_poll_loop for hot-reloads
- Improve deny reason with symlink debugging hint
- Add 18 new tests including hot-reload and Linux symlink e2e tests
Closes#770
* fix(sandbox): skip procfs-dependent tests when /proc/<pid>/root/ is inaccessible
The Linux-specific symlink resolution tests depend on /proc/<pid>/root/
being readable, which requires CAP_SYS_PTRACE or permissive ptrace
scope. This is unavailable in CI containers, rootless containers, and
hardened hosts. Add a procfs_root_accessible() guard that skips these
tests gracefully instead of failing.
* fix(sandbox): add explicit logging when symlink resolution fails and improve deny messages
When /proc/<pid>/root/ is inaccessible (restricted ptrace, rootless
containers, hardened hosts), resolve_binary_in_container now logs a
per-binary warning with the specific error, the path it tried, and
actionable guidance (use canonical path or grant CAP_SYS_PTRACE).
Previously this was completely silent.
The Rego deny reason for binary mismatches now leads with 'SYMLINK HINT'
and includes a concrete fix command ('readlink -f' inside the sandbox)
plus what to look for in logs if automatic resolution isn't working.
* fix(sandbox): use read_link instead of canonicalize for symlink resolution
std::fs::canonicalize resolves /proc/<pid>/root itself (a kernel
pseudo-symlink to /) which strips the prefix needed for path extraction.
This caused resolution to silently fail in all environments, not just CI.
Replace with an iterative read_link loop that walks the symlink chain
within the container namespace without resolving the /proc mount point.
Add normalize_path helper for relative symlink targets containing ..
components. Update procfs_root_accessible test guard to actually probe
the full resolution path instead of just checking path existence.
* fix(sandbox): defer symlink resolution until container filesystem is ready
The one-shot resolve ran immediately after ProcessHandle::spawn, before
the child's mount namespace and /proc/<pid>/root/ were populated. This
caused symlink_metadata to fail with ENOENT on every binary, and the
poll loop never retried because it only reloads when the policy hash
changes on the server.
Replace the synchronous resolve with an async task that probes
/proc/<pid>/root/ with retries (10 attempts, 500ms apart, 5s total).
The child's mount namespace is typically ready within a few hundred ms.
Also inline error values into warning message strings so they appear in
default log output (not just as structured tracing fields that may be
elided), and add debug-level logs before each symlink_metadata call to
aid diagnosis.
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
Policies with allowed_ips entries targeting loopback, link-local, or
unspecified ranges now fail at connection time instead of being silently
blocked at runtime. The shorthand log format for DENIED events includes
a [reason:...] suffix so operators can distinguish 'allowlist miss' from
'structurally un-allowable'. The mechanistic mapper skips proposals for
always-blocked destinations, preventing the infinite TUI notification
loop. The gateway validates proposed rules on approval as defense-in-depth.
- Extract shared IP helpers (is_always_blocked_ip, is_always_blocked_net,
is_internal_ip) to openshell_core::net
- Reject always-blocked entries in parse_allowed_ips with hard error
- Skip implicit allowed_ips synthesis for always-blocked literal IP hosts
- Add status_detail to HttpActivityBuilder for denial reason propagation
- Enrich NET and HTTP shorthand with [reason:...] for DENIED events
- Add engine: tag to HTTP shorthand (consistency with NET shorthand)
- Filter always-blocked proposals in mechanistic mapper generate_proposals
- Add validate_rule_not_always_blocked server-side defense-in-depth
- Update architecture docs, published docs, and E2E test assertions
The drop_privileges_succeeds_for_current_user test failed on non-root
systems because initgroups(3) requires CAP_SETGID even when switching
to the current user. Add a new drop_privileges_succeeds_for_current_group
test that omits run_as_user, bypassing initgroups while still exercising
setgid() and the GID post-condition verification. Mark the original
full-path test with #[ignore] so it can be run explicitly as root.
Fixes#622
Drew Newberry