mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-24 06:34:33 +08:00
krishicks
8091f66877
Since #2726 the canonical main process's stdout and stderr are captured in pipes that feed only the in-memory replay buffer used by sandbox connect. Agent output therefore never reaches the container's own stdout and stderr, so it is missing from kubectl logs, docker logs, and podman logs and from anything that collects container logs. Before #2726 the entrypoint inherited the container's descriptors and its output appeared there. Copy the main process's output to the launcher's stdout and stderr in addition to the replay buffer, restoring the earlier behavior: - Output is copied byte for byte to the matching stream from a forwarder thread per stream, after it is published to the replay buffer. When the container runtime falls behind on a stream, that stream's reader waits instead of dropping output, so backpressure reaches the agent as it did with inherited descriptors, while the other stream and attachments keep receiving output. - Before the main process's exit is published, the output readers finish and queued output is drained to the container log, so an agent's final lines are not lost at shutdown. A 30 second deadline covers both; when it expires, readers waiting on the container log are released and drain the pipes into the replay buffer only, so a stalled container log cannot block exit reporting. - PTY-mode processes are not copied. The terminal stream carries escape sequences and echoed input, and terminal commands never reached the container log before #2726. - Exec, SSH, and SFTP sessions are not copied. Launcher log lines keep their existing format and remain in the container's stderr. They are written as whole lines, and a newline is inserted first when the agent left stderr mid-line, so launcher and agent lines do not merge. The Docker and VM drivers appended the tail of the workload's output to failure messages: Docker the workload container's log, and the VM driver the guest console, which carries the launcher's stdout and stderr. Those messages land in the sandbox's Ready condition and in platform events that the gateway republishes to the sandbox event stream. With agent output in that log, those messages would carry arbitrary agent output, including anything sensitive the agent prints, into gateway status and events. The supervisor starts its health endpoint only after the agent starts, so every Docker failure path could include agent output, and the VM driver reports one whenever the VM or host supervisor exits. Forward only the supervisor's log tail, matching the Podman driver, which reads the workload log solely to match fixed launcher markers and never forwards raw workload output. The workload's output remains available through docker logs and the VM's rootfs-console.log. Document where main process output appears in the logging docs and the cluster debugging skill. Closes #3928 Signed-off-by: Kris Hicks <khicks@nvidia.com>
126 lines
6.8 KiB
Plaintext
126 lines
6.8 KiB
Plaintext
---
|
|
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
title: "Accessing Logs"
|
|
description: "How to view sandbox logs through the CLI, TUI, and directly on the sandbox filesystem."
|
|
keywords: "Generative AI, Cybersecurity, Logging, CLI, TUI, Observability"
|
|
---
|
|
|
|
OpenShell provides three ways to access sandbox logs: the CLI, the TUI, and direct filesystem access inside the sandbox.
|
|
|
|
## CLI
|
|
|
|
Use `openshell logs` to stream logs from a running sandbox:
|
|
|
|
```shell
|
|
openshell logs smoke-l4 --source sandbox
|
|
```
|
|
|
|
The CLI receives logs from the gateway over gRPC. Each line includes a timestamp, source, level, and message:
|
|
|
|
```text
|
|
[1775014132.118] [sandbox] [OCSF ] [ocsf] NET:OPEN [INFO] ALLOWED /usr/bin/curl(58) -> api.github.com:443 [policy:github_api engine:opa]
|
|
[1775014132.190] [sandbox] [OCSF ] [ocsf] HTTP:GET [INFO] ALLOWED GET http://api.github.com/zen [policy:github_api]
|
|
[1775014132.690] [sandbox] [OCSF ] [ocsf] NET:OPEN [MED] DENIED /usr/bin/curl(64) -> httpbin.org:443 [policy:- engine:opa]
|
|
[1775014113.058] [sandbox] [INFO ] [openshell_sandbox] Starting sandbox
|
|
```
|
|
|
|
OCSF structured events show `OCSF` as the level. Standard tracing events show `INFO`, `WARN`, or `ERROR`.
|
|
|
|
Gateway-originated policy mutations also appear in this stream. When the gateway merges `openshell policy update` operations or approves or removes draft policy chunks, it emits `gateway` `OCSF` `CONFIG:*` lines for the affected sandbox so you can see the exact logical change that produced a new policy revision.
|
|
|
|
## TUI
|
|
|
|
The TUI dashboard displays sandbox logs in real time. Logs appear in the log panel with the same format as the CLI.
|
|
|
|
## Main Process Output
|
|
|
|
The sandbox's main process writes its stdout and stderr to the sandbox container's stdout and stderr, alongside warnings from the sandbox runtime. Read it with the container tooling for the compute driver:
|
|
|
|
```shell
|
|
kubectl -n <namespace> logs <sandbox-pod> -c agent
|
|
docker logs <sandbox-container>
|
|
podman logs <sandbox-container>
|
|
```
|
|
|
|
A main process started with a TTY writes only to its attachment; use `openshell sandbox connect` to view that output. Output from `sandbox exec` and SSH sessions does not reach the container log, and `openshell logs` does not include main process output.
|
|
|
|
When the container runtime falls behind reading the log, the main process blocks on writes, as with any container.
|
|
|
|
## Gateway Log Storage
|
|
|
|
The sandbox pushes logs to the gateway over gRPC in real time. The gateway stores a bounded buffer of recent log lines per sandbox. This buffer is not persisted to disk and is lost when the gateway restarts.
|
|
|
|
For durable log storage, use the log files inside the sandbox or enable [OCSF JSON export](/observability/ocsf-json-export) and ship the JSONL files to an external log aggregator.
|
|
|
|
## Loss Awareness and Resume
|
|
|
|
The watch stream behind `openshell logs` is loss-aware. Each resumable event (log line or platform event) carries an opaque `cursor` token. Status snapshots and warnings carry an empty cursor.
|
|
|
|
Do not parse a cursor. The only supported operation is comparing two cursors observed on the same stream and keeping the greater one as the resume point. That comparison is a plain string comparison in any language.
|
|
|
|
The gateway distinguishes recoverable from unrecoverable loss:
|
|
|
|
- **Recoverable lag.** When a consumer falls behind and the gateway skips ahead in its buffer, the stream emits a warning event and keeps running. The warning is the signal that events were skipped.
|
|
- **Unrecoverable gap.** When a client reconnects and asks to resume after a cursor the gateway has already trimmed from its buffer, the stream ends with an `OUT_OF_RANGE` status. The client should restart observation and, if it needs the missing lines, read them from the log files inside the sandbox.
|
|
|
|
A cursor is bound to the cursor space that issued it. A gateway restart, teardown of the sandbox's buffers, or a reconnect that lands on a different gateway replica starts a new space, and cursors from the previous one no longer refer to anything. The gateway rejects them with `OUT_OF_RANGE` instead of treating them as caught up — which would silently suppress the new space's events. A malformed cursor is rejected with `INVALID_ARGUMENT`.
|
|
|
|
`OUT_OF_RANGE` is terminal for that cursor. Restart the watch with an empty resume cursor; retrying the same one fails identically.
|
|
|
|
On reconnect, a client passes the highest cursor it processed as the resume point. The gateway replays only events after that cursor — logs and platform events merged in order — then resumes live delivery. The handoff from replay to live delivery is exact: an event buffered while the stream was reopening is delivered once, never twice. It is not a guarantee that nothing was lost — a warning event or an `OUT_OF_RANGE` status still reports loss, both before and after a reconnect.
|
|
|
|
The gateway merges the log and platform event sources before emitting, so events normally arrive in ascending cursor order. That applies to the buffered tail you receive when the stream opens, to a replay after a resume, and to live delivery. The gateway does not delay an event to wait for a lower cursor that has not been published yet, so a cursor can still arrive late under concurrent publication. Track the highest cursor seen as the resume point rather than the last one received.
|
|
|
|
## Direct Filesystem Access
|
|
|
|
Start an independent shell with `sandbox exec` to read log files directly:
|
|
|
|
```text
|
|
openshell sandbox exec --name my-sandbox --tty -- /bin/bash -l
|
|
sandbox@my-sandbox:~$ cat /var/log/openshell.2026-04-01.log
|
|
```
|
|
|
|
Or run a one-off command without an interactive shell:
|
|
|
|
```shell
|
|
openshell sandbox exec --name my-sandbox -- cat /var/log/openshell.2026-04-01.log
|
|
```
|
|
|
|
`sandbox connect` attaches to the sandbox's existing canonical main process; it
|
|
does not start a new shell.
|
|
|
|
The log files inside the sandbox contain the complete record, including events that the gRPC push channel can drop under load. The push channel is bounded and drops events rather than blocking.
|
|
|
|
## Filtering by Event Type
|
|
|
|
The shorthand format is designed for `grep`. Some useful patterns:
|
|
|
|
```shell
|
|
# All denied connections
|
|
grep "DENIED\|BLOCKED" /var/log/openshell.*.log
|
|
|
|
# All network events
|
|
grep "OCSF NET:" /var/log/openshell.*.log
|
|
|
|
# All L7 enforcement decisions
|
|
grep "OCSF HTTP:" /var/log/openshell.*.log
|
|
|
|
# Security findings only
|
|
grep "OCSF FINDING:" /var/log/openshell.*.log
|
|
|
|
# Policy changes
|
|
grep "OCSF CONFIG:" /var/log/openshell.*.log
|
|
|
|
# All OCSF events, excluding standard tracing
|
|
grep "^.* OCSF " /var/log/openshell.*.log
|
|
|
|
# Events at medium severity or above
|
|
grep "\[MED\]\|\[HIGH\]\|\[CRIT\]\|\[FATAL\]" /var/log/openshell.*.log
|
|
```
|
|
|
|
## Next Steps
|
|
|
|
- Learn how the [log formats](/observability/logging) work and how to read the shorthand.
|
|
- [Enable OCSF JSON export](/observability/ocsf-json-export) for machine-readable structured output.
|