Files
OpenShell/proto
Roland Huss fdce6ef260 feat(supervisor): relay agent OTLP traces to the collector (#4276)
* feat(supervisor): relay agent OTLP traces to the collector

Serve the reserved endpoint 192.0.0.8:4318 inside the supervisor, attribute each batch with openshell.sandbox.id and openshell.telemetry.source=agent at the protobuf wire level, buffer it with fixed bounds, and forward it over OTLP/gRPC to the collector the supervisor already uses, flushing before the exit report and keeping the relay up for exec sessions.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* refactor(supervisor): let the OTLP relay exporter take a prepared channel

Expose Exporter::with_channel as the seam for a collector transport built once in openshell-otel and shared by the gateway, the supervisor's own spans, and the relay. Exporter::new keeps building the plaintext channel for now.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* feat(sandbox): leave caller-set OTLP endpoints alone

Inject the relay variables only when the sandbox creation request names no OTLP endpoint, generic or traces-specific. A caller who names one has chosen where the agent exports; overwriting it would silently redirect an agent that deliberately exports elsewhere. A protocol or compression setting alone does not suppress injection. Applies to both spawn paths and to exec children, with the docs updated to match.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* fix(supervisor): address OTLP relay review feedback

Inject only the generic OTEL_EXPORTER_OTLP_ENDPOINT and
OTEL_EXPORTER_OTLP_PROTOCOL as lowest-precedence defaults, and only
while the relay is active. The supervisor now starts the relay before
attach and signals it to the sandbox runtime through
AgentSpec::otlp_relay_active. An endpoint in the creation request, the
image ENV, or an exec environment suppresses the defaults, and a
provider-supplied traces endpoint is removed.

Harden the relay itself:
- skip the exit flush and shutdown wait when the last export failed
- count batches still pending on Drop as discarded and log them
- validate the collector endpoint with the shared openshell-otel helper
- accept Content-Encoding: identity
- fix the 413 and shutdown 503 messages and the memory bound docs
- narrow the attribution claim to resource attributes

Document best-effort delivery and the 404 for metrics and logs.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* feat(supervisor): gate the agent trace relay behind [otlp.agent]

The relay started whenever a sandbox supervisor had a trace endpoint, so
enabling gateway trace export also changed every workload's environment.
Make it opt-in: `[openshell.gateway.otlp.agent]` with `enabled = true`
turns it on, and its optional `endpoint` falls back to the shared
`[openshell.gateway.otlp]` endpoint, like the OCSF table.

The gateway validates the endpoint and drivers pass it to supervisors as
OPENSHELL_OTLP_AGENT_ENDPOINT, separate from OPENSHELL_OTLP_ENDPOINT.
Docker, Podman (including the Podman Machine host alias), Kubernetes and
VM drivers carry it, in-process and as standalone binaries. The shared
Kubernetes supervisor still does not relay.

The relay now acknowledges OTLP metrics and logs exports with an empty
success and discards them, so agent SDKs that export every signal to the
generic endpoint stop retrying. Discards are counted and logged once.

Local dev gateway tasks, the Skaffold values and the e2e harness enable
the table; OPENSHELL_E2E_OTLP_AGENT=0 runs the inert case with trace
export still on.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* docs(supervisor): describe the agent trace relay architecture

Add an "Agent traces" section to the architecture page and document the
relay in the crate READMEs: the reserved-destination hook in the network
supervisor, the OTEL defaults the sandbox runtime injects, and how the
Docker, Podman, Kubernetes, and VM drivers pass the agent endpoint.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* fix(sandbox): disable OTLP compression in the relay env and cover exec children

The relay refuses compressed bodies, so an image or creation request that
sets gzip compression without naming an endpoint received the relay default
and lost every trace export to a 415. The relay default now also sets
OTEL_EXPORTER_OTLP_COMPRESSION and OTEL_EXPORTER_OTLP_TRACES_COMPRESSION to
none, as #4275 asks.

Boundary exec children start from a cleared environment and never see the
image ENV, yet the relay default was suppressed when the sandbox's own
environment named an endpoint, which left exec children with no endpoint at
all. apply_otlp_relay_env now takes a ChildEnvironment so only children that
inherit the sandbox environment consult it.

Add a probe test for both TLS environment modes together with the relay
default, now that the mode-aware TLS layering lives in
compose_canonical_process_environment, and describe the whole composition
at its call sites.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* fix(supervisor): flush the agent trace relay before suspending

The suspend lifecycle from #4765 returned without shutting the relay down,
so Drop discarded batches the relay had already acknowledged to the agent,
which does not resend them after resume. Shut the relay down, bounded by
FLUSH_TIMEOUT, once the workload is frozen and before networking is
dropped.

Also tidy what the rebases left behind: drop a stray blank line inside the
upstream startup block, rename SandboxInstanceConfig::agent_otlp_endpoint
to otlp_agent_endpoint to match every other identifier, serialize the OCSF
table through the generic serialize_enabled, correct the UnsupportedScheme
doc comment (the supervisor's own exporter does speak TLS), and build the
reserved-destination proxy test identities with host-native paths.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* test(e2e): scope the Podman OTLP default and restore upstream lockfiles

The Podman wrapper enabled OTLP export and the agent relay for every e2e
run, including single-target CI jobs such as additional_ca. Enable it only
for the full suite or a selection that includes otlp_relay, following the
SPIFFE fixture pattern; the Docker wrapper stays opt-in.

The e2e and features lockfiles had downgraded libsqlite3-sys,
crossbeam-queue and hybrid-array against upstream with no manifest change
requiring it. Restore upstream's lockfiles and add only the packages the
new e2e dependencies pull in.

Signed-off-by: Roland Huß <rhuss@redhat.com>

---------

Signed-off-by: Roland Huß <rhuss@redhat.com>
2026-10-28 15:26:52 +00:00
..

Protobuf API conventions

This directory defines OpenShell's gRPC contracts. These conventions are the source of truth for public protobuf API design. Apply them to new APIs and when changing existing APIs; generated SDK naming follows from these definitions.

Entity references

  • Use name for the primary resource targeted by an RPC.
  • Use the resource's role for a related resource reference, such as sandbox, provider, service, or rule.
  • Do not append _name to a canonical entity reference. Its value is already the canonical name. Descriptive values, local map keys, implementation labels, and configured registration names that are not entity references may retain the suffix. Examples include display_name, file_name, driver_name, runtime_class_name, rule_name, and middleware_name.
  • Public callers reference entities by canonical name. Keep immutable IDs at authentication, persistence, compute-driver, and other internal boundaries.

For example:

message GetSandboxRequest {
  openshell.datamodel.v1.WorkspaceSelector workspace_scope = 2;
  string name = 1;
}

message ExposeServiceRequest {
  openshell.datamodel.v1.WorkspaceSelector workspace_scope = 5;
  string sandbox = 1;
  string name = 2;
}

GetSandboxRequest.name identifies the RPC's primary resource. ExposeServiceRequest.sandbox identifies a related sandbox while name identifies the service being exposed.

Workspace scope

  • Public workspace-scoped requests declare workspace_scope before every other field. Existing wire field numbers do not need to follow declaration order.
  • Type workspace_scope as openshell.datamodel.v1.WorkspaceSelector.
  • Requests operating in one workspace require a non-empty canonical WorkspaceSelector.workspace. The default workspace is an explicit name, not an omitted-value fallback.
  • Accept WorkspaceSelector.all_workspaces only on collection-list RPCs that explicitly document and authorize cross-workspace access. The supported public collections are sandboxes, sandbox templates, providers, and services.
  • Requests whose primary resource is a workspace use name, not workspace_scope.
  • Document every request that permits an omitted selector. Current exceptions are platform provider-profile scope and authenticated sandbox bootstrap.

Field and message design

  • Prefer a dedicated request and response message for each RPC, including requests that are currently empty.
  • Use google.protobuf.Timestamp for absolute time and google.protobuf.Duration for elapsed time.
  • Use optional presence when omitted and explicitly empty values have different meanings. Do not infer presence from a protobuf scalar's default value.
  • Keep public request fields in semantic reading order. Field numbers preserve wire identity and may therefore differ from declaration order.
  • Document authorization-sensitive selector variants and omission semantics on the containing request.

Schema evolution

  • Treat field numbers and fully qualified message names as durable wire identities.
  • When removing or renaming a field, reserve its old number and source name. Do not reuse either for a different meaning.
  • Review changes against both the public descriptor closure and durable stored protobuf closure. The public_and_durable_schema_inventories_are_complete test in openshell-server owns both inventories.
  • Regenerate Rust, Python, Go, and TypeScript bindings after contract changes. Run mise run pre-commit, the affected SDK checks, and relevant server tests before submitting the change.

Supervisor host protocol

supervisor_host.proto is the private stream between a gateway and the supervisor process it shares among its sandboxes: the supervisor attaches, the gateway sends complete assignment snapshots, and the supervisor reports what it hosts. It is a version-locked deployment protocol served on the gateway listener but authorized only for the gateway-minted supervisor token (auth_mode: "supervisor"). It is not imported by openshell.proto, so SDK generation and gRPC reflection exclude it; the server's schema inventory test classifies it explicitly. SandboxInstanceAssignment.launch_material carries the existing JSON sandbox instance registration of openshell-supervisor-control and is marked secret.

Isolation backend protocol

isolation_backend.proto defines backend discovery, boundary binding, and optional suspend/resume RPCs. sandbox_protocol.proto defines typed workload control, process I/O, and network mediation over the bound boundary. Both are supervisor-to-backend contracts, not gateway RPCs. Backend authors can generate bindings independently from these definitions. The negotiated Sandbox Protocol version covers the gRPC services, typed messages, streaming, and lifecycle semantics.