mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-31 07:44:57 +08:00
Roland Huss
fdce6ef260
* feat(supervisor): relay agent OTLP traces to the collector Serve the reserved endpoint 192.0.0.8:4318 inside the supervisor, attribute each batch with openshell.sandbox.id and openshell.telemetry.source=agent at the protobuf wire level, buffer it with fixed bounds, and forward it over OTLP/gRPC to the collector the supervisor already uses, flushing before the exit report and keeping the relay up for exec sessions. Signed-off-by: Roland Huß <rhuss@redhat.com> * refactor(supervisor): let the OTLP relay exporter take a prepared channel Expose Exporter::with_channel as the seam for a collector transport built once in openshell-otel and shared by the gateway, the supervisor's own spans, and the relay. Exporter::new keeps building the plaintext channel for now. Signed-off-by: Roland Huß <rhuss@redhat.com> * feat(sandbox): leave caller-set OTLP endpoints alone Inject the relay variables only when the sandbox creation request names no OTLP endpoint, generic or traces-specific. A caller who names one has chosen where the agent exports; overwriting it would silently redirect an agent that deliberately exports elsewhere. A protocol or compression setting alone does not suppress injection. Applies to both spawn paths and to exec children, with the docs updated to match. Signed-off-by: Roland Huß <rhuss@redhat.com> * fix(supervisor): address OTLP relay review feedback Inject only the generic OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL as lowest-precedence defaults, and only while the relay is active. The supervisor now starts the relay before attach and signals it to the sandbox runtime through AgentSpec::otlp_relay_active. An endpoint in the creation request, the image ENV, or an exec environment suppresses the defaults, and a provider-supplied traces endpoint is removed. Harden the relay itself: - skip the exit flush and shutdown wait when the last export failed - count batches still pending on Drop as discarded and log them - validate the collector endpoint with the shared openshell-otel helper - accept Content-Encoding: identity - fix the 413 and shutdown 503 messages and the memory bound docs - narrow the attribution claim to resource attributes Document best-effort delivery and the 404 for metrics and logs. Signed-off-by: Roland Huß <rhuss@redhat.com> * feat(supervisor): gate the agent trace relay behind [otlp.agent] The relay started whenever a sandbox supervisor had a trace endpoint, so enabling gateway trace export also changed every workload's environment. Make it opt-in: `[openshell.gateway.otlp.agent]` with `enabled = true` turns it on, and its optional `endpoint` falls back to the shared `[openshell.gateway.otlp]` endpoint, like the OCSF table. The gateway validates the endpoint and drivers pass it to supervisors as OPENSHELL_OTLP_AGENT_ENDPOINT, separate from OPENSHELL_OTLP_ENDPOINT. Docker, Podman (including the Podman Machine host alias), Kubernetes and VM drivers carry it, in-process and as standalone binaries. The shared Kubernetes supervisor still does not relay. The relay now acknowledges OTLP metrics and logs exports with an empty success and discards them, so agent SDKs that export every signal to the generic endpoint stop retrying. Discards are counted and logged once. Local dev gateway tasks, the Skaffold values and the e2e harness enable the table; OPENSHELL_E2E_OTLP_AGENT=0 runs the inert case with trace export still on. Signed-off-by: Roland Huß <rhuss@redhat.com> * docs(supervisor): describe the agent trace relay architecture Add an "Agent traces" section to the architecture page and document the relay in the crate READMEs: the reserved-destination hook in the network supervisor, the OTEL defaults the sandbox runtime injects, and how the Docker, Podman, Kubernetes, and VM drivers pass the agent endpoint. Signed-off-by: Roland Huß <rhuss@redhat.com> * fix(sandbox): disable OTLP compression in the relay env and cover exec children The relay refuses compressed bodies, so an image or creation request that sets gzip compression without naming an endpoint received the relay default and lost every trace export to a 415. The relay default now also sets OTEL_EXPORTER_OTLP_COMPRESSION and OTEL_EXPORTER_OTLP_TRACES_COMPRESSION to none, as #4275 asks. Boundary exec children start from a cleared environment and never see the image ENV, yet the relay default was suppressed when the sandbox's own environment named an endpoint, which left exec children with no endpoint at all. apply_otlp_relay_env now takes a ChildEnvironment so only children that inherit the sandbox environment consult it. Add a probe test for both TLS environment modes together with the relay default, now that the mode-aware TLS layering lives in compose_canonical_process_environment, and describe the whole composition at its call sites. Signed-off-by: Roland Huß <rhuss@redhat.com> * fix(supervisor): flush the agent trace relay before suspending The suspend lifecycle from #4765 returned without shutting the relay down, so Drop discarded batches the relay had already acknowledged to the agent, which does not resend them after resume. Shut the relay down, bounded by FLUSH_TIMEOUT, once the workload is frozen and before networking is dropped. Also tidy what the rebases left behind: drop a stray blank line inside the upstream startup block, rename SandboxInstanceConfig::agent_otlp_endpoint to otlp_agent_endpoint to match every other identifier, serialize the OCSF table through the generic serialize_enabled, correct the UnsupportedScheme doc comment (the supervisor's own exporter does speak TLS), and build the reserved-destination proxy test identities with host-native paths. Signed-off-by: Roland Huß <rhuss@redhat.com> * test(e2e): scope the Podman OTLP default and restore upstream lockfiles The Podman wrapper enabled OTLP export and the agent relay for every e2e run, including single-target CI jobs such as additional_ca. Enable it only for the full suite or a selection that includes otlp_relay, following the SPIFFE fixture pattern; the Docker wrapper stays opt-in. The e2e and features lockfiles had downgraded libsqlite3-sys, crossbeam-queue and hybrid-array against upstream with no manifest change requiring it. Restore upstream's lockfiles and add only the packages the new e2e dependencies pull in. Signed-off-by: Roland Huß <rhuss@redhat.com> --------- Signed-off-by: Roland Huß <rhuss@redhat.com>