mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-15 05:09:44 +08:00
krishicks
fa24299092
Add an opt-in OTLP/gRPC trace exporter to the gateway. Export is enabled
by the presence of an `[openshell.gateway.otlp]` table with an endpoint;
there is no separate toggle.
Instrumented:
- Inbound request server spans, named for the RPC (`$service/$method`) or
`{method} {path}` for plain HTTP. They continue valid W3C `traceparent`
context when present and start a new trace otherwise. gRPC spans also
carry `rpc.system`, `rpc.service`, `rpc.method`, and trailer-derived
`rpc.grpc.status_code`.
- Compute driver calls (create, delete, list, get, validate, watch) as
client spans anchored on the `ComputeDriver` contract.
- Store reads and writes as children of the current request or loop span.
- Work with no inbound request: compute driver initialization, the sandbox
reconcile sweep, provider credential refresh tick, and driver watch events.
Each roots one operation trace so its child work does not arrive as anonymous
single-span traces.
This is deliberately not exhaustive. Auth, policy evaluation, and
middleware remain uninstrumented, as do store lifecycle calls (`ping`,
`close`) that a readiness poll would turn into a span per tick. The aim is
a useful trace tree at a reviewable size; coverage can grow against real
traces.
Design notes:
- The TOML table owns whether and where to export. The SDK `OTEL_*`
variables own how; sampling, batching, and limits are not mirrored into
gateway config.
- The OpenTelemetry layer exports spans only. Existing `tracing` events
remain on the stdout and sandbox-log paths and are not copied into trace
payloads.
- Telemetry never blocks the gateway. A malformed endpoint logs an error
and disables export rather than failing startup, and buffered spans are
drained during graceful shutdown.
- Failed spans carry error status without a separate `error.type` attribute.
Request spans use HTTP status and gRPC response trailers; driver spans use
the returned gRPC status; autonomous loop spans record failed results
explicitly. Store spans exempt `UniqueViolation` and `Conflict`, because
those errors report expected contention such as a held lease or an
optimistic-concurrency retry.
- The compute driver is reachable only through `TracedDriver::call`, so a
call cannot skip its span. This is the client half of a client/server pair
and the single place to inject context if drivers move out of process.
- Tests share one process-wide subscriber and in-memory exporter because
`tracing` caches callsite interest globally.
Inbound W3C trace context is propagated into gateway request spans. Context
is not yet injected into outbound driver calls, so a future out-of-process
driver would still need propagation at the `TracedDriver` seam.
The Helm chart is intentionally unchanged, so OTLP export cannot yet be
enabled on a chart-deployed gateway.
Refs #2507
Signed-off-by: Kris Hicks <khicks@nvidia.com>
48 lines
1.5 KiB
Rust
48 lines
1.5 KiB
Rust
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
// SPDX-License-Identifier: Apache-2.0
|
|
|
|
//! Process-wide tracing subscriber setup for the gateway.
|
|
//!
|
|
//! This module routes gateway logs and spans to configured diagnostic outputs.
|
|
//! `OpenShell` product telemetry collected for maintainers is handled by
|
|
//! [`crate::telemetry`].
|
|
|
|
use opentelemetry_sdk::trace::SdkTracerProvider;
|
|
use tracing_subscriber::EnvFilter;
|
|
use tracing_subscriber::prelude::*;
|
|
|
|
use crate::config_file::OtlpConfig;
|
|
use crate::otel_tracing::SetupError;
|
|
use crate::tracing_bus::TracingLogBus;
|
|
|
|
pub struct TracingHandle {
|
|
tracer_provider: Option<SdkTracerProvider>,
|
|
}
|
|
|
|
impl TracingHandle {
|
|
pub fn shutdown(&self) {
|
|
if let Some(provider) = &self.tracer_provider
|
|
&& let Err(err) = provider.shutdown()
|
|
{
|
|
tracing::warn!(error = %err, "OTLP tracer provider shutdown failed");
|
|
}
|
|
}
|
|
}
|
|
|
|
pub fn install(
|
|
env_filter: EnvFilter,
|
|
tracing_log_bus: &TracingLogBus,
|
|
otlp_config: Option<&OtlpConfig>,
|
|
) -> (TracingHandle, Option<SetupError>) {
|
|
let (tracer_provider, setup_error) = crate::otel_tracing::provider_for(otlp_config);
|
|
|
|
tracing_subscriber::registry()
|
|
.with(env_filter)
|
|
.with(tracing_subscriber::fmt::layer())
|
|
.with(tracing_log_bus.layer())
|
|
.with(tracer_provider.as_ref().map(crate::otel_tracing::layer))
|
|
.init();
|
|
|
|
(TracingHandle { tracer_provider }, setup_error)
|
|
}
|