trending/OpenShell
1
0
Fork 0
mirror of https://github.com/NVIDIA/OpenShell.git synced 2026-10-24 06:34:33 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(auth): classify rejected gateway credentials in authentication events (#4655)

Browse Source 
Authenticators now return the mechanism and a reason category with the
status they reject a request with, and the gateway router records both
in the Authentication event. Callers still receive the original status.

Refs #2912

Signed-off-by: Kris Hicks <khicks@nvidia.com>
This commit is contained in:
krishicks
2026-10-16 20:14:45 +00:00
committed by GitHub
parent 77929d9c14
commit 35a3390dbc
11 changed files with 1212 additions and 298 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/openshell-server/src/auth/audit.rs
+41 -37
View File
@@ -7,47 +7,51 @@
//! credentials or raw verification errors.
use openshell_ocsf::{
ActionId, AuthenticationBuilder, DispositionId, HttpMethod, HttpRequest, Service, SeverityId,
StatusId, Url, User, ocsf_emit,
ActionId, AuthProtocolId, AuthenticationBuilder, DispositionId, HttpMethod, HttpRequest,
Service, SeverityId, StatusId, Url, User, ocsf_emit,
};
/// Why an authentication attempt failed.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AuthFailureReason {
/// No credential was presented.
MissingCredential,
/// Credentials for more than one mechanism were presented together.
ConflictingCredentials,
}
use super::authenticator::{AuthFailureReason, AuthMechanism};
impl AuthFailureReason {
#[must_use]
pub fn as_str(self) -> &'static str {
match self {
Self::MissingCredential => "missing_credential",
Self::ConflictingCredentials => "conflicting_credentials",
}
}
fn severity(self) -> SeverityId {
match self {
Self::MissingCredential => SeverityId::Low,
Self::ConflictingCredentials => SeverityId::Medium,
}
fn category(reason: AuthFailureReason) -> &'static str {
match reason {
AuthFailureReason::MissingCredential => "missing_credential",
AuthFailureReason::ConflictingCredentials => "conflicting_credentials",
AuthFailureReason::MalformedCredential => "malformed_credential",
AuthFailureReason::InvalidCredential => "invalid_credential",
AuthFailureReason::ExpiredCredential => "expired_credential",
AuthFailureReason::AudienceOrIssuerMismatch => "audience_or_issuer_mismatch",
AuthFailureReason::UnknownIdentity => "unknown_identity",
AuthFailureReason::StaleIdentity => "stale_identity",
AuthFailureReason::IdentityProviderUnavailable => "identity_provider_unavailable",
}
}
/// The mechanism a caller attempted to authenticate with.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AuthMechanism {
Mtls,
fn severity(reason: AuthFailureReason) -> SeverityId {
match reason {
AuthFailureReason::MissingCredential | AuthFailureReason::ExpiredCredential => {
SeverityId::Low
}
AuthFailureReason::ConflictingCredentials
| AuthFailureReason::MalformedCredential
| AuthFailureReason::InvalidCredential
| AuthFailureReason::AudienceOrIssuerMismatch
| AuthFailureReason::UnknownIdentity
| AuthFailureReason::StaleIdentity
| AuthFailureReason::IdentityProviderUnavailable => SeverityId::Medium,
}
}
impl AuthMechanism {
fn label(self) -> &'static str {
match self {
Self::Mtls => "mtls",
}
fn with_protocol(
builder: AuthenticationBuilder<'_>,
mechanism: AuthMechanism,
) -> AuthenticationBuilder<'_> {
match mechanism {
AuthMechanism::Oidc => builder.auth_protocol(AuthProtocolId::OpenId),
AuthMechanism::Mtls => builder.auth_protocol_other("mtls"),
AuthMechanism::SandboxJwt => builder.auth_protocol_other("sandbox_jwt"),
AuthMechanism::ComputeDriver => builder.auth_protocol_other("compute_driver"),
AuthMechanism::GatewayPeer => builder.auth_protocol_other("k8s_service_account"),
}
}
@@ -65,16 +69,16 @@ pub fn emit_authentication_failure(
)
.action(ActionId::Denied)
.disposition(DispositionId::Blocked)
.severity(reason.severity())
.severity(severity(reason))
.status(StatusId::Failure)
.status_detail(reason.as_str())
.status_detail(category(reason))
.http_request(HttpRequest {
http_method: HttpMethod::Post,
url: Some(Url::from_path(path)),
})
.message(format!("authentication failed: {}", reason.as_str()));
.message(format!("authentication failed: {}", category(reason)));
if let Some(mechanism) = mechanism {
builder = builder.auth_protocol_other(mechanism.label());
builder = with_protocol(builder, mechanism);
}
ocsf_emit!(builder.build());
}
crates/openshell-server/src/auth/authenticator.rs
+198 -37
View File
@@ -7,10 +7,12 @@
//! [`AuthenticatorChain`] of [`Authenticator`] implementations. The chain
//! evaluates authenticators in order; the first one that recognizes the
//! caller produces the [`Principal`]. An authenticator that does not apply
//! (e.g. an OIDC authenticator seeing no Bearer header) returns `Ok(None)`
//! so the chain falls through to the next. An authenticator that *does*
//! apply but rejects the caller returns `Err(Status)`, which terminates
//! the chain — fail-closed.
//! (e.g. an OIDC authenticator seeing no Bearer header) returns
//! [`Authentication::NotApplicable`] so the chain falls through to the next.
//! An authenticator that rejects a credential another authenticator may still
//! accept returns [`Authentication::Declined`], which also falls through. An
//! authenticator that *does* apply but rejects the caller returns
//! `Err(AuthRejection)`, which terminates the chain — fail-closed.
//!
//! Live authenticators slotting into the chain:
//! - [`super::sandbox_jwt::SandboxSessionJwtAuthenticator`] — generation-bound
@@ -31,20 +33,91 @@ use tonic::Status;
pub trait Authenticator: Send + Sync + 'static {
/// Inspect an inbound request and return the authenticated principal.
///
/// - `Ok(Some(principal))` — this authenticator recognized the caller.
/// - `Accepted(principal)` — this authenticator recognized the caller.
/// The chain stops and the principal is inserted into request
/// extensions.
/// - `Ok(None)` — this authenticator does not apply (e.g. no Bearer
/// - `NotApplicable` — this authenticator does not apply (e.g. no Bearer
/// token for an OIDC authenticator). The chain falls through to
/// the next authenticator.
/// - `Err(status)` — this authenticator applies but rejected the
/// caller. The chain terminates and the status is returned to the
/// client. Fail-closed.
/// - `Declined(decline)` — this authenticator rejected the credential,
/// but another may accept it. The chain falls through, keeping the
/// decline to classify the request if nothing accepts it.
/// - `Err(rejection)` — this authenticator applies but rejected the
/// caller. The chain terminates and the rejection's status is returned
/// to the client. Fail-closed.
async fn authenticate(
&self,
headers: &http::HeaderMap,
path: &str,
) -> Result<Option<Principal>, Status>;
) -> Result<Authentication, AuthRejection>;
}
/// Outcome of an authenticator, or of the whole chain, that did not fail
/// closed.
#[derive(Debug, Clone)]
pub enum Authentication {
Accepted(Principal),
NotApplicable,
Declined(AuthDecline),
}
/// A credential an authenticator rejected without ending the chain.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct AuthDecline {
pub mechanism: AuthMechanism,
pub reason: AuthFailureReason,
}
/// Why an authentication attempt failed.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AuthFailureReason {
/// No credential was presented.
MissingCredential,
/// Credentials for more than one mechanism were presented together.
ConflictingCredentials,
/// The credential could not be parsed.
MalformedCredential,
/// The credential failed signature, key, or claim verification.
InvalidCredential,
/// The credential was valid but has expired.
ExpiredCredential,
/// The credential was issued by or for a different party.
AudienceOrIssuerMismatch,
/// The credential names an identity the gateway does not know.
UnknownIdentity,
/// The credential names an identity that has since been replaced or revoked.
StaleIdentity,
/// The service that verifies the credential could not be reached.
IdentityProviderUnavailable,
}
/// The mechanism a caller attempted to authenticate with.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AuthMechanism {
Mtls,
Oidc,
SandboxJwt,
ComputeDriver,
GatewayPeer,
}
/// A rejected authentication attempt: the status returned to the caller and
/// the category recorded in the audit event.
#[derive(Debug, Clone)]
pub struct AuthRejection {
pub mechanism: AuthMechanism,
pub reason: AuthFailureReason,
pub status: Status,
}
impl AuthRejection {
pub fn new(mechanism: AuthMechanism, reason: AuthFailureReason, status: Status) -> Self {
Self {
mechanism,
reason,
status,
}
}
}
/// Bearer credential from the request's `authorization` header, if any.
@@ -77,20 +150,27 @@ impl AuthenticatorChain {
}
}
/// Run the chain. Returns the first principal produced. If every
/// authenticator returns `Ok(None)`, the result is `Ok(None)` — the
/// router translates that to `unauthenticated`.
/// Run the chain. Returns the first principal produced, or the first
/// rejection. If neither occurs, the result is the first decline, or
/// `NotApplicable` — the router translates either to `unauthenticated`.
pub async fn authenticate(
&self,
headers: &http::HeaderMap,
path: &str,
) -> Result<Option<Principal>, Status> {
) -> Result<Authentication, AuthRejection> {
let mut declined: Option<AuthDecline> = None;
for authenticator in self.authenticators.iter() {
if let Some(principal) = authenticator.authenticate(headers, path).await? {
return Ok(Some(principal));
match authenticator.authenticate(headers, path).await? {
Authentication::Accepted(principal) => {
return Ok(Authentication::Accepted(principal));
}
Authentication::NotApplicable => {}
Authentication::Declined(decline) => {
declined.get_or_insert(decline);
}
}
}
Ok(None)
Ok(declined.map_or(Authentication::NotApplicable, Authentication::Declined))
}
}
@@ -112,12 +192,12 @@ pub mod test_support {
/// crypto. Each call records the path it was invoked with so tests can
/// assert chain ordering.
pub struct MockAuthenticator {
pub outcome: Result<Option<Principal>, Status>,
pub outcome: Result<Authentication, AuthRejection>,
pub calls: Mutex<Vec<String>>,
}
impl MockAuthenticator {
pub fn returning(outcome: Result<Option<Principal>, Status>) -> Self {
pub fn returning(outcome: Result<Authentication, AuthRejection>) -> Self {
Self {
outcome,
calls: Mutex::new(Vec::new()),
@@ -135,11 +215,20 @@ pub mod test_support {
&self,
_headers: &http::HeaderMap,
path: &str,
) -> Result<Option<Principal>, Status> {
) -> Result<Authentication, AuthRejection> {
self.calls.lock().unwrap().push(path.to_string());
self.outcome.clone()
}
}
impl Authentication {
pub fn principal(self) -> Option<Principal> {
match self {
Self::Accepted(principal) => Some(principal),
Self::NotApplicable | Self::Declined(_) => None,
}
}
}
}
#[cfg(test)]
@@ -179,17 +268,18 @@ mod tests {
#[tokio::test]
async fn chain_returns_first_match() {
let first = Arc::new(MockAuthenticator::returning(Ok(Some(user_principal(
"alice",
)))));
let second = Arc::new(MockAuthenticator::returning(Ok(Some(user_principal(
"bob",
)))));
let first = Arc::new(MockAuthenticator::returning(Ok(Authentication::Accepted(
user_principal("alice"),
))));
let second = Arc::new(MockAuthenticator::returning(Ok(Authentication::Accepted(
user_principal("bob"),
))));
let chain = AuthenticatorChain::new(vec![first.clone(), second.clone()]);
let result = chain
.authenticate(&http::HeaderMap::new(), "/some/path")
.await
.unwrap()
.principal()
.expect("expected a principal");
match result {
Principal::User(u) => assert_eq!(u.identity.subject, "alice"),
@@ -205,15 +295,18 @@ mod tests {
#[tokio::test]
async fn chain_falls_through_on_none() {
let first = Arc::new(MockAuthenticator::returning(Ok(None)));
let second = Arc::new(MockAuthenticator::returning(Ok(Some(user_principal(
"bob",
)))));
let first = Arc::new(MockAuthenticator::returning(Ok(
Authentication::NotApplicable,
)));
let second = Arc::new(MockAuthenticator::returning(Ok(Authentication::Accepted(
user_principal("bob"),
))));
let chain = AuthenticatorChain::new(vec![first.clone(), second.clone()]);
let result = chain
.authenticate(&http::HeaderMap::new(), "/some/path")
.await
.unwrap()
.principal()
.expect("expected a principal");
match result {
Principal::User(u) => assert_eq!(u.identity.subject, "bob"),
@@ -225,18 +318,20 @@ mod tests {
#[tokio::test]
async fn chain_fails_closed_on_first_error() {
let first = Arc::new(MockAuthenticator::returning(Err(Status::unauthenticated(
"bad token",
let first = Arc::new(MockAuthenticator::returning(Err(AuthRejection::new(
AuthMechanism::Oidc,
AuthFailureReason::InvalidCredential,
Status::unauthenticated("bad token"),
))));
let second = Arc::new(MockAuthenticator::returning(Ok(Authentication::Accepted(
user_principal("bob"),
))));
let second = Arc::new(MockAuthenticator::returning(Ok(Some(user_principal(
"bob",
)))));
let chain = AuthenticatorChain::new(vec![first.clone(), second.clone()]);
let err = chain
.authenticate(&http::HeaderMap::new(), "/some/path")
.await
.expect_err("must short-circuit on error");
assert_eq!(err.code(), tonic::Code::Unauthenticated);
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
assert_eq!(first.call_count(), 1);
assert_eq!(
second.call_count(),
@@ -252,6 +347,72 @@ mod tests {
.authenticate(&http::HeaderMap::new(), "/some/path")
.await
.unwrap();
assert!(result.is_none());
assert!(matches!(result, Authentication::NotApplicable));
}
fn peer_decline() -> AuthDecline {
AuthDecline {
mechanism: AuthMechanism::GatewayPeer,
reason: AuthFailureReason::ExpiredCredential,
}
}
#[tokio::test]
async fn chain_falls_through_a_decline_and_reports_it_when_nothing_accepts() {
let first = Arc::new(MockAuthenticator::returning(Ok(Authentication::Declined(
peer_decline(),
))));
let second = Arc::new(MockAuthenticator::returning(Ok(
Authentication::NotApplicable,
)));
let chain = AuthenticatorChain::new(vec![first, second.clone()]);
let result = chain
.authenticate(&http::HeaderMap::new(), "/some/path")
.await
.unwrap();
assert_eq!(second.call_count(), 1);
assert!(matches!(result, Authentication::Declined(d) if d == peer_decline()));
}
#[tokio::test]
async fn chain_accepts_a_principal_after_a_decline() {
let first = Arc::new(MockAuthenticator::returning(Ok(Authentication::Declined(
peer_decline(),
))));
let second = Arc::new(MockAuthenticator::returning(Ok(Authentication::Accepted(
user_principal("bob"),
))));
let chain = AuthenticatorChain::new(vec![first, second]);
let result = chain
.authenticate(&http::HeaderMap::new(), "/some/path")
.await
.unwrap();
assert!(result.principal().is_some());
}
#[tokio::test]
async fn rejection_after_a_decline_keeps_its_own_classification() {
let first = Arc::new(MockAuthenticator::returning(Ok(Authentication::Declined(
peer_decline(),
))));
let second = Arc::new(MockAuthenticator::returning(Err(AuthRejection::new(
AuthMechanism::Oidc,
AuthFailureReason::InvalidCredential,
Status::unauthenticated("invalid token: unknown signing key"),
))));
let chain = AuthenticatorChain::new(vec![first, second]);
let err = chain
.authenticate(&http::HeaderMap::new(), "/some/path")
.await
.expect_err("a later rejection still fails closed");
assert_eq!(err.status.message(), "invalid token: unknown signing key");
assert_eq!(err.mechanism, AuthMechanism::Oidc);
assert_eq!(err.reason, AuthFailureReason::InvalidCredential);
}
}
crates/openshell-server/src/auth/compute_driver.rs
+87 -25
View File
@@ -3,11 +3,12 @@
//! Compute-driver delegated sandbox bootstrap authentication.
use super::authenticator::{Authenticator, bearer_credential};
use super::authenticator::{AuthFailureReason, AuthMechanism};
use super::authenticator::{AuthRejection, Authentication, Authenticator, bearer_credential};
use super::principal::{Principal, SandboxIdentitySource, SandboxPrincipal};
use crate::compute::ComputeRuntime;
use async_trait::async_trait;
use tonic::Status;
use tonic::{Code, Status};
/// The only public gateway method on which driver-native credentials apply.
pub const ISSUE_SANDBOX_TOKEN_PATH: &str = "/openshell.v1.OpenShell/IssueSandboxToken";
@@ -29,36 +30,59 @@ impl Authenticator for ComputeDriverAuthenticator {
&self,
headers: &http::HeaderMap,
path: &str,
) -> Result<Option<Principal>, Status> {
) -> Result<Authentication, AuthRejection> {
if path != ISSUE_SANDBOX_TOKEN_PATH {
return Ok(None);
return Ok(Authentication::NotApplicable);
}
let Some(credential) = bearer_credential(headers) else {
return Ok(None);
return Ok(Authentication::NotApplicable);
};
let authenticated = self.compute.authenticate_sandbox(credential).await?;
let authenticated = self
.compute
.authenticate_sandbox(credential)
.await
.map_err(|status| reject(driver_failure_reason(status.code()), status))?;
let sandbox_id = authenticated.sandbox_id;
if sandbox_id.is_empty() {
return Err(Status::permission_denied(
"compute driver returned an empty sandbox identity",
return Err(reject(
AuthFailureReason::UnknownIdentity,
Status::permission_denied("compute driver returned an empty sandbox identity"),
));
}
if authenticated.runtime_identity.is_empty() {
return Err(Status::permission_denied(
"compute driver returned an empty runtime identity",
return Err(reject(
AuthFailureReason::UnknownIdentity,
Status::permission_denied("compute driver returned an empty runtime identity"),
));
}
Ok(Some(Principal::Sandbox(SandboxPrincipal {
sandbox_id,
source: SandboxIdentitySource::ComputeDriver {
driver_name: self.compute.configured_driver_name().to_string(),
runtime_identity: authenticated.runtime_identity,
Ok(Authentication::Accepted(Principal::Sandbox(
SandboxPrincipal {
sandbox_id,
source: SandboxIdentitySource::ComputeDriver {
driver_name: self.compute.configured_driver_name().to_string(),
runtime_identity: authenticated.runtime_identity,
},
trust_domain: Some("openshell".to_string()),
},
trust_domain: Some("openshell".to_string()),
})))
)))
}
}
fn reject(reason: AuthFailureReason, status: Status) -> AuthRejection {
AuthRejection::new(AuthMechanism::ComputeDriver, reason, status)
}
fn driver_failure_reason(code: Code) -> AuthFailureReason {
match code {
Code::Unauthenticated
| Code::PermissionDenied
| Code::InvalidArgument
| Code::FailedPrecondition => AuthFailureReason::InvalidCredential,
Code::NotFound => AuthFailureReason::UnknownIdentity,
_ => AuthFailureReason::IdentityProviderUnavailable,
}
}
@@ -69,7 +93,6 @@ mod tests {
use crate::compute::{NoopTestDriver, new_test_runtime_with_driver};
use crate::persistence::Store;
use std::sync::Arc;
use tonic::Code;
fn bearer_headers(token: &str) -> http::HeaderMap {
let mut headers = http::HeaderMap::new();
@@ -97,6 +120,7 @@ mod tests {
)
.await
.unwrap()
.principal()
.expect("driver credential should authenticate");
let Principal::Sandbox(principal) = principal else {
@@ -128,7 +152,7 @@ mod tests {
.await
.unwrap();
assert!(principal.is_some());
assert!(principal.principal().is_some());
}
#[tokio::test]
@@ -147,7 +171,7 @@ mod tests {
.await
.unwrap();
assert!(result.is_none());
assert!(matches!(result, Authentication::NotApplicable));
}
#[tokio::test]
@@ -159,7 +183,7 @@ mod tests {
.await
.unwrap();
assert!(result.is_none());
assert!(matches!(result, Authentication::NotApplicable));
}
#[tokio::test]
@@ -174,7 +198,8 @@ mod tests {
.await
.expect_err("empty identity must fail closed");
assert_eq!(error.code(), Code::PermissionDenied);
assert_eq!(error.status.code(), Code::PermissionDenied);
assert_eq!(error.reason, AuthFailureReason::UnknownIdentity);
}
#[tokio::test]
@@ -193,7 +218,8 @@ mod tests {
.await
.expect_err("empty runtime identity must fail closed");
assert_eq!(error.code(), Code::PermissionDenied);
assert_eq!(error.status.code(), Code::PermissionDenied);
assert_eq!(error.reason, AuthFailureReason::UnknownIdentity);
}
#[tokio::test]
@@ -212,7 +238,43 @@ mod tests {
.await
.expect_err("driver errors must propagate");
assert_eq!(error.code(), Code::Unavailable);
assert_eq!(error.message(), "driver unavailable");
assert_eq!(error.status.code(), Code::Unavailable);
assert_eq!(error.status.message(), "driver unavailable");
assert_eq!(error.mechanism, AuthMechanism::ComputeDriver);
assert_eq!(error.reason, AuthFailureReason::IdentityProviderUnavailable);
}
#[tokio::test]
async fn driver_credential_rejection_is_classified_by_status() {
for (code, reason) in [
(Code::Unauthenticated, AuthFailureReason::InvalidCredential),
(Code::PermissionDenied, AuthFailureReason::InvalidCredential),
(Code::NotFound, AuthFailureReason::UnknownIdentity),
(
Code::DeadlineExceeded,
AuthFailureReason::IdentityProviderUnavailable,
),
(
Code::Internal,
AuthFailureReason::IdentityProviderUnavailable,
),
] {
let auth = authenticator(NoopTestDriver::failing_sandbox_authentication(
code,
"driver says no",
))
.await;
let error = auth
.authenticate(
&bearer_headers("driver-credential"),
ISSUE_SANDBOX_TOKEN_PATH,
)
.await
.expect_err("driver errors must propagate");
assert_eq!(error.status.code(), code);
assert_eq!(error.reason, reason, "{code:?}");
}
}
}
crates/openshell-server/src/auth/oidc.rs
+138 -32
View File
@@ -10,7 +10,8 @@
//! This module owns authentication (verifying who the caller is).
//! Authorization (deciding what the caller can do) is in `authz.rs`.
use super::authenticator::{Authenticator, bearer_credential};
use super::authenticator::{AuthFailureReason, AuthMechanism};
use super::authenticator::{AuthRejection, Authentication, Authenticator, bearer_credential};
use super::identity::{Identity, IdentityProvider};
use super::principal::{Principal, UserPrincipal};
use async_trait::async_trait;
@@ -792,23 +793,32 @@ impl JwksCache {
///
/// This is the authentication step — it verifies the caller's identity
/// but does not check authorization (that's `authz::AuthzPolicy::check`).
pub async fn validate_token(&self, token: &str) -> Result<Identity, Status> {
pub async fn validate_token(&self, token: &str) -> Result<Identity, AuthRejection> {
crate::install_jsonwebtoken_crypto_provider();
self.refresh_if_stale().await.map_err(|e| {
warn!(error = %e, "JWKS refresh failed");
Status::internal("OIDC key refresh failed")
reject(
AuthFailureReason::IdentityProviderUnavailable,
Status::internal("OIDC key refresh failed"),
)
})?;
// Decode the header to find the key ID.
let header = decode_header(token).map_err(|e| {
debug!(error = %e, "Failed to decode JWT header");
Status::unauthenticated("invalid token")
reject(
AuthFailureReason::MalformedCredential,
Status::unauthenticated("invalid token"),
)
})?;
let kid = header.kid.ok_or_else(|| {
debug!("JWT has no kid in header");
Status::unauthenticated("invalid token: missing kid")
reject(
AuthFailureReason::MalformedCredential,
Status::unauthenticated("invalid token: missing kid"),
)
})?;
// Refresh once on kid miss.
@@ -817,7 +827,10 @@ impl JwksCache {
drop(keys_guard);
self.refresh_keys_coalesced().await.map_err(|e| {
warn!(error = %e, "JWKS refresh on kid miss failed");
Status::internal("OIDC key refresh failed")
reject(
AuthFailureReason::IdentityProviderUnavailable,
Status::internal("OIDC key refresh failed"),
)
})?;
keys_guard = self.keys.read().await;
}
@@ -827,7 +840,10 @@ impl JwksCache {
.map(|(dk, alg)| (dk.clone(), *alg))
.ok_or_else(|| {
debug!(kid = %kid, "JWT kid not found in JWKS");
Status::unauthenticated("invalid token: unknown signing key")
reject(
AuthFailureReason::InvalidCredential,
Status::unauthenticated("invalid token: unknown signing key"),
)
})?;
drop(keys_guard);
@@ -837,7 +853,10 @@ impl JwksCache {
expected = ?cached_algorithm,
"JWT algorithm mismatch for cached signing key"
);
return Err(Status::unauthenticated("invalid token"));
return Err(reject(
AuthFailureReason::InvalidCredential,
Status::unauthenticated("invalid token"),
));
}
let mut validation = Validation::new(cached_algorithm);
@@ -847,7 +866,10 @@ impl JwksCache {
let token_data = decode::<OidcClaims>(token, &decoding_key, &validation).map_err(|e| {
debug!(error = %e, "JWT validation failed");
Status::unauthenticated(format!("invalid token: {e}"))
reject(
decode_failure_reason(e.kind()),
Status::unauthenticated(format!("invalid token: {e}")),
)
})?;
let mut claims = token_data.claims;
@@ -869,10 +891,31 @@ impl JwksCache {
}
}
fn reject(reason: AuthFailureReason, status: Status) -> AuthRejection {
AuthRejection::new(AuthMechanism::Oidc, reason, status)
}
fn decode_failure_reason(kind: &jsonwebtoken::errors::ErrorKind) -> AuthFailureReason {
use jsonwebtoken::errors::ErrorKind;
match kind {
ErrorKind::ExpiredSignature => AuthFailureReason::ExpiredCredential,
ErrorKind::InvalidIssuer | ErrorKind::InvalidAudience => {
AuthFailureReason::AudienceOrIssuerMismatch
}
ErrorKind::InvalidToken
| ErrorKind::MissingRequiredClaim(_)
| ErrorKind::InvalidClaimFormat(_)
| ErrorKind::Base64(_)
| ErrorKind::Json(_)
| ErrorKind::Utf8(_) => AuthFailureReason::MalformedCredential,
_ => AuthFailureReason::InvalidCredential,
}
}
/// Authenticator that validates `Authorization: Bearer <jwt>` headers against
/// the configured OIDC issuer.
///
/// Returns `Ok(None)` when no Bearer header is present, so the chain can fall
/// Returns `NotApplicable` when no Bearer header is present, so the chain can fall
/// through to other authenticators (e.g. the gateway-minted sandbox JWT
/// authenticator).
pub struct OidcAuthenticator {
@@ -891,13 +934,15 @@ impl Authenticator for OidcAuthenticator {
&self,
headers: &http::HeaderMap,
_path: &str,
) -> Result<Option<Principal>, Status> {
) -> Result<Authentication, AuthRejection> {
let Some(token) = bearer_credential(headers) else {
return Ok(None);
return Ok(Authentication::NotApplicable);
};
let identity = self.cache.validate_token(token).await?;
Ok(Some(Principal::User(UserPrincipal { identity })))
Ok(Authentication::Accepted(Principal::User(UserPrincipal {
identity,
})))
}
}
@@ -1479,10 +1524,12 @@ mod tests {
};
let forged = format!("{}.{forged_payload}.{}", segments[0], segments[2]);
cache
let rejection = cache
.validate_token(&forged)
.await
.expect_err("a swapped payload must fail the signature check");
assert_eq!(rejection.mechanism, AuthMechanism::Oidc);
assert_eq!(rejection.reason, AuthFailureReason::InvalidCredential);
}
#[tokio::test]
@@ -1502,10 +1549,12 @@ mod tests {
)
.expect("sign with unrelated key");
cache
let rejection = cache
.validate_token(&token)
.await
.expect_err("a token signed by a key outside the JWKS must be rejected");
assert_eq!(rejection.mechanism, AuthMechanism::Oidc);
assert_eq!(rejection.reason, AuthFailureReason::InvalidCredential);
}
#[tokio::test]
@@ -1519,10 +1568,12 @@ mod tests {
TEST_KID,
);
cache
let rejection = cache
.validate_token(&token)
.await
.expect_err("an expired token must be rejected");
assert_eq!(rejection.mechanism, AuthMechanism::Oidc);
assert_eq!(rejection.reason, AuthFailureReason::ExpiredCredential);
}
#[tokio::test]
@@ -1535,10 +1586,15 @@ mod tests {
TEST_KID,
);
cache
let rejection = cache
.validate_token(&token)
.await
.expect_err("a token from another issuer must be rejected");
assert_eq!(rejection.mechanism, AuthMechanism::Oidc);
assert_eq!(
rejection.reason,
AuthFailureReason::AudienceOrIssuerMismatch
);
}
#[tokio::test]
@@ -1551,10 +1607,15 @@ mod tests {
TEST_KID,
);
cache
let rejection = cache
.validate_token(&token)
.await
.expect_err("a token minted for another audience must be rejected");
assert_eq!(rejection.mechanism, AuthMechanism::Oidc);
assert_eq!(
rejection.reason,
AuthFailureReason::AudienceOrIssuerMismatch
);
}
#[tokio::test]
@@ -1567,10 +1628,54 @@ mod tests {
"rotated-away-key",
);
cache
let rejection = cache
.validate_token(&token)
.await
.expect_err("a token naming a kid absent from the JWKS must be rejected");
assert_eq!(rejection.mechanism, AuthMechanism::Oidc);
assert_eq!(rejection.reason, AuthFailureReason::InvalidCredential);
}
#[tokio::test]
async fn unparseable_token_is_rejected_as_malformed() {
let server = wiremock::MockServer::start().await;
let cache = cache_with_mock_issuer(&server).await;
let rejection = cache
.validate_token("not-a-jwt")
.await
.expect_err("an unparseable token must be rejected");
assert_eq!(rejection.reason, AuthFailureReason::MalformedCredential);
assert_eq!(rejection.status.code(), tonic::Code::Unauthenticated);
}
#[tokio::test]
async fn jwks_outage_is_rejected_as_identity_provider_unavailable() {
use wiremock::matchers::{method, path};
use wiremock::{Mock, ResponseTemplate};
let server = wiremock::MockServer::start().await;
let cache = cache_with_mock_issuer(&server).await;
server.reset().await;
Mock::given(method("GET"))
.and(path("/jwks"))
.respond_with(ResponseTemplate::new(503))
.mount(&server)
.await;
let token = mint_rs256(
&claims_for(&server.uri(), TEST_AUDIENCE, now_secs() + 3600),
"rotated-in-key",
);
let rejection = cache
.validate_token(&token)
.await
.expect_err("a token cannot be verified while the JWKS is unreachable");
assert_eq!(
rejection.reason,
AuthFailureReason::IdentityProviderUnavailable
);
assert_eq!(rejection.status.code(), tonic::Code::Internal);
}
mod jwks_validation {
@@ -1936,7 +2041,7 @@ mod tests {
});
let unknown_kid_token = encode(&header, &claims, &good.encoding_key).unwrap();
let err = cache.validate_token(&unknown_kid_token).await.unwrap_err();
assert_eq!(err.code(), tonic::Code::Unauthenticated);
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
assert!(cache.validate_token(&token).await.is_ok());
}
@@ -2021,7 +2126,7 @@ mod tests {
.checked_sub(grace + Duration::from_secs(1))
.unwrap();
let err = cache.validate_token(&token).await.unwrap_err();
assert_eq!(err.code(), tonic::Code::Unauthenticated);
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
}
#[tokio::test]
@@ -2083,7 +2188,7 @@ mod tests {
let previous_refresh = *cache.last_kid_miss_refresh.read().await;
let err = cache.validate_token(&unknown_kid_token).await.unwrap_err();
assert_eq!(err.code(), tonic::Code::Unauthenticated);
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
assert!(*cache.last_kid_miss_refresh.read().await > previous_refresh);
// Keep the remaining lookups deterministically inside the same
@@ -2093,7 +2198,7 @@ mod tests {
Instant::now().checked_add(Duration::from_mins(1)).unwrap();
for _ in 0..4 {
let err = cache.validate_token(&unknown_kid_token).await.unwrap_err();
assert_eq!(err.code(), tonic::Code::Unauthenticated);
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
}
let requests = server.received_requests().await.unwrap();
@@ -2114,7 +2219,7 @@ mod tests {
let token = sign_token(&ec, &issuer, &config.audience, "user-1");
let forged = token_with_header_alg(&token, Algorithm::HS256);
let err = cache.validate_token(&forged).await.unwrap_err();
assert_eq!(err.code(), tonic::Code::Unauthenticated);
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
}
#[tokio::test]
@@ -2131,7 +2236,8 @@ mod tests {
});
let token = encode(&header, &claims, &key.encoding_key).unwrap();
let err = cache.validate_token(&token).await.unwrap_err();
assert_eq!(err.code(), tonic::Code::Unauthenticated);
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
assert_eq!(err.reason, AuthFailureReason::MalformedCredential);
}
#[tokio::test]
@@ -2143,7 +2249,7 @@ mod tests {
let (cache, issuer, config, _server) = cache_from_jwks(jwks).await;
let token = sign_token(&signing, &issuer, &config.audience, "user-1");
let err = cache.validate_token(&token).await.unwrap_err();
assert_eq!(err.code(), tonic::Code::Unauthenticated);
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
}
#[tokio::test]
@@ -2171,7 +2277,7 @@ mod tests {
rs256_variant.algorithm = Algorithm::RS256;
let mismatched = sign_token(&rs256_variant, &issuer, &config.audience, "user-1");
let err = cache.validate_token(&mismatched).await.unwrap_err();
assert_eq!(err.code(), tonic::Code::Unauthenticated);
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
}
#[tokio::test]
@@ -2204,7 +2310,7 @@ mod tests {
let sign_token_str = sign_token(&sign_key, &issuer, &config.audience, "sign-user");
let err = cache.validate_token(&sign_token_str).await.unwrap_err();
assert_eq!(err.code(), tonic::Code::Unauthenticated);
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
}
#[tokio::test]
@@ -2218,7 +2324,7 @@ mod tests {
let (cache, issuer, config, _server) = cache_from_jwks(jwks).await;
let token = sign_token(&key, &issuer, &config.audience, "user-1");
let err = cache.validate_token(&token).await.unwrap_err();
assert_eq!(err.code(), tonic::Code::Unauthenticated);
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
}
#[tokio::test]
@@ -2267,7 +2373,7 @@ mod tests {
});
let second_token = encode(&header, &claims, &second.encoding_key).unwrap();
let err = cache.validate_token(&second_token).await.unwrap_err();
assert_eq!(err.code(), tonic::Code::Unauthenticated);
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
}
#[tokio::test]
@@ -2279,11 +2385,11 @@ mod tests {
let rsa_token = sign_token(&rsa, &issuer, &config.audience, "rsa-user");
let err = cache.validate_token(&rsa_token).await.unwrap_err();
assert_eq!(err.code(), tonic::Code::Unauthenticated);
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
let eddsa_token = sign_token(&eddsa, &issuer, &config.audience, "eddsa-user");
let err = cache.validate_token(&eddsa_token).await.unwrap_err();
assert_eq!(err.code(), tonic::Code::Unauthenticated);
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
}
}
}
crates/openshell-server/src/auth/peer.rs
+188 -52
View File
@@ -9,7 +9,10 @@
//! the apiserver `TokenReview` API, checks the live pod UID and required labels,
//! and only then produces a [`Principal::Peer`].
use super::authenticator::{Authenticator, bearer_credential};
use super::authenticator::{
AuthDecline, AuthRejection, Authentication, Authenticator, bearer_credential,
};
use super::authenticator::{AuthFailureReason, AuthMechanism};
use super::principal::{PeerPrincipal, Principal};
use async_trait::async_trait;
use k8s_openapi::api::{
@@ -56,7 +59,10 @@ pub struct ResolvedGatewayPeerIdentity {
#[async_trait]
pub trait GatewayPeerIdentityResolver: Send + Sync + 'static {
async fn resolve(&self, token: &str) -> Result<Option<ResolvedGatewayPeerIdentity>, Status>;
async fn resolve(
&self,
token: &str,
) -> Result<Option<ResolvedGatewayPeerIdentity>, AuthRejection>;
}
#[derive(Debug)]
@@ -88,18 +94,21 @@ impl Authenticator for PeerServiceAccountAuthenticator {
&self,
headers: &http::HeaderMap,
path: &str,
) -> Result<Option<Principal>, Status> {
) -> Result<Authentication, AuthRejection> {
if !crate::auth::method_authz::is_peer_callable(path) {
return Ok(None);
return Ok(Authentication::NotApplicable);
}
let Some(token) = bearer_credential(headers) else {
return Ok(None);
return Ok(Authentication::NotApplicable);
};
let Some(resolved) = self.resolver.resolve(token).await? else {
debug!("K8s gateway peer token did not authenticate; falling through");
return Ok(None);
return Ok(Authentication::Declined(AuthDecline {
mechanism: AuthMechanism::GatewayPeer,
reason: AuthFailureReason::InvalidCredential,
}));
};
if let Some(claimed_replica) = headers
@@ -113,18 +122,23 @@ impl Authenticator for PeerServiceAccountAuthenticator {
pod_name = %resolved.pod_name,
"gateway peer replica header does not match authenticated pod"
);
return Err(Status::permission_denied(
"gateway peer replica does not match authenticated pod",
return Err(reject(
AuthFailureReason::InvalidCredential,
Status::permission_denied("gateway peer replica does not match authenticated pod"),
));
}
Ok(Some(Principal::Peer(PeerPrincipal {
Ok(Authentication::Accepted(Principal::Peer(PeerPrincipal {
replica_id: resolved.pod_name,
pod_uid: resolved.pod_uid,
})))
}
}
fn reject(reason: AuthFailureReason, status: Status) -> AuthRejection {
AuthRejection::new(AuthMechanism::GatewayPeer, reason, status)
}
/// Resolver backed by Kubernetes `TokenReview` and a live Pod lookup.
pub struct LiveGatewayPeerResolver {
token_reviews_api: Api<TokenReview>,
@@ -158,7 +172,10 @@ impl LiveGatewayPeerResolver {
#[async_trait]
impl GatewayPeerIdentityResolver for LiveGatewayPeerResolver {
async fn resolve(&self, token: &str) -> Result<Option<ResolvedGatewayPeerIdentity>, Status> {
async fn resolve(
&self,
token: &str,
) -> Result<Option<ResolvedGatewayPeerIdentity>, AuthRejection> {
let review = TokenReview {
metadata: ObjectMeta::default(),
spec: TokenReviewSpec {
@@ -174,11 +191,17 @@ impl GatewayPeerIdentityResolver for LiveGatewayPeerResolver {
.await
.map_err(|err| {
warn!(error = %err, "K8s TokenReview failed for gateway peer");
Status::internal(format!("peer tokenreview failed: {err}"))
reject(
AuthFailureReason::IdentityProviderUnavailable,
Status::internal(format!("peer tokenreview failed: {err}")),
)
})?;
let status = review
.status
.ok_or_else(|| Status::internal("TokenReview response missing status"))?;
let status = review.status.ok_or_else(|| {
reject(
AuthFailureReason::IdentityProviderUnavailable,
Status::internal("TokenReview response missing status"),
)
})?;
let Some(identity) = peer_token_review_identity(
&status,
&self.expected_audience,
@@ -199,14 +222,20 @@ impl GatewayPeerIdentityResolver for LiveGatewayPeerResolver {
error = %err,
"failed to fetch gateway peer pod"
);
Status::internal(format!("gateway peer pod GET failed: {err}"))
reject(
AuthFailureReason::IdentityProviderUnavailable,
Status::internal(format!("gateway peer pod GET failed: {err}")),
)
})?;
let Some(pod) = pod else {
warn!(
pod = %identity.pod_name,
"gateway peer pod referenced by SA token not found"
);
return Err(Status::not_found("gateway peer pod not found"));
return Err(reject(
AuthFailureReason::StaleIdentity,
Status::not_found("gateway peer pod not found"),
));
};
let actual_uid = pod.metadata.uid.as_deref().unwrap_or_default();
@@ -217,8 +246,9 @@ impl GatewayPeerIdentityResolver for LiveGatewayPeerResolver {
actual_uid,
"gateway peer SA token pod UID does not match live pod"
);
return Err(Status::permission_denied(
"gateway peer SA token pod UID mismatch",
return Err(reject(
AuthFailureReason::StaleIdentity,
Status::permission_denied("gateway peer SA token pod UID mismatch"),
));
}
@@ -234,8 +264,9 @@ impl GatewayPeerIdentityResolver for LiveGatewayPeerResolver {
expected_service_account = %self.expected_service_account,
"gateway peer pod service account does not match TokenReview principal"
);
return Err(Status::permission_denied(
"gateway peer pod service account mismatch",
return Err(reject(
AuthFailureReason::InvalidCredential,
Status::permission_denied("gateway peer pod service account mismatch"),
));
}
@@ -322,7 +353,10 @@ impl CachingGatewayPeerResolver {
#[async_trait]
impl GatewayPeerIdentityResolver for CachingGatewayPeerResolver {
async fn resolve(&self, token: &str) -> Result<Option<ResolvedGatewayPeerIdentity>, Status> {
async fn resolve(
&self,
token: &str,
) -> Result<Option<ResolvedGatewayPeerIdentity>, AuthRejection> {
let key = token_cache_key(token);
if let Some(identity) = self.cached(&key) {
return Ok(Some(identity));
@@ -375,7 +409,7 @@ fn peer_token_review_identity(
expected_audience: &str,
namespace: &str,
expected_service_account: &str,
) -> Result<Option<PeerTokenReviewIdentity>, Status> {
) -> Result<Option<PeerTokenReviewIdentity>, AuthRejection> {
if status.authenticated != Some(true) {
debug!(
error = status.error.as_deref().unwrap_or_default(),
@@ -391,19 +425,24 @@ fn peer_token_review_identity(
audiences = ?audiences,
"K8s TokenReview authenticated gateway peer token without expected audience"
);
return Err(Status::unauthenticated(
"gateway peer token audience not accepted",
return Err(reject(
AuthFailureReason::AudienceOrIssuerMismatch,
Status::unauthenticated("gateway peer token audience not accepted"),
));
}
let user = status
.user
.as_ref()
.ok_or_else(|| Status::permission_denied("TokenReview response missing user info"))?;
let username = user
.username
.as_deref()
.ok_or_else(|| Status::permission_denied("TokenReview response missing username"))?;
let user = status.user.as_ref().ok_or_else(|| {
reject(
AuthFailureReason::InvalidCredential,
Status::permission_denied("TokenReview response missing user info"),
)
})?;
let username = user.username.as_deref().ok_or_else(|| {
reject(
AuthFailureReason::InvalidCredential,
Status::permission_denied("TokenReview response missing username"),
)
})?;
let expected_username = format!("system:serviceaccount:{namespace}:{expected_service_account}");
if username != expected_username {
warn!(
@@ -412,8 +451,11 @@ fn peer_token_review_identity(
service_account = %expected_service_account,
"K8s TokenReview principal is not the configured gateway service account"
);
return Err(Status::permission_denied(
"gateway peer token is not from the configured service account",
return Err(reject(
AuthFailureReason::UnknownIdentity,
Status::permission_denied(
"gateway peer token is not from the configured service account",
),
));
}
@@ -423,15 +465,17 @@ fn peer_token_review_identity(
}
#[allow(clippy::result_large_err)]
fn user_extra_one(user: &UserInfo, key: &str) -> Result<String, Status> {
fn user_extra_one(user: &UserInfo, key: &str) -> Result<String, AuthRejection> {
let Some(values) = user.extra.as_ref().and_then(|extra| extra.get(key)) else {
return Err(Status::permission_denied(
"gateway peer token is not pod-bound",
return Err(reject(
AuthFailureReason::InvalidCredential,
Status::permission_denied("gateway peer token is not pod-bound"),
));
};
if values.len() != 1 || values[0].is_empty() {
return Err(Status::permission_denied(
"gateway peer token has invalid pod binding",
return Err(reject(
AuthFailureReason::InvalidCredential,
Status::permission_denied("gateway peer token has invalid pod binding"),
));
}
Ok(values[0].clone())
@@ -441,7 +485,7 @@ fn user_extra_one(user: &UserInfo, key: &str) -> Result<String, Status> {
fn validate_required_pod_labels(
pod: &Pod,
required_labels: &[(String, String)],
) -> Result<(), Status> {
) -> Result<(), AuthRejection> {
let labels = pod.metadata.labels.as_ref();
for (key, expected) in required_labels {
let actual = labels
@@ -456,8 +500,9 @@ fn validate_required_pod_labels(
actual,
"gateway peer pod missing required label"
);
return Err(Status::permission_denied(
"gateway peer pod labels do not match",
return Err(reject(
AuthFailureReason::UnknownIdentity,
Status::permission_denied("gateway peer pod labels do not match"),
));
}
}
@@ -535,12 +580,14 @@ pub mod test_support {
use std::sync::Mutex;
pub struct FakeGatewayPeerResolver {
pub outcome: Result<Option<ResolvedGatewayPeerIdentity>, Status>,
pub outcome: Result<Option<ResolvedGatewayPeerIdentity>, AuthRejection>,
pub seen_tokens: Mutex<Vec<String>>,
}
impl FakeGatewayPeerResolver {
pub fn returning(outcome: Result<Option<ResolvedGatewayPeerIdentity>, Status>) -> Self {
pub fn returning(
outcome: Result<Option<ResolvedGatewayPeerIdentity>, AuthRejection>,
) -> Self {
Self {
outcome,
seen_tokens: Mutex::new(Vec::new()),
@@ -553,12 +600,9 @@ pub mod test_support {
async fn resolve(
&self,
token: &str,
) -> Result<Option<ResolvedGatewayPeerIdentity>, Status> {
) -> Result<Option<ResolvedGatewayPeerIdentity>, AuthRejection> {
self.seen_tokens.lock().unwrap().push(token.to_string());
match &self.outcome {
Ok(opt) => Ok(opt.clone()),
Err(status) => Err(Status::new(status.code(), status.message())),
}
self.outcome.clone()
}
}
}
@@ -644,7 +688,29 @@ mod tests {
"openshell",
)
.expect_err("wrong service account must fail closed");
assert_eq!(err.code(), tonic::Code::PermissionDenied);
assert_eq!(err.status.code(), tonic::Code::PermissionDenied);
assert_eq!(err.mechanism, AuthMechanism::GatewayPeer);
assert_eq!(err.reason, AuthFailureReason::UnknownIdentity);
}
#[test]
fn peer_token_review_identity_rejects_other_audiences() {
let status = token_review_status(
true,
vec!["kubernetes.default.svc"],
"system:serviceaccount:openshell:openshell",
vec![(POD_NAME_EXTRA, "openshell-0"), (POD_UID_EXTRA, "uid-a")],
);
let err = peer_token_review_identity(
&status,
DEFAULT_PEER_TOKEN_AUDIENCE,
"openshell",
"openshell",
)
.expect_err("a token for another audience must fail closed");
assert_eq!(err.status.code(), tonic::Code::Unauthenticated);
assert_eq!(err.reason, AuthFailureReason::AudienceOrIssuerMismatch);
}
#[test]
@@ -669,7 +735,75 @@ mod tests {
)],
)
.expect_err("missing required label must fail");
assert_eq!(err.code(), tonic::Code::PermissionDenied);
assert_eq!(err.status.code(), tonic::Code::PermissionDenied);
assert_eq!(err.reason, AuthFailureReason::UnknownIdentity);
}
async fn decline_for(token: &str) -> AuthDecline {
let resolver = Arc::new(FakeGatewayPeerResolver::returning(Ok(None)));
let auth = PeerServiceAccountAuthenticator::new(resolver);
match auth
.authenticate(&bearer_headers(token), PEER_RELAY_PATH)
.await
.expect("a token TokenReview rejects must fall through")
{
Authentication::Declined(decline) => decline,
other => panic!("expected a decline, got {other:?}"),
}
}
#[tokio::test]
async fn authenticator_declines_tokens_tokenreview_rejects_as_invalid() {
// The payload's exp is unverified, so a past exp is no evidence of expiry.
for token in [
token_expiring_in(-60),
token_expiring_in(3600),
"opaque-token".to_string(),
] {
let decline = decline_for(&token).await;
assert_eq!(decline.mechanism, AuthMechanism::GatewayPeer);
assert_eq!(
decline.reason,
AuthFailureReason::InvalidCredential,
"{token}"
);
}
}
#[tokio::test]
async fn authenticator_rejects_a_replica_header_naming_another_pod() {
let resolver = Arc::new(FakeGatewayPeerResolver::returning(Ok(Some(identity()))));
let auth = PeerServiceAccountAuthenticator::new(resolver);
let mut headers = bearer_headers("token-a");
headers.insert("x-openshell-peer-replica", "openshell-1".parse().unwrap());
let err = auth
.authenticate(&headers, PEER_RELAY_PATH)
.await
.expect_err("a mismatched replica claim must be rejected");
assert_eq!(err.status.code(), tonic::Code::PermissionDenied);
assert_eq!(err.mechanism, AuthMechanism::GatewayPeer);
assert_eq!(err.reason, AuthFailureReason::InvalidCredential);
}
#[tokio::test]
async fn authenticator_preserves_the_resolver_rejection() {
let resolver = Arc::new(FakeGatewayPeerResolver::returning(Err(AuthRejection::new(
AuthMechanism::GatewayPeer,
AuthFailureReason::StaleIdentity,
Status::permission_denied("gateway peer SA token pod UID mismatch"),
))));
let auth = PeerServiceAccountAuthenticator::new(resolver);
let err = auth
.authenticate(&bearer_headers("token-a"), PEER_RELAY_PATH)
.await
.expect_err("resolver rejection must fail closed");
assert_eq!(
err.status.message(),
"gateway peer SA token pod UID mismatch"
);
assert_eq!(err.reason, AuthFailureReason::StaleIdentity);
}
#[tokio::test]
@@ -686,6 +820,7 @@ mod tests {
.authenticate(&bearer_headers("token-a"), PEER_RELAY_PATH)
.await
.unwrap()
.principal()
.expect("principal");
let Principal::Peer(peer) = principal else {
@@ -714,13 +849,14 @@ mod tests {
] {
assert!(matches!(
auth.authenticate(&headers, path).await.unwrap(),
Some(Principal::Peer(_))
Authentication::Accepted(Principal::Peer(_))
));
}
assert!(
auth.authenticate(&headers, "/openshell.v1.OpenShell/ListSandboxes")
.await
.unwrap()
.principal()
.is_none()
);
assert_eq!(resolver.seen_tokens.lock().unwrap().len(), 4);
crates/openshell-server/src/auth/sandbox_jwt.rs
+77 -17
View File
@@ -11,7 +11,8 @@
//! Algorithm: `EdDSA` (Ed25519). Pinned via `Validation::algorithms` to
//! prevent algorithm-confusion attacks.
use super::authenticator::{Authenticator, bearer_credential};
use super::authenticator::{AuthFailureReason, AuthMechanism};
use super::authenticator::{AuthRejection, Authentication, Authenticator, bearer_credential};
use super::principal::{Principal, SandboxIdentitySource, SandboxPrincipal};
use async_trait::async_trait;
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
@@ -34,8 +35,9 @@ use x509_parser::{oid_registry::OID_SIG_ED25519, prelude::FromDer, x509::Subject
use openshell_core::SandboxSessionId;
use openshell_core::jwt::{
AuthenticatedSandboxSession, CredentialEpoch, GATEWAY_SESSION_JWT_TYPE, SandboxId,
SandboxLaunchAuthentication, SandboxRuntimeIdentity, SessionJwtIssuer, SessionJwtVerifier,
SessionTokenProfile, SessionVerificationKey, SupervisorAuthBundle, SystemJwtClock,
SandboxLaunchAuthentication, SandboxRuntimeIdentity, SessionJwtError, SessionJwtIssuer,
SessionJwtVerifier, SessionTokenProfile, SessionVerificationKey, SupervisorAuthBundle,
SystemJwtClock,
};
use openshell_core::sandbox_generation::SandboxGenerationId;
@@ -255,10 +257,17 @@ impl SandboxSessionJwtAuthority {
})
}
pub fn verify_gateway_token(&self, token: &str) -> Result<AuthenticatedSandboxSession, Status> {
self.gateway_verifier
.verify(token)
.map_err(|error| Status::unauthenticated(format!("invalid gateway session: {error}")))
pub fn verify_gateway_token(
&self,
token: &str,
) -> Result<AuthenticatedSandboxSession, AuthRejection> {
self.gateway_verifier.verify(token).map_err(|error| {
AuthRejection::new(
AuthMechanism::SandboxJwt,
session_failure_reason(&error),
Status::unauthenticated(format!("invalid gateway session: {error}")),
)
})
}
}
@@ -292,15 +301,15 @@ impl Authenticator for SandboxSessionJwtAuthenticator {
&self,
headers: &http::HeaderMap,
path: &str,
) -> Result<Option<Principal>, Status> {
) -> Result<Authentication, AuthRejection> {
let Some(token) = bearer_credential(headers) else {
return Ok(None);
return Ok(Authentication::NotApplicable);
};
let Ok(header) = decode_header(token) else {
return Ok(None);
return Ok(Authentication::NotApplicable);
};
if header.typ.as_deref() != Some(GATEWAY_SESSION_JWT_TYPE) {
return Ok(None);
return Ok(Authentication::NotApplicable);
}
let authenticated = self.authority.verify_gateway_token(token)?;
// Refresh performs its own lineage check so the immediately consumed
@@ -309,13 +318,28 @@ impl Authenticator for SandboxSessionJwtAuthenticator {
if path != "/openshell.v1.OpenShell/RefreshSandboxToken" {
crate::auth::sandbox_session::authorize_persisted(&self.store, &authenticated).await?;
}
Ok(Some(Principal::Sandbox(SandboxPrincipal {
sandbox_id: authenticated.sandbox_id.to_string(),
source: SandboxIdentitySource::BootstrapJwt {
issuer: "launch-session".to_string(),
Ok(Authentication::Accepted(Principal::Sandbox(
SandboxPrincipal {
sandbox_id: authenticated.sandbox_id.to_string(),
source: SandboxIdentitySource::BootstrapJwt {
issuer: "launch-session".to_string(),
},
trust_domain: Some("openshell".to_string()),
},
trust_domain: Some("openshell".to_string()),
})))
)))
}
}
fn session_failure_reason(error: &SessionJwtError) -> AuthFailureReason {
match error {
SessionJwtError::Expired => AuthFailureReason::ExpiredCredential,
SessionJwtError::WrongIssuer | SessionJwtError::WrongAudience => {
AuthFailureReason::AudienceOrIssuerMismatch
}
SessionJwtError::MissingKeyId | SessionJwtError::InvalidJti => {
AuthFailureReason::MalformedCredential
}
_ => AuthFailureReason::InvalidCredential,
}
}
@@ -562,6 +586,42 @@ mod tests {
);
}
#[test]
fn session_verification_errors_map_to_reason_categories() {
for (error, reason) in [
(
SessionJwtError::Expired,
AuthFailureReason::ExpiredCredential,
),
(
SessionJwtError::WrongIssuer,
AuthFailureReason::AudienceOrIssuerMismatch,
),
(
SessionJwtError::WrongAudience,
AuthFailureReason::AudienceOrIssuerMismatch,
),
(
SessionJwtError::MissingKeyId,
AuthFailureReason::MalformedCredential,
),
(
SessionJwtError::InvalidJti,
AuthFailureReason::MalformedCredential,
),
(
SessionJwtError::InvalidToken,
AuthFailureReason::InvalidCredential,
),
(
SessionJwtError::UnknownKeyId,
AuthFailureReason::InvalidCredential,
),
] {
assert_eq!(session_failure_reason(&error), reason, "{error:?}");
}
}
#[test]
fn zero_ttl_is_rejected() {
let mat = generate_jwt_key().expect("jwt key");
crates/openshell-server/src/auth/sandbox_session.rs
+64 -30
View File
@@ -12,6 +12,8 @@ use sha2::{Digest as _, Sha256};
use tonic::Status;
use uuid::Uuid;
use crate::auth::authenticator::AuthRejection;
use crate::auth::authenticator::{AuthFailureReason, AuthMechanism};
use crate::persistence::Store;
pub const RUNTIME_GENERATION_ANNOTATION: &str = "internal.openshell.ai/runtime-generation";
@@ -244,16 +246,37 @@ impl PersistedSandboxIdentity {
}
}
fn reject(reason: AuthFailureReason, status: Status) -> AuthRejection {
AuthRejection::new(AuthMechanism::SandboxJwt, reason, status)
}
fn stale_identity() -> AuthRejection {
reject(
AuthFailureReason::StaleIdentity,
Status::unauthenticated("gateway token does not match the active sandbox identity"),
)
}
#[allow(clippy::result_large_err)]
async fn load_identity(
store: &Store,
principal: &AuthenticatedSandboxSession,
) -> Result<(Sandbox, PersistedSandboxIdentity), Status> {
) -> Result<(Sandbox, PersistedSandboxIdentity), AuthRejection> {
let sandbox = store
.get_message::<Sandbox>(principal.sandbox_id.as_str())
.await
.map_err(|error| Status::unavailable(format!("load sandbox identity failed: {error}")))?
.ok_or_else(|| Status::unauthenticated("sandbox identity does not exist"))?;
.map_err(|error| {
reject(
AuthFailureReason::IdentityProviderUnavailable,
Status::unavailable(format!("load sandbox identity failed: {error}")),
)
})?
.ok_or_else(|| {
reject(
AuthFailureReason::UnknownIdentity,
Status::unauthenticated("sandbox identity does not exist"),
)
})?;
let phase = SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown);
if !matches!(
@@ -264,23 +287,28 @@ async fn load_identity(
| SandboxPhase::Completed
| SandboxPhase::Error
) {
return Err(Status::failed_precondition(
"sandbox runtime identity is not active",
return Err(reject(
AuthFailureReason::StaleIdentity,
Status::failed_precondition("sandbox runtime identity is not active"),
));
}
let metadata = sandbox
.metadata
.as_ref()
.ok_or_else(|| Status::unauthenticated("sandbox identity metadata is missing"))?;
let identity = PersistedSandboxIdentity::read(&metadata.annotations)
.map_err(|_| Status::unauthenticated("sandbox runtime identity is invalid"))?;
let metadata = sandbox.metadata.as_ref().ok_or_else(|| {
reject(
AuthFailureReason::UnknownIdentity,
Status::unauthenticated("sandbox identity metadata is missing"),
)
})?;
let identity = PersistedSandboxIdentity::read(&metadata.annotations).map_err(|_| {
reject(
AuthFailureReason::UnknownIdentity,
Status::unauthenticated("sandbox runtime identity is invalid"),
)
})?;
if principal.runtime_generation != identity.runtime_generation
|| principal.auth_epoch != identity.auth_epoch
{
return Err(Status::unauthenticated(
"gateway token does not match the active sandbox identity",
));
return Err(stale_identity());
}
Ok((sandbox, identity))
}
@@ -289,12 +317,10 @@ async fn load_identity(
pub async fn authorize_persisted(
store: &Store,
principal: &AuthenticatedSandboxSession,
) -> Result<PersistedSandboxIdentity, Status> {
) -> Result<PersistedSandboxIdentity, AuthRejection> {
let (_, identity) = load_identity(store, principal).await?;
if principal.token_id != identity.gateway_token_id {
return Err(Status::unauthenticated(
"gateway token does not match the active sandbox identity",
));
return Err(stale_identity());
}
Ok(identity)
}
@@ -311,22 +337,21 @@ pub async fn authorize_refresh(
principal: &AuthenticatedSandboxSession,
request_hash: &RefreshRequestHash,
now: i64,
) -> Result<RefreshAuthorization, Status> {
) -> Result<RefreshAuthorization, AuthRejection> {
let (_, identity) = load_identity(store, principal).await?;
if principal.token_id == identity.gateway_token_id {
return Ok(RefreshAuthorization::Current(identity));
}
let Some(replay) = &identity.refresh_replay else {
return Err(Status::unauthenticated(
"gateway token does not match the active sandbox identity",
));
return Err(stale_identity());
};
if principal.token_id != replay.previous_gateway_token_id
|| request_hash != &replay.request_hash
|| now > replay.replay_until
{
return Err(Status::unauthenticated(
"gateway refresh retry does not match the active lineage",
return Err(reject(
AuthFailureReason::StaleIdentity,
Status::unauthenticated("gateway refresh retry does not match the active lineage"),
));
}
Ok(RefreshAuthorization::Replay(identity))
@@ -343,7 +368,11 @@ pub async fn rotate_gateway_token(
principal: &AuthenticatedSandboxSession,
next: &PersistedSandboxIdentity,
) -> Result<PersistedSandboxIdentity, Status> {
let (sandbox, current) = load_identity(store, principal).await?;
// `authorize_refresh` already audited this bearer; a rejection here means
// the sandbox changed during the refresh.
let (sandbox, current) = load_identity(store, principal)
.await
.map_err(|rejection| rejection.status)?;
if principal.token_id != current.gateway_token_id {
return Err(Status::unauthenticated(
"gateway token does not match the active sandbox identity",
@@ -435,7 +464,8 @@ mod tests {
let error = authorize_persisted(&store, &principal(1, Uuid::from_u128(2)))
.await
.expect_err("a different token lineage must be rejected");
assert_eq!(error.code(), tonic::Code::Unauthenticated);
assert_eq!(error.status.code(), tonic::Code::Unauthenticated);
assert_eq!(error.reason, AuthFailureReason::StaleIdentity);
store
.update_message_cas::<Sandbox, _>("sandbox-a", 0, |sandbox| {
@@ -454,7 +484,8 @@ mod tests {
let error = authorize_persisted(&store, &principal(1, Uuid::from_u128(1)))
.await
.expect_err("old epoch must be revoked on every replica");
assert_eq!(error.code(), tonic::Code::Unauthenticated);
assert_eq!(error.status.code(), tonic::Code::Unauthenticated);
assert_eq!(error.reason, AuthFailureReason::StaleIdentity);
authorize_persisted(&store, &principal(2, Uuid::from_u128(2)))
.await
.expect("new epoch is active");
@@ -468,7 +499,8 @@ mod tests {
let error = authorize_persisted(&store, &principal(2, Uuid::from_u128(2)))
.await
.expect_err("stopped runtime must reject its token");
assert_eq!(error.code(), tonic::Code::FailedPrecondition);
assert_eq!(error.status.code(), tonic::Code::FailedPrecondition);
assert_eq!(error.reason, AuthFailureReason::StaleIdentity);
}
#[tokio::test]
@@ -502,7 +534,8 @@ mod tests {
let error = authorize_persisted(&store, &first)
.await
.expect_err("consumed gateway token must be rejected");
assert_eq!(error.code(), tonic::Code::Unauthenticated);
assert_eq!(error.status.code(), tonic::Code::Unauthenticated);
assert_eq!(error.reason, AuthFailureReason::StaleIdentity);
assert!(matches!(
authorize_refresh(&store, &first, &request_hash, 130)
.await
@@ -512,7 +545,8 @@ mod tests {
let error = authorize_refresh(&store, &first, &request_hash, 131)
.await
.expect_err("predecessor retry must expire");
assert_eq!(error.code(), tonic::Code::Unauthenticated);
assert_eq!(error.status.code(), tonic::Code::Unauthenticated);
assert_eq!(error.reason, AuthFailureReason::StaleIdentity);
authorize_persisted(&store, &principal(1, next.gateway_token_id))
.await
.expect("successor gateway token is current");
crates/openshell-server/src/compute/mod.rs
+1
View File
@@ -12500,6 +12500,7 @@ mod tests {
)
.await
.unwrap_err()
.status
.code(),
Code::Unauthenticated,
);
crates/openshell-server/src/grpc/auth_rpc.rs
+240 -34
View File
@@ -15,6 +15,8 @@
//! ordinary RPCs or select another successor.
use crate::ServerState;
use crate::auth::audit;
use crate::auth::authenticator::{AuthFailureReason, AuthMechanism, AuthRejection};
use crate::auth::identity::IdentityProvider;
use crate::auth::principal::{Principal, SandboxIdentitySource};
use openshell_core::proto::{
@@ -147,6 +149,71 @@ pub async fn handle_issue_sandbox_token(
}
#[allow(clippy::result_large_err, clippy::unused_async)]
const REFRESH_SANDBOX_TOKEN_PATH: &str = "/openshell.v1.OpenShell/RefreshSandboxToken";
/// The authenticator chain defers lineage checks for refresh to this handler,
/// so its rejections are audited here.
async fn authenticate_refresh(
state: &ServerState,
session_authority: &crate::auth::sandbox_jwt::SandboxSessionJwtAuthority,
metadata: &tonic::metadata::MetadataMap,
sandbox_id: &str,
request_hash: &crate::auth::sandbox_session::RefreshRequestHash,
now: i64,
) -> Result<
(
openshell_core::jwt::AuthenticatedSandboxSession,
crate::auth::sandbox_session::RefreshAuthorization,
),
AuthRejection,
> {
let reject = |reason, message: &str| {
AuthRejection::new(
AuthMechanism::SandboxJwt,
reason,
Status::unauthenticated(message),
)
};
let mut authorization_values = metadata.get_all("authorization").iter();
let authorization = authorization_values.next().ok_or_else(|| {
reject(
AuthFailureReason::MissingCredential,
"missing authorization metadata",
)
})?;
if authorization_values.next().is_some() {
return Err(reject(
AuthFailureReason::ConflictingCredentials,
"duplicate authorization metadata",
));
}
let gateway_token = authorization
.to_str()
.ok()
.and_then(openshell_core::auth::strip_bearer_scheme)
.ok_or_else(|| {
reject(
AuthFailureReason::MalformedCredential,
"invalid bearer authorization metadata",
)
})?;
let principal = session_authority.verify_gateway_token(gateway_token)?;
if principal.sandbox_id.as_str() != sandbox_id {
return Err(reject(
AuthFailureReason::InvalidCredential,
"gateway token does not match the authenticated sandbox",
));
}
let authorization = crate::auth::sandbox_session::authorize_refresh(
&state.store,
&principal,
request_hash,
now,
)
.await?;
Ok((principal, authorization))
}
pub async fn handle_refresh_sandbox_token(
state: &Arc<ServerState>,
request: Request<RefreshSandboxTokenRequest>,
@@ -188,36 +255,27 @@ pub async fn handle_refresh_sandbox_token(
.as_ref()
.ok_or_else(|| Status::unavailable("sandbox session minting is not configured"))?;
let authorization_values = request.metadata().get_all("authorization");
let mut authorization_values = authorization_values.iter();
let authorization = authorization_values
.next()
.ok_or_else(|| Status::unauthenticated("missing authorization metadata"))?;
if authorization_values.next().is_some() {
return Err(Status::unauthenticated("duplicate authorization metadata"));
}
let gateway_token = authorization
.to_str()
.ok()
.and_then(openshell_core::auth::strip_bearer_scheme)
.ok_or_else(|| Status::unauthenticated("invalid bearer authorization metadata"))?;
let principal = session_authority.verify_gateway_token(gateway_token)?;
if principal.sandbox_id.as_str() != sandbox.sandbox_id {
return Err(Status::unauthenticated(
"gateway token does not match the authenticated sandbox",
));
}
let request_hash = crate::auth::sandbox_session::RefreshRequestHash::from_extension_services(
&requested_extension_services,
);
let issued_at = current_unix_seconds();
let authorization = crate::auth::sandbox_session::authorize_refresh(
&state.store,
&principal,
let (principal, authorization) = authenticate_refresh(
state,
session_authority,
request.metadata(),
&sandbox.sandbox_id,
&request_hash,
issued_at,
)
.await?;
.await
.map_err(|rejection| {
audit::emit_authentication_failure(
REFRESH_SANDBOX_TOKEN_PATH,
Some(rejection.mechanism),
rejection.reason,
);
rejection.status
})?;
let (successor, should_rotate) = match authorization {
crate::auth::sandbox_session::RefreshAuthorization::Current(identity) => (
identity.next_gateway_token(
@@ -695,6 +753,94 @@ mod tests {
assert_eq!(error.code(), tonic::Code::Unauthenticated);
}
#[tokio::test]
async fn refresh_with_a_consumed_bearer_emits_stale_identity() {
use openshell_ocsf::test_support::OcsfCapture;
let state = state_with_issuer().await;
let request = || {
let mut request = Request::new(RefreshSandboxTokenRequest {
extension_service_names: Vec::new(),
});
request
.extensions_mut()
.insert(sandbox_principal("sandbox-a"));
request
};
let mut first_request = request();
let first = authorize_refresh(&state, &mut first_request).await;
let second = handle_refresh_sandbox_token(&state, first_request)
.await
.expect("first refresh")
.into_inner();
let mut second_request = request();
set_refresh_authorization(&mut second_request, &second.token);
handle_refresh_sandbox_token(&state, second_request)
.await
.expect("successor refresh");
let capture = OcsfCapture::default();
let error = {
let _guard = capture.install();
let mut replay = request();
set_refresh_authorization(&mut replay, &first);
handle_refresh_sandbox_token(&state, replay)
.await
.expect_err("consumed predecessor must be rejected")
};
assert_eq!(error.code(), tonic::Code::Unauthenticated);
assert_eq!(
error.message(),
"gateway refresh retry does not match the active lineage"
);
let events: Vec<_> = capture
.events()
.iter()
.map(|event| event.to_json().unwrap())
.collect();
assert_eq!(events.len(), 1, "{events:?}");
assert_eq!(events[0]["class_uid"], 3002);
assert_eq!(events[0]["status_detail"], "stale_identity");
assert_eq!(events[0]["auth_protocol"], "sandbox_jwt");
assert_eq!(
events[0]["http_request"]["url"]["path"],
"/openshell.v1.OpenShell/RefreshSandboxToken"
);
assert!(!events[0].to_string().contains(&first));
}
#[tokio::test]
async fn successful_or_unauthorized_refresh_emits_no_authentication_event() {
use openshell_ocsf::test_support::OcsfCapture;
let state = state_with_issuer().await;
let capture = OcsfCapture::default();
let _guard = capture.install();
let mut unauthorized = Request::new(RefreshSandboxTokenRequest {
extension_service_names: vec!["unknown-service".to_string()],
});
unauthorized
.extensions_mut()
.insert(sandbox_principal("sandbox-a"));
let current = authorize_refresh(&state, &mut unauthorized).await;
handle_refresh_sandbox_token(&state, unauthorized)
.await
.expect_err("unknown extension service must be rejected");
let mut refresh = Request::new(RefreshSandboxTokenRequest {
extension_service_names: Vec::new(),
});
refresh
.extensions_mut()
.insert(sandbox_principal("sandbox-a"));
set_refresh_authorization(&mut refresh, &current);
handle_refresh_sandbox_token(&state, refresh)
.await
.expect("refresh OK");
assert!(capture.events().is_empty(), "{:?}", capture.events());
}
#[tokio::test]
async fn failed_refresh_does_not_consume_the_gateway_bearer() {
let state = state_with_issuer().await;
@@ -828,13 +974,14 @@ mod tests {
assert_eq!(err.code(), tonic::Code::Unauthenticated);
}
#[tokio::test]
async fn issue_returns_generation_bound_token_and_rejects_it_after_runtime_replacement() {
use crate::auth::authenticator::Authenticator;
async fn issue_session_bearer(
state: &Arc<ServerState>,
) -> (
crate::auth::sandbox_jwt::SandboxSessionJwtAuthenticator,
String,
) {
use crate::auth::principal::SandboxIdentitySource;
use crate::auth::sandbox_jwt::SandboxSessionJwtAuthenticator;
let state = state_with_issuer().await;
let mut req = Request::new(IssueSandboxTokenRequest {});
req.extensions_mut()
.insert(Principal::Sandbox(SandboxPrincipal {
@@ -845,32 +992,46 @@ mod tests {
},
trust_domain: Some("openshell".to_string()),
}));
let resp = handle_issue_sandbox_token(&state, req)
let resp = handle_issue_sandbox_token(state, req)
.await
.expect("issue OK")
.into_inner();
assert!(!resp.token.is_empty());
assert!(resp.expiration_time.is_some());
let authenticator = SandboxSessionJwtAuthenticator::new(
let authenticator = crate::auth::sandbox_jwt::SandboxSessionJwtAuthenticator::new(
state
.sandbox_session_jwt_authority
.clone()
.expect("session authority"),
state.store.clone(),
);
(authenticator, resp.token)
}
fn bearer_headers(token: &str) -> http::HeaderMap {
let mut headers = http::HeaderMap::new();
headers.insert(
"authorization",
format!("Bearer {}", resp.token)
.parse()
.expect("bearer header"),
format!("Bearer {token}").parse().expect("bearer header"),
);
headers
}
#[tokio::test]
async fn issue_returns_generation_bound_token_and_rejects_it_after_runtime_replacement() {
use crate::auth::authenticator::Authenticator;
use crate::auth::authenticator::{AuthFailureReason, AuthMechanism};
let state = state_with_issuer().await;
let (authenticator, token) = issue_session_bearer(&state).await;
let headers = bearer_headers(&token);
let provider_path = "/openshell.v1.OpenShell/GetSandboxProviderEnvironment";
let principal = authenticator
.authenticate(&headers, provider_path)
.await
.expect("active runtime token must authenticate")
.principal()
.expect("session authenticator must recognize its token");
assert!(matches!(principal, Principal::Sandbox(_)));
@@ -887,7 +1048,52 @@ mod tests {
.authenticate(&headers, provider_path)
.await
.expect_err("obsolete runtime token must not reach provider access");
assert_eq!(error.code(), tonic::Code::Unauthenticated);
assert_eq!(error.status.code(), tonic::Code::Unauthenticated);
assert_eq!(error.mechanism, AuthMechanism::SandboxJwt);
assert_eq!(error.reason, AuthFailureReason::StaleIdentity);
}
#[tokio::test]
async fn session_bearer_for_deleted_sandbox_is_rejected_as_unknown_identity() {
use crate::auth::authenticator::AuthFailureReason;
use crate::auth::authenticator::Authenticator;
let state = state_with_issuer().await;
let (authenticator, token) = issue_session_bearer(&state).await;
assert!(state.store.delete("sandbox", "sandbox-a").await.unwrap());
let error = authenticator
.authenticate(
&bearer_headers(&token),
"/openshell.v1.OpenShell/GetSandboxProviderEnvironment",
)
.await
.expect_err("a bearer for a deleted sandbox must be rejected");
assert_eq!(error.status.code(), tonic::Code::Unauthenticated);
assert_eq!(error.reason, AuthFailureReason::UnknownIdentity);
}
#[tokio::test]
async fn tampered_session_bearer_is_rejected_as_invalid() {
use crate::auth::authenticator::Authenticator;
use crate::auth::authenticator::{AuthFailureReason, AuthMechanism};
let state = state_with_issuer().await;
let (authenticator, token) = issue_session_bearer(&state).await;
let (signed, signature) = token.rsplit_once('.').expect("JWT segments");
let flipped = if signature.starts_with('A') { 'B' } else { 'A' };
let tampered = format!("{signed}.{flipped}{}", &signature[1..]);
let error = authenticator
.authenticate(
&bearer_headers(&tampered),
"/openshell.v1.OpenShell/GetSandboxProviderEnvironment",
)
.await
.expect_err("a tampered bearer must be rejected");
assert_eq!(error.status.code(), tonic::Code::Unauthenticated);
assert_eq!(error.mechanism, AuthMechanism::SandboxJwt);
assert_eq!(error.reason, AuthFailureReason::InvalidCredential);
}
#[tokio::test]
crates/openshell-server/src/multiplex.rs
+163 -33
View File
@@ -43,8 +43,8 @@ use tracing_opentelemetry::OpenTelemetrySpanExt as _;
use crate::{
OpenShellService, ServerState,
auth::audit::{self, AuthFailureReason, AuthMechanism},
auth::authenticator::AuthenticatorChain,
auth::audit,
auth::authenticator::{AuthFailureReason, AuthMechanism, Authentication, AuthenticatorChain},
auth::authz::AuthzPolicy,
auth::identity::Identity,
auth::oidc::{self, OidcAuthenticator},
@@ -1137,22 +1137,31 @@ where
Principal::Operator(operator)
} else if let Some(chain) = chain {
match chain.authenticate(req.headers(), &path).await {
Ok(Some(p)) => p,
Ok(None) => match (mtls_auth_enabled, peer_identity) {
Ok(Authentication::Accepted(p)) => p,
Ok(outcome) => match (mtls_auth_enabled, peer_identity) {
(true, Some(identity)) => Principal::User(UserPrincipal { identity }),
_ if allow_unauthenticated_users => unauthenticated_dev_user_principal(),
_ => {
audit::emit_authentication_failure(
&path,
None,
AuthFailureReason::MissingCredential,
);
let (mechanism, reason) = match outcome {
Authentication::Declined(decline) => {
(Some(decline.mechanism), decline.reason)
}
_ => (None, AuthFailureReason::MissingCredential),
};
audit::emit_authentication_failure(&path, mechanism, reason);
return Ok(status_response(tonic::Status::unauthenticated(
"missing authorization header",
)));
}
},
Err(status) => return Ok(status_response(status)),
Err(rejection) => {
audit::emit_authentication_failure(
&path,
Some(rejection.mechanism),
rejection.reason,
);
return Ok(status_response(rejection.status));
}
}
} else if mtls_auth_enabled {
let Some(identity) = peer_identity else {
@@ -2784,7 +2793,11 @@ mod tests {
Ok::<_, Infallible>(tonic::Status::unimplemented("test fallback").into_http())
});
let rejecting_oidc = Arc::new(MockAuthenticator::returning(Err(
tonic::Status::unauthenticated("OIDC credentials required"),
crate::auth::authenticator::AuthRejection::new(
AuthMechanism::Oidc,
AuthFailureReason::InvalidCredential,
tonic::Status::unauthenticated("OIDC credentials required"),
),
)));
let grpc = AuthGrpcRouter::with_peer_identity(
GrpcRouter::new(unrouted, reflection),
@@ -3217,6 +3230,7 @@ mod tests {
mod auth_router {
use super::*;
use crate::auth::authenticator::AuthRejection;
use crate::auth::authenticator::test_support::MockAuthenticator;
use crate::auth::identity::{Identity, IdentityProvider};
use crate::auth::principal::{
@@ -3394,7 +3408,7 @@ mod tests {
}),
] {
let chain = AuthenticatorChain::new(vec![Arc::new(MockAuthenticator::returning(
Ok(Some(principal)),
Ok(Authentication::Accepted(principal)),
))]);
let (recorder, _) = PrincipalRecorder::new();
let mut router = AuthGrpcRouter::new(recorder, Some(chain), None);
@@ -3449,7 +3463,9 @@ mod tests {
#[tokio::test]
async fn mtls_peer_identity_fills_missing_principal_when_enabled() {
let mock = Arc::new(MockAuthenticator::returning(Ok(None)));
let mock = Arc::new(MockAuthenticator::returning(Ok(
Authentication::NotApplicable,
)));
let chain = AuthenticatorChain::new(vec![mock]);
let (recorder, seen) = PrincipalRecorder::new();
let mut router = AuthGrpcRouter::with_peer_identity(
@@ -3518,9 +3534,11 @@ mod tests {
#[tokio::test]
async fn reflection_bypasses_oidc_and_mtls_user_authentication() {
let oidc = Arc::new(MockAuthenticator::returning(Err(
let oidc = Arc::new(MockAuthenticator::returning(Err(AuthRejection::new(
AuthMechanism::Oidc,
AuthFailureReason::InvalidCredential,
tonic::Status::unauthenticated("OIDC credentials required"),
)));
))));
let chain = AuthenticatorChain::new(vec![oidc]);
let (recorder, seen) = PrincipalRecorder::new();
let mut router =
@@ -3543,7 +3561,9 @@ mod tests {
#[tokio::test]
async fn unauthenticated_dev_user_fills_missing_principal_when_enabled() {
let mock = Arc::new(MockAuthenticator::returning(Ok(None)));
let mock = Arc::new(MockAuthenticator::returning(Ok(
Authentication::NotApplicable,
)));
let chain = AuthenticatorChain::new(vec![mock]);
let (recorder, seen) = PrincipalRecorder::new();
let mut router =
@@ -3586,9 +3606,9 @@ mod tests {
#[tokio::test]
async fn user_principal_lands_in_request_extensions() {
let mock = Arc::new(MockAuthenticator::returning(Ok(Some(user_principal(
"alice",
)))));
let mock = Arc::new(MockAuthenticator::returning(Ok(Authentication::Accepted(
user_principal("alice"),
))));
let chain = AuthenticatorChain::new(vec![mock]);
let (recorder, seen) = PrincipalRecorder::new();
let mut router = AuthGrpcRouter::new(recorder, Some(chain), None);
@@ -3605,7 +3625,9 @@ mod tests {
#[tokio::test]
async fn sandbox_principal_lands_in_request_extensions() {
let mock = Arc::new(MockAuthenticator::returning(Ok(Some(sandbox_principal()))));
let mock = Arc::new(MockAuthenticator::returning(Ok(Authentication::Accepted(
sandbox_principal(),
))));
let chain = AuthenticatorChain::new(vec![mock]);
let (recorder, seen) = PrincipalRecorder::new();
let mut router = AuthGrpcRouter::new(recorder, Some(chain), None);
@@ -3622,7 +3644,9 @@ mod tests {
#[tokio::test]
async fn sandbox_principal_can_call_allowlisted_method() {
let mock = Arc::new(MockAuthenticator::returning(Ok(Some(sandbox_principal()))));
let mock = Arc::new(MockAuthenticator::returning(Ok(Authentication::Accepted(
sandbox_principal(),
))));
let chain = AuthenticatorChain::new(vec![mock]);
let (recorder, seen) = PrincipalRecorder::new();
let mut router = AuthGrpcRouter::new(recorder, Some(chain), None);
@@ -3673,7 +3697,9 @@ mod tests {
"/openshell.v1.OpenShell/IssueSandboxToken",
"/openshell.v1.OpenShell/RefreshSandboxToken",
] {
let mock = Arc::new(MockAuthenticator::returning(Ok(Some(admin_user()))));
let mock = Arc::new(MockAuthenticator::returning(Ok(Authentication::Accepted(
admin_user(),
))));
let chain = AuthenticatorChain::new(vec![mock]);
let (recorder, seen) = PrincipalRecorder::new();
let mut router = AuthGrpcRouter::new(recorder, Some(chain), Some(policy.clone()));
@@ -3694,7 +3720,7 @@ mod tests {
scopes_enabled: true,
};
let mock = Arc::new(MockAuthenticator::returning(Ok(Some(
let mock = Arc::new(MockAuthenticator::returning(Ok(Authentication::Accepted(
provider_writer_principal(),
))));
let chain = AuthenticatorChain::new(vec![mock]);
@@ -3707,7 +3733,7 @@ mod tests {
assert_eq!(response.status(), StatusCode::OK);
assert!(seen.lock().unwrap().is_some());
let mock = Arc::new(MockAuthenticator::returning(Ok(Some(
let mock = Arc::new(MockAuthenticator::returning(Ok(Authentication::Accepted(
provider_writer_principal(),
))));
let chain = AuthenticatorChain::new(vec![mock]);
@@ -3729,7 +3755,9 @@ mod tests {
"/openshell.v1.OpenShell/CreateProvider",
"/openshell.v1.OpenShell/ApproveDraftChunk",
] {
let mock = Arc::new(MockAuthenticator::returning(Ok(Some(sandbox_principal()))));
let mock = Arc::new(MockAuthenticator::returning(Ok(Authentication::Accepted(
sandbox_principal(),
))));
let chain = AuthenticatorChain::new(vec![mock]);
let (recorder, seen) = PrincipalRecorder::new();
let mut router = AuthGrpcRouter::new(recorder, Some(chain), None);
@@ -3743,7 +3771,9 @@ mod tests {
#[tokio::test]
async fn missing_principal_returns_unauthenticated() {
let mock = Arc::new(MockAuthenticator::returning(Ok(None)));
let mock = Arc::new(MockAuthenticator::returning(Ok(
Authentication::NotApplicable,
)));
let chain = AuthenticatorChain::new(vec![mock]);
let (recorder, seen) = PrincipalRecorder::new();
let mut router = AuthGrpcRouter::new(recorder, Some(chain), None);
@@ -3804,9 +3834,11 @@ mod tests {
#[tokio::test]
async fn authenticator_error_short_circuits() {
let mock = Arc::new(MockAuthenticator::returning(Err(
let mock = Arc::new(MockAuthenticator::returning(Err(AuthRejection::new(
AuthMechanism::Oidc,
AuthFailureReason::InvalidCredential,
tonic::Status::unauthenticated("forged"),
)));
))));
let chain = AuthenticatorChain::new(vec![mock]);
let (recorder, seen) = PrincipalRecorder::new();
let mut router = AuthGrpcRouter::new(recorder, Some(chain), None);
@@ -3852,8 +3884,9 @@ mod tests {
async fn missing_credential_emits_authentication_failure() {
let capture = OcsfCapture::default();
let _guard = capture.install();
let chain =
AuthenticatorChain::new(vec![Arc::new(MockAuthenticator::returning(Ok(None)))]);
let chain = AuthenticatorChain::new(vec![Arc::new(MockAuthenticator::returning(Ok(
Authentication::NotApplicable,
)))]);
let (recorder, _) = PrincipalRecorder::new();
let mut router = AuthGrpcRouter::new(recorder, Some(chain), None);
let path = "/openshell.v1.OpenShell/ListSandboxes";
@@ -3913,12 +3946,107 @@ mod tests {
assert!(!events[0].to_string().contains("canary-secret-token"));
}
async fn call_with_rejection(
rejection: AuthRejection,
path: &str,
) -> (tonic::Status, Vec<serde_json::Value>) {
let capture = OcsfCapture::default();
let _guard = capture.install();
let chain = AuthenticatorChain::new(vec![Arc::new(MockAuthenticator::returning(Err(
rejection,
)))]);
let (recorder, seen) = PrincipalRecorder::new();
let mut router = AuthGrpcRouter::new(recorder, Some(chain), None);
let request = Request::builder()
.uri(path)
.header("authorization", "Bearer canary-secret-token-9f2c")
.body(Full::new(Bytes::new()))
.unwrap();
let res = router.call(request).await.unwrap();
assert!(seen.lock().unwrap().is_none());
let status = tonic::Status::from_header_map(res.headers()).expect("grpc status");
(status, captured_json(&capture))
}
#[tokio::test]
async fn declined_credential_emits_its_classification_with_the_missing_credential_status() {
let capture = OcsfCapture::default();
let _guard = capture.install();
let chain = AuthenticatorChain::new(vec![Arc::new(MockAuthenticator::returning(Ok(
Authentication::Declined(crate::auth::authenticator::AuthDecline {
mechanism: AuthMechanism::GatewayPeer,
reason: AuthFailureReason::InvalidCredential,
}),
)))]);
let (recorder, seen) = PrincipalRecorder::new();
let mut router = AuthGrpcRouter::new(recorder, Some(chain), None);
let path = "/openshell.v1.OpenShell/PeerRelay";
let res = router.call(empty_request(path)).await.unwrap();
assert!(seen.lock().unwrap().is_none());
let status = tonic::Status::from_header_map(res.headers()).expect("grpc status");
assert_eq!(status.code(), tonic::Code::Unauthenticated);
assert_eq!(status.message(), "missing authorization header");
let events = captured_json(&capture);
assert_eq!(events.len(), 1, "{events:?}");
assert_authentication_failure(&events[0], path, "invalid_credential", 3);
assert_eq!(events[0]["auth_protocol"], "k8s_service_account");
}
#[tokio::test]
async fn rejected_oidc_credential_emits_classified_authentication_failure() {
let path = "/openshell.v1.OpenShell/ListSandboxes";
let (status, events) = call_with_rejection(
AuthRejection::new(
AuthMechanism::Oidc,
AuthFailureReason::ExpiredCredential,
tonic::Status::unauthenticated("invalid token: ExpiredSignature"),
),
path,
)
.await;
assert_eq!(status.code(), tonic::Code::Unauthenticated);
assert_eq!(status.message(), "invalid token: ExpiredSignature");
assert_eq!(events.len(), 1, "{events:?}");
assert_authentication_failure(&events[0], path, "expired_credential", 2);
assert_eq!(events[0]["auth_protocol_id"], 4);
assert_eq!(events[0]["auth_protocol"], "OpenID");
assert!(!events[0].to_string().contains("canary-secret-token"));
assert!(!events[0].to_string().contains("ExpiredSignature"));
}
#[tokio::test]
async fn rejected_sandbox_credential_emits_classified_authentication_failure() {
let path = "/openshell.v1.OpenShell/GetSandboxProviderEnvironment";
let (status, events) = call_with_rejection(
AuthRejection::new(
AuthMechanism::SandboxJwt,
AuthFailureReason::StaleIdentity,
tonic::Status::unauthenticated(
"gateway token does not match the active sandbox identity",
),
),
path,
)
.await;
assert_eq!(status.code(), tonic::Code::Unauthenticated);
assert_eq!(events.len(), 1, "{events:?}");
assert_authentication_failure(&events[0], path, "stale_identity", 3);
assert_eq!(events[0]["auth_protocol_id"], 99);
assert_eq!(events[0]["auth_protocol"], "sandbox_jwt");
}
#[tokio::test]
async fn authenticated_and_exempt_requests_emit_no_authentication_event() {
let capture = OcsfCapture::default();
let _guard = capture.install();
let chain = AuthenticatorChain::new(vec![Arc::new(MockAuthenticator::returning(Ok(
Some(user_principal("alice")),
Authentication::Accepted(user_principal("alice")),
)))]);
let (recorder, _) = PrincipalRecorder::new();
let mut router = AuthGrpcRouter::new(recorder, Some(chain), None);
@@ -3939,9 +4067,11 @@ mod tests {
async fn health_methods_bypass_chain() {
// Authenticator is wired to fail-closed; the request still gets
// through because the path is exempt.
let mock = Arc::new(MockAuthenticator::returning(Err(
let mock = Arc::new(MockAuthenticator::returning(Err(AuthRejection::new(
AuthMechanism::Oidc,
AuthFailureReason::InvalidCredential,
tonic::Status::unauthenticated("would reject"),
)));
))));
let chain = AuthenticatorChain::new(vec![mock.clone()]);
let (recorder, _) = PrincipalRecorder::new();
let mut router = AuthGrpcRouter::new(recorder, Some(chain), None);
docs/observability/ocsf-json-export.mdx
+15 -1
View File
@@ -21,7 +21,21 @@ This output is separate from the supervisor stderr and file output described bel
Gateway records use the configured gateway name for both `device.uid` and `device.name`. Replicas of one installation share that identity; `device.hostname` identifies the emitting replica. Assign distinct gateway names to installations whose events are collected together. Reusing a name, including the default name, groups those installations under the same identity. Renaming a gateway changes its audit identity.
The gateway emits an Authentication (`class_uid: 3002`) event when it rejects a gRPC request that presents no credential, or an operator certificate together with an `authorization` header. `status_detail` holds the reason category: `missing_credential` or `conflicting_credentials`. These events never include presented credentials or headers.
The gateway emits an Authentication (`class_uid: 3002`) event when it rejects a gRPC request during authentication. The caller receives the same error either way. `auth_protocol` names the mechanism involved, when one applies: `OpenID` for OIDC bearer tokens, or `mtls`, `sandbox_jwt`, `compute_driver`, or `k8s_service_account`. `status_detail` holds the reason category:
| Reason | Severity | Meaning |
|---|---|---|
| `missing_credential` | Low | No credential was presented. |
| `expired_credential` | Low | The credential has expired. |
| `conflicting_credentials` | Medium | An operator certificate arrived together with an `authorization` header. |
| `malformed_credential` | Medium | The credential could not be parsed. |
| `invalid_credential` | Medium | The credential failed signature, key, or claim verification. |
| `audience_or_issuer_mismatch` | Medium | The credential was issued by or for a different party. |
| `unknown_identity` | Medium | The credential names an identity the gateway does not recognize. |
| `stale_identity` | Medium | The credential names a sandbox runtime or gateway pod that has since been replaced or stopped. |
| `identity_provider_unavailable` | Medium | The OIDC key endpoint, Kubernetes API, compute driver, or gateway database could not verify the credential. |
These events never include presented credentials, headers, or verification error text.
Without JSONL enabled, gateway OCSF events still appear as shorthand in console output, subject to the diagnostic log filter. Events associated with a sandbox also appear in its gateway log stream; gateway-wide events have no sandbox association.