mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-02 07:34:45 +08:00
refactor(network): consolidate proxy egress pipeline (#2373)
* refactor(network): introduce shared egress pipeline Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): cover shared proxy egress paths Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * refactor(network): make destination authorization explicit Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * refactor(network): pin proxy relay policy context Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): lock relay generation contracts Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): establish phase zero compatibility baseline Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * feat(policy): detect ambiguous network endpoints Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * feat(sandbox): fail closed on invalid policy updates Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * refactor(network): invalidate relays on policy changes Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * docs(policy): document validation failure posture Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): cover validation and middleware egress Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(config): move policy failure mode to gateway toml Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): name proxy contracts by behavior Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(network): align overlap validation with endpoint selection Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): expect hard loopback denial Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): match declared endpoint denial Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(policy): preserve path-specific endpoint overrides Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): respect hard-blocked host gateways Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(network): reconcile proxy refactor with main Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(network): preserve CONNECT policy generation Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(policy): cover runtime endpoint glob semantics Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(server): reject ambiguous policies before persistence Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * docs(policy): explain ambiguity preflight behavior Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(policy): compare body limits within protocol Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(proxy): avoid global tracing capture race Signed-off-by: John Myers <johntmyers@users.noreply.github.com> * chore(server): format rebased provider tests Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(server): authenticate rebased policy requests Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(sandbox): retain runtime on middleware outage Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(server): preflight provider composition activation Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(sandbox): distinguish runtime failure transitions Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Signed-off-by: John Myers <johntmyers@users.noreply.github.com> Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
This commit is contained in:
John T. Myers
co-authored by
John Myers
John Myers
parent
770d4e6b99
commit
905b554c7c
@@ -106,6 +106,30 @@ The middleware service must start before the gateway and be reachable from both
|
||||
|
||||
At request time, distinguish an explicit `middleware_denied` result from `middleware_failed`. A denial is always enforced. A failure follows the policy-local `on_error`: `fail_closed` blocks the request, while `fail_open` bypasses only that stage and emits a detection finding. If a running supervisor cannot install a new registry, it preserves its last-known-good generation and emits a configuration failure event.
|
||||
|
||||
For network policy validation failures, first distinguish a gateway mutation
|
||||
rejection from a supervisor runtime rejection. Direct policy updates,
|
||||
incremental merges and approvals, provider attachments, and provider-profile
|
||||
fanout are validated against the complete effective policy before persistence
|
||||
when the gateway knows the affected sandbox scope. A `FAILED_PRECONDITION`
|
||||
ambiguity response means no invalid revision or partial fanout was stored.
|
||||
Supervisor validation remains defense in depth for startup, races, and policy
|
||||
sources outside those mutation paths.
|
||||
|
||||
Runtime rejection behavior is configured only in `gateway.toml`:
|
||||
|
||||
```toml
|
||||
[openshell.gateway]
|
||||
policy_validation_failure_mode = "fail_closed"
|
||||
```
|
||||
|
||||
The default `fail_closed` mode deactivates the previous generation, closes
|
||||
pinned relays, and quarantines new egress until a valid generation loads.
|
||||
`retain_last_valid` explicitly keeps the previous valid policy active; without
|
||||
one it still fails closed. Restart the gateway after changing this field.
|
||||
Inspect sandbox OCSF configuration and finding events for the validation
|
||||
rationale, configured and effective modes, active generation, and the explicit
|
||||
`previous_policy_active` state.
|
||||
|
||||
### Step 4: Check Docker-Backed Gateways
|
||||
|
||||
```bash
|
||||
@@ -417,6 +441,8 @@ openshell logs <sandbox-name>
|
||||
| Provider profiles disappear after enabling an interceptor catalog | `provider_profile_sources` selected only an authoritative interceptor or returned invalid/duplicate IDs | Inspect source list and interceptor `Describe`/catalog logs; include `builtin` and `user` when intended |
|
||||
| Gateway fails after registering supervisor middleware | Service unavailable, invalid manifest, duplicate binding, reserved name, or invalid body/timeout limit | Middleware service and gateway logs; `[[openshell.supervisor.middleware]]`; `Describe` response |
|
||||
| Policy update rejects `network_middlewares` | Unknown middleware name, implementation-owned config invalid, duplicate order, broad/invalid host selector, or fail-closed coverage of `tls: skip` | Policy error, gateway logs, middleware `ValidateConfig`, selector and order fields |
|
||||
| Policy mutation returns `FAILED_PRECONDITION` for endpoint ambiguity | Equally specific effective endpoint selectors disagree on connection or request-processing metadata | CLI error, base and provider-composed policy, affected profile attachments; confirm no new revision was stored |
|
||||
| Supervisor enters policy quarantine | A runtime candidate failed validation while `policy_validation_failure_mode = "fail_closed"` | Sandbox OCSF config/finding events, validation rationale, active generation, `previous_policy_active` |
|
||||
| HTTP request returns `middleware_failed` or `middleware_denied` | Selected stage failed or explicitly denied the admitted request | Sandbox OCSF logs; policy-local middleware config; service availability; `on_error` |
|
||||
| Custom compute driver is unavailable | Driver process/socket missing, inaccessible, or configured with a reserved/mismatched name | Socket ownership/mode, driver service logs, gateway `GetCapabilities` logs |
|
||||
| Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials |
|
||||
|
||||
@@ -237,7 +237,10 @@ Only needed for the **Moderate** and **Full** tiers. Translate API path paramete
|
||||
| `/api/v1/models/{model_id}/versions/{version}` | `/api/v1/models/*/versions/*` |
|
||||
| All sub-paths under `/api/v1/` | `/api/v1/**` |
|
||||
|
||||
Remember: `*` does not cross `/` boundaries. Use `**` for recursive matching across path segments.
|
||||
Path matching uses the runtime `glob` engine. Both `*` and `**` may cross `/`
|
||||
boundaries; `?` matches one character, and bracket classes such as `[0-9]` and
|
||||
`[!0]` are supported. Prefer segment-shaped patterns such as
|
||||
`/repos/*/issues` for readability, but do not rely on `*` to stop at `/`.
|
||||
|
||||
### Building the Explicit Rules List
|
||||
|
||||
@@ -439,7 +442,7 @@ The policy needs to go somewhere. Determine which mode applies:
|
||||
|
||||
2. **Check for conflicts**:
|
||||
- Does a policy with the same key already exist? If so, ask the user whether to **replace** it, **merge** new endpoints/binaries into it, or use a different key.
|
||||
- Does an existing policy already cover the same host:port? Warn the user — overlapping endpoint coverage across policies causes OPA evaluation errors (complete rule conflict).
|
||||
- Does an existing endpoint selector overlap the new selector? Compatible overlaps are allowed and can intentionally aggregate allow and deny rules. Reject or revise equally specific overlaps that disagree on connection or request-processing metadata, including TLS, destination constraints, protocol/parser behavior, enforcement, or credential handling. A more-specific path selector may override broader request-processing metadata.
|
||||
|
||||
3. **Apply the change**:
|
||||
- **Adding a new policy**: Insert the new policy block under `network_policies`, maintaining the file's existing indentation and style.
|
||||
|
||||
@@ -727,7 +727,9 @@ An exact IP is treated as `/32` — only that specific address is permitted.
|
||||
**Agent workflow**:
|
||||
|
||||
1. Read `sandbox-policy.yaml`
|
||||
2. Check that no existing policy already covers `api.github.com:443` — if one does, warn about overlap
|
||||
2. Check existing selectors for `api.github.com:443`. Compatible overlaps may
|
||||
aggregate request rules; revise equally specific overlaps that disagree on
|
||||
TLS, destination, protocol/parser, enforcement, or credential behavior.
|
||||
3. Check that the key `github_readonly` doesn't already exist
|
||||
4. Insert the new policy under `network_policies`:
|
||||
|
||||
|
||||
@@ -354,6 +354,13 @@ Edit `current-policy.yaml` to allow the blocked actions. **For policy content au
|
||||
openshell policy set dev --policy current-policy.yaml --wait
|
||||
```
|
||||
|
||||
The gateway validates the complete effective candidate—including attached
|
||||
provider-profile policy—before it stores a direct update, incremental merge,
|
||||
approved proposal, provider attachment, or profile update that affects attached
|
||||
sandboxes. An ambiguity failure returns `FAILED_PRECONDITION`; the rejected
|
||||
candidate does not create a policy revision or partially update affected
|
||||
sandboxes. Fix the conflicting endpoint selectors and submit again.
|
||||
|
||||
The `--wait` flag blocks until the sandbox confirms the policy is loaded (polls every second). Exit codes:
|
||||
- **0**: Policy loaded successfully
|
||||
- **1**: Policy load failed
|
||||
@@ -578,6 +585,13 @@ openshell settings set --global --key providers_v2_enabled --value true
|
||||
|
||||
Global mutations prompt for confirmation. Use `--yes` only in reviewed automation.
|
||||
|
||||
`policy_validation_failure_mode` is gateway startup configuration, not a
|
||||
mutable `openshell settings` key. Set it under `[openshell.gateway]` in
|
||||
`gateway.toml` and restart the gateway. The security-first default is
|
||||
`fail_closed`; `retain_last_valid` is an explicit availability tradeoff. OCSF
|
||||
configuration events state whether the previous generation is active after a
|
||||
runtime validation failure.
|
||||
|
||||
## Workflow 10: Service Access
|
||||
|
||||
Use `forward` for local access and `service` for a gateway-managed HTTP endpoint:
|
||||
|
||||
@@ -57,6 +57,21 @@ unsafe internal destinations, and evaluates the active policy. On Linux, it
|
||||
maps an accepted proxy connection back to the workload socket by matching the
|
||||
complete local-to-remote TCP tuple before resolving every process that owns the
|
||||
socket inode.
|
||||
|
||||
CONNECT and absolute-form forward HTTP are explicit-proxy adapters over the same
|
||||
egress pipeline. Each adapter normalizes its request into an egress intent, and
|
||||
the shared authorization result carries the process evidence used by destination
|
||||
validation and relay selection. During the compatibility migration, endpoint
|
||||
state is hydrated at the adapters' existing policy query points; it is not yet
|
||||
one atomic, generation-consistent authorization result. Destination validation
|
||||
returns an unopened connector so adapters retain their existing response and
|
||||
upstream-dial timing. CONNECT prepares a generation-pinned relay context before
|
||||
entering shared TLS-terminated or plaintext HTTP relays; non-HTTP traffic uses
|
||||
the shared raw byte relay after the existing adapter gates. Forward HTTP retains
|
||||
its guarded single-request relay while sharing authorization, request context,
|
||||
policy-pinning, and destination boundaries.
|
||||
Adapter-specific response and OCSF event shapes remain at the protocol boundary.
|
||||
|
||||
For inspected HTTP traffic, the proxy can enforce REST method/path rules,
|
||||
WebSocket upgrade and text-message rules, GraphQL operation rules, and
|
||||
MCP method, tool, and supported params rules or generic JSON-RPC method rules
|
||||
|
||||
@@ -82,9 +82,9 @@ metadata before forwarding. The proxy also supports credential injection on
|
||||
terminated HTTP streams when policy allows the endpoint.
|
||||
|
||||
Raw streams and long-lived response bodies are connection scoped. Policy
|
||||
reloads affect the next connection or the next parsed HTTP request; they do not
|
||||
rewrite bytes already being relayed. HTTP upgrades switch to raw relay by
|
||||
default. A `protocol: rest` endpoint can opt in to
|
||||
generation changes close relays pinned to the previous generation instead of
|
||||
allowing them to continue under stale authorization. HTTP upgrades switch to
|
||||
raw relay by default. A `protocol: rest` endpoint can opt in to
|
||||
`websocket_credential_rewrite` for client-to-server WebSocket text messages
|
||||
after an allowed `101` upgrade; server-to-client traffic and all other upgraded
|
||||
protocols remain raw passthrough.
|
||||
@@ -98,10 +98,37 @@ supervisor polls for config revisions and attempts to load new dynamic policy
|
||||
into the in-process OPA engine; CLI reads of the latest sandbox policy use the
|
||||
same effective configuration path.
|
||||
|
||||
If a new policy fails validation or loading, the supervisor reports the failure
|
||||
and keeps the last-known-good policy. Static controls, such as filesystem
|
||||
allowlists and process identity, require a new sandbox because they are applied
|
||||
before the child process starts.
|
||||
The supervisor validates complete effective policy generations before
|
||||
activation. Overlapping endpoint selectors may contribute request allow and
|
||||
deny rules only when their connection and request-processing metadata agree;
|
||||
conflicting TLS, destination, credential, parser, or enforcement metadata
|
||||
rejects the complete generation. Plain L4 endpoints do not contribute
|
||||
request-processing metadata, so they may overlap an L7 endpoint when their
|
||||
connection metadata agrees. When request paths overlap, a path endpoint with a
|
||||
higher specificity rank deterministically overrides broader request-processing
|
||||
metadata. Equally specific overlapping endpoints must agree.
|
||||
|
||||
Gateway mutation paths validate the complete effective candidate before
|
||||
persistence when the affected sandbox scope is known. Direct replacements,
|
||||
incremental merges and approvals, provider attachment, and profile fanout reject
|
||||
ambiguity atomically, without creating an invalid revision or partially
|
||||
activating an update. Supervisor validation remains the defense-in-depth
|
||||
boundary for startup, concurrent changes, and sources outside those mutations.
|
||||
|
||||
The `[openshell.gateway] policy_validation_failure_mode` configuration controls
|
||||
candidates rejected by supervisor runtime validation. Gateway preflight
|
||||
rejections never become generations and leave the active policy unchanged. The
|
||||
runtime mode defaults to `fail_closed`, which publishes a quarantine generation,
|
||||
denies new egress, invalidates existing relays, and leaves the previous policy
|
||||
inactive. Operators may explicitly select
|
||||
`retain_last_valid`, which keeps the previous generation active. With no
|
||||
previous valid generation, the effective mode remains `fail_closed` regardless
|
||||
of the configured mode. The gateway distributes this startup configuration to
|
||||
sandbox supervisors with each effective policy snapshot. OCSF configuration and finding events state the
|
||||
candidate version, validation rationale, configured and effective modes, active
|
||||
generation, and whether the previous policy is active. Static controls,
|
||||
such as filesystem allowlists and process identity, require a new sandbox
|
||||
because they are applied before the child process starts.
|
||||
|
||||
Gateway-global policy can override sandbox-scoped policy. Use it sparingly
|
||||
because it changes the effective access model for every sandbox on the gateway.
|
||||
|
||||
@@ -36,6 +36,43 @@ pub const DEFAULT_DOCKER_NETWORK_NAME: &str = "openshell-docker";
|
||||
/// Default domain used for browser-facing sandbox service URLs.
|
||||
pub const DEFAULT_SERVICE_ROUTING_DOMAIN: &str = "openshell.localhost";
|
||||
|
||||
/// Gateway posture when a sandbox rejects a candidate policy generation.
|
||||
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum PolicyValidationFailureMode {
|
||||
/// Deactivate the previous policy and deny new egress until a valid
|
||||
/// generation is loaded.
|
||||
#[default]
|
||||
FailClosed,
|
||||
/// Keep the last valid generation active when a newer candidate fails
|
||||
/// validation. Startup still fails closed when no valid generation exists.
|
||||
RetainLastValid,
|
||||
}
|
||||
|
||||
impl PolicyValidationFailureMode {
|
||||
#[must_use]
|
||||
pub const fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Self::FailClosed => "fail_closed",
|
||||
Self::RetainLastValid => "retain_last_valid",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl FromStr for PolicyValidationFailureMode {
|
||||
type Err = String;
|
||||
|
||||
fn from_str(value: &str) -> Result<Self, Self::Err> {
|
||||
match value {
|
||||
"fail_closed" => Ok(Self::FailClosed),
|
||||
"retain_last_valid" => Ok(Self::RetainLastValid),
|
||||
_ => Err(format!(
|
||||
"invalid policy validation failure mode '{value}'; expected fail_closed or retain_last_valid"
|
||||
)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Default OCI repository for the supervisor image (no tag).
|
||||
pub const DEFAULT_SUPERVISOR_IMAGE_REPO: &str = "ghcr.io/nvidia/openshell/supervisor";
|
||||
|
||||
@@ -396,6 +433,9 @@ pub struct Config {
|
||||
/// Log level (trace, debug, info, warn, error).
|
||||
pub log_level: String,
|
||||
|
||||
/// Security posture for rejected sandbox policy generations.
|
||||
pub policy_validation_failure_mode: PolicyValidationFailureMode,
|
||||
|
||||
/// TLS configuration. When `None`, the server listens on plaintext HTTP.
|
||||
pub tls: Option<TlsConfig>,
|
||||
|
||||
@@ -737,6 +777,7 @@ impl Config {
|
||||
health_bind_address: None,
|
||||
metrics_bind_address: None,
|
||||
log_level: default_log_level(),
|
||||
policy_validation_failure_mode: PolicyValidationFailureMode::default(),
|
||||
tls,
|
||||
oidc: None,
|
||||
auth: GatewayAuthConfig::default(),
|
||||
@@ -981,10 +1022,10 @@ mod tests {
|
||||
use super::{
|
||||
ComputeDriverKind, Config, DEFAULT_SERVICE_ROUTING_DOMAIN, GatewayInterceptorBindingPolicy,
|
||||
GatewayInterceptorConfig, GatewayInterceptorFailurePolicy, GatewayJwtConfig,
|
||||
GatewayProviderProfileSourceConfig, detect_docker_socket_from_candidates, detect_driver,
|
||||
detect_podman_socket_from_candidates, docker_host_unix_socket_path, docker_socket_responds,
|
||||
is_unix_socket, normalize_compute_driver_name, podman_socket_candidates_from_env,
|
||||
podman_socket_responds,
|
||||
GatewayProviderProfileSourceConfig, PolicyValidationFailureMode,
|
||||
detect_docker_socket_from_candidates, detect_driver, detect_podman_socket_from_candidates,
|
||||
docker_host_unix_socket_path, docker_socket_responds, is_unix_socket,
|
||||
normalize_compute_driver_name, podman_socket_candidates_from_env, podman_socket_responds,
|
||||
};
|
||||
#[cfg(unix)]
|
||||
use std::io::{Read as _, Write as _};
|
||||
@@ -1020,6 +1061,21 @@ mod tests {
|
||||
assert!(err.contains("unsupported compute driver 'firecracker'"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn policy_validation_failure_mode_is_secure_by_default() {
|
||||
assert_eq!(
|
||||
Config::new(None).policy_validation_failure_mode,
|
||||
PolicyValidationFailureMode::FailClosed
|
||||
);
|
||||
assert_eq!(
|
||||
"retain_last_valid"
|
||||
.parse::<PolicyValidationFailureMode>()
|
||||
.unwrap(),
|
||||
PolicyValidationFailureMode::RetainLastValid
|
||||
);
|
||||
assert!("keep_old".parse::<PolicyValidationFailureMode>().is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compute_driver_name_normalization_accepts_builtin_and_custom_names() {
|
||||
assert_eq!(normalize_compute_driver_name(" VM ").unwrap(), "vm");
|
||||
|
||||
@@ -780,6 +780,8 @@ pub struct SettingsPollResult {
|
||||
pub supervisor_middleware_services: Vec<crate::proto::SupervisorMiddlewareService>,
|
||||
/// Workspace the sandbox belongs to.
|
||||
pub workspace: String,
|
||||
/// Gateway-configured posture for rejected policy generations.
|
||||
pub policy_validation_failure_mode: crate::PolicyValidationFailureMode,
|
||||
}
|
||||
|
||||
fn settings_poll_result(inner: crate::proto::GetSandboxConfigResponse) -> SettingsPollResult {
|
||||
@@ -795,6 +797,41 @@ fn settings_poll_result(inner: crate::proto::GetSandboxConfigResponse) -> Settin
|
||||
provider_env_revision: inner.provider_env_revision,
|
||||
supervisor_middleware_services: inner.supervisor_middleware_services,
|
||||
workspace: inner.workspace,
|
||||
policy_validation_failure_mode: inner
|
||||
.policy_validation_failure_mode
|
||||
.parse()
|
||||
.unwrap_or_default(),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod settings_poll_tests {
|
||||
use super::settings_poll_result;
|
||||
use crate::PolicyValidationFailureMode;
|
||||
use crate::proto::GetSandboxConfigResponse;
|
||||
|
||||
#[test]
|
||||
fn validation_failure_mode_round_trips_from_gateway_config() {
|
||||
let result = settings_poll_result(GetSandboxConfigResponse {
|
||||
policy_validation_failure_mode: "retain_last_valid".to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
assert_eq!(
|
||||
result.policy_validation_failure_mode,
|
||||
PolicyValidationFailureMode::RetainLastValid
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_validation_failure_mode_fails_closed() {
|
||||
let result = settings_poll_result(GetSandboxConfigResponse {
|
||||
policy_validation_failure_mode: "future_mode".to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
assert_eq!(
|
||||
result.policy_validation_failure_mode,
|
||||
PolicyValidationFailureMode::FailClosed
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -45,7 +45,7 @@ pub use config::{
|
||||
ComputeDriverKind, Config, GatewayAuthConfig, GatewayInterceptorBindingOverride,
|
||||
GatewayInterceptorBindingPolicy, GatewayInterceptorConfig, GatewayInterceptorFailurePolicy,
|
||||
GatewayInterceptorPhaseConfig, GatewayJwtConfig, GatewayProviderProfileSourceConfig,
|
||||
MtlsAuthConfig, OidcConfig, TlsConfig,
|
||||
MtlsAuthConfig, OidcConfig, PolicyValidationFailureMode, TlsConfig,
|
||||
};
|
||||
pub use error::{ComputeDriverError, Error, Result};
|
||||
pub use metadata::{
|
||||
|
||||
@@ -0,0 +1,992 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Validation for endpoint selectors whose policy-derived behavior conflicts.
|
||||
|
||||
use openshell_core::proto::{NetworkEndpoint, SandboxPolicy};
|
||||
use std::collections::{BTreeSet, HashSet, VecDeque};
|
||||
use std::fmt;
|
||||
|
||||
/// One pair of endpoints that can authorize the same request but disagree on
|
||||
/// policy-derived behavior that must have a single deterministic value.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct EndpointAmbiguity {
|
||||
pub left_policy: String,
|
||||
pub left_endpoint_index: usize,
|
||||
pub left_selector: String,
|
||||
pub right_policy: String,
|
||||
pub right_endpoint_index: usize,
|
||||
pub right_selector: String,
|
||||
pub overlapping_ports: Vec<u32>,
|
||||
pub conflicts: Vec<String>,
|
||||
}
|
||||
|
||||
impl fmt::Display for EndpointAmbiguity {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
write!(
|
||||
f,
|
||||
"network policies '{}' endpoint[{}] ({}) and '{}' endpoint[{}] ({}) overlap on port(s) {} with conflicting metadata: {}",
|
||||
self.left_policy,
|
||||
self.left_endpoint_index,
|
||||
self.left_selector,
|
||||
self.right_policy,
|
||||
self.right_endpoint_index,
|
||||
self.right_selector,
|
||||
self.overlapping_ports
|
||||
.iter()
|
||||
.map(u32::to_string)
|
||||
.collect::<Vec<_>>()
|
||||
.join(","),
|
||||
self.conflicts.join("; "),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
struct EndpointRef<'a> {
|
||||
policy: &'a str,
|
||||
index: usize,
|
||||
endpoint: &'a NetworkEndpoint,
|
||||
}
|
||||
|
||||
/// Reject endpoint metadata ambiguity before a policy generation is activated.
|
||||
///
|
||||
/// Request authorization rules (`access`, `rules`, and `deny_rules`) may be
|
||||
/// contributed by multiple compatible endpoints. Metadata used to establish
|
||||
/// or parse a connection must agree whenever the endpoint host, port, and (for
|
||||
/// request-specific metadata) path selectors can match the same request.
|
||||
#[must_use]
|
||||
pub fn find_endpoint_ambiguities(policy: &SandboxPolicy) -> Vec<EndpointAmbiguity> {
|
||||
let endpoints = policy
|
||||
.network_policies
|
||||
.iter()
|
||||
.flat_map(|(key, rule)| {
|
||||
let policy_name = if rule.name.is_empty() {
|
||||
key.as_str()
|
||||
} else {
|
||||
rule.name.as_str()
|
||||
};
|
||||
rule.endpoints
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(move |(index, endpoint)| EndpointRef {
|
||||
policy: policy_name,
|
||||
index,
|
||||
endpoint,
|
||||
})
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
let mut ambiguities = Vec::new();
|
||||
for left_index in 0..endpoints.len() {
|
||||
for right_index in (left_index + 1)..endpoints.len() {
|
||||
let left = &endpoints[left_index];
|
||||
let right = &endpoints[right_index];
|
||||
let overlapping_ports = overlapping_ports(left.endpoint, right.endpoint);
|
||||
if overlapping_ports.is_empty()
|
||||
|| !host_patterns_overlap(&left.endpoint.host, &right.endpoint.host)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
let mut conflicts = connection_conflicts(left.endpoint, right.endpoint);
|
||||
if endpoint_contributes_request_pipeline_metadata(left.endpoint)
|
||||
&& endpoint_contributes_request_pipeline_metadata(right.endpoint)
|
||||
&& path_patterns_overlap(&left.endpoint.path, &right.endpoint.path)
|
||||
&& path_selector_specificity(&left.endpoint.path)
|
||||
== path_selector_specificity(&right.endpoint.path)
|
||||
{
|
||||
conflicts.extend(request_pipeline_conflicts(left.endpoint, right.endpoint));
|
||||
}
|
||||
if conflicts.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
ambiguities.push(EndpointAmbiguity {
|
||||
left_policy: left.policy.to_string(),
|
||||
left_endpoint_index: left.index,
|
||||
left_selector: endpoint_selector(left.endpoint),
|
||||
right_policy: right.policy.to_string(),
|
||||
right_endpoint_index: right.index,
|
||||
right_selector: endpoint_selector(right.endpoint),
|
||||
overlapping_ports,
|
||||
conflicts,
|
||||
});
|
||||
}
|
||||
}
|
||||
ambiguities
|
||||
}
|
||||
|
||||
fn endpoint_selector(endpoint: &NetworkEndpoint) -> String {
|
||||
let host = if endpoint.host.is_empty() {
|
||||
"<any-host>"
|
||||
} else {
|
||||
&endpoint.host
|
||||
};
|
||||
let path = if endpoint.path.is_empty() {
|
||||
""
|
||||
} else {
|
||||
endpoint.path.as_str()
|
||||
};
|
||||
format!("{host}:{}{}", display_ports(endpoint), path)
|
||||
}
|
||||
|
||||
fn display_ports(endpoint: &NetworkEndpoint) -> String {
|
||||
effective_ports(endpoint)
|
||||
.iter()
|
||||
.map(u32::to_string)
|
||||
.collect::<Vec<_>>()
|
||||
.join(",")
|
||||
}
|
||||
|
||||
fn effective_ports(endpoint: &NetworkEndpoint) -> BTreeSet<u32> {
|
||||
if endpoint.ports.is_empty() {
|
||||
(endpoint.port > 0)
|
||||
.then_some(endpoint.port)
|
||||
.into_iter()
|
||||
.collect()
|
||||
} else {
|
||||
endpoint
|
||||
.ports
|
||||
.iter()
|
||||
.copied()
|
||||
.filter(|port| *port > 0)
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
|
||||
fn overlapping_ports(left: &NetworkEndpoint, right: &NetworkEndpoint) -> Vec<u32> {
|
||||
effective_ports(left)
|
||||
.intersection(&effective_ports(right))
|
||||
.copied()
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn connection_conflicts(left: &NetworkEndpoint, right: &NetworkEndpoint) -> Vec<String> {
|
||||
let mut conflicts = Vec::new();
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"tls",
|
||||
&normalized_tls(&left.tls),
|
||||
&normalized_tls(&right.tls),
|
||||
);
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"allowed_ips",
|
||||
&normalized_strings(&left.allowed_ips),
|
||||
&normalized_strings(&right.allowed_ips),
|
||||
);
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"advisor_proposed",
|
||||
&left.advisor_proposed,
|
||||
&right.advisor_proposed,
|
||||
);
|
||||
conflicts
|
||||
}
|
||||
|
||||
/// Keep request-pipeline ambiguity checks aligned with Rego's
|
||||
/// `endpoint_has_extended_config` predicate. Plain L4 endpoints authorize a
|
||||
/// destination but do not participate in endpoint-config selection, so they
|
||||
/// cannot compete with the single L7/connection-config endpoint selected for
|
||||
/// that request.
|
||||
fn endpoint_contributes_request_pipeline_metadata(endpoint: &NetworkEndpoint) -> bool {
|
||||
!endpoint.protocol.is_empty() || !endpoint.allowed_ips.is_empty() || !endpoint.tls.is_empty()
|
||||
}
|
||||
|
||||
fn request_pipeline_conflicts(left: &NetworkEndpoint, right: &NetworkEndpoint) -> Vec<String> {
|
||||
let mut conflicts = Vec::new();
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"protocol",
|
||||
&left.protocol.to_ascii_lowercase(),
|
||||
&right.protocol.to_ascii_lowercase(),
|
||||
);
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"enforcement",
|
||||
&normalized_enforcement(&left.enforcement),
|
||||
&normalized_enforcement(&right.enforcement),
|
||||
);
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"allow_encoded_slash",
|
||||
&left.allow_encoded_slash,
|
||||
&right.allow_encoded_slash,
|
||||
);
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"websocket_credential_rewrite",
|
||||
&left.websocket_credential_rewrite,
|
||||
&right.websocket_credential_rewrite,
|
||||
);
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"request_body_credential_rewrite",
|
||||
&left.request_body_credential_rewrite,
|
||||
&right.request_body_credential_rewrite,
|
||||
);
|
||||
if left.protocol.eq_ignore_ascii_case("websocket")
|
||||
&& right.protocol.eq_ignore_ascii_case("websocket")
|
||||
{
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"websocket_graphql_policy",
|
||||
&websocket_graphql_policy(left),
|
||||
&websocket_graphql_policy(right),
|
||||
);
|
||||
}
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"credential_signing",
|
||||
&left.credential_signing,
|
||||
&right.credential_signing,
|
||||
);
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"signing_service",
|
||||
&left.signing_service,
|
||||
&right.signing_service,
|
||||
);
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"signing_region",
|
||||
&left.signing_region,
|
||||
&right.signing_region,
|
||||
);
|
||||
|
||||
if left.protocol.eq_ignore_ascii_case("graphql")
|
||||
&& right.protocol.eq_ignore_ascii_case("graphql")
|
||||
{
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"graphql_max_body_bytes",
|
||||
&normalized_body_limit(left.graphql_max_body_bytes),
|
||||
&normalized_body_limit(right.graphql_max_body_bytes),
|
||||
);
|
||||
}
|
||||
if left.protocol.eq_ignore_ascii_case(&right.protocol) && is_json_rpc_family(&left.protocol) {
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"json_rpc_max_body_bytes",
|
||||
&normalized_body_limit(left.json_rpc_max_body_bytes),
|
||||
&normalized_body_limit(right.json_rpc_max_body_bytes),
|
||||
);
|
||||
}
|
||||
if left.protocol.eq_ignore_ascii_case("mcp") && right.protocol.eq_ignore_ascii_case("mcp") {
|
||||
push_conflict(
|
||||
&mut conflicts,
|
||||
"mcp.strict_tool_names",
|
||||
&normalized_mcp_strict_tool_names(left),
|
||||
&normalized_mcp_strict_tool_names(right),
|
||||
);
|
||||
}
|
||||
conflicts
|
||||
}
|
||||
|
||||
fn websocket_graphql_policy(endpoint: &NetworkEndpoint) -> bool {
|
||||
let allow_rule_has_graphql_fields = endpoint.rules.iter().any(|rule| {
|
||||
rule.allow.as_ref().is_some_and(|allow| {
|
||||
!allow.operation_type.is_empty()
|
||||
|| !allow.operation_name.is_empty()
|
||||
|| !allow.fields.is_empty()
|
||||
})
|
||||
});
|
||||
let deny_rule_has_graphql_fields = endpoint.deny_rules.iter().any(|deny| {
|
||||
!deny.operation_type.is_empty()
|
||||
|| !deny.operation_name.is_empty()
|
||||
|| !deny.fields.is_empty()
|
||||
});
|
||||
|
||||
!endpoint.graphql_persisted_queries.is_empty()
|
||||
|| (!endpoint.persisted_queries.is_empty() && endpoint.persisted_queries != "deny")
|
||||
|| allow_rule_has_graphql_fields
|
||||
|| deny_rule_has_graphql_fields
|
||||
}
|
||||
|
||||
fn push_conflict<T: fmt::Debug + PartialEq>(
|
||||
conflicts: &mut Vec<String>,
|
||||
field: &str,
|
||||
left: &T,
|
||||
right: &T,
|
||||
) {
|
||||
if left != right {
|
||||
conflicts.push(format!("{field}={left:?} vs {right:?}"));
|
||||
}
|
||||
}
|
||||
|
||||
fn normalized_tls(value: &str) -> &'static str {
|
||||
if value.eq_ignore_ascii_case("skip") {
|
||||
"skip"
|
||||
} else {
|
||||
"auto"
|
||||
}
|
||||
}
|
||||
|
||||
fn normalized_enforcement(value: &str) -> &'static str {
|
||||
if value.eq_ignore_ascii_case("enforce") {
|
||||
"enforce"
|
||||
} else {
|
||||
"audit"
|
||||
}
|
||||
}
|
||||
|
||||
fn normalized_strings(values: &[String]) -> Vec<String> {
|
||||
values
|
||||
.iter()
|
||||
.map(|value| value.trim().to_ascii_lowercase())
|
||||
.collect::<BTreeSet<_>>()
|
||||
.into_iter()
|
||||
.collect()
|
||||
}
|
||||
|
||||
const DEFAULT_BODY_LIMIT: u32 = 65_536;
|
||||
|
||||
fn normalized_body_limit(value: u32) -> u32 {
|
||||
if value == 0 {
|
||||
DEFAULT_BODY_LIMIT
|
||||
} else {
|
||||
value
|
||||
}
|
||||
}
|
||||
|
||||
fn is_json_rpc_family(protocol: &str) -> bool {
|
||||
protocol.eq_ignore_ascii_case("json-rpc") || protocol.eq_ignore_ascii_case("mcp")
|
||||
}
|
||||
|
||||
fn normalized_mcp_strict_tool_names(endpoint: &NetworkEndpoint) -> bool {
|
||||
endpoint
|
||||
.mcp
|
||||
.as_ref()
|
||||
.and_then(|options| options.strict_tool_names)
|
||||
.unwrap_or(true)
|
||||
}
|
||||
|
||||
fn host_patterns_overlap(left: &str, right: &str) -> bool {
|
||||
if left.is_empty() || right.is_empty() {
|
||||
return true;
|
||||
}
|
||||
glob_patterns_overlap(&left.to_ascii_lowercase(), &right.to_ascii_lowercase(), '.')
|
||||
}
|
||||
|
||||
fn path_patterns_overlap(left: &str, right: &str) -> bool {
|
||||
if left.is_empty()
|
||||
|| right.is_empty()
|
||||
|| matches!(left, "**" | "/**")
|
||||
|| matches!(right, "**" | "/**")
|
||||
{
|
||||
return true;
|
||||
}
|
||||
runtime_path_patterns_overlap(left, right)
|
||||
}
|
||||
|
||||
/// Match the runtime route-selection rank used by `L7EndpointConfig`.
|
||||
///
|
||||
/// Overlapping endpoints with different ranks do not compete for request
|
||||
/// metadata: the endpoint with the more-specific path wins. Equal-rank
|
||||
/// overlaps must agree because iteration order would otherwise decide which
|
||||
/// parser, credential handling, or enforcement behavior applies.
|
||||
fn path_selector_specificity(path: &str) -> usize {
|
||||
if path.is_empty() {
|
||||
0
|
||||
} else {
|
||||
path.chars().filter(|character| *character != '*').count()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
struct CharacterRange {
|
||||
start: char,
|
||||
end: char,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
enum GlobToken {
|
||||
Literal(char),
|
||||
AnyChar,
|
||||
CharacterClass {
|
||||
ranges: Vec<CharacterRange>,
|
||||
negated: bool,
|
||||
},
|
||||
Star {
|
||||
crosses_delimiter: bool,
|
||||
},
|
||||
}
|
||||
|
||||
fn tokenize_delimited_glob(pattern: &str) -> Vec<GlobToken> {
|
||||
let chars = pattern.chars().collect::<Vec<_>>();
|
||||
let mut tokens = Vec::new();
|
||||
let mut index = 0;
|
||||
while index < chars.len() {
|
||||
if chars[index] != '*' {
|
||||
tokens.push(GlobToken::Literal(chars[index]));
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
let start = index;
|
||||
while index < chars.len() && chars[index] == '*' {
|
||||
index += 1;
|
||||
}
|
||||
tokens.push(GlobToken::Star {
|
||||
crosses_delimiter: index - start >= 2,
|
||||
});
|
||||
}
|
||||
tokens
|
||||
}
|
||||
|
||||
fn tokenize_runtime_path_glob(pattern: &str) -> Option<Vec<GlobToken>> {
|
||||
let chars = pattern.chars().collect::<Vec<_>>();
|
||||
let mut tokens = Vec::new();
|
||||
let mut index = 0;
|
||||
while index < chars.len() {
|
||||
match chars[index] {
|
||||
'?' => {
|
||||
tokens.push(GlobToken::AnyChar);
|
||||
index += 1;
|
||||
}
|
||||
'*' => {
|
||||
let start = index;
|
||||
while index < chars.len() && chars[index] == '*' {
|
||||
index += 1;
|
||||
}
|
||||
let count = index - start;
|
||||
if count > 2 {
|
||||
return None;
|
||||
}
|
||||
if count == 2 {
|
||||
let starts_component = start == 0 || chars[start - 1] == '/';
|
||||
let ends_component = index == chars.len() || chars[index] == '/';
|
||||
if !starts_component || !ends_component {
|
||||
return None;
|
||||
}
|
||||
if index < chars.len() && chars[index] == '/' {
|
||||
index += 1;
|
||||
}
|
||||
}
|
||||
// `glob::Pattern::matches` uses `require_literal_separator:
|
||||
// false`, so both `*` and `**` can consume `/`.
|
||||
tokens.push(GlobToken::Star {
|
||||
crosses_delimiter: true,
|
||||
});
|
||||
}
|
||||
'[' => {
|
||||
let negated = chars.get(index + 1) == Some(&'!');
|
||||
let content_start = index + if negated { 2 } else { 1 };
|
||||
let close = chars[content_start..]
|
||||
.iter()
|
||||
.position(|character| *character == ']')
|
||||
.map(|offset| content_start + offset)?;
|
||||
if close == content_start {
|
||||
return None;
|
||||
}
|
||||
tokens.push(GlobToken::CharacterClass {
|
||||
ranges: parse_character_ranges(&chars[content_start..close]),
|
||||
negated,
|
||||
});
|
||||
index = close + 1;
|
||||
}
|
||||
literal => {
|
||||
tokens.push(GlobToken::Literal(literal));
|
||||
index += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
Some(tokens)
|
||||
}
|
||||
|
||||
fn parse_character_ranges(characters: &[char]) -> Vec<CharacterRange> {
|
||||
let mut ranges = Vec::new();
|
||||
let mut index = 0;
|
||||
while index < characters.len() {
|
||||
if index + 2 < characters.len() && characters[index + 1] == '-' {
|
||||
ranges.push(CharacterRange {
|
||||
start: characters[index],
|
||||
end: characters[index + 2],
|
||||
});
|
||||
index += 3;
|
||||
} else {
|
||||
ranges.push(CharacterRange {
|
||||
start: characters[index],
|
||||
end: characters[index],
|
||||
});
|
||||
index += 1;
|
||||
}
|
||||
}
|
||||
ranges
|
||||
}
|
||||
|
||||
fn runtime_path_patterns_overlap(left: &str, right: &str) -> bool {
|
||||
let (Some(left), Some(right)) = (
|
||||
tokenize_runtime_path_glob(left),
|
||||
tokenize_runtime_path_glob(right),
|
||||
) else {
|
||||
// Invalid path globs are rejected by ordinary policy validation. Keep
|
||||
// ambiguity validation conservative if it is called independently.
|
||||
return true;
|
||||
};
|
||||
token_languages_overlap(&left, &right, '/')
|
||||
}
|
||||
|
||||
/// Decide whether two delimiter-aware glob languages intersect.
|
||||
///
|
||||
/// This is a small product-NFA search. `*` consumes any character except the
|
||||
/// delimiter and `**` consumes any character, including the delimiter. Star
|
||||
/// epsilon transitions and self-loops make the state space finite.
|
||||
fn glob_patterns_overlap(left: &str, right: &str, delimiter: char) -> bool {
|
||||
let left = tokenize_delimited_glob(left);
|
||||
let right = tokenize_delimited_glob(right);
|
||||
token_languages_overlap(&left, &right, delimiter)
|
||||
}
|
||||
|
||||
fn token_languages_overlap(left: &[GlobToken], right: &[GlobToken], delimiter: char) -> bool {
|
||||
let mut queue = VecDeque::from([(0_usize, 0_usize)]);
|
||||
let mut seen = HashSet::new();
|
||||
|
||||
while let Some((left_index, right_index)) = queue.pop_front() {
|
||||
if !seen.insert((left_index, right_index)) {
|
||||
continue;
|
||||
}
|
||||
if left_index == left.len() && right_index == right.len() {
|
||||
return true;
|
||||
}
|
||||
|
||||
if matches!(left.get(left_index), Some(GlobToken::Star { .. })) {
|
||||
queue.push_back((left_index + 1, right_index));
|
||||
}
|
||||
if matches!(right.get(right_index), Some(GlobToken::Star { .. })) {
|
||||
queue.push_back((left_index, right_index + 1));
|
||||
}
|
||||
|
||||
let Some(left_token) = left.get(left_index) else {
|
||||
continue;
|
||||
};
|
||||
let Some(right_token) = right.get(right_index) else {
|
||||
continue;
|
||||
};
|
||||
if tokens_share_character(left_token, right_token, delimiter) {
|
||||
let next_left = if matches!(left_token, GlobToken::Star { .. }) {
|
||||
left_index
|
||||
} else {
|
||||
left_index + 1
|
||||
};
|
||||
let next_right = if matches!(right_token, GlobToken::Star { .. }) {
|
||||
right_index
|
||||
} else {
|
||||
right_index + 1
|
||||
};
|
||||
queue.push_back((next_left, next_right));
|
||||
}
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
fn tokens_share_character(left: &GlobToken, right: &GlobToken, delimiter: char) -> bool {
|
||||
let left_ranges = token_character_ranges(left, delimiter);
|
||||
let right_ranges = token_character_ranges(right, delimiter);
|
||||
left_ranges.iter().any(|left| {
|
||||
right_ranges
|
||||
.iter()
|
||||
.any(|right| left.0 <= right.1 && right.0 <= left.1)
|
||||
})
|
||||
}
|
||||
|
||||
fn token_character_ranges(token: &GlobToken, delimiter: char) -> Vec<(u32, u32)> {
|
||||
match token {
|
||||
GlobToken::Literal(value) => vec![(u32::from(*value), u32::from(*value))],
|
||||
GlobToken::AnyChar
|
||||
| GlobToken::Star {
|
||||
crosses_delimiter: true,
|
||||
} => unicode_scalar_ranges(),
|
||||
GlobToken::Star {
|
||||
crosses_delimiter: false,
|
||||
} => complement_ranges(&[(u32::from(delimiter), u32::from(delimiter))]),
|
||||
GlobToken::CharacterClass { ranges, negated } => {
|
||||
let ranges = normalize_ranges(
|
||||
ranges
|
||||
.iter()
|
||||
.filter(|range| range.start <= range.end)
|
||||
.map(|range| (u32::from(range.start), u32::from(range.end)))
|
||||
.collect(),
|
||||
);
|
||||
if *negated {
|
||||
complement_ranges(&ranges)
|
||||
} else {
|
||||
ranges
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn unicode_scalar_ranges() -> Vec<(u32, u32)> {
|
||||
vec![(0, 0xD7FF), (0xE000, 0x0010_FFFF)]
|
||||
}
|
||||
|
||||
fn normalize_ranges(mut ranges: Vec<(u32, u32)>) -> Vec<(u32, u32)> {
|
||||
ranges.sort_unstable();
|
||||
let mut normalized: Vec<(u32, u32)> = Vec::new();
|
||||
for (start, end) in ranges {
|
||||
for (start, end) in intersect_with_unicode_scalars(start, end) {
|
||||
if let Some(last) = normalized.last_mut()
|
||||
&& start <= last.1.saturating_add(1)
|
||||
{
|
||||
last.1 = last.1.max(end);
|
||||
} else {
|
||||
normalized.push((start, end));
|
||||
}
|
||||
}
|
||||
}
|
||||
normalized
|
||||
}
|
||||
|
||||
fn intersect_with_unicode_scalars(start: u32, end: u32) -> Vec<(u32, u32)> {
|
||||
unicode_scalar_ranges()
|
||||
.into_iter()
|
||||
.filter_map(|(scalar_start, scalar_end)| {
|
||||
let start = start.max(scalar_start);
|
||||
let end = end.min(scalar_end);
|
||||
(start <= end).then_some((start, end))
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn complement_ranges(ranges: &[(u32, u32)]) -> Vec<(u32, u32)> {
|
||||
let ranges = normalize_ranges(ranges.to_vec());
|
||||
let mut complement = Vec::new();
|
||||
for (universe_start, universe_end) in unicode_scalar_ranges() {
|
||||
let mut cursor = universe_start;
|
||||
for &(start, end) in &ranges {
|
||||
if end < universe_start || start > universe_end {
|
||||
continue;
|
||||
}
|
||||
let start = start.max(universe_start);
|
||||
let end = end.min(universe_end);
|
||||
if cursor < start {
|
||||
complement.push((cursor, start - 1));
|
||||
}
|
||||
cursor = end.saturating_add(1);
|
||||
if cursor > universe_end {
|
||||
break;
|
||||
}
|
||||
}
|
||||
if cursor <= universe_end {
|
||||
complement.push((cursor, universe_end));
|
||||
}
|
||||
}
|
||||
complement
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use openshell_core::proto::{L7Allow, L7Rule, NetworkBinary, NetworkPolicyRule};
|
||||
|
||||
fn endpoint(host: &str, port: u32) -> NetworkEndpoint {
|
||||
NetworkEndpoint {
|
||||
host: host.to_string(),
|
||||
port,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
fn policy_with(left: NetworkEndpoint, right: NetworkEndpoint) -> SandboxPolicy {
|
||||
let mut policy = SandboxPolicy::default();
|
||||
policy.network_policies.insert(
|
||||
"left".to_string(),
|
||||
NetworkPolicyRule {
|
||||
name: "left".to_string(),
|
||||
endpoints: vec![left],
|
||||
binaries: vec![NetworkBinary {
|
||||
path: "/usr/bin/curl".to_string(),
|
||||
..Default::default()
|
||||
}],
|
||||
},
|
||||
);
|
||||
policy.network_policies.insert(
|
||||
"right".to_string(),
|
||||
NetworkPolicyRule {
|
||||
name: "right".to_string(),
|
||||
endpoints: vec![right],
|
||||
binaries: vec![NetworkBinary {
|
||||
path: "/usr/bin/bash".to_string(),
|
||||
..Default::default()
|
||||
}],
|
||||
},
|
||||
);
|
||||
policy
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn exact_and_wildcard_hosts_overlap() {
|
||||
assert!(host_patterns_overlap("api.example.com", "*.example.com"));
|
||||
assert!(host_patterns_overlap(
|
||||
"us-aiplatform.googleapis.com",
|
||||
"*-aiplatform.googleapis.com"
|
||||
));
|
||||
assert!(!host_patterns_overlap("api.example.com", "*.other.com"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn intersecting_wildcards_are_detected() {
|
||||
assert!(host_patterns_overlap("*.example.com", "api.*.com"));
|
||||
assert!(host_patterns_overlap("**.example.com", "api.example.com"));
|
||||
assert!(!host_patterns_overlap("*.example.com", "*.example.org"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn disjoint_ports_do_not_overlap() {
|
||||
let mut left = endpoint("api.example.com", 443);
|
||||
left.tls = "skip".to_string();
|
||||
let right = endpoint("api.example.com", 8443);
|
||||
assert!(find_endpoint_ambiguities(&policy_with(left, right)).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compatible_request_rules_may_overlap() {
|
||||
let mut left = endpoint("api.example.com", 443);
|
||||
left.protocol = "rest".to_string();
|
||||
left.tls = "skip".to_string();
|
||||
let mut right = left.clone();
|
||||
left.access = "read-only".to_string();
|
||||
right.access = "read-write".to_string();
|
||||
|
||||
assert!(find_endpoint_ambiguities(&policy_with(left, right)).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plain_l4_endpoint_does_not_compete_with_l7_endpoint_metadata() {
|
||||
let left = endpoint("api.example.com", 443);
|
||||
let mut right = endpoint("api.example.com", 443);
|
||||
right.protocol = "rest".to_string();
|
||||
right.enforcement = "enforce".to_string();
|
||||
|
||||
assert!(find_endpoint_ambiguities(&policy_with(left, right)).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn disjoint_path_specific_protocols_may_overlap() {
|
||||
let mut left = endpoint("api.example.com", 443);
|
||||
left.path = "/graphql".to_string();
|
||||
left.protocol = "graphql".to_string();
|
||||
let mut right = endpoint("api.example.com", 443);
|
||||
right.path = "/repos/**".to_string();
|
||||
right.protocol = "rest".to_string();
|
||||
|
||||
assert!(find_endpoint_ambiguities(&policy_with(left, right)).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn question_mark_path_overlap_is_detected() {
|
||||
let mut left = endpoint("api.example.com", 443);
|
||||
left.path = "/v?".to_string();
|
||||
left.protocol = "rest".to_string();
|
||||
let mut right = endpoint("api.example.com", 443);
|
||||
right.path = "/v1".to_string();
|
||||
right.protocol = "graphql".to_string();
|
||||
|
||||
let ambiguities = find_endpoint_ambiguities(&policy_with(left, right));
|
||||
assert_eq!(ambiguities.len(), 1);
|
||||
assert!(
|
||||
ambiguities[0]
|
||||
.conflicts
|
||||
.iter()
|
||||
.any(|field| field.contains("protocol"))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn overlapping_character_class_paths_are_detected() {
|
||||
let mut left = endpoint("api.example.com", 443);
|
||||
left.path = "/v[12]".to_string();
|
||||
left.protocol = "rest".to_string();
|
||||
let mut right = endpoint("api.example.com", 443);
|
||||
right.path = "/v[23]".to_string();
|
||||
right.protocol = "graphql".to_string();
|
||||
|
||||
assert_eq!(
|
||||
find_endpoint_ambiguities(&policy_with(left, right)).len(),
|
||||
1
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn disjoint_character_class_paths_may_overlap_by_host_and_port() {
|
||||
let mut left = endpoint("api.example.com", 443);
|
||||
left.path = "/v[12]".to_string();
|
||||
left.protocol = "rest".to_string();
|
||||
let mut right = endpoint("api.example.com", 443);
|
||||
right.path = "/v[34]".to_string();
|
||||
right.protocol = "graphql".to_string();
|
||||
|
||||
assert!(find_endpoint_ambiguities(&policy_with(left, right)).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn negated_character_class_paths_follow_runtime_glob_semantics() {
|
||||
assert!(runtime_path_patterns_overlap("/v[!0]", "/v1"));
|
||||
assert!(!runtime_path_patterns_overlap("/v[!0]", "/v0"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn more_specific_path_may_override_request_pipeline_metadata() {
|
||||
let mut left = endpoint("api.example.com", 443);
|
||||
left.protocol = "rest".to_string();
|
||||
left.enforcement = "enforce".to_string();
|
||||
let mut right = endpoint("api.example.com", 443);
|
||||
right.path = "/graphql".to_string();
|
||||
right.protocol = "graphql".to_string();
|
||||
right.enforcement = "enforce".to_string();
|
||||
|
||||
assert!(find_endpoint_ambiguities(&policy_with(left, right)).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn exact_wildcard_tls_conflict_is_rejected() {
|
||||
let mut left = endpoint("*.example.com", 443);
|
||||
left.tls = "skip".to_string();
|
||||
let right = endpoint("api.example.com", 443);
|
||||
let ambiguities = find_endpoint_ambiguities(&policy_with(left, right));
|
||||
|
||||
assert_eq!(ambiguities.len(), 1);
|
||||
assert!(ambiguities[0].conflicts[0].contains("tls"));
|
||||
assert!(ambiguities[0].to_string().contains("left"));
|
||||
assert!(ambiguities[0].to_string().contains("right"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn allowed_ip_conflict_is_rejected_regardless_of_order() {
|
||||
let mut left = endpoint("api.example.com", 443);
|
||||
left.allowed_ips = vec!["10.0.1.0/24".to_string(), "10.0.0.0/24".to_string()];
|
||||
let mut compatible = endpoint("api.example.com", 443);
|
||||
compatible.allowed_ips = vec!["10.0.0.0/24".to_string(), "10.0.1.0/24".to_string()];
|
||||
assert!(find_endpoint_ambiguities(&policy_with(left.clone(), compatible)).is_empty());
|
||||
|
||||
let mut conflicting = endpoint("api.example.com", 443);
|
||||
conflicting.allowed_ips = vec!["10.0.2.0/24".to_string()];
|
||||
let ambiguities = find_endpoint_ambiguities(&policy_with(left, conflicting));
|
||||
assert!(
|
||||
ambiguities[0]
|
||||
.conflicts
|
||||
.iter()
|
||||
.any(|field| field.contains("allowed_ips"))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn credential_and_parser_conflicts_are_rejected_on_same_path() {
|
||||
let mut left = endpoint("api.example.com", 443);
|
||||
left.protocol = "rest".to_string();
|
||||
left.credential_signing = "sigv4".to_string();
|
||||
left.signing_service = "execute-api".to_string();
|
||||
let mut right = left.clone();
|
||||
right.signing_service = "bedrock".to_string();
|
||||
right.allow_encoded_slash = true;
|
||||
|
||||
let ambiguities = find_endpoint_ambiguities(&policy_with(left, right));
|
||||
assert!(
|
||||
ambiguities[0]
|
||||
.conflicts
|
||||
.iter()
|
||||
.any(|field| field.contains("signing_service"))
|
||||
);
|
||||
assert!(
|
||||
ambiguities[0]
|
||||
.conflicts
|
||||
.iter()
|
||||
.any(|field| field.contains("allow_encoded_slash"))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn json_rpc_body_limit_is_compared_only_within_the_same_protocol() {
|
||||
let mut json_rpc = endpoint("api.example.com", 443);
|
||||
json_rpc.protocol = "json-rpc".to_string();
|
||||
json_rpc.json_rpc_max_body_bytes = 1_024;
|
||||
|
||||
let mut mcp = endpoint("api.example.com", 443);
|
||||
mcp.protocol = "mcp".to_string();
|
||||
mcp.json_rpc_max_body_bytes = 2_048;
|
||||
|
||||
let mixed_protocol = find_endpoint_ambiguities(&policy_with(json_rpc, mcp.clone()));
|
||||
assert_eq!(mixed_protocol.len(), 1);
|
||||
assert!(
|
||||
mixed_protocol[0]
|
||||
.conflicts
|
||||
.iter()
|
||||
.any(|field| field.contains("protocol"))
|
||||
);
|
||||
assert!(
|
||||
!mixed_protocol[0]
|
||||
.conflicts
|
||||
.iter()
|
||||
.any(|field| field.contains("json_rpc_max_body_bytes"))
|
||||
);
|
||||
|
||||
let mut other_mcp = mcp.clone();
|
||||
other_mcp.json_rpc_max_body_bytes = 4_096;
|
||||
let same_protocol = find_endpoint_ambiguities(&policy_with(mcp, other_mcp));
|
||||
assert_eq!(same_protocol.len(), 1);
|
||||
assert!(
|
||||
same_protocol[0]
|
||||
.conflicts
|
||||
.iter()
|
||||
.any(|field| field.contains("json_rpc_max_body_bytes"))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn websocket_graphql_classification_conflict_is_rejected() {
|
||||
let mut graphql = endpoint("api.example.com", 443);
|
||||
graphql.protocol = "websocket".to_string();
|
||||
graphql.rules.push(L7Rule {
|
||||
allow: Some(L7Allow {
|
||||
operation_type: "subscription".to_string(),
|
||||
..Default::default()
|
||||
}),
|
||||
});
|
||||
let mut transport = endpoint("api.example.com", 443);
|
||||
transport.protocol = "websocket".to_string();
|
||||
transport.rules.push(L7Rule {
|
||||
allow: Some(L7Allow {
|
||||
method: "WEBSOCKET_TEXT".to_string(),
|
||||
..Default::default()
|
||||
}),
|
||||
});
|
||||
|
||||
let ambiguities = find_endpoint_ambiguities(&policy_with(graphql, transport));
|
||||
assert_eq!(ambiguities.len(), 1);
|
||||
assert!(
|
||||
ambiguities[0]
|
||||
.conflicts
|
||||
.iter()
|
||||
.any(|field| field.contains("websocket_graphql_policy"))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn matching_websocket_graphql_classification_is_compatible() {
|
||||
let mut left = endpoint("api.example.com", 443);
|
||||
left.protocol = "websocket".to_string();
|
||||
left.persisted_queries = "allow_registered".to_string();
|
||||
let mut right = left.clone();
|
||||
right.rules.push(L7Rule {
|
||||
allow: Some(L7Allow {
|
||||
operation_name: "Events".to_string(),
|
||||
..Default::default()
|
||||
}),
|
||||
});
|
||||
|
||||
assert!(find_endpoint_ambiguities(&policy_with(left, right)).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn different_binary_lists_do_not_hide_endpoint_ambiguity() {
|
||||
let mut left = endpoint("api.example.com", 443);
|
||||
left.tls = "skip".to_string();
|
||||
let right = endpoint("api.example.com", 443);
|
||||
|
||||
assert_eq!(
|
||||
find_endpoint_ambiguities(&policy_with(left, right)).len(),
|
||||
1
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -18,6 +18,10 @@ use std::collections::{BTreeMap, HashMap};
|
||||
use std::fmt;
|
||||
use std::path::Path;
|
||||
|
||||
mod ambiguity;
|
||||
|
||||
pub use ambiguity::{EndpointAmbiguity, find_endpoint_ambiguities};
|
||||
|
||||
use miette::{IntoDiagnostic, Result, WrapErr};
|
||||
use openshell_core::proto::{
|
||||
FilesystemPolicy, GraphqlOperation, L7Allow, L7DenyRule, L7QueryMatcher, L7Rule,
|
||||
|
||||
@@ -23,9 +23,12 @@ use std::sync::atomic::{AtomicBool, AtomicU32};
|
||||
use std::time::Duration;
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
use openshell_core::PolicyValidationFailureMode;
|
||||
|
||||
use openshell_ocsf::{
|
||||
ActionId, ActivityId, AppLifecycleBuilder, ConfigStateChangeBuilder, DetectionFindingBuilder,
|
||||
DispositionId, FindingInfo, SandboxContext, SeverityId, StateId, StatusId, ocsf_emit,
|
||||
DispositionId, FindingInfo, OcsfEvent, SandboxContext, SeverityId, StateId, StatusId,
|
||||
ocsf_emit,
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -852,7 +855,10 @@ fn load_policy_from_sidecar_bootstrap(
|
||||
policy,
|
||||
opa_engine,
|
||||
Some(proto),
|
||||
LoadedPolicyOrigin::Gateway { revision: None },
|
||||
LoadedPolicyOrigin::Gateway {
|
||||
revision: None,
|
||||
has_last_valid_policy: true,
|
||||
},
|
||||
))
|
||||
}
|
||||
|
||||
@@ -1992,7 +1998,7 @@ async fn load_policy(
|
||||
}
|
||||
}
|
||||
|
||||
let loaded_policy_revision =
|
||||
let mut loaded_policy_revision =
|
||||
policy_bound_to_snapshot.then(|| LoadedPolicyRevision::from_snapshot(&snapshot));
|
||||
|
||||
// Build OPA engine from baked-in rules + typed proto data.
|
||||
@@ -2002,12 +2008,37 @@ async fn load_policy(
|
||||
// container hasn't started yet. After the entrypoint spawns, the
|
||||
// engine is rebuilt with the real PID for symlink resolution.
|
||||
info!("Creating OPA engine from proto policy data");
|
||||
let mut has_last_valid_policy = true;
|
||||
let engine = match OpaEngine::from_proto(&proto_policy) {
|
||||
Ok(engine) => engine,
|
||||
Ok(engine) => Arc::new(engine),
|
||||
Err(e) => {
|
||||
report_initial_policy_failure(endpoint, id, loaded_policy_revision.as_ref(), &e)
|
||||
.await;
|
||||
return Err(e);
|
||||
let validation_error = e.to_string();
|
||||
let candidate_version = snapshot.version;
|
||||
let candidate_hash = snapshot.policy_hash.clone();
|
||||
// There is no in-memory last-known-good generation during
|
||||
// startup, so both configured modes necessarily fail closed.
|
||||
// Load the restrictive default atomically and keep the
|
||||
// rejected revision unacknowledged for poll reconciliation.
|
||||
has_last_valid_policy = false;
|
||||
proto_policy = openshell_policy::restrictive_default_policy();
|
||||
let engine = Arc::new(OpaEngine::from_proto(&proto_policy)?);
|
||||
let disposition = apply_policy_validation_failure(
|
||||
&engine,
|
||||
snapshot.policy_validation_failure_mode,
|
||||
has_last_valid_policy,
|
||||
candidate_version,
|
||||
&validation_error,
|
||||
)?;
|
||||
emit_policy_validation_failure(
|
||||
&disposition,
|
||||
candidate_version,
|
||||
&candidate_hash,
|
||||
&validation_error,
|
||||
);
|
||||
loaded_policy_revision = None;
|
||||
engine
|
||||
}
|
||||
};
|
||||
|
||||
@@ -2050,7 +2081,7 @@ async fn load_policy(
|
||||
} else {
|
||||
MiddlewareRegistryStatus::Synchronized
|
||||
};
|
||||
let opa_engine = Some(Arc::new(engine));
|
||||
let opa_engine = Some(engine);
|
||||
|
||||
let policy = match SandboxPolicy::try_from(proto_policy.clone()) {
|
||||
Ok(policy) => policy,
|
||||
@@ -2067,6 +2098,7 @@ async fn load_policy(
|
||||
middleware_registry_status,
|
||||
LoadedPolicyOrigin::Gateway {
|
||||
revision: loaded_policy_revision,
|
||||
has_last_valid_policy,
|
||||
},
|
||||
agent_proposals_enabled_from_settings(&snapshot.settings),
|
||||
));
|
||||
@@ -2194,6 +2226,72 @@ enum MiddlewareRegistryStatus {
|
||||
NeedsReconciliation,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
enum GatewayRuntimeReloadError {
|
||||
PolicyValidation(miette::Report),
|
||||
MiddlewareRegistry(miette::Report),
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
enum GatewayRuntimeFailureClass {
|
||||
PolicyValidation,
|
||||
MiddlewareRegistry,
|
||||
}
|
||||
|
||||
impl GatewayRuntimeReloadError {
|
||||
fn class(&self) -> GatewayRuntimeFailureClass {
|
||||
match self {
|
||||
Self::PolicyValidation(_) => GatewayRuntimeFailureClass::PolicyValidation,
|
||||
Self::MiddlewareRegistry(_) => GatewayRuntimeFailureClass::MiddlewareRegistry,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
struct FailedRuntimeRevision {
|
||||
config_revision: u64,
|
||||
policy_hash: String,
|
||||
failure_class: GatewayRuntimeFailureClass,
|
||||
}
|
||||
|
||||
impl FailedRuntimeRevision {
|
||||
fn new(config_revision: u64, policy_hash: &str, failure: &GatewayRuntimeReloadError) -> Self {
|
||||
Self {
|
||||
config_revision,
|
||||
policy_hash: policy_hash.to_string(),
|
||||
failure_class: failure.class(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn reload_gateway_policy_runtime(
|
||||
engine: &OpaEngine,
|
||||
policy: Option<&openshell_core::proto::SandboxPolicy>,
|
||||
entrypoint_pid: u32,
|
||||
desired_services: &[openshell_core::proto::SupervisorMiddlewareService],
|
||||
middleware_registry_changed: bool,
|
||||
) -> std::result::Result<(), GatewayRuntimeReloadError> {
|
||||
match policy {
|
||||
Some(policy) if middleware_registry_changed => {
|
||||
let registry = connect_middleware_registry(desired_services)
|
||||
.await
|
||||
.map_err(GatewayRuntimeReloadError::MiddlewareRegistry)?;
|
||||
engine
|
||||
.reload_policy_and_middleware_from_proto_with_pid(policy, entrypoint_pid, registry)
|
||||
.map_err(GatewayRuntimeReloadError::PolicyValidation)
|
||||
}
|
||||
// Policy-only change: the installed registry already matches the
|
||||
// delivered service set, so swap the engine alone. This must not
|
||||
// require middleware reachability.
|
||||
Some(policy) => engine
|
||||
.reload_from_proto_with_pid(policy, entrypoint_pid)
|
||||
.map_err(GatewayRuntimeReloadError::PolicyValidation),
|
||||
None => Err(GatewayRuntimeReloadError::PolicyValidation(
|
||||
miette::miette!("runtime reload requires a policy payload but none was returned"),
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
/// True when the installed middleware registry no longer matches the desired
|
||||
/// service set and must be rebuilt (reconnecting every delivered service).
|
||||
///
|
||||
@@ -2249,6 +2347,7 @@ enum LoadedPolicyOrigin {
|
||||
LocalOverride,
|
||||
Gateway {
|
||||
revision: Option<LoadedPolicyRevision>,
|
||||
has_last_valid_policy: bool,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -2256,6 +2355,16 @@ impl LoadedPolicyOrigin {
|
||||
fn allows_gateway_policy_reload(&self) -> bool {
|
||||
matches!(self, Self::Gateway { .. })
|
||||
}
|
||||
|
||||
fn has_last_valid_policy(&self) -> bool {
|
||||
match self {
|
||||
Self::LocalOverride => true,
|
||||
Self::Gateway {
|
||||
has_last_valid_policy,
|
||||
..
|
||||
} => *has_last_valid_policy,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl LoadedPolicyRevision {
|
||||
@@ -2362,7 +2471,7 @@ fn initial_poll_disposition(
|
||||
) -> InitialPollDisposition {
|
||||
match origin {
|
||||
LoadedPolicyOrigin::LocalOverride => InitialPollDisposition::TrackOnly,
|
||||
LoadedPolicyOrigin::Gateway { revision } => {
|
||||
LoadedPolicyOrigin::Gateway { revision, .. } => {
|
||||
initial_policy_ack_candidate(revision.as_ref(), canonical).map_or(
|
||||
InitialPollDisposition::Reconcile,
|
||||
InitialPollDisposition::Acknowledge,
|
||||
@@ -2594,6 +2703,187 @@ async fn reconcile_middleware_registry(
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
struct PolicyValidationFailureDisposition {
|
||||
configured_mode: PolicyValidationFailureMode,
|
||||
mode: PolicyValidationFailureMode,
|
||||
previous_policy_active: bool,
|
||||
active_generation: u64,
|
||||
}
|
||||
|
||||
struct RejectedPolicyGeneration {
|
||||
version: u32,
|
||||
policy_hash: String,
|
||||
validation_error: String,
|
||||
configured_mode: PolicyValidationFailureMode,
|
||||
}
|
||||
|
||||
enum GatewayRuntimeFailureDisposition {
|
||||
PolicyRejected {
|
||||
error: String,
|
||||
disposition: PolicyValidationFailureDisposition,
|
||||
},
|
||||
MiddlewareUnavailable {
|
||||
error: String,
|
||||
},
|
||||
}
|
||||
|
||||
fn apply_gateway_runtime_reload_failure(
|
||||
engine: &OpaEngine,
|
||||
failure: GatewayRuntimeReloadError,
|
||||
configured_mode: PolicyValidationFailureMode,
|
||||
has_last_valid_policy: bool,
|
||||
version: u32,
|
||||
) -> Result<GatewayRuntimeFailureDisposition> {
|
||||
match failure {
|
||||
GatewayRuntimeReloadError::PolicyValidation(error) => {
|
||||
let error = error.to_string();
|
||||
let disposition = apply_policy_validation_failure(
|
||||
engine,
|
||||
configured_mode,
|
||||
has_last_valid_policy,
|
||||
version,
|
||||
&error,
|
||||
)?;
|
||||
Ok(GatewayRuntimeFailureDisposition::PolicyRejected { error, disposition })
|
||||
}
|
||||
GatewayRuntimeReloadError::MiddlewareRegistry(error) => {
|
||||
Ok(GatewayRuntimeFailureDisposition::MiddlewareUnavailable {
|
||||
error: error.to_string(),
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn apply_policy_validation_failure(
|
||||
engine: &OpaEngine,
|
||||
configured_mode: PolicyValidationFailureMode,
|
||||
has_last_valid_policy: bool,
|
||||
version: u32,
|
||||
error: &str,
|
||||
) -> Result<PolicyValidationFailureDisposition> {
|
||||
let mode = if has_last_valid_policy {
|
||||
configured_mode
|
||||
} else {
|
||||
PolicyValidationFailureMode::FailClosed
|
||||
};
|
||||
match mode {
|
||||
PolicyValidationFailureMode::FailClosed => {
|
||||
let reason = format!(
|
||||
"policy validation failed; fail-closed quarantine is active; candidate version {version} rejected: {error}"
|
||||
);
|
||||
let active_generation = engine.enter_fail_closed(reason)?;
|
||||
Ok(PolicyValidationFailureDisposition {
|
||||
configured_mode,
|
||||
mode,
|
||||
previous_policy_active: false,
|
||||
active_generation,
|
||||
})
|
||||
}
|
||||
PolicyValidationFailureMode::RetainLastValid => {
|
||||
let active_generation = engine.exit_fail_closed()?;
|
||||
Ok(PolicyValidationFailureDisposition {
|
||||
configured_mode,
|
||||
mode,
|
||||
previous_policy_active: true,
|
||||
active_generation,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn policy_validation_failure_events(
|
||||
disposition: &PolicyValidationFailureDisposition,
|
||||
version: u32,
|
||||
policy_hash: &str,
|
||||
error: &str,
|
||||
) -> [OcsfEvent; 2] {
|
||||
let previous_policy_state = if disposition.previous_policy_active {
|
||||
"IS active"
|
||||
} else {
|
||||
"IS NOT active"
|
||||
};
|
||||
let state = if disposition.previous_policy_active {
|
||||
(StateId::Enabled, "retained_last_valid")
|
||||
} else {
|
||||
(StateId::Disabled, "fail_closed")
|
||||
};
|
||||
let message = format!(
|
||||
"Policy validation failed; configured_mode={} effective_mode={}; previous policy {previous_policy_state} [version:{version} active_generation:{} error:{error}]",
|
||||
disposition.configured_mode.as_str(),
|
||||
disposition.mode.as_str(),
|
||||
disposition.active_generation,
|
||||
);
|
||||
let finding_uid = format!("policy-validation-failed-{version}");
|
||||
let version_string = version.to_string();
|
||||
let config = ConfigStateChangeBuilder::new(ocsf_ctx())
|
||||
.severity(SeverityId::High)
|
||||
.status(StatusId::Failure)
|
||||
.state(state.0, state.1)
|
||||
.unmapped("candidate_version", serde_json::json!(version))
|
||||
.unmapped("candidate_policy_hash", serde_json::json!(policy_hash))
|
||||
.unmapped(
|
||||
"validation_failure_mode",
|
||||
serde_json::json!(disposition.mode.as_str()),
|
||||
)
|
||||
.unmapped(
|
||||
"configured_validation_failure_mode",
|
||||
serde_json::json!(disposition.configured_mode.as_str()),
|
||||
)
|
||||
.unmapped(
|
||||
"previous_policy_active",
|
||||
serde_json::json!(disposition.previous_policy_active),
|
||||
)
|
||||
.unmapped(
|
||||
"active_generation",
|
||||
serde_json::json!(disposition.active_generation),
|
||||
)
|
||||
.unmapped("validation_error", serde_json::json!(error))
|
||||
.message(message.clone())
|
||||
.build();
|
||||
let finding = DetectionFindingBuilder::new(ocsf_ctx())
|
||||
.activity(ActivityId::Open)
|
||||
.action(ActionId::Denied)
|
||||
.disposition(DispositionId::Blocked)
|
||||
.severity(SeverityId::High)
|
||||
.is_alert(true)
|
||||
.finding_info(
|
||||
FindingInfo::new(&finding_uid, "Invalid policy generation rejected").with_desc(error),
|
||||
)
|
||||
.evidence_pairs(&[
|
||||
("candidate_version", &version_string),
|
||||
("candidate_policy_hash", policy_hash),
|
||||
("validation_failure_mode", disposition.mode.as_str()),
|
||||
(
|
||||
"configured_validation_failure_mode",
|
||||
disposition.configured_mode.as_str(),
|
||||
),
|
||||
(
|
||||
"previous_policy_active",
|
||||
if disposition.previous_policy_active {
|
||||
"true"
|
||||
} else {
|
||||
"false"
|
||||
},
|
||||
),
|
||||
])
|
||||
.remediation("Submit a valid, unambiguous policy generation")
|
||||
.message(message)
|
||||
.build();
|
||||
[config, finding]
|
||||
}
|
||||
|
||||
fn emit_policy_validation_failure(
|
||||
disposition: &PolicyValidationFailureDisposition,
|
||||
version: u32,
|
||||
policy_hash: &str,
|
||||
error: &str,
|
||||
) {
|
||||
for event in policy_validation_failure_events(disposition, version, policy_hash, error) {
|
||||
ocsf_emit!(event);
|
||||
}
|
||||
}
|
||||
|
||||
async fn run_policy_poll_loop(ctx: PolicyPollLoopContext) -> Result<()> {
|
||||
use openshell_core::grpc_client::CachedOpenShellClient;
|
||||
use openshell_core::proto::PolicySource;
|
||||
@@ -2617,7 +2907,9 @@ async fn run_policy_poll_loop(ctx: PolicyPollLoopContext) -> Result<()> {
|
||||
openshell_core::proto::EffectiveSetting,
|
||||
> = std::collections::HashMap::new();
|
||||
let reloads_gateway_policy = ctx.loaded_policy_origin.allows_gateway_policy_reload();
|
||||
let mut last_failed_runtime_revision: Option<(u64, String)> = None;
|
||||
let mut last_failed_runtime_revision: Option<FailedRuntimeRevision> = None;
|
||||
let mut rejected_policy_generation: Option<RejectedPolicyGeneration> = None;
|
||||
let mut has_last_valid_policy = ctx.loaded_policy_origin.has_last_valid_policy();
|
||||
|
||||
// A first poll that does not match the policy already loaded into OPA must
|
||||
// pass through the normal reconciliation path immediately. It must never
|
||||
@@ -2707,14 +2999,23 @@ async fn run_policy_poll_loop(ctx: PolicyPollLoopContext) -> Result<()> {
|
||||
¤t_middleware_services,
|
||||
&result.supervisor_middleware_services,
|
||||
);
|
||||
let policy_runtime_changed = gateway_policy_runtime_needs_reconciliation(
|
||||
reloads_gateway_policy,
|
||||
¤t_policy_hash,
|
||||
&result.policy_hash,
|
||||
¤t_middleware_services,
|
||||
&result.supervisor_middleware_services,
|
||||
middleware_registry_status,
|
||||
);
|
||||
// A valid candidate may intentionally restore byte-for-byte policy
|
||||
// content that was active before a rejected update. Its hash then
|
||||
// equals `current_policy_hash`, but the runtime is still quarantined
|
||||
// and must reload (or it would remain deny-all indefinitely).
|
||||
let recovering_rejected_policy = reloads_gateway_policy
|
||||
&& rejected_policy_generation
|
||||
.as_ref()
|
||||
.is_some_and(|rejected| rejected.policy_hash != result.policy_hash);
|
||||
let policy_runtime_changed = recovering_rejected_policy
|
||||
|| gateway_policy_runtime_needs_reconciliation(
|
||||
reloads_gateway_policy,
|
||||
¤t_policy_hash,
|
||||
&result.policy_hash,
|
||||
¤t_middleware_services,
|
||||
&result.supervisor_middleware_services,
|
||||
middleware_registry_status,
|
||||
);
|
||||
|
||||
// A local policy override is not coupled to the gateway policy
|
||||
// snapshot, so its service registry can still be reconciled alone.
|
||||
@@ -2738,6 +3039,30 @@ async fn run_policy_poll_loop(ctx: PolicyPollLoopContext) -> Result<()> {
|
||||
// Log which settings changed.
|
||||
log_setting_changes(¤t_settings, &result.settings);
|
||||
|
||||
// A posture change after a rejected update takes effect immediately.
|
||||
// The compiled last-known-good engine remains available beneath a
|
||||
// fail-closed quarantine, so an explicit retain_last_valid selection
|
||||
// can reactivate it without accepting any part of the invalid policy.
|
||||
if !policy_changed && let Some(rejected) = rejected_policy_generation.as_mut() {
|
||||
let mode = result.policy_validation_failure_mode;
|
||||
if mode != rejected.configured_mode {
|
||||
let disposition = apply_policy_validation_failure(
|
||||
&ctx.opa_engine,
|
||||
mode,
|
||||
has_last_valid_policy,
|
||||
rejected.version,
|
||||
&rejected.validation_error,
|
||||
)?;
|
||||
emit_policy_validation_failure(
|
||||
&disposition,
|
||||
rejected.version,
|
||||
&rejected.policy_hash,
|
||||
&rejected.validation_error,
|
||||
);
|
||||
rejected.configured_mode = mode;
|
||||
}
|
||||
}
|
||||
|
||||
ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx())
|
||||
.severity(SeverityId::Informational)
|
||||
.status(StatusId::Success)
|
||||
@@ -2804,26 +3129,14 @@ async fn run_policy_poll_loop(ctx: PolicyPollLoopContext) -> Result<()> {
|
||||
|
||||
if policy_runtime_changed {
|
||||
let pid = ctx.entrypoint_pid.load(Ordering::Acquire);
|
||||
let runtime_result = match result.policy.as_ref() {
|
||||
Some(policy) if middleware_registry_changed => {
|
||||
match connect_middleware_registry(&result.supervisor_middleware_services).await
|
||||
{
|
||||
Ok(registry) => ctx
|
||||
.opa_engine
|
||||
.reload_policy_and_middleware_from_proto_with_pid(
|
||||
policy, pid, registry,
|
||||
),
|
||||
Err(error) => Err(error),
|
||||
}
|
||||
}
|
||||
// Policy-only change: the installed registry already matches
|
||||
// the delivered service set, so swap the engine alone. This
|
||||
// must not require middleware reachability.
|
||||
Some(policy) => ctx.opa_engine.reload_from_proto_with_pid(policy, pid),
|
||||
None => Err(miette::miette!(
|
||||
"runtime reload requires a policy payload but none was returned"
|
||||
)),
|
||||
};
|
||||
let runtime_result = reload_gateway_policy_runtime(
|
||||
&ctx.opa_engine,
|
||||
result.policy.as_ref(),
|
||||
pid,
|
||||
&result.supervisor_middleware_services,
|
||||
middleware_registry_changed,
|
||||
)
|
||||
.await;
|
||||
|
||||
match runtime_result {
|
||||
Ok(()) => {
|
||||
@@ -2831,6 +3144,8 @@ async fn run_policy_poll_loop(ctx: PolicyPollLoopContext) -> Result<()> {
|
||||
.policy
|
||||
.as_ref()
|
||||
.expect("successful runtime reload requires a policy payload");
|
||||
has_last_valid_policy = true;
|
||||
rejected_policy_generation = None;
|
||||
if policy_changed {
|
||||
if let Some(policy_local_ctx) = ctx.policy_local_ctx.as_ref() {
|
||||
policy_local_ctx.set_current_policy(policy.clone()).await;
|
||||
@@ -2875,6 +3190,26 @@ async fn run_policy_poll_loop(ctx: PolicyPollLoopContext) -> Result<()> {
|
||||
PolicyStatusUpdate::loaded(result.version),
|
||||
);
|
||||
}
|
||||
} else if recovering_rejected_policy
|
||||
&& result.version > 0
|
||||
&& result.policy_source == PolicySource::Sandbox
|
||||
{
|
||||
ocsf_emit!(
|
||||
ConfigStateChangeBuilder::new(ocsf_ctx())
|
||||
.severity(SeverityId::Informational)
|
||||
.status(StatusId::Success)
|
||||
.state(StateId::Enabled, "loaded")
|
||||
.unmapped("policy_hash", serde_json::json!(&result.policy_hash))
|
||||
.message(format!(
|
||||
"Policy reloaded successfully and fail-closed quarantine cleared [policy_hash:{}]",
|
||||
result.policy_hash
|
||||
))
|
||||
.build()
|
||||
);
|
||||
enqueue_policy_status(
|
||||
&status_sender,
|
||||
PolicyStatusUpdate::loaded(result.version),
|
||||
);
|
||||
}
|
||||
|
||||
if middleware_registry_changed {
|
||||
@@ -2898,28 +3233,61 @@ async fn run_policy_poll_loop(ctx: PolicyPollLoopContext) -> Result<()> {
|
||||
middleware_registry_status = MiddlewareRegistryStatus::Synchronized;
|
||||
last_failed_runtime_revision = None;
|
||||
}
|
||||
Err(e) => {
|
||||
let failed_revision = (result.config_revision, result.policy_hash.clone());
|
||||
Err(failure) => {
|
||||
let failed_revision = FailedRuntimeRevision::new(
|
||||
result.config_revision,
|
||||
&result.policy_hash,
|
||||
&failure,
|
||||
);
|
||||
if last_failed_runtime_revision.as_ref() != Some(&failed_revision) {
|
||||
ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx())
|
||||
.severity(SeverityId::Medium)
|
||||
.status(StatusId::Failure)
|
||||
.state(StateId::Other, "failed")
|
||||
.unmapped("version", serde_json::json!(result.version))
|
||||
.unmapped("error", serde_json::json!(e.to_string()))
|
||||
.message(format!(
|
||||
"Policy and middleware runtime reload failed, keeping last-known-good runtime [version:{} error:{e}]",
|
||||
result.version
|
||||
))
|
||||
.build());
|
||||
if policy_changed
|
||||
&& result.version > 0
|
||||
&& result.policy_source == PolicySource::Sandbox
|
||||
{
|
||||
enqueue_policy_status(
|
||||
&status_sender,
|
||||
PolicyStatusUpdate::failed(result.version, e.to_string()),
|
||||
);
|
||||
let failure_mode = result.policy_validation_failure_mode;
|
||||
match apply_gateway_runtime_reload_failure(
|
||||
&ctx.opa_engine,
|
||||
failure,
|
||||
failure_mode,
|
||||
has_last_valid_policy,
|
||||
result.version,
|
||||
)? {
|
||||
GatewayRuntimeFailureDisposition::PolicyRejected {
|
||||
error,
|
||||
disposition,
|
||||
} => {
|
||||
emit_policy_validation_failure(
|
||||
&disposition,
|
||||
result.version,
|
||||
&result.policy_hash,
|
||||
&error,
|
||||
);
|
||||
rejected_policy_generation = Some(RejectedPolicyGeneration {
|
||||
version: result.version,
|
||||
policy_hash: result.policy_hash.clone(),
|
||||
validation_error: error.clone(),
|
||||
configured_mode: failure_mode,
|
||||
});
|
||||
if policy_changed
|
||||
&& result.version > 0
|
||||
&& result.policy_source == PolicySource::Sandbox
|
||||
{
|
||||
enqueue_policy_status(
|
||||
&status_sender,
|
||||
PolicyStatusUpdate::failed(result.version, error),
|
||||
);
|
||||
}
|
||||
}
|
||||
GatewayRuntimeFailureDisposition::MiddlewareUnavailable { error } => {
|
||||
ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx())
|
||||
.severity(SeverityId::Medium)
|
||||
.status(StatusId::Failure)
|
||||
.state(StateId::Other, "failed")
|
||||
.unmapped("version", serde_json::json!(result.version))
|
||||
.unmapped("error", serde_json::json!(&error))
|
||||
.unmapped("previous_policy_active", serde_json::json!(true))
|
||||
.message(format!(
|
||||
"Supervisor middleware registry unavailable, keeping last-known-good policy runtime active [version:{} error:{error}]",
|
||||
result.version
|
||||
))
|
||||
.build());
|
||||
}
|
||||
}
|
||||
}
|
||||
last_failed_runtime_revision = Some(failed_revision);
|
||||
@@ -3458,6 +3826,7 @@ filesystem_policy:
|
||||
provider_env_revision: 0,
|
||||
supervisor_middleware_services: Vec::new(),
|
||||
workspace: String::new(),
|
||||
policy_validation_failure_mode: PolicyValidationFailureMode::default(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3483,6 +3852,96 @@ filesystem_policy:
|
||||
assert_eq!(engine.current_generation(), builtins_generation);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unavailable_middleware_reload_keeps_last_known_good_runtime_active() {
|
||||
let engine = OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine");
|
||||
install_builtin_middleware_registry(&engine)
|
||||
.await
|
||||
.expect("install built-in middleware registry");
|
||||
let active_generation = engine.current_generation();
|
||||
let unavailable_service = openshell_core::proto::SupervisorMiddlewareService {
|
||||
name: "unavailable-guard".into(),
|
||||
grpc_endpoint: "http://127.0.0.1:1".into(),
|
||||
max_body_bytes: 1024,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let failure = reload_gateway_policy_runtime(
|
||||
&engine,
|
||||
Some(&proto_policy_fixture()),
|
||||
0,
|
||||
&[unavailable_service],
|
||||
true,
|
||||
)
|
||||
.await
|
||||
.expect_err("unavailable middleware must fail candidate preparation");
|
||||
let disposition = apply_gateway_runtime_reload_failure(
|
||||
&engine,
|
||||
failure,
|
||||
PolicyValidationFailureMode::FailClosed,
|
||||
true,
|
||||
2,
|
||||
)
|
||||
.expect("middleware failure handling must succeed");
|
||||
|
||||
assert!(matches!(
|
||||
disposition,
|
||||
GatewayRuntimeFailureDisposition::MiddlewareUnavailable { .. }
|
||||
));
|
||||
assert_eq!(engine.current_generation(), active_generation);
|
||||
assert!(engine.fail_closed_reason().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn policy_rejection_after_middleware_outage_is_not_deduplicated() {
|
||||
let engine = OpaEngine::from_strings(
|
||||
include_str!("../../openshell-supervisor-network/data/sandbox-policy.rego"),
|
||||
"network_policies: {}\n",
|
||||
)
|
||||
.unwrap();
|
||||
let middleware_failure = GatewayRuntimeReloadError::MiddlewareRegistry(miette::miette!(
|
||||
"middleware service unavailable"
|
||||
));
|
||||
let first_failure = FailedRuntimeRevision::new(42, "sha256:candidate", &middleware_failure);
|
||||
let middleware_disposition = apply_gateway_runtime_reload_failure(
|
||||
&engine,
|
||||
middleware_failure,
|
||||
PolicyValidationFailureMode::FailClosed,
|
||||
true,
|
||||
7,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert!(matches!(
|
||||
middleware_disposition,
|
||||
GatewayRuntimeFailureDisposition::MiddlewareUnavailable { .. }
|
||||
));
|
||||
assert!(engine.fail_closed_reason().is_none());
|
||||
|
||||
let policy_failure = GatewayRuntimeReloadError::PolicyValidation(miette::miette!(
|
||||
"conflicting endpoint metadata"
|
||||
));
|
||||
let second_failure = FailedRuntimeRevision::new(42, "sha256:candidate", &policy_failure);
|
||||
assert_ne!(
|
||||
first_failure, second_failure,
|
||||
"a changed failure class for the same candidate must be handled"
|
||||
);
|
||||
|
||||
let policy_disposition = apply_gateway_runtime_reload_failure(
|
||||
&engine,
|
||||
policy_failure,
|
||||
PolicyValidationFailureMode::FailClosed,
|
||||
true,
|
||||
7,
|
||||
)
|
||||
.unwrap();
|
||||
assert!(matches!(
|
||||
policy_disposition,
|
||||
GatewayRuntimeFailureDisposition::PolicyRejected { .. }
|
||||
));
|
||||
assert!(engine.fail_closed_reason().is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn failed_gateway_runtime_snapshot_is_retried_without_revision_change() {
|
||||
let services = Vec::new();
|
||||
@@ -3684,6 +4143,7 @@ filesystem_policy:
|
||||
initial_poll_disposition(
|
||||
&LoadedPolicyOrigin::Gateway {
|
||||
revision: Some(loaded),
|
||||
has_last_valid_policy: true,
|
||||
},
|
||||
&canonical,
|
||||
),
|
||||
@@ -3713,7 +4173,10 @@ filesystem_policy:
|
||||
2,
|
||||
openshell_core::proto::PolicySource::Sandbox,
|
||||
);
|
||||
let origin = LoadedPolicyOrigin::Gateway { revision: None };
|
||||
let origin = LoadedPolicyOrigin::Gateway {
|
||||
revision: None,
|
||||
has_last_valid_policy: true,
|
||||
};
|
||||
|
||||
assert_eq!(
|
||||
initial_poll_disposition(&origin, &canonical),
|
||||
@@ -3753,4 +4216,167 @@ filesystem_policy:
|
||||
"workspace must survive the snapshot so sync_policy_and_fetch_snapshot receives it"
|
||||
);
|
||||
}
|
||||
#[test]
|
||||
fn fail_closed_validation_failure_deactivates_previous_generation() {
|
||||
let engine = OpaEngine::from_strings(
|
||||
include_str!("../../openshell-supervisor-network/data/sandbox-policy.rego"),
|
||||
"network_policies: {}\n",
|
||||
)
|
||||
.unwrap();
|
||||
let previous_generation = engine.current_generation();
|
||||
|
||||
let disposition = apply_policy_validation_failure(
|
||||
&engine,
|
||||
PolicyValidationFailureMode::FailClosed,
|
||||
true,
|
||||
7,
|
||||
"conflicting tls metadata",
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert!(!disposition.previous_policy_active);
|
||||
assert!(disposition.active_generation > previous_generation);
|
||||
assert!(
|
||||
engine
|
||||
.fail_closed_reason()
|
||||
.expect("quarantine reason")
|
||||
.contains("candidate version 7 rejected")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retain_validation_failure_keeps_previous_generation_active() {
|
||||
let engine = OpaEngine::from_strings(
|
||||
include_str!("../../openshell-supervisor-network/data/sandbox-policy.rego"),
|
||||
"network_policies: {}\n",
|
||||
)
|
||||
.unwrap();
|
||||
let previous_generation = engine.current_generation();
|
||||
|
||||
let quarantined = apply_policy_validation_failure(
|
||||
&engine,
|
||||
PolicyValidationFailureMode::FailClosed,
|
||||
true,
|
||||
6,
|
||||
"conflicting tls metadata",
|
||||
)
|
||||
.unwrap();
|
||||
assert!(!quarantined.previous_policy_active);
|
||||
|
||||
let disposition = apply_policy_validation_failure(
|
||||
&engine,
|
||||
PolicyValidationFailureMode::RetainLastValid,
|
||||
true,
|
||||
7,
|
||||
"conflicting tls metadata",
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert!(disposition.previous_policy_active);
|
||||
assert!(disposition.active_generation > quarantined.active_generation);
|
||||
assert!(disposition.active_generation > previous_generation);
|
||||
assert!(engine.fail_closed_reason().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retain_validation_failure_without_last_valid_policy_stays_fail_closed() {
|
||||
let engine = OpaEngine::from_strings(
|
||||
include_str!("../../openshell-supervisor-network/data/sandbox-policy.rego"),
|
||||
"network_policies: {}\n",
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let disposition = apply_policy_validation_failure(
|
||||
&engine,
|
||||
PolicyValidationFailureMode::RetainLastValid,
|
||||
false,
|
||||
1,
|
||||
"conflicting tls metadata",
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(
|
||||
disposition.configured_mode,
|
||||
PolicyValidationFailureMode::RetainLastValid
|
||||
);
|
||||
assert_eq!(disposition.mode, PolicyValidationFailureMode::FailClosed);
|
||||
assert!(!disposition.previous_policy_active);
|
||||
assert!(engine.fail_closed_reason().is_some());
|
||||
|
||||
let [config, _] = policy_validation_failure_events(
|
||||
&disposition,
|
||||
1,
|
||||
"sha256:test",
|
||||
"conflicting tls metadata",
|
||||
);
|
||||
let config = config.to_json().unwrap();
|
||||
assert_eq!(config["unmapped"]["validation_failure_mode"], "fail_closed");
|
||||
assert_eq!(
|
||||
config["unmapped"]["configured_validation_failure_mode"],
|
||||
"retain_last_valid"
|
||||
);
|
||||
assert!(
|
||||
config["message"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.contains("previous policy IS NOT active")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validation_failure_ocsf_states_whether_previous_policy_is_active() {
|
||||
let fail_closed = PolicyValidationFailureDisposition {
|
||||
configured_mode: PolicyValidationFailureMode::FailClosed,
|
||||
mode: PolicyValidationFailureMode::FailClosed,
|
||||
previous_policy_active: false,
|
||||
active_generation: 9,
|
||||
};
|
||||
let [config, finding] = policy_validation_failure_events(
|
||||
&fail_closed,
|
||||
8,
|
||||
"sha256:test",
|
||||
"conflicting tls metadata",
|
||||
);
|
||||
let config = config.to_json().unwrap();
|
||||
assert_eq!(config["class_uid"], 5019);
|
||||
assert_eq!(config["status"], "Failure");
|
||||
assert_eq!(config["unmapped"]["validation_failure_mode"], "fail_closed");
|
||||
assert_eq!(
|
||||
config["unmapped"]["configured_validation_failure_mode"],
|
||||
"fail_closed"
|
||||
);
|
||||
assert_eq!(config["unmapped"]["previous_policy_active"], false);
|
||||
assert!(
|
||||
config["message"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.contains("previous policy IS NOT active")
|
||||
);
|
||||
|
||||
let finding = finding.to_json().unwrap();
|
||||
assert_eq!(finding["class_uid"], 2004);
|
||||
assert_eq!(finding["action"], "Denied");
|
||||
assert_eq!(finding["disposition"], "Blocked");
|
||||
|
||||
let retained = PolicyValidationFailureDisposition {
|
||||
configured_mode: PolicyValidationFailureMode::RetainLastValid,
|
||||
mode: PolicyValidationFailureMode::RetainLastValid,
|
||||
previous_policy_active: true,
|
||||
active_generation: 4,
|
||||
};
|
||||
let [config, _] = policy_validation_failure_events(
|
||||
&retained,
|
||||
8,
|
||||
"sha256:test",
|
||||
"conflicting tls metadata",
|
||||
);
|
||||
let config = config.to_json().unwrap();
|
||||
assert_eq!(config["unmapped"]["previous_policy_active"], true);
|
||||
assert!(
|
||||
config["message"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.contains("previous policy IS active")
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -135,3 +135,93 @@ async fn handle_connection<H: MetadataHandler>(
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
use tokio::sync::mpsc;
|
||||
|
||||
struct RecordingHandler {
|
||||
requests: mpsc::UnboundedSender<(String, String)>,
|
||||
}
|
||||
|
||||
impl MetadataHandler for RecordingHandler {
|
||||
async fn handle<S: AsyncRead + AsyncWrite + Unpin + Send>(
|
||||
&self,
|
||||
method: &str,
|
||||
path: &str,
|
||||
_request: &[u8],
|
||||
stream: &mut S,
|
||||
) -> Result<()> {
|
||||
self.requests
|
||||
.send((method.to_string(), path.to_string()))
|
||||
.unwrap();
|
||||
stream
|
||||
.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok")
|
||||
.await
|
||||
.map_err(|error| miette::miette!("{error}"))?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
async fn connection_pair() -> (tokio::net::TcpStream, tokio::net::TcpStream) {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let client = tokio::net::TcpStream::connect(listener.local_addr().unwrap())
|
||||
.await
|
||||
.unwrap();
|
||||
let (server, _) = listener.accept().await.unwrap();
|
||||
(client, server)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn metadata_loopback_dispatches_method_path_and_response() {
|
||||
let (requests_tx, mut requests_rx) = mpsc::unbounded_channel();
|
||||
let handler = RecordingHandler {
|
||||
requests: requests_tx,
|
||||
};
|
||||
let (mut client, server) = connection_pair().await;
|
||||
let server_task = tokio::spawn(async move { handle_connection(&handler, server).await });
|
||||
|
||||
client
|
||||
.write_all(b"GET /computeMetadata/v1/instance HTTP/1.1\r\nHost: metadata\r\n\r\n")
|
||||
.await
|
||||
.unwrap();
|
||||
let mut response = Vec::new();
|
||||
client.read_to_end(&mut response).await.unwrap();
|
||||
server_task.await.unwrap().unwrap();
|
||||
|
||||
assert_eq!(
|
||||
requests_rx.try_recv().unwrap(),
|
||||
(
|
||||
"GET".to_string(),
|
||||
"/computeMetadata/v1/instance".to_string()
|
||||
)
|
||||
);
|
||||
assert_eq!(response, b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn metadata_loopback_rejects_oversized_headers_before_handler() {
|
||||
let (requests_tx, mut requests_rx) = mpsc::unbounded_channel();
|
||||
let handler = RecordingHandler {
|
||||
requests: requests_tx,
|
||||
};
|
||||
let (mut client, server) = connection_pair().await;
|
||||
let server_task = tokio::spawn(async move { handle_connection(&handler, server).await });
|
||||
|
||||
client
|
||||
.write_all(&vec![b'x'; MAX_REQUEST_BYTES])
|
||||
.await
|
||||
.unwrap();
|
||||
let mut response = Vec::new();
|
||||
client.read_to_end(&mut response).await.unwrap();
|
||||
server_task.await.unwrap().unwrap();
|
||||
|
||||
assert_eq!(
|
||||
response,
|
||||
b"HTTP/1.1 413 Request Entity Too Large\r\nContent-Length: 0\r\n\r\n"
|
||||
);
|
||||
assert!(requests_rx.try_recv().is_err());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -404,6 +404,13 @@ fn prepare_server_config(args: &mut RunArgs, matches: &ArgMatches) -> Result<Ser
|
||||
config = config.with_ssh_session_ttl_secs(ttl);
|
||||
}
|
||||
|
||||
if let Some(mode) = file
|
||||
.as_ref()
|
||||
.and_then(|f| f.openshell.gateway.policy_validation_failure_mode)
|
||||
{
|
||||
config.policy_validation_failure_mode = mode;
|
||||
}
|
||||
|
||||
if let Some(issuer) = args.oidc_issuer.clone() {
|
||||
config = config.with_oidc(openshell_core::OidcConfig {
|
||||
issuer,
|
||||
@@ -1765,6 +1772,9 @@ enable_loopback_service_http = false
|
||||
std::fs::write(
|
||||
&config_path,
|
||||
r#"
|
||||
[openshell.gateway]
|
||||
policy_validation_failure_mode = "retain_last_valid"
|
||||
|
||||
[openshell.drivers.docker]
|
||||
unknown_docker_key = true
|
||||
|
||||
@@ -1789,6 +1799,10 @@ mem_mib = "not-a-number"
|
||||
super::prepare_server_config(&mut args, &matches).expect("server config is prepared");
|
||||
|
||||
assert_eq!(prepared.config.compute_drivers, vec!["podman".to_string()]);
|
||||
assert_eq!(
|
||||
prepared.config.policy_validation_failure_mode,
|
||||
openshell_core::PolicyValidationFailureMode::RetainLastValid
|
||||
);
|
||||
let file = prepared.config_file.expect("config file is preserved");
|
||||
assert!(file.openshell.drivers.contains_key("docker"));
|
||||
assert!(file.openshell.drivers.contains_key("vm"));
|
||||
|
||||
@@ -105,6 +105,9 @@ pub struct GatewayFileSection {
|
||||
pub grpc_rate_limit_requests: Option<u64>,
|
||||
#[serde(default)]
|
||||
pub grpc_rate_limit_window_seconds: Option<u64>,
|
||||
/// Security posture when a sandbox rejects a candidate policy generation.
|
||||
#[serde(default)]
|
||||
pub policy_validation_failure_mode: Option<openshell_core::PolicyValidationFailureMode>,
|
||||
|
||||
// ── Service routing ──────────────────────────────────────────────────
|
||||
/// Subject Alternative Names configured on the gateway server certificate.
|
||||
@@ -422,6 +425,7 @@ compute_drivers = ["kubernetes"]
|
||||
sandbox_namespace = "agents"
|
||||
grpc_rate_limit_requests = 120
|
||||
grpc_rate_limit_window_seconds = 60
|
||||
policy_validation_failure_mode = "retain_last_valid"
|
||||
default_image = "ghcr.io/nvidia/openshell/sandbox:latest"
|
||||
supervisor_image = "ghcr.io/nvidia/openshell/supervisor:latest"
|
||||
client_tls_secret_name = "openshell-sandbox-tls"
|
||||
@@ -450,6 +454,10 @@ grpc_endpoint = "https://openshell-gateway.agents.svc:8080"
|
||||
);
|
||||
assert_eq!(gw.grpc_rate_limit_requests, Some(120));
|
||||
assert_eq!(gw.grpc_rate_limit_window_seconds, Some(60));
|
||||
assert_eq!(
|
||||
gw.policy_validation_failure_mode,
|
||||
Some(openshell_core::PolicyValidationFailureMode::RetainLastValid)
|
||||
);
|
||||
assert!(gw.tls.is_some());
|
||||
assert!(gw.oidc.is_some());
|
||||
assert!(file.openshell.drivers.contains_key("kubernetes"));
|
||||
@@ -513,6 +521,18 @@ sampler = "traceidratio"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_unknown_policy_validation_failure_mode() {
|
||||
let tmp = write_tmp(
|
||||
r#"
|
||||
[openshell.gateway]
|
||||
policy_validation_failure_mode = "keep_old"
|
||||
"#,
|
||||
);
|
||||
let error = load(tmp.path()).expect_err("unknown posture must fail TOML validation");
|
||||
assert!(error.to_string().contains("policy_validation_failure_mode"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_gateway_auth_config() {
|
||||
let toml = r"
|
||||
|
||||
@@ -987,9 +987,27 @@ async fn auto_approve_chunk(
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let (version, hash) =
|
||||
merge_chunk_into_policy(state.store.as_ref(), sandbox_id, context.workspace, &chunk)
|
||||
.await?;
|
||||
let provider_names = context
|
||||
.sandbox
|
||||
.spec
|
||||
.as_ref()
|
||||
.map(|spec| spec.providers.as_slice())
|
||||
.unwrap_or_default();
|
||||
let provider_layers = provider_policy_layers_for_sandbox(
|
||||
state,
|
||||
context.workspace,
|
||||
context.sandbox,
|
||||
provider_names,
|
||||
)
|
||||
.await?;
|
||||
let (version, hash) = merge_chunk_into_policy(
|
||||
state.store.as_ref(),
|
||||
sandbox_id,
|
||||
context.workspace,
|
||||
&chunk,
|
||||
&provider_layers,
|
||||
)
|
||||
.await?;
|
||||
let chunk_summary = summarize_draft_chunk_rule(&chunk)?;
|
||||
|
||||
let now_ms = current_time_ms();
|
||||
@@ -1091,6 +1109,168 @@ async fn current_effective_policy_for_sandbox(
|
||||
Ok(policy)
|
||||
}
|
||||
|
||||
fn validate_endpoint_ambiguities(policy: &ProtoSandboxPolicy) -> Result<(), Status> {
|
||||
let ambiguities = openshell_policy::find_endpoint_ambiguities(policy);
|
||||
if ambiguities.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
Err(Status::failed_precondition(format!(
|
||||
"network endpoint ambiguity validation failed:\n{}",
|
||||
ambiguities
|
||||
.iter()
|
||||
.map(ToString::to_string)
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n")
|
||||
)))
|
||||
}
|
||||
|
||||
pub(super) fn validate_candidate_effective_policy(
|
||||
base_policy: &ProtoSandboxPolicy,
|
||||
provider_layers: &[ProviderPolicyLayer],
|
||||
) -> Result<(), Status> {
|
||||
let effective_policy = if provider_layers.is_empty() {
|
||||
base_policy.clone()
|
||||
} else {
|
||||
compose_effective_policy(base_policy, provider_layers)
|
||||
};
|
||||
validate_endpoint_ambiguities(&effective_policy)
|
||||
}
|
||||
|
||||
async fn provider_policy_layers_for_sandbox(
|
||||
state: &ServerState,
|
||||
workspace: &str,
|
||||
sandbox: &Sandbox,
|
||||
provider_names: &[String],
|
||||
) -> Result<Vec<ProviderPolicyLayer>, Status> {
|
||||
let global_settings = load_global_settings(state.store.as_ref()).await?;
|
||||
if decode_policy_from_global_settings(&global_settings)?.is_some()
|
||||
|| !bool_setting_enabled(&global_settings, settings::PROVIDERS_V2_ENABLED_KEY)?
|
||||
{
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let catalog = state
|
||||
.provider_profile_sources
|
||||
.snapshot_catalog(state.store.as_ref(), workspace)
|
||||
.await?;
|
||||
let layers = profile_provider_policy_layers_with_catalog(
|
||||
state.store.as_ref(),
|
||||
&catalog,
|
||||
workspace,
|
||||
provider_names,
|
||||
)
|
||||
.await?;
|
||||
debug!(
|
||||
sandbox_id = %sandbox.object_id(),
|
||||
provider_layer_count = layers.len(),
|
||||
"Composed candidate provider policy layers for ambiguity validation"
|
||||
);
|
||||
Ok(layers)
|
||||
}
|
||||
|
||||
pub(super) async fn current_base_policy_for_sandbox(
|
||||
store: &Store,
|
||||
sandbox: &Sandbox,
|
||||
) -> Result<ProtoSandboxPolicy, Status> {
|
||||
if let Some(record) = store
|
||||
.get_latest_policy(sandbox.object_id())
|
||||
.await
|
||||
.map_err(|e| Status::internal(format!("fetch latest policy failed: {e}")))?
|
||||
{
|
||||
return ProtoSandboxPolicy::decode(record.policy_payload.as_slice())
|
||||
.map_err(|e| Status::internal(format!("decode current policy failed: {e}")));
|
||||
}
|
||||
Ok(sandbox
|
||||
.spec
|
||||
.as_ref()
|
||||
.and_then(|spec| spec.policy.clone())
|
||||
.unwrap_or_default())
|
||||
}
|
||||
|
||||
pub(super) async fn validate_candidate_provider_attachments(
|
||||
state: &ServerState,
|
||||
workspace: &str,
|
||||
sandbox: &Sandbox,
|
||||
provider_names: &[String],
|
||||
) -> Result<(), Status> {
|
||||
let base_policy = current_base_policy_for_sandbox(state.store.as_ref(), sandbox).await?;
|
||||
let provider_layers =
|
||||
provider_policy_layers_for_sandbox(state, workspace, sandbox, provider_names).await?;
|
||||
validate_candidate_effective_policy(&base_policy, &provider_layers)
|
||||
}
|
||||
|
||||
pub(super) async fn provider_policy_composition_enabled(store: &Store) -> Result<bool, Status> {
|
||||
let global_settings = load_global_settings(store).await?;
|
||||
provider_policy_composition_enabled_in(&global_settings)
|
||||
}
|
||||
|
||||
fn provider_policy_composition_enabled_in(settings: &StoredSettings) -> Result<bool, Status> {
|
||||
Ok(decode_policy_from_global_settings(settings)?.is_none()
|
||||
&& bool_setting_enabled(settings, settings::PROVIDERS_V2_ENABLED_KEY)?)
|
||||
}
|
||||
|
||||
async fn validate_provider_composition_for_existing_sandboxes(
|
||||
state: &ServerState,
|
||||
) -> Result<(), Status> {
|
||||
let mut offset = 0;
|
||||
let mut catalogs = HashMap::<String, EffectiveProviderProfileCatalog>::new();
|
||||
|
||||
loop {
|
||||
let sandboxes = state
|
||||
.store
|
||||
.list_all_messages::<Sandbox>(MAX_PAGE_SIZE, offset)
|
||||
.await
|
||||
.map_err(|e| Status::internal(format!("list sandboxes failed: {e}")))?;
|
||||
let page_len = sandboxes.len();
|
||||
|
||||
for sandbox in sandboxes {
|
||||
let provider_names = sandbox
|
||||
.spec
|
||||
.as_ref()
|
||||
.map(|spec| spec.providers.as_slice())
|
||||
.unwrap_or_default();
|
||||
if provider_names.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let workspace = sandbox.object_workspace().to_string();
|
||||
if !catalogs.contains_key(&workspace) {
|
||||
let catalog = state
|
||||
.provider_profile_sources
|
||||
.snapshot_catalog(state.store.as_ref(), &workspace)
|
||||
.await?;
|
||||
catalogs.insert(workspace.clone(), catalog);
|
||||
}
|
||||
let catalog = catalogs
|
||||
.get(&workspace)
|
||||
.expect("catalog was inserted for sandbox workspace");
|
||||
let base_policy =
|
||||
current_base_policy_for_sandbox(state.store.as_ref(), &sandbox).await?;
|
||||
let provider_layers = profile_provider_policy_layers_with_catalog(
|
||||
state.store.as_ref(),
|
||||
catalog,
|
||||
&workspace,
|
||||
provider_names,
|
||||
)
|
||||
.await?;
|
||||
validate_candidate_effective_policy(&base_policy, &provider_layers).map_err(|error| {
|
||||
Status::failed_precondition(format!(
|
||||
"cannot activate provider policy composition: sandbox '{}/{}' has an invalid effective policy: {}",
|
||||
workspace,
|
||||
sandbox.object_name(),
|
||||
error.message()
|
||||
))
|
||||
})?;
|
||||
}
|
||||
|
||||
if page_len < MAX_PAGE_SIZE as usize {
|
||||
break;
|
||||
}
|
||||
offset = offset.saturating_add(MAX_PAGE_SIZE);
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn truncate_for_log(input: &str, max_chars: usize) -> String {
|
||||
let mut chars = input.chars();
|
||||
let truncated: String = chars.by_ref().take(max_chars).collect();
|
||||
@@ -1423,11 +1603,12 @@ pub(super) async fn handle_get_sandbox_config(
|
||||
let settings = merge_effective_settings(&global_settings, &sandbox_settings)?;
|
||||
let supervisor_middleware_services =
|
||||
state.middleware_registry.required_services(policy.as_ref());
|
||||
let config_revision = compute_config_revision(
|
||||
let config_revision = compute_config_revision_with_validation_mode(
|
||||
policy.as_ref(),
|
||||
&settings,
|
||||
policy_source,
|
||||
&supervisor_middleware_services,
|
||||
state.config.policy_validation_failure_mode,
|
||||
);
|
||||
let provider_env_revision = compute_provider_env_revision_with_catalog(
|
||||
state.store.as_ref(),
|
||||
@@ -1448,6 +1629,11 @@ pub(super) async fn handle_get_sandbox_config(
|
||||
provider_env_revision,
|
||||
supervisor_middleware_services,
|
||||
workspace,
|
||||
policy_validation_failure_mode: state
|
||||
.config
|
||||
.policy_validation_failure_mode
|
||||
.as_str()
|
||||
.to_string(),
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -1731,7 +1917,6 @@ async fn handle_update_config_inner(
|
||||
"one of policy, setting_key, or merge_operations must be provided",
|
||||
));
|
||||
}
|
||||
|
||||
if req.global {
|
||||
if !req.annotations.is_empty() {
|
||||
return Err(Status::invalid_argument(
|
||||
@@ -1760,6 +1945,7 @@ async fn handle_update_config_inner(
|
||||
validate_policy_safety(&new_policy)?;
|
||||
crate::middleware::validate_policy(state.middleware_registry.as_ref(), &new_policy)
|
||||
.await?;
|
||||
validate_candidate_effective_policy(&new_policy, &[])?;
|
||||
|
||||
let payload = new_policy.encode_to_vec();
|
||||
let hash = deterministic_policy_hash(&new_policy);
|
||||
@@ -1862,9 +2048,30 @@ async fn handle_update_config_inner(
|
||||
}
|
||||
|
||||
let mut global_settings = load_global_settings(state.store.as_ref()).await?;
|
||||
let provider_composition_was_enabled =
|
||||
provider_policy_composition_enabled_in(&global_settings)?;
|
||||
let changed = if req.delete_setting {
|
||||
let removed = global_settings.settings.remove(key).is_some();
|
||||
if removed
|
||||
global_settings.settings.remove(key).is_some()
|
||||
} else {
|
||||
let setting = req
|
||||
.setting_value
|
||||
.as_ref()
|
||||
.ok_or_else(|| Status::invalid_argument("setting_value is required"))?;
|
||||
let stored = proto_setting_to_stored(key, setting)?;
|
||||
upsert_setting_value(&mut global_settings.settings, key, stored)
|
||||
};
|
||||
|
||||
if changed {
|
||||
let provider_composition_is_enabled =
|
||||
provider_policy_composition_enabled_in(&global_settings)?;
|
||||
if !provider_composition_was_enabled && provider_composition_is_enabled {
|
||||
validate_provider_composition_for_existing_sandboxes(state).await?;
|
||||
}
|
||||
|
||||
global_settings.revision = global_settings.revision.wrapping_add(1);
|
||||
save_global_settings(state.store.as_ref(), &global_settings).await?;
|
||||
|
||||
if req.delete_setting
|
||||
&& key == POLICY_SETTING_KEY
|
||||
&& let Ok(Some(latest)) = state
|
||||
.store
|
||||
@@ -1876,19 +2083,6 @@ async fn handle_update_config_inner(
|
||||
.supersede_older_policies(GLOBAL_POLICY_SANDBOX_ID, latest.version + 1)
|
||||
.await;
|
||||
}
|
||||
removed
|
||||
} else {
|
||||
let setting = req
|
||||
.setting_value
|
||||
.as_ref()
|
||||
.ok_or_else(|| Status::invalid_argument("setting_value is required"))?;
|
||||
let stored = proto_setting_to_stored(key, setting)?;
|
||||
upsert_setting_value(&mut global_settings.settings, key, stored)
|
||||
};
|
||||
|
||||
if changed {
|
||||
global_settings.revision = global_settings.revision.wrapping_add(1);
|
||||
save_global_settings(state.store.as_ref(), &global_settings).await?;
|
||||
}
|
||||
|
||||
return Ok(update_config_response(
|
||||
@@ -2026,6 +2220,9 @@ async fn handle_update_config_inner(
|
||||
.ok_or_else(|| Status::internal("sandbox has no spec"))?;
|
||||
let merge_ops = parse_merge_operations(&req.merge_operations)?;
|
||||
validate_merge_operations_for_server(&merge_ops)?;
|
||||
let provider_layers =
|
||||
provider_policy_layers_for_sandbox(state, &workspace, &sandbox, &spec.providers)
|
||||
.await?;
|
||||
let atomic_context = AtomicPolicyWriteContext {
|
||||
expected_resource_version: req.expected_resource_version,
|
||||
provenance: &req.annotations,
|
||||
@@ -2041,6 +2238,7 @@ async fn handle_update_config_inner(
|
||||
&workspace,
|
||||
baseline_policy.as_ref(),
|
||||
&merge_ops,
|
||||
&provider_layers,
|
||||
Some(&atomic_context),
|
||||
)
|
||||
.await?;
|
||||
@@ -2144,6 +2342,9 @@ async fn handle_update_config_inner(
|
||||
|
||||
validate_policy_safety(&new_policy)?;
|
||||
crate::middleware::validate_policy(state.middleware_registry.as_ref(), &new_policy).await?;
|
||||
let provider_layers =
|
||||
provider_policy_layers_for_sandbox(state, &workspace, &sandbox, &spec.providers).await?;
|
||||
validate_candidate_effective_policy(&new_policy, &provider_layers)?;
|
||||
|
||||
let _sandbox_sync_guard = if backfill_policy.is_some() {
|
||||
Some(state.compute.sandbox_sync_guard().await)
|
||||
@@ -3063,8 +3264,21 @@ async fn handle_approve_draft_chunk_inner(
|
||||
"ApproveDraftChunk: merging rule into active policy"
|
||||
);
|
||||
|
||||
let (version, hash) =
|
||||
merge_chunk_into_policy(state.store.as_ref(), &sandbox_id, &workspace, &chunk).await?;
|
||||
let provider_names = sandbox
|
||||
.spec
|
||||
.as_ref()
|
||||
.map(|spec| spec.providers.as_slice())
|
||||
.unwrap_or_default();
|
||||
let provider_layers =
|
||||
provider_policy_layers_for_sandbox(state, &workspace, &sandbox, provider_names).await?;
|
||||
let (version, hash) = merge_chunk_into_policy(
|
||||
state.store.as_ref(),
|
||||
&sandbox_id,
|
||||
&workspace,
|
||||
&chunk,
|
||||
&provider_layers,
|
||||
)
|
||||
.await?;
|
||||
let chunk_summary = summarize_draft_chunk_rule(&chunk)?;
|
||||
|
||||
let now_ms = current_time_ms();
|
||||
@@ -3277,6 +3491,33 @@ async fn handle_approve_all_draft_chunks_inner(
|
||||
let mut chunks_skipped: u32 = 0;
|
||||
let mut last_version: i64 = 0;
|
||||
let mut last_hash = String::new();
|
||||
let provider_names = sandbox
|
||||
.spec
|
||||
.as_ref()
|
||||
.map(|spec| spec.providers.as_slice())
|
||||
.unwrap_or_default();
|
||||
let provider_layers =
|
||||
provider_policy_layers_for_sandbox(state, &workspace, &sandbox, provider_names).await?;
|
||||
let mut bulk_candidate =
|
||||
current_base_policy_for_sandbox(state.store.as_ref(), &sandbox).await?;
|
||||
for chunk in &pending_chunks {
|
||||
let security_notes = current_draft_chunk_security_notes(chunk)?;
|
||||
if !req.include_security_flagged && !security_notes.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let rule = NetworkPolicyRule::decode(chunk.proposed_rule.as_slice())
|
||||
.map_err(|e| Status::internal(format!("decode proposed_rule failed: {e}")))?;
|
||||
let operations = [PolicyMergeOp::AddRule {
|
||||
rule_name: chunk.rule_name.clone(),
|
||||
rule,
|
||||
}];
|
||||
validate_merge_operations_for_server(&operations)?;
|
||||
bulk_candidate = merge_policy(bulk_candidate, &operations)
|
||||
.map_err(map_policy_merge_error)?
|
||||
.policy;
|
||||
}
|
||||
validate_policy_safety(&bulk_candidate)?;
|
||||
validate_candidate_effective_policy(&bulk_candidate, &provider_layers)?;
|
||||
|
||||
for chunk in &pending_chunks {
|
||||
let security_notes = current_draft_chunk_security_notes(chunk)?;
|
||||
@@ -3301,8 +3542,14 @@ async fn handle_approve_all_draft_chunks_inner(
|
||||
"ApproveAllDraftChunks: merging chunk"
|
||||
);
|
||||
|
||||
let (version, hash) =
|
||||
merge_chunk_into_policy(state.store.as_ref(), &sandbox_id, &workspace, chunk).await?;
|
||||
let (version, hash) = merge_chunk_into_policy(
|
||||
state.store.as_ref(),
|
||||
&sandbox_id,
|
||||
&workspace,
|
||||
chunk,
|
||||
&provider_layers,
|
||||
)
|
||||
.await?;
|
||||
last_version = version;
|
||||
last_hash = hash;
|
||||
let chunk_summary = summarize_draft_chunk_rule(chunk)?;
|
||||
@@ -3692,14 +3939,16 @@ fn deterministic_policy_hash(policy: &ProtoSandboxPolicy) -> String {
|
||||
}
|
||||
|
||||
/// Compute a fingerprint for the effective sandbox configuration.
|
||||
fn compute_config_revision(
|
||||
fn compute_config_revision_with_validation_mode(
|
||||
policy: Option<&ProtoSandboxPolicy>,
|
||||
settings: &HashMap<String, EffectiveSetting>,
|
||||
policy_source: PolicySource,
|
||||
supervisor_middleware_services: &[openshell_core::proto::SupervisorMiddlewareService],
|
||||
policy_validation_failure_mode: openshell_core::PolicyValidationFailureMode,
|
||||
) -> u64 {
|
||||
let mut hasher = Sha256::new();
|
||||
hasher.update((policy_source as i32).to_le_bytes());
|
||||
hasher.update(policy_validation_failure_mode.as_str().as_bytes());
|
||||
if let Some(policy) = policy {
|
||||
hasher.update(deterministic_policy_hash(policy).as_bytes());
|
||||
}
|
||||
@@ -3741,6 +3990,22 @@ fn compute_config_revision(
|
||||
u64::from_le_bytes(bytes)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn compute_config_revision(
|
||||
policy: Option<&ProtoSandboxPolicy>,
|
||||
settings: &HashMap<String, EffectiveSetting>,
|
||||
policy_source: PolicySource,
|
||||
supervisor_middleware_services: &[openshell_core::proto::SupervisorMiddlewareService],
|
||||
) -> u64 {
|
||||
compute_config_revision_with_validation_mode(
|
||||
policy,
|
||||
settings,
|
||||
policy_source,
|
||||
supervisor_middleware_services,
|
||||
openshell_core::PolicyValidationFailureMode::default(),
|
||||
)
|
||||
}
|
||||
|
||||
fn decode_draft_chunk_rule(record: &DraftChunkRecord) -> Result<Option<NetworkPolicyRule>, Status> {
|
||||
if record.proposed_rule.is_empty() {
|
||||
Ok(None)
|
||||
@@ -4156,6 +4421,7 @@ async fn apply_merge_operations_with_retry(
|
||||
workspace: &str,
|
||||
baseline_policy: Option<&ProtoSandboxPolicy>,
|
||||
operations: &[PolicyMergeOp],
|
||||
provider_layers: &[ProviderPolicyLayer],
|
||||
atomic_context: Option<&AtomicPolicyWriteContext<'_>>,
|
||||
) -> Result<(i64, String, Option<Sandbox>), Status> {
|
||||
for attempt in 1..=MERGE_RETRY_LIMIT {
|
||||
@@ -4179,6 +4445,7 @@ async fn apply_merge_operations_with_retry(
|
||||
validate_static_fields_unchanged(baseline_policy, &new_policy)?;
|
||||
}
|
||||
validate_policy_safety(&new_policy)?;
|
||||
validate_candidate_effective_policy(&new_policy, provider_layers)?;
|
||||
|
||||
if let Some(ref current) = latest
|
||||
&& current.policy_hash == hash
|
||||
@@ -4274,6 +4541,7 @@ pub(super) async fn merge_chunk_into_policy(
|
||||
sandbox_id: &str,
|
||||
workspace: &str,
|
||||
chunk: &DraftChunkRecord,
|
||||
provider_layers: &[ProviderPolicyLayer],
|
||||
) -> Result<(i64, String), Status> {
|
||||
let rule = NetworkPolicyRule::decode(chunk.proposed_rule.as_slice())
|
||||
.map_err(|e| Status::internal(format!("decode proposed_rule failed: {e}")))?;
|
||||
@@ -4282,9 +4550,17 @@ pub(super) async fn merge_chunk_into_policy(
|
||||
rule,
|
||||
}];
|
||||
validate_merge_operations_for_server(&operations)?;
|
||||
apply_merge_operations_with_retry(store, sandbox_id, workspace, None, &operations, None)
|
||||
.await
|
||||
.map(|(version, hash, _)| (version, hash))
|
||||
apply_merge_operations_with_retry(
|
||||
store,
|
||||
sandbox_id,
|
||||
workspace,
|
||||
None,
|
||||
&operations,
|
||||
provider_layers,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.map(|(version, hash, _)| (version, hash))
|
||||
}
|
||||
|
||||
async fn remove_chunk_from_policy(
|
||||
@@ -4302,6 +4578,7 @@ async fn remove_chunk_from_policy(
|
||||
rule_name: chunk.rule_name.clone(),
|
||||
binary_path: chunk.binary.clone(),
|
||||
}],
|
||||
&[],
|
||||
None,
|
||||
)
|
||||
.await
|
||||
@@ -5409,6 +5686,14 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
fn test_ambiguous_policy() -> ProtoSandboxPolicy {
|
||||
let mut left = test_policy_with_rule("left", "api.example.com");
|
||||
left.network_policies.get_mut("left").unwrap().endpoints[0].tls = "skip".to_string();
|
||||
let right = test_policy_with_rule("right", "api.example.com");
|
||||
left.network_policies.extend(right.network_policies);
|
||||
left
|
||||
}
|
||||
|
||||
fn test_sandbox(
|
||||
id: &str,
|
||||
name: &str,
|
||||
@@ -6111,6 +6396,171 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn candidate_effective_policy_rejects_provider_endpoint_ambiguity() {
|
||||
let base = test_policy_with_rule("base", "api.example.com");
|
||||
let mut provider_rule = test_policy_with_rule("provider", "api.example.com")
|
||||
.network_policies
|
||||
.remove("provider")
|
||||
.unwrap();
|
||||
provider_rule.endpoints[0].tls = "skip".to_string();
|
||||
let layers = [ProviderPolicyLayer {
|
||||
rule_name: "_provider_test".to_string(),
|
||||
rule: provider_rule,
|
||||
}];
|
||||
|
||||
let error = validate_candidate_effective_policy(&base, &layers)
|
||||
.expect_err("provider composition must reject endpoint ambiguity");
|
||||
assert_eq!(error.code(), Code::FailedPrecondition);
|
||||
assert!(error.message().contains("api.example.com"));
|
||||
assert!(error.message().contains("tls"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_config_rejects_ambiguous_policy_before_persisting_revision() {
|
||||
let state = test_server_state().await;
|
||||
let mut sandbox = test_sandbox(
|
||||
"sb-ambiguous-update",
|
||||
"ambiguous-update",
|
||||
ProtoSandboxPolicy::default(),
|
||||
Vec::new(),
|
||||
);
|
||||
sandbox.spec.as_mut().unwrap().policy = None;
|
||||
state.store.put_message(&sandbox).await.unwrap();
|
||||
|
||||
let error = handle_update_config(
|
||||
&state,
|
||||
with_user(Request::new(UpdateConfigRequest {
|
||||
name: "ambiguous-update".to_string(),
|
||||
workspace: "default".to_string(),
|
||||
policy: Some(test_ambiguous_policy()),
|
||||
..Default::default()
|
||||
})),
|
||||
)
|
||||
.await
|
||||
.expect_err("ambiguous policy must fail before persistence");
|
||||
|
||||
assert_eq!(error.code(), Code::FailedPrecondition);
|
||||
assert!(error.message().contains("ambiguity validation failed"));
|
||||
assert!(
|
||||
state
|
||||
.store
|
||||
.get_latest_policy("sb-ambiguous-update")
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none(),
|
||||
"invalid policy must not leave a revision in history"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn merge_operations_reject_ambiguity_before_persisting_revision() {
|
||||
let state = test_server_state().await;
|
||||
let mut policy = test_ambiguous_policy();
|
||||
policy.network_policies.get_mut("left").unwrap().endpoints[0].path = "/v1/*".to_string();
|
||||
policy.network_policies.get_mut("right").unwrap().endpoints[0].path =
|
||||
"/v1/users".to_string();
|
||||
let operations = policy
|
||||
.network_policies
|
||||
.into_iter()
|
||||
.map(|(rule_name, rule)| PolicyMergeOp::AddRule { rule_name, rule })
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
let error = apply_merge_operations_with_retry(
|
||||
state.store.as_ref(),
|
||||
"sb-ambiguous-merge",
|
||||
"default",
|
||||
None,
|
||||
&operations,
|
||||
&[],
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.expect_err("ambiguous merge must fail before persistence");
|
||||
|
||||
assert_eq!(error.code(), Code::FailedPrecondition);
|
||||
assert!(
|
||||
state
|
||||
.store
|
||||
.get_latest_policy("sb-ambiguous-merge")
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn provider_attachment_preflight_rejects_composed_ambiguity() {
|
||||
use openshell_core::proto::{
|
||||
ProviderProfile, ProviderProfileCategory, StoredProviderProfile,
|
||||
};
|
||||
|
||||
let state = test_server_state().await;
|
||||
enable_providers_v2(&state).await;
|
||||
state
|
||||
.store
|
||||
.put_message(&StoredProviderProfile {
|
||||
metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta {
|
||||
id: "profile-ambiguous".to_string(),
|
||||
name: "ambiguous".to_string(),
|
||||
created_at_ms: 1_000_000,
|
||||
labels: HashMap::new(),
|
||||
resource_version: 0,
|
||||
annotations: HashMap::new(),
|
||||
workspace: "default".to_string(),
|
||||
deletion_timestamp_ms: 0,
|
||||
}),
|
||||
profile: Some(ProviderProfile {
|
||||
id: "ambiguous".to_string(),
|
||||
display_name: "Ambiguous".to_string(),
|
||||
category: ProviderProfileCategory::Other as i32,
|
||||
endpoints: vec![NetworkEndpoint {
|
||||
host: "api.example.com".to_string(),
|
||||
port: 443,
|
||||
tls: "skip".to_string(),
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
}),
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
state
|
||||
.store
|
||||
.put_message(&test_provider("candidate-provider", "ambiguous"))
|
||||
.await
|
||||
.unwrap();
|
||||
let sandbox = test_sandbox(
|
||||
"sb-provider-ambiguity",
|
||||
"provider-ambiguity",
|
||||
test_policy_with_rule("base", "api.example.com"),
|
||||
Vec::new(),
|
||||
);
|
||||
state.store.put_message(&sandbox).await.unwrap();
|
||||
|
||||
let error = super::super::sandbox::handle_attach_sandbox_provider(
|
||||
&state,
|
||||
authed_request(openshell_core::proto::AttachSandboxProviderRequest {
|
||||
sandbox_name: "provider-ambiguity".to_string(),
|
||||
provider_name: "candidate-provider".to_string(),
|
||||
expected_resource_version: 0,
|
||||
workspace: "default".to_string(),
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.expect_err("provider attachment must validate the composed policy");
|
||||
|
||||
assert_eq!(error.code(), Code::FailedPrecondition);
|
||||
assert!(error.message().contains("tls"));
|
||||
let stored = state
|
||||
.store
|
||||
.get_message_by_name::<Sandbox>("default", "provider-ambiguity")
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert!(stored.spec.unwrap().providers.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn sandbox_config_rejects_invalid_provider_composed_policy() {
|
||||
use openshell_core::proto::{
|
||||
@@ -10827,9 +11277,10 @@ mod tests {
|
||||
rejection_reason: String::new(),
|
||||
};
|
||||
|
||||
let (version, _) = merge_chunk_into_policy(&store, &chunk.sandbox_id, "default", &chunk)
|
||||
.await
|
||||
.unwrap();
|
||||
let (version, _) =
|
||||
merge_chunk_into_policy(&store, &chunk.sandbox_id, "default", &chunk, &[])
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(version, 1);
|
||||
|
||||
@@ -10924,7 +11375,7 @@ mod tests {
|
||||
rejection_reason: String::new(),
|
||||
};
|
||||
|
||||
let (version, _) = merge_chunk_into_policy(&store, sandbox_id, "default", &chunk)
|
||||
let (version, _) = merge_chunk_into_policy(&store, sandbox_id, "default", &chunk, &[])
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(version, 2);
|
||||
@@ -11026,7 +11477,7 @@ mod tests {
|
||||
rejection_reason: String::new(),
|
||||
};
|
||||
|
||||
let (version, _) = merge_chunk_into_policy(&store, sandbox_id, "default", &chunk)
|
||||
let (version, _) = merge_chunk_into_policy(&store, sandbox_id, "default", &chunk, &[])
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(version, 2);
|
||||
@@ -11108,9 +11559,23 @@ mod tests {
|
||||
|
||||
let (left, right) = tokio::join!(
|
||||
apply_merge_operations_with_retry(
|
||||
&store, sandbox_id, "default", None, &add_allow, None
|
||||
&store,
|
||||
sandbox_id,
|
||||
"default",
|
||||
None,
|
||||
&add_allow,
|
||||
&[],
|
||||
None
|
||||
),
|
||||
apply_merge_operations_with_retry(
|
||||
&store,
|
||||
sandbox_id,
|
||||
"default",
|
||||
None,
|
||||
&add_deny,
|
||||
&[],
|
||||
None
|
||||
),
|
||||
apply_merge_operations_with_retry(&store, sandbox_id, "default", None, &add_deny, None),
|
||||
);
|
||||
|
||||
let mut versions = vec![left.unwrap().0, right.unwrap().0];
|
||||
@@ -11497,6 +11962,159 @@ mod tests {
|
||||
assert!(err.message().contains("reserved '_provider_' prefix"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_config_global_policy_rejects_ambiguity_before_persisting() {
|
||||
let state = test_server_state().await;
|
||||
|
||||
let error = handle_update_config(
|
||||
&state,
|
||||
with_user(Request::new(UpdateConfigRequest {
|
||||
global: true,
|
||||
policy: Some(test_ambiguous_policy()),
|
||||
..Default::default()
|
||||
})),
|
||||
)
|
||||
.await
|
||||
.expect_err("ambiguous global policy must fail before persistence");
|
||||
|
||||
assert_eq!(error.code(), Code::FailedPrecondition);
|
||||
assert!(
|
||||
state
|
||||
.store
|
||||
.get_latest_policy(GLOBAL_POLICY_SANDBOX_ID)
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
let settings = load_global_settings(state.store.as_ref()).await.unwrap();
|
||||
assert!(!settings.settings.contains_key(POLICY_SETTING_KEY));
|
||||
}
|
||||
|
||||
async fn install_ambiguous_provider_binding(state: &Arc<ServerState>, suffix: &str) {
|
||||
use openshell_core::proto::{
|
||||
ProviderProfile, ProviderProfileCategory, StoredProviderProfile,
|
||||
};
|
||||
|
||||
let profile_name = format!("ambiguous-{suffix}");
|
||||
let provider_name = format!("provider-{suffix}");
|
||||
state
|
||||
.store
|
||||
.put_message(&StoredProviderProfile {
|
||||
metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta {
|
||||
id: format!("profile-{suffix}"),
|
||||
name: profile_name.clone(),
|
||||
created_at_ms: 1_000_000,
|
||||
labels: HashMap::new(),
|
||||
resource_version: 0,
|
||||
annotations: HashMap::new(),
|
||||
workspace: "default".to_string(),
|
||||
deletion_timestamp_ms: 0,
|
||||
}),
|
||||
profile: Some(ProviderProfile {
|
||||
id: profile_name.clone(),
|
||||
display_name: "Ambiguous".to_string(),
|
||||
category: ProviderProfileCategory::Other as i32,
|
||||
endpoints: vec![NetworkEndpoint {
|
||||
host: "api.example.com".to_string(),
|
||||
port: 443,
|
||||
tls: "skip".to_string(),
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
}),
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
state
|
||||
.store
|
||||
.put_message(&test_provider(&provider_name, &profile_name))
|
||||
.await
|
||||
.unwrap();
|
||||
state
|
||||
.store
|
||||
.put_message(&test_sandbox(
|
||||
&format!("sandbox-{suffix}"),
|
||||
&format!("sandbox-{suffix}"),
|
||||
test_policy_with_rule("base", "api.example.com"),
|
||||
vec![provider_name],
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn enabling_provider_composition_rejects_existing_ambiguous_binding() {
|
||||
let state = test_server_state().await;
|
||||
install_ambiguous_provider_binding(&state, "enable").await;
|
||||
|
||||
let error = handle_update_config(
|
||||
&state,
|
||||
with_user(Request::new(UpdateConfigRequest {
|
||||
global: true,
|
||||
setting_key: settings::PROVIDERS_V2_ENABLED_KEY.to_string(),
|
||||
setting_value: Some(SettingValue {
|
||||
value: Some(setting_value::Value::BoolValue(true)),
|
||||
}),
|
||||
..Default::default()
|
||||
})),
|
||||
)
|
||||
.await
|
||||
.expect_err("provider composition must be validated before activation");
|
||||
|
||||
assert_eq!(error.code(), Code::FailedPrecondition);
|
||||
assert!(error.message().contains("sandbox-enable"));
|
||||
assert!(error.message().contains("tls"));
|
||||
let settings = load_global_settings(state.store.as_ref()).await.unwrap();
|
||||
assert!(!bool_setting_enabled(&settings, settings::PROVIDERS_V2_ENABLED_KEY).unwrap());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn deleting_global_policy_rejects_reactivated_ambiguous_provider_binding() {
|
||||
let state = test_server_state().await;
|
||||
install_ambiguous_provider_binding(&state, "delete-policy").await;
|
||||
|
||||
handle_update_config(
|
||||
&state,
|
||||
with_user(Request::new(UpdateConfigRequest {
|
||||
global: true,
|
||||
policy: Some(test_policy_with_rule("global", "global.example.com")),
|
||||
..Default::default()
|
||||
})),
|
||||
)
|
||||
.await
|
||||
.expect("global policy should suppress provider composition");
|
||||
handle_update_config(
|
||||
&state,
|
||||
with_user(Request::new(UpdateConfigRequest {
|
||||
global: true,
|
||||
setting_key: settings::PROVIDERS_V2_ENABLED_KEY.to_string(),
|
||||
setting_value: Some(SettingValue {
|
||||
value: Some(setting_value::Value::BoolValue(true)),
|
||||
}),
|
||||
..Default::default()
|
||||
})),
|
||||
)
|
||||
.await
|
||||
.expect("providers may be enabled while a global policy is active");
|
||||
|
||||
let error = handle_update_config(
|
||||
&state,
|
||||
with_user(Request::new(UpdateConfigRequest {
|
||||
global: true,
|
||||
setting_key: POLICY_SETTING_KEY.to_string(),
|
||||
delete_setting: true,
|
||||
..Default::default()
|
||||
})),
|
||||
)
|
||||
.await
|
||||
.expect_err("global policy deletion must validate reactivated provider composition");
|
||||
|
||||
assert_eq!(error.code(), Code::FailedPrecondition);
|
||||
assert!(error.message().contains("sandbox-delete-policy"));
|
||||
let settings = load_global_settings(state.store.as_ref()).await.unwrap();
|
||||
assert!(settings.settings.contains_key(POLICY_SETTING_KEY));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn merge_effective_settings_global_overrides_sandbox_key() {
|
||||
let global = StoredSettings {
|
||||
@@ -11744,6 +12362,28 @@ mod tests {
|
||||
assert_ne!(rev_a, rev_b);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn config_revision_changes_when_validation_failure_mode_changes() {
|
||||
let policy = ProtoSandboxPolicy::default();
|
||||
let settings = HashMap::new();
|
||||
|
||||
let fail_closed = compute_config_revision_with_validation_mode(
|
||||
Some(&policy),
|
||||
&settings,
|
||||
PolicySource::Sandbox,
|
||||
&[],
|
||||
openshell_core::PolicyValidationFailureMode::FailClosed,
|
||||
);
|
||||
let retain_last_valid = compute_config_revision_with_validation_mode(
|
||||
Some(&policy),
|
||||
&settings,
|
||||
PolicySource::Sandbox,
|
||||
&[],
|
||||
openshell_core::PolicyValidationFailureMode::RetainLastValid,
|
||||
);
|
||||
assert_ne!(fail_closed, retain_last_valid);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn config_revision_changes_when_supervisor_middleware_services_change() {
|
||||
let policy = ProtoSandboxPolicy::default();
|
||||
|
||||
@@ -20,6 +20,7 @@ use openshell_core::proto::{
|
||||
use openshell_core::telemetry::{
|
||||
LifecycleOperation, ProviderProfile as TelemetryProviderProfile, TelemetryOutcome,
|
||||
};
|
||||
use openshell_policy::ProviderPolicyLayer;
|
||||
use prost::Message;
|
||||
use std::collections::HashMap;
|
||||
use tonic::Status;
|
||||
@@ -2193,12 +2194,12 @@ async fn profile_attached_sandbox_diagnostics(
|
||||
profiles: &[(String, ProviderTypeProfile)],
|
||||
operation: &str,
|
||||
) -> Result<Vec<ProfileValidationDiagnostic>, Status> {
|
||||
let mut candidate_profiles = HashMap::<String, (String, ProviderProfile)>::new();
|
||||
let mut candidate_profiles = HashMap::<String, (String, ProviderTypeProfile)>::new();
|
||||
for (source, profile) in profiles {
|
||||
let Some(id) = normalize_profile_id(&profile.id) else {
|
||||
continue;
|
||||
};
|
||||
candidate_profiles.insert(id, (source.clone(), profile.to_proto()));
|
||||
candidate_profiles.insert(id, (source.clone(), profile.clone()));
|
||||
}
|
||||
if candidate_profiles.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
@@ -2226,11 +2227,14 @@ async fn profile_attached_sandbox_diagnostics(
|
||||
.await?
|
||||
};
|
||||
let mut diagnostics = Vec::new();
|
||||
let validate_policy_composition =
|
||||
super::policy::provider_policy_composition_enabled(store).await?;
|
||||
for sandbox in sandboxes {
|
||||
let sandbox_name = sandbox.object_name().to_string();
|
||||
let sandbox_workspace = sandbox.object_workspace().to_string();
|
||||
let spec = sandbox.spec.as_ref().expect("filtered by scan_sandboxes");
|
||||
let mut bindings = Vec::new();
|
||||
let mut provider_layers = Vec::new();
|
||||
let mut imported_profiles_used = Vec::<(String, String)>::new();
|
||||
|
||||
for provider_name in &spec.providers {
|
||||
@@ -2246,21 +2250,41 @@ async fn profile_attached_sandbox_diagnostics(
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
let profile_id =
|
||||
normalize_provider_type(&provider.r#type).unwrap_or(provider.r#type.as_str());
|
||||
let scope_mismatch = (is_platform_scope && !provider.profile_workspace.is_empty())
|
||||
|| (!is_platform_scope && provider.profile_workspace.is_empty());
|
||||
if scope_mismatch {
|
||||
bindings.extend(dynamic_token_grant_bindings_for_provider_with_catalog(
|
||||
catalog, &provider,
|
||||
));
|
||||
if validate_policy_composition
|
||||
&& let Some(profile) = get_provider_type_profile_for_scope(
|
||||
catalog,
|
||||
profile_id,
|
||||
&provider.profile_workspace,
|
||||
)
|
||||
{
|
||||
let rule_name = openshell_policy::provider_rule_name(provider.object_name());
|
||||
provider_layers.push(ProviderPolicyLayer {
|
||||
rule: profile.network_policy_rule(&rule_name),
|
||||
rule_name,
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
let profile_id =
|
||||
normalize_provider_type(&provider.r#type).unwrap_or(provider.r#type.as_str());
|
||||
if let Some((source, profile)) = candidate_profiles.get(profile_id) {
|
||||
bindings.extend(dynamic_token_grant_bindings_for_profile(
|
||||
provider.object_name(),
|
||||
profile,
|
||||
&profile.to_proto(),
|
||||
));
|
||||
if validate_policy_composition {
|
||||
let rule_name = openshell_policy::provider_rule_name(provider.object_name());
|
||||
provider_layers.push(ProviderPolicyLayer {
|
||||
rule: profile.network_policy_rule(&rule_name),
|
||||
rule_name,
|
||||
});
|
||||
}
|
||||
let used = (source.clone(), profile_id.to_string());
|
||||
if !imported_profiles_used.contains(&used) {
|
||||
imported_profiles_used.push(used);
|
||||
@@ -2269,6 +2293,19 @@ async fn profile_attached_sandbox_diagnostics(
|
||||
bindings.extend(dynamic_token_grant_bindings_for_provider_with_catalog(
|
||||
catalog, &provider,
|
||||
));
|
||||
if validate_policy_composition
|
||||
&& let Some(profile) = get_provider_type_profile_for_scope(
|
||||
catalog,
|
||||
profile_id,
|
||||
&provider.profile_workspace,
|
||||
)
|
||||
{
|
||||
let rule_name = openshell_policy::provider_rule_name(provider.object_name());
|
||||
provider_layers.push(ProviderPolicyLayer {
|
||||
rule: profile.network_policy_rule(&rule_name),
|
||||
rule_name,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2289,6 +2326,27 @@ async fn profile_attached_sandbox_diagnostics(
|
||||
});
|
||||
}
|
||||
}
|
||||
if validate_policy_composition {
|
||||
let base_policy =
|
||||
super::policy::current_base_policy_for_sandbox(store, &sandbox).await?;
|
||||
if let Err(error) =
|
||||
super::policy::validate_candidate_effective_policy(&base_policy, &provider_layers)
|
||||
{
|
||||
for (source, profile_id) in &imported_profiles_used {
|
||||
diagnostics.push(ProfileValidationDiagnostic {
|
||||
source: source.clone(),
|
||||
profile_id: profile_id.clone(),
|
||||
field: "endpoints".to_string(),
|
||||
message: format!(
|
||||
"{operation} would create ambiguous network endpoints on sandbox \
|
||||
'{sandbox_name}': {}",
|
||||
error.message()
|
||||
),
|
||||
severity: "error".to_string(),
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(diagnostics)
|
||||
@@ -3063,11 +3121,11 @@ mod tests {
|
||||
GetProviderProfileRequest, GetProviderRefreshStatusRequest, GetProviderRequest,
|
||||
ImportProviderProfilesRequest, L7Allow, L7Rule, LintProviderProfilesRequest,
|
||||
ListProviderProfilesRequest, ListProvidersRequest, NetworkBinary, NetworkEndpoint,
|
||||
ProviderCredentialRefresh, ProviderCredentialRefreshMaterial, ProviderCredentialTokenGrant,
|
||||
ProviderCredentialTokenGrantAudienceOverride, ProviderProfile, ProviderProfileCategory,
|
||||
ProviderProfileCredential, ProviderProfileImportItem, RotateProviderCredentialRequest,
|
||||
Sandbox, SandboxSpec, StoredProviderProfile, UpdateProviderProfilesRequest,
|
||||
UpdateProviderRequest,
|
||||
NetworkPolicyRule, ProviderCredentialRefresh, ProviderCredentialRefreshMaterial,
|
||||
ProviderCredentialTokenGrant, ProviderCredentialTokenGrantAudienceOverride,
|
||||
ProviderProfile, ProviderProfileCategory, ProviderProfileCredential,
|
||||
ProviderProfileImportItem, RotateProviderCredentialRequest, Sandbox, SandboxPolicy,
|
||||
SandboxSpec, StoredProviderProfile, UpdateProviderProfilesRequest, UpdateProviderRequest,
|
||||
};
|
||||
use openshell_core::{ObjectId, ObjectName};
|
||||
use tonic::{Code, Request};
|
||||
@@ -4015,6 +4073,135 @@ mod tests {
|
||||
assert_eq!(fetched.id, "custom-api");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn profile_update_rejects_fanout_endpoint_ambiguity_without_persisting() {
|
||||
let state = test_server_state().await;
|
||||
crate::grpc::policy::save_global_settings(
|
||||
state.store.as_ref(),
|
||||
&crate::grpc::StoredSettings {
|
||||
revision: 1,
|
||||
settings: std::iter::once((
|
||||
openshell_core::settings::PROVIDERS_V2_ENABLED_KEY.to_string(),
|
||||
crate::grpc::StoredSettingValue::Bool(true),
|
||||
))
|
||||
.collect(),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let mut initial_profile = custom_profile("fanout-ambiguity");
|
||||
initial_profile.endpoints.push(NetworkEndpoint {
|
||||
host: "other.example.com".to_string(),
|
||||
port: 443,
|
||||
..Default::default()
|
||||
});
|
||||
let imported = handle_import_provider_profiles(
|
||||
&state,
|
||||
authed_request(ImportProviderProfilesRequest {
|
||||
profiles: vec![ProviderProfileImportItem {
|
||||
profile: Some(initial_profile),
|
||||
source: "fanout.yaml".to_string(),
|
||||
}],
|
||||
workspace: "default".to_string(),
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap()
|
||||
.into_inner();
|
||||
assert!(imported.imported);
|
||||
let resource_version = imported.profiles[0].resource_version;
|
||||
|
||||
create_provider_record(
|
||||
state.store.as_ref(),
|
||||
"default",
|
||||
provider_with_values("fanout-provider", "fanout-ambiguity"),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
state
|
||||
.store
|
||||
.put_message(&Sandbox {
|
||||
metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta {
|
||||
id: "fanout-sandbox-id".to_string(),
|
||||
name: "fanout-sandbox".to_string(),
|
||||
created_at_ms: 0,
|
||||
labels: HashMap::new(),
|
||||
resource_version: 0,
|
||||
annotations: HashMap::new(),
|
||||
workspace: "default".to_string(),
|
||||
deletion_timestamp_ms: 0,
|
||||
}),
|
||||
spec: Some(SandboxSpec {
|
||||
providers: vec!["fanout-provider".to_string()],
|
||||
policy: Some(SandboxPolicy {
|
||||
network_policies: HashMap::from([(
|
||||
"base".to_string(),
|
||||
NetworkPolicyRule {
|
||||
name: "base".to_string(),
|
||||
endpoints: vec![NetworkEndpoint {
|
||||
host: "api.example.com".to_string(),
|
||||
port: 443,
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
},
|
||||
)]),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let mut conflicting_profile = custom_profile("fanout-ambiguity");
|
||||
conflicting_profile.resource_version = resource_version;
|
||||
conflicting_profile.endpoints.push(NetworkEndpoint {
|
||||
host: "api.example.com".to_string(),
|
||||
port: 443,
|
||||
tls: "skip".to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
let response = handle_update_provider_profiles(
|
||||
&state,
|
||||
authed_request(UpdateProviderProfilesRequest {
|
||||
profile: Some(ProviderProfileImportItem {
|
||||
profile: Some(conflicting_profile),
|
||||
source: "fanout.yaml".to_string(),
|
||||
}),
|
||||
expected_resource_version: resource_version,
|
||||
id: "fanout-ambiguity".to_string(),
|
||||
workspace: "default".to_string(),
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap()
|
||||
.into_inner();
|
||||
|
||||
assert!(!response.updated);
|
||||
assert!(response.diagnostics.iter().any(|diagnostic| {
|
||||
diagnostic.field == "endpoints"
|
||||
&& diagnostic.message.contains("fanout-sandbox")
|
||||
&& diagnostic.message.contains("tls")
|
||||
}));
|
||||
let stored = handle_get_provider_profile(
|
||||
&state,
|
||||
authed_request(GetProviderProfileRequest {
|
||||
id: "fanout-ambiguity".to_string(),
|
||||
workspace: "default".to_string(),
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap()
|
||||
.into_inner()
|
||||
.profile
|
||||
.unwrap();
|
||||
assert_eq!(stored.endpoints[0].host, "other.example.com");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn import_provider_profile_rejects_builtin_overwrite() {
|
||||
let state = test_server_state().await;
|
||||
|
||||
@@ -504,6 +504,13 @@ pub(super) async fn handle_attach_sandbox_provider(
|
||||
&candidate_spec.providers,
|
||||
)
|
||||
.await?;
|
||||
super::policy::validate_candidate_provider_attachments(
|
||||
state,
|
||||
&workspace,
|
||||
&sandbox,
|
||||
&candidate_spec.providers,
|
||||
)
|
||||
.await?;
|
||||
|
||||
let provider_name = request.provider_name.clone();
|
||||
let attached = Arc::new(AtomicBool::new(false));
|
||||
|
||||
@@ -20,3 +20,63 @@ pub mod sigv4;
|
||||
mod spiffe_endpoint;
|
||||
mod token_grant;
|
||||
pub mod upstream_proxy;
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) mod test_alloc {
|
||||
use std::alloc::{GlobalAlloc, Layout, System};
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
struct CountingAllocator;
|
||||
|
||||
static ALLOCATIONS: AtomicU64 = AtomicU64::new(0);
|
||||
static ALLOCATED_BYTES: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
#[allow(unsafe_code)]
|
||||
unsafe impl GlobalAlloc for CountingAllocator {
|
||||
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
|
||||
let pointer = unsafe { System.alloc(layout) };
|
||||
if !pointer.is_null() {
|
||||
ALLOCATIONS.fetch_add(1, Ordering::Relaxed);
|
||||
ALLOCATED_BYTES.fetch_add(layout.size() as u64, Ordering::Relaxed);
|
||||
}
|
||||
pointer
|
||||
}
|
||||
|
||||
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
|
||||
let pointer = unsafe { System.alloc_zeroed(layout) };
|
||||
if !pointer.is_null() {
|
||||
ALLOCATIONS.fetch_add(1, Ordering::Relaxed);
|
||||
ALLOCATED_BYTES.fetch_add(layout.size() as u64, Ordering::Relaxed);
|
||||
}
|
||||
pointer
|
||||
}
|
||||
|
||||
unsafe fn dealloc(&self, pointer: *mut u8, layout: Layout) {
|
||||
unsafe { System.dealloc(pointer, layout) };
|
||||
}
|
||||
|
||||
unsafe fn realloc(&self, pointer: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
|
||||
let pointer = unsafe { System.realloc(pointer, layout, new_size) };
|
||||
if !pointer.is_null() {
|
||||
ALLOCATIONS.fetch_add(1, Ordering::Relaxed);
|
||||
ALLOCATED_BYTES.fetch_add(new_size as u64, Ordering::Relaxed);
|
||||
}
|
||||
pointer
|
||||
}
|
||||
}
|
||||
|
||||
#[global_allocator]
|
||||
static GLOBAL: CountingAllocator = CountingAllocator;
|
||||
|
||||
pub fn reset() {
|
||||
ALLOCATIONS.store(0, Ordering::SeqCst);
|
||||
ALLOCATED_BYTES.store(0, Ordering::SeqCst);
|
||||
}
|
||||
|
||||
pub fn snapshot() -> (u64, u64) {
|
||||
(
|
||||
ALLOCATIONS.load(Ordering::SeqCst),
|
||||
ALLOCATED_BYTES.load(Ordering::SeqCst),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -20,6 +20,7 @@ use std::sync::{
|
||||
Arc, Mutex, RwLock,
|
||||
atomic::{AtomicU64, Ordering},
|
||||
};
|
||||
use tokio::sync::watch;
|
||||
use tracing::info;
|
||||
|
||||
/// Baked-in rego rules for OPA policy evaluation.
|
||||
@@ -123,6 +124,26 @@ pub struct OpaEngine {
|
||||
engine: Mutex<regorus::Engine>,
|
||||
generation: Arc<AtomicU64>,
|
||||
middleware_runner: RwLock<ChainRunner>,
|
||||
generation_tx: watch::Sender<u64>,
|
||||
fail_closed_reason: RwLock<Option<String>>,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
static TEST_OPA_QUERY_COUNT: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
#[cfg(test)]
|
||||
fn record_test_opa_query() {
|
||||
TEST_OPA_QUERY_COUNT.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn reset_test_opa_query_count() {
|
||||
TEST_OPA_QUERY_COUNT.store(0, Ordering::SeqCst);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn test_opa_query_count() -> u64 {
|
||||
TEST_OPA_QUERY_COUNT.load(Ordering::SeqCst)
|
||||
}
|
||||
|
||||
/// Generation guard captured when an HTTP tunnel or request path starts.
|
||||
@@ -130,6 +151,7 @@ pub struct OpaEngine {
|
||||
pub struct PolicyGenerationGuard {
|
||||
captured_generation: u64,
|
||||
current_generation: Arc<AtomicU64>,
|
||||
generation_rx: watch::Receiver<u64>,
|
||||
}
|
||||
|
||||
impl PolicyGenerationGuard {
|
||||
@@ -155,6 +177,19 @@ impl PolicyGenerationGuard {
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Wait until the policy generation changes.
|
||||
///
|
||||
/// Relay boundaries use this to close even an idle or raw stream as soon
|
||||
/// as a new generation (including fail-closed quarantine) is published.
|
||||
pub async fn wait_until_stale(&self) {
|
||||
let mut receiver = self.generation_rx.clone();
|
||||
while !self.is_stale() {
|
||||
if receiver.changed().await.is_err() {
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Per-tunnel L7 policy evaluator bound to the engine generation captured when
|
||||
@@ -201,6 +236,33 @@ impl TunnelPolicyEngine {
|
||||
}
|
||||
|
||||
impl OpaEngine {
|
||||
fn with_engine(engine: regorus::Engine) -> Self {
|
||||
let generation = Arc::new(AtomicU64::new(0));
|
||||
let (generation_tx, _) = watch::channel(0);
|
||||
Self {
|
||||
engine: Mutex::new(engine),
|
||||
generation,
|
||||
middleware_runner: RwLock::new(ChainRunner::default()),
|
||||
generation_tx,
|
||||
fail_closed_reason: RwLock::new(None),
|
||||
}
|
||||
}
|
||||
|
||||
fn advance_generation(&self) -> u64 {
|
||||
let generation = self.generation.fetch_add(1, Ordering::AcqRel) + 1;
|
||||
self.generation_tx.send_replace(generation);
|
||||
generation
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn poison_lock_for_test(&self) {
|
||||
let _ = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
|
||||
let _guard = self.engine.lock().expect("test engine lock");
|
||||
panic!("poison OPA engine lock for compatibility fallback test");
|
||||
}));
|
||||
assert!(self.engine.is_poisoned());
|
||||
}
|
||||
|
||||
/// Load policy from a `.rego` rules file and data from a YAML file.
|
||||
///
|
||||
/// Preprocesses the YAML data to expand access presets and validate L7 config.
|
||||
@@ -232,11 +294,7 @@ impl OpaEngine {
|
||||
engine
|
||||
.add_data_json(&data_json)
|
||||
.map_err(|e| miette::miette!("{e}"))?;
|
||||
Ok(Self {
|
||||
engine: Mutex::new(engine),
|
||||
generation: Arc::new(AtomicU64::new(0)),
|
||||
middleware_runner: RwLock::new(ChainRunner::default()),
|
||||
})
|
||||
Ok(Self::with_engine(engine))
|
||||
}
|
||||
|
||||
/// Load policy rules and data from strings (data is YAML).
|
||||
@@ -287,11 +345,7 @@ impl OpaEngine {
|
||||
engine
|
||||
.add_data_json(&data_json)
|
||||
.map_err(|e| miette::miette!("{e}"))?;
|
||||
Ok(Self {
|
||||
engine: Mutex::new(engine),
|
||||
generation: Arc::new(AtomicU64::new(0)),
|
||||
middleware_runner: RwLock::new(ChainRunner::default()),
|
||||
})
|
||||
Ok(Self::with_engine(engine))
|
||||
}
|
||||
|
||||
/// Create OPA engine from a typed proto policy.
|
||||
@@ -325,6 +379,18 @@ impl OpaEngine {
|
||||
entrypoint_pid: u32,
|
||||
require_binary_identity: bool,
|
||||
) -> Result<Self> {
|
||||
let ambiguities = openshell_policy::find_endpoint_ambiguities(proto);
|
||||
if !ambiguities.is_empty() {
|
||||
return Err(miette::miette!(
|
||||
"network endpoint ambiguity validation failed:\n{}",
|
||||
ambiguities
|
||||
.iter()
|
||||
.map(ToString::to_string)
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n")
|
||||
));
|
||||
}
|
||||
|
||||
emit_binary_identity_mode(require_binary_identity, "proto");
|
||||
if let Err(violations) = openshell_policy::validate_sandbox_policy(proto) {
|
||||
let errors = violations
|
||||
@@ -366,11 +432,7 @@ impl OpaEngine {
|
||||
engine
|
||||
.add_data_json(&data_json)
|
||||
.map_err(|e| miette::miette!("{e}"))?;
|
||||
Ok(Self {
|
||||
engine: Mutex::new(engine),
|
||||
generation: Arc::new(AtomicU64::new(0)),
|
||||
middleware_runner: RwLock::new(ChainRunner::default()),
|
||||
})
|
||||
Ok(Self::with_engine(engine))
|
||||
}
|
||||
|
||||
/// Evaluate a network access request against the loaded policy.
|
||||
@@ -386,6 +448,19 @@ impl OpaEngine {
|
||||
.lock()
|
||||
.map_err(|_| miette::miette!("OPA engine lock poisoned"))?;
|
||||
|
||||
let fail_closed_reason = self
|
||||
.fail_closed_reason
|
||||
.read()
|
||||
.map_err(|_| miette::miette!("OPA fail-closed state lock poisoned"))?
|
||||
.clone();
|
||||
if let Some(reason) = fail_closed_reason {
|
||||
return Ok(PolicyDecision {
|
||||
allowed: false,
|
||||
reason,
|
||||
matched_policy: None,
|
||||
});
|
||||
}
|
||||
|
||||
engine
|
||||
.set_input_json(&input_json.to_string())
|
||||
.map_err(|e| miette::miette!("{e}"))?;
|
||||
@@ -429,6 +504,9 @@ impl OpaEngine {
|
||||
&self,
|
||||
input: &NetworkInput,
|
||||
) -> Result<(NetworkAction, u64)> {
|
||||
#[cfg(test)]
|
||||
record_test_opa_query();
|
||||
|
||||
let input_json = network_input_json(input);
|
||||
|
||||
let mut engine = self
|
||||
@@ -437,6 +515,15 @@ impl OpaEngine {
|
||||
.map_err(|_| miette::miette!("OPA engine lock poisoned"))?;
|
||||
let generation = self.current_generation();
|
||||
|
||||
let fail_closed_reason = self
|
||||
.fail_closed_reason
|
||||
.read()
|
||||
.map_err(|_| miette::miette!("OPA fail-closed state lock poisoned"))?
|
||||
.clone();
|
||||
if let Some(reason) = fail_closed_reason {
|
||||
return Ok((NetworkAction::Deny { reason }, generation));
|
||||
}
|
||||
|
||||
engine
|
||||
.set_input_json(&input_json.to_string())
|
||||
.map_err(|e| miette::miette!("{e}"))?;
|
||||
@@ -483,7 +570,11 @@ impl OpaEngine {
|
||||
.lock()
|
||||
.map_err(|_| miette::miette!("OPA engine lock poisoned"))?;
|
||||
*engine = new_engine;
|
||||
self.generation.fetch_add(1, Ordering::AcqRel);
|
||||
*self
|
||||
.fail_closed_reason
|
||||
.write()
|
||||
.map_err(|_| miette::miette!("OPA fail-closed state lock poisoned"))? = None;
|
||||
self.advance_generation();
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -518,7 +609,11 @@ impl OpaEngine {
|
||||
.lock()
|
||||
.map_err(|_| miette::miette!("OPA engine lock poisoned"))?;
|
||||
*engine = new_engine;
|
||||
self.generation.fetch_add(1, Ordering::AcqRel);
|
||||
*self
|
||||
.fail_closed_reason
|
||||
.write()
|
||||
.map_err(|_| miette::miette!("OPA fail-closed state lock poisoned"))? = None;
|
||||
self.advance_generation();
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -553,10 +648,61 @@ impl OpaEngine {
|
||||
.map_err(|_| miette::miette!("middleware runner lock poisoned"))?;
|
||||
*engine = new_engine;
|
||||
*runner = new_runner;
|
||||
self.generation.fetch_add(1, Ordering::AcqRel);
|
||||
*self
|
||||
.fail_closed_reason
|
||||
.write()
|
||||
.map_err(|_| miette::miette!("OPA fail-closed state lock poisoned"))? = None;
|
||||
self.advance_generation();
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Publish a deny-all quarantine generation without activating any part
|
||||
/// of the invalid candidate policy.
|
||||
///
|
||||
/// The existing compiled engine remains available for an explicit
|
||||
/// `retain_last_valid` posture or a later valid reload, but all new network
|
||||
/// decisions deny with `reason` while the quarantine is active. Advancing
|
||||
/// the generation invalidates and wakes every pinned relay.
|
||||
pub fn enter_fail_closed(&self, reason: impl Into<String>) -> Result<u64> {
|
||||
let _engine = self
|
||||
.engine
|
||||
.lock()
|
||||
.map_err(|_| miette::miette!("OPA engine lock poisoned"))?;
|
||||
*self
|
||||
.fail_closed_reason
|
||||
.write()
|
||||
.map_err(|_| miette::miette!("OPA fail-closed state lock poisoned"))? =
|
||||
Some(reason.into());
|
||||
Ok(self.advance_generation())
|
||||
}
|
||||
|
||||
pub fn fail_closed_reason(&self) -> Option<String> {
|
||||
self.fail_closed_reason
|
||||
.read()
|
||||
.ok()
|
||||
.and_then(|reason| reason.clone())
|
||||
}
|
||||
|
||||
/// Reactivate the compiled last-known-good engine after an operator
|
||||
/// explicitly selects the availability-oriented retention posture.
|
||||
pub fn exit_fail_closed(&self) -> Result<u64> {
|
||||
let _engine = self
|
||||
.engine
|
||||
.lock()
|
||||
.map_err(|_| miette::miette!("OPA engine lock poisoned"))?;
|
||||
let was_fail_closed = self
|
||||
.fail_closed_reason
|
||||
.write()
|
||||
.map_err(|_| miette::miette!("OPA fail-closed state lock poisoned"))?
|
||||
.take()
|
||||
.is_some();
|
||||
if was_fail_closed {
|
||||
Ok(self.advance_generation())
|
||||
} else {
|
||||
Ok(self.current_generation())
|
||||
}
|
||||
}
|
||||
|
||||
/// Current policy generation. Successful reloads increment this value.
|
||||
pub fn current_generation(&self) -> u64 {
|
||||
self.generation.load(Ordering::Acquire)
|
||||
@@ -570,7 +716,7 @@ impl OpaEngine {
|
||||
.write()
|
||||
.map_err(|_| miette::miette!("middleware runner lock poisoned"))?;
|
||||
*runner = ChainRunner::from_registry(registry);
|
||||
self.generation.fetch_add(1, Ordering::AcqRel);
|
||||
self.advance_generation();
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -603,6 +749,7 @@ impl OpaEngine {
|
||||
Ok(PolicyGenerationGuard {
|
||||
captured_generation: generation,
|
||||
current_generation: Arc::clone(&self.generation),
|
||||
generation_rx: self.generation_tx.subscribe(),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -665,6 +812,9 @@ impl OpaEngine {
|
||||
&self,
|
||||
input: &NetworkInput,
|
||||
) -> Result<(Vec<regorus::Value>, u64)> {
|
||||
#[cfg(test)]
|
||||
record_test_opa_query();
|
||||
|
||||
let input_json = network_input_json(input);
|
||||
|
||||
let mut engine = self
|
||||
@@ -723,6 +873,9 @@ impl OpaEngine {
|
||||
/// denial while preserving separate handling for `allowed_ips` and advisor
|
||||
/// proposals.
|
||||
pub fn query_exact_declared_endpoint_host(&self, input: &NetworkInput) -> Result<bool> {
|
||||
#[cfg(test)]
|
||||
record_test_opa_query();
|
||||
|
||||
let input_json = network_input_json(input);
|
||||
|
||||
let mut engine = self
|
||||
@@ -762,6 +915,7 @@ impl OpaEngine {
|
||||
generation_guard: PolicyGenerationGuard {
|
||||
captured_generation: generation,
|
||||
current_generation: Arc::clone(&self.generation),
|
||||
generation_rx: self.generation_tx.subscribe(),
|
||||
},
|
||||
middleware_runner: self.middleware_runner()?,
|
||||
})
|
||||
@@ -3043,11 +3197,7 @@ network_policies:
|
||||
.expect("policy should load");
|
||||
rego.add_data_json(&data_json.to_string())
|
||||
.expect("data should load");
|
||||
let engine = OpaEngine {
|
||||
engine: Mutex::new(rego),
|
||||
generation: Arc::new(AtomicU64::new(0)),
|
||||
middleware_runner: RwLock::new(ChainRunner::default()),
|
||||
};
|
||||
let engine = OpaEngine::with_engine(rego);
|
||||
let input = l7_websocket_graphql_input(
|
||||
"realtime.graphql.com",
|
||||
serde_json::json!([{
|
||||
@@ -4676,6 +4826,97 @@ network_policies:
|
||||
assert_eq!(val, regorus::Value::from(true));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn proto_load_rejects_ambiguous_endpoint_metadata_with_rationale() {
|
||||
let mut policy = ProtoSandboxPolicy::default();
|
||||
policy.network_policies.insert(
|
||||
"wildcard".to_string(),
|
||||
NetworkPolicyRule {
|
||||
name: "wildcard".to_string(),
|
||||
endpoints: vec![NetworkEndpoint {
|
||||
host: "*.example.com".to_string(),
|
||||
port: 443,
|
||||
tls: "skip".to_string(),
|
||||
..Default::default()
|
||||
}],
|
||||
binaries: vec![NetworkBinary {
|
||||
path: "/usr/bin/curl".to_string(),
|
||||
..Default::default()
|
||||
}],
|
||||
},
|
||||
);
|
||||
policy.network_policies.insert(
|
||||
"exact".to_string(),
|
||||
NetworkPolicyRule {
|
||||
name: "exact".to_string(),
|
||||
endpoints: vec![NetworkEndpoint {
|
||||
host: "api.example.com".to_string(),
|
||||
port: 443,
|
||||
..Default::default()
|
||||
}],
|
||||
binaries: vec![NetworkBinary {
|
||||
path: "/usr/bin/bash".to_string(),
|
||||
..Default::default()
|
||||
}],
|
||||
},
|
||||
);
|
||||
|
||||
let Err(error) = OpaEngine::from_proto(&policy) else {
|
||||
panic!("ambiguity must reject activation");
|
||||
};
|
||||
let message = error.to_string();
|
||||
assert!(message.contains("ambiguity validation failed"));
|
||||
assert!(message.contains("wildcard"));
|
||||
assert!(message.contains("exact"));
|
||||
assert!(message.contains("tls"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn fail_closed_quarantine_denies_and_wakes_generation_guards() {
|
||||
let engine = test_engine();
|
||||
let guard = engine
|
||||
.generation_guard(engine.current_generation())
|
||||
.unwrap();
|
||||
let stale = guard.wait_until_stale();
|
||||
|
||||
let generation = engine
|
||||
.enter_fail_closed("candidate policy validation failed: conflicting tls")
|
||||
.unwrap();
|
||||
tokio::time::timeout(std::time::Duration::from_secs(1), stale)
|
||||
.await
|
||||
.expect("generation waiter should wake");
|
||||
assert_eq!(generation, 1);
|
||||
assert!(guard.is_stale());
|
||||
|
||||
let input = NetworkInput {
|
||||
host: "api.anthropic.com".to_string(),
|
||||
port: 443,
|
||||
binary_path: PathBuf::from("/usr/bin/curl"),
|
||||
binary_sha256: String::new(),
|
||||
ancestors: vec![],
|
||||
cmdline_paths: vec![],
|
||||
};
|
||||
let action = engine.evaluate_network_action(&input).unwrap();
|
||||
assert_eq!(
|
||||
action,
|
||||
NetworkAction::Deny {
|
||||
reason: "candidate policy validation failed: conflicting tls".to_string()
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn valid_reload_exits_fail_closed_quarantine() {
|
||||
let engine = test_engine();
|
||||
engine.enter_fail_closed("invalid candidate").unwrap();
|
||||
assert!(engine.fail_closed_reason().is_some());
|
||||
|
||||
engine.reload(TEST_POLICY, TEST_DATA_YAML).unwrap();
|
||||
|
||||
assert!(engine.fail_closed_reason().is_none());
|
||||
assert_eq!(engine.current_generation(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn endpoint_config_generation_matches_query_generation() {
|
||||
let engine = l7_engine();
|
||||
@@ -5003,6 +5244,7 @@ network_policies:
|
||||
port: 8567
|
||||
protocol: rest
|
||||
enforcement: enforce
|
||||
tls: skip
|
||||
allowed_ips:
|
||||
- 192.168.1.100
|
||||
rules:
|
||||
@@ -5041,7 +5283,7 @@ process:
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn overlapping_policies_endpoint_config_returns_result() {
|
||||
fn overlapping_policy_outputs_are_snapshotted_independently() {
|
||||
let engine = OpaEngine::from_strings(TEST_POLICY, OVERLAPPING_L7_TEST_DATA)
|
||||
.expect("engine should load overlapping data");
|
||||
let input = NetworkInput {
|
||||
@@ -5052,12 +5294,29 @@ process:
|
||||
ancestors: vec![],
|
||||
cmdline_paths: vec![],
|
||||
};
|
||||
// Should return config from one of the entries without error.
|
||||
let config = engine.query_endpoint_config(&input).unwrap();
|
||||
assert!(
|
||||
config.is_some(),
|
||||
"Expected endpoint config for overlapping policies"
|
||||
assert_eq!(
|
||||
engine.evaluate_network_action(&input).unwrap(),
|
||||
NetworkAction::Allow {
|
||||
matched_policy: Some("allow_192_168_1_100_8567".to_string())
|
||||
}
|
||||
);
|
||||
|
||||
let (configs, generation) = engine
|
||||
.query_endpoint_configs_with_generation(&input)
|
||||
.unwrap();
|
||||
assert_eq!(generation, engine.current_generation());
|
||||
assert_eq!(configs.len(), 2);
|
||||
assert_eq!(get_str(&configs[0], "tls").as_deref(), Some("skip"));
|
||||
assert_eq!(get_str_array(&configs[0], "allowed_ips"), ["192.168.1.100"]);
|
||||
assert_eq!(get_str(&configs[1], "tls"), None);
|
||||
|
||||
let selected = engine.query_endpoint_config(&input).unwrap().unwrap();
|
||||
assert_eq!(
|
||||
crate::l7::parse_tls_mode(&selected),
|
||||
crate::l7::TlsMode::Skip
|
||||
);
|
||||
assert_eq!(engine.query_allowed_ips(&input).unwrap(), ["192.168.1.100"]);
|
||||
assert!(engine.query_exact_declared_endpoint_host(&input).unwrap());
|
||||
}
|
||||
|
||||
// ========================================================================
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,303 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Shared external destination validation and upstream dial boundary.
|
||||
|
||||
use super::{
|
||||
implicit_allowed_ips_for_ip_host, is_host_gateway_alias, parse_allowed_ips,
|
||||
resolve_and_check_allowed_ips, resolve_and_check_declared_endpoint,
|
||||
resolve_and_check_trusted_gateway, resolve_and_reject_internal,
|
||||
};
|
||||
use ipnet::IpNet;
|
||||
use std::net::{IpAddr, SocketAddr};
|
||||
use tokio::net::TcpStream;
|
||||
|
||||
/// Address-validation mode selected from the current endpoint configuration.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub(super) enum AddressAuthorization {
|
||||
DefaultPublicOnly,
|
||||
ExplicitAllowedIps(Vec<IpNet>),
|
||||
ExactDeclaredHost,
|
||||
ImplicitIpLiteral(IpAddr),
|
||||
TrustedGatewayAlias { expected_ip: IpAddr },
|
||||
}
|
||||
|
||||
/// Fully materialized input to shared destination validation.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub(super) struct DestinationValidationPlan {
|
||||
pub(super) address_authorization: AddressAuthorization,
|
||||
}
|
||||
|
||||
/// Inputs needed to apply the current SSRF and endpoint destination policy.
|
||||
pub(super) struct DestinationRequest<'a> {
|
||||
pub(super) host: &'a str,
|
||||
pub(super) port: u16,
|
||||
pub(super) sandbox_entrypoint_pid: u32,
|
||||
pub(super) plan: &'a DestinationValidationPlan,
|
||||
}
|
||||
|
||||
/// Destination-validation branch that rejected an egress request.
|
||||
///
|
||||
/// Adapters use this classification to preserve their existing HTTP response
|
||||
/// and OCSF message shapes while sharing the underlying validation logic.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub(super) enum DestinationDenialKind {
|
||||
TrustedGateway,
|
||||
InvalidAllowedIps,
|
||||
AllowedIps,
|
||||
DeclaredEndpoint,
|
||||
InternalAddress,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub(super) struct DestinationDenial {
|
||||
pub(super) kind: DestinationDenialKind,
|
||||
pub(super) reason: String,
|
||||
}
|
||||
|
||||
impl DestinationDenial {
|
||||
fn new(kind: DestinationDenialKind, reason: String) -> Self {
|
||||
Self { kind, reason }
|
||||
}
|
||||
}
|
||||
|
||||
/// Select one current destination-validation mode without changing precedence.
|
||||
pub(super) fn build_validation_plan(
|
||||
host: &str,
|
||||
normalized_host: &str,
|
||||
trusted_host_gateway: Option<IpAddr>,
|
||||
raw_allowed_ips: &[String],
|
||||
exact_declared_endpoint_host: bool,
|
||||
) -> Result<DestinationValidationPlan, DestinationDenial> {
|
||||
let address_authorization = if is_host_gateway_alias(normalized_host)
|
||||
&& let Some(expected_ip) = trusted_host_gateway
|
||||
{
|
||||
AddressAuthorization::TrustedGatewayAlias { expected_ip }
|
||||
} else if !raw_allowed_ips.is_empty() {
|
||||
AddressAuthorization::ExplicitAllowedIps(parse_allowed_ips(raw_allowed_ips).map_err(
|
||||
|reason| DestinationDenial::new(DestinationDenialKind::InvalidAllowedIps, reason),
|
||||
)?)
|
||||
} else if let Some(ip) = implicit_allowed_ips_for_ip_host(host)
|
||||
.first()
|
||||
.and_then(|raw| raw.parse::<IpAddr>().ok())
|
||||
{
|
||||
AddressAuthorization::ImplicitIpLiteral(ip)
|
||||
} else if exact_declared_endpoint_host {
|
||||
AddressAuthorization::ExactDeclaredHost
|
||||
} else {
|
||||
AddressAuthorization::DefaultPublicOnly
|
||||
};
|
||||
|
||||
Ok(DestinationValidationPlan {
|
||||
address_authorization,
|
||||
})
|
||||
}
|
||||
|
||||
/// Validated, but not yet opened, upstream destination.
|
||||
///
|
||||
/// The explicit proxy adapter controls when `connect` is called so CONNECT and
|
||||
/// forward HTTP retain their current upstream-dial timing during the refactor.
|
||||
pub(super) struct UpstreamConnector {
|
||||
host: String,
|
||||
port: u16,
|
||||
addrs: Vec<SocketAddr>,
|
||||
}
|
||||
|
||||
impl UpstreamConnector {
|
||||
pub(super) fn addrs(&self) -> &[SocketAddr] {
|
||||
&self.addrs
|
||||
}
|
||||
|
||||
pub(super) async fn connect(&self) -> std::io::Result<TcpStream> {
|
||||
tracing::debug!(
|
||||
host = %self.host,
|
||||
port = self.port,
|
||||
address_count = self.addrs.len(),
|
||||
"Opening validated upstream connection"
|
||||
);
|
||||
TcpStream::connect(self.addrs.as_slice()).await
|
||||
}
|
||||
|
||||
fn new(host: &str, port: u16, addrs: Vec<SocketAddr>) -> Self {
|
||||
Self {
|
||||
host: host.to_string(),
|
||||
port,
|
||||
addrs,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve and validate a destination using the existing proxy security rules.
|
||||
pub(super) async fn validate_destination(
|
||||
request: DestinationRequest<'_>,
|
||||
) -> Result<UpstreamConnector, DestinationDenial> {
|
||||
let DestinationRequest {
|
||||
host,
|
||||
port,
|
||||
sandbox_entrypoint_pid,
|
||||
plan,
|
||||
} = request;
|
||||
|
||||
let addrs = match &plan.address_authorization {
|
||||
AddressAuthorization::TrustedGatewayAlias { expected_ip } => {
|
||||
resolve_and_check_trusted_gateway(host, port, *expected_ip, sandbox_entrypoint_pid)
|
||||
.await
|
||||
.map_err(|reason| {
|
||||
DestinationDenial::new(DestinationDenialKind::TrustedGateway, reason)
|
||||
})?
|
||||
}
|
||||
AddressAuthorization::ExplicitAllowedIps(networks) => {
|
||||
resolve_and_check_allowed_ips(host, port, networks, sandbox_entrypoint_pid)
|
||||
.await
|
||||
.map_err(|reason| {
|
||||
DestinationDenial::new(DestinationDenialKind::AllowedIps, reason)
|
||||
})?
|
||||
}
|
||||
AddressAuthorization::ImplicitIpLiteral(ip) => {
|
||||
let network = IpNet::from(*ip);
|
||||
resolve_and_check_allowed_ips(host, port, &[network], sandbox_entrypoint_pid)
|
||||
.await
|
||||
.map_err(|reason| {
|
||||
DestinationDenial::new(DestinationDenialKind::AllowedIps, reason)
|
||||
})?
|
||||
}
|
||||
AddressAuthorization::ExactDeclaredHost => {
|
||||
resolve_and_check_declared_endpoint(host, port, sandbox_entrypoint_pid)
|
||||
.await
|
||||
.map_err(|reason| {
|
||||
DestinationDenial::new(DestinationDenialKind::DeclaredEndpoint, reason)
|
||||
})?
|
||||
}
|
||||
AddressAuthorization::DefaultPublicOnly => {
|
||||
resolve_and_reject_internal(host, port, sandbox_entrypoint_pid)
|
||||
.await
|
||||
.map_err(|reason| {
|
||||
DestinationDenial::new(DestinationDenialKind::InternalAddress, reason)
|
||||
})?
|
||||
}
|
||||
};
|
||||
|
||||
Ok(UpstreamConnector::new(host, port, addrs))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::net::{IpAddr, Ipv4Addr};
|
||||
|
||||
fn request<'a>(host: &'a str, plan: &'a DestinationValidationPlan) -> DestinationRequest<'a> {
|
||||
DestinationRequest {
|
||||
host,
|
||||
port: 80,
|
||||
sandbox_entrypoint_pid: 0,
|
||||
plan,
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn default_mode_classifies_loopback_as_internal_address() {
|
||||
let plan = DestinationValidationPlan {
|
||||
address_authorization: AddressAuthorization::DefaultPublicOnly,
|
||||
};
|
||||
let denial = validate_destination(request("127.0.0.1", &plan))
|
||||
.await
|
||||
.err()
|
||||
.expect("loopback must be denied");
|
||||
|
||||
assert_eq!(denial.kind, DestinationDenialKind::InternalAddress);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn invalid_allowed_ips_has_a_distinct_denial_kind() {
|
||||
let denial = build_validation_plan(
|
||||
"api.example.test",
|
||||
"api.example.test",
|
||||
None,
|
||||
&["not-an-ip".to_string()],
|
||||
false,
|
||||
)
|
||||
.expect_err("invalid allowed_ips must be denied");
|
||||
|
||||
assert_eq!(denial.kind, DestinationDenialKind::InvalidAllowedIps);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn declared_endpoint_preserves_its_denial_classification() {
|
||||
let plan = DestinationValidationPlan {
|
||||
address_authorization: AddressAuthorization::ExactDeclaredHost,
|
||||
};
|
||||
let denial = validate_destination(request("127.0.0.1", &plan))
|
||||
.await
|
||||
.err()
|
||||
.expect("declared loopback must remain denied");
|
||||
|
||||
assert_eq!(denial.kind, DestinationDenialKind::DeclaredEndpoint);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn trusted_gateway_preserves_its_denial_classification() {
|
||||
let plan = DestinationValidationPlan {
|
||||
address_authorization: AddressAuthorization::TrustedGatewayAlias {
|
||||
expected_ip: IpAddr::V4(Ipv4Addr::LOCALHOST),
|
||||
},
|
||||
};
|
||||
let denial = validate_destination(request("host.openshell.internal", &plan))
|
||||
.await
|
||||
.err()
|
||||
.expect("loopback cannot be a trusted gateway");
|
||||
|
||||
assert_eq!(denial.kind, DestinationDenialKind::TrustedGateway);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validation_mode_precedence_is_explicit_and_stable() {
|
||||
let trusted_ip = IpAddr::V4(Ipv4Addr::new(169, 254, 1, 2));
|
||||
let trusted = build_validation_plan(
|
||||
"host.openshell.internal",
|
||||
"host.openshell.internal",
|
||||
Some(trusted_ip),
|
||||
&["10.0.0.0/8".to_string()],
|
||||
true,
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
trusted.address_authorization,
|
||||
AddressAuthorization::TrustedGatewayAlias {
|
||||
expected_ip: trusted_ip
|
||||
}
|
||||
);
|
||||
|
||||
let explicit = build_validation_plan(
|
||||
"10.2.3.4",
|
||||
"10.2.3.4",
|
||||
None,
|
||||
&["10.0.0.0/8".to_string()],
|
||||
true,
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
explicit.address_authorization,
|
||||
AddressAuthorization::ExplicitAllowedIps(vec!["10.0.0.0/8".parse().unwrap()])
|
||||
);
|
||||
|
||||
let implicit = build_validation_plan("10.2.3.4", "10.2.3.4", None, &[], true).unwrap();
|
||||
assert_eq!(
|
||||
implicit.address_authorization,
|
||||
AddressAuthorization::ImplicitIpLiteral("10.2.3.4".parse().unwrap())
|
||||
);
|
||||
|
||||
let declared =
|
||||
build_validation_plan("private.example", "private.example", None, &[], true).unwrap();
|
||||
assert_eq!(
|
||||
declared.address_authorization,
|
||||
AddressAuthorization::ExactDeclaredHost
|
||||
);
|
||||
|
||||
let default =
|
||||
build_validation_plan("*.example.com", "*.example.com", None, &[], false).unwrap();
|
||||
assert_eq!(
|
||||
default.address_authorization,
|
||||
AddressAuthorization::DefaultPublicOnly
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Transport-neutral egress inputs and authorization results.
|
||||
//!
|
||||
//! Explicit proxy adapters normalize their protocol-specific request into an
|
||||
//! [`EgressIntent`]. Authorization then returns an [`EgressDecision`] that is
|
||||
//! consumed by destination validation and relay selection. Keeping these types
|
||||
//! independent of CONNECT and forward HTTP prevents policy behavior from
|
||||
//! drifting as more adapters are added.
|
||||
|
||||
use super::destination::DestinationValidationPlan;
|
||||
use crate::opa::NetworkAction;
|
||||
use std::path::PathBuf;
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub(super) struct L7ConfigSnapshot {
|
||||
pub(super) config: crate::l7::L7EndpointConfig,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub(super) struct L7RouteSnapshot {
|
||||
pub(super) configs: Vec<L7ConfigSnapshot>,
|
||||
/// Policy generation used to materialize this L7 route.
|
||||
pub(super) l7_policy_generation: u64,
|
||||
}
|
||||
|
||||
/// Endpoint metadata materialized for an allowed egress decision.
|
||||
///
|
||||
/// The migration hydrates these fields at the same points the legacy handlers
|
||||
/// queried them so policy-reload and upstream-connect timing remain unchanged.
|
||||
#[derive(Debug, Clone)]
|
||||
pub(super) struct EndpointDecision {
|
||||
pub(super) tls_mode: crate::l7::TlsMode,
|
||||
pub(super) l7_route: Option<L7RouteSnapshot>,
|
||||
/// Destination authorization selected at the legacy hydration point.
|
||||
pub(super) destination: Option<DestinationValidationPlan>,
|
||||
}
|
||||
|
||||
impl Default for EndpointDecision {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
tls_mode: crate::l7::TlsMode::Auto,
|
||||
l7_route: None,
|
||||
destination: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Userland surface through which an external egress request arrived.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub(super) enum EgressTransport {
|
||||
Connect,
|
||||
ForwardHttp,
|
||||
}
|
||||
|
||||
/// Destination requested by an explicit proxy adapter.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub(super) struct RequestedDestination {
|
||||
pub(super) host: String,
|
||||
pub(super) port: u16,
|
||||
}
|
||||
|
||||
/// Transport-neutral description of an external egress request.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub(super) struct EgressIntent {
|
||||
pub(super) transport: EgressTransport,
|
||||
pub(super) destination: RequestedDestination,
|
||||
}
|
||||
|
||||
impl EgressIntent {
|
||||
pub(super) fn connect(host: String, port: u16) -> Self {
|
||||
Self::new(EgressTransport::Connect, host, port)
|
||||
}
|
||||
|
||||
pub(super) fn forward_http(host: String, port: u16) -> Self {
|
||||
Self::new(EgressTransport::ForwardHttp, host, port)
|
||||
}
|
||||
|
||||
fn new(transport: EgressTransport, host: String, port: u16) -> Self {
|
||||
Self {
|
||||
transport,
|
||||
destination: RequestedDestination { host, port },
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Why process identity is absent from an egress decision.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
#[cfg_attr(not(target_os = "linux"), allow(dead_code))]
|
||||
pub(super) enum IdentityUnavailableReason {
|
||||
EndpointOnlyMode,
|
||||
LookupFailed,
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
UnsupportedPlatform,
|
||||
}
|
||||
|
||||
/// Process evidence captured for policy evaluation and audit logging.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
#[cfg_attr(not(target_os = "linux"), allow(dead_code))]
|
||||
pub(super) enum ProcessIdentityEvidence {
|
||||
Available,
|
||||
Unavailable(IdentityUnavailableReason),
|
||||
}
|
||||
|
||||
/// Result of authorizing a normalized egress intent.
|
||||
///
|
||||
/// The identity fields intentionally mirror the former CONNECT-specific
|
||||
/// decision during the compatibility migration. Endpoint configuration is
|
||||
/// hydrated at the legacy query points without changing lookup precedence or
|
||||
/// failure defaults.
|
||||
pub(super) struct EgressDecision {
|
||||
pub(super) intent: EgressIntent,
|
||||
pub(super) action: NetworkAction,
|
||||
/// Policy generation used for the L4 network decision.
|
||||
pub(super) l4_policy_generation: u64,
|
||||
/// Whether process identity evidence was available to policy evaluation.
|
||||
pub(super) identity: ProcessIdentityEvidence,
|
||||
/// Endpoint behavior hydrated for destination validation and relays.
|
||||
pub(super) endpoint: EndpointDecision,
|
||||
/// Resolved binary path.
|
||||
pub(super) binary: Option<PathBuf>,
|
||||
/// PID owning the socket.
|
||||
pub(super) binary_pid: Option<u32>,
|
||||
/// Ancestor binary paths from process tree walk.
|
||||
pub(super) ancestors: Vec<PathBuf>,
|
||||
/// Cmdline-derived absolute paths (for script detection).
|
||||
pub(super) cmdline_paths: Vec<PathBuf>,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn adapters_create_transport_specific_intents() {
|
||||
let connect = EgressIntent::connect("api.example.com".to_string(), 443);
|
||||
let forward = EgressIntent::forward_http("api.example.com".to_string(), 80);
|
||||
|
||||
assert_eq!(connect.transport, EgressTransport::Connect);
|
||||
assert_eq!(connect.destination.host, "api.example.com");
|
||||
assert_eq!(connect.destination.port, 443);
|
||||
assert_eq!(forward.transport, EgressTransport::ForwardHttp);
|
||||
assert_eq!(forward.destination.port, 80);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,468 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Shared relay primitives for authorized explicit-proxy egress.
|
||||
|
||||
use super::{EgressDecision, L7RouteSnapshot, emit_l7_tunnel_close_after_policy_change};
|
||||
use crate::l7::relay::L7EvalContext;
|
||||
use crate::opa::{NetworkAction, OpaEngine, PolicyGenerationGuard, TunnelPolicyEngine};
|
||||
use miette::{IntoDiagnostic, Result};
|
||||
use openshell_core::activity::ActivitySender;
|
||||
use openshell_core::proto::ProviderProfileCredential;
|
||||
use openshell_core::secrets::SecretResolver;
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use tokio::io::{AsyncRead, AsyncWrite};
|
||||
|
||||
type DynamicCredentials = Arc<std::sync::RwLock<HashMap<String, ProviderProfileCredential>>>;
|
||||
|
||||
enum PreparedHttpPolicy {
|
||||
Inspect {
|
||||
configs: Vec<crate::l7::L7EndpointConfig>,
|
||||
evaluator: Box<TunnelPolicyEngine>,
|
||||
},
|
||||
Passthrough {
|
||||
generation_guard: PolicyGenerationGuard,
|
||||
},
|
||||
}
|
||||
|
||||
/// Everything an HTTP relay needs after authorization is complete.
|
||||
///
|
||||
/// The relay deliberately owns a generation-pinned policy primitive instead
|
||||
/// of retaining access to the mutable OPA engine. Policy reloads therefore
|
||||
/// fail closed through the guard or tunnel evaluator already attached here.
|
||||
pub(super) struct RelayContext<'a> {
|
||||
request: &'a L7EvalContext,
|
||||
policy: PreparedHttpPolicy,
|
||||
middleware_engine: &'a OpaEngine,
|
||||
}
|
||||
|
||||
/// Build the request-processing context shared by CONNECT and forward HTTP.
|
||||
pub(super) fn http_context(
|
||||
decision: &EgressDecision,
|
||||
secret_resolver: Option<Arc<SecretResolver>>,
|
||||
activity_tx: Option<ActivitySender>,
|
||||
dynamic_credentials: Option<DynamicCredentials>,
|
||||
agent_proposals: openshell_core::proposals::AgentProposals,
|
||||
) -> L7EvalContext {
|
||||
let policy_name = match &decision.action {
|
||||
NetworkAction::Allow { matched_policy } => matched_policy.clone().unwrap_or_default(),
|
||||
NetworkAction::Deny { .. } => String::new(),
|
||||
};
|
||||
|
||||
L7EvalContext {
|
||||
host: decision.intent.destination.host.clone(),
|
||||
port: decision.intent.destination.port,
|
||||
policy_name,
|
||||
binary_path: decision
|
||||
.binary
|
||||
.as_ref()
|
||||
.map(|path| path.to_string_lossy().into_owned())
|
||||
.unwrap_or_default(),
|
||||
ancestors: decision
|
||||
.ancestors
|
||||
.iter()
|
||||
.map(|path| path.to_string_lossy().into_owned())
|
||||
.collect(),
|
||||
cmdline_paths: decision
|
||||
.cmdline_paths
|
||||
.iter()
|
||||
.map(|path| path.to_string_lossy().into_owned())
|
||||
.collect(),
|
||||
secret_resolver,
|
||||
activity_tx,
|
||||
dynamic_credentials: dynamic_credentials.clone(),
|
||||
token_grant_resolver: dynamic_credentials
|
||||
.as_ref()
|
||||
.map(|_| crate::l7::token_grant_injection::default_resolver()),
|
||||
agent_proposals,
|
||||
}
|
||||
}
|
||||
|
||||
/// Pin a generation for a relay or the forward HTTP single-request path.
|
||||
pub(super) fn pin_policy_generation(
|
||||
opa_engine: &OpaEngine,
|
||||
expected_generation: u64,
|
||||
) -> Result<PolicyGenerationGuard> {
|
||||
opa_engine.generation_guard(expected_generation)
|
||||
}
|
||||
|
||||
/// Clone an L7 evaluator for a relay or the forward HTTP single-request path.
|
||||
pub(super) fn pin_l7_evaluator(
|
||||
opa_engine: &OpaEngine,
|
||||
expected_generation: u64,
|
||||
) -> Result<TunnelPolicyEngine> {
|
||||
opa_engine.clone_engine_for_tunnel(expected_generation)
|
||||
}
|
||||
|
||||
pub(super) fn validate_route_generation(
|
||||
route: Option<&L7RouteSnapshot>,
|
||||
expected_generation: u64,
|
||||
) -> Result<()> {
|
||||
if let Some(route) = route
|
||||
&& route.l7_policy_generation != expected_generation
|
||||
{
|
||||
return Err(miette::miette!(
|
||||
"policy changed before CONNECT route hydration \
|
||||
[l4_generation:{} l7_generation:{}]",
|
||||
expected_generation,
|
||||
route.l7_policy_generation,
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Prepare a generation-pinned HTTP relay at the adapter boundary.
|
||||
///
|
||||
/// A stale generation preserves the established CONNECT behavior: emit the
|
||||
/// policy-change close event and let the adapter close the live tunnel without
|
||||
/// attempting to write an HTTP response into it.
|
||||
pub(super) fn prepare_http_relay<'a>(
|
||||
route: Option<&L7RouteSnapshot>,
|
||||
opa_engine: &'a OpaEngine,
|
||||
decision: &EgressDecision,
|
||||
request: &'a L7EvalContext,
|
||||
) -> Option<RelayContext<'a>> {
|
||||
if let Err(error) = validate_route_generation(route, decision.l4_policy_generation) {
|
||||
emit_l7_tunnel_close_after_policy_change(
|
||||
&decision.intent.destination.host,
|
||||
decision.intent.destination.port,
|
||||
error,
|
||||
);
|
||||
return None;
|
||||
}
|
||||
|
||||
let policy = if let Some(route) = route.filter(|route| !route.configs.is_empty()) {
|
||||
let evaluator = match pin_l7_evaluator(opa_engine, decision.l4_policy_generation) {
|
||||
Ok(evaluator) => evaluator,
|
||||
Err(error) => {
|
||||
emit_l7_tunnel_close_after_policy_change(
|
||||
&decision.intent.destination.host,
|
||||
decision.intent.destination.port,
|
||||
error,
|
||||
);
|
||||
return None;
|
||||
}
|
||||
};
|
||||
let configs = route
|
||||
.configs
|
||||
.iter()
|
||||
.map(|snapshot| snapshot.config.clone())
|
||||
.collect();
|
||||
PreparedHttpPolicy::Inspect {
|
||||
configs,
|
||||
evaluator: Box::new(evaluator),
|
||||
}
|
||||
} else {
|
||||
let generation_guard =
|
||||
match pin_policy_generation(opa_engine, decision.l4_policy_generation) {
|
||||
Ok(guard) => guard,
|
||||
Err(error) => {
|
||||
emit_l7_tunnel_close_after_policy_change(
|
||||
&decision.intent.destination.host,
|
||||
decision.intent.destination.port,
|
||||
error,
|
||||
);
|
||||
return None;
|
||||
}
|
||||
};
|
||||
PreparedHttpPolicy::Passthrough { generation_guard }
|
||||
};
|
||||
|
||||
Some(RelayContext {
|
||||
request,
|
||||
policy,
|
||||
middleware_engine: opa_engine,
|
||||
})
|
||||
}
|
||||
|
||||
/// Pin the generation used by a raw relay so policy activation or quarantine
|
||||
/// closes streams that otherwise have no request boundary at which to notice
|
||||
/// a stale decision.
|
||||
pub(super) fn prepare_raw_relay(
|
||||
route: Option<&L7RouteSnapshot>,
|
||||
opa_engine: &OpaEngine,
|
||||
decision: &EgressDecision,
|
||||
) -> Option<PolicyGenerationGuard> {
|
||||
if let Err(error) = validate_route_generation(route, decision.l4_policy_generation) {
|
||||
emit_l7_tunnel_close_after_policy_change(
|
||||
&decision.intent.destination.host,
|
||||
decision.intent.destination.port,
|
||||
error,
|
||||
);
|
||||
return None;
|
||||
}
|
||||
|
||||
match pin_policy_generation(opa_engine, decision.l4_policy_generation) {
|
||||
Ok(guard) => Some(guard),
|
||||
Err(error) => {
|
||||
emit_l7_tunnel_close_after_policy_change(
|
||||
&decision.intent.destination.host,
|
||||
decision.intent.destination.port,
|
||||
error,
|
||||
);
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Relay an HTTP/1 stream using an already-authorized, generation-pinned context.
|
||||
///
|
||||
/// CONNECT plaintext and TLS-terminated streams both enter through this
|
||||
/// function. Forward HTTP will provide a buffered first request to the same
|
||||
/// boundary in the next migration step.
|
||||
pub(super) async fn relay_http_stream<C, U>(
|
||||
client: &mut C,
|
||||
upstream: &mut U,
|
||||
context: RelayContext<'_>,
|
||||
) -> Result<()>
|
||||
where
|
||||
C: AsyncRead + AsyncWrite + Unpin + Send,
|
||||
U: AsyncRead + AsyncWrite + Unpin + Send,
|
||||
{
|
||||
match context.policy {
|
||||
PreparedHttpPolicy::Inspect { configs, evaluator } if configs.len() == 1 => {
|
||||
let generation_guard = evaluator.generation_guard().clone();
|
||||
tokio::select! {
|
||||
result = crate::l7::relay::relay_with_inspection(
|
||||
&configs[0],
|
||||
*evaluator,
|
||||
client,
|
||||
upstream,
|
||||
context.request,
|
||||
) => result,
|
||||
() = generation_guard.wait_until_stale() => {
|
||||
emit_stale_relay_close(context.request, &generation_guard);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
}
|
||||
PreparedHttpPolicy::Inspect { configs, evaluator } => {
|
||||
let generation_guard = evaluator.generation_guard().clone();
|
||||
tokio::select! {
|
||||
result = crate::l7::relay::relay_with_route_selection(
|
||||
&configs,
|
||||
*evaluator,
|
||||
client,
|
||||
upstream,
|
||||
context.request,
|
||||
) => result,
|
||||
() = generation_guard.wait_until_stale() => {
|
||||
emit_stale_relay_close(context.request, &generation_guard);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
}
|
||||
PreparedHttpPolicy::Passthrough { generation_guard } => {
|
||||
tokio::select! {
|
||||
result = crate::l7::relay::relay_passthrough_with_credentials(
|
||||
client,
|
||||
upstream,
|
||||
context.request,
|
||||
&generation_guard,
|
||||
Some(context.middleware_engine),
|
||||
) => result,
|
||||
() = generation_guard.wait_until_stale() => {
|
||||
emit_stale_relay_close(context.request, &generation_guard);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Relay a policy-authorized raw TCP stream.
|
||||
pub(super) async fn relay_tcp<C, U>(
|
||||
client: &mut C,
|
||||
upstream: &mut U,
|
||||
generation_guard: &PolicyGenerationGuard,
|
||||
request: &L7EvalContext,
|
||||
) -> Result<()>
|
||||
where
|
||||
C: AsyncRead + AsyncWrite + Unpin,
|
||||
U: AsyncRead + AsyncWrite + Unpin,
|
||||
{
|
||||
tokio::select! {
|
||||
result = tokio::io::copy_bidirectional(client, upstream) => {
|
||||
result.into_diagnostic()?;
|
||||
}
|
||||
() = generation_guard.wait_until_stale() => {
|
||||
emit_stale_relay_close(request, generation_guard);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn emit_stale_relay_close(request: &L7EvalContext, guard: &PolicyGenerationGuard) {
|
||||
emit_l7_tunnel_close_after_policy_change(
|
||||
&request.host,
|
||||
request.port,
|
||||
miette::miette!(
|
||||
"policy generation is stale [captured_generation:{} current_generation:{}]",
|
||||
guard.captured_generation(),
|
||||
guard.current_generation(),
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::super::{EgressIntent, EndpointDecision, ProcessIdentityEvidence};
|
||||
use super::*;
|
||||
|
||||
const POLICY_REGO: &str = include_str!("../../data/sandbox-policy.rego");
|
||||
const EMPTY_POLICY_DATA: &str = "network_policies: {}\n";
|
||||
|
||||
fn decision(l4_policy_generation: u64) -> EgressDecision {
|
||||
EgressDecision {
|
||||
intent: EgressIntent::connect("example.com".to_string(), 80),
|
||||
action: NetworkAction::Allow {
|
||||
matched_policy: Some("test".to_string()),
|
||||
},
|
||||
l4_policy_generation,
|
||||
identity: ProcessIdentityEvidence::Available,
|
||||
endpoint: EndpointDecision::default(),
|
||||
binary: None,
|
||||
binary_pid: None,
|
||||
ancestors: vec![],
|
||||
cmdline_paths: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
fn request_context() -> L7EvalContext {
|
||||
L7EvalContext {
|
||||
host: "example.com".to_string(),
|
||||
port: 80,
|
||||
policy_name: "test".to_string(),
|
||||
binary_path: String::new(),
|
||||
ancestors: vec![],
|
||||
cmdline_paths: vec![],
|
||||
secret_resolver: None,
|
||||
activity_tx: None,
|
||||
dynamic_credentials: None,
|
||||
token_grant_resolver: None,
|
||||
agent_proposals: openshell_core::proposals::AgentProposals::default(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn relay_without_route_pins_l4_decision_generation() {
|
||||
let engine = OpaEngine::from_strings(POLICY_REGO, EMPTY_POLICY_DATA).unwrap();
|
||||
let decision = decision(engine.current_generation());
|
||||
let request = request_context();
|
||||
|
||||
let context = prepare_http_relay(None, &engine, &decision, &request)
|
||||
.expect("current L4 generation should prepare a relay");
|
||||
let PreparedHttpPolicy::Passthrough { generation_guard } = context.policy else {
|
||||
panic!("route-less relay should use a generation guard");
|
||||
};
|
||||
|
||||
assert_eq!(
|
||||
generation_guard.captured_generation(),
|
||||
decision.l4_policy_generation
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_hydrated_route_cannot_replace_stale_l4_generation() {
|
||||
let engine = OpaEngine::from_strings(POLICY_REGO, EMPTY_POLICY_DATA).unwrap();
|
||||
let decision = decision(u64::MAX);
|
||||
let route = L7RouteSnapshot {
|
||||
configs: vec![],
|
||||
l7_policy_generation: engine.current_generation(),
|
||||
};
|
||||
let request = request_context();
|
||||
|
||||
assert!(
|
||||
prepare_http_relay(Some(&route), &engine, &decision, &request).is_none(),
|
||||
"a current L7 lookup must not freshen a stale L4 allow"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inspected_route_cannot_replace_stale_l4_generation() {
|
||||
let engine = OpaEngine::from_strings(POLICY_REGO, EMPTY_POLICY_DATA).unwrap();
|
||||
let decision = decision(u64::MAX);
|
||||
let route = L7RouteSnapshot {
|
||||
configs: vec![super::super::L7ConfigSnapshot {
|
||||
config: crate::l7::L7EndpointConfig {
|
||||
protocol: crate::l7::L7Protocol::Rest,
|
||||
path: "/**".to_string(),
|
||||
tls: crate::l7::TlsMode::Auto,
|
||||
enforcement: crate::l7::EnforcementMode::Enforce,
|
||||
graphql_max_body_bytes: crate::l7::graphql::DEFAULT_MAX_BODY_BYTES,
|
||||
json_rpc_max_body_bytes: crate::l7::jsonrpc::DEFAULT_MAX_BODY_BYTES,
|
||||
mcp_strict_tool_names: true,
|
||||
allow_encoded_slash: false,
|
||||
websocket_credential_rewrite: false,
|
||||
request_body_credential_rewrite: false,
|
||||
websocket_graphql_policy: false,
|
||||
credential_signing: crate::l7::CredentialSigning::None,
|
||||
signing_service: String::new(),
|
||||
signing_region: String::new(),
|
||||
},
|
||||
}],
|
||||
l7_policy_generation: engine.current_generation(),
|
||||
};
|
||||
let request = request_context();
|
||||
|
||||
assert!(
|
||||
prepare_http_relay(Some(&route), &engine, &decision, &request).is_none(),
|
||||
"an inspected route must use the generation that authorized CONNECT"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn raw_route_cannot_replace_stale_l4_generation() {
|
||||
let engine = OpaEngine::from_strings(POLICY_REGO, EMPTY_POLICY_DATA).unwrap();
|
||||
let decision = decision(u64::MAX);
|
||||
let route = L7RouteSnapshot {
|
||||
configs: vec![],
|
||||
l7_policy_generation: engine.current_generation(),
|
||||
};
|
||||
|
||||
assert!(
|
||||
prepare_raw_relay(Some(&route), &engine, &decision).is_none(),
|
||||
"a raw relay must not freshen a stale L4 allow"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stale_generation_fails_before_relay_context_is_created() {
|
||||
let engine = OpaEngine::from_strings(POLICY_REGO, EMPTY_POLICY_DATA).unwrap();
|
||||
let decision = decision(engine.current_generation());
|
||||
let request = request_context();
|
||||
engine.reload(POLICY_REGO, EMPTY_POLICY_DATA).unwrap();
|
||||
|
||||
assert!(
|
||||
prepare_http_relay(None, &engine, &decision, &request).is_none(),
|
||||
"policy reload must prevent a stale relay from starting"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn raw_relay_closes_immediately_when_fail_closed_generation_is_published() {
|
||||
let engine = Arc::new(OpaEngine::from_strings(POLICY_REGO, EMPTY_POLICY_DATA).unwrap());
|
||||
let guard = engine
|
||||
.generation_guard(engine.current_generation())
|
||||
.unwrap();
|
||||
let request = request_context();
|
||||
let (_client_peer, mut proxy_client) = tokio::io::duplex(64);
|
||||
let (_upstream_peer, mut proxy_upstream) = tokio::io::duplex(64);
|
||||
|
||||
let relay = tokio::spawn(async move {
|
||||
relay_tcp(&mut proxy_client, &mut proxy_upstream, &guard, &request).await
|
||||
});
|
||||
tokio::task::yield_now().await;
|
||||
|
||||
engine
|
||||
.enter_fail_closed("candidate policy validation failed")
|
||||
.unwrap();
|
||||
|
||||
tokio::time::timeout(std::time::Duration::from_secs(1), relay)
|
||||
.await
|
||||
.expect("raw relay should close when its generation becomes stale")
|
||||
.expect("relay task should not panic")
|
||||
.expect("stale relay closure should be clean");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,609 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Compatibility and regression contracts for the shared proxy egress pipeline.
|
||||
|
||||
use super::*;
|
||||
use std::sync::Arc;
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
|
||||
fn allowed_decision(intent: EgressIntent) -> EgressDecision {
|
||||
EgressDecision {
|
||||
intent,
|
||||
action: NetworkAction::Allow {
|
||||
matched_policy: Some("proxy_compatibility".to_string()),
|
||||
},
|
||||
l4_policy_generation: 0,
|
||||
identity: ProcessIdentityEvidence::Available,
|
||||
endpoint: EndpointDecision::default(),
|
||||
binary: Some(PathBuf::from("/usr/bin/curl")),
|
||||
binary_pid: Some(42),
|
||||
ancestors: vec![PathBuf::from("/usr/bin/sh")],
|
||||
cmdline_paths: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
async fn tcp_pair() -> (TcpStream, TcpStream) {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let client = TcpStream::connect(listener.local_addr().unwrap())
|
||||
.await
|
||||
.unwrap();
|
||||
let (server, _) = listener.accept().await.unwrap();
|
||||
(client, server)
|
||||
}
|
||||
|
||||
fn assert_json_response(
|
||||
response: &[u8],
|
||||
expected_status: &str,
|
||||
expected_error: &str,
|
||||
expected_detail: &str,
|
||||
) {
|
||||
let (headers, body) = response
|
||||
.windows(4)
|
||||
.position(|window| window == b"\r\n\r\n")
|
||||
.map(|end| (&response[..end + 4], &response[end + 4..]))
|
||||
.expect("complete HTTP response");
|
||||
let headers = String::from_utf8(headers.to_vec()).unwrap();
|
||||
assert!(headers.starts_with(expected_status));
|
||||
assert!(headers.contains("Content-Type: application/json\r\n"));
|
||||
assert!(headers.contains(&format!("Content-Length: {}\r\n", body.len())));
|
||||
assert!(headers.contains("Connection: close\r\n"));
|
||||
assert_eq!(
|
||||
serde_json::from_slice::<serde_json::Value>(body).unwrap(),
|
||||
serde_json::json!({
|
||||
"error": expected_error,
|
||||
"detail": expected_detail,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn destination_denials_preserve_adapter_specific_wire_contracts() {
|
||||
let cases = [
|
||||
(
|
||||
DestinationDenialKind::TrustedGateway,
|
||||
"trusted-gateway check failed",
|
||||
),
|
||||
(
|
||||
DestinationDenialKind::InvalidAllowedIps,
|
||||
"invalid allowed_ips in policy",
|
||||
),
|
||||
(
|
||||
DestinationDenialKind::AllowedIps,
|
||||
"allowed_ips check failed",
|
||||
),
|
||||
(
|
||||
DestinationDenialKind::DeclaredEndpoint,
|
||||
"declared endpoint check failed",
|
||||
),
|
||||
(DestinationDenialKind::InternalAddress, "internal address"),
|
||||
];
|
||||
|
||||
for (kind, detail) in cases {
|
||||
let denial = DestinationDenial {
|
||||
kind,
|
||||
reason: "proxy compatibility destination failure".to_string(),
|
||||
};
|
||||
let peer: SocketAddr = "127.0.0.1:41000".parse().unwrap();
|
||||
|
||||
let (mut app, mut proxy) = tcp_pair().await;
|
||||
deny_connect_destination(
|
||||
&mut proxy,
|
||||
&denial,
|
||||
peer,
|
||||
"target.example",
|
||||
8443,
|
||||
"/usr/bin/curl",
|
||||
"42",
|
||||
"/usr/bin/sh",
|
||||
"curl",
|
||||
&allowed_decision(EgressIntent::connect("target.example".to_string(), 8443)),
|
||||
&None,
|
||||
&None,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
proxy.shutdown().await.unwrap();
|
||||
let mut response = Vec::new();
|
||||
app.read_to_end(&mut response).await.unwrap();
|
||||
assert_json_response(
|
||||
&response,
|
||||
"HTTP/1.1 403 Forbidden\r\n",
|
||||
"ssrf_denied",
|
||||
&format!("CONNECT target.example:8443 blocked: {detail}"),
|
||||
);
|
||||
|
||||
let (mut app, mut proxy) = tcp_pair().await;
|
||||
deny_forward_destination(
|
||||
&mut proxy,
|
||||
&denial,
|
||||
peer,
|
||||
"POST",
|
||||
"target.example",
|
||||
8080,
|
||||
"/v1/items",
|
||||
"/usr/bin/curl",
|
||||
"42",
|
||||
"/usr/bin/sh",
|
||||
"curl",
|
||||
"proxy_compatibility",
|
||||
&allowed_decision(EgressIntent::forward_http(
|
||||
"target.example".to_string(),
|
||||
8080,
|
||||
)),
|
||||
None,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
proxy.shutdown().await.unwrap();
|
||||
let mut response = Vec::new();
|
||||
app.read_to_end(&mut response).await.unwrap();
|
||||
assert_json_response(
|
||||
&response,
|
||||
"HTTP/1.1 403 Forbidden\r\n",
|
||||
"ssrf_denied",
|
||||
&format!("POST target.example:8080 blocked: {detail}"),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn representative_adapter_denials_preserve_ocsf_fields() {
|
||||
let denial_reason = "target.example resolves to internal address 10.0.0.5";
|
||||
let denial = DestinationDenial {
|
||||
kind: DestinationDenialKind::InternalAddress,
|
||||
reason: denial_reason.to_string(),
|
||||
};
|
||||
let peer: SocketAddr = "127.0.0.1:41000".parse().unwrap();
|
||||
|
||||
// Build the production events directly rather than routing through the
|
||||
// global tracing pipeline. Its callsite-interest cache is process-global,
|
||||
// so parallel tests can otherwise make captured-event assertions flaky.
|
||||
let connect = serde_json::to_value(build_connect_destination_deny_ocsf_event(
|
||||
&denial,
|
||||
peer,
|
||||
"target.example",
|
||||
8443,
|
||||
"/usr/bin/curl",
|
||||
"42",
|
||||
"/usr/bin/sh",
|
||||
"curl --proxy",
|
||||
))
|
||||
.unwrap();
|
||||
assert_eq!(connect["class_name"], "Network Activity");
|
||||
assert_eq!(connect["activity_name"], "Open");
|
||||
assert_eq!(connect["action"], "Denied");
|
||||
assert_eq!(connect["disposition"], "Blocked");
|
||||
assert_eq!(connect["severity"], "Medium");
|
||||
assert_eq!(connect["status"], "Failure");
|
||||
assert_eq!(connect["dst_endpoint"]["domain"], "target.example");
|
||||
assert_eq!(connect["dst_endpoint"]["port"], 8443);
|
||||
assert_eq!(connect["actor"]["process"]["name"], "/usr/bin/curl");
|
||||
assert_eq!(connect["actor"]["process"]["pid"], 42);
|
||||
assert_eq!(connect["firewall_rule"]["name"], "-");
|
||||
assert_eq!(connect["firewall_rule"]["type"], "ssrf");
|
||||
assert_eq!(
|
||||
connect["message"],
|
||||
"CONNECT blocked: internal address target.example:8443"
|
||||
);
|
||||
assert_eq!(connect["status_detail"], denial_reason);
|
||||
|
||||
let forward = serde_json::to_value(build_forward_destination_deny_ocsf_event(
|
||||
&denial,
|
||||
peer,
|
||||
"POST",
|
||||
"target.example",
|
||||
8080,
|
||||
"/v1/items",
|
||||
"/usr/bin/curl",
|
||||
"42",
|
||||
"/usr/bin/sh",
|
||||
"curl --proxy",
|
||||
"proxy_compatibility",
|
||||
))
|
||||
.unwrap();
|
||||
assert_eq!(forward["class_name"], "HTTP Activity");
|
||||
assert_eq!(forward["activity_name"], "Other");
|
||||
assert_eq!(forward["action"], "Denied");
|
||||
assert_eq!(forward["disposition"], "Blocked");
|
||||
assert_eq!(forward["severity"], "Medium");
|
||||
assert_eq!(forward["status"], "Failure");
|
||||
assert_eq!(forward["dst_endpoint"]["domain"], "target.example");
|
||||
assert_eq!(forward["dst_endpoint"]["port"], 8080);
|
||||
assert_eq!(forward["http_request"]["http_method"], "POST");
|
||||
assert_eq!(forward["firewall_rule"]["name"], "proxy_compatibility");
|
||||
assert_eq!(forward["firewall_rule"]["type"], "ssrf");
|
||||
assert_eq!(
|
||||
forward["message"],
|
||||
"FORWARD blocked: internal IP without allowed_ips for target.example:8080"
|
||||
);
|
||||
assert_eq!(forward["status_detail"], denial_reason);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn representative_adapter_allows_preserve_ocsf_fields() {
|
||||
let peer: SocketAddr = "127.0.0.1:41000".parse().unwrap();
|
||||
let connect = serde_json::to_value(build_connect_allow_ocsf_event(
|
||||
peer,
|
||||
"target.example",
|
||||
8443,
|
||||
"/usr/bin/curl",
|
||||
"42",
|
||||
"/usr/bin/sh",
|
||||
"curl --proxy",
|
||||
"proxy_compatibility",
|
||||
true,
|
||||
))
|
||||
.unwrap();
|
||||
assert_eq!(connect["class_name"], "Network Activity");
|
||||
assert_eq!(connect["activity_name"], "Open");
|
||||
assert_eq!(connect["action"], "Allowed");
|
||||
assert_eq!(connect["disposition"], "Allowed");
|
||||
assert_eq!(connect["severity"], "Informational");
|
||||
assert_eq!(connect["status"], "Success");
|
||||
assert_eq!(connect["dst_endpoint"]["domain"], "target.example");
|
||||
assert_eq!(connect["dst_endpoint"]["port"], 8443);
|
||||
assert_eq!(connect["actor"]["process"]["name"], "/usr/bin/curl");
|
||||
assert_eq!(connect["firewall_rule"]["name"], "proxy_compatibility");
|
||||
assert_eq!(connect["firewall_rule"]["type"], "opa");
|
||||
assert_eq!(connect["message"], "CONNECT_L7 allowed target.example:8443");
|
||||
|
||||
let forward = serde_json::to_value(build_forward_allow_ocsf_event(
|
||||
peer,
|
||||
"GET",
|
||||
"target.example",
|
||||
8080,
|
||||
"/v1/items",
|
||||
"/usr/bin/curl",
|
||||
"42",
|
||||
"/usr/bin/sh",
|
||||
"curl --proxy",
|
||||
"proxy_compatibility",
|
||||
))
|
||||
.unwrap();
|
||||
assert_eq!(forward["class_name"], "HTTP Activity");
|
||||
assert_eq!(forward["activity_name"], "Other");
|
||||
assert_eq!(forward["action"], "Allowed");
|
||||
assert_eq!(forward["disposition"], "Allowed");
|
||||
assert_eq!(forward["severity"], "Informational");
|
||||
assert_eq!(forward["status"], "Success");
|
||||
assert_eq!(forward["dst_endpoint"]["domain"], "target.example");
|
||||
assert_eq!(forward["dst_endpoint"]["port"], 8080);
|
||||
assert_eq!(forward["http_request"]["http_method"], "GET");
|
||||
assert_eq!(forward["firewall_rule"]["name"], "proxy_compatibility");
|
||||
assert_eq!(forward["firewall_rule"]["type"], "opa");
|
||||
assert_eq!(
|
||||
forward["message"],
|
||||
"FORWARD allowed GET target.example:8080/v1/items"
|
||||
);
|
||||
}
|
||||
|
||||
fn poisoned_engine() -> OpaEngine {
|
||||
let engine = OpaEngine::from_strings(
|
||||
include_str!("../../../data/sandbox-policy.rego"),
|
||||
r#"
|
||||
network_policies:
|
||||
proxy_compatibility:
|
||||
name: proxy_compatibility
|
||||
endpoints:
|
||||
- host: target.example
|
||||
port: 443
|
||||
protocol: rest
|
||||
enforcement: enforce
|
||||
tls: skip
|
||||
allowed_ips: ["10.0.0.0/8"]
|
||||
rules:
|
||||
- allow: { method: GET, path: "/**" }
|
||||
binaries:
|
||||
- path: /usr/bin/curl
|
||||
"#,
|
||||
)
|
||||
.unwrap();
|
||||
engine.poison_lock_for_test();
|
||||
engine
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn l7_query_failure_preserves_l4_only_fallback() {
|
||||
let engine = poisoned_engine();
|
||||
let decision = allowed_decision(EgressIntent::connect("target.example".to_string(), 443));
|
||||
assert!(query_l7_route_snapshot(&engine, &decision, "target.example", 443).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tls_query_failure_preserves_auto_fallback() {
|
||||
let engine = poisoned_engine();
|
||||
let decision = allowed_decision(EgressIntent::connect("target.example".to_string(), 443));
|
||||
assert_eq!(
|
||||
query_tls_mode(&engine, &decision, "target.example", 443),
|
||||
crate::l7::TlsMode::Auto
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn allowed_ips_query_failure_preserves_empty_fallback() {
|
||||
let engine = poisoned_engine();
|
||||
let decision = allowed_decision(EgressIntent::connect("target.example".to_string(), 443));
|
||||
assert!(query_allowed_ips(&engine, &decision, "target.example", 443).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn exact_host_query_failure_preserves_false_fallback() {
|
||||
let engine = poisoned_engine();
|
||||
let decision = allowed_decision(EgressIntent::connect("target.example".to_string(), 443));
|
||||
assert!(!query_exact_declared_endpoint_host(
|
||||
&engine,
|
||||
&decision,
|
||||
"target.example",
|
||||
443
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn identity_required_policy_accepts_real_binary_and_rejects_empty_exec_path() {
|
||||
let engine = OpaEngine::from_strings(
|
||||
include_str!("../../../data/sandbox-policy.rego"),
|
||||
r#"
|
||||
network_policies:
|
||||
proxy_compatibility:
|
||||
name: proxy_compatibility
|
||||
endpoints:
|
||||
- host: target.example
|
||||
port: 443
|
||||
binaries:
|
||||
- path: /usr/bin/curl
|
||||
"#,
|
||||
)
|
||||
.unwrap();
|
||||
let input = |binary_path: PathBuf| crate::opa::NetworkInput {
|
||||
host: "target.example".to_string(),
|
||||
port: 443,
|
||||
binary_path,
|
||||
binary_sha256: String::new(),
|
||||
ancestors: vec![],
|
||||
cmdline_paths: vec![],
|
||||
};
|
||||
|
||||
assert!(matches!(
|
||||
engine
|
||||
.evaluate_network_action(&input(PathBuf::from("/usr/bin/curl")))
|
||||
.unwrap(),
|
||||
NetworkAction::Allow { .. }
|
||||
));
|
||||
assert!(matches!(
|
||||
engine
|
||||
.evaluate_network_action(&input(PathBuf::new()))
|
||||
.unwrap(),
|
||||
NetworkAction::Deny { .. }
|
||||
));
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
#[test]
|
||||
fn identity_required_mode_is_explicitly_unsupported_off_linux() {
|
||||
let engine = OpaEngine::from_strings(
|
||||
include_str!("../../../data/sandbox-policy.rego"),
|
||||
"network_policies: {}\n",
|
||||
)
|
||||
.unwrap();
|
||||
let decision = authorize_egress_intent(
|
||||
crate::procfs::WorkloadProxyTcpConnection::new(
|
||||
"127.0.0.1:41000".parse().unwrap(),
|
||||
"127.0.0.1:3000".parse().unwrap(),
|
||||
),
|
||||
&engine,
|
||||
&BinaryIdentityCache::new(),
|
||||
&AtomicU32::new(1),
|
||||
EgressIntent::connect("target.example".to_string(), 443),
|
||||
);
|
||||
|
||||
assert!(matches!(decision.action, NetworkAction::Deny { .. }));
|
||||
assert_eq!(
|
||||
decision.identity,
|
||||
ProcessIdentityEvidence::Unavailable(IdentityUnavailableReason::UnsupportedPlatform)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn forward_rewrite_does_not_treat_a_pipelined_request_as_body_overflow() {
|
||||
let raw = b"GET http://target.example/allowed HTTP/1.1\r\n\
|
||||
Host: target.example\r\n\
|
||||
Connection: keep-alive\r\n\r\n\
|
||||
POST http://target.example/blocked HTTP/1.1\r\n\
|
||||
Host: target.example\r\n\
|
||||
Content-Length: 0\r\n\r\n";
|
||||
let rewritten =
|
||||
rewrite_forward_request(raw, raw.len(), "/allowed", "target.example", None, false).unwrap();
|
||||
let rewritten = String::from_utf8(rewritten).unwrap();
|
||||
|
||||
assert!(rewritten.starts_with("GET /allowed HTTP/1.1\r\n"));
|
||||
assert!(rewritten.contains("Connection: close\r\n"));
|
||||
assert!(!rewritten.contains("POST http://target.example/blocked"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn forward_rewrite_trims_pipeline_after_content_length_body() {
|
||||
let raw = b"POST http://target.example/allowed HTTP/1.1\r\n\
|
||||
Host: target.example\r\n\
|
||||
Content-Length: 4\r\n\r\n\
|
||||
body\
|
||||
GET http://target.example/blocked HTTP/1.1\r\n\
|
||||
Host: target.example\r\n\r\n";
|
||||
let rewritten =
|
||||
rewrite_forward_request(raw, raw.len(), "/allowed", "target.example", None, false).unwrap();
|
||||
let rewritten = String::from_utf8(rewritten).unwrap();
|
||||
|
||||
assert!(rewritten.ends_with("\r\n\r\nbody"));
|
||||
assert!(!rewritten.contains("GET http://target.example/blocked"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn forward_rewrite_trims_pipeline_after_complete_chunked_body() {
|
||||
let raw = b"POST http://target.example/allowed HTTP/1.1\r\n\
|
||||
Host: target.example\r\n\
|
||||
Transfer-Encoding: chunked\r\n\r\n\
|
||||
4\r\nbody\r\n0\r\n\r\n\
|
||||
GET http://target.example/blocked HTTP/1.1\r\n\
|
||||
Host: target.example\r\n\r\n";
|
||||
let rewritten =
|
||||
rewrite_forward_request(raw, raw.len(), "/allowed", "target.example", None, false).unwrap();
|
||||
let rewritten = String::from_utf8(rewritten).unwrap();
|
||||
|
||||
assert!(rewritten.ends_with("4\r\nbody\r\n0\r\n\r\n"));
|
||||
assert!(!rewritten.contains("GET http://target.example/blocked"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn forward_https_absolute_form_rejection_is_snapshotted() {
|
||||
let (response, denial_stages) = drive_forward_through_handler(
|
||||
" - { host: \"target.example\", port: 443 }\n",
|
||||
"https://target.example/private",
|
||||
)
|
||||
.await;
|
||||
|
||||
assert_eq!(
|
||||
response,
|
||||
b"HTTP/1.1 400 Bad Request\r\nContent-Length: 27\r\n\r\nUse CONNECT for HTTPS URLs"
|
||||
);
|
||||
assert!(denial_stages.is_empty());
|
||||
}
|
||||
|
||||
async fn exercise_benchmark_request(proxy_addr: SocketAddr, target: SocketAddr, connect: bool) {
|
||||
let mut client = TcpStream::connect(proxy_addr).await.unwrap();
|
||||
let authority = target.to_string();
|
||||
let request = if connect {
|
||||
format!("CONNECT {authority} HTTP/1.1\r\nHost: {authority}\r\n\r\n")
|
||||
} else {
|
||||
format!(
|
||||
"GET http://{authority}/proxy-baseline HTTP/1.1\r\nHost: {authority}\r\nConnection: close\r\n\r\n"
|
||||
)
|
||||
};
|
||||
client.write_all(request.as_bytes()).await.unwrap();
|
||||
let mut response = Vec::new();
|
||||
client.read_to_end(&mut response).await.unwrap();
|
||||
assert!(response.starts_with(b"HTTP/1.1 403 Forbidden"));
|
||||
}
|
||||
|
||||
/// Run with:
|
||||
/// `cargo test -p openshell-supervisor-network proxy_performance_baseline -- --ignored --nocapture --test-threads=1`
|
||||
#[test]
|
||||
#[ignore = "manual proxy allocation/query/latency baseline"]
|
||||
fn proxy_performance_baseline() {
|
||||
temp_env::with_vars(
|
||||
[(
|
||||
openshell_core::sandbox_env::NETWORK_BINARY_IDENTITY,
|
||||
Some("endpoint-only"),
|
||||
)],
|
||||
|| {
|
||||
tokio::runtime::Builder::new_multi_thread()
|
||||
.worker_threads(2)
|
||||
.enable_all()
|
||||
.build()
|
||||
.unwrap()
|
||||
.block_on(async {
|
||||
// Benchmark the full fail-closed path using a declared loopback
|
||||
// destination. This is deterministic and never opens a listener
|
||||
// outside the local process, so it does not trigger host firewall
|
||||
// prompts during manual baseline collection.
|
||||
let target: SocketAddr = "127.0.0.1:18080".parse().unwrap();
|
||||
|
||||
let policy = format!(
|
||||
r#"
|
||||
network_policies:
|
||||
proxy_compatibility:
|
||||
name: proxy_compatibility
|
||||
endpoints:
|
||||
- host: {host}
|
||||
port: {port}
|
||||
tls: skip
|
||||
binaries:
|
||||
- path: "/**"
|
||||
"#,
|
||||
host = target.ip(),
|
||||
port = target.port(),
|
||||
);
|
||||
let engine = Arc::new(
|
||||
OpaEngine::from_strings_with_binary_identity_required(
|
||||
include_str!("../../../data/sandbox-policy.rego"),
|
||||
&policy,
|
||||
false,
|
||||
)
|
||||
.unwrap(),
|
||||
);
|
||||
let proxy_listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let proxy_addr = proxy_listener.local_addr().unwrap();
|
||||
let proxy_engine = engine.clone();
|
||||
let proxy_task = tokio::spawn(async move {
|
||||
while let Ok((stream, _)) = proxy_listener.accept().await {
|
||||
let engine = proxy_engine.clone();
|
||||
tokio::spawn(async move {
|
||||
Box::pin(handle_tcp_connection(
|
||||
stream,
|
||||
engine,
|
||||
Arc::new(BinaryIdentityCache::new()),
|
||||
Arc::new(AtomicU32::new(0)),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
AgentProposals::default(),
|
||||
Arc::new(None),
|
||||
Arc::new(None),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
for connect in [true, false] {
|
||||
exercise_benchmark_request(proxy_addr, target, connect).await;
|
||||
}
|
||||
|
||||
let iterations = std::env::var("OPENSHELL_PROXY_BASELINE_ITERATIONS")
|
||||
.ok()
|
||||
.and_then(|value| value.parse::<u64>().ok())
|
||||
.filter(|value| *value > 0)
|
||||
.unwrap_or(25);
|
||||
let mut results = serde_json::Map::new();
|
||||
for (name, connect) in [("connect", true), ("forward", false)] {
|
||||
crate::test_alloc::reset();
|
||||
crate::opa::reset_test_opa_query_count();
|
||||
let started = std::time::Instant::now();
|
||||
for _ in 0..iterations {
|
||||
exercise_benchmark_request(proxy_addr, target, connect).await;
|
||||
}
|
||||
let elapsed = started.elapsed();
|
||||
let queries = crate::opa::test_opa_query_count();
|
||||
let (allocations, allocated_bytes) = crate::test_alloc::snapshot();
|
||||
let expected_queries = 4;
|
||||
assert_eq!(queries, expected_queries * iterations);
|
||||
results.insert(
|
||||
name.to_string(),
|
||||
serde_json::json!({
|
||||
"allocated_bytes_per_request": allocated_bytes / iterations,
|
||||
"allocations_per_request": allocations / iterations,
|
||||
"latency_ns_per_request": elapsed.as_nanos() / u128::from(iterations),
|
||||
"opa_queries_per_request": queries / iterations,
|
||||
}),
|
||||
);
|
||||
}
|
||||
println!(
|
||||
"{}",
|
||||
serde_json::json!({
|
||||
"iterations": iterations,
|
||||
"proxy_performance_baseline": results,
|
||||
"scenario": "declared_loopback_destination_denied",
|
||||
"schema_version": 1,
|
||||
})
|
||||
);
|
||||
|
||||
proxy_task.abort();
|
||||
});
|
||||
},
|
||||
);
|
||||
}
|
||||
@@ -77,6 +77,61 @@ pub fn parse_kmsg_line(line: &str, namespace_prefix: &str) -> Option<BypassEvent
|
||||
})
|
||||
}
|
||||
|
||||
fn build_bypass_ocsf_events(
|
||||
event: &BypassEvent,
|
||||
binary: &str,
|
||||
binary_pid: &str,
|
||||
ancestors: &str,
|
||||
) -> (openshell_ocsf::OcsfEvent, openshell_ocsf::OcsfEvent) {
|
||||
let hint = hint_for_event(event);
|
||||
let reason = "direct connection bypassed HTTP CONNECT proxy";
|
||||
let dst_port = event.dst_port.to_string();
|
||||
let dst_ep = event.dst_addr.parse::<std::net::IpAddr>().map_or_else(
|
||||
|_| Endpoint::from_domain(&event.dst_addr, event.dst_port),
|
||||
|ip| Endpoint::from_ip(ip, event.dst_port),
|
||||
);
|
||||
|
||||
let net_event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx())
|
||||
.activity(ActivityId::Refuse)
|
||||
.action(ActionId::Denied)
|
||||
.disposition(DispositionId::Blocked)
|
||||
.severity(SeverityId::Medium)
|
||||
.dst_endpoint(dst_ep)
|
||||
.actor_process(Process::from_bypass(binary, binary_pid, ancestors))
|
||||
.firewall_rule("bypass-detect", "nftables")
|
||||
.observation_point(3)
|
||||
.message(format!(
|
||||
"BYPASS_DETECT {}:{} proto={} binary={binary} action=reject reason={reason}",
|
||||
event.dst_addr, event.dst_port, event.proto,
|
||||
))
|
||||
.build();
|
||||
|
||||
let finding_event = DetectionFindingBuilder::new(openshell_ocsf::ctx::ctx())
|
||||
.activity(ActivityId::Open)
|
||||
.action(ActionId::Denied)
|
||||
.disposition(DispositionId::Blocked)
|
||||
.severity(SeverityId::Medium)
|
||||
.is_alert(true)
|
||||
.confidence(ConfidenceId::High)
|
||||
.finding_info(FindingInfo::new("bypass-detect", "Proxy Bypass Detected").with_desc(reason))
|
||||
.remediation(hint)
|
||||
.evidence_pairs(&[
|
||||
("dst_addr", event.dst_addr.as_str()),
|
||||
("dst_port", dst_port.as_str()),
|
||||
("proto", event.proto.as_str()),
|
||||
("binary", binary),
|
||||
("binary_pid", binary_pid),
|
||||
("ancestors", ancestors),
|
||||
])
|
||||
.message(format!(
|
||||
"BYPASS_DETECT {}:{} proto={} binary={binary} hint={hint}",
|
||||
event.dst_addr, event.dst_port, event.proto,
|
||||
))
|
||||
.build();
|
||||
|
||||
(net_event, finding_event)
|
||||
}
|
||||
|
||||
/// Extract a single space-delimited field value from a nftables log line.
|
||||
///
|
||||
/// Given `"DST="` and a string like `"...DST=93.184.216.34 LEN=60..."`,
|
||||
@@ -207,60 +262,11 @@ pub fn spawn(
|
||||
("-".to_string(), "-".to_string(), "-".to_string())
|
||||
};
|
||||
|
||||
let hint = hint_for_event(&event);
|
||||
let reason = "direct connection bypassed HTTP CONNECT proxy";
|
||||
|
||||
// Dual-emit: Network Activity [4001] + Detection Finding [2004]
|
||||
{
|
||||
let dst_ep = if let Ok(ip) = event.dst_addr.parse::<std::net::IpAddr>() {
|
||||
Endpoint::from_ip(ip, event.dst_port)
|
||||
} else {
|
||||
Endpoint::from_domain(&event.dst_addr, event.dst_port)
|
||||
};
|
||||
|
||||
let net_event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx())
|
||||
.activity(ActivityId::Refuse)
|
||||
.action(ActionId::Denied)
|
||||
.disposition(DispositionId::Blocked)
|
||||
.severity(SeverityId::Medium)
|
||||
.dst_endpoint(dst_ep.clone())
|
||||
.actor_process(Process::from_bypass(&binary, &binary_pid, &ancestors))
|
||||
.firewall_rule("bypass-detect", "nftables")
|
||||
.observation_point(3)
|
||||
.message(format!(
|
||||
"BYPASS_DETECT {}:{} proto={} binary={binary} action=reject reason={reason}",
|
||||
event.dst_addr, event.dst_port, event.proto,
|
||||
))
|
||||
.build();
|
||||
ocsf_emit!(net_event);
|
||||
|
||||
let finding_event = DetectionFindingBuilder::new(openshell_ocsf::ctx::ctx())
|
||||
.activity(ActivityId::Open)
|
||||
.action(ActionId::Denied)
|
||||
.disposition(DispositionId::Blocked)
|
||||
.severity(SeverityId::Medium)
|
||||
.is_alert(true)
|
||||
.confidence(ConfidenceId::High)
|
||||
.finding_info(
|
||||
FindingInfo::new("bypass-detect", "Proxy Bypass Detected")
|
||||
.with_desc(reason),
|
||||
)
|
||||
.remediation(hint)
|
||||
.evidence_pairs(&[
|
||||
("dst_addr", &event.dst_addr),
|
||||
("dst_port", &event.dst_port.to_string()),
|
||||
("proto", &event.proto),
|
||||
("binary", &binary),
|
||||
("binary_pid", &binary_pid),
|
||||
("ancestors", &ancestors),
|
||||
])
|
||||
.message(format!(
|
||||
"BYPASS_DETECT {}:{} proto={} binary={binary} hint={hint}",
|
||||
event.dst_addr, event.dst_port, event.proto,
|
||||
))
|
||||
.build();
|
||||
ocsf_emit!(finding_event);
|
||||
}
|
||||
let (net_event, finding_event) =
|
||||
build_bypass_ocsf_events(&event, &binary, &binary_pid, &ancestors);
|
||||
ocsf_emit!(net_event);
|
||||
ocsf_emit!(finding_event);
|
||||
|
||||
// Send to denial aggregator if available.
|
||||
if let Some(ref tx) = denial_tx {
|
||||
@@ -488,6 +494,49 @@ mod tests {
|
||||
assert!(hint_for_event(&event).contains("UDP"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bypass_ocsf_contract_is_stable() {
|
||||
let event = BypassEvent {
|
||||
dst_addr: "93.184.216.34".to_string(),
|
||||
dst_port: 443,
|
||||
src_port: 48012,
|
||||
proto: "tcp".to_string(),
|
||||
uid: Some(1000),
|
||||
};
|
||||
let (network, finding) =
|
||||
build_bypass_ocsf_events(&event, "/usr/bin/curl", "42", "/usr/bin/sh");
|
||||
let network = serde_json::to_value(network).unwrap();
|
||||
assert_eq!(network["class_name"], "Network Activity");
|
||||
assert_eq!(network["activity_name"], "Refuse");
|
||||
assert_eq!(network["action"], "Denied");
|
||||
assert_eq!(network["disposition"], "Blocked");
|
||||
assert_eq!(network["severity"], "Medium");
|
||||
assert!(network.get("status").is_none());
|
||||
assert_eq!(network["dst_endpoint"]["ip"], "93.184.216.34");
|
||||
assert_eq!(network["dst_endpoint"]["port"], 443);
|
||||
assert_eq!(network["actor"]["process"]["name"], "/usr/bin/curl");
|
||||
assert_eq!(network["firewall_rule"]["name"], "bypass-detect");
|
||||
assert_eq!(network["firewall_rule"]["type"], "nftables");
|
||||
assert_eq!(network["observation_point_id"], 3);
|
||||
assert!(
|
||||
network["message"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.contains("action=reject")
|
||||
);
|
||||
|
||||
let finding = serde_json::to_value(finding).unwrap();
|
||||
assert_eq!(finding["class_name"], "Detection Finding");
|
||||
assert_eq!(finding["action"], "Denied");
|
||||
assert_eq!(finding["disposition"], "Blocked");
|
||||
assert_eq!(finding["severity"], "Medium");
|
||||
assert_eq!(finding["confidence"], "High");
|
||||
assert_eq!(finding["is_alert"], true);
|
||||
assert_eq!(finding["finding_info"]["uid"], "bypass-detect");
|
||||
assert_eq!(finding["finding_info"]["title"], "Proxy Bypass Detected");
|
||||
assert_eq!(finding["evidences"][0]["data"]["dst_port"], "443");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_process_identity_surfaces_ambiguous_shared_socket() {
|
||||
use std::ffi::CString;
|
||||
|
||||
@@ -214,6 +214,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files.
|
||||
| server.oidc.rolesClaim | string | `""` | Dot-separated path to the roles array in the JWT claims. Keycloak: "realm_access.roles", Entra ID: "roles", Okta: "groups". |
|
||||
| server.oidc.scopesClaim | string | `""` | Dot-separated path to the scopes array in the JWT claims. |
|
||||
| server.oidc.userRole | string | `""` | Role name for standard user access. |
|
||||
| server.policyValidationFailureMode | string | `"fail_closed"` | Posture when a candidate sandbox policy fails validation. `fail_closed` deactivates the previous policy; `retain_last_valid` keeps it active. |
|
||||
| server.providerTokenGrants.spiffe.enabled | bool | `false` | Mount the SPIFFE Workload API socket into sandbox pods for dynamic provider token grants. |
|
||||
| server.providerTokenGrants.spiffe.workloadApiSocketPath | string | `"/spiffe-workload-api/spire-agent.sock"` | Path to the SPIFFE Workload API socket mounted into sandbox pods. |
|
||||
| server.sandboxImage | string | `"ghcr.io/nvidia/openshell-community/sandboxes/base:latest"` | Default sandbox image used when requests do not specify one. |
|
||||
|
||||
@@ -33,6 +33,11 @@ data:
|
||||
{{- end }}
|
||||
log_level = {{ .Values.server.logLevel | quote }}
|
||||
sandbox_namespace = {{ include "openshell.sandboxNamespace" . | quote }}
|
||||
{{- $policyValidationFailureMode := .Values.server.policyValidationFailureMode }}
|
||||
{{- if not (has $policyValidationFailureMode (list "fail_closed" "retain_last_valid")) }}
|
||||
{{- fail "server.policyValidationFailureMode must be fail_closed or retain_last_valid" }}
|
||||
{{- end }}
|
||||
policy_validation_failure_mode = {{ $policyValidationFailureMode | quote }}
|
||||
default_image = {{ .Values.server.sandboxImage | quote }}
|
||||
{{- if include "openshell.supervisorImageOverrideEnabled" . }}
|
||||
supervisor_image = {{ include "openshell.supervisorImage" . | quote }}
|
||||
|
||||
@@ -229,6 +229,22 @@ tests:
|
||||
path: data["gateway.toml"]
|
||||
pattern: 'grpc_rate_limit_window_seconds\s*='
|
||||
|
||||
- it: renders fail-closed policy validation posture by default
|
||||
template: templates/gateway-config.yaml
|
||||
asserts:
|
||||
- matchRegex:
|
||||
path: data["gateway.toml"]
|
||||
pattern: '(?ms)\[openshell\.gateway\].*?policy_validation_failure_mode\s*=\s*"fail_closed"'
|
||||
|
||||
- it: renders retain-last-valid policy validation posture
|
||||
template: templates/gateway-config.yaml
|
||||
set:
|
||||
server.policyValidationFailureMode: retain_last_valid
|
||||
asserts:
|
||||
- matchRegex:
|
||||
path: data["gateway.toml"]
|
||||
pattern: '(?ms)\[openshell\.gateway\].*?policy_validation_failure_mode\s*=\s*"retain_last_valid"'
|
||||
|
||||
- it: renders the gRPC rate limit under [openshell.gateway] when both values are positive
|
||||
template: templates/gateway-config.yaml
|
||||
set:
|
||||
|
||||
@@ -229,6 +229,9 @@ server:
|
||||
# -- Enable plaintext HTTP routing for loopback sandbox service URLs on
|
||||
# TLS-enabled gateways.
|
||||
enableLoopbackServiceHttp: true
|
||||
# -- Posture when a candidate sandbox policy fails validation. `fail_closed`
|
||||
# deactivates the previous policy; `retain_last_valid` keeps it active.
|
||||
policyValidationFailureMode: fail_closed
|
||||
# Optional gateway-wide gRPC request rate limit. Applies only to gRPC API
|
||||
# traffic after protocol multiplexing; health, metrics, and loopback service
|
||||
# HTTP routes are not rate limited. Both values must be positive to enable the
|
||||
|
||||
@@ -79,6 +79,10 @@ compute_drivers = ["kubernetes"]
|
||||
sandbox_namespace = "openshell"
|
||||
ssh_session_ttl_secs = 3600
|
||||
|
||||
# Reject invalid policy generations securely by default. Set
|
||||
# "retain_last_valid" only when availability takes priority.
|
||||
policy_validation_failure_mode = "fail_closed"
|
||||
|
||||
# Subject Alternative Names baked into the gateway server certificate.
|
||||
# Wildcard DNS SANs (e.g. "*.dev.openshell.localhost") also enable sandbox
|
||||
# service URLs under that domain.
|
||||
@@ -180,6 +184,8 @@ phases = ["validate"]
|
||||
|
||||
Local Docker, Podman, and VM gateways can also set `[openshell.gateway.mtls_auth] enabled = true` to authenticate CLI callers from verified client certificates. Kubernetes deployments must leave this unset and use OIDC or a trusted access proxy; the Helm chart does not render this table.
|
||||
|
||||
`[openshell.gateway] policy_validation_failure_mode` controls what sandbox supervisors do when a complete candidate policy fails runtime validation. The default, `fail_closed`, deactivates the previous network policy, closes relays pinned to it, and denies new egress until a valid generation loads. `retain_last_valid` leaves the previous valid generation active. Both modes reject the candidate atomically; startup always fails closed when no previous valid generation exists. Gateway mutation paths that can preflight a known effective scope reject invalid candidates before persistence and leave the active policy unchanged regardless of this setting. Changing the value requires restarting the gateway so it can reload `gateway.toml` and distribute the new posture to sandbox supervisors.
|
||||
|
||||
`[openshell.gateway.gateway_jwt] ttl_secs` controls gateway-minted sandbox JWT lifetime. When omitted, it defaults to `0`: the token `exp` claim and `expires_at_ms` response field become `0`, and the sandbox JWT does not expire. Use that default only for local single-player Docker, Podman, or VM gateways. Kubernetes and other shared deployments should set a positive TTL; Helm renders `3600` seconds by default, and the gateway logs a warning when a Kubernetes gateway uses `0`.
|
||||
|
||||
`[openshell.gateway.auth] allow_unauthenticated_users = true` is an unsafe local-development and trusted-proxy escape hatch. It accepts user-facing CLI/API calls without OIDC or mTLS credentials while sandbox supervisors still authenticate with gateway-minted sandbox JWTs. Leave it false for shared and production gateways.
|
||||
|
||||
@@ -223,7 +223,7 @@ REST allow rules match HTTP requests by method, path, and optional query paramet
|
||||
| Field | Type | Required | Description |
|
||||
|---|---|---|---|
|
||||
| `method` | string | Yes | HTTP method to allow (for example, `GET`, `POST`). `*` matches any method. |
|
||||
| `path` | string | Yes | URL path pattern. Supports `*` and `**` glob syntax. |
|
||||
| `path` | string | Yes | URL path glob. `*` and `**` match zero or more characters and may cross `/`; `?` matches one character; bracket classes such as `[0-9]` and `[!0]` are supported. |
|
||||
| `query` | map | No | Query parameter matchers keyed by decoded param name. Matcher value can be a glob string (`tag: "foo-*"`) or an object with `any` (`tag: { any: ["foo-*", "bar-*"] }`). |
|
||||
|
||||
Example REST allow rules:
|
||||
|
||||
@@ -61,8 +61,7 @@ network_middlewares:
|
||||
|
||||
Static sections are locked at sandbox creation. Changing them requires destroying and recreating the sandbox.
|
||||
Dynamic sections can be updated on a running sandbox with `openshell policy update` for incremental merges or `openshell policy set` for full replacement, and take effect without restarting.
|
||||
When a hot reload changes rules on an active HTTP L7 endpoint, existing keep-alive tunnels are closed before forwarding another parsed request. Credential-injection-only HTTP passthrough tunnels use the same reload boundary. Most HTTP clients reconnect automatically, and the next request is evaluated against the current policy.
|
||||
Raw streams are connection-scoped and outside L7 live-reload guarantees. This includes `tls: skip`, non-HTTP TCP payloads, HTTP upgrades such as WebSocket, and long-lived response streams such as SSE. A reload applies to the next connection or next parsed HTTP request; it does not interrupt an already-forwarded raw stream. Use `protocol: websocket` when policy should stay attached to the RFC 6455 upgrade and client text messages after the allowed upgrade. Add `websocket_credential_rewrite: true` only when the relay should rewrite credential placeholders in client-to-server WebSocket text messages. Add `request_body_credential_rewrite: true` only on inspected REST endpoints that need OpenShell to rewrite placeholders in supported text request bodies.
|
||||
When a hot reload changes rules, the supervisor publishes a new policy generation and closes connections pinned to the previous generation. This includes HTTP keep-alive tunnels, `tls: skip`, non-HTTP payloads, HTTP upgrades such as WebSocket, and long-lived response streams such as SSE. Most clients reconnect automatically, and the next connection or request is evaluated against the current policy. Use `protocol: websocket` when policy should stay attached to the RFC 6455 upgrade and client text messages after the allowed upgrade. Add `websocket_credential_rewrite: true` only when the relay should rewrite credential placeholders in client-to-server WebSocket text messages. Add `request_body_credential_rewrite: true` only on inspected REST endpoints that need OpenShell to rewrite placeholders in supported text request bodies.
|
||||
|
||||
| Section | Type | Description |
|
||||
|---|---|---|
|
||||
@@ -202,6 +201,44 @@ The following steps outline the hot-reload policy update workflow.
|
||||
openshell policy list <name>
|
||||
```
|
||||
|
||||
### Validation failures
|
||||
|
||||
OpenShell validates a complete candidate policy before activating any part of it. Endpoints may overlap when their connection and request-processing metadata agree. For example, two `api.example.com:443` REST entries can contribute different allow and deny rules when they use the same TLS, destination, credential, parser, and enforcement settings. A plain L4 endpoint may overlap an L7 endpoint because it authorizes the destination without contributing request-processing metadata. A more-specific path endpoint may override request-processing metadata from a broader endpoint, such as a `/graphql` GraphQL endpoint alongside a general REST endpoint for the same host. OpenShell rejects the candidate when overlapping exact or wildcard host selectors can both contribute equally specific endpoint configuration and disagree on those fields.
|
||||
|
||||
When the gateway knows the affected sandbox scope, it validates the complete
|
||||
effective candidate before persistence. This covers direct policy replacement,
|
||||
incremental merges and proposal approvals, provider attachment, and
|
||||
provider-profile updates that fan out to attached sandboxes. An ambiguity
|
||||
failure returns `FAILED_PRECONDITION`; OpenShell stores no invalid policy
|
||||
revision and does not partially apply a profile update. Supervisor validation
|
||||
remains a defense-in-depth boundary for startup, concurrent changes, and policy
|
||||
sources outside those mutation paths.
|
||||
|
||||
A gateway preflight rejection leaves the currently active policy unchanged
|
||||
regardless of failure mode because the candidate is never persisted or
|
||||
distributed. If a candidate reaches a supervisor and fails runtime validation,
|
||||
the gateway's `policy_validation_failure_mode` configuration determines the
|
||||
supervisor posture. Set it under `[openshell.gateway]` in `gateway.toml`. Its
|
||||
default is `fail_closed`:
|
||||
|
||||
```toml
|
||||
[openshell.gateway]
|
||||
policy_validation_failure_mode = "fail_closed"
|
||||
```
|
||||
|
||||
In `fail_closed` mode, the supervisor publishes a quarantine generation, denies new egress, and closes connections pinned to the previous generation. The previous policy is not active. A later valid policy exits quarantine automatically.
|
||||
|
||||
Operators that explicitly prioritize availability can retain the previous generation:
|
||||
|
||||
```toml
|
||||
[openshell.gateway]
|
||||
policy_validation_failure_mode = "retain_last_valid"
|
||||
```
|
||||
|
||||
In `retain_last_valid` mode, the rejected candidate remains inactive and the previous valid generation remains active. If no previous valid generation exists, such as during initial startup, OpenShell still fails closed. Restart the gateway after changing `gateway.toml`; connected sandbox supervisors receive the configured posture from the restarted gateway. Individual sandboxes cannot override it.
|
||||
|
||||
OCSF configuration and finding events identify the rejected candidate, validation rationale, configured and effective modes, active generation, and whether the previous policy is active. When `retain_last_valid` is configured without a previous valid generation, the effective mode remains `fail_closed`. Connection denials during quarantine include the validation failure as their policy denial rationale.
|
||||
|
||||
## Incremental Policy Updates
|
||||
|
||||
Use `openshell policy update` when you want to merge network policy changes into the current live policy instead of replacing the whole YAML document. This command only updates the dynamic `network_policies` section.
|
||||
@@ -331,13 +368,14 @@ means:
|
||||
- match the endpoint `api.github.com:443`.
|
||||
- match HTTP method `POST`.
|
||||
- match paths like `/repos/acme/issues`.
|
||||
- do not match deeper paths like `/repos/acme/project/issues/123` because `*` matches one path segment.
|
||||
- also match deeper paths when the surrounding literals align, because `*` may include `/`.
|
||||
|
||||
Path globs follow the same semantics as YAML allow and deny rules:
|
||||
|
||||
- `*` matches one path segment.
|
||||
- `**` matches any number of segments.
|
||||
- `/repos/*/issues` matches one repository owner or name segment in the middle.
|
||||
- `*` and `**` match zero or more characters and may cross `/` boundaries.
|
||||
- `?` matches exactly one character.
|
||||
- bracket classes such as `[0-9]` and negated classes such as `[!0]` are supported.
|
||||
- `/repos/*/issues` matches any intervening text, including multiple path segments.
|
||||
- `/repos/**` matches everything under `/repos/`.
|
||||
|
||||
The rule-level commands only modify method and path constraints. They do not change binaries, hostnames, ports, protocol settings, or WebSocket message payload matching.
|
||||
|
||||
@@ -1950,15 +1950,14 @@ def test_host_wildcard_rejects_deep_subdomain(
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def test_overlapping_policies_do_not_crash_opa(
|
||||
def test_overlapping_policies_with_conflicting_destination_metadata_are_rejected(
|
||||
sandbox: Callable[..., Sandbox],
|
||||
) -> None:
|
||||
"""OVL-1: Two policies covering the same host:port must not crash OPA.
|
||||
"""OVL-1: Conflicting metadata on the same host:port fails closed.
|
||||
|
||||
After a draft rule approval, the merged policy can contain two entries
|
||||
for the same (host, port). The OPA engine must handle this without
|
||||
a 'duplicated definition of local variable' error. This test creates
|
||||
the overlap directly to simulate the post-approval state.
|
||||
One endpoint permits any resolved address while the other constrains
|
||||
``allowed_ips``. The complete candidate is ambiguous and must not activate
|
||||
either entry.
|
||||
"""
|
||||
policy = _base_policy(
|
||||
network_policies={
|
||||
@@ -1992,8 +1991,9 @@ def test_overlapping_policies_do_not_crash_opa(
|
||||
args=(_PROXY_HOST, _PROXY_PORT, _SANDBOX_IP, _FORWARD_PROXY_PORT),
|
||||
)
|
||||
assert result.exit_code == 0, result.stderr
|
||||
assert "200" in result.stdout, (
|
||||
f"Overlapping policies should not crash; expected 200, got: {result.stdout}"
|
||||
assert "403" in result.stdout, (
|
||||
"Conflicting overlapping policies should fail closed; "
|
||||
f"expected 403, got: {result.stdout}"
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -123,6 +123,11 @@ name = "workspace_lifecycle"
|
||||
path = "tests/workspace_lifecycle.rs"
|
||||
required-features = ["e2e"]
|
||||
|
||||
[[test]]
|
||||
name = "proxy_egress_pipeline"
|
||||
path = "tests/proxy_egress_pipeline.rs"
|
||||
required-features = ["e2e-host-gateway"]
|
||||
|
||||
[[test]]
|
||||
name = "gpu"
|
||||
path = "tests/gpu.rs"
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -373,7 +373,7 @@ def proxy_parts():
|
||||
raise RuntimeError(f"invalid proxy URL: {{proxy_url!r}}")
|
||||
return parsed.hostname, parsed.port or 80
|
||||
|
||||
def connect_with_retry(host, port, timeout_seconds=20):
|
||||
def proxy_socket_with_retry(host, port, mode, timeout_seconds=20):
|
||||
proxy_host, proxy_port = proxy_parts()
|
||||
target = f"{{host}}:{{port}}"
|
||||
deadline = time.monotonic() + timeout_seconds
|
||||
@@ -382,13 +382,14 @@ def connect_with_retry(host, port, timeout_seconds=20):
|
||||
sock = None
|
||||
try:
|
||||
sock = socket.create_connection((proxy_host, proxy_port), timeout=5)
|
||||
request = f"CONNECT {{target}} HTTP/1.1\r\nHost: {{target}}\r\n\r\n"
|
||||
sock.sendall(request.encode("ascii"))
|
||||
response = recv_until(sock, b"\r\n\r\n").decode("iso-8859-1", "replace")
|
||||
if response.startswith("HTTP/1.1 200") or response.startswith("HTTP/1.0 200"):
|
||||
return sock
|
||||
first_line = response.splitlines()[0] if response else "<empty response>"
|
||||
raise RuntimeError(f"proxy CONNECT failed: {{first_line}}")
|
||||
if mode == "connect":
|
||||
request = f"CONNECT {{target}} HTTP/1.1\r\nHost: {{target}}\r\n\r\n"
|
||||
sock.sendall(request.encode("ascii"))
|
||||
response = recv_until(sock, b"\r\n\r\n").decode("iso-8859-1", "replace")
|
||||
if not (response.startswith("HTTP/1.1 200") or response.startswith("HTTP/1.0 200")):
|
||||
first_line = response.splitlines()[0] if response else "<empty response>"
|
||||
raise RuntimeError(f"proxy CONNECT failed: {{first_line}}")
|
||||
return sock
|
||||
except (OSError, RuntimeError) as error:
|
||||
if sock is not None:
|
||||
sock.close()
|
||||
@@ -398,25 +399,28 @@ def connect_with_retry(host, port, timeout_seconds=20):
|
||||
|
||||
token = os.environ[TOKEN_ENV]
|
||||
payload = json.dumps({{"authorization": "Bearer " + token}}, sort_keys=True)
|
||||
key = base64.b64encode(os.urandom(16)).decode("ascii")
|
||||
|
||||
with connect_with_retry(HOST, PORT) as sock:
|
||||
request = (
|
||||
f"GET /ws HTTP/1.1\r\n"
|
||||
f"Host: {{HOST}}:{{PORT}}\r\n"
|
||||
"Upgrade: websocket\r\n"
|
||||
"Connection: Upgrade\r\n"
|
||||
f"Sec-WebSocket-Key: {{key}}\r\n"
|
||||
"Sec-WebSocket-Version: 13\r\n"
|
||||
"\r\n"
|
||||
)
|
||||
sock.sendall(request.encode("ascii"))
|
||||
response = recv_until(sock, b"\r\n\r\n").decode("iso-8859-1", "replace")
|
||||
if not response.startswith("HTTP/1.1 101"):
|
||||
raise RuntimeError("websocket upgrade failed")
|
||||
sock.sendall(masked_text_frame(payload))
|
||||
_, response_payload = read_frame(sock)
|
||||
print(response_payload.decode("utf-8"))
|
||||
results = {{}}
|
||||
for mode in ("connect", "forward"):
|
||||
key = base64.b64encode(os.urandom(16)).decode("ascii")
|
||||
with proxy_socket_with_retry(HOST, PORT, mode) as sock:
|
||||
request_target = "/ws" if mode == "connect" else f"http://{{HOST}}:{{PORT}}/ws"
|
||||
request = (
|
||||
f"GET {{request_target}} HTTP/1.1\r\n"
|
||||
f"Host: {{HOST}}:{{PORT}}\r\n"
|
||||
"Upgrade: websocket\r\n"
|
||||
"Connection: Upgrade\r\n"
|
||||
f"Sec-WebSocket-Key: {{key}}\r\n"
|
||||
"Sec-WebSocket-Version: 13\r\n"
|
||||
"\r\n"
|
||||
)
|
||||
sock.sendall(request.encode("ascii"))
|
||||
response = recv_until(sock, b"\r\n\r\n").decode("iso-8859-1", "replace")
|
||||
if not response.startswith("HTTP/1.1 101"):
|
||||
raise RuntimeError(f"{{mode}} websocket upgrade failed: {{response!r}}")
|
||||
sock.sendall(masked_text_frame(payload))
|
||||
_, response_payload = read_frame(sock)
|
||||
results[mode] = json.loads(response_payload.decode("utf-8"))
|
||||
print(json.dumps(results, sort_keys=True))
|
||||
"#,
|
||||
host = host,
|
||||
port = port,
|
||||
@@ -425,7 +429,7 @@ with connect_with_retry(HOST, PORT) as sock:
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn websocket_text_placeholder_is_rewritten_in_sandbox() {
|
||||
async fn websocket_text_placeholder_is_rewritten_through_both_adapters() {
|
||||
let _provider_lock = PROVIDER_LOCK
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
@@ -465,7 +469,14 @@ async fn websocket_text_placeholder_is_rewritten_in_sandbox() {
|
||||
assert!(
|
||||
guard
|
||||
.create_output
|
||||
.contains(r#"{"saw_placeholder": false, "saw_secret": true}"#),
|
||||
.contains(r#""connect": {"saw_placeholder": false, "saw_secret": true}"#),
|
||||
"expected CONNECT upstream to see only the resolved secret marker:\n{}",
|
||||
guard.create_output
|
||||
);
|
||||
assert!(
|
||||
guard
|
||||
.create_output
|
||||
.contains(r#""forward": {"saw_placeholder": false, "saw_secret": true}"#),
|
||||
"expected upstream to see only the resolved secret marker:\n{}",
|
||||
guard.create_output
|
||||
);
|
||||
|
||||
@@ -367,6 +367,10 @@ message GetSandboxConfigResponse {
|
||||
// Workspace the sandbox belongs to. Allows the supervisor to learn its
|
||||
// workspace context for subsequent workspace-scoped RPCs.
|
||||
string workspace = 10;
|
||||
// Gateway-configured posture for rejected policy generations. Valid values
|
||||
// are "fail_closed" and "retain_last_valid". Unknown or empty values must
|
||||
// be treated as fail_closed by the supervisor.
|
||||
string policy_validation_failure_mode = 11;
|
||||
}
|
||||
|
||||
// Connection details for one operator-registered supervisor middleware service.
|
||||
|
||||
Reference in New Issue
Block a user