mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-25 06:42:52 +08:00
docs(rfc): update sandbox proxy adapter proposal
Signed-off-by: John Myers <johntmyers@users.noreply.github.com>
This commit is contained in:
John Myers
@@ -1,420 +0,0 @@
|
||||
---
|
||||
authors:
|
||||
- "@johntmyers"
|
||||
state: draft
|
||||
links:
|
||||
- https://github.com/NVIDIA/OpenShell/issues/1107
|
||||
- https://github.com/NVIDIA/OpenShell/pull/1083
|
||||
- https://github.com/NVIDIA/OpenShell/pull/1151
|
||||
---
|
||||
|
||||
# RFC 0004 - Sandbox Proxy Egress Adapter Model
|
||||
|
||||
<!--
|
||||
See rfc/README.md for the full RFC process and state definitions.
|
||||
-->
|
||||
|
||||
## Summary
|
||||
|
||||
Refactor sandbox egress around one shared authorization and relay pipeline.
|
||||
CONNECT, forward HTTP proxy, transparent native TCP, policy DNS,
|
||||
`inference.local`, and `policy.local` should become adapters that translate
|
||||
userland entry points into a common egress intent. Policy evaluation,
|
||||
destination validation, credential injection, request-body rewrite,
|
||||
WebSocket upgrade handling, protocol parsing, and relay ownership should happen
|
||||
behind shared boundaries.
|
||||
|
||||
This RFC keeps the main direction in this document. Supporting detail lives in:
|
||||
|
||||
- [Current shape appendix](current-shape.md)
|
||||
- [Technical design appendix](technical-design.md)
|
||||
- [Implementation plan](implementation-plan.md)
|
||||
|
||||
## Motivation
|
||||
|
||||
The sandbox proxy has accumulated separate egress paths for CONNECT, forward
|
||||
HTTP, local services, inference routing, endpoint metadata, credential
|
||||
injection, and L7 policy. That makes security changes easy to apply to one path
|
||||
and miss in another.
|
||||
|
||||
The target shape separates three concerns:
|
||||
|
||||
- **Adapters** describe how userland reached the proxy.
|
||||
- **Authorization** decides whether that egress is allowed and what endpoint
|
||||
behavior applies.
|
||||
- **Relays** own bytes, credentials, protocol parsing, and upstream dialing.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Replace CONNECT with forward proxy as the only explicit proxy mode.
|
||||
- Add SOCKS support.
|
||||
- Add HTTP/2 L7 parsing in this refactor.
|
||||
- Redesign provider credential storage.
|
||||
- Reintroduce iptables as the sandbox packet filtering backend.
|
||||
- Use eBPF connect hooks for transparent capture. Native TCP capture needs a
|
||||
userland proxy in the byte stream for TLS termination and protocol parsing.
|
||||
|
||||
## Proposal
|
||||
|
||||
### Migration Big Rocks
|
||||
|
||||
1. **Transport adapters.** CONNECT, forward HTTP, transparent TCP, policy DNS,
|
||||
and local service routes become small entry adapters. They parse their
|
||||
surface and produce either an egress intent, a local response, or a DNS
|
||||
answer. They do not duplicate policy evaluation.
|
||||
2. **Egress intent and decision.** The shared authorization boundary evaluates
|
||||
L4 policy once per connection intent and returns one decision containing the
|
||||
matched policy, matched endpoint, process identity, allowed IP metadata, TLS
|
||||
behavior, and protocol enforcement.
|
||||
3. **Relays.** Relays receive an authorized destination connector, not an
|
||||
already-open upstream socket. HTTP relays evaluate every request before
|
||||
dialing, own REST request-body credential rewrite, and hand allowed
|
||||
WebSocket upgrades to the WebSocket relay. TCP application parsers own their
|
||||
protocol loop and decide when a validated upstream connection is needed.
|
||||
|
||||
### Unified Adapter Flow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
User["Userland payload / harness"]
|
||||
|
||||
subgraph ExplicitProxy["Explicit proxy listener"]
|
||||
ProxyBytes["HTTP proxy bytes"]
|
||||
IsConnect{"CONNECT request?"}
|
||||
Connect["CONNECT adapter"]
|
||||
Forward["Forward HTTP adapter"]
|
||||
ProxyBytes --> IsConnect
|
||||
IsConnect -- Yes --> Connect
|
||||
IsConnect -- No --> Forward
|
||||
end
|
||||
|
||||
subgraph NativeTcp["Policy DNS + native TCP"]
|
||||
NameLookup["Userland DNS lookup"]
|
||||
PolicyDns["Policy DNS adapter"]
|
||||
DnsAnswer["DNS answer"]
|
||||
NativeConnect["Userland connect(ip:port)"]
|
||||
TcpAdapter["Transparent TCP adapter"]
|
||||
NameLookup --> PolicyDns
|
||||
PolicyDns --> DnsAnswer
|
||||
DnsAnswer --> NativeConnect
|
||||
NativeConnect --> TcpAdapter
|
||||
end
|
||||
|
||||
subgraph LocalApis["Sandbox-local services"]
|
||||
InferenceReq["Request to inference.local"]
|
||||
PolicyReq["Request to policy.local"]
|
||||
InferenceAdapter["Inference local adapter"]
|
||||
PolicyAdapter["Policy local adapter"]
|
||||
InferenceReq --> InferenceAdapter
|
||||
PolicyReq --> PolicyAdapter
|
||||
end
|
||||
|
||||
subgraph Shared["Shared egress pipeline"]
|
||||
Intent["Egress intent"]
|
||||
Auth["Authorize and select endpoint"]
|
||||
Decision["Egress decision"]
|
||||
Validate["Resolve and validate destination"]
|
||||
Relay["Relay"]
|
||||
Deny["Adapter-specific deny response"]
|
||||
Intent --> Auth
|
||||
Auth --> Allowed{"Allowed?"}
|
||||
Allowed -- No --> Deny
|
||||
Allowed -- Yes --> Decision
|
||||
Decision --> Validate
|
||||
Validate --> Relay
|
||||
end
|
||||
|
||||
User --> ProxyBytes
|
||||
User --> NameLookup
|
||||
User --> NativeConnect
|
||||
User --> InferenceReq
|
||||
User --> PolicyReq
|
||||
|
||||
Connect --> Intent
|
||||
Forward --> Intent
|
||||
TcpAdapter --> Intent
|
||||
InferenceAdapter --> InferenceResp["Local inference response"]
|
||||
PolicyAdapter --> PolicyResp["Local policy response"]
|
||||
```
|
||||
|
||||
### Relay Flow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start["Authorized egress + destination connector"]
|
||||
Start --> HasFirst{"First HTTP request already parsed?"}
|
||||
|
||||
HasFirst -- Yes --> ForwardMode{"Selected enforcement"}
|
||||
ForwardMode -- "L4 only" --> HttpCred["HTTP relay<br/>credential injection only"]
|
||||
ForwardMode -- "HTTP rules" --> HttpL7["HTTP relay<br/>REST/GraphQL/WebSocket policy"]
|
||||
ForwardMode -- "TCP app rules" --> BadForward["Deny: HTTP request for TCP app endpoint"]
|
||||
|
||||
HasFirst -- No --> Inspect["Inspect tunnel or native stream bytes"]
|
||||
Inspect --> SkipTls{"Endpoint says skip TLS handling?"}
|
||||
SkipTls -- Yes --> TcpBytes["TCP relay<br/>byte copy"]
|
||||
SkipTls -- No --> Peek["Peek client bytes"]
|
||||
Peek --> IsTls{"TLS ClientHello?"}
|
||||
IsTls -- Yes --> Tls["Shared TLS terminator"]
|
||||
IsTls -- No --> Readable["Readable client stream"]
|
||||
Tls --> Readable
|
||||
|
||||
Readable --> Mode{"Selected enforcement"}
|
||||
Mode -- "L4 only" --> SniffHttp{"Looks like HTTP?"}
|
||||
SniffHttp -- Yes --> HttpCred
|
||||
SniffHttp -- No --> TcpBytes
|
||||
|
||||
Mode -- "HTTP rules" --> MustHttp{"Looks like HTTP?"}
|
||||
MustHttp -- Yes --> HttpL7
|
||||
MustHttp -- No --> DenyHttp["Deny: expected HTTP"]
|
||||
|
||||
Mode -- "TCP app rules" --> TcpParser["TCP relay<br/>application parser owns loop"]
|
||||
|
||||
HttpCred --> Creds["Resolve and redact credentials"]
|
||||
HttpL7 --> CredsL7["Resolve and redact credentials"]
|
||||
CredsL7 --> ParseHttp["Parse and evaluate each HTTP request"]
|
||||
ParseHttp --> HttpAllowed{"Request allowed?"}
|
||||
HttpAllowed -- No --> HttpDeny["Local HTTP deny<br/>no upstream connect"]
|
||||
HttpAllowed -- Yes --> Rewrite["Rewrite configured credential slots"]
|
||||
Creds --> Rewrite
|
||||
Rewrite --> HttpDial["Connect or reuse upstream"]
|
||||
HttpDial --> HttpResponse["Write request and relay response"]
|
||||
HttpResponse --> Upgrade{"101 WebSocket upgrade?"}
|
||||
Upgrade -- No --> NextHttp["Continue HTTP request loop"]
|
||||
Upgrade -- Yes --> WsMode{"WebSocket inspection needed?"}
|
||||
WsMode -- No --> RawUpgrade["Raw upgraded stream"]
|
||||
WsMode -- Yes --> WsRelay["WebSocket relay<br/>text-frame rewrite / message policy"]
|
||||
NextHttp --> ParseHttp
|
||||
|
||||
TcpParser --> ParserDial["Parser dials upstream when protocol allows"]
|
||||
TcpBytes --> TcpDial["Connect upstream"]
|
||||
TcpDial --> ByteCopy["Copy bytes"]
|
||||
```
|
||||
|
||||
Relay rules:
|
||||
|
||||
- HTTP credential injection happens in both HTTP modes: L4-only HTTP and
|
||||
HTTP-inspected.
|
||||
- Credential injection includes request target, query, headers, opt-in REST
|
||||
request bodies, and opt-in client-to-server WebSocket text frames.
|
||||
- HTTP L7 policy is evaluated before upstream dial for each request.
|
||||
- WebSocket upgrade policy is evaluated as HTTP first. After an allowed `101`
|
||||
upgrade, the WebSocket relay owns frame parsing when text-frame credential
|
||||
rewrite, WebSocket transport policy, GraphQL-over-WebSocket policy, or safe
|
||||
compression handling is configured. Other upgraded streams remain raw.
|
||||
- Forward HTTP must stay in the shared HTTP relay loop. It must not evaluate
|
||||
one request and then switch to raw bidirectional copy. Keeping forward HTTP
|
||||
single-request with `Connection: close` is also acceptable, but the invariant
|
||||
is that no follow-on request bytes reach upstream unevaluated.
|
||||
- `protocol: tcp` means L4 authorization plus byte copy unless HTTP is detected
|
||||
for credential injection.
|
||||
- Future TCP application parsers, such as Redis or Postgres, own the full
|
||||
message loop and can parse multiple commands over one TCP session.
|
||||
|
||||
### CONNECT Adapter
|
||||
|
||||
CONNECT remains the standard explicit proxy tunnel for HTTPS and arbitrary TCP.
|
||||
It parses the CONNECT line into an egress intent, then waits for the shared
|
||||
relay to decide if and when an upstream connection should be opened.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client["Client sends CONNECT host:port"] --> Parse["Parse target"]
|
||||
Parse --> Intent["Build egress intent"]
|
||||
Intent --> Auth["Shared authorization"]
|
||||
Auth --> Allowed{"Destination allowed?"}
|
||||
Allowed -- No --> Deny["CONNECT deny response"]
|
||||
Allowed -- Yes --> Ready["Return tunnel-ready response"]
|
||||
Ready --> Relay["Relay inspects tunneled bytes"]
|
||||
Relay --> Dial["Relay or parser connects upstream when allowed"]
|
||||
```
|
||||
|
||||
CONNECT should stay because forward proxy is only a plaintext HTTP request
|
||||
format. CONNECT is still the generic explicit proxy mode for TLS and non-HTTP
|
||||
TCP clients.
|
||||
|
||||
### Forward HTTP Adapter
|
||||
|
||||
Forward HTTP is compatibility for clients that send absolute-form HTTP requests.
|
||||
The adapter parses the first request and hands it to the shared HTTP relay or
|
||||
an equivalent guarded single-request relay.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Req["Absolute-form HTTP request"] --> Parse["Parse URI and first request"]
|
||||
Parse --> Intent["Build egress intent"]
|
||||
Intent --> Auth["Shared authorization"]
|
||||
Auth --> Allowed{"Allowed?"}
|
||||
Allowed -- No --> Deny["HTTP deny response"]
|
||||
Allowed -- Yes --> Relay["Shared or guarded HTTP relay"]
|
||||
Relay --> Mode{"Connection mode"}
|
||||
Mode -- "Persistent" --> Loop["Evaluate every request on this connection"]
|
||||
Mode -- "Single request" --> Close["Force Connection: close"]
|
||||
```
|
||||
|
||||
### Transparent TCP Adapter
|
||||
|
||||
Transparent TCP supports native clients that do not know they are using a
|
||||
proxy. The capture mechanism should be network namespace interception into a
|
||||
userland proxy listener. Since main now uses nftables for sandbox bypass
|
||||
enforcement, transparent capture should be designed as nftables
|
||||
REDIRECT/TPROXY state in the inner sandbox network namespace, not as an
|
||||
iptables path.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Policy["Policy load / reload"] --> Register["Register native TCP names"]
|
||||
Lookup["Userland DNS lookup"] --> Dns["Policy DNS adapter"]
|
||||
Register --> Dns
|
||||
Dns --> Answer["Return approved IPs"]
|
||||
Answer --> Capture["Enable capture for active IP:port"]
|
||||
Connect["Userland connect(ip:port)"] --> Capture
|
||||
Capture --> Adapter["Transparent TCP adapter"]
|
||||
Adapter --> Intent["Build egress intent from original destination"]
|
||||
Intent --> Shared["Shared authorization and relay"]
|
||||
```
|
||||
|
||||
### Policy DNS
|
||||
|
||||
Policy DNS replaces static `/etc/hosts` snapshots for native TCP names. It is
|
||||
query-driven: check whether the name is policy-eligible, resolve through trusted
|
||||
DNS, filter returned IPs, publish the active endpoint mapping, and answer
|
||||
userland.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Query["DNS query from userland"] --> Adapter["Policy DNS adapter"]
|
||||
Adapter --> Known{"Registered native TCP policy name?"}
|
||||
Known -- No --> Refuse["NXDOMAIN / REFUSED / SERVFAIL"]
|
||||
Known -- Yes --> Upstream["Trusted upstream DNS lookup"]
|
||||
Upstream --> Filter["Filter answers against endpoint policy"]
|
||||
Filter --> Publish["Publish active mapping and capture rule"]
|
||||
Publish --> Answer["DNS answer"]
|
||||
```
|
||||
|
||||
The later `connect(ip:port)` still creates the egress intent and runs through
|
||||
normal authorization.
|
||||
|
||||
### Network Enforcement Substrate
|
||||
|
||||
Current main uses nftables for bypass enforcement. It accepts proxy-bound
|
||||
traffic and loopback, accepts established flows, then rejects and optionally
|
||||
logs other TCP/UDP traffic for the bypass monitor. That is enforcement, not
|
||||
native TCP capture.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Conn["Userland packet"] --> ProxyDest{"Proxy destination?"}
|
||||
ProxyDest -- Yes --> AcceptProxy["nftables accept"]
|
||||
ProxyDest -- No --> Capture{"Future native TCP capture match?"}
|
||||
Capture -- Yes --> Redirect["nftables redirect/TPROXY to transparent adapter"]
|
||||
Capture -- No --> Reject["nftables log + reject bypass"]
|
||||
Reject --> Monitor["Bypass monitor emits OCSF"]
|
||||
```
|
||||
|
||||
The transparent TCP work should extend this nftables model with explicit
|
||||
capture rules that run before the reject path and are scoped to active policy
|
||||
DNS mappings.
|
||||
|
||||
### Local Service Adapters
|
||||
|
||||
`inference.local` and `policy.local` are sandbox-local APIs. They should use
|
||||
the adapter model, but they do not represent normal external egress.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["Request to inference.local"] --> B["Inference local adapter"]
|
||||
B --> C{"TLS and inference context available?"}
|
||||
C -- No --> D["Local denial and log"]
|
||||
C -- Yes --> E["Terminate client TLS"]
|
||||
E --> F["Parse HTTP request"]
|
||||
F --> G{"Known inference route?"}
|
||||
G -- Yes --> H["Route through openshell-router"]
|
||||
H --> I["Strip caller auth and inject provider auth/model"]
|
||||
I --> J["Stream response with limits"]
|
||||
G -- No --> K["403 and close"]
|
||||
```
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["Request to policy.local"] --> B["Policy local adapter"]
|
||||
B --> C{"Local route"}
|
||||
C -- "Current policy" --> D["Policy snapshot response"]
|
||||
C -- "Recent denials" --> E["Bounded denial summaries"]
|
||||
C -- "Policy proposal" --> F["Validate and submit proposal"]
|
||||
D --> G["HTTP response"]
|
||||
E --> G
|
||||
F --> G
|
||||
```
|
||||
|
||||
### Deployment Modes
|
||||
|
||||
The first implementation can remain embedded in `openshell-sandbox`, but the
|
||||
proxy should be shaped around explicit runtime contracts.
|
||||
|
||||
| Mode | Shape | Main concern |
|
||||
|------|-------|--------------|
|
||||
| Embedded | Current sandbox process owns proxy modules | Lowest migration cost |
|
||||
| Standalone process | Sandbox supervisor launches a proxy binary | Clear process/API boundary |
|
||||
| Sidecar | Proxy runs outside the payload container but inside the sandbox boundary | Reliable process identity across namespaces |
|
||||
|
||||
A pluggable proxy must expose the configured userland surfaces, implement the
|
||||
gateway APIs it needs, and prove equivalent policy enforcement through tests.
|
||||
The nftables rules that force or reject userland traffic belong to the sandbox
|
||||
network boundary even if the proxy process later moves into a standalone binary
|
||||
or sidecar.
|
||||
|
||||
## Implementation plan
|
||||
|
||||
The migration plan lives in [implementation-plan.md](implementation-plan.md).
|
||||
The intended order is: first add regression coverage, then introduce the shared
|
||||
authorization result and destination validation, then preserve the current
|
||||
forward HTTP single-request/guarded-relay invariant, then add shared TLS
|
||||
handling, TCP parser boundaries, nftables-backed policy DNS capture, local
|
||||
service adapters, and finally the runtime boundary cleanup.
|
||||
|
||||
## Risks
|
||||
|
||||
- Tightening endpoint metadata failures from fail-open to deny may expose
|
||||
latent policy or Rego errors.
|
||||
- Deterministic endpoint selection may reject policies that currently load but
|
||||
only work by accident.
|
||||
- Transparent TCP capture adds network namespace interception complexity.
|
||||
- Transparent TCP capture must coexist with the current nftables bypass
|
||||
reject/log table without creating gaps where direct egress bypasses the proxy.
|
||||
- Sidecar mode needs a reliable identity source for binary/path scoped policy.
|
||||
- `policy.local` expands the sandbox-local control surface and needs strict
|
||||
route validation, body limits, redaction, and gateway authentication.
|
||||
|
||||
## Alternatives
|
||||
|
||||
- Keep patching each current proxy path separately. This has the lowest short
|
||||
term cost but keeps the security surface duplicated.
|
||||
- Replace CONNECT with forward proxy. This does not work for arbitrary TCP and
|
||||
is not a replacement for HTTPS tunnels.
|
||||
- Build only transparent TCP. This helps native clients but does not replace
|
||||
explicit proxy support used by common HTTP tooling.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Should overlapping endpoint metadata be rejected at policy load time, or
|
||||
should policy name plus endpoint index define precedence?
|
||||
2. Should missing TLS state fail closed for credential-capable or inspected
|
||||
endpoints?
|
||||
3. Should direct IP connects to a policy-DNS-resolved TCP endpoint be accepted,
|
||||
or should DNS query correlation be required for stricter modes?
|
||||
4. What TTL cap and stale-generation grace period should policy DNS use?
|
||||
5. Which process identity source should sidecar mode use when it cannot inspect
|
||||
payload process metadata through local `/proc`?
|
||||
6. Which proxy capabilities should be negotiated with the gateway at startup?
|
||||
|
||||
## Expected result
|
||||
|
||||
Adding a new HTTP-family protocol parser should require parser code, policy
|
||||
schema/Rego support, tests, and docs. It should not require new CONNECT and
|
||||
forward-proxy branches. REST, GraphQL, WebSocket upgrade policy, request-body
|
||||
credential rewrite, and WebSocket text-frame rewrite should all remain behind
|
||||
the shared HTTP/WebSocket relay boundary.
|
||||
|
||||
Adding a native TCP application parser should require policy DNS/capture
|
||||
support, a TCP application parser, policy rules, tests, and docs. Plain
|
||||
`protocol: tcp` remains L4 authorization plus byte relay.
|
||||
@@ -1,167 +0,0 @@
|
||||
# Current Shape Appendix
|
||||
|
||||
This appendix records the current proxy shape and the review findings that
|
||||
motivate the adapter model. The main RFC intentionally keeps these details out
|
||||
of the direction document.
|
||||
|
||||
## Current Entry Points
|
||||
|
||||
The sandbox proxy currently handles multiple userland-facing paths in the same
|
||||
large module:
|
||||
|
||||
- CONNECT proxy traffic for HTTPS and generic TCP tunnels.
|
||||
- Forward HTTP proxy traffic for absolute-form HTTP requests.
|
||||
- Local service routes such as `inference.local`.
|
||||
- Network namespace bypass enforcement through nftables reject/log rules.
|
||||
- Policy and endpoint metadata lookups through OPA/Rego.
|
||||
- DNS resolution and endpoint validation for CONNECT and forward HTTP egress.
|
||||
- Credential injection and redaction for provider-backed HTTP egress.
|
||||
- Opt-in REST request-body credential rewrite.
|
||||
- L7 REST, GraphQL, WebSocket, and GraphQL-over-WebSocket enforcement.
|
||||
|
||||
The issue is not that these features exist. The issue is that entry mechanisms,
|
||||
policy evaluation, endpoint metadata lookup, credential injection, and byte
|
||||
relay decisions are interleaved.
|
||||
|
||||
## Current CONNECT Shape
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client["Client CONNECT host:port"] --> Parse["Parse CONNECT target"]
|
||||
Parse --> L4["Evaluate network policy"]
|
||||
L4 --> Allowed{"Allowed?"}
|
||||
Allowed -- No --> Deny["CONNECT denial"]
|
||||
Allowed -- Yes --> Meta["Query endpoint metadata"]
|
||||
Meta --> Config{"L7 or credential config?"}
|
||||
Config -- No --> Raw["Open upstream and copy bytes"]
|
||||
Config -- Yes --> Tunnel["Return tunnel-ready response"]
|
||||
Tunnel --> Inspect["Parse tunneled HTTP when possible"]
|
||||
Inspect --> L7["Evaluate HTTP policy"]
|
||||
L7 --> Inject["Inject credentials if configured"]
|
||||
Inject --> Upstream["Write upstream and relay response"]
|
||||
```
|
||||
|
||||
This path has the strongest HTTP relay behavior because it can keep parsing
|
||||
requests on a long-lived tunnel and enforce L7 rules per request.
|
||||
|
||||
## Current Forward HTTP Shape
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client["Absolute-form HTTP request"] --> Parse["Parse first request"]
|
||||
Parse --> L4["Evaluate network policy"]
|
||||
L4 --> Allowed{"Allowed?"}
|
||||
Allowed -- No --> Deny["HTTP denial"]
|
||||
Allowed -- Yes --> L7{"Matching L7 endpoint?"}
|
||||
L7 -- Yes --> Eval["Evaluate REST/GraphQL/WebSocket policy"]
|
||||
Eval --> Rewrite["Rewrite to origin-form + configured credentials"]
|
||||
L7 -- No --> Rewrite
|
||||
Rewrite --> Close["Force Connection: close except WebSocket upgrade"]
|
||||
Close --> Upstream["Open upstream"]
|
||||
Upstream --> Relay["Guarded HTTP relay / upgrade relay"]
|
||||
```
|
||||
|
||||
The latest main branch no longer has the old raw-copy-after-first-request shape
|
||||
for ordinary forward HTTP. It rewrites ordinary requests with `Connection:
|
||||
close`, uses guarded HTTP relay helpers for body handling, and sends allowed
|
||||
WebSocket upgrades through the same upgrade relay. That is a narrower surface
|
||||
than the historical bidirectional copy, but it is still implemented separately
|
||||
from the CONNECT relay path.
|
||||
|
||||
## Current Network Namespace Enforcement
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start["Process in sandbox network namespace"] --> Dest{"Destination"}
|
||||
Dest -- "Proxy host_ip:port" --> Proxy["Accept to sandbox proxy"]
|
||||
Dest -- "Loopback" --> Loopback["Accept loopback"]
|
||||
Dest -- "Established/related" --> Established["Accept response packet"]
|
||||
Dest -- "Other TCP/UDP" --> Reject["nftables log + reject"]
|
||||
Reject --> Monitor["Bypass monitor reads dmesg"]
|
||||
Monitor --> OCSF["OCSF network + detection events"]
|
||||
```
|
||||
|
||||
The sandbox now installs an `inet` nftables filter table for bypass
|
||||
enforcement. The table accepts proxy-bound traffic, loopback, and established
|
||||
flows, then rejects and optionally logs other TCP/UDP traffic. It does not
|
||||
currently redirect native TCP connections into the proxy.
|
||||
|
||||
## Current Local Service Shape
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Request["Request to local name"] --> Match{"Known local route?"}
|
||||
Match -- "inference.local" --> Inference["Inference routing logic"]
|
||||
Match -- "policy.local" --> Policy["Policy local logic"]
|
||||
Match -- No --> External["Normal egress path"]
|
||||
Inference --> LocalResponse["Local response"]
|
||||
Policy --> LocalResponse
|
||||
```
|
||||
|
||||
Local routes are userland-facing proxy surfaces. They should stay distinct from
|
||||
external egress while still fitting the adapter model.
|
||||
|
||||
## Findings To Preserve
|
||||
|
||||
### Invariant: forward proxy must not relay unevaluated follow-on HTTP bytes
|
||||
|
||||
The historical forward path evaluated at most the first absolute-form request,
|
||||
rewrote it, then switched to bidirectional copy. Bytes already buffered after
|
||||
the first header block, or later pipelined requests on the same client/upstream
|
||||
connection, could reach upstream without the CONNECT L7 relay's per-request
|
||||
parser/evaluator.
|
||||
|
||||
Latest main mitigates this by forcing ordinary forward HTTP to one request per
|
||||
connection and by using guarded relay helpers. The adapter model should
|
||||
preserve the invariant either by keeping forward HTTP single-request/close or
|
||||
by passing the first parsed request into a shared HTTP relay loop.
|
||||
|
||||
### Endpoint config is not tied to deterministic matched policy
|
||||
|
||||
The policy name used for L4 authorization and logging can be selected through a
|
||||
different precedence rule than endpoint metadata. With overlapping host, port,
|
||||
and binary rules, allowed IPs, TLS behavior, enforcement, and
|
||||
`allow_encoded_slash` can come from a different endpoint than the policy name
|
||||
logged and used for L4 allow.
|
||||
|
||||
The adapter model requires authorization to return one decision with one
|
||||
deterministic matched endpoint.
|
||||
|
||||
### Endpoint metadata query failures fail open to L4 behavior
|
||||
|
||||
If endpoint metadata lookup fails, callers can interpret the result as no L7
|
||||
configuration and downgrade to credential-only or raw L4 relay.
|
||||
|
||||
The adapter model treats endpoint metadata as part of the authorization result.
|
||||
Failure to materialize required metadata should deny rather than erase extended
|
||||
configuration.
|
||||
|
||||
### Control-plane port block only applies on one resolution path
|
||||
|
||||
Blocked control-plane ports are enforced inside one allowed-IPs validation
|
||||
path, while the normal host-based path uses a different validation route.
|
||||
|
||||
The adapter model moves resolution, allowed IP checks, SSRF checks, and
|
||||
control-plane port blocks into shared destination validation.
|
||||
|
||||
## Existing Feature Inventory
|
||||
|
||||
The refactor should preserve:
|
||||
|
||||
- CONNECT explicit proxy support.
|
||||
- Forward HTTP explicit proxy support.
|
||||
- nftables bypass reject/log enforcement.
|
||||
- Provider credential injection and redaction.
|
||||
- REST request-body credential rewrite.
|
||||
- WebSocket text-frame credential rewrite.
|
||||
- REST endpoint method/path policy.
|
||||
- GraphQL L7 policy.
|
||||
- WebSocket transport and GraphQL-over-WebSocket policy.
|
||||
- Inference routing through `inference.local`.
|
||||
- Agent-facing policy routes through `policy.local`.
|
||||
- Timeout and resource tracking for client, upstream, and local service work.
|
||||
- Structured OCSF logging for network and HTTP policy outcomes.
|
||||
- SSRF and internal address protections.
|
||||
- Control-plane port protection.
|
||||
- `allowed_ips` endpoint restrictions.
|
||||
- TLS termination for inspectable client connections.
|
||||
@@ -0,0 +1,404 @@
|
||||
---
|
||||
authors:
|
||||
- "@johntmyers"
|
||||
state: draft
|
||||
links:
|
||||
- https://github.com/NVIDIA/OpenShell/issues/1107
|
||||
- https://github.com/NVIDIA/OpenShell/pull/1083
|
||||
- https://github.com/NVIDIA/OpenShell/pull/1151
|
||||
- https://github.com/NVIDIA/OpenShell/pull/1286
|
||||
- https://github.com/NVIDIA/OpenShell/pull/1511
|
||||
---
|
||||
|
||||
# RFC 0005 - Sandbox Proxy Egress Adapter Model
|
||||
|
||||
<!--
|
||||
See rfc/README.md for the full RFC process and state definitions.
|
||||
-->
|
||||
|
||||
## Summary
|
||||
|
||||
Refactor sandbox egress around one shared authorization and relay pipeline.
|
||||
CONNECT, forward HTTP, native TCP capture, policy DNS, `inference.local`,
|
||||
`policy.local`, and metadata loopback should become narrow adapters that
|
||||
translate userland entry points into common runtime intents. Policy evaluation,
|
||||
destination validation, credential injection, request-body rewrite, WebSocket
|
||||
handling, protocol parsing, and upstream dialing should happen behind shared
|
||||
boundaries.
|
||||
|
||||
The codebase has already moved in this direction by splitting networking into
|
||||
`openshell-supervisor-network` and process/netns work into
|
||||
`openshell-supervisor-process`. This RFC proposes the next internal boundary:
|
||||
make proxy entry mechanisms pluggable without duplicating authorization,
|
||||
destination validation, or relay behavior.
|
||||
|
||||
Supporting detail lives in:
|
||||
|
||||
- [Current shape appendix](current-shape.md)
|
||||
- [Technical design appendix](technical-design.md)
|
||||
- [Implementation plan](implementation-plan.md)
|
||||
|
||||
## Motivation
|
||||
|
||||
The sandbox proxy supports several connection surfaces: explicit CONNECT,
|
||||
forward HTTP, local inference and policy APIs, metadata loopback, TLS
|
||||
termination, REST and GraphQL inspection, WebSocket inspection, credential
|
||||
injection, and nftables-backed bypass detection. These features are valuable,
|
||||
but changes to policy and enforcement still tend to touch multiple entry paths.
|
||||
|
||||
The risk is asymmetric enforcement. A security fix can be added to CONNECT and
|
||||
missed in forward HTTP; endpoint metadata can be selected differently from the
|
||||
logged policy; a credential path can gain request-body or WebSocket support
|
||||
without the same behavior existing in another relay mode.
|
||||
|
||||
The target shape separates three concerns:
|
||||
|
||||
- **Adapters** describe how userland reached the networking component.
|
||||
- **Authorization** decides whether the egress is allowed and what endpoint
|
||||
behavior applies.
|
||||
- **Relays** own bytes, credentials, protocol parsing, and upstream dialing.
|
||||
|
||||
This also prepares the proxy for future deployment modes. Today the proxy runs
|
||||
inside the sandbox supervisor process. The networking leaf can already run in a
|
||||
network-only mode, and a future standalone binary or sidecar should be possible
|
||||
if it implements the same userland surfaces, gateway APIs, and policy
|
||||
enforcement contracts.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Replace CONNECT with forward proxy as the only explicit proxy mode.
|
||||
- Add SOCKS support.
|
||||
- Add HTTP/2 L7 parsing in this refactor. Inspected HTTP paths should continue
|
||||
to reject unsupported h2c upgrades instead of silently upgrading to raw
|
||||
traffic.
|
||||
- Redesign provider credential storage.
|
||||
- Reintroduce iptables as the sandbox packet filtering backend.
|
||||
- Use eBPF connect hooks for transparent capture. Native TCP capture needs a
|
||||
userland proxy in the byte stream for TLS termination and protocol parsing.
|
||||
|
||||
## Proposal
|
||||
|
||||
### Migration Big Rocks
|
||||
|
||||
1. **Transport and local-service adapters.** CONNECT, forward HTTP,
|
||||
transparent TCP, policy DNS, `inference.local`, `policy.local`, and metadata
|
||||
loopback become small adapters. They parse their surface and produce either
|
||||
an egress intent, a local response, or a DNS answer. They do not duplicate
|
||||
policy evaluation.
|
||||
2. **Egress intent and decision.** Shared authorization evaluates L4 policy and
|
||||
endpoint selection once per connection intent and returns one decision
|
||||
containing the matched policy, matched endpoint, process identity, allowed
|
||||
IP metadata, TLS behavior, protocol enforcement, and credential injection
|
||||
plan.
|
||||
3. **Relays.** Relays receive an authorized destination connector, not an
|
||||
already-open upstream socket. HTTP relays evaluate every request before
|
||||
upstream write. TCP relays copy bytes for L4-only endpoints or hand the
|
||||
stream to a protocol parser when endpoint policy requires native protocol
|
||||
enforcement.
|
||||
|
||||
### Unified Adapter Flow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
User["Userland payload / harness"]
|
||||
|
||||
subgraph ExplicitProxy["Explicit proxy listener"]
|
||||
ProxyBytes["HTTP proxy bytes"]
|
||||
IsConnect{"CONNECT request?"}
|
||||
Connect["CONNECT adapter"]
|
||||
Forward["Forward HTTP adapter"]
|
||||
ProxyBytes --> IsConnect
|
||||
IsConnect -- Yes --> Connect
|
||||
IsConnect -- No --> Forward
|
||||
end
|
||||
|
||||
subgraph NativeTcp["Policy DNS + native TCP"]
|
||||
NameLookup["Userland DNS lookup"]
|
||||
PolicyDns["Policy DNS adapter"]
|
||||
DnsAnswer["DNS answer + active mapping"]
|
||||
NativeConnect["Userland connect(ip:port)"]
|
||||
TcpAdapter["Transparent TCP adapter"]
|
||||
NameLookup --> PolicyDns
|
||||
PolicyDns --> DnsAnswer
|
||||
DnsAnswer --> NativeConnect
|
||||
NativeConnect --> TcpAdapter
|
||||
end
|
||||
|
||||
subgraph LocalApis["Sandbox-local services"]
|
||||
InferenceReq["Request to inference.local"]
|
||||
PolicyReq["Request to policy.local"]
|
||||
MetadataReq["Request to metadata loopback"]
|
||||
InferenceAdapter["Inference local adapter"]
|
||||
PolicyAdapter["Policy local adapter"]
|
||||
MetadataAdapter["Metadata loopback adapter"]
|
||||
InferenceReq --> InferenceAdapter
|
||||
PolicyReq --> PolicyAdapter
|
||||
MetadataReq --> MetadataAdapter
|
||||
end
|
||||
|
||||
subgraph Shared["Shared external egress pipeline"]
|
||||
Intent["EgressIntent"]
|
||||
Auth["Authorize and select endpoint"]
|
||||
Decision["EgressDecision"]
|
||||
Validate["Resolve and validate destination"]
|
||||
Relay["Relay"]
|
||||
Deny["Adapter-specific deny response"]
|
||||
Intent --> Auth
|
||||
Auth --> Allowed{"Allowed?"}
|
||||
Allowed -- No --> Deny
|
||||
Allowed -- Yes --> Decision
|
||||
Decision --> Validate
|
||||
Validate --> Relay
|
||||
end
|
||||
|
||||
User --> ProxyBytes
|
||||
User --> NameLookup
|
||||
User --> NativeConnect
|
||||
User --> InferenceReq
|
||||
User --> PolicyReq
|
||||
User --> MetadataReq
|
||||
|
||||
Connect --> Intent
|
||||
Forward --> Intent
|
||||
TcpAdapter --> Intent
|
||||
InferenceAdapter --> InferenceResp["Local inference response"]
|
||||
PolicyAdapter --> PolicyResp["Local policy response"]
|
||||
MetadataAdapter --> MetadataResp["Local metadata credential response"]
|
||||
```
|
||||
|
||||
Each adapter still owns its response shape. If authorization denies a CONNECT
|
||||
intent, the CONNECT adapter returns a tunnel denial. If forward HTTP is denied,
|
||||
the forward adapter returns an HTTP denial. If policy DNS refuses a name, it
|
||||
returns the appropriate DNS response. The shared layer decides the outcome; the
|
||||
adapter renders it for its protocol.
|
||||
|
||||
### Relay Flow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start["Authorized egress + destination connector"]
|
||||
Start --> FirstReq{"First HTTP request already parsed?"}
|
||||
|
||||
FirstReq -- Yes --> ForwardMode{"decision.endpoint.enforcement"}
|
||||
ForwardMode -- "None" --> HttpCred["HTTP relay<br/>credential injection only"]
|
||||
ForwardMode -- "Http" --> HttpL7["HTTP relay<br/>REST/GraphQL/WebSocket policy"]
|
||||
ForwardMode -- "TcpApplication" --> BadForward["Deny: HTTP request for TCP app endpoint"]
|
||||
|
||||
FirstReq -- No --> TlsPolicy{"TLS handling skipped?"}
|
||||
TlsPolicy -- Yes --> Readable["Readable client stream"]
|
||||
TlsPolicy -- No --> Peek["Peek client bytes"]
|
||||
Peek --> Tls{"TLS ClientHello?"}
|
||||
Tls -- Yes --> Terminate["Shared TLS terminator"]
|
||||
Tls -- No --> Readable
|
||||
Terminate --> Readable
|
||||
|
||||
Readable --> Enforce{"decision.endpoint.enforcement"}
|
||||
Enforce -- "None" --> Sniff{"Looks like HTTP?"}
|
||||
Sniff -- Yes --> HttpCred
|
||||
Sniff -- No --> TcpRelay["TcpRelay<br/>byte copy"]
|
||||
|
||||
Enforce -- "Http" --> MustHttp{"Looks like HTTP?"}
|
||||
MustHttp -- Yes --> HttpL7
|
||||
MustHttp -- No --> DenyHttp["Deny: expected HTTP"]
|
||||
|
||||
Enforce -- "TcpApplication" --> TcpParser["TcpRelay<br/>protocol parser owns loop"]
|
||||
|
||||
HttpCred --> ReqLoop["HTTP request loop"]
|
||||
HttpL7 --> ReqLoop
|
||||
ReqLoop --> ReqPolicy{"Request allowed?"}
|
||||
ReqPolicy -- No --> ReqDeny["Local HTTP deny<br/>no upstream write"]
|
||||
ReqPolicy -- Yes --> StaticCreds["Resolve static placeholders"]
|
||||
StaticCreds --> TokenGrant["Apply endpoint token grant if configured"]
|
||||
TokenGrant --> Rewrite["Rewrite configured credential slots"]
|
||||
Rewrite --> HttpDial["Connect or reuse upstream"]
|
||||
HttpDial --> HttpResponse["Write request and relay response"]
|
||||
HttpResponse --> Upgrade{"101 WebSocket upgrade?"}
|
||||
Upgrade -- No --> ReqLoop
|
||||
Upgrade -- Yes --> WsInspect{"WebSocket inspection or rewrite configured?"}
|
||||
WsInspect -- No --> RawUpgrade["Raw upgraded stream"]
|
||||
WsInspect -- Yes --> WsRelay["WebSocket relay<br/>text-frame rewrite / message policy"]
|
||||
|
||||
TcpParser --> ParserDial["Parser calls connector when protocol allows"]
|
||||
TcpRelay --> TcpDial["Connect upstream"]
|
||||
TcpDial --> ByteCopy["Copy bytes"]
|
||||
```
|
||||
|
||||
Relay rules:
|
||||
|
||||
- HTTP credential injection happens in both HTTP modes: L4-only HTTP and
|
||||
HTTP-inspected.
|
||||
- Credential injection includes static placeholder rewrite and endpoint-bound
|
||||
dynamic token grants. Token grants run after policy allow and before upstream
|
||||
write; failures deny without forwarding the request.
|
||||
- Static credential rewrite covers request target, query, headers, opt-in REST
|
||||
request bodies, and opt-in client-to-server WebSocket text frames.
|
||||
- HTTP L7 policy is evaluated before upstream write for each request.
|
||||
- WebSocket upgrade policy is evaluated as HTTP first. After an allowed `101`
|
||||
upgrade, the WebSocket relay owns frame parsing when text-frame credential
|
||||
rewrite, WebSocket transport policy, GraphQL-over-WebSocket policy, or safe
|
||||
compression handling is configured. Other upgraded streams remain raw.
|
||||
- Forward HTTP must stay in the shared HTTP relay loop or in an equivalent
|
||||
guarded single-request relay. It must not evaluate one request and then
|
||||
switch to raw bidirectional copy.
|
||||
- `protocol: tcp` or an omitted protocol means L4 authorization plus byte copy,
|
||||
except that HTTP-looking streams may still use HTTP credential injection.
|
||||
- Future native protocol parsers, such as Redis, Postgres, or MySQL, own the
|
||||
full message loop and can parse multiple commands or queries on one TCP
|
||||
session.
|
||||
|
||||
### Adapter Responsibilities
|
||||
|
||||
CONNECT remains the generic explicit proxy mode for HTTPS and arbitrary TCP.
|
||||
The CONNECT adapter parses `CONNECT host:port` into an `EgressIntent`, asks the
|
||||
shared authorization boundary for an `EgressDecision`, returns the tunnel-ready
|
||||
response only after the connection is allowed, and then hands the tunnel to the
|
||||
relay. The upstream connection is opened by the HTTP relay or TCP parser when
|
||||
payload policy allows it.
|
||||
|
||||
Forward HTTP is compatibility for clients that send absolute-form HTTP
|
||||
requests. The adapter parses the first request, rewrites proxy framing only at
|
||||
the relay boundary, rejects `https://` absolute-form requests, rejects
|
||||
unsupported h2c upgrades on inspected routes, and either stays in a shared HTTP
|
||||
request loop or forces `Connection: close` for a guarded single request.
|
||||
|
||||
Transparent TCP is for native clients that do not know they are using a proxy.
|
||||
It depends on policy DNS and nftables capture: DNS answers create active
|
||||
endpoint mappings, userland later calls `connect(ip:port)`, nftables redirects
|
||||
matching traffic to a userland listener, and the TCP adapter recovers the
|
||||
original destination before building an intent.
|
||||
|
||||
Policy DNS replaces static `/etc/hosts` snapshots for native TCP names. It is
|
||||
query-driven: check whether the name is policy-eligible, resolve through
|
||||
trusted DNS, filter returned IPs, publish the active endpoint mapping, and
|
||||
answer userland. The later `connect(ip:port)` still runs through normal
|
||||
authorization.
|
||||
|
||||
Local service adapters stay outside the normal external egress relay:
|
||||
`inference.local` routes chat, completion, model discovery, embeddings, and
|
||||
provider-specific inference traffic through the router with local limits;
|
||||
`policy.local` exposes current policy, denial summaries, proposal submission,
|
||||
and proposal wait routes; metadata loopback serves provider metadata
|
||||
credentials to SDKs that bypass HTTP proxy variables.
|
||||
|
||||
### Network Enforcement Substrate
|
||||
|
||||
Current main uses nftables for sandbox bypass enforcement. It accepts
|
||||
proxy-bound traffic, loopback, and established flows, then rejects and
|
||||
optionally logs other TCP/UDP traffic for the bypass monitor. That is
|
||||
enforcement, not native TCP capture.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Packet["Userland packet"] --> ProxyDest{"Proxy destination?"}
|
||||
ProxyDest -- Yes --> AcceptProxy["nftables accept"]
|
||||
ProxyDest -- No --> Capture{"Future native TCP capture match?"}
|
||||
Capture -- Yes --> Redirect["nftables redirect/TPROXY to transparent adapter"]
|
||||
Capture -- No --> Reject["nftables log + reject bypass"]
|
||||
Reject --> Monitor["Bypass monitor emits OCSF"]
|
||||
```
|
||||
|
||||
Transparent TCP work should extend this nftables model with explicit capture
|
||||
rules that run before the reject path and are scoped to active policy DNS
|
||||
mappings. It should not add a parallel iptables path.
|
||||
|
||||
### Deployment Modes
|
||||
|
||||
| Mode | Shape | Status |
|
||||
|------|-------|--------|
|
||||
| Embedded supervisor | `openshell-sandbox` orchestrates `openshell-supervisor-network` and `openshell-supervisor-process` | Current |
|
||||
| Network-only supervisor | Networking, policy, proxy, local services, and background tasks run without a payload process leaf | Current runtime mode |
|
||||
| Standalone proxy binary | Supervisor launches networking as a separate process with explicit APIs | Future packaging/API work |
|
||||
| Sidecar proxy | Proxy runs outside the payload container but inside the sandbox boundary | Future isolation mode |
|
||||
|
||||
A pluggable proxy must expose the right userland surfaces, implement the
|
||||
gateway APIs it needs, and prove equivalent policy enforcement through tests.
|
||||
The nftables rules that force or reject userland traffic belong to the sandbox
|
||||
network boundary even if the proxy process later moves into a standalone binary
|
||||
or sidecar.
|
||||
|
||||
## Implementation plan
|
||||
|
||||
The detailed migration plan lives in [implementation-plan.md](implementation-plan.md).
|
||||
The intended order is:
|
||||
|
||||
1. Add regression coverage around the current split, forward HTTP invariants,
|
||||
endpoint selection, token grants, WebSocket/body rewrite, metadata loopback,
|
||||
and nftables bypass enforcement.
|
||||
2. Introduce `EgressIntent` and `EgressDecision` inside
|
||||
`openshell-supervisor-network`.
|
||||
3. Move destination validation and endpoint metadata materialization behind the
|
||||
shared decision and connector boundary.
|
||||
4. Consolidate forward HTTP, CONNECT HTTP inspection, credential injection,
|
||||
request-body rewrite, and WebSocket handling behind shared HTTP/WebSocket
|
||||
relay code.
|
||||
5. Move TLS detection and termination ahead of the HTTP/TCP relay split.
|
||||
6. Add the TCP relay/parser boundary, then policy DNS and native TCP capture.
|
||||
7. Treat local services and deployment modes as explicit runtime contracts.
|
||||
|
||||
## Risks
|
||||
|
||||
- Tightening endpoint metadata failures from fail-open to deny may expose
|
||||
latent policy or Rego errors.
|
||||
- Deterministic endpoint selection may reject policies that currently load but
|
||||
only work by accident.
|
||||
- Token grants add a runtime dependency on SPIFFE Workload API and token
|
||||
endpoints. Failures should remain fail-closed and sanitized.
|
||||
- Transparent TCP capture adds network namespace interception complexity and
|
||||
must coexist with the nftables bypass reject/log table.
|
||||
- Sidecar mode needs a reliable identity source for binary/path scoped policy.
|
||||
- Metadata loopback and `policy.local` expand sandbox-local control surfaces
|
||||
and need strict route validation, body limits, redaction, and authentication
|
||||
boundaries.
|
||||
- Provider-composed policy rules use a reserved namespace. Decisions and logs
|
||||
must distinguish provider-derived policy from user-authored policy without
|
||||
exposing provider rules as editable sandbox proposals.
|
||||
|
||||
## Alternatives
|
||||
|
||||
### Keep patching each entry path
|
||||
|
||||
This has the lowest short-term cost but keeps security behavior duplicated
|
||||
across CONNECT, forward HTTP, and local services. It also makes future TCP
|
||||
application protocol support harder because each parser must be wired through
|
||||
multiple entry mechanisms.
|
||||
|
||||
### Replace CONNECT with forward proxy
|
||||
|
||||
Forward proxy only covers plaintext absolute-form HTTP requests. It is not a
|
||||
replacement for HTTPS tunnels, WebSocket tunnels, or arbitrary TCP clients.
|
||||
CONNECT should remain the generic explicit proxy mode.
|
||||
|
||||
### Build only transparent TCP
|
||||
|
||||
Transparent TCP helps native clients but does not replace explicit proxy
|
||||
support used by common HTTP tooling. It also requires policy DNS and nftables
|
||||
capture before it can safely preserve endpoint identity.
|
||||
|
||||
## Prior art
|
||||
|
||||
The current `openshell-supervisor-network` split is the immediate prior step:
|
||||
it already separates proxy, OPA, L7, inference routing, policy-local routes,
|
||||
TLS, and token grants from process supervision.
|
||||
|
||||
The current `openshell-supervisor-process` netns and bypass monitor are the
|
||||
packet-enforcement substrate. Transparent TCP should extend that nftables
|
||||
model rather than creating a second firewall path.
|
||||
|
||||
The existing L7 relay is the behavioral prior art for this RFC. It already
|
||||
proves per-request HTTP evaluation, GraphQL parsing, WebSocket frame handling,
|
||||
request-body rewrite, and token-grant injection can live behind relay
|
||||
boundaries.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Should overlapping endpoint metadata be rejected at policy load time, or
|
||||
should policy name plus endpoint index define precedence?
|
||||
2. Should direct IP connects to a policy-DNS-resolved TCP endpoint be accepted,
|
||||
or should DNS query correlation be required for stricter modes?
|
||||
3. What TTL cap and stale-generation grace period should policy DNS use?
|
||||
4. Which process identity source should sidecar mode use when it cannot inspect
|
||||
payload process metadata through local `/proc`?
|
||||
5. Which proxy capabilities should be negotiated with the gateway at startup?
|
||||
6. Should metadata loopback be modeled as an adapter inside
|
||||
`openshell-supervisor-network`, or remain orchestrated by `openshell-sandbox`
|
||||
with shared credential/provider helpers?
|
||||
@@ -0,0 +1,223 @@
|
||||
# Current Shape Appendix
|
||||
|
||||
This appendix records the current proxy shape and the review findings that
|
||||
motivate the adapter model. The main RFC intentionally keeps these details out
|
||||
of the direction document.
|
||||
|
||||
## Current Runtime Split
|
||||
|
||||
The proxy is no longer only a large module inside `openshell-sandbox`.
|
||||
Current main has three relevant runtime owners:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Sandbox["openshell-sandbox<br/>orchestrator"]
|
||||
Network["openshell-supervisor-network<br/>proxy, OPA, L7, TLS, inference,<br/>policy.local, token grants"]
|
||||
Process["openshell-supervisor-process<br/>process leaf, SSH, netns,<br/>nftables, bypass monitor"]
|
||||
Denials["Denial/activity aggregators"]
|
||||
Gateway["Gateway policy/provider APIs"]
|
||||
|
||||
Sandbox --> Network
|
||||
Sandbox --> Process
|
||||
Network --> Denials
|
||||
Process --> Denials
|
||||
Sandbox --> Gateway
|
||||
Network --> Gateway
|
||||
```
|
||||
|
||||
`openshell-sandbox` creates the shared network namespace, owns denial/activity
|
||||
channels, starts the policy poll loop, starts networking, starts the metadata
|
||||
loopback server when needed, and then optionally starts the process leaf. If
|
||||
`process_enabled` is false, the supervisor can run in network-only mode and
|
||||
keep networking/background tasks alive until shutdown.
|
||||
|
||||
`openshell-supervisor-network` owns the explicit proxy listener, OPA engine
|
||||
integration, L7 enforcement, TLS termination, inference routing, policy-local
|
||||
routes, identity cache, provider credential injection, and token grants.
|
||||
|
||||
`openshell-supervisor-process` owns process execution, SSH, network namespace
|
||||
helpers, nftables bypass rules, and the bypass monitor that turns nftables LOG
|
||||
entries into OCSF events.
|
||||
|
||||
## Current Userland-Facing Surfaces
|
||||
|
||||
The networking surface currently includes:
|
||||
|
||||
- CONNECT proxy traffic for HTTPS and generic TCP tunnels.
|
||||
- Forward HTTP proxy traffic for absolute-form HTTP requests.
|
||||
- `inference.local` for local inference routing.
|
||||
- `policy.local` for current policy, denial summaries, proposal submission,
|
||||
and proposal wait routes.
|
||||
- GCE metadata loopback for SDKs that bypass HTTP proxy variables.
|
||||
- nftables bypass enforcement for direct TCP/UDP egress that does not enter
|
||||
the proxy.
|
||||
- OPA/Rego policy and endpoint metadata lookups.
|
||||
- DNS resolution and endpoint validation for CONNECT and forward HTTP egress.
|
||||
- Static provider credential injection and redaction.
|
||||
- Endpoint-bound dynamic token grant injection.
|
||||
- Opt-in REST request-body credential rewrite.
|
||||
- L7 REST, GraphQL, WebSocket, and GraphQL-over-WebSocket enforcement.
|
||||
|
||||
The issue is not that these features exist. The issue is that entry mechanisms,
|
||||
policy evaluation, endpoint metadata lookup, credential injection, and byte
|
||||
relay decisions are still interleaved.
|
||||
|
||||
## Current CONNECT Shape
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client["Client CONNECT host:port"] --> Parse["Parse CONNECT target"]
|
||||
Parse --> L4["Evaluate network policy"]
|
||||
L4 --> Allowed{"Allowed?"}
|
||||
Allowed -- No --> Deny["CONNECT denial"]
|
||||
Allowed -- Yes --> Meta["Query endpoint metadata"]
|
||||
Meta --> Config{"L7, TLS, or credential config?"}
|
||||
Config -- No --> Tunnel["Return tunnel-ready response"]
|
||||
Config -- Yes --> Tunnel
|
||||
Tunnel --> Inspect["Inspect tunneled bytes when possible"]
|
||||
Inspect --> Relay["HTTP/WebSocket/TCP relay selection"]
|
||||
Relay --> Inject["Static credentials and token grants if configured"]
|
||||
Inject --> Upstream["Open upstream when relay policy allows"]
|
||||
```
|
||||
|
||||
CONNECT is still the strongest entry shape because the tunnel relay can keep
|
||||
parsing HTTP requests on long-lived connections and enforce request policy per
|
||||
request.
|
||||
|
||||
## Current Forward HTTP Shape
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client["Absolute-form HTTP request"] --> Parse["Parse first request"]
|
||||
Parse --> L4["Evaluate network policy"]
|
||||
L4 --> Allowed{"Allowed?"}
|
||||
Allowed -- No --> Deny["HTTP denial"]
|
||||
Allowed -- Yes --> L7{"Matching L7 endpoint?"}
|
||||
L7 -- Yes --> Eval["Evaluate REST/GraphQL/WebSocket policy"]
|
||||
Eval --> Guard["Reject unsupported h2c upgrade when inspected"]
|
||||
Guard --> Rewrite["Rewrite to origin-form + configured credentials"]
|
||||
L7 -- No --> Rewrite
|
||||
Rewrite --> Token["Apply token grant if endpoint-bound"]
|
||||
Token --> Close["Force Connection: close except WebSocket upgrade"]
|
||||
Close --> Upstream["Open upstream"]
|
||||
Upstream --> Relay["Guarded HTTP relay / upgrade relay"]
|
||||
```
|
||||
|
||||
Latest main no longer has the old raw-copy-after-first-request shape for
|
||||
ordinary forward HTTP. It rewrites ordinary requests with `Connection: close`,
|
||||
uses guarded HTTP relay helpers for body handling, rejects inspected h2c
|
||||
upgrades, injects token grants, and sends allowed WebSocket upgrades through
|
||||
the upgrade relay. That is a narrower surface than the historical bidirectional
|
||||
copy, but it is still orchestrated separately from the CONNECT relay path.
|
||||
|
||||
## Current Local Service Shape
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Request["Request to local name"] --> Match{"Known local route?"}
|
||||
Match -- "inference.local" --> Inference["Inference route adapter"]
|
||||
Match -- "policy.local" --> Policy["Policy local adapter"]
|
||||
Match -- "metadata loopback" --> Metadata["Metadata credential server"]
|
||||
Match -- No --> External["Normal egress path"]
|
||||
Inference --> InferenceResp["Local inference response"]
|
||||
Policy --> PolicyResp["Local policy response"]
|
||||
Metadata --> MetadataResp["Metadata response"]
|
||||
```
|
||||
|
||||
`inference.local` now covers buffered and streaming inference shapes including
|
||||
chat/completion routes, model discovery, embeddings, and provider-specific
|
||||
routes. `policy.local` supports the agentic approval loop: agents can submit
|
||||
narrow proposals and wait on approval/reload before retrying. Metadata
|
||||
loopback exists for provider credentials consumed by SDKs that do not honor
|
||||
HTTP proxy variables.
|
||||
|
||||
These are userland-facing network surfaces. They should stay distinct from
|
||||
external egress while still fitting the adapter model.
|
||||
|
||||
## Current Network Namespace Enforcement
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start["Process in sandbox network namespace"] --> Dest{"Destination"}
|
||||
Dest -- "Proxy host_ip:port" --> Proxy["Accept to sandbox proxy"]
|
||||
Dest -- "Loopback" --> Loopback["Accept loopback"]
|
||||
Dest -- "Established/related" --> Established["Accept response packet"]
|
||||
Dest -- "Other TCP/UDP" --> Reject["nftables log + reject"]
|
||||
Reject --> Monitor["Bypass monitor reads dmesg"]
|
||||
Monitor --> OCSF["OCSF network + detection events"]
|
||||
```
|
||||
|
||||
The process leaf installs an `inet` nftables filter table for bypass
|
||||
enforcement. The table accepts proxy-bound traffic, loopback, and established
|
||||
flows, then rejects and optionally logs other TCP/UDP traffic. It does not
|
||||
currently redirect native TCP connections into the proxy.
|
||||
|
||||
## Findings To Preserve
|
||||
|
||||
### Invariant: forward proxy must not relay unevaluated follow-on HTTP bytes
|
||||
|
||||
The historical forward path evaluated at most the first absolute-form request,
|
||||
rewrote it, then switched to bidirectional copy. Bytes already buffered after
|
||||
the first header block, or later pipelined requests on the same client/upstream
|
||||
connection, could reach upstream without the CONNECT L7 relay's per-request
|
||||
parser/evaluator.
|
||||
|
||||
Latest main mitigates this by forcing ordinary forward HTTP to one request per
|
||||
connection and by using guarded relay helpers. The adapter model should
|
||||
preserve the invariant either by keeping forward HTTP single-request/close or
|
||||
by passing the first parsed request into a shared HTTP relay loop.
|
||||
|
||||
### Endpoint config is not tied to deterministic matched policy
|
||||
|
||||
The policy name used for L4 authorization and logging can be selected through a
|
||||
different precedence rule than endpoint metadata. With overlapping host, port,
|
||||
and binary rules, allowed IPs, TLS behavior, enforcement, and
|
||||
`allow_encoded_slash` can come from a different endpoint than the policy name
|
||||
logged and used for L4 allow.
|
||||
|
||||
The adapter model requires authorization to return one decision with one
|
||||
deterministic matched endpoint.
|
||||
|
||||
### Endpoint metadata query failures should not erase enforcement
|
||||
|
||||
If endpoint metadata lookup fails, callers can interpret the result as no L7
|
||||
configuration and downgrade to credential-only or raw L4 relay.
|
||||
|
||||
The adapter model treats endpoint metadata as part of the authorization result.
|
||||
Failure to materialize required metadata should deny rather than erase extended
|
||||
configuration.
|
||||
|
||||
### Destination validation must be shared
|
||||
|
||||
Private address checks, `allowed_ips`, exact declared private endpoint trust,
|
||||
trusted gateway aliases, SSRF checks, and control-plane port blocks have grown
|
||||
over time. They should be centralized so CONNECT, forward HTTP, future
|
||||
transparent TCP, and local-service egress use the same resolved-destination
|
||||
rules.
|
||||
|
||||
## Existing Feature Inventory
|
||||
|
||||
The refactor should preserve:
|
||||
|
||||
- CONNECT explicit proxy support.
|
||||
- Forward HTTP explicit proxy support.
|
||||
- Network-only supervisor mode.
|
||||
- nftables bypass reject/log enforcement.
|
||||
- Provider credential injection and redaction.
|
||||
- Dynamic token grant injection through SPIFFE-backed provider credentials.
|
||||
- REST request-body credential rewrite.
|
||||
- WebSocket text-frame credential rewrite.
|
||||
- REST endpoint method/path policy.
|
||||
- GraphQL-over-HTTP policy.
|
||||
- WebSocket transport and GraphQL-over-WebSocket policy.
|
||||
- h2c rejection on inspected HTTP routes.
|
||||
- Inference routing through `inference.local`, including embeddings.
|
||||
- Agent-facing policy advisor routes through `policy.local`.
|
||||
- GCE metadata loopback for supported provider credentials.
|
||||
- Timeout and resource tracking for client, upstream, and local service work.
|
||||
- Structured OCSF logging for network and HTTP policy outcomes.
|
||||
- SSRF and internal address protections.
|
||||
- Exact declared private endpoint handling.
|
||||
- Control-plane port protection.
|
||||
- `allowed_ips` endpoint restrictions.
|
||||
- TLS auto-detection and termination for inspectable client connections.
|
||||
+57
-16
@@ -7,28 +7,46 @@ direction-focused.
|
||||
|
||||
- Add tests for forward HTTP pipelining and keep-alive follow-on requests,
|
||||
including the current `Connection: close` mitigation.
|
||||
- Add tests for forward HTTP h2c rejection on inspected endpoints.
|
||||
- Add tests for overlapping endpoint metadata selection.
|
||||
- Add tests for endpoint metadata query failures.
|
||||
- Add tests for control-plane port blocking through all destination validation
|
||||
paths.
|
||||
- Add tests for exact declared private endpoint trust and `allowed_ips`
|
||||
behavior across CONNECT and forward HTTP.
|
||||
- Add tests proving static credential injection works in L4-only HTTP and
|
||||
HTTP-inspected paths.
|
||||
- Add tests proving token grant success injects the configured header and token
|
||||
grant failure does not forward upstream.
|
||||
- Add tests for REST request-body credential rewrite, WebSocket text-frame
|
||||
credential rewrite, WebSocket GraphQL policy, and compression handling.
|
||||
- Add tests for `policy.local` proposal wait behavior and `inference.local`
|
||||
buffered/streaming route limits.
|
||||
- Add tests for metadata loopback startup/failure behavior when provider
|
||||
credentials require it.
|
||||
- Add nftables bypass enforcement tests that verify proxy-bound traffic is
|
||||
accepted while direct TCP/UDP egress is rejected and logged when available.
|
||||
|
||||
## Phase 1 - Authorization Result
|
||||
|
||||
- Introduce `EgressIntent` and `EgressDecision`.
|
||||
- Introduce `EgressIntent` and `EgressDecision` inside
|
||||
`openshell-supervisor-network`.
|
||||
- Make authorization return matched policy and matched endpoint metadata
|
||||
together.
|
||||
- Include policy source on the decision: user-authored, provider-derived, or
|
||||
local-service internal.
|
||||
- Include protocol enforcement and credential injection plan on the decision.
|
||||
- Fail closed when required endpoint metadata cannot be materialized.
|
||||
- Emit consistent OCSF network denial events from the shared boundary.
|
||||
|
||||
## Phase 2 - Shared Destination Validation
|
||||
|
||||
- Move DNS resolution, allowed IP filtering, SSRF checks, and control-plane port
|
||||
checks into one destination validation path.
|
||||
- Move DNS resolution, allowed IP filtering, SSRF checks, exact declared
|
||||
endpoint handling, trusted gateway aliases, and control-plane port checks
|
||||
into one destination validation path.
|
||||
- Return an `UpstreamConnector` rather than an opened upstream socket.
|
||||
- Add tests proving CONNECT, forward HTTP, and transparent TCP use the same
|
||||
validation behavior.
|
||||
- Add tests proving CONNECT, forward HTTP, and future transparent TCP use the
|
||||
same validation behavior.
|
||||
|
||||
## Phase 3 - Forward HTTP Adapter
|
||||
|
||||
@@ -36,15 +54,20 @@ direction-focused.
|
||||
request and builds an egress intent.
|
||||
- Route the parsed first request into the shared HTTP relay or preserve the
|
||||
current guarded single-request relay behavior.
|
||||
- Preserve `https://` absolute-form rejection.
|
||||
- Preserve h2c rejection on inspected routes.
|
||||
- Keep the no-raw-copy invariant after the first request.
|
||||
|
||||
## Phase 4 - HTTP And WebSocket Relay Consolidation
|
||||
## Phase 4 - HTTP, WebSocket, And Credential Relay Consolidation
|
||||
|
||||
- Centralize HTTP request parsing, REST policy, GraphQL policy, WebSocket
|
||||
upgrade policy, credential resolution, redaction, request rewrite, upstream
|
||||
dial, and response relay.
|
||||
- Evaluate every HTTP request before upstream write.
|
||||
- Ensure denied HTTP requests do not create upstream TCP sessions.
|
||||
- Preserve static placeholder rewrite for target, query, and headers.
|
||||
- Preserve dynamic token grant injection after request allow and before
|
||||
upstream write.
|
||||
- Preserve opt-in REST request-body credential rewrite behind the shared HTTP
|
||||
relay, including bounded buffering, supported content-type handling,
|
||||
`Content-Length` recomputation, and fail-closed unresolved placeholders.
|
||||
@@ -58,13 +81,14 @@ direction-focused.
|
||||
- Move client-side TLS detection and termination before the HTTP/TCP relay
|
||||
split.
|
||||
- Keep endpoint TLS behavior on `EgressDecision`.
|
||||
- Treat `tls: skip` as the explicit opt-out for TLS handling.
|
||||
- Remove duplicate HTTP-specific and TCP-specific TLS termination decisions.
|
||||
|
||||
## Phase 6 - TCP Relay And Parser Boundary
|
||||
|
||||
- Rename raw TCP relay concepts to `TcpRelay`.
|
||||
- Use `TcpRelay` for byte relay and TCP application parser dispatch.
|
||||
- Keep `protocol: tcp` or omitted protocol as L4 authorization plus byte copy.
|
||||
- Add a TCP application parser dispatch point for future protocol enforcement.
|
||||
- Keep `protocol: tcp` as L4 authorization plus byte copy.
|
||||
- Let TCP application parsers own their message loop and call the connector
|
||||
when protocol state allows.
|
||||
|
||||
@@ -75,6 +99,7 @@ direction-focused.
|
||||
- Publish active DNS answer state and capture rules.
|
||||
- Implement nftables REDIRECT/TPROXY capture rules ahead of the bypass reject
|
||||
path; do not add a parallel iptables path.
|
||||
- Coordinate capture rule ownership with `openshell-supervisor-process::netns`.
|
||||
- Implement transparent TCP adapter lookup from captured original destination
|
||||
to active endpoint generation.
|
||||
- Decide TTL and stale-generation behavior.
|
||||
@@ -82,35 +107,51 @@ direction-focused.
|
||||
## Phase 8 - Local Service Adapters
|
||||
|
||||
- Model `inference.local` as a local adapter with TLS termination, route
|
||||
validation, provider auth injection, streaming limits, and OCSF logging.
|
||||
validation, provider auth injection, streaming/buffered limits, and OCSF
|
||||
logging.
|
||||
- Model `policy.local` as a local adapter for current policy, bounded denial
|
||||
summaries, and policy proposals.
|
||||
- Keep both paths outside normal external egress relay.
|
||||
summaries, policy proposals, and proposal wait.
|
||||
- Decide whether metadata loopback remains orchestrated in `openshell-sandbox`
|
||||
or moves behind a local adapter boundary in `openshell-supervisor-network`.
|
||||
- Keep these paths outside normal external egress relay while preserving
|
||||
credential redaction and route validation.
|
||||
|
||||
## Phase 9 - Runtime Boundary
|
||||
|
||||
- Keep embedded mode for the first migration.
|
||||
- Keep embedded supervisor mode as the first migration target.
|
||||
- Treat the existing `openshell-supervisor-network` and
|
||||
`openshell-supervisor-process` split as the structural baseline.
|
||||
- Define the proxy runtime API needed for a future standalone binary:
|
||||
configured listeners, policy updates, gateway calls, telemetry, and shutdown.
|
||||
configured listeners, policy updates, provider credentials, token grants,
|
||||
gateway calls, telemetry, denial/activity events, and shutdown.
|
||||
- Identify process identity requirements for standalone and sidecar modes.
|
||||
- Add capability negotiation with the gateway if standalone proxy versions can
|
||||
differ from gateway versions.
|
||||
|
||||
## Phase 10 - Cleanup
|
||||
|
||||
- Remove duplicated endpoint metadata queries from relay paths.
|
||||
- Remove duplicated deny rendering where adapters can own response shape.
|
||||
- Remove duplicated destination validation and deny rendering where adapters
|
||||
can own response shape.
|
||||
- Remove any remaining forward HTTP raw-copy fallback.
|
||||
- Remove stale references to iptables or static `/etc/hosts` native TCP
|
||||
mapping from proxy design docs.
|
||||
- Update architecture docs once implementation lands.
|
||||
|
||||
## Testing Plan
|
||||
|
||||
- Unit-test each adapter's intent construction and deny response shape.
|
||||
- Unit-test authorization precedence for overlapping policy and endpoint rules.
|
||||
- Unit-test provider-derived rule namespace handling and `policy.local`
|
||||
filtering.
|
||||
- Integration-test shared destination validation across CONNECT, forward HTTP,
|
||||
and transparent TCP.
|
||||
- Integration-test HTTP keep-alive and pipelined requests with REST, GraphQL,
|
||||
and WebSocket upgrade enforcement.
|
||||
- Integration-test credential injection in L4-only HTTP and HTTP-inspected
|
||||
paths.
|
||||
- Integration-test token grant success, cache hit, malformed token, resolver
|
||||
unavailable, and token endpoint failure.
|
||||
- Integration-test REST request-body credential rewrite for JSON,
|
||||
form-url-encoded, `text/*`, unsupported content types, chunked framing, body
|
||||
caps, and unresolved placeholders.
|
||||
@@ -123,5 +164,5 @@ direction-focused.
|
||||
application parsers.
|
||||
- Integration-test policy DNS TTL, stale generation handling, and captured
|
||||
connect correlation.
|
||||
- Integration-test `inference.local` and `policy.local` body limits, timeout
|
||||
behavior, redaction, and local denial responses.
|
||||
- Integration-test `inference.local`, `policy.local`, and metadata loopback
|
||||
body limits, timeout behavior, redaction, and local denial responses.
|
||||
+117
-38
@@ -1,7 +1,19 @@
|
||||
# Technical Design Appendix
|
||||
|
||||
This appendix carries the implementation-level design details behind the main
|
||||
RFC.
|
||||
This appendix carries implementation-level design details behind the main RFC.
|
||||
|
||||
## Existing Runtime Boundary
|
||||
|
||||
`openshell-supervisor-network::run::run_networking` is the current networking
|
||||
startup boundary. It builds policy-local context, waits for policy binary
|
||||
symlink resolution, creates the identity cache, writes the TLS CA, builds TLS
|
||||
state, resolves inference routes, wires provider credentials and token grants,
|
||||
and starts the proxy.
|
||||
|
||||
This is a useful outer boundary, but it is not yet the proxy adapter boundary.
|
||||
The proxy still needs internal `EgressIntent` and `EgressDecision` boundaries
|
||||
so CONNECT, forward HTTP, local routes, and future native TCP capture do not
|
||||
duplicate policy and relay orchestration.
|
||||
|
||||
## Shared Data Boundaries
|
||||
|
||||
@@ -11,14 +23,16 @@ RFC.
|
||||
|
||||
It should carry:
|
||||
|
||||
- entry transport: CONNECT, forward HTTP, transparent TCP, or local HTTP;
|
||||
- entry transport: CONNECT, forward HTTP, transparent TCP, local HTTP, policy
|
||||
DNS, or metadata loopback;
|
||||
- requested destination host/port or captured original IP/port;
|
||||
- process identity inputs collected by the adapter/runtime;
|
||||
- optional first HTTP request for forward proxy traffic;
|
||||
- optional local service route.
|
||||
- optional local service route;
|
||||
- policy generation or DNS mapping generation when relevant.
|
||||
|
||||
Adapters build intents. They should not query endpoint metadata or select
|
||||
relays.
|
||||
Adapters build intents. They should not query endpoint metadata, select TLS
|
||||
mode, or select relays.
|
||||
|
||||
### EgressDecision
|
||||
|
||||
@@ -28,15 +42,19 @@ It should carry:
|
||||
|
||||
- allow or deny;
|
||||
- deterministic matched policy identifier;
|
||||
- whether the policy is user-authored, provider-derived, or local-service
|
||||
internal;
|
||||
- deterministic matched endpoint identifier and endpoint metadata;
|
||||
- process identity used for evaluation;
|
||||
- destination and allowed IP constraints;
|
||||
- TLS behavior;
|
||||
- protocol enforcement;
|
||||
- credential injection plan;
|
||||
- logging context and denial reason.
|
||||
|
||||
Relay code should read this decision. It should not query OPA again for
|
||||
endpoint metadata, TLS mode, allowed IPs, or parser selection.
|
||||
endpoint metadata, TLS mode, allowed IPs, credential behavior, or parser
|
||||
selection.
|
||||
|
||||
## Protocol Enforcement
|
||||
|
||||
@@ -46,15 +64,14 @@ Use a protocol enforcement value derived from endpoint policy:
|
||||
|-----------------|-------------|----------------|
|
||||
| omitted / `tcp` | None | L4 authorization plus byte relay, with optional HTTP sniff for credential injection |
|
||||
| `rest` | HTTP | HTTP request parser with REST rules, plus opt-in request-body and WebSocket text-frame credential rewrite |
|
||||
| `graphql` | HTTP | HTTP request parser with GraphQL rules |
|
||||
| `graphql` | HTTP | HTTP request parser with GraphQL-over-HTTP rules |
|
||||
| `websocket` | HTTP | HTTP upgrade policy followed by WebSocket frame policy or GraphQL-over-WebSocket policy |
|
||||
| future `redis`, `postgres`, `mysql`, ... | TCP application | Protocol-specific TCP parser owns the message loop |
|
||||
|
||||
`protocol: tcp` is effectively the default L4 mode. It should not run TCP
|
||||
application parsers.
|
||||
|
||||
Avoid using the term "provider" for these parser concepts because providers
|
||||
are already a first-class credential and routing domain in OpenShell.
|
||||
application parsers. Avoid using the term "provider" for parser concepts
|
||||
because providers are already a first-class credential and routing domain in
|
||||
OpenShell.
|
||||
|
||||
## Suggested Types
|
||||
|
||||
@@ -65,7 +82,9 @@ enum EgressTransport {
|
||||
Connect,
|
||||
ForwardHttp,
|
||||
TransparentTcp,
|
||||
PolicyDns,
|
||||
LocalHttp,
|
||||
MetadataLoopback,
|
||||
}
|
||||
|
||||
struct EgressIntent {
|
||||
@@ -74,6 +93,7 @@ struct EgressIntent {
|
||||
process: ProcessIdentity,
|
||||
first_request: Option<ParsedHttpRequest>,
|
||||
local_route: Option<LocalRoute>,
|
||||
generation: Option<PolicyGeneration>,
|
||||
}
|
||||
|
||||
struct EgressDecision {
|
||||
@@ -83,11 +103,23 @@ struct EgressDecision {
|
||||
log_context: EgressLogContext,
|
||||
}
|
||||
|
||||
struct MatchedPolicy {
|
||||
id: PolicyId,
|
||||
source: PolicySource,
|
||||
}
|
||||
|
||||
enum PolicySource {
|
||||
User,
|
||||
ProviderDerived,
|
||||
LocalService,
|
||||
}
|
||||
|
||||
struct MatchedEndpoint {
|
||||
id: EndpointId,
|
||||
allowed_ips: AllowedIpPolicy,
|
||||
tls: TlsPolicy,
|
||||
enforcement: ProtocolEnforcement,
|
||||
credentials: CredentialInjectionPlan,
|
||||
}
|
||||
|
||||
enum ProtocolEnforcement {
|
||||
@@ -104,10 +136,30 @@ enum HttpL7Protocol {
|
||||
|
||||
struct HttpL7Config {
|
||||
protocol: HttpL7Protocol,
|
||||
path: EndpointPathScope,
|
||||
allow_encoded_slash: bool,
|
||||
enforcement_mode: L7EnforcementMode,
|
||||
websocket_credential_rewrite: bool,
|
||||
request_body_credential_rewrite: bool,
|
||||
websocket_graphql_policy: bool,
|
||||
graphql_max_body_bytes: usize,
|
||||
}
|
||||
|
||||
struct CredentialInjectionPlan {
|
||||
static_placeholders: StaticPlaceholderPlan,
|
||||
token_grant: Option<TokenGrantPlan>,
|
||||
}
|
||||
|
||||
struct StaticPlaceholderPlan {
|
||||
http_target_query_header: bool,
|
||||
rest_request_body: bool,
|
||||
websocket_text_frames: bool,
|
||||
}
|
||||
|
||||
struct TokenGrantPlan {
|
||||
provider_key: String,
|
||||
auth_style: TokenGrantAuthStyle,
|
||||
token_endpoint: String,
|
||||
}
|
||||
|
||||
struct RelayContext {
|
||||
@@ -122,26 +174,19 @@ struct RelayContext {
|
||||
validated destination and lets relays/parsers open an upstream connection only
|
||||
after protocol policy allows it.
|
||||
|
||||
## Module Layout
|
||||
## Current Owners And Proposed Cleanup
|
||||
|
||||
A future split could look like:
|
||||
|
||||
| Module | Responsibility |
|
||||
|--------|----------------|
|
||||
| `proxy::adapter::connect` | Parse CONNECT and render CONNECT responses |
|
||||
| `proxy::adapter::forward_http` | Parse absolute-form HTTP and preserve first request |
|
||||
| `proxy::adapter::transparent_tcp` | Recover captured original destination |
|
||||
| `proxy::adapter::policy_dns` | Answer eligible DNS queries and publish active mappings |
|
||||
| `proxy::adapter::local` | Implement `inference.local` and `policy.local` surfaces |
|
||||
| `proxy::auth` | Build decisions from intents and OPA results |
|
||||
| `proxy::destination` | Resolve, filter, and validate destinations |
|
||||
| `proxy::netfilter` | Own nftables bypass and future transparent capture rules |
|
||||
| `proxy::relay::http` | HTTP request loop, credentials, REST/GraphQL/WebSocket upgrade policy |
|
||||
| `proxy::relay::websocket` | WebSocket frame validation, text-frame rewrite, and message policy |
|
||||
| `proxy::relay::tcp` | TCP byte relay and TCP application parser dispatch |
|
||||
| `proxy::relay::tls` | Shared client-side TLS termination |
|
||||
| `proxy::parser` | HTTP, WebSocket, and TCP application parser traits/config |
|
||||
| `proxy::telemetry` | OCSF and tracing helpers |
|
||||
| Current owner | Current responsibility | Proposed cleanup |
|
||||
|---------------|------------------------|------------------|
|
||||
| `openshell-sandbox` | Orchestrator, policy poll loop, denial/activity channels, metadata loopback startup, network-only lifecycle | Keep as orchestration; avoid embedding per-entry proxy policy decisions |
|
||||
| `openshell-supervisor-network::run` | Networking startup and handles | Become the stable runtime API for embedded and future standalone modes |
|
||||
| `openshell-supervisor-network::proxy` | CONNECT, forward HTTP, local route dispatch, destination validation, denial rendering | Split into adapters, authorization, destination, relay selection, and adapter response rendering |
|
||||
| `openshell-supervisor-network::opa` | Policy engine and Rego queries | Return deterministic `EgressDecision` data instead of separate policy and endpoint lookups |
|
||||
| `openshell-supervisor-network::l7` | REST, GraphQL, WebSocket, inference helpers, TLS, token grants | Keep as protocol/relay implementation behind shared relay boundaries |
|
||||
| `openshell-supervisor-network::policy_local` | `policy.local` state and routes | Model as a local adapter with explicit limits and proposal/wait behavior |
|
||||
| `openshell-supervisor-process::netns` | nftables bypass rules and namespace helpers | Remain owner of bypass enforcement; coordinate future capture rules with network proxy mappings |
|
||||
| `openshell-supervisor-process::bypass_monitor` | nftables LOG parsing and OCSF bypass telemetry | Remain telemetry producer for bypass violations |
|
||||
| `openshell-core::secrets` and provider credential state | Static placeholder sources and dynamic credential metadata | Feed credential injection plans; do not leak secrets into decision logs |
|
||||
|
||||
## Policy DNS And Resolved TCP State
|
||||
|
||||
@@ -194,15 +239,23 @@ Two acceptable approaches:
|
||||
Endpoint metadata query failures should fail closed when metadata is required
|
||||
for the selected endpoint. They should not silently downgrade to L4 behavior.
|
||||
|
||||
Provider-derived policies use a reserved rule-name namespace. The gateway and
|
||||
sandbox sync should prevent user-authored `_provider_*` rules, and
|
||||
`policy.local` proposal surfaces should not expose provider-derived rules as
|
||||
editable user policy. `EgressDecision` should still identify provider-derived
|
||||
matches for logging and debugging.
|
||||
|
||||
## Credential Injection Boundary
|
||||
|
||||
Credential injection belongs in the HTTP relay:
|
||||
Credential injection belongs in the HTTP/WebSocket relay after policy allow and
|
||||
before upstream write.
|
||||
|
||||
1. Authorization selects the endpoint and confirms credentials may be used.
|
||||
2. The HTTP relay resolves credentials only when it has an allowed HTTP request.
|
||||
3. Secrets are redacted from logs and policy-visible metadata.
|
||||
4. The final upstream request or frame is rewritten with real credentials
|
||||
immediately before write.
|
||||
1. Authorization selects the endpoint and computes a credential injection plan.
|
||||
2. The HTTP relay resolves credentials only when it has an allowed request.
|
||||
3. Static placeholder values are resolved and redacted from logs.
|
||||
4. Endpoint-bound token grants obtain or reuse a dynamic access token.
|
||||
5. The final upstream request or WebSocket frame is rewritten immediately
|
||||
before write.
|
||||
|
||||
Both L4-only HTTP and HTTP-inspected paths can inject credentials. The
|
||||
difference is whether REST, GraphQL, or WebSocket policy is evaluated before
|
||||
@@ -215,7 +268,8 @@ Credential rewrite slots should be explicit:
|
||||
- client-to-server WebSocket text frames only when
|
||||
`websocket_credential_rewrite` is enabled;
|
||||
- GraphQL-over-WebSocket connection/control messages when they are carried in
|
||||
text frames and the endpoint enables the WebSocket rewrite path.
|
||||
text frames and the endpoint enables the WebSocket rewrite path;
|
||||
- token grant headers for endpoint-bound provider credentials.
|
||||
|
||||
Request-body rewrite is REST-only. It should buffer bounded UTF-8 textual
|
||||
bodies, including JSON, form-url-encoded, and `text/*`, recompute
|
||||
@@ -223,6 +277,12 @@ bodies, including JSON, form-url-encoded, and `text/*`, recompute
|
||||
credential markers, and fail closed when a reserved placeholder cannot be
|
||||
resolved safely. Binary WebSocket frames are not rewritten.
|
||||
|
||||
Token grants are dynamic credential injection. They use provider metadata to
|
||||
request a SPIFFE JWT-SVID, exchange it for an OAuth2 access token, cache the
|
||||
token, and inject either an `Authorization: Bearer` header or a configured
|
||||
custom header. Token grant failures should return a local relay error and must
|
||||
not forward the request upstream.
|
||||
|
||||
## Parser Boundary
|
||||
|
||||
Protocol parsers operate on streams owned by the relay.
|
||||
@@ -241,6 +301,24 @@ Protocol parsers operate on streams owned by the relay.
|
||||
This avoids a separate dial strategy enum. The parser knows which protocol
|
||||
milestone is sufficient to call the validated connector.
|
||||
|
||||
## Local Service Adapter Boundary
|
||||
|
||||
Local services are network surfaces but not normal external egress:
|
||||
|
||||
- `inference.local` terminates local client traffic, validates known inference
|
||||
routes, strips caller auth, injects provider routing/auth, and applies
|
||||
streaming or buffered limits based on route type.
|
||||
- `policy.local` serves policy snapshots, denial summaries, proposal
|
||||
submission, and proposal wait. It should never expose secrets or provider
|
||||
rules as editable policy.
|
||||
- Metadata loopback serves provider metadata credentials for SDKs that bypass
|
||||
HTTP proxy variables. It should use the same provider credential state and
|
||||
redaction discipline as other credential paths.
|
||||
|
||||
These adapters may call gateway APIs or local credential helpers, but they
|
||||
should not bypass policy and credential invariants that apply to external
|
||||
egress.
|
||||
|
||||
## Timeout And Resource Ownership
|
||||
|
||||
| Owner | Resource |
|
||||
@@ -254,6 +332,7 @@ milestone is sufficient to call the validated connector.
|
||||
| TCP relay | Byte-copy idle timeout and half-close handling |
|
||||
| TCP parser | Protocol message timeouts and parser-specific limits |
|
||||
| Local service adapter | Local route body limits, response caps, gateway call timeout |
|
||||
| Token grant resolver | SPIFFE Workload API timeout, token endpoint timeout, cache TTL |
|
||||
|
||||
Timeouts should be recorded in telemetry at the owner boundary that can explain
|
||||
the failure.
|
||||
Reference in New Issue
Block a user