trending/substrate
1
0
Fork 0
mirror of https://github.com/agent-substrate/substrate.git synced 2026-10-02 03:24:42 +08:00
Code Issues Packages Projects Releases Wiki Activity

remove ssl_cert_dir and add clearer guidance for actor trust store (#2044)

Browse Source 
We should not override the whole actor trust store with SSL_CERT_DIR,
removing that to defer using SSL_CERT_FILE, which is additive.

Additionally, this does not cover all languages and runtimes, adding
guidance for other env vars required and other runtimes that need union
of the trust stores in a single file.

In a followup, we are going to remove all "sdsmint" and "plain gateway"
language and just have one gateway thats capable of minting or not based
on the sni.

> It's a good idea to open an issue first for discussion.

- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
This commit is contained in:
Lior Lieberman
2026-10-01 04:59:44 +00:00
committed by GitHub
parent e8cad55675
commit 7526b9cc75
3 changed files with 105 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files
demos/egress/egress-microvm-mitm-template.yaml.tmpl
+5 -11
View File
@@ -18,7 +18,7 @@
# projection is worth proving on both.
#
# Kept in step with egress-microvm-template.yaml.tmpl; the deltas
# are the names, the projected trust bundle, and the two SSL_CERT_* variables.
# are the names, the projected trust bundle, and the SSL_CERT_FILE variable.
metadata:
atespace: ate-demo-egress-microvm-mitm
@@ -30,16 +30,10 @@ containers:
- name: egress
image: ko://github.com/agent-substrate/substrate/demos/egress
command: ["/ko-app/egress"]
# The demo fetches with a stock net/http client, so its roots come from
# crypto/x509's system pool — which on Unix reads both of these.
#
# SSL_CERT_FILE alone is not enough to make the projected bundle the whole
# story: it replaces the default cert FILE list, but the default cert
# DIRECTORY list is still scanned, and the base image keeps its public
# roots in /etc/ssl/certs. Pointing SSL_CERT_DIR at the projection too
# makes the anchor set exactly the gateway CA, so a successful HTTPS fetch
# proves the projected bundle validated the minted leaf. When both passthrough
# TLS and MITM policies are configured both public and ATE roots are needed.
# The demo fetches with a stock net/http client. SSL_CERT_FILE adds the
# gateway CA, and Go still scans /etc/ssl/certs, so the image's public
# roots stay available for tls_passthrough origins. SSL_CERT_DIR would
# drop them, so it is deliberately not set.
env:
- name: SSL_CERT_FILE
value: /run/ate/trust-bundle.pem
demos/egress/egress-mitm-template.yaml.tmpl
+6 -12
View File
@@ -13,14 +13,14 @@
# limitations under the License.
# MITM variant of the egress demo template: the same workload as
# egress-template.yaml.tmpl, but trusting only the egress gateway
# egress-template.yaml.tmpl, but also trusting the egress gateway
# CA. The networking suite builds its egress actors from this fixture instead
# of the plain one when E2E_EGRESS_MITM is set, because under an sdsmint
# gateway TestActorEgressHTTPS's origin certificate is a per-SNI leaf the
# gateway minted, which chains to no public CA.
#
# Kept in step with egress-template.yaml.tmpl; the deltas are the
# names, the projected trust bundle, and the two SSL_CERT_* variables.
# names, the projected trust bundle, and the SSL_CERT_FILE variable.
metadata:
atespace: ate-demo-egress-mitm
@@ -32,16 +32,10 @@ containers:
- name: egress
image: ko://github.com/agent-substrate/substrate/demos/egress
command: ["/ko-app/egress"]
# The demo fetches with a stock net/http client, so its roots come from
# crypto/x509's system pool — which on Unix reads both of these.
#
# SSL_CERT_FILE alone is not enough to make the projected bundle the whole
# story: it replaces the default cert FILE list, but the default cert
# DIRECTORY list is still scanned, and the base image keeps its public
# roots in /etc/ssl/certs. Pointing SSL_CERT_DIR at the projection too
# makes the anchor set exactly the gateway CA, so a successful HTTPS fetch
# proves the projected bundle validated the minted leaf. When both passthrough
# TLS and MITM policies are configured both public and ATE roots are needed.
# The demo fetches with a stock net/http client. SSL_CERT_FILE adds the
# gateway CA, and Go still scans /etc/ssl/certs, so the image's public
# roots stay available for tls_passthrough origins. SSL_CERT_DIR would
# drop them, so it is deliberately not set.
env:
- name: SSL_CERT_FILE
value: /run/ate/trust-bundle.pem
docs/egress-trust-bundle.md
+94 -39
View File
@@ -1,14 +1,19 @@
# Enabling man-in-the-middle (MITM) interception for Actor Egress policy
Under an sdsmint install, the egress gateway terminates every TLS connection an
actor opens and re-originates it. The certificate the actor sees is therefore
not the origin's — it is a per-SNI leaf the gateway minted, which chains to the
gateway's own CA and to no public root. An actor that validates against only the
public roots rejects it, and every HTTPS request the actor makes fails with a
Under an sdsmint install, the egress gateway terminates the TLS connections an
actor's `https` rules allow and re-originates them. The certificate the actor
sees is then not the origin's: it is a per-SNI leaf the gateway minted, which
chains to the gateway's own CA and to no public root. An actor that validates
against only the public roots rejects it, and every such request fails with a
certificate error.
Connections a `tls_passthrough` rule allows are not terminated. The actor sees
the origin's own chain and verifies it against the public roots. An actor whose
policy mixes the two therefore has to trust **both** the gateway CA and the
public roots.
This guide covers how to project the gateway's CA into an actor's filesystem
and how to point the actor's TLS client at it.
and how to add it to the actor's trust store without losing the public roots.
## DNS and egress policy
@@ -28,9 +33,8 @@ You need it when **all** of the following hold:
--deploy-atenet --experimental-use-sdsmint`).
* The actor makes **HTTPS** (or any TLS) requests.
If you configure this on a non-sdsmint install you will *break* the actor:
the steps below make the gateway CA the actor's only trust anchor, and without
a MITM gateway in front of it nothing the actor dials will chain to that CA.
On an install without sdsmint the bundle does not exist, and an actor that
declares it does not start (see [Operational notes](#operational-notes)).
## Project the bundle
@@ -73,52 +77,90 @@ depends on the first block being a particular certificate.
## Point the runtime at it
Projecting the file is not enough; each TLS stack has to be told to use it.
The rule is **append, never replace**: the gateway CA is added to the public
roots, not substituted for them. A trust store that holds only the gateway CA
rejects every `tls_passthrough` origin.
### Go, and anything linked against OpenSSL
### Runtimes with an additive setting
These need one environment variable and no other work.
| Runtime | Setting | Why it keeps the public roots |
|---|---|---|
| Go, stock `net/http` | `SSL_CERT_FILE=/run/ate/trust-bundle.pem` | Go reads that file in place of the default bundle, then still scans `/etc/ssl/certs`. |
| Anything on OpenSSL's default verify paths: Python `ssl`, `urllib`, `aiohttp`, `psql`, ... | `SSL_CERT_FILE=/run/ate/trust-bundle.pem` | Same: the default certificate directory is still scanned. |
| Node.js | `NODE_EXTRA_CA_CERTS=/run/ate/trust-bundle.pem` | Adds to Node's bundled roots. |
| Deno | `DENO_CERT=/run/ate/trust-bundle.pem` | Adds to Deno's bundled roots. |
Never set `SSL_CERT_DIR`. It replaces the default directory list, and with it
the public roots.
```yaml
env:
- name: SSL_CERT_FILE
value: /run/ate/trust-bundle.pem
- name: SSL_CERT_DIR
value: /run/ate
```
Set **both**. `SSL_CERT_FILE` replaces the default certificate *file* list, but
the default certificate *directory* list is still scanned, and most base images
keep their public roots in `/etc/ssl/certs`. With `SSL_CERT_FILE` alone the
actor trusts the gateway CA *in addition to* every public CA. Pointing
`SSL_CERT_DIR` at the projection as well makes the anchor set exactly the
gateway CA — so a successful HTTPS fetch proves the projected bundle is what
validated the minted leaf, rather than a public root happening to work.
A Go program can also skip the variable and append the projected PEM to the
pool returned by `x509.SystemCertPool()`.
Under sdsmint the public roots are useless anyway: every TLS origin the actor
can reach is fronted by the gateway.
### Runtimes that take a bundle file
### Other runtimes
These treat the file they are given as the whole trust store, so they need a
file that holds the public roots **and** the gateway CA.
| Runtime | Variable | Note |
|---|---|---|
| Node.js | `NODE_EXTRA_CA_CERTS=/run/ate/trust-bundle.pem` | *Adds* to Node's bundled roots rather than replacing them; the actor keeps trusting public CAs. |
| Python `requests` | `REQUESTS_CA_BUNDLE=/run/ate/trust-bundle.pem` | `requests` defaults to certifi and ignores `SSL_CERT_FILE`. |
| Python `ssl` / `urllib` | `SSL_CERT_FILE`, `SSL_CERT_DIR` | Honored via OpenSSL's default verify paths. |
| Python `httpx`, other Certifi-based clients | — | No environment variable; Certifi is pinned in code. Pass the path explicitly, e.g. `httpx.Client(verify="/run/ate/trust-bundle.pem")`. |
| curl | `CURL_CA_BUNDLE=/run/ate/trust-bundle.pem` | |
| git over HTTPS | `GIT_SSL_CAINFO=/run/ate/trust-bundle.pem` | |
| Java | — | No environment variable. Convert the PEM to a PKCS#12 or JKS truststore at startup and pass `-Djavax.net.ssl.trustStore`. |
| Runtime | Setting |
|---|---|
| Python `requests` | `REQUESTS_CA_BUNDLE` (it ignores `SSL_CERT_FILE`) |
| Python `httpx` | `SSL_CERT_FILE`, or `httpx.Client(verify=...)` |
| pip | `PIP_CERT` |
| curl | `CURL_CA_BUNDLE` |
| git over HTTPS | `GIT_SSL_CAINFO` |
Build that file at start with an entrypoint wrapper. Set the wrapper as the
container's `command` and the real program as `args`; the projection is
read-only, so write the result somewhere writable.
```sh
#!/bin/sh
set -eu
sys=/etc/ssl/certs/ca-certificates.crt # RHEL family: /etc/pki/tls/certs/ca-bundle.crt
ca=/run/ate/trust-bundle.pem
out=/tmp/ca-bundle.pem
{ cat "$sys"; echo; cat "$ca"; } > "$out"
export SSL_CERT_FILE="$out" REQUESTS_CA_BUNDLE="$out" CURL_CA_BUNDLE="$out" GIT_SSL_CAINFO="$out" PIP_CERT="$out"
export NODE_EXTRA_CA_CERTS="$ca" DENO_CERT="$ca"
exec "$@"
```
The `echo` keeps two PEM blocks from running together when the system bundle
lacks a trailing newline.
Java reads none of these variables. Copy the JDK's `cacerts` to a writable
path, import each certificate from the bundle with `keytool -importcert` (one
call per certificate), and pass
`-Djavax.net.ssl.trustStore=<copy>` through `JAVA_TOOL_OPTIONS`.
An image without a shell cannot run a wrapper. Use the in-code route for Go, or
pick a runtime from the first table.
Do not append the bundle to certifi's own `cacert.pem`, and do not bake it into
the image at build time: the first breaks on the next package upgrade, the
second ties the image to one cluster's CA and breaks on rotation.
## Verify
`demos/egress/egress-mitm.yaml.tmpl` is a complete working template that does
exactly this. Deploy it against an sdsmint install:
`demos/egress/egress-mitm-template.yaml.tmpl` is a complete working template
that does exactly this. Deploy it against an sdsmint install:
```bash
./hack/install-ate.sh --deploy-demo-egress-mitm
```
Then drive an actor's egress at an HTTPS URL and confirm it returns a response
rather than a certificate error. Because the demo sets `SSL_CERT_DIR` as well,
a `200` is positive evidence that the projected bundle did the validating.
Then drive an actor's egress at an HTTPS URL an `https` rule allows and
confirm it returns a response rather than a certificate error. The minted leaf
chains to no public root, so a `200` is positive evidence that the projected
bundle did the validating.
## Operational notes
@@ -154,13 +196,26 @@ wrote. The other failure modes differ only in the innermost clause:
| Name not on the allowlist | `trust bundle "my-own-bundle" is not supported by this deployment (supported: egress-mitm.ate.dev)` |
| Bundle present but empty or unparseable | `trust bundle "egress-mitm.ate.dev": unusable ClusterTrustBundle "egress-mitm.ate.dev:mitm:primary-bundle": …` |
**A certificate error is a trust-store problem, not a gateway one.** Substrate
cannot see how the actor loaded the bundle, so the first signal is the
application's own error: `x509: certificate signed by unknown authority` from
Go, `CERTIFICATE_VERIFY_FAILED` from Python, exit code 60 from curl.
| Symptom | Likely cause |
|---|---|
| Fails on an `https`-rule host, `tls_passthrough` hosts work | The gateway CA is missing: projection absent, wrong path, or a variable the runtime does not read |
| Fails on a `tls_passthrough` host, `https`-rule hosts work | The public roots are missing: `SSL_CERT_DIR` is set, or a bundle-file variable points at the projection alone |
| Both fail | Wrong file, or an image with no system bundle at the path the wrapper reads |
| Worked, then fails after a resume | The CA rotated and the process still holds its old pool; restart the process |
**Rotation is picked up on the next resume.** atelet re-resolves the bundle on
both `Run` and `Restore`, so a suspended actor gets the current anchors when it
comes back. A long-running actor that never suspends keeps the copy made when it
started — and a process that has already loaded the file into memory (Go caches
its system pool after first use) will not see a change on disk either way. Plan
CA rotation around a resume, with an overlap window that covers the actors that
do not suspend.
its system pool after first use) will not see a change on disk either way. A
wrapper-built union file is likewise rebuilt only on a cold start. Plan CA
rotation around a resume, with an overlap window that covers the actors that do
not suspend.
**The bundle is not a substitute for authenticating the actor.** It lets the
actor verify the *gateway*. It says nothing to an origin about which actor is