Keith Mattix II
abd45ad081
Run agentgateway data plane tests as a part of substrate CI (non-blocking to start so we can confirm it's not flaky). Also, change the `--atenet-router` flag to `--atenet-dataplane` to make it clearer that the flag controls ingress and egress. I've run the e2es locally across gVisor and microVM plus the MITM variants for both. The only skip we do for agentgateway is `TestIngressProtocolDowngrade` because 1. the behavior its testing only exists on the non-CONNECT atunnel ingress path and agentgateway only sends CONNECT to atunnel and 2. I'm not sure that we want this to be a part of the contract that substrate is bound by (e.g. do we really want to commit to atunnel always parsing HTTP?). My goal with getting both dataplanes into CI is to start taking steps to codify the proxy (router + egress PEP) contract for substrate. The telemetry they emit, atunnel expectations, etc. are all important contracts to explicitly call out so that they don't become too coupled to a single dataplane implementation. > It's a good idea to open an issue first for discussion. - [X] Tests pass - [X] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>
E2E testing
$ source .ate-dev-env.sh
$ go test -v ./internal/e2e/suites/... -args --e2e
Principles
- Keep it simple -- use go test for the harness.
- e2e tests live under
internal/e2e/suites/<suite> - Each suite should implement TestMain using e2e.RunTestMain()
- e2e tests will be skipped for ordinary unit tests unless the
--e2eflag is set e.g.go test ./internal/e2e/suites/... -args --e2e
- e2e tests will be skipped for ordinary unit tests unless the
- Helper libraries live under
internal/e2e - Setup and Teardown are on a per-component basis and the component's author's responsibility.
Preconditions
The e2e tests assume you have a cluster set up with Agent Substrate installed,
for example via hack/install-ate.sh --deploy-ate-system or
hack/install-ate-kind.sh --deploy-ate-system.
Sandbox classes
The suites are runtime-agnostic: the same tests run against gVisor and against
the micro-VM (kata + cloud-hypervisor) sandbox class. E2E_SANDBOX_CLASS
selects which, by repointing every fixture at its variant --- see
e2e.CounterFixture, e2e.EgressFixture and e2e.RenderFixtureManifest in
sandbox.go. Unset means gVisor.
# gVisor (the default), against the demos install-ate-kind.sh deploys
$ hack/run-e2e-kind.sh -v -args --no-color
# micro-VM, against the counter-microvm and egress-microvm demos
$ E2E_SANDBOX_CLASS=microvm hack/run-e2e-kind.sh -v -args --no-color
The micro-VM lane needs its fixtures installed first, which also needs a node
with /dev/kvm (hack/create-kind-cluster.sh detects one and labels the node):
$ hack/run-microvm-demo-kind.sh # counter-microvm + assets
$ hack/install-ate-kind.sh --deploy-demo-egress-microvm # egress-microvm
A handful of knobs override the class defaults, mostly for a cluster that
installs the fixtures elsewhere: E2E_SUBSTRATE_TEMPLATE_ATESPACE /
E2E_SUBSTRATE_TEMPLATE_NAME / E2E_SUBSTRATE_POOL_NAMESPACE /
E2E_SUBSTRATE_POOL_NAME point the counter fixture somewhere else, and
E2E_TEMPLATE_READY_TIMEOUT replaces the golden-snapshot budget (90s on
gVisor, 10m on micro-VM, where the golden is a cloud-hypervisor cold boot plus
a checkpoint).
After a failure
A suite deletes the namespaces it created only when it passed. A failed run keeps them, because the failure is usually explained inside a worker pod (the ateom logs, and for a micro-VM worker the guest's console tail), and deleting the namespace takes those pods with it:
$ kubectl logs -n <kept-namespace> <worker-pod>
Nothing reclaims them afterwards, and each namespace holds a WorkerPool's worth of running pods, so clean up once you are done reading:
$ hack/cleanup-e2e.sh # deletes every namespace labeled ate.dev/e2e
Creating a new test suite
Copy testmain_test.go from internal/e2e/suites/example into your new suite. It will
look like this:
func run(m *testing.M) int {
Setup()
defer Teardown()
// return allows the deferred Teardown to run.
return e2e.RunTestMain(m)
}
func TestMain(m *testing.M) { os.Exit(run(m)) }
This will handle the standard flags and checks for running an e2e test suite.