mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-10 10:13:50 +08:00
Benjamin Elder
e4844d0fd2
When an e2e test failed, the evidence was deleted before anyone could read it. The suite deleted every namespace it created on the way out, taking the worker pods with it, and the workflow's post-failure dump only looked at three fixed namespaces — never the suites' randomly-named ones. So a failure inside an actor (#619: a micro-VM resume where the guest died at boot) left nothing behind but the RPC error the test printed. Keep the namespaces when the suite failed, and dump every worker pod in every namespace, so the ateom logs — which carry the guest's console tail — reach the failed run's output. Kept namespaces are nobody's to reclaim, and each holds a WorkerPool's worth of running pods, so they now carry an ate.dev/e2e label and hack/cleanup-e2e.sh deletes them once the logs have served their purpose. CI throws its cluster away, but a development cluster accumulates them run after run. Fixes #<issue_number_goes_here> Built while debugging https://github.com/agent-substrate/substrate/issues/619 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
53 lines
1.9 KiB
Bash
Executable File
53 lines
1.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
|
|
# Copyright 2026 Google LLC
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
set -o errexit -o nounset -o pipefail
|
|
|
|
# Deletes the namespaces a failed e2e run left behind.
|
|
#
|
|
# A failed suite keeps its namespaces so the worker pods — and the ateom logs
|
|
# inside them, which is where an actor failure is actually explained — survive
|
|
# long enough to read (see RetainNamespaces in internal/e2e/namespace.go).
|
|
# Nothing reclaims them afterwards, and each holds a WorkerPool's worth of
|
|
# running pods, so run this once the logs have served their purpose.
|
|
#
|
|
# This deletes EVERY e2e namespace in the cluster, so don't run it while a suite
|
|
# is running.
|
|
#
|
|
# Env: KUBECTL_CONTEXT (optional) kube context to target; KIND_CLUSTER_NAME
|
|
# (optional) shorthand for the matching kind context.
|
|
|
|
if [[ -z "${KUBECTL_CONTEXT:-}" && -n "${KIND_CLUSTER_NAME:-}" ]]; then
|
|
KUBECTL_CONTEXT="kind-${KIND_CLUSTER_NAME}"
|
|
fi
|
|
kubectl_args=()
|
|
if [[ -n "${KUBECTL_CONTEXT:-}" ]]; then
|
|
kubectl_args+=("--context=${KUBECTL_CONTEXT}")
|
|
fi
|
|
|
|
# The label CreateNamespace stamps on every namespace the suites create.
|
|
selector="ate.dev/e2e"
|
|
|
|
leftover="$(kubectl "${kubectl_args[@]}" get namespaces -l "${selector}" -o name)"
|
|
if [[ -z "${leftover}" ]]; then
|
|
echo "No leftover e2e namespaces."
|
|
exit 0
|
|
fi
|
|
|
|
echo "Deleting leftover e2e namespaces:"
|
|
echo "${leftover}"
|
|
kubectl "${kubectl_args[@]}" delete namespaces -l "${selector}" --ignore-not-found
|