feat(docker): fail Docker calls fast while the daemon is unreachable (#9047)

A circuit breaker sits under the retry layer of both Docker clients and
counts consecutive transport failures: an I/O error or an unusable
socket, never an HTTP response or a wait for a pooled connection. After
floci.docker.breaker-failure-threshold (3) of them it pings the daemon
off the caller's thread, over a one-connection transport of its own;
only when that ping fails too does every Docker call fail at once with
DockerUnavailableException, which the retry layer never replays. While
open it pings every floci.docker.breaker-probe-interval-seconds (5), and
the first answer closes it. Any HTTP status to the ping is an answer,
and a ping that never reaches the daemon neither opens nor closes the
breaker. A threshold of 0 turns it off.
This commit is contained in:
Hector Ventura
2026-11-20 14:16:14 -04:00
committed by GitHub
parent 3991e5ce44
commit 5cded523e3
12 changed files with 761 additions and 2 deletions
@@ -134,6 +134,8 @@ floci:
max-connections: 1024 # Docker client connection pool for short calls
streaming-max-connections: 3072 # Separate pool for streams; each live Lambda container holds 2, plus 1 per extension
pool-gauge-interval-seconds: 60 # Log Docker pool use every N seconds while a pool is busy; 0 = off
breaker-failure-threshold: 3 # Consecutive Docker call attempts (each retry counts) failing at the transport before Floci pings the daemon and fails calls fast if it is gone; 0 = off
breaker-probe-interval-seconds: 5 # Seconds between pings while the daemon is unreachable
docker-config-path: "" # Path to dir containing Docker's config.json (e.g. /root/.docker)
registry-credentials: [] # Per-registry explicit credentials for private registries
+15
View File
@@ -122,6 +122,19 @@ floci:
Environment variable: `FLOCI_DOCKER_POOL_GAUGE_INTERVAL_SECONDS`
### Unreachable daemon
When the Docker daemon stops answering, Floci stops waiting on it. After `breaker-failure-threshold` attempts in a row (3 by default) fail before the daemon answers (a broken connection, a refused or missing socket, a call that times out), Floci pings the daemon. Each retry of a call is an attempt, so a single call that Floci retries can reach the threshold on its own; the ping is what keeps that from turning Docker off. If the ping fails too, every Docker call fails at once instead of waiting out its own connect, response and retry timeouts, so a container-backed request reports a missing daemon in about a second. Floci logs one warning when this starts, pings the daemon every `breaker-probe-interval-seconds` (5 by default), and logs one line when it answers again. An error status from the daemon (a 404, a 500) is an answer and never counts, for a call or for the ping, and a ping that never reaches the daemon (Floci shutting down) neither turns Docker off nor back on. The ping uses a connection of its own and gives up after 5 seconds. Set the threshold to 0 to turn this off:
```yaml
floci:
docker:
breaker-failure-threshold: 3
breaker-probe-interval-seconds: 5
```
Environment variables: `FLOCI_DOCKER_BREAKER_FAILURE_THRESHOLD`, `FLOCI_DOCKER_BREAKER_PROBE_INTERVAL_SECONDS`
## Private Registry Authentication
Any service that pulls a container image from a private registry (Lambda image functions, custom OpenSearch images, private Postgres images, etc.) needs Docker credentials. Two approaches are supported and can be combined.
@@ -368,6 +381,8 @@ caller keeps its existing failure handling after retries are exhausted.
| `FLOCI_DOCKER_DOCKER_HOST` | `unix:///var/run/docker.sock` | Docker daemon socket |
| `FLOCI_DOCKER_DOCKER_CONFIG_PATH` | _(unset)_ | Path to directory containing Docker's `config.json` |
| `FLOCI_DOCKER_MAX_CONNECTIONS` | `1024` | Connection pool size for Floci's short Docker calls |
| `FLOCI_DOCKER_BREAKER_FAILURE_THRESHOLD` | `3` | Docker call attempts in a row (each retry counts) that fail before the daemon answers, after which a failed ping makes every Docker call fail at once until the daemon answers again; `0` turns this off |
| `FLOCI_DOCKER_BREAKER_PROBE_INTERVAL_SECONDS` | `5` | Seconds between the pings that check whether an unreachable Docker daemon is back |
| `FLOCI_DOCKER_STREAMING_MAX_CONNECTIONS` | `3072` | Connection pool size for streams that last a container's lifetime or a long command's run. Each live Lambda container holds two (log stream and exit watcher) plus one per extension, and each CodeBuild phase, SSM command, ECS Exec session, EKS audit follower, user data script and EMR bootstrap action one |
| `FLOCI_DOCKER_REGISTRY_CREDENTIALS_0__SERVER` | _(unset)_ | Registry hostname for credential entry 0 |
| `FLOCI_DOCKER_REGISTRY_CREDENTIALS_0__USERNAME` | _(unset)_ | Username for credential entry 0 |
@@ -149,6 +149,8 @@ This foundation release provides opt-in security-group filtering for EC2 Docker
| `FLOCI_DOCKER_MAX_CONNECTIONS` | `1024` | Connection pool size for Floci's short Docker calls. See [Docker Configuration](docker.md#connection-pool) |
| `FLOCI_DOCKER_STREAMING_MAX_CONNECTIONS` | `3072` | Connection pool size for streams that last a container's lifetime or a long command's run, kept apart from the main pool so they never starve create, start, stop and remove. Each live Lambda container holds two (log stream and exit watcher) plus one per extension, and each CodeBuild phase, SSM command, ECS Exec session, EKS audit follower, user data script and EMR bootstrap action one. See [Docker Configuration](docker.md#connection-pool) |
| `FLOCI_DOCKER_POOL_GAUGE_INTERVAL_SECONDS` | `60` | Seconds between the log lines that report each Docker pool's leased, available and pending connections and its open streams, written only while a pool is in use; `0` turns them off. See [Docker Configuration](docker.md#connection-pool) |
| `FLOCI_DOCKER_BREAKER_FAILURE_THRESHOLD` | `3` | Docker call attempts in a row (each retry counts) that fail before the daemon answers, after which a failed ping makes every Docker call fail at once until the daemon answers again; `0` turns this off. See [Docker Configuration](docker.md#unreachable-daemon) |
| `FLOCI_DOCKER_BREAKER_PROBE_INTERVAL_SECONDS` | `5` | Seconds between the pings that check whether an unreachable Docker daemon is back |
| `FLOCI_DOCKER_IMAGE_REGISTRY_BASE` | _(none)_ | Optional registry/repository base for every Docker image Floci launches. When set, `postgres:16-alpine` resolves as `<base>/postgres:16-alpine` and `public.ecr.aws/docker/library/ubuntu:24.04` resolves as `<base>/public.ecr.aws/docker/library/ubuntu:24.04`. An image in a registry on `localhost` or a loopback address, Floci's own ECR among them, is launched as it is |
| `FLOCI_DOCKER_LOG_MAX_SIZE` | `10m` | Log rotation max size for spawned containers (e.g. `10m`, `1g`) |
| `FLOCI_DOCKER_LOG_MAX_FILE` | `3` | Number of rotated log files to keep for spawned containers |
@@ -4496,6 +4496,20 @@ public interface EmulatorConfig {
@WithDefault("60")
int poolGaugeIntervalSeconds();
/**
* Consecutive Docker call attempts that must fail at the transport (an I/O error or an
* unusable socket, never an HTTP response) before Floci pings the daemon and, when the ping
* fails too, fails every Docker call at once until the daemon answers again. Each retry of a
* call is an attempt, so one call the retry layer replays can reach the threshold alone; the
* ping keeps that from opening the breaker over a daemon that answers. 0 turns this off.
*/
@WithDefault("3")
int breakerFailureThreshold();
/** Seconds between the pings that check whether an unreachable Docker daemon is back. */
@WithDefault("5")
int breakerProbeIntervalSeconds();
/**
* Optional namespace inserted into Floci-managed child container and volume names.
* Useful when multiple Floci processes share one Docker daemon.
@@ -0,0 +1,45 @@
package io.floci.aws.core.common.docker;
import com.github.dockerjava.transport.DockerHttpClient;
import java.io.IOException;
/**
* Puts {@link DockerCircuitBreaker} in front of a Docker transport: while the breaker is open a call
* fails at once with {@link DockerUnavailableException}; otherwise the call runs, an HTTP response of
* any status tells the breaker the daemon is there, and a transport failure counts toward opening it.
* It sits under {@link RetryingDockerHttpClient}, which never replays the fail-fast exception.
*/
final class CircuitBreakingDockerHttpClient implements DockerHttpClient {
private final DockerHttpClient delegate;
private final DockerCircuitBreaker breaker;
CircuitBreakingDockerHttpClient(DockerHttpClient delegate, DockerCircuitBreaker breaker) {
this.delegate = delegate;
this.breaker = breaker;
}
@Override
public Response execute(Request request) {
if (breaker.isOpen()) {
throw new RuntimeException(breaker.unavailable());
}
Response response;
try {
response = delegate.execute(request);
} catch (RuntimeException e) {
if (DockerRetry.isDaemonFailure(e)) {
breaker.recordFailure(e);
}
throw e;
}
breaker.recordResponse();
return response;
}
@Override
public void close() throws IOException {
delegate.close();
}
}
@@ -0,0 +1,234 @@
package io.floci.aws.core.common.docker;
import com.github.dockerjava.transport.DockerHttpClient;
import org.jboss.logging.Logger;
import java.time.Duration;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
/**
* One view of whether the Docker daemon is reachable, shared by both Docker clients. After
* {@code threshold} consecutive attempts fail at the transport (an I/O error or an unusable socket,
* never an HTTP response) it pings the daemon off the caller's thread; only when that ping fails at
* the transport too does it open, and every Docker call then fails at once with
* {@link DockerUnavailableException} instead of waiting out its own connect, response and retry
* timeouts. While open it pings every {@code probeInterval}, and the first answer closes it. A ping
* is judged by the same rules as a call: any HTTP status is an answer, and a failure that says nothing
* about the daemon (an interruption, a wait for a pooled connection) neither opens nor closes it. A
* {@code threshold} of 0 disables it.
*
* <p>The confirming ping matters because the breaker sits under the retry layer and so counts
* attempts, not calls: a burst of broken pipes during a fan-out build is a few failed attempts in a
* row that the next attempt rides out, and must not take Docker away from every service.
*/
final class DockerCircuitBreaker implements AutoCloseable {
private static final Logger LOG = Logger.getLogger(DockerCircuitBreaker.class);
/** What a ping showed about the daemon. */
enum PingResult {
/** An HTTP response, whatever its status: the daemon is there. */
ANSWERED,
/** A transport failure, as {@link DockerRetry#isDaemonFailure} classifies it. */
UNREACHABLE,
/** A failure that says nothing about the daemon, such as an interruption. */
INCONCLUSIVE
}
/** Pings the daemon. */
@FunctionalInterface
interface Probe {
PingResult ping();
}
private final int threshold;
private final Duration probeInterval;
private final AtomicInteger consecutiveFailures = new AtomicInteger();
private final AtomicBoolean probeScheduled = new AtomicBoolean();
private final AtomicInteger probesRun = new AtomicInteger();
private final Object schedulerLock = new Object();
private volatile boolean open;
private volatile String lastFailure = "";
private volatile Probe probe;
private ScheduledExecutorService scheduler;
private boolean closed;
DockerCircuitBreaker(int threshold, Duration probeInterval) {
this.threshold = threshold;
this.probeInterval = probeInterval;
}
/**
* A ping over {@code transport}, below the breaker, so it reaches the daemon while the breaker is
* open. The producer gives it a transport of its own, so a full pool never delays it and its short
* response timeout bounds how long a daemon that accepts a connection but never answers holds it.
*/
static Probe pingOver(DockerHttpClient transport) {
return () -> {
DockerHttpClient.Request ping = DockerHttpClient.Request.builder()
.method(DockerHttpClient.Request.Method.GET)
.path("/_ping")
.build();
try (DockerHttpClient.Response ignored = transport.execute(ping)) {
return PingResult.ANSWERED;
} catch (RuntimeException e) {
LOG.debugv("Docker daemon ping failed: {0}", e.getMessage());
return DockerRetry.isDaemonFailure(e) ? PingResult.UNREACHABLE : PingResult.INCONCLUSIVE;
}
};
}
boolean isEnabled() {
return threshold > 0;
}
boolean isOpen() {
return open;
}
/** Sets how the breaker pings the daemon. */
void useProbe(Probe newProbe) {
probe = newProbe;
}
/** How many pings have finished, for tests that wait on one. */
int probesRun() {
return probesRun.get();
}
/** Whether a ping is scheduled or running. */
boolean isProbeScheduled() {
return probeScheduled.get();
}
/** The exception an open breaker raises in place of a Docker call. */
DockerUnavailableException unavailable() {
return new DockerUnavailableException("Docker daemon is unreachable: " + threshold
+ " consecutive Docker call attempts failed at the transport and it has not answered a ping since"
+ " (last failure: " + lastFailure + ")");
}
/** A Docker call got an HTTP response, whatever its status: the daemon is there. */
void recordResponse() {
consecutiveFailures.set(0);
if (open) {
closeCircuit();
}
}
/** A Docker call failed at the transport, as {@link DockerRetry#isDaemonFailure} classifies it. */
void recordFailure(Throwable failure) {
if (!isEnabled()) {
return;
}
lastFailure = String.valueOf(failure.getMessage());
if (consecutiveFailures.incrementAndGet() >= threshold && !open) {
scheduleProbe(0);
}
}
private void scheduleProbe(long delayMillis) {
if (probe == null || !probeScheduled.compareAndSet(false, true)) {
return;
}
synchronized (schedulerLock) {
if (closed) {
probeScheduled.set(false);
return;
}
if (scheduler == null) {
ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(1, runnable -> {
Thread thread = new Thread(runnable, "floci-docker-breaker-probe");
thread.setDaemon(true);
return thread;
});
executor.setRemoveOnCancelPolicy(true);
scheduler = executor;
}
scheduler.schedule(this::runProbe, delayMillis, TimeUnit.MILLISECONDS);
}
}
private void runProbe() {
PingResult result;
try {
result = probe.ping();
} catch (RuntimeException e) {
LOG.debugv("Docker daemon ping failed: {0}", e.getMessage());
result = DockerRetry.isDaemonFailure(e) ? PingResult.UNREACHABLE : PingResult.INCONCLUSIVE;
}
probeScheduled.set(false);
try {
switch (result) {
case ANSWERED -> {
consecutiveFailures.set(0);
if (open) {
closeCircuit();
}
}
case UNREACHABLE -> {
if (!open && !openCircuit()) {
return;
}
scheduleProbe(probeInterval.toMillis());
}
case INCONCLUSIVE -> {
// Neither opens nor closes: a closed breaker pings again on the next transport failure.
if (open) {
scheduleProbe(probeInterval.toMillis());
}
}
}
} finally {
probesRun.incrementAndGet();
}
}
/**
* Opens the breaker unless a call got an answer since the failures that triggered the ping. The
* open is written before the count is read again, and {@link #recordResponse} resets the count
* before it reads the open, so a response racing this either shows here as a reset count or sees
* the open and closes it: it can never leave the breaker open behind it.
*/
private boolean openCircuit() {
if (consecutiveFailures.get() < threshold) {
return false;
}
synchronized (schedulerLock) {
open = true;
}
if (consecutiveFailures.get() < threshold) {
closeCircuit();
return false;
}
LOG.warnv("Docker daemon is unreachable: {0} consecutive Docker call attempts failed at the transport"
+ " and a ping failed too, so Docker calls fail at once until the daemon answers a ping (tried"
+ " every {1} s). Last failure: {2}", threshold, probeInterval.toSeconds(), lastFailure);
return true;
}
private void closeCircuit() {
synchronized (schedulerLock) {
if (!open) {
return;
}
open = false;
}
LOG.info("Docker daemon answered again; Docker calls go through");
}
@Override
public void close() {
synchronized (schedulerLock) {
closed = true;
if (scheduler != null) {
scheduler.shutdownNow();
scheduler = null;
}
}
}
}
@@ -55,14 +55,23 @@ public class DockerClientProducer {
private static final ObjectMapper DOCKER_CONTEXT_JSON_MAPPER = new ObjectMapper();
/** Bounds the connect and the answer of the Docker circuit breaker's confirming ping. */
static final Duration PING_TIMEOUT = Duration.ofSeconds(5);
private final EmulatorConfig config;
private final DockerPoolGaugeLogger gaugeLogger;
private final DockerCircuitBreaker breaker;
@Inject
public DockerClientProducer(EmulatorConfig config) {
this.config = config;
this.gaugeLogger = new DockerPoolGaugeLogger(
validatePoolGaugeIntervalSeconds(config.docker().poolGaugeIntervalSeconds()));
int breakerThreshold = validateBreakerFailureThreshold(config.docker().breakerFailureThreshold());
// The probe interval only matters, and is only checked, while the breaker is on.
Duration probeInterval = breakerThreshold == 0 ? Duration.ZERO
: Duration.ofSeconds(validateBreakerProbeIntervalSeconds(config.docker().breakerProbeIntervalSeconds()));
this.breaker = new DockerCircuitBreaker(breakerThreshold, probeInterval);
}
/**
@@ -426,6 +435,20 @@ public class DockerClientProducer {
return seconds;
}
/** 0 turns the Docker circuit breaker off; a negative threshold is a misconfiguration. */
static int validateBreakerFailureThreshold(int threshold) {
if (threshold < 0) {
throw new IllegalArgumentException("floci.docker.breaker-failure-threshold "
+ "(FLOCI_DOCKER_BREAKER_FAILURE_THRESHOLD) must be 0 or more, got: " + threshold);
}
return threshold;
}
static int validateBreakerProbeIntervalSeconds(int seconds) {
return requireAtLeastOne(seconds, "floci.docker.breaker-probe-interval-seconds",
"FLOCI_DOCKER_BREAKER_PROBE_INTERVAL_SECONDS");
}
private static int requireAtLeastOne(int value, String setting, String envVar) {
if (value < 1) {
throw new IllegalArgumentException(setting + " (" + envVar + ") must be at least 1, got: " + value);
@@ -465,9 +488,43 @@ public class DockerClientProducer {
void disposeDockerClient(@Disposes DockerClient client) throws IOException {
gaugeLogger.unregister("control-plane");
breaker.close();
client.close();
}
/**
* Gives the breaker a ping that opens a one-connection transport of its own for each ping and closes it
* after, so a ping never waits for a pooled connection behind the calls whose failures it is confirming,
* {@link #PING_TIMEOUT} bounds the connect and the answer, and no transport outlives the client whose
* build set the probe. Pings are rare: one after a run of transport failures, then one per probe
* interval while the breaker is open.
*/
private void usePingTransport(DefaultDockerClientConfig clientConfig) {
if (!breaker.isEnabled()) {
return;
}
breaker.useProbe(() -> {
try (FlociDockerHttpClient transport = newPingTransport(clientConfig, PING_TIMEOUT)) {
return DockerCircuitBreaker.pingOver(transport).ping();
} catch (IOException e) {
LOG.debugv("Closing the Docker breaker's ping transport failed: {0}", e.getMessage());
return DockerCircuitBreaker.PingResult.INCONCLUSIVE;
}
});
}
/** One connection whose connect, lease and answer are each bounded by {@code timeout}. */
static FlociDockerHttpClient newPingTransport(DefaultDockerClientConfig clientConfig, Duration timeout) {
return new FlociDockerHttpClient.Builder()
.dockerHost(clientConfig.getDockerHost())
.sslConfig(clientConfig.getSSLConfig())
.maxConnections(1)
.connectionTimeout(timeout)
.responseTimeout(timeout)
.connectionRequestTimeout(timeout)
.build();
}
void disposeStreamingDockerClient(@Disposes @StreamingDocker DockerClient client) throws IOException {
gaugeLogger.unregister("streaming");
client.close();
@@ -518,7 +575,8 @@ public class DockerClientProducer {
Duration.ofSeconds(validateConnectionRequestTimeoutSeconds(
config.docker().connectionRequestTimeoutSeconds())));
gaugeLogger.register(role, transport::gauges);
return DockerClientImpl.getInstance(clientConfig, wrapForRole(transport, role, apiTimeout));
usePingTransport(clientConfig);
return DockerClientImpl.getInstance(clientConfig, wrapForRole(transport, role, apiTimeout, breaker));
}
static FlociDockerHttpClient newHttpClient(DefaultDockerClientConfig clientConfig, int maxConnections,
@@ -543,6 +601,20 @@ public class DockerClientProducer {
* silent daemon, not the length of each of its attempts.
*/
static DockerHttpClient wrapForRole(DockerHttpClient transport, String role, Duration callTimeout) {
return wrapForRole(transport, role, callTimeout, null);
}
/**
* As {@link #wrapForRole(DockerHttpClient, String, Duration)}, with {@code breaker} (shared by both
* roles) between the retry layer and the transport, so an open breaker fails a call before the
* retry layer could replay it, and every attempt's outcome reaches the breaker. A null or disabled
* breaker leaves the transport as it is.
*/
static DockerHttpClient wrapForRole(DockerHttpClient transport, String role, Duration callTimeout,
DockerCircuitBreaker breaker) {
if (breaker != null && breaker.isEnabled()) {
transport = new CircuitBreakingDockerHttpClient(transport, breaker);
}
if (!"control-plane".equals(role)) {
// The streaming transport replays only container waits, long-lived streams the call timeout never bounds.
return RetryingDockerHttpClient.forContainerWaits(transport);
@@ -96,7 +96,7 @@ public final class DockerRetry {
if (c instanceof InterruptedIOException && !(c instanceof SocketTimeoutException)) {
return false;
}
if (c instanceof LocalFailure) {
if (c instanceof LocalFailure || c instanceof DockerUnavailableException) {
return false;
}
if (isUnusableSocket(c)) {
@@ -128,6 +128,30 @@ public final class DockerRetry {
return false;
}
/**
* True when {@code t} says the daemon could not be reached at the transport, what
* {@link DockerCircuitBreaker} counts: a transient I/O failure ({@link #isTransientIo}) or a
* socket that is missing or may not be opened. A wait for a pooled connection, an interruption, a
* failure raised on this host ({@link LocalFailure}) and the breaker's own fail-fast exception say
* nothing about the daemon and do not count.
*/
static boolean isDaemonFailure(Throwable t) {
boolean unusableSocket = false;
for (Throwable c = t; c != null; c = c.getCause()) {
if (c instanceof InterruptedIOException && !(c instanceof SocketTimeoutException)) {
return false;
}
if (c instanceof LocalFailure || c instanceof DockerUnavailableException) {
return false;
}
unusableSocket |= isUnusableSocket(c);
if (c.getCause() == c) {
break;
}
}
return unusableSocket || isTransientIo(t);
}
/**
* True when {@code t} reports that the docker socket file does not exist ({@code No such
* file or directory}), as when Floci runs without a mounted docker socket, or that this
@@ -0,0 +1,16 @@
package io.floci.aws.core.common.docker;
import java.io.IOException;
/**
* Raised instead of a Docker call while {@link DockerCircuitBreaker} is open: the daemon failed
* several calls in a row at the transport and did not answer a ping since, so the call fails at once
* rather than waiting out its own timeouts. {@link DockerRetry#isTransientIo} never treats it as
* transient, so no retry layer replays it.
*/
public final class DockerUnavailableException extends IOException {
public DockerUnavailableException(String message) {
super(message);
}
}
+7
View File
@@ -433,6 +433,13 @@ floci:
# threads, written only while a pool is in use; 0 turns them off. Keep in step with the
# @WithDefault on EmulatorConfig.DockerConfig.poolGaugeIntervalSeconds: a value set here WINS over the annotation.
pool-gauge-interval-seconds: 60
# FLOCI_DOCKER_BREAKER_FAILURE_THRESHOLD: consecutive Docker call attempts (each retry counts)
# that fail at the transport before Floci pings the daemon and, if that fails too, fails every Docker call at once until the
# daemon answers; 0 turns this off. FLOCI_DOCKER_BREAKER_PROBE_INTERVAL_SECONDS: seconds between
# those pings. Keep both in step with the @WithDefault on EmulatorConfig.DockerConfig: a value set
# here WINS over the annotation.
breaker-failure-threshold: 3
breaker-probe-interval-seconds: 5
resource-namespace: "" # FLOCI_DOCKER_RESOURCE_NAMESPACE; scopes child container/volume names
docker-config-path: ""
# extra-labels: # Extra Docker labels applied to every Floci-created container and volume
@@ -220,4 +220,18 @@ class ApplicationDefaultsTest {
assertEquals("60", fallback.value(),
"the @WithDefault must match application.yml, which wins over it");
}
// Catches: the Docker circuit breaker keys missing from the shipped application.yml, or a yml value
// that silently disagrees with its @WithDefault fallback.
@Test
void productionConfigDeclaresTheDockerBreakerAtItsDefaults() throws IOException, NoSuchMethodException {
JsonNode docker = productionConfig().path("floci").path("docker");
assertEquals(3, docker.path("breaker-failure-threshold").asInt(-1));
assertEquals(5, docker.path("breaker-probe-interval-seconds").asInt(-1));
assertEquals("3", EmulatorConfig.DockerConfig.class.getMethod("breakerFailureThreshold")
.getAnnotation(WithDefault.class).value(), "the @WithDefault must match application.yml");
assertEquals("5", EmulatorConfig.DockerConfig.class.getMethod("breakerProbeIntervalSeconds")
.getAnnotation(WithDefault.class).value(), "the @WithDefault must match application.yml");
}
}
@@ -0,0 +1,314 @@
package io.floci.aws.core.common.docker;
import com.github.dockerjava.core.DefaultDockerClientConfig;
import com.github.dockerjava.transport.DockerHttpClient;
import com.github.dockerjava.transport.DockerHttpClient.Request;
import com.github.dockerjava.transport.DockerHttpClient.Response;
import io.floci.aws.core.common.docker.DockerCircuitBreaker.PingResult;
import org.apache.hc.core5.http.ConnectionRequestTimeoutException;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.InetAddress;
import java.net.ServerSocket;
import java.net.SocketException;
import java.nio.file.NoSuchFileException;
import java.time.Duration;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.BooleanSupplier;
import java.util.function.Supplier;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertInstanceOf;
import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTimeoutPreemptively;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assertions.fail;
class DockerCircuitBreakerTest {
private static final Request GET_INFO = Request.builder().method(Request.Method.GET).path("/info").build();
private DockerCircuitBreaker breaker;
@AfterEach
void closeBreaker() {
if (breaker != null) {
breaker.close();
}
}
// Catches: a dead daemon still costing every call its own connect, response and retry timeouts.
@Test
void opensAfterConsecutiveTransportFailuresAndAFailedPing_thenFailsCallsWithoutTheTransport() {
breaker = new DockerCircuitBreaker(3, Duration.ofMillis(50));
AtomicInteger pings = new AtomicInteger();
breaker.useProbe(() -> {
pings.incrementAndGet();
return PingResult.UNREACHABLE;
});
CountingTransport transport = CountingTransport.failing(() -> true);
DockerHttpClient client = new CircuitBreakingDockerHttpClient(transport, breaker);
for (int i = 0; i < 3; i++) {
assertThrows(RuntimeException.class, () -> client.execute(GET_INFO));
}
awaitTrue(breaker::isOpen, "three transport failures and a failed ping should open the breaker");
RuntimeException failFast = assertThrows(RuntimeException.class, () -> client.execute(GET_INFO));
assertInstanceOf(DockerUnavailableException.class, failFast.getCause());
assertTrue(failFast.getCause().getMessage().contains("Broken pipe"),
"the fail-fast message should carry the last transport failure: " + failFast.getCause().getMessage());
assertEquals(3, transport.calls.get(), "an open breaker must not reach the transport");
assertTrue(pings.get() >= 1);
}
// Catches: a burst of broken pipes during a fan-out build taking Docker away from every service.
@Test
void staysClosedWhenThePingAnswers() {
breaker = new DockerCircuitBreaker(3, Duration.ofMillis(50));
AtomicInteger pings = new AtomicInteger();
breaker.useProbe(() -> {
pings.incrementAndGet();
return PingResult.ANSWERED;
});
DockerHttpClient client = new CircuitBreakingDockerHttpClient(CountingTransport.failing(() -> true), breaker);
for (int i = 0; i < 3; i++) {
assertThrows(RuntimeException.class, () -> client.execute(GET_INFO));
}
awaitTrue(() -> breaker.probesRun() == 1, "the third failure should ping the daemon");
assertFalse(breaker.isOpen(), "a daemon that answers the ping must not open the breaker");
assertFalse(breaker.isProbeScheduled(), "a closed breaker should not schedule another ping");
assertEquals(1, pings.get());
}
// Catches: a daemon that answers its ping with an error status (a 500, a 503) being taken for gone.
@Test
void aPingAnsweredWithAnErrorStatusKeepsTheBreakerClosed() {
DockerCircuitBreaker.Probe ping = DockerCircuitBreaker.pingOver(
new CountingTransport(() -> new StatusResponse(500)));
assertEquals(PingResult.ANSWERED, ping.ping());
breaker = new DockerCircuitBreaker(1, Duration.ofMillis(50));
breaker.useProbe(ping);
DockerHttpClient client = new CircuitBreakingDockerHttpClient(CountingTransport.failing(() -> true), breaker);
assertThrows(RuntimeException.class, () -> client.execute(GET_INFO));
awaitTrue(() -> breaker.probesRun() == 1, "the failure should ping the daemon");
assertFalse(breaker.isOpen(), "an HTTP status from the ping is an answer");
}
// Catches: a ping that only waited for a pooled connection, during the fan-out the ping is there
// for, opening the breaker over a healthy daemon.
@Test
void aPingThatOnlyWaitsForAConnectionNeitherOpensNorStopsLaterPings() {
DockerCircuitBreaker.Probe ping = DockerCircuitBreaker.pingOver(new CountingTransport(() -> {
throw new RuntimeException(new ConnectionRequestTimeoutException("lease"));
}));
assertEquals(PingResult.INCONCLUSIVE, ping.ping());
assertEquals(PingResult.UNREACHABLE,
DockerCircuitBreaker.pingOver(CountingTransport.failing(() -> true)).ping());
breaker = new DockerCircuitBreaker(1, Duration.ofMillis(50));
breaker.useProbe(ping);
DockerHttpClient client = new CircuitBreakingDockerHttpClient(CountingTransport.failing(() -> true), breaker);
assertThrows(RuntimeException.class, () -> client.execute(GET_INFO));
awaitTrue(() -> breaker.probesRun() == 1, "the failure should ping the daemon");
assertFalse(breaker.isOpen(), "a ping that never reached the daemon must not open the breaker");
assertThrows(RuntimeException.class, () -> client.execute(GET_INFO));
awaitTrue(() -> breaker.probesRun() == 2, "the next transport failure should ping again");
}
// Catches: the confirming ping waiting out the API timeout (300 s by default) on a daemon that
// accepts the connection and never answers, while every call keeps entering the transport.
@Test
void thePingTransportGivesUpOnASilentDaemonWithinItsTimeout() throws IOException {
try (ServerSocket silent = new ServerSocket(0, 1, InetAddress.getLoopbackAddress())) {
DefaultDockerClientConfig config = DefaultDockerClientConfig.createDefaultConfigBuilder()
.withDockerHost("tcp://127.0.0.1:" + silent.getLocalPort())
.withDockerTlsVerify(false)
.build();
try (FlociDockerHttpClient transport = DockerClientProducer.newPingTransport(config, Duration.ofMillis(300))) {
DockerCircuitBreaker.Probe ping = DockerCircuitBreaker.pingOver(transport);
PingResult result = assertTimeoutPreemptively(Duration.ofSeconds(10), ping::ping,
"the ping must give up after its own timeout, not the API timeout");
assertEquals(PingResult.UNREACHABLE, result);
}
}
}
// Catches: a daemon that answers with an error status (a 500, a 404) being treated as gone.
@Test
void anHttpErrorResponseResetsTheCount() {
breaker = new DockerCircuitBreaker(2, Duration.ofMillis(50));
AtomicBoolean pinged = new AtomicBoolean();
breaker.useProbe(() -> {
pinged.set(true);
return PingResult.UNREACHABLE;
});
AtomicInteger call = new AtomicInteger();
CountingTransport transport = new CountingTransport(() -> {
if (call.incrementAndGet() % 2 == 1) {
throw new RuntimeException(new SocketException("Broken pipe"));
}
return new StatusResponse(500);
});
DockerHttpClient client = new CircuitBreakingDockerHttpClient(transport, breaker);
for (int i = 0; i < 6; i++) {
try {
client.execute(GET_INFO).close();
} catch (RuntimeException expected) {
// Every other call fails at the transport; the 500 in between proves the daemon is there.
}
}
assertFalse(pinged.get(), "failures interleaved with HTTP responses never reach the threshold");
assertFalse(breaker.isOpen());
}
// Catches: the retry layer replaying the breaker's fail-fast exception for about fifteen seconds.
@Test
void theRetryLayerOverAnOpenBreakerMakesOneAttemptAndNeverBacksOff() {
breaker = new DockerCircuitBreaker(1, Duration.ofMillis(50));
breaker.useProbe(() -> PingResult.UNREACHABLE);
CountingTransport transport = CountingTransport.failing(() -> true);
DockerHttpClient breaking = new CircuitBreakingDockerHttpClient(transport, breaker);
assertThrows(RuntimeException.class, () -> breaking.execute(GET_INFO));
awaitTrue(breaker::isOpen, "one failure and a failed ping should open a threshold-1 breaker");
DockerHttpClient retrying = new RetryingDockerHttpClient(breaking, 6, 500L, null, System::nanoTime,
millis -> fail("an open breaker must not be retried, but the retry layer backed off " + millis + " ms"));
RuntimeException e = assertThrows(RuntimeException.class, () -> retrying.execute(GET_INFO));
assertInstanceOf(DockerUnavailableException.class, e.getCause());
assertEquals(1, transport.calls.get(), "only the call that opened the breaker reached the transport");
}
// Catches: a breaker that never closes after the daemon comes back.
@Test
void aPingThatAnswersClosesAnOpenBreaker() {
breaker = new DockerCircuitBreaker(1, Duration.ofMillis(50));
AtomicBoolean daemonBack = new AtomicBoolean();
breaker.useProbe(() -> daemonBack.get() ? PingResult.ANSWERED : PingResult.UNREACHABLE);
AtomicBoolean failing = new AtomicBoolean(true);
CountingTransport transport = CountingTransport.failing(failing::get);
DockerHttpClient client = new CircuitBreakingDockerHttpClient(transport, breaker);
assertThrows(RuntimeException.class, () -> client.execute(GET_INFO));
awaitTrue(breaker::isOpen, "the breaker should open");
daemonBack.set(true);
failing.set(false);
awaitTrue(() -> !breaker.isOpen(), "a ping that answers should close the breaker");
client.execute(GET_INFO).close();
assertEquals(2, transport.calls.get(), "calls reach the transport again once the breaker closes");
}
// Catches: a missing Docker socket going uncounted, or a full pool or the fail-fast exception being
// read as a dead daemon.
@Test
void countsUnusableSocketsAndTransientIoButNotLeaseWaitsOrItsOwnException() {
assertTrue(DockerRetry.isDaemonFailure(new RuntimeException(new SocketException("Connection reset"))));
assertTrue(DockerRetry.isDaemonFailure(new RuntimeException(new NoSuchFileException("/var/run/docker.sock"))));
assertFalse(DockerRetry.isDaemonFailure(new RuntimeException(new ConnectionRequestTimeoutException("lease"))));
assertFalse(DockerRetry.isDaemonFailure(new RuntimeException(new DockerUnavailableException("open"))));
assertFalse(DockerRetry.isTransientIo(new RuntimeException(new DockerUnavailableException("open"))),
"the retry layer must never replay the breaker's fail-fast exception");
}
// Catches: the producer leaving the breaker out of the transport stack, or putting it above the retry
// layer, where an open breaker would still be retried.
@Test
void producerPutsTheBreakerUnderTheRetryLayerOfBothTransports() {
breaker = new DockerCircuitBreaker(1, Duration.ofMillis(50));
breaker.useProbe(() -> PingResult.UNREACHABLE);
CountingTransport transport = CountingTransport.failing(() -> true);
DockerHttpClient control = DockerClientProducer.wrapForRole(transport, "control-plane",
Duration.ofSeconds(30), breaker);
assertSame(RetryingDockerHttpClient.class, control.getClass());
Request ping = Request.builder().method(Request.Method.GET).path("/_ping").build();
assertThrows(RuntimeException.class, () -> control.execute(ping));
awaitTrue(breaker::isOpen, "a failure under the control-plane transport should open the breaker");
int callsWhenOpened = transport.calls.get();
DockerHttpClient streaming = DockerClientProducer.wrapForRole(transport, "streaming",
Duration.ofSeconds(30), breaker);
RuntimeException e = assertThrows(RuntimeException.class, () -> streaming.execute(GET_INFO));
assertInstanceOf(DockerUnavailableException.class, e.getCause());
assertEquals(callsWhenOpened, transport.calls.get(), "the shared breaker must also stop the streaming transport");
}
private static void awaitTrue(BooleanSupplier condition, String message) {
long deadline = System.nanoTime() + Duration.ofSeconds(5).toNanos();
while (!condition.getAsBoolean()) {
if (System.nanoTime() > deadline) {
fail(message);
}
Thread.onSpinWait();
}
}
private static final class CountingTransport implements DockerHttpClient {
private final Supplier<Response> behaviour;
final AtomicInteger calls = new AtomicInteger();
CountingTransport(Supplier<Response> behaviour) {
this.behaviour = behaviour;
}
static CountingTransport failing(BooleanSupplier fails) {
return new CountingTransport(() -> {
if (fails.getAsBoolean()) {
throw new RuntimeException(new SocketException("Broken pipe"));
}
return new StatusResponse(200);
});
}
@Override
public Response execute(Request request) {
calls.incrementAndGet();
return behaviour.get();
}
@Override
public void close() throws IOException {
// Nothing to release.
}
}
private record StatusResponse(int status) implements Response {
@Override
public int getStatusCode() {
return status;
}
@Override
public Map<String, List<String>> getHeaders() {
return Map.of();
}
@Override
public InputStream getBody() {
return new ByteArrayInputStream(new byte[0]);
}
@Override
public void close() {
// Nothing to release.
}
}
}