substrate/docs/request-parking.md

# Request Parking (atenet router)

## Summary

**Request parking** lets the `atenet` router hold ("park") an inbound request
whose target actor cannot be served *yet* because of transient worker-pool
saturation, retrying the resume until the actor becomes routable or a bounded
wait elapses — instead of immediately returning `503` to the client.

## Motivation

When a request arrives for a suspended actor, the router resumes it before
routing:

```
Envoy --(ext_proc RequestHeaders)--> router.handleRequestHeaders
    --> ActorResumer.ResumeActor --> ateapi ResumeActor (gRPC)
```

`ateapi`'s `AssignWorkerStep` claims a free worker from the actor's `WorkerPool`.
In an oversubscribed system — the core premise of Substrate, where many actors
multiplex onto few workers — a burst of traffic can momentarily exhaust the
pool. `AssignWorkerStep` then returns `ResourceExhausted: "no free workers
available"`.

Previously the router mapped that straight to an HTTP `503` and failed the
request. But such saturation is usually momentary: another actor suspends within
milliseconds and frees its worker. Failing fast turns a sub-second blip into a
user-visible error.

## Behavior

With parking enabled (the default), the router treats `ResourceExhausted`,
`FailedPrecondition` and `Unavailable` from `ResumeActor` as **retryable**
conditions (alongside the existing `Aborted` concurrent-resume conflict) — a
parked request rides out transient pool saturation and control-plane blips
(e.g. an ateapi rolling restart) alike. The request is *parked*: the resumer
keeps retrying with exponential backoff until either

- the resume succeeds (the actor is `RUNNING` and has a worker IP) — the request
  is then routed normally; or
- the **park budget** (`--parked-request-budget`, default `5s`) elapses — the
  underlying capacity error is returned, surfacing as `503 "actor <id>
  unavailable: no free workers available"`.

**The budget bounds retries, not a committed resume.** When the budget elapses
the router stops starting new resume attempts, but an attempt already in
flight is **never canceled**: by then the control plane has committed work to
it, and canceling would discard an in-progress restore. Instead the router
waits for that attempt's real result: a restore that overshoots the budget
(routine under node contention) is **served late** rather than failed, and a
late retryable error still surfaces as the capacity `503`. The attempt is
bounded by the control plane's own server-side RPC deadline, and Envoy's
ext_proc message timeout (`budget + 5s`) remains the ceiling on how long a
client is held either way.

**Worst-case occupancy.** A parked request's lot slot is held until its caller
stops waiting; with the Envoy dataplane that is bounded by the ext_proc
message timeout (`budget + 5s`). The resume attempt itself can outlive every
caller: it carries no client-side deadline and runs until ateapi's
server-side maximum RPC deadline, holding that actor's flight entry.
New requests for the same actor during that window do not start another
control-plane call — they join the in-flight attempt, and if it has not
resolved by their own stream deadline they are ended by the dataplane's
timeout rather than a router verdict.

To bound resource use and provide backpressure, parked requests are admitted to
a **parking lot** of fixed capacity (`--parked-request-max`, default `1024`). A
request occupies a slot only from the moment it actually parks — its resume
flight's first retryable failure. A request resolved on the flight's first
attempt (the actor was already running) never occupies a slot. When the lot is
full, a request reaching its park transition is shed with `503 "actor <id>
unavailable: router at capacity"` rather than queueing without bound — at the
cost of exactly the one resume attempt that revealed it would have to wait.

Every parked request holds one ext_proc stream — one active request against
Envoy's ext_proc cluster — for its entire wait, while ordinary requests hold
one only for a millisecond-scale header exchange. The cluster's circuit breaker
is therefore the hard ceiling on concurrent parked requests. By default the
router **derives** it as twice `--parked-request-max` (minimum `1024`), so the
lot always fits and an equal share of **fast-path headroom** remains — a
saturated lot cannot starve requests to already-running actors, at any lot
size. The lot upholds the same guarantee on its side: admission happens at the
park transition, so fast-path requests never compete for slots (#1081).
`--extproc-max-requests` overrides the derivation; explicit values are
validated `>= --parked-request-max` at startup, because a breaker below the lot
would silently truncate it — Envoy would reject the overflow itself, with 503s
that never reach the lot and never count in `parking.rejected`.

Concurrent requests for the *same* actor are de-duplicated by the resumer's
per-actor flight registry: they share a single in-flight `ResumeActor` call and
all park on its result, so a hot actor consumes N parking slots but only one
control-plane RPC.

**The park budget is per-flight, not per-request.** The budget clock starts
when a flight's first caller begins the resume; every later request for the
same actor joins that flight and shares its remaining budget and outcome. A
request that joins late may therefore see `budget_exhausted` after waiting far
less than a full budget itself — the accepted cost of collapsing a hot actor's
requests into one control-plane call. (`parking.wait.duration` records each
request's *own* parked time, so sub-budget `budget_exhausted` samples are
expected under sustained saturation.)

### What is *not* parked

Only transient conditions — capacity (`ResourceExhausted`), transient actor
state (`FailedPrecondition`), concurrency (`Aborted`), and control-plane
unavailability (`Unavailable`) — are parked. Errors that will not resolve by
waiting are returned immediately (fail fast):

| Resume result                          | Behavior                          |
| -------------------------------------- | --------------------------------- |
| `OK`                                   | Route to worker                   |
| `Aborted` (concurrent resume)          | Retry (always)                    |
| `ResourceExhausted` (no free worker)   | **Park & retry** (when enabled)   |
| `FailedPrecondition` (transient state) | **Park & retry** (when enabled)   |
| `Unavailable` (control-plane blip)     | **Park & retry** (when enabled)   |
| `NotFound`                             | Fail fast → `404`                 |
| `DeadlineExceeded`                     | Fail fast → `504`                 |
| `PermissionDenied` / `Unauthenticated` | Fail fast → `403` / `401`         |

When parking is **disabled** (`--parked-request-max=0`), the router fails fast:
`FailedPrecondition` and `Unavailable` are returned immediately, there is no
admission cap, and only `Aborted` (concurrent-resume) conflicts are retried,
within a `15s` budget.

### Parked requests survive router shutdown

A request parked when the router pod receives SIGTERM is **not** reset: the
shutdown sequence keeps the ext_proc server (and, via a preStop handshake, the
Envoy sidecar) alive until in-flight streams finish, and the ext_proc drain
deadline (`--drain-timeout`) defaults to a value derived from
`--parked-request-budget` and is validated at startup to be `>=` the budget —
so a parked request always gets its full budget and a normal verdict (routed
`200` or capacity `503`) even mid-termination. See the graceful-shutdown knobs
(`--drain-delay`, `--drain-timeout`) in `manifests/ate-install/atenet-router.yaml`.

## Configuration

| Flag                             | Default | Meaning                                                            |
| -------------------------------- | ------- | ------------------------------------------------------------------ |
| `--parked-request-budget`         | `5s`    | Park budget per resume *flight*; requests de-duplicated onto an in-flight resume share its remaining budget (see Behavior). |
| `--parked-request-max`            | `1024`  | Max concurrent **parked** requests (a slot is taken at the park transition, never for a first-attempt lookup); requests parking beyond it are shed (503). `0` disables parking. |
| `--parked-request-retry-interval` | `100ms` | Delay before a parked request's first resume retry.                |
| `--parked-request-retry-factor`   | `1.1`   | Multiplier applied to the retry delay after each attempt (>= 1).   |
| `--parked-request-retry-jitter`   | `0.1`   | Random fraction in `[0, 1)` added per retry to de-synchronize parked requests. |
| `--extproc-max-requests`          | `0` (auto) | Envoy circuit-breaker `max_requests` for the ext_proc cluster. `0` derives twice `--parked-request-max` (min `1024`); explicit values must be `>= --parked-request-max` (enforced at startup). The excess is fast-path headroom (see Behavior). |

The retry backoff deliberately has no cap and no attempt limit: the budget alone
bounds the wait.

## Observability

**Metrics** (OpenTelemetry, meter `atenet-router`):

- `atenet.router.parking.active` — up/down counter: requests currently parked.
- `atenet.router.parking.wait.duration` — histogram (seconds) of time spent
  parked. Recorded **exactly once per parked request**, at the moment its wait
  ends; never recorded for requests served on their flight's first attempt
  (those never park), for shed requests (those only increment
  `parking.rejected`), nor when parking is disabled. The `outcome` label says
  how the park ended:

  | `outcome`          | When it is set                                                              |
  | ------------------ | --------------------------------------------------------------------------- |
  | `served`           | The resume succeeded and the request was routed to its worker.              |
  | `budget_exhausted` | The park budget elapsed while the resume was still blocked on a retryable condition (pool saturated, a concurrent operation holding the actor, or the control plane unavailable) — the signal that capacity, not a fault, is the bottleneck. |
  | `canceled`         | The client disconnected while parked (request context canceled).            |
  | `timeout`          | The request's own deadline expired while parked (distinct from the park budget). |
  | `error`            | The resume failed with a non-retryable error (`NotFound`, `PermissionDenied`, ...). |

- `atenet.router.parking.rejected` — counter: requests shed because the lot was
  full.

**Status page** (`/statusz`): a "Request Parking" card shows whether parking is
enabled, the current vs. maximum parked count, and the max wait.