Merge pull request #7476 from ranxianglei/fix/429-governor-default-off

[PR] fix: 429 governor off by default — upstream 429s pass through to the client
This commit is contained in:
ranxianglei
2026-11-21 23:43:42 +08:00
committed by GitHub
13 changed files with 219 additions and 43 deletions
+2 -2
View File
@@ -170,7 +170,7 @@
| `network.preflightDeadEndCooldownMs` | number | 300000 | BILI_PREFLIGHT_DEAD_END_COOLDOWN_MS | 行き止まりのプリフライト後、同じセッションの調整を再試行するまでのクールダウン。 |
| `network.replayRetryMax` | number | 3 (1 disables replays) | BILI_REPLAY_RETRY_MAX | エラーを表面化させる前に、同一リクエストの最大トランスポート失敗リプレイ回数。 |
| `network.replayRetryBaseMs` | number | 1500 (0 disables the delay) | BILI_REPLAY_RETRY_BASE_MS | リプレイ間のベースバックオフ遅延(指数関数的)。 |
| `network.throttle429Enabled` | boolean | true | BILI_THROTTLE_429 | エグレス側の 429 ガバナナ(#4204):繰り返される上流 429 は転送されるのではなくローカルで応答される。アカウント制限および未認識/不正なレスポンスは origin+credential を保持する;KeyRing の正確な keyring_model_temporarily_unavailable コードと一致するリクエストモデルのみが origin+credential+model を保持し、他のモデルを適格とする。モデルの不一致または欠落はアカウント全体に適用される;false は従来の常に転送する動作を復元する。 |
| `network.throttle429Enabled` | boolean | false | BILI_THROTTLE_429 | エグレス側の 429 ガバナナ(#4204)、#6639/#5853 以降既定でオフ:上流の 429 は変更なしにクライアントへ転送され、bili は 1 件あたり [429-passthrough] warn ログを記録する。true(または BILI_THROTTLE_429=1)にするとエスケープハッチとして再有効化:繰り返される上流 429 が転送されずローカルで応答される。アカウント制限および未認識/不正なレスポンスは origin+credential を保持;KeyRing の正確な keyring_model_temporarily_unavailable コードと一致するリクエストモデルのみが origin+credential+model を保持し、他のモデルを適格とする。モデルの不一致または欠落はアカウント全体に適用される。 |
| `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | ホールドがターミナル(既知のすべての SDK リトライセットの外側のローカル 424)に変わるまでの連続上流 429 の回数;バックオフウィンドウごとに1回のプローブが、アカウントホールドをクリアする非 429 レスポンス、またはそのモデルホールドをクリアする一致するモデルの非 429 レスポンスがあるまでチェックを続ける。 |
| `network.throttle429BaseDelayMs` | number | 2000 | BILI_THROTTLE_429_BASE_DELAY_MS | 429 ホールドのベースバックオフ(連続する 429 ごとに倍増し、[0.5×, 1×] にジャッターされ、throttle429MaxDelayMs で上限が設定される;上流の Retry-After が大きい場合は優先される)。 |
| `network.throttle429MaxDelayMs` | number | 30000 | BILI_THROTTLE_429_MAX_DELAY_MS | 429 のバックオフおよびターミナルフェーズのプローブ間隔の上限。 |
@@ -2049,7 +2049,7 @@ ACP ネイティブエージェント(現在は `pi` 拡張機能)は、プ
| `BILI_STREAM_ERROR_SHAPE` | `compat.streamErrorShape` | protocol |
| `BILI_STREAM_KEEPALIVE_MS` | `network.streamKeepAliveMs` | 15000 (0 disables) |
| `BILI_SUBAGENT_SPLIT` | `subagentSplit` | true |
| `BILI_THROTTLE_429` | `network.throttle429Enabled` | true |
| `BILI_THROTTLE_429` | `network.throttle429Enabled` | false |
| `BILI_THROTTLE_429_BASE_DELAY_MS` | `network.throttle429BaseDelayMs` | 2000 |
| `BILI_THROTTLE_429_MAX_CONSECUTIVE` | `network.throttle429MaxConsecutive` | 5 |
| `BILI_THROTTLE_429_MAX_DELAY_MS` | `network.throttle429MaxDelayMs` | 30000 |
+2 -2
View File
@@ -173,7 +173,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed
| `network.preflightDeadEndCooldownMs` | number | 300000 | BILI_PREFLIGHT_DEAD_END_COOLDOWN_MS | Cooldown after a dead-end preflight before retrying reconciliation for the same session. |
| `network.replayRetryMax` | number | 3 (1 disables replays) | BILI_REPLAY_RETRY_MAX | Maximum transport-failure replays of an identical request before surfacing the error. |
| `network.replayRetryBaseMs` | number | 1500 (0 disables the delay) | BILI_REPLAY_RETRY_BASE_MS | Base backoff delay between replays (exponential). |
| `network.throttle429Enabled` | boolean | true | BILI_THROTTLE_429 | Egress-side 429 governor (#4204): repeated upstream 429s are answered locally instead of forwarded. Account limits and unrecognized/malformed responses hold origin+credential; only KeyRing's exact `keyring_model_temporarily_unavailable` code with a matching request model holds origin+credential+model, leaving other models eligible. A mismatch or missing model remains account-wide; false restores the legacy always-forward behavior. |
| `network.throttle429Enabled` | boolean | false | BILI_THROTTLE_429 | Egress-side 429 governor (#4204), OFF by default since #6639/#5853: upstream 429s are forwarded to the client unchanged and bili logs one [429-passthrough] warn per observed 429. Set true (or BILI_THROTTLE_429=1) to opt back in as an escape hatch: repeated upstream 429s are then answered locally instead of forwarded — account limits and unrecognized/malformed responses hold origin+credential; only KeyRing's exact `keyring_model_temporarily_unavailable` code with a matching request model holds origin+credential+model, leaving other models eligible. A mismatch or missing model remains account-wide. |
| `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | Consecutive upstream 429s before the hold turns terminal (local 424 outside every known SDK retry set); one probe per backoff window keeps checking until any non-429 response clears an account hold, or the matching model's non-429 response clears its model hold. |
| `network.throttle429BaseDelayMs` | number | 2000 | BILI_THROTTLE_429_BASE_DELAY_MS | Base backoff for the 429 hold (doubles per consecutive 429, jittered into [0.5×, 1×], capped by throttle429MaxDelayMs; an upstream Retry-After wins when larger). |
| `network.throttle429MaxDelayMs` | number | 30000 | BILI_THROTTLE_429_MAX_DELAY_MS | Ceiling for the 429 hold backoff and for the terminal-phase probe interval. |
@@ -2064,7 +2064,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes
| `BILI_STREAM_ERROR_SHAPE` | `compat.streamErrorShape` | protocol |
| `BILI_STREAM_KEEPALIVE_MS` | `network.streamKeepAliveMs` | 15000 (0 disables) |
| `BILI_SUBAGENT_SPLIT` | `subagentSplit` | true |
| `BILI_THROTTLE_429` | `network.throttle429Enabled` | true |
| `BILI_THROTTLE_429` | `network.throttle429Enabled` | false |
| `BILI_THROTTLE_429_BASE_DELAY_MS` | `network.throttle429BaseDelayMs` | 2000 |
| `BILI_THROTTLE_429_MAX_CONSECUTIVE` | `network.throttle429MaxConsecutive` | 5 |
| `BILI_THROTTLE_429_MAX_DELAY_MS` | `network.throttle429MaxDelayMs` | 30000 |
+2 -2
View File
@@ -173,7 +173,7 @@
| `network.preflightDeadEndCooldownMs` | number | 300000 | BILI_PREFLIGHT_DEAD_END_COOLDOWN_MS | 死胡同 preflight 后,对同一会话重试校准的冷却时长。 |
| `network.replayRetryMax` | number | 3 (1 disables replays) | BILI_REPLAY_RETRY_MAX | 同一请求在向上游暴露错误前的最大传输失败重放次数。 |
| `network.replayRetryBaseMs` | number | 1500 (0 disables the delay) | BILI_REPLAY_RETRY_BASE_MS | 重放之间的基础退避延迟(指数增长)。 |
| `network.throttle429Enabled` | boolean | true | BILI_THROTTLE_429 | 出站 429 治理器(#4204):连续上游 429 会在本地应答而不再转发。账号限额及无法识别/格式错误的响应按 origin+凭据持有;只有 KeyRing 精确返回 `keyring_model_temporarily_unavailable` 且响应模型与请求模型完全相同时,才按 origin+凭据+模型持有,让其他模型继续可用。模型不匹配或缺失仍按账号级处理;设为 false 恢复旧的总是转发行为。 |
| `network.throttle429Enabled` | boolean | false | BILI_THROTTLE_429 | 出站 429 治理器(#4204),自 #6639/#5853 起默认关闭:上游 429 原样转发给客户端,bili 每观测到一次 429 打一条 [429-passthrough] warn 日志。设为 true(或 BILI_THROTTLE_429=1)重新启用,作为逃生舱:连续上游 429 会在本地应答而不再转发。账号限额及无法识别/格式错误的响应按 origin+凭据持有;只有 KeyRing 精确返回 `keyring_model_temporarily_unavailable` 且响应模型与请求模型完全相同时,才按 origin+凭据+模型持有,让其他模型继续可用。模型不匹配或缺失仍按账号级处理。 |
| `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | 连续上游 429 达到该次数后 hold 转为终局(本地 424,不在任何已知 SDK 重试集合内);每个退避窗口仍放行一次探测,直到账号级 hold 收到任意非 429 响应,或模型级 hold 收到同一模型的非 429 响应后清除。 |
| `network.throttle429BaseDelayMs` | number | 2000 | BILI_THROTTLE_429_BASE_DELAY_MS | 429 hold 的基础退避(每连续一次 429 翻倍、抖动到 [0.5×, 1×]、以 throttle429MaxDelayMs 封顶;上游 Retry-After 更大时优先)。 |
| `network.throttle429MaxDelayMs` | number | 30000 | BILI_THROTTLE_429_MAX_DELAY_MS | 429 hold 退避的上限,也是终局阶段探测间隔的上限。 |
@@ -2041,7 +2041,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报
| `BILI_STREAM_ERROR_SHAPE` | `compat.streamErrorShape` | protocol |
| `BILI_STREAM_KEEPALIVE_MS` | `network.streamKeepAliveMs` | 15000 (0 disables) |
| `BILI_SUBAGENT_SPLIT` | `subagentSplit` | true |
| `BILI_THROTTLE_429` | `network.throttle429Enabled` | true |
| `BILI_THROTTLE_429` | `network.throttle429Enabled` | false |
| `BILI_THROTTLE_429_BASE_DELAY_MS` | `network.throttle429BaseDelayMs` | 2000 |
| `BILI_THROTTLE_429_MAX_CONSECUTIVE` | `network.throttle429MaxConsecutive` | 5 |
| `BILI_THROTTLE_429_MAX_DELAY_MS` | `network.throttle429MaxDelayMs` | 30000 |
+6 -4
View File
@@ -2342,10 +2342,12 @@ type FileConfig = {
replayRetryMax?: number;
/** Replay backoff base delay (was BILI_REPLAY_RETRY_BASE_MS; default 1500). */
replayRetryBaseMs?: number;
/** Egress-side 429 throttle governor master switch (#4204; was env-only BILI_THROTTLE_429; default true).
* After consecutive upstream 429s, further sends to that origin+credential are answered locally
* (429 + Retry-After, then a terminal 424) instead of being forwarded, so host-driven retry loops
* cannot pin a concurrency-style provider limit. false restores the legacy always-forward behavior. */
/** Egress-side 429 throttle governor master switch (#4204; was env-only BILI_THROTTLE_429; default false,
* #6639/#5853). OFF by default: upstream 429s are forwarded to the client untouched — the host owns the
* retry decision — and bili only logs a warn per observed 429. Set true for accounts where host-driven
* retry loops would pin a shared concurrency-style provider limit: after consecutive upstream 429s,
* further sends to that origin+credential are then answered locally (429 + Retry-After, then a terminal
* 424) instead of being forwarded. */
throttle429Enabled?: boolean;
/** Consecutive upstream 429s before the hold turns terminal — local 424 answers outside every known
* client-SDK retry set, so hosts stop re-driving (#4204; was env-only BILI_THROTTLE_429_MAX_CONSECUTIVE; default 5). */
+8 -6
View File
@@ -135,15 +135,17 @@ const THROTTLE_429_MAX_CONSECUTIVE = 5;
const THROTTLE_429_BASE_DELAY_MS = 2_000;
const THROTTLE_429_MAX_DELAY_MS = 30_000;
/** BILI_THROTTLE_429 > network.throttle429Enabled > true. The egress 429
* governor (#4204) is ON by default: consecutive upstream 429s put sends to
* that origin+credential on local backoff instead of amplifying a
* concurrency-style provider limit. "0"/"false" restores the legacy
* always-forward behavior. Read per request so the switch flips live. */
/** BILI_THROTTLE_429 > network.throttle429Enabled > false. The egress 429
* governor (#4204) is OFF by default (#6639/#5853): upstream 429s are
* forwarded to the client untouched — the host owns the retry decision —
* and bili only warns in the log. Opt in ("1"/"true" here, or
* network.throttle429Enabled: true) for accounts where host retry storms
* pin a shared concurrency-style provider limit (#4204). Read per request
* so the switch flips live. */
export function throttle429Enabled(): boolean {
const envRaw = process.env.BILI_THROTTLE_429;
if (envRaw !== undefined) return !(envRaw === "0" || envRaw === "false");
return fileNetwork().throttle429Enabled !== false;
return fileNetwork().throttle429Enabled === true;
}
/** BILI_THROTTLE_429_MAX_CONSECUTIVE > network.throttle429MaxConsecutive > 5.
+36 -8
View File
@@ -1,13 +1,20 @@
// #4204 — egress-side 429 throttle governor.
//
// Problem: a concurrency-style provider limit (z.ai GLM coding plan: ~9-18
// allowed in parallel, dynamic, no Retry-After) plus a host that re-drives a
// failed turn every few seconds with the full history. bili forwarded every
// re-driven request, so the retry stream itself occupied the provider's slots
// and pinned the WHOLE account at the limit for minutes — every other session
// failed too, until the client (and proxy process) was restarted.
// OFF BY DEFAULT (#6639/#5853 decision): upstream 429s are forwarded to the
// client untouched — the host owns the retry decision — and bili only logs a
// warn per observed 429. The governor below is an opt-in escape hatch for
// accounts where an unbounded host retry storm pins a SHARED concurrency-style
// limit and starves the other sessions on the same credential.
//
// Fix: consecutive 429s create a local hold by origin+credential. A verified
// Problem (when enabled): a concurrency-style provider limit (z.ai GLM coding
// plan: ~9-18 allowed in parallel, dynamic, no Retry-After) plus a host that
// re-drives a failed turn every few seconds with the full history. bili
// forwarded every re-driven request, so the retry stream itself occupied the
// provider's slots and pinned the WHOLE account at the limit for minutes —
// every other session failed too, until the client (and proxy process) was
// restarted.
//
// Fix (enabled mode): consecutive 429s create a local hold by origin+credential. A verified
// KeyRing model-endpoint refusal is keyed by origin+credential+request-model;
// account quota and every ambiguous/unrecognized 429 remain origin+credential.
// Further sends covered by that scope are answered LOCALLY instead of forwarded:
@@ -541,6 +548,21 @@ function createHoldState(
};
}
/** Governor disabled (#6639): the 429 is forwarded to the client as-is — the
* host owns the retry decision. Say so once per observed 429 so an operator
* can tell in the log that bili did NOT intercept it. */
function logThrottle429Passthrough(
host: string,
model: string | undefined,
retryAfterHeader: string | null,
now: number,
): void {
const raMs = parseRetryAfterMs(retryAfterHeader, now);
const raNote = raMs !== null ? `, upstream Retry-After ~${Math.round(raMs / 1000)}s (passed through, client decides)` : "";
const modelNote = model === undefined || model.length === 0 ? "" : ` for model "${model}"`;
loggerLog("warn", `[acp-proxy] [429-passthrough] ${host}${modelNote}: upstream 429 forwarded to the client unchanged (429 governor disabled — no local hold, no 424)${raNote} (#4204/#6639)`);
}
function logThrottle429(state: HoldState, retryAfterHeader: string | null, now: number): void {
const streak = state.streak;
const maxC = throttle429MaxConsecutive();
@@ -749,8 +771,12 @@ export function noteUpstreamThrottle429(
modelScope?: string,
probeToken?: symbol,
): void {
if (!throttle429Enabled()) return;
const { key, host, cred } = holdKey(url, headers);
if (!throttle429Enabled()) {
logThrottle429Passthrough(host, modelScope, retryAfterHeader, now);
releaseProbeToken(probeToken, now, key);
return;
}
releaseProbeToken(probeToken, now, key);
const scopeModel = modelScope === undefined || modelScope.length === 0 ? undefined : modelScope;
const stateKey = scopeModel === undefined ? key : modelHoldKey(key, scopeModel);
@@ -792,6 +818,8 @@ export function beginProvisionalUpstreamThrottle429(
): (modelScope?: string) => void {
if (!throttle429Enabled()) {
reservation?.release();
const { host } = holdKey(url, headers);
logThrottle429Passthrough(host, requestModel, retryAfterHeader, now);
return () => undefined;
}
const { key, host, cred } = holdKey(url, headers);
@@ -85,6 +85,10 @@ test("a bodyless continuation response still returns null", async () => {
});
test("a 429 continuation keeps its body for model-scope classification", async () => {
// #6639: governor is OFF by default now — this pin verifies the ENABLED
// path, so opt in explicitly (restored in finally).
const previousToggle = process.env.BILI_THROTTLE_429;
process.env.BILI_THROTTLE_429 = "1";
_resetUpstreamThrottleForTest();
let canceled = 0;
const response = new Response(new ReadableStream<Uint8Array>({
@@ -105,6 +109,8 @@ test("a 429 continuation keeps its body for model-scope classification", async (
assert.equal(_liveUpstreamTimersForTest(), 0);
} finally {
_resetUpstreamThrottleForTest();
if (previousToggle === undefined) delete process.env.BILI_THROTTLE_429;
else process.env.BILI_THROTTLE_429 = previousToggle;
}
});
+9
View File
@@ -24,8 +24,13 @@ function throttleReject(url: string, headers: Record<string, string>) {
// The loop-side chokepoint wiring is exercised by the e2e lanes.
test("429 governor contract: consecutive 429 verdicts arm the local gate; a resolved verdict lifts it", () => {
// #6639: the governor is OFF by default now — this pin verifies the
// ENABLED contract, so opt in explicitly (restored in finally).
const previousToggle = process.env.BILI_THROTTLE_429;
process.env.BILI_THROTTLE_429 = "1";
const url = "http://loop-observe-4725.test/v1/chat/completions";
const headers: Record<string, string> = {};
try {
// resolved verdict first: no hold exists — stays clean
noteUpstreamThrottleResolved(url, headers, 200);
assert.equal(throttleReject(url, headers), undefined, "no hold before any 429");
@@ -37,4 +42,8 @@ test("429 governor contract: consecutive 429 verdicts arm the local gate; a reso
// a settled success (exactly what the loop notes on 2xx) lifts the hold
noteUpstreamThrottleResolved(url, headers, 200);
assert.equal(throttleReject(url, headers), undefined, "resolved verdict lifts the hold");
} finally {
if (previousToggle === undefined) delete process.env.BILI_THROTTLE_429;
else process.env.BILI_THROTTLE_429 = previousToggle;
}
});
+23 -10
View File
@@ -42,6 +42,19 @@ function opts(url: string, label: string, signal: AbortSignal = new AbortControl
};
}
/** #6639: the 429 governor is OFF by default now — these pins verify the
* ENABLED behavior, so opt in explicitly via env (restored on every exit). */
function withGovernor<T>(fn: () => Promise<T>): Promise<T> {
const previous = process.env.BILI_THROTTLE_429;
process.env.BILI_THROTTLE_429 = "1";
return Promise.resolve()
.then(fn)
.finally(() => {
if (previous === undefined) delete process.env.BILI_THROTTLE_429;
else process.env.BILI_THROTTLE_429 = previous;
});
}
async function recordedServer(status: number, body: string): Promise<{ url: string; hits: () => number; close: () => Promise<void> }> {
let hitCount = 0;
const server = http.createServer((_req, res) => {
@@ -59,7 +72,7 @@ async function recordedServer(status: number, body: string): Promise<{ url: stri
};
}
test("#4743 A: an armed 429 hold answers the continuation refetch locally — the empty turn passes through and the upstream records ZERO hits", async () => {
test("#4743 A: an armed 429 hold answers the continuation refetch locally — the empty turn passes through and the upstream records ZERO hits", async () => withGovernor(async () => {
const srv = await recordedServer(200, "event: message_start\ndata: {}\n\n");
try {
// arm the governor for this exact upstream (same streak shape as the
@@ -76,9 +89,9 @@ test("#4743 A: an armed 429 hold answers the continuation refetch locally — th
await srv.close();
noteUpstreamThrottleResolved(srv.url, {}, 200);
}
});
}));
test("#4743 B: the refetch's own 429 verdict arms the governor; a resolved retry lifts the hold", async () => {
test("#4743 B: the refetch's own 429 verdict arms the governor; a resolved retry lifts the hold", async () => withGovernor(async () => {
const rejected = await recordedServer(429, JSON.stringify({ error: { code: "keyring_all_limited", model: "m", message: "all keys limited" } }));
const ok = await recordedServer(200, "event: message_start\ndata: {}\n\n");
try {
@@ -103,9 +116,9 @@ test("#4743 B: the refetch's own 429 verdict arms the governor; a resolved retry
await rejected.close();
await ok.close();
}
});
}));
test("#4743 C: a structured KeyRing model refusal holds only the matching model", async () => {
test("#4743 C: a structured KeyRing model refusal holds only the matching model", async () => withGovernor(async () => {
const rejected = await recordedServer(
429,
JSON.stringify({ error: { code: "keyring_model_temporarily_unavailable", model: "m", message: "model unavailable" } }),
@@ -127,11 +140,11 @@ test("#4743 C: a structured KeyRing model refusal holds only the matching model"
await rejected.close();
noteUpstreamThrottleResolved(rejected.url, {}, 200, Date.now(), "m");
}
});
}));
// Real timers are intentional: both concurrent response bodies finish after
// the former 1s cutoff, while the continuation callbacks stop waiting at 250ms.
test("#4743 D: concurrent delayed KeyRing bodies over 1s refine their own provisional holds without delaying callbacks", { timeout: 5_000 }, async () => {
test("#4743 D: concurrent delayed KeyRing bodies over 1s refine their own provisional holds without delaying callbacks", { timeout: 5_000 }, async () => withGovernor(async () => {
const url = "http://delayed-keyring-429.invalid/v1/chat/completions";
const host = new URL(url).host;
const timers: Array<ReturnType<typeof setTimeout>> = [];
@@ -223,9 +236,9 @@ test("#4743 D: concurrent delayed KeyRing bodies over 1s refine their own provis
noteUpstreamThrottleResolved(url, {}, 200, Date.now(), "m", mProbeToken);
noteUpstreamThrottleResolved(url, {}, 200, Date.now(), "n", nProbeToken);
}
});
}));
test("#4743 E: an oversized 429 body with a never-settling cancellation promise returns promptly with account scope", { timeout: 1_000 }, async () => {
test("#4743 E: an oversized 429 body with a never-settling cancellation promise returns promptly with account scope", { timeout: 1_000 }, async () => withGovernor(async () => {
const url = "http://oversized-keyring-429.invalid/v1/chat/completions";
let cancelCalled = false;
const transport = async () => new Response(
@@ -255,7 +268,7 @@ test("#4743 E: an oversized 429 body with a never-settling cancellation promise
} finally {
noteUpstreamThrottleResolved(url, {}, 200, Date.now(), "m");
}
});
}));
test("#4743 F: an aborted retry releases its own probe token without clearing the model hold", { timeout: 5_000 }, async () => {
const url = "http://aborted-keyring-429.invalid/v1/chat/completions";
@@ -0,0 +1,116 @@
// #6639/#5853 decision: the egress 429 governor (#4204) is OFF by default.
// With no BILI_THROTTLE_429 env and no network.throttle429Enabled config,
// upstream 429s are forwarded to the client untouched — no local hold, no
// terminal 424 — and bili logs one [429-passthrough] warn per observed 429.
// The governor stays available as an opt-in escape hatch (env "1" or config
// true) for accounts where host retry storms pin a shared concurrency limit.
import { test } from "node:test";
import assert from "node:assert/strict";
import {
activeThrottleHolds,
beginProvisionalUpstreamThrottle429,
noteUpstreamThrottle429,
upstreamThrottleGate,
upstreamThrottlePeekReject,
_resetUpstreamThrottleForTest,
} from "../src/upstream-throttle.ts";
import { throttle429Enabled } from "../src/knobs.ts";
import { setLogCapture } from "../src/logger.ts";
const URL_A = "http://z.example/api/coding/paas/v4/chat/completions";
const H_CRED_A: Record<string, string> = { authorization: "Bearer AAA" };
const T0 = 1_000_000;
const KNOB_ENVS = [
"BILI_THROTTLE_429",
"BILI_THROTTLE_429_MAX_CONSECUTIVE",
"BILI_THROTTLE_429_BASE_DELAY_MS",
"BILI_THROTTLE_429_MAX_DELAY_MS",
"BILI_THROTTLE_429_SCOPE",
"BILI_CONFIG_FILE",
] as const;
/** No throttle env, no reachable config file — the pure default. */
function withDefaults(fn: () => void): void {
const saved: Record<string, string | undefined> = {};
for (const k of KNOB_ENVS) saved[k] = process.env[k];
try {
for (const k of KNOB_ENVS) {
if (k === "BILI_CONFIG_FILE") process.env.BILI_CONFIG_FILE = "/nonexistent-bili-6639-config.json";
else delete process.env[k];
}
_resetUpstreamThrottleForTest();
fn();
} finally {
for (const k of KNOB_ENVS) {
if (saved[k] === undefined) delete process.env[k];
else process.env[k] = saved[k];
}
}
}
test("default (no env, no config): governor is OFF — the passthrough posture (#6639)", () => {
withDefaults(() => {
assert.equal(throttle429Enabled(), false);
});
});
test("default: 8 sequential upstream 429s never produce a local hold or a 424", () => {
withDefaults(() => {
for (let i = 0; i < 8; i++) {
const gate = upstreamThrottleGate(URL_A, H_CRED_A, T0 + i * 1000, "glm-5.2");
assert.equal(gate.reject, undefined, `request ${i + 1}: no local rejection`);
assert.equal(gate.probeToken, undefined, `request ${i + 1}: no probe token minted`);
assert.equal(upstreamThrottlePeekReject(URL_A, H_CRED_A, T0 + i * 1000 + 500, "glm-5.2"), undefined);
noteUpstreamThrottle429(URL_A, H_CRED_A, "600", T0 + i * 1000, "glm-5.2");
}
assert.equal(activeThrottleHolds().length, 0, "no hold state stored while disabled");
});
});
test("default: every observed 429 logs one [429-passthrough] warn (gate + provisional paths)", () => {
withDefaults(() => {
const captured: { level: string; msg: string }[] = [];
setLogCapture((level, msg) => captured.push({ level, msg }));
try {
noteUpstreamThrottle429(URL_A, H_CRED_A, "600", T0, "glm-5.2");
const settle = beginProvisionalUpstreamThrottle429(
URL_A, H_CRED_A, null, T0 + 1000, "glm-5.2",
);
settle("glm-5.2"); // settle must stay a no-op while disabled
const warns = captured.filter((c) => c.level === "warn" && c.msg.includes("[429-passthrough]"));
assert.equal(warns.length, 2, `expected 2 passthrough warns, got: ${captured.map((c) => c.msg).join(" | ")}`);
assert.match(warns[0]!.msg, /upstream 429 forwarded to the client unchanged/);
assert.match(warns[0]!.msg, /for model "glm-5\.2"/);
assert.match(warns[0]!.msg, /Retry-After ~600s/);
assert.match(warns[1]!.msg, /z\.example/);
// The provisional settler released its state: nothing armed.
assert.equal(activeThrottleHolds().length, 0);
} finally {
setLogCapture(null);
}
});
});
test("opt-in escape hatch: BILI_THROTTLE_429=1 re-arms the governor (#4204)", () => {
const savedEnv = process.env.BILI_THROTTLE_429;
const savedCfg = process.env.BILI_CONFIG_FILE;
process.env.BILI_THROTTLE_429 = "1";
process.env.BILI_CONFIG_FILE = "/nonexistent-bili-6639-config.json";
try {
_resetUpstreamThrottleForTest();
assert.equal(throttle429Enabled(), true);
noteUpstreamThrottle429(URL_A, H_CRED_A, "600", T0, "glm-5.2");
const hold = activeThrottleHolds()[0];
assert.ok(hold, "opt-in: hold state stored");
assert.equal(hold.streak, 1);
const gate = upstreamThrottleGate(URL_A, H_CRED_A, T0 + 10, "glm-5.2");
assert.ok(gate.reject, "opt-in: subsequent send rejected locally");
} finally {
_resetUpstreamThrottleForTest();
if (savedEnv === undefined) delete process.env.BILI_THROTTLE_429;
else process.env.BILI_THROTTLE_429 = savedEnv;
if (savedCfg === undefined) delete process.env.BILI_CONFIG_FILE;
else process.env.BILI_CONFIG_FILE = savedCfg;
}
});
+6 -6
View File
@@ -11,8 +11,8 @@ entries:
desc_en: "Transport timeouts, watchdogs and replay policy for upstream calls."
desc_zh: "上游调用的传输超时、看门狗与重放策略。"
desc_ja: "上流呼び出しのタイムアウト、ウォッチドッグ、リプレイポリシー。"
site_en: "<code>network</code> — transport timeouts, replays &amp; the 429 egress governor: <span class='mech'>upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential</span>."
site_zh: "<code>network</code> —— 传输超时、重放与 429 出站治理:<span class='mech'>upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential</span>。"
site_en: "<code>network</code> — transport timeouts, replays &amp; the 429 egress governor: <span class='mech'>upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled false · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential</span>."
site_zh: "<code>network</code> —— 传输超时、重放与 429 出站治理:<span class='mech'>upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled false · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential</span>。"
- key: network.upstreamTimeoutMs
type: number
default: "720000"
@@ -103,12 +103,12 @@ entries:
desc_ja: "リプレイ間のベースバックオフ遅延(指数関数的)。"
- key: network.throttle429Enabled
type: boolean
default: "true"
default: "false"
env: BILI_THROTTLE_429
group: process
desc_en: "Egress-side 429 governor (#4204): repeated upstream 429s are answered locally instead of forwarded. Account limits and unrecognized/malformed responses hold origin+credential; only KeyRing's exact `keyring_model_temporarily_unavailable` code with a matching request model holds origin+credential+model, leaving other models eligible. A mismatch or missing model remains account-wide; false restores the legacy always-forward behavior."
desc_zh: "出站 429 治理器(#4204):连续上游 429 会在本地应答而不再转发。账号限额及无法识别/格式错误的响应按 origin+凭据持有;只有 KeyRing 精确返回 `keyring_model_temporarily_unavailable` 且响应模型与请求模型完全相同时,才按 origin+凭据+模型持有,让其他模型继续可用。模型不匹配或缺失仍按账号级处理;设为 false 恢复旧的总是转发行为。"
desc_ja: "エグレス側の 429 ガバナナ(#4204):繰り返される上流 429 は転送されるのではなくローカルで応答される。アカウント制限および未認識/不正なレスポンスは origin+credential を保持する;KeyRing の正確な keyring_model_temporarily_unavailable コードと一致するリクエストモデルのみが origin+credential+model を保持し、他のモデルを適格とする。モデルの不一致または欠落はアカウント全体に適用される;false は従来の常に転送する動作を復元する。"
desc_en: "Egress-side 429 governor (#4204), OFF by default since #6639/#5853: upstream 429s are forwarded to the client unchanged and bili logs one [429-passthrough] warn per observed 429. Set true (or BILI_THROTTLE_429=1) to opt back in as an escape hatch: repeated upstream 429s are then answered locally instead of forwarded — account limits and unrecognized/malformed responses hold origin+credential; only KeyRing's exact `keyring_model_temporarily_unavailable` code with a matching request model holds origin+credential+model, leaving other models eligible. A mismatch or missing model remains account-wide."
desc_zh: "出站 429 治理器(#4204),自 #6639/#5853 起默认关闭:上游 429 原样转发给客户端,bili 每观测到一次 429 打一条 [429-passthrough] warn 日志。设为 true(或 BILI_THROTTLE_429=1)重新启用,作为逃生舱:连续上游 429 会在本地应答而不再转发。账号限额及无法识别/格式错误的响应按 origin+凭据持有;只有 KeyRing 精确返回 `keyring_model_temporarily_unavailable` 且响应模型与请求模型完全相同时,才按 origin+凭据+模型持有,让其他模型继续可用。模型不匹配或缺失仍按账号级处理。"
desc_ja: "エグレス側の 429 ガバナナ(#4204)、#6639/#5853 以降既定でオフ:上流の 429 は変更なしにクライアントへ転送され、bili は 1 件あたり [429-passthrough] warn ログを記録する。true(または BILI_THROTTLE_429=1)にするとエスケープハッチとして再有効化:繰り返される上流 429 が転送されずローカルで応答される。アカウント制限および未認識/不正なレスポンスは origin+credential を保持;KeyRing の正確な keyring_model_temporarily_unavailable コードと一致するリクエストモデルのみが origin+credential+model を保持し、他のモデルを適格とする。モデルの不一致または欠落はアカウント全体に適用される。"
- key: network.throttle429MaxConsecutive
type: number
default: "5"
+1 -1
View File
@@ -298,7 +298,7 @@ qwen / mcode / aider / copilot / amp / goose (full list: bili --help)</code
<li data-i18n-html="tt.mitm"><code>mitm.enabled</code> / <code>mitm.domains</code> — HTTPS interception switch and whitelist domains for login clients.</li>
<li data-i18n-html="tt.mitm2"><code>mitm.handshakeTimeoutMs</code> — TLS handshake timeout (default 10000).</li>
<li class="tg" data-i18n-html="tg.proc"><strong>Process blocks</strong></li>
<li data-i18n-html="tt.network"><code>network</code> — transport timeouts, replays &amp; the 429 egress governor: <span class='mech'>upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential</span>.</li>
<li data-i18n-html="tt.network"><code>network</code> — transport timeouts, replays &amp; the 429 egress governor: <span class='mech'>upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled false · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential</span>.</li>
<li data-i18n-html="tt.guardrails"><code>guardrails</code> — pi/omp plugin tool-output + stream text-run guardrails: <span class='mech'>bashTimeout 60 · maxOutputBytes 50000 · repetition true · textRunThreshold 200</span>.</li>
<li data-i18n-html="tt.persist"><code>persist</code> — session persistence to disk: <span class='mech'>enabled true · zstd false · debounceMs 500 · maxDelayMs 5000 · tailTokens 16384 · epermAlertThreshold 5 · epermAlertRepeatMs 0</span>.</li>
<li data-i18n-html="tt.storage"><code>storage</code> — unified storage sidecar: <span class='mech'>unified legacy</span> — file tier selects the lane-wide storage mode; env BILI_STORAGE_UNIFIED wins when set.</li>
+2 -2
View File
@@ -133,7 +133,7 @@
"tt.ccx": "<code>codexCompact</code> — intercept | pass (default intercept): handle Codex's compaction requests in bili or forward upstream — explicit forms plus the inline auto-compact transiting as an ordinary turn (prompt-form detection, #3008; overridden compact_prompt transits verbatim).",
"ttl.ttl": "<code>ccrRetrievalTtlMs</code> — lifetime of CCR retrieval pointers (default 600000; 0 disables retrieval).",
"ttl.tmp": "<code>decompressTmpCap</code> — size cap for temporary decompressed blocks (default 50).",
"tt.network": "<code>network</code> — transport timeouts, replays &amp; the 429 egress governor: <span class='mech'>upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential</span>.",
"tt.network": "<code>network</code> — transport timeouts, replays &amp; the 429 egress governor: <span class='mech'>upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled false · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential</span>.",
"tt.guardrails": "<code>guardrails</code> — pi/omp plugin tool-output + stream text-run guardrails: <span class='mech'>bashTimeout 60 · maxOutputBytes 50000 · repetition true · textRunThreshold 200</span>.",
"tt.persist": "<code>persist</code> — session persistence to disk: <span class='mech'>enabled true · zstd false · debounceMs 500 · maxDelayMs 5000 · tailTokens 16384 · epermAlertThreshold 5 · epermAlertRepeatMs 0</span>.",
"tt.storage": "<code>storage</code> — unified storage sidecar: <span class='mech'>unified legacy</span> — file tier selects the lane-wide storage mode; env BILI_STORAGE_UNIFIED wins when set.",
@@ -392,7 +392,7 @@
"tt.ccx": "<code>codexCompact</code> —— intercept | pass(默认 intercept):在 bili 内处理 Codex 压缩请求或转给上游 —— 显式形态 + 以普通请求形态过线的内联自动压缩(prompt 文本检测,#3008;自定义 compact_prompt 按原样转发)。",
"ttl.ttl": "<code>ccrRetrievalTtlMs</code> —— CCR 回取指针有效期(默认 600000;0 禁用回取)。",
"ttl.tmp": "<code>decompressTmpCap</code> —— 临时展开块的大小上限(默认 50)。",
"tt.network": "<code>network</code> —— 传输超时、重放与 429 出站治理:<span class='mech'>upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential</span>。",
"tt.network": "<code>network</code> —— 传输超时、重放与 429 出站治理:<span class='mech'>upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled false · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential</span>。",
"tt.guardrails": "<code>guardrails</code> —— pi/omp 插件工具输出与流式 text-run 守卫:<span class='mech'>bashTimeout 60 · maxOutputBytes 50000 · repetition true · textRunThreshold 200</span>。",
"tt.persist": "<code>persist</code> —— 会话落盘持久化:<span class='mech'>enabled true · zstd false · debounceMs 500 · maxDelayMs 5000 · tailTokens 16384 · epermAlertThreshold 5 · epermAlertRepeatMs 0</span>。",
"tt.storage": "<code>storage</code> —— 统一存储边车:<span class='mech'>unified legacy</span> —— 文件层选定全 lane 存储模式;设置了 BILI_STORAGE_UNIFIED 时环境变量优先。",