Files
billion-context/reference/testing.md
T
ework-agent b8cad881be ci: shard the test suite across the matrix to cut wall time ~3x (#3143)
The suite outgrew one runner: ~656 top-level files (~190k LOC) serialized
through --test-concurrency=4 on a single 4-core leg took ~17-20 min (windows
slowest; measured 1225s on the last full green run). Add a `shard` matrix
dimension to the test job so each leg runs a balanced slice; run-ci-tests.mjs
now partitions deterministically + totally (LPT bin-packing over file size)
so the union of all shards == allFiles and no test is skipped. Same tests,
assertions, env, OS and node versions — zero coverage loss. Wall drops ~3x
(~20 -> ~7 min).
2026-10-14 14:57:38 +08:00

203 lines
10 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Testing Reference
> **Not auto-loaded.** On-demand mechanics pulled out of `AGENTS.md` §3 to keep
> the auto-loaded spec lean. The operative rules (run e2e before pipeline
> changes, never remove the gates) stay in [`AGENTS.md` §3](../AGENTS.md#3-development-standards);
> this file holds the exact commands, env vars, phases, and CI trigger list.
## Build Commands
```bash
npm run build # tsup bundle (inlines acp-kernel)
npm run typecheck # tsc --noEmit --project tsconfig.json (src + tests)
npm test # node --import tsx --test tests/*.test.ts
```
## CI: test schedule (#2816)
The main CI `test` job does not run the flat `npm test` glob — it runs
`node scripts/run-ci-tests.mjs`, which runs the same top-level suites with an
explicit `--test-concurrency=4`. Baseline logs prove the old runner default
(`availableParallelism()-1`) was already concurrent — the sum of per-test
durations (~630s ubuntu / ~1110s windows) fits inside the step wall
(373s / 688s) only if several files ran at once; pinning c=4 makes that
explicit with one slot per core. A two-phase prototype (serial "live" phase +
concurrent pool) was measured first and rejected: the serial phase idled the
other cores for 3-4 min without buying stability, because live/process suites
have run mixed with the rest on these dedicated runners for months (#1248's
flakes were loaded local sandboxes — gated by `ACP_TEST_CLAUDE_NATIVE`, not CI).
The script self-checks on every run: a new top-level `tests/*.test.ts` that
matches a process-spawn pattern but is NOT registered in its `LIVE_FILES` list
fails the job. When you add a suite that spawns processes, register it in
`scripts/run-ci-tests.mjs` and review its concurrency assumptions (#1248:
per-file XDG homes, `listen(0)` ports, no machine-global state) before it
rides c=4 unreviewed.
Scheduling only — same tests, same assertions, same env vars; local
`npm test` is untouched.
Since #3143 the `test` job also SHARDS across the matrix: each leg runs one of
three balanced slices (`CI_TEST_SHARD="i/3"`), partitioned by
`scripts/run-ci-tests.mjs` with deterministic LPT bin-packing over file size —
total (union of shards == every file) and disjoint (no file in two shards), so no
test is ever skipped and the LIVE_FILES tripwire still scans the full set in every
shard. The suite had grown past one runner's ~c=4 ceiling (~656 files / ~190k LOC →
~20 min wall on windows), so sharding cut the wall ~3x (~7 min) at zero coverage
cost. Tune by editing the `shard:` list in `.github/workflows/ci.yml` AND its `/3`
denominator together (they must stay equal).
The `static-gates` job (same os×node matrix) runs typecheck + kernel suite +
pi-subagents suite + build IN PARALLEL with the `test` job; they used to be
sequential steps ahead of the tests (~90s of critical path per leg).
## Local Testing & Install
Test against the REAL published artifact:
```bash
npm run build
npm install -g billion-context@latest # install from registry
bili start --port 8787
```
`npm install -g . --install-links` also works (installs a real copy of the local
build). The `--install-links` flag is REQUIRED on npm ≥ 9: without it, local-directory
global installs default to a SYMLINK (`install-links=false`), which leaks dev-tree
edits into the "installed" tool and silently re-links on reinstall (#1225). Verify with
`readlink $(npm root -g)/billion-context` — empty output means a real copy. The
registry install remains preferred for testing the real published artifact.
## E2E Regression (real client through bili)
`tests/e2e/e2e-codex.test.ts` drives the **real `codex` CLI** through a built proxy
against a real Responses-compatible upstream and asserts the full context lifecycle
end-to-end: warmup → load growth → ACP compress → purity → native-compact
interception (last phase gated by `E2E_FORGE=1`). Full phase details, env vars, and
mechanics: `tests/e2e/README.md`.
```bash
# zero-token preflight (codex binary + dist + upstream reachable)
E2E_CHECK=1 node --import tsx --test tests/e2e/e2e-codex.test.ts
# full run (defaults to local sglang at http://127.0.0.1:8199/v1, zero cost)
npm run build
ACP_TEST_E2E=1 node --import tsx --test tests/e2e/e2e-codex.test.ts
# + native-compact interception phase
ACP_TEST_E2E=1 E2E_FORGE=1 node --import tsx --test tests/e2e/e2e-codex.test.ts
```
Rules:
- The suite **skips by default** so `npm test` stays free; never remove the
`ACP_TEST_E2E` gate.
- Run it (at least the 4-phase core) before merging changes to the request
pipeline — `server.ts`, `src/loop/`, adapters, preflight/compact paths. It is
the only coverage that exercises real client behavior (codex UA, wire quirks,
retry loops).
- Any Responses-compatible upstream works via `E2E_UPSTREAM_URL` /
`E2E_UPSTREAM_KEY`; the provider is configured as `name = "OpenAI"` so codex
stays on the remote compaction (V2) path — do not "fix" this.
- CI (`.github/workflows/ci-e2e.yml`) auto-runs on PRs whose diff touches the
request-pipeline hot files (`src/server.ts`, `src/server/**`, `src/loop/**`,
`src/agent/**`, adapters/stream/persist, `tests/e2e/**`, `package-lock.json`
— an acp-kernel pin bump IS a pipeline change) plus manual dispatch for anything
else. It needs repo secrets `E2E_UPSTREAM_URL` / `E2E_UPSTREAM_KEY`; a hosted
runner cannot reach `127.0.0.1` upstreams, and events without secret access
(fork PRs) skip the run gracefully instead of failing.
## CI: registry installs must be retry-protected (#2611)
A transient runner↔registry blip (`ECONNRESET` / `network aborted`) during a
setup-phase `npm i -g <cli>` or `npm ci` used to red an otherwise-green PR with
no test code ever running (#2541 run 38032380794; rerun-failed-jobs turned it
green on the same head). Root cause: npm's built-in fetch retry defaults to only
2 tries. Every job that touches the npm registry therefore carries this
job-level `env:` block — it raises the retry budget so a ~10s jitter rides out
instead of aborting setup, and because it is job-level it covers EVERY registry
op in the job (the global CLI install AND `npm ci`, plus any build-time fetch):
```yaml
env:
npm_config_fetch_retries: '5'
npm_config_fetch_retry_mintimeout: '5000'
npm_config_fetch_retry_maxtimeout: '60000'
```
Bounded by design: worst case ~+2 min of backoff before a genuine sustained
outage still fails fast (each job also carries its own `timeout-minutes`). Copy
the block verbatim into any NEW e2e/setup lane that installs from the registry so
the pattern never re-scatters unpinned. Deliberately NOT applied around
`npm publish` (the artifact/release lanes): a publish is non-idempotent — if the
first upload commits server-side but its response resets, a fetch-retry re-POSTs
and can trip E409 / double-publish, which is worse than the original failure mode.
## E2E: Real-Image Billing Lane (`npm run test:e2e:image`)
`tests/e2e/e2e-image-billing.test.ts` is the hermetic twin of the manual
verification that cleared PR #1857 (#1843/#1848): a corpus of structurally
real images (valid PNG/JPEG/WebP/GIF containers, real dimensions, a 4.7MB
4032x3024 photo, EXIF-orientation, truncated and pure-garbage blobs) is
synthesized deterministically in pure Node — no PIL/ImageMagick, no network,
zero tokens — and driven through the real billing pipeline against a mock
upstream that bills the Qwen2-VL formula. It pins the three failure classes
#1857 fixed: header-parser truth for every container, the >=700x bytes-prior
poison ratio, and the learned-cost rescue (arbitrated 12.6MB forward ->
usage report teaches the per-image cost -> stale-high usage baseline no
longer bricks the session -> baseline regresses to real billing) plus the
default auto->pixels route riding a malformed blob. CI:
`.github/workflows/ci-image.yml` auto-runs on PRs touching the image/
window-gate surfaces; it always runs ungated (needs nothing external), ~1s.
## E2E: Hermetic Local Registry (`ACP_TEST_REGISTRY`)
`tests/e2e/e2e-registry.test.ts` brings its own verdaccio instance (random loopback
port read back from `listen(0)`, isolated storage/config/home) and exercises the REAL
self-update chain end-to-end — dist-tag resolve → tarball download → sha512 verify →
staged extract → in-place install → disk flip — plus post-update
`plugin install opencode`. Loopback only; zero external network, zero secrets,
zero tokens.
`tests/e2e/e2e-advisory-rollback.test.ts` (same gate, same fixture infra)
drives a resident `bili start` against a rollback-form advisory document and
asserts the #1588 contract end-to-end: control self-update, forced rollback
to the older target + persistent restart banner, no ping-pong across cycles
(pre-restart), and the candidate gate refusing the affected latest
(post-restart).
```bash
npm run build
ACP_TEST_REGISTRY=1 node --import tsx --test \
tests/e2e/e2e-registry.test.ts tests/e2e/e2e-advisory-rollback.test.ts
```
Rules:
- Gated by `ACP_TEST_REGISTRY=1`; skips by default, and the `npm test` glob does
not cover `tests/e2e/` anyway. CI job: `.github/workflows/ci-registry.yml`.
- Run it before merging changes to `src/update.ts`, `src/advisory.ts`, or the
install/uninstall pipeline (`src/plugin-install.ts`).
- The updater's registry base URL and check interval are overridable via
`BILI_UPDATE_REGISTRY` / `BILI_UPDATE_CHECK_INTERVAL_MS` (defaults unchanged when
unset) — these seams exist for this suite (#1153); keep them default-invariant.
- The fixture MUST bring its own registry instance — never point it at an external
(even internal) registry service.
Three additional phases harden the #1628 failure class:
- **Unwritable install dir** (posix only; skipped on windows) — chmods the install
root `0o555` and drives `bili update` through the real chain: assert the loud
failure line, the one-per-process `install location` diagnostic, zero disk flip,
and no leaked update lock.
- **Real published baseline** (`BILI_E2E_REAL_BASELINE=<published version>`) — the
only phase that reads upstream: downloads ONE real published tarball, then proves
the LIVE old updater hops onto the locally built dist. This is the local form of
the no-op validation protocol (`reference/release.md`); it cannot replace a real
no-op release, but it catches "current dist can't be installed by an old updater"
per PR. CI resolves the baseline to `npm view billion-context version` before the
run; locally pass it explicitly.
- **Cache-lane doctor** — lays out an opencode cache lane
(`npm/billion-context@latest/<ts>/node_modules/billion-context`) and asserts
`doctor --json` resolves the update target to the running copy, not a global root.