Files
billion-context/reference/testing.md
T
ework-agent b8cad881be ci: shard the test suite across the matrix to cut wall time ~3x (#3143)
The suite outgrew one runner: ~656 top-level files (~190k LOC) serialized
through --test-concurrency=4 on a single 4-core leg took ~17-20 min (windows
slowest; measured 1225s on the last full green run). Add a `shard` matrix
dimension to the test job so each leg runs a balanced slice; run-ci-tests.mjs
now partitions deterministically + totally (LPT bin-packing over file size)
so the union of all shards == allFiles and no test is skipped. Same tests,
assertions, env, OS and node versions — zero coverage loss. Wall drops ~3x
(~20 -> ~7 min).
2026-10-14 14:57:38 +08:00

10 KiB
Raw Blame History

Testing Reference

Not auto-loaded. On-demand mechanics pulled out of AGENTS.md §3 to keep the auto-loaded spec lean. The operative rules (run e2e before pipeline changes, never remove the gates) stay in AGENTS.md §3; this file holds the exact commands, env vars, phases, and CI trigger list.

Build Commands

npm run build          # tsup bundle (inlines acp-kernel)
npm run typecheck      # tsc --noEmit --project tsconfig.json (src + tests)
npm test               # node --import tsx --test tests/*.test.ts

CI: test schedule (#2816)

The main CI test job does not run the flat npm test glob — it runs node scripts/run-ci-tests.mjs, which runs the same top-level suites with an explicit --test-concurrency=4. Baseline logs prove the old runner default (availableParallelism()-1) was already concurrent — the sum of per-test durations (~630s ubuntu / ~1110s windows) fits inside the step wall (373s / 688s) only if several files ran at once; pinning c=4 makes that explicit with one slot per core. A two-phase prototype (serial "live" phase + concurrent pool) was measured first and rejected: the serial phase idled the other cores for 3-4 min without buying stability, because live/process suites have run mixed with the rest on these dedicated runners for months (#1248's flakes were loaded local sandboxes — gated by ACP_TEST_CLAUDE_NATIVE, not CI).

The script self-checks on every run: a new top-level tests/*.test.ts that matches a process-spawn pattern but is NOT registered in its LIVE_FILES list fails the job. When you add a suite that spawns processes, register it in scripts/run-ci-tests.mjs and review its concurrency assumptions (#1248: per-file XDG homes, listen(0) ports, no machine-global state) before it rides c=4 unreviewed.

Scheduling only — same tests, same assertions, same env vars; local npm test is untouched.

Since #3143 the test job also SHARDS across the matrix: each leg runs one of three balanced slices (CI_TEST_SHARD="i/3"), partitioned by scripts/run-ci-tests.mjs with deterministic LPT bin-packing over file size — total (union of shards == every file) and disjoint (no file in two shards), so no test is ever skipped and the LIVE_FILES tripwire still scans the full set in every shard. The suite had grown past one runner's ~c=4 ceiling (~656 files / ~190k LOC → ~20 min wall on windows), so sharding cut the wall ~3x (~7 min) at zero coverage cost. Tune by editing the shard: list in .github/workflows/ci.yml AND its /3 denominator together (they must stay equal).

The static-gates job (same os×node matrix) runs typecheck + kernel suite + pi-subagents suite + build IN PARALLEL with the test job; they used to be sequential steps ahead of the tests (~90s of critical path per leg).

Local Testing & Install

Test against the REAL published artifact:

npm run build
npm install -g billion-context@latest   # install from registry
bili start --port 8787

npm install -g . --install-links also works (installs a real copy of the local build). The --install-links flag is REQUIRED on npm ≥ 9: without it, local-directory global installs default to a SYMLINK (install-links=false), which leaks dev-tree edits into the "installed" tool and silently re-links on reinstall (#1225). Verify with readlink $(npm root -g)/billion-context — empty output means a real copy. The registry install remains preferred for testing the real published artifact.

E2E Regression (real client through bili)

tests/e2e/e2e-codex.test.ts drives the real codex CLI through a built proxy against a real Responses-compatible upstream and asserts the full context lifecycle end-to-end: warmup → load growth → ACP compress → purity → native-compact interception (last phase gated by E2E_FORGE=1). Full phase details, env vars, and mechanics: tests/e2e/README.md.

# zero-token preflight (codex binary + dist + upstream reachable)
E2E_CHECK=1 node --import tsx --test tests/e2e/e2e-codex.test.ts

# full run (defaults to local sglang at http://127.0.0.1:8199/v1, zero cost)
npm run build
ACP_TEST_E2E=1 node --import tsx --test tests/e2e/e2e-codex.test.ts

# + native-compact interception phase
ACP_TEST_E2E=1 E2E_FORGE=1 node --import tsx --test tests/e2e/e2e-codex.test.ts

Rules:

  • The suite skips by default so npm test stays free; never remove the ACP_TEST_E2E gate.
  • Run it (at least the 4-phase core) before merging changes to the request pipeline — server.ts, src/loop/, adapters, preflight/compact paths. It is the only coverage that exercises real client behavior (codex UA, wire quirks, retry loops).
  • Any Responses-compatible upstream works via E2E_UPSTREAM_URL / E2E_UPSTREAM_KEY; the provider is configured as name = "OpenAI" so codex stays on the remote compaction (V2) path — do not "fix" this.
  • CI (.github/workflows/ci-e2e.yml) auto-runs on PRs whose diff touches the request-pipeline hot files (src/server.ts, src/server/**, src/loop/**, src/agent/**, adapters/stream/persist, tests/e2e/**, package-lock.json — an acp-kernel pin bump IS a pipeline change) plus manual dispatch for anything else. It needs repo secrets E2E_UPSTREAM_URL / E2E_UPSTREAM_KEY; a hosted runner cannot reach 127.0.0.1 upstreams, and events without secret access (fork PRs) skip the run gracefully instead of failing.

CI: registry installs must be retry-protected (#2611)

A transient runner↔registry blip (ECONNRESET / network aborted) during a setup-phase npm i -g <cli> or npm ci used to red an otherwise-green PR with no test code ever running (#2541 run 38032380794; rerun-failed-jobs turned it green on the same head). Root cause: npm's built-in fetch retry defaults to only 2 tries. Every job that touches the npm registry therefore carries this job-level env: block — it raises the retry budget so a ~10s jitter rides out instead of aborting setup, and because it is job-level it covers EVERY registry op in the job (the global CLI install AND npm ci, plus any build-time fetch):

    env:
      npm_config_fetch_retries: '5'
      npm_config_fetch_retry_mintimeout: '5000'
      npm_config_fetch_retry_maxtimeout: '60000'

Bounded by design: worst case ~+2 min of backoff before a genuine sustained outage still fails fast (each job also carries its own timeout-minutes). Copy the block verbatim into any NEW e2e/setup lane that installs from the registry so the pattern never re-scatters unpinned. Deliberately NOT applied around npm publish (the artifact/release lanes): a publish is non-idempotent — if the first upload commits server-side but its response resets, a fetch-retry re-POSTs and can trip E409 / double-publish, which is worse than the original failure mode.

E2E: Real-Image Billing Lane (npm run test:e2e:image)

tests/e2e/e2e-image-billing.test.ts is the hermetic twin of the manual verification that cleared PR #1857 (#1843/#1848): a corpus of structurally real images (valid PNG/JPEG/WebP/GIF containers, real dimensions, a 4.7MB 4032x3024 photo, EXIF-orientation, truncated and pure-garbage blobs) is synthesized deterministically in pure Node — no PIL/ImageMagick, no network, zero tokens — and driven through the real billing pipeline against a mock upstream that bills the Qwen2-VL formula. It pins the three failure classes #1857 fixed: header-parser truth for every container, the >=700x bytes-prior poison ratio, and the learned-cost rescue (arbitrated 12.6MB forward -> usage report teaches the per-image cost -> stale-high usage baseline no longer bricks the session -> baseline regresses to real billing) plus the default auto->pixels route riding a malformed blob. CI: .github/workflows/ci-image.yml auto-runs on PRs touching the image/ window-gate surfaces; it always runs ungated (needs nothing external), ~1s.

E2E: Hermetic Local Registry (ACP_TEST_REGISTRY)

tests/e2e/e2e-registry.test.ts brings its own verdaccio instance (random loopback port read back from listen(0), isolated storage/config/home) and exercises the REAL self-update chain end-to-end — dist-tag resolve → tarball download → sha512 verify → staged extract → in-place install → disk flip — plus post-update plugin install opencode. Loopback only; zero external network, zero secrets, zero tokens.

tests/e2e/e2e-advisory-rollback.test.ts (same gate, same fixture infra) drives a resident bili start against a rollback-form advisory document and asserts the #1588 contract end-to-end: control self-update, forced rollback to the older target + persistent restart banner, no ping-pong across cycles (pre-restart), and the candidate gate refusing the affected latest (post-restart).

npm run build
ACP_TEST_REGISTRY=1 node --import tsx --test \
  tests/e2e/e2e-registry.test.ts tests/e2e/e2e-advisory-rollback.test.ts

Rules:

  • Gated by ACP_TEST_REGISTRY=1; skips by default, and the npm test glob does not cover tests/e2e/ anyway. CI job: .github/workflows/ci-registry.yml.
  • Run it before merging changes to src/update.ts, src/advisory.ts, or the install/uninstall pipeline (src/plugin-install.ts).
  • The updater's registry base URL and check interval are overridable via BILI_UPDATE_REGISTRY / BILI_UPDATE_CHECK_INTERVAL_MS (defaults unchanged when unset) — these seams exist for this suite (#1153); keep them default-invariant.
  • The fixture MUST bring its own registry instance — never point it at an external (even internal) registry service.

Three additional phases harden the #1628 failure class:

  • Unwritable install dir (posix only; skipped on windows) — chmods the install root 0o555 and drives bili update through the real chain: assert the loud failure line, the one-per-process install location diagnostic, zero disk flip, and no leaked update lock.
  • Real published baseline (BILI_E2E_REAL_BASELINE=<published version>) — the only phase that reads upstream: downloads ONE real published tarball, then proves the LIVE old updater hops onto the locally built dist. This is the local form of the no-op validation protocol (reference/release.md); it cannot replace a real no-op release, but it catches "current dist can't be installed by an old updater" per PR. CI resolves the baseline to npm view billion-context version before the run; locally pass it explicitly.
  • Cache-lane doctor — lays out an opencode cache lane (npm/billion-context@latest/<ts>/node_modules/billion-context) and asserts doctor --json resolves the update target to the running copy, not a global root.