Growth investigations were guessing, because the diagnostics snapshot only
carried process totals: RSS and committed bytes say memory grew, never
where. The map adds allocator-live bytes and a size-class histogram to each
snapshot, so a distinctive class identifies the allocation without
per-callsite instrumentation.
The design constraint is honesty about coverage. Walking the allocator
reaches the aggregate heap plus this thread's heap and its abandoned pages,
which is not the whole process, and in a build where malloc does not route
through the allocator at all it reaches nothing. A per-subsystem tally
would report zeros there and read as a clean bill of health. So every
snapshot carries three independent totals and an explicit residual --
os_committed, allocator-live, residual = the difference -- and the triple
localises growth instead of implying it:
live grows leak on a walked heap; buckets say which size
residual grows, flat another thread, retention, or non-allocator memory
both flat, RSS grows mappings or stacks, not the heap
That guard paid for itself immediately. Pointed at the open Windows growth
in #581 it reported live=0 with zero blocks and the residual carrying all
of it -- not 'the heap is innocent' but 'this walk saw nothing', which is
the correct reading for a process demonstrably holding tens of megabytes.
Together with the growth surviving a build with the allocator override
compiled out, that says the Windows allocations under investigation never
reach the allocator, so an allocator-based map cannot localise them and the
next instrument has to sit lower down.
Covered by a test that allocates a known volume in a known size class and
requires the map to attribute it there, so the map can never regress into
silently reporting zeros.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The macOS Intel leg failed the fixture contract on every run while every
other platform passed. Removing the readiness fsync did not change it; the
hardened diagnostics named the real cause on the first remote run:
waited=30.0s, exit_status=alive, port_file=absent, staged_temp_files=none,
empty startup log -- the process was healthy but had never reached
publish_port.
http.server.HTTPServer.server_bind() resolves socket.getfqdn(host). On a
host whose resolver does not answer for the bind address that call blocks
for the resolver timeout, so the constructor never returns and no port is
ever published. The fixture now binds through a subclass that keeps the
threading server but skips the FQDN resolution, which only feeds CGI-style
variables this fixture never serves.
Proven both directions locally by forcing socket.getfqdn to hang: the
subclass publishes its port immediately, the stock server never does. That
forcing hook is kept as a permanent contract guard, so the reverse-DNS
dependency cannot return without turning the gate red on every platform
rather than on one runner nobody can reproduce.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The native Windows soak could never start its server, so that gate had
never actually measured anything. The root cause was the harness, not the
daemon: an MSYS filesystem FIFO feeding a native Windows process delivers
empty stdin (reproduced 5/5), while a Bash anonymous coprocess carries the
full JSON-RPC initialize exchange to the same protected payload, and the
failing daemon log showed an orderly client disconnect rather than a
daemon-side eviction. The soak now uses a named coprocess on native
Windows only, duplicating its endpoints to stable fd3/fd4 and closing the
originals so closing fd3 still delivers EOF; POSIX keeps the FIFO path.
The coproc syntax sits inside eval because macOS system Bash 3.2 must
still parse this file even though only MSYS2 Bash 5 executes that branch.
With transport repaired the next exact failure surfaced: native Windows
Python cannot open an MSYS /c/... diagnostics path. Both consumers now
pipe the file through stdin, the pattern already established elsewhere in
the repo, and the soak recovery contract forbids reintroducing a direct
native-Python open.
Also on the release path:
- The smoke fixture server no longer fsyncs before atomically publishing
its port. This is ephemeral readiness signalling, not crash-durable
state, and the macOS Intel runner failed inside that durability sync.
When the contract does fail it now reports the observable state --
waited, exit status, port file, staged temp files, interpreter, startup
log -- because the previous verdict named nothing on the one runner we
cannot reproduce locally.
- The POSIX publication test hook no longer compiles into Windows builds,
fixing an exact -Werror unused-variable failure. The setter keeps a
parameter-consuming Windows stub because it is public API.
- The Wine leg assembles the real release layout (payload plus canonical
launcher) and version-checks both, running the launcher through cmd so
it has a Windows-visible parent. The unsupported Wine soak is removed:
Wine stays a fast compile/package/version check, and native Windows is
authoritative for daemon, locking, ACL and process-lifetime semantics.
- run.sh soak-windows routes to the native Windows VM, which validates the
payload, builds the protected per-user temp root, stamps ACLs, and
refuses success without a completion summary.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The parity harness drives the pipeline passes with a bare ctx and never
ran the production teardown, so the return-type table pass_calls builds
leaked — latent until the fixture gained typed Java methods, and visible
only under Linux LeakSanitizer. Both runners now free it exactly the way
pipeline.c does.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The parallel per-file resolve hardcoded INHERITS for every base-class
relation while the sequential semantic pass made the Interface split, so
any corpus large enough for the parallel path had its explicit implements
demoted: elasticsearch carried 11,179 INHERITS edges pointing at Interface
nodes and 3 IMPLEMENTS. Both venues now share one decision keyed off the
target node's label — the graph truth the edge attaches to.
OVERRIDE edges existed only for Go's implicit interface satisfaction. A
serial full-graph tail run by both pipelines now matches class methods
against explicit bases by name and emits Method-to-Method OVERRIDE edges
for the explicit languages, with Go excluded to avoid double cover.
The venue parity fixture gains a Java interface/implements/extends trio —
IMPLEMENTS parity used to compare 0 == 0 and guarded nothing — plus an
exact-count test so the parity can never go vacuous again. At scale:
elasticsearch IMPLEMENTS 3 to 11,182 and OVERRIDE 0 to 49,695, TypeScript
0 to 4,496 and 594, dotnet/runtime 0 to 12,953 and 45,667, with INHERITS
plus IMPLEMENTS sums conserved exactly.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A directory with more immediate subdirectories than the fixed 512-frame
walk stack made the whole discovery walk hard-fail with files=0 and a
misleading 'check repo_path' hint — dotnet/runtime carries 855 sibling
dirs in one JIT regression folder and could not be indexed at all. The
stack now grows geometrically from the same initial capacity; the parent
frame is a popped copy, so growth cannot dangle it. Covered by a 600-dir
fanout reproduction.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The build system now stamps CBM_SANITIZED_BUILD for any non-empty SANITIZE,
so timing budgets key off build-system truth instead of compiler-specific
__SANITIZE_ADDRESS__ probes that miss trap-UBSan and TSan; the two bench
suites use it.
The httpd interrupt test proved 'the interrupt worked' with a wall-clock
bound, which is a lottery under sanitizers. A send-deadline test hook pins
the deadline out of reach so the join itself is the causality proof, and
the stop watchdog becomes a true ten-second hang detector.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two changes to the index coordinator, developed together and validated as
one state:
Determinism seams. A cancel racing a worker start was only observable on a
loaded runner; now the interleaving is forced by construction: a test hold
parks every job thread before its first pre-start cancel check, and a
background-initialize completion counter turns sleep-then-assert negatives
into positive-signal waits. The cancel-before-worker-start case is pinned
by a dedicated test; three auto-index admission tests now wait for the
admission tail instead of sleeping through it.
Queued admission. An explicit index_repository request that hits the
physical job limit used to surface a raw busy error. The request thread
already blocks for the whole index, so it now queues: a cancellable,
shutdown-aware wait-and-resubscribe loop replaces the error path, with a
wait counter as the parked-state observable. The programmatic coordinator
API keeps its non-blocking busy answer for callers with their own retry
policy (watcher, auto-index). The job-limit test asserts the new contract:
parked without a second worker start, admitted and completed once the
occupying job finishes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The suite's observation waits shared a 2000 ms budget that is a hang
detector, not a latency assertion — every waiter returns the moment its
condition holds, so green runs never pay it. Under ThreadSanitizer's
5-20x slowdown on a loaded 3-core macOS CI runner the cancel-delivery
wait lost the tail of that distribution once in seven otherwise-green
TSan rounds (the cancel itself had succeeded; only the fake worker's
counter observation timed out). The budget is now 10000 ms.
Verified: nine consecutive daemon_application runs under native macOS
TSan with the widened budget, all green, zero ThreadSanitizer
warnings.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Local runs left most of the machine idle: the serial tail ran sixteen
suites one at a time on otherwise-idle cores, the three platform legs
were launched by hand (usually sequentially), and every container-leg
object write crossed the virtiofs bind mount.
- run-tests-parallel.sh tail scheduling in two phases: the FLEX suites
(timing-shaped but free of the shared per-account daemon runtime
namespace) run CBM_TAIL_JOBS-wide (default 2), then the EXCL group —
daemon-family plus the suites that drive daemon one-shots or
supervisor rendezvous — runs strictly sequentially on a machine
exactly as quiet as the old fully-serial tail gave it. The wave was
already fed longest-first by the shard dealing order, so the drain-out
no longer ends on a heavy straggler.
- ladder.sh: one maintained entry point for the full local push gate
with the legs overlapped — lint, the Linux container suite, and the
Windows VM suite in the background, the macOS suite in the
foreground, one verdict per leg, logs kept per leg. A missing
prerequisite fails its leg loudly instead of silently skipping.
- win.sh test-par now runs through vm-run-tests.sh (--par mode): the
full parallel harness under the CI-shaped protected temp root with
complete output. It previously ran under the MSYS-shared /tmp and
piped through `tail -25` — the same truncated-blindness class that
hid 40 Windows failures from the `test` command.
- docker-compose: build artifacts and the incremental fixture cache
move to named volumes on the container VM's native filesystem. Object
writes over the virtiofs bind mount are the container legs' largest
avoidable I/O cost, and the fixture cache now survives across
container runs.
- test_mem(win): the first valid full-parallel VM run proved working-set
trimming beats the re-touch mitigation (19 MB resident of a 256 MB
double-touch at 18 parallel suites). The RSS probe now VirtualLocks a
64 MB span — locked pages are exempt from trimming, making the
measurement pressure-immune — with bounded touch-and-sample retries
when the lock is unavailable. Red-to-green under the same 18-job load.
- cli: the portable install's staging error now appends the activation
refusal note (predicate, SID, object) like the managed path already
does — a bare "activation transaction I/O failed" on a CI-only
failure is undiagnosable without it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The incremental suite cloned fastapi from the network on every run and
indexed a DIFFERENT corpus per environment: CI used a sparse checkout
without docs/ and tests/ (the assertion thresholds are sized for it)
while local runs indexed the full tree — twice the files for the same
assertions, a per-run network dependency, and the suite's dominant cost
on the sanitized container leg.
The fixture is now cloned from the network at most once per machine
into a persistent cache (committed with an atomic rename so a torn
download can never pose as a valid cache; CBM_TEST_FIXTURE_CACHE
overrides the location), every run local-clones from it offline, and
docs/ plus tests/ are trimmed everywhere — the portable equivalent of
CI's historical sparse profile, so all environments now index the same
corpus the assertions were written against.
Same 163 tests, same assertions: 74s -> 31s native, with the container
leg's cut scaling with the halved indexing volume, and the "network
offline" flake source reduced to first-run-per-machine.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The first full Windows x64 CI run of this branch (earlier contract-step
walls had always aborted the leg) surfaced 40 failures: the managed
launcher's fail-closed install/update/uninstall gates and the suite's
POSIX-modeled activation tests were both new in this branch and had
never been reconciled, and they hid three real production bugs.
Production fixes:
- ui: config publishes with POSIX rename semantics. MoveFileExW's legacy
replace cannot supersede a destination a reader holds open (the name
lingers until the last handle closes, even with FILE_SHARE_DELETE), so
a config save racing an open reader failed. The temp file is opened
with DELETE access and renamed in-hand (bare-path target, NUL-padded
rename record) before close, falling back to MoveFileExW on any
rename-by-handle failure so non-NTFS config homes keep the previous
behavior.
- subprocess(win): the child's redirected log handle now shares write
access. POSIX places no lock on the redirected log, so a second
producer may append while the child runs; the missing FILE_SHARE_WRITE
mandatory-locked every other writer out, which the index-supervisor
callback-isolation test legitimately exercises.
- cli(win): the portable uninstall path composes the managed binary name
with its .exe suffix; it could never find (or remove) the installed
binary before.
- activation: security-predicate refusals are now diagnosable. The
transaction's predicates refuse without a usable OS last-error, which
collapsed five distinct refusal classes into one blind "status -3,
os 0" at every caller. Each refusing predicate records what refused —
predicate, untrusted SID, and object path — first-wins so the deepest
cause survives, and the staging error line appends it. This is what
identified every Windows refusal in this batch, including the real
install-failure class on drive-root-rooted payload directories
(Authenticated Users holds inherited Modify there; profile-rooted
directories do not).
Windows test dispatch (release binaries unchanged):
- Test builds (CBM_CLI_ENABLE_TEST_API, the existing test-object macro)
compile the portable activation flow on Windows, reachable solely
through the installed activation test seam and announced loudly on
stderr; release binaries compile exclusively the managed transaction,
so the dispatch polarity fails closed. The suite's install/update/
uninstall calls route through a seam-if-absent wrapper, keeping the
shared activation and agent-config semantics unit-covered on Windows;
POSIX arms keep exercising the real activation machinery.
- Two new release-gate tests assert the fail-closed contract with the
seam OFF: a portable payload may not self-update/self-uninstall, and
install refuses to adopt or replace a foreign file at the canonical
launcher path even under --force.
- Test-bug fixes the first valid Windows runs exposed: the version-
cohort crash test reused one conflict record across the probe and the
turnover acquire (whose entry memset erased the probe's detail before
the assertions read it); the activation staged-path assertion now
checks containment rather than a separator (the \\?\ namespace
performs no forward-slash translation); the ui atomic test captures,
cleans up, then asserts so a failure cannot poison later tests; the
PATH-visibility guard is POSIX-only (Windows configures PATH through
the user registry, not a shell rc).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fixes surfaced while bringing the shared-coordination-daemon branch
green across macOS, the Linux containers, and the real Windows-ARM64 VM,
plus a new stability guard that caught one of them.
- daemon(win): the CLI teardown transition latched a failed release as
permanent, but a Windows participant-state release must briefly
try-hold the shared startup/legacy gates and legitimately collides
with a concurrent one-shot's teardown; the release is retriable by
contract (it always retains the transition), so parallel one-shot
commands no longer report "CLI coordination cleanup failed" despite
succeeding. Found by the new stability guard's churn section.
- cli/main(win): install/update/uninstall and the Augment hook-script
removal derive the managed launcher's identity from its plain
drive-form path (cli_windows_plain_utf8), not the \\?\ extended-length
form, so agent-config ownership matches on Windows and uninstall no
longer leaves the MCP entry or owned hook scripts behind.
- security: the `daemon start --open` browser launch is now shell-free
(ShellExecuteW on Windows, cbm_exec_no_shell elsewhere) instead of
system(); the CORS origin check spells out its two literal loopback
URLs so the static URL audit sees a complete value. Both clear the
Layer-1 allow-list audit.
- coordination: version-cohort lock retries now sleep a per-process
jittered interval — fixed-period retries can phase-lock two
participants so one starves. The activation-quiesce test's
observation window now covers worst-case candidate staging (three
tamper-defense hashes over a ~1 GB sanitizer binary exceed the old
30 s budget on container I/O).
- smoke/soak: the agent-config smoke matches Windows config paths
through their escaped-backslash quoted form; the soak parser skips the
update-available banner before the JSON summary and gained a one-shot
CLI admission-churn phase inside the RSS/FD leak window. The build-dir
safety fixture creates a real symlink/junction (MSYS2 ln -s otherwise
deep-copies) so its traversal-refusal contract is genuinely exercised,
and the symlinked-agent-roots test demotes its planted links to an
unprivileged owner when run as root so the refusal is really tested.
- tests(win): a new daemon-stability guard exercises the parameter
surface, hook fail-open with its rate-limited notice, start-twice and
occupied-port handling, busy-stop refusal against a live MCP session,
kill -9 crash recovery, and sequential/parallel one-shot churn.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.
Daemon lifecycle:
- daemon start/stop/status subcommands. `daemon start` launches a
PERMANENT daemon (spawn shape is byte-exact argv; survives idle
periods and session ends) and reports an already-active daemon
instead of failing. Permanence is honored at every stop latch:
last-committed-client disconnect, host initial-client window,
coordinator release, and application final-session close — a
permanent daemon also keeps admitting new sessions after its last
one closes.
- daemon stop refuses while sessions are active and lists the blocking
peers (pid/role) that must finish first; an idle daemon drains
through the activation-shutdown machinery with the ACK ordered after
connection interrupts. A second stop is idempotent. The wire ops are
no-cohort first-frame requests with peer fingerprint authentication,
so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
keep their local supervised path). A cold CLI run that had to spawn a
temporary daemon prints a hint that `daemon start` removes the
per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
emits a visible, rate-limited notice (Claude-dialect systemMessage
plus stderr for other dialects) and always exits 0 — augmentation is
never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
older-build permanent daemon (strict semantic-version triples only;
dev builds never auto-drain) and the build-conflict message names
`cbm daemon stop` as the manual escape hatch.
Windows IPC/runtime (real-VM verified):
- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
path used to destroy its listening pipe instance on every 20 ms poll
timeout; a client attaching in the teardown window was severed or left
on an orphaned pipe object whose HELLO no server handle could ever
read, absorbing the connect until the client's own timeout expired.
The pending connect now survives poll timeouts and nothing is
destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
server handle can discard a just-sent response before the peer reads
it (POSIX stream sockets never lose buffered data on close). A bounded
cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
immediate on interrupted connections) now precedes close in
runtime_worker_finish and runtime_reject_inline, so hello-conflict,
capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
transport half-close, so close_begin now announces departure with an
explicit frame (ordered after APPLICATION_CANCEL, before the local
interrupt); the server releases the client's admission on receipt
instead of waiting for the handle to close. Admission-drop timing is
now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
request publication under the send lock, best-effort sends the active
token's APPLICATION_CANCEL frame, then interrupts local I/O; the
server cancels MCP/subprocess work promptly. Contract tests accept
both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
the requester's license to act on "snapshotted and draining", so every
connection interrupt is now initiated before the ACK is sent; a
session could previously get one more request serviced after the
requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
timer granularity) burned ~1.6 s against permanently obstructed paths,
stalling hello rejections past the client's timeout. The retry budget
is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
nonblocking send() on Windows is absorbed wholesale into AFD kernel
buffering regardless of SO_SNDBUF, so send deadlines and interrupts
could never engage against a slow peer (and the full payload was
pinned in nonpaged pool). Slicing restores a deterministic
backpressure point; a test hook pins SO_SNDBUF for the deadline and
interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
deadline, explicit connection states, refusal to free a server while
a listener-owned connection is active.
Windows long-path support:
- Central path-aware wide conversion (canonicalize via GetFullPathNameW
and prepend the extended-length prefix for absolute paths >=240) at
the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
store opens, and the daemon build-fingerprint/log paths. Deep managed
installs (a 64-hex generation directory routinely exceeds MAX_PATH)
now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
ancestry validators now operate in the extended-length namespace;
the launcher path is canonicalized (and prefixed when deep) once at
entry so every downstream exact-string comparison stays
form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
GetFileAttributesW) so non-ASCII install paths survive argv[0]
resolution.
Windows launcher install/uninstall transaction:
- FileRenameInfoEx names are NUL-terminated in an over-allocated
buffer. FileNameLength governs per the contract, but filter drivers
read FileName as NUL-terminated and appended adjacent heap bytes to
created names — a flaky, garbage-suffixed rename target. Both the CLI
and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
.cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
chars so the bare rename target stays under the FileRenameInfoEx
NT-conversion ceiling at guard depths; 64 bits still uniquely
identify the generation.
- When the running launcher's mapped generation backings pin .cbm
against rename, the backings are relocated to activation-<pid>-N
.retired tombstones beside the install (a mapped image may be renamed,
never deleted; the launcher's liveness-guarded sweep reclaims stale
tombstones). Every relocation is recorded, and a FAILED uninstall
reverses the moves after restoring .cbm — via MoveFileExW with
extended-length paths on both arguments, since the deep generation
target is beyond the handle-based rename's bare-path reach — so a
restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
out so the tree is shallow enough for the detached cleanup's rd, and
the cleanup's working directory strips the extended-length prefix
(CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
(CopyFileW destinations, CREATE_NEW tombstones, probe directories)
are explicitly owner-stamped so the exact-owner validators hold on
runner images; guard fixtures stamp hand-built trees the same way.
Diagnostics, tests and infra:
- diagnostics: discovery is now an always-delivered JSON control record
(new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
paths containing spaces; placement honors $TMPDIR with /tmp fallback
via a diagnostics-local helper; the soak parser reads the JSON record;
documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
log levels, TMPDIR-with-spaces, and native Windows output-contract
assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
parents with production-shaped ancestry (LocalAppData on Windows, via
th_secure_runtime_parent_new) — the runtime ancestry validation
correctly refuses temp roots whose ancestors grant mutation rights to
Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
drive the documented startup-owner publication flow before reading
generation-bound endpoint addresses. This turns the 26 Windows
failures previously visible in CI's full-test job green without
weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
contract including failed-uninstall restore and immediate reinstall
after uninstall; new daemon lifecycle and reworked hook-augment
guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
(it was set by CI but never read, so perf suites ran everywhere);
four throughput/bench suites are classified as perf, the heavy
store_arch suite moved to the slow-timeout tier, and two
wall-clock-sensitive assertions were rewritten as invariant checks
with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
contract, soak daemon-recovery contract, path-safety helper, the
Windows VM worktree-sync contract wired into scripts/test.sh, and
vm/win.sh guards building its clean embedded-UI product in an
isolated BUILD_DIR so it cannot clobber the incremental test build.
provision-windows.sh now installs Node.js for the guards UI build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Windows-VM bootstrap embedded a personal SSH public key and the
drivers connected with StrictHostKeyChecking=no. Key material and host
identity are now strictly operator-supplied and live outside the
repository (~/.claude/cbm-vm/, never committed):
- windows-bootstrap.ps1 takes the operator's public key as a mandatory
-SshPublicKeyPath argument (validated single Ed25519 line) and prints
the VM host-key fingerprint to pin. Generating your own key is
documented in the script and README:
ssh-keygen -t ed25519 -N "" -C <label> -f ~/.claude/cbm-vm/id_ed25519
- win.sh and provision-windows.sh verify the pinned
CBM_VM_HOST_KEY_SHA256 fingerprint before every connection
(StrictHostKeyChecking=yes against a keyscanned known_hosts, Ed25519
only, IdentitiesOnly), validate branch names before remote
interpolation, and fail remote pipelines loudly (set -e -o pipefail).
- The MSYS2 installer download is pinned to a versioned release URL with
SHA-256 verification instead of an unpinned latest.
- win.sh gains a sync subcommand mirroring the uncommitted worktree onto
the VM; its manifest logic is guarded by
tests/test_vm_worktree_manifest.sh, wired into scripts/test.sh.
guards now runs the maintained PowerShell driver against a clean
product build; smoke-install drives vm-smoke.sh.
- run.sh drops the stale duplicate windows-vm/mac-vm legs (the former
carried its own unpinned ssh path) in favor of the vm/ drivers.
The previously committed public key has been rotated off the VM and is
no longer authorized anywhere.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The backpressure/EOF fixtures waited only 10s for the forked daemon to
announce readiness, but the daemon's own startup includes the cohort-claim
path whose internal budget is HOST_DAEMON_CLAIM_TIMEOUT_MS (12s). Under the
4-core oversubscription of CI runners a slow-but-legitimate startup blew
past 10s and was SIGKILLed before it could announce, so daemon_frontend
backpressure failed almost every ubuntu run while passing on fast machines.
Raise the announce deadline to 30s, comfortably above the daemon's own
worst-case budget. Verified: the backpressure failures no longer reproduce
under a local CI-fidelity 4-core run.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every spawned worker/daemon re-hashes the entire executable image for
the exact-build fingerprint; for the ASan test-runner (hundreds of MB)
that is tens of seconds per spawn on constrained CI runners and was the
sole cause of the whole daemon-family readiness-timeout flake class
(index_supervisor, daemon_frontend, cli, and much of the Windows
daemon surface). Add a test-only seam under CBM_CLI_ENABLE_TEST_API:
when CBM_TEST_BUILD_FINGERPRINT holds a valid stub the hash is skipped.
test_main.c sets it once; every forked child and re-exec'd worker
inherits it, so exact-build match AND mismatch behaviour is preserved
(a mismatch test still passes a different fingerprint via argv). Never
compiled into production. Local mac run: index_supervisor 6/6, the four
daemon suites 111/0, exact-build test green. The generous frontend
catastrophic-alarm values are kept as backstops.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The macOS legs went green with the prior calibration but the 3-core
ubuntu-24.04-arm runner is slower still: each spawned worker re-hashes
the entire ASan test-runner image (hundreds of MB) for the exact-build
fingerprint before signalling readiness, which there takes tens of
seconds. Raise the hang-guard deadlines generously - index_supervisor
60s ready / 90s terminal, frontend 30-90s, cli 45s. These are hang
guards, not benchmarks: large values only slow genuine-hang detection,
never mask a regression. (Root-cause fix - a test-only fingerprint
injection seam that skips the hash - is tracked for follow-up.)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Same class as the index-supervisor calibration, proven by the serial
tail: the daemon_frontend backpressure/EOF tests and the cli forced-
install quiesce child failed on the 4-core ubuntu leg even running
alone on a quiet machine. Their 2 s request/cleanup/acquire budgets are
below real startup cost there (identity capture, frontend spawn, lock
acquisition on cold IO). Raised to 10-30 s; still hang guards - a
wedged run fails loudly.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The worker-log dumps settled the CI macOS failures: workers were alive
with created-but-EMPTY logs when the 3 s readiness / 5 s terminal
deadlines expired - busy in startup, not hung and not refused. Worker
startup re-hashes the entire executable image for the exact-build
fingerprint before its first write; for the ASan test-runner that is a
multi-hundred-MB read+SHA-256, which the 3-core CI runner VMs with cold
IO cannot finish inside the old budgets, while a warm 18-thread dev
machine finishes in well under a second.
Raise the fixture budgets to 15 s readiness / 20 s terminal. These are
hang guards, not benchmarks: a wedged worker still fails loudly, and
the failure path now dumps the (empty) worker log as proof either way.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The four readiness/terminal failures on the CI macOS legs persist even
with the suite serialized on a quiet machine, so the worker is exiting
rather than starving - and its exit reason lives in the worker log and
response files, which teardown deletes before the asserts print.
Dump them at each failure site (async readiness, probe non-terminal,
backlog logging, oversize containment) so the next failing run names
the worker's actual error instead of a bare ASSERT.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The parallel runner's isolated-process execution exposed three cli tests
that were passing for the wrong reason on Linux: cbm_zed_config_dir (and
the VS Code/vendor path resolvers) prefer $XDG_CONFIG_HOME over
$HOME/.config, CI runners export it, and an isolated `test-runner cli`
process therefore resolved config paths OUTSIDE the test fixture — the
tests only stayed green in the all-suites single process because an
earlier suite's env mutations leaked into them (hidden cross-suite
coupling, deterministic failure in isolation, macOS unaffected since its
resolvers do not consult XDG).
- cli_detect_agents_finds_zed: save/set XDG_CONFIG_HOME to the fixture's
.config for the detection call, restore after.
- cli_vscode_profile_mcp_uninstall: same pin alongside its existing
HOME/PATH/APPDATA handling.
- cli_durable_profiles_follow_current_vendor_paths: XDG_CONFIG_HOME joins
its unset-and-restore env list.
All 222 cli tests pass locally with a hostile XDG_CONFIG_HOME exported.
No production code changed; no assertion weakened.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Follow-up correcting d8a78e8, whose message claimed a broken merge cap
crashes "within the first second" so the 10s alarm could not mask it.
Falsification disproved that claim AND found the guard was vacuous on
POSIX from the start:
- With CBM_TS_STACK_MERGE_MAX_DEPTH deleted outright, the ORIGINAL
unbounded test stayed GREEN (220s full grind, no crash) on macOS ASan.
- Small thread stacks (2 MB, then 512 KB) did not change that.
- Instrumenting the vendored recursion showed why: the recursive merge
path is NEVER ENTERED for this input in the observable window — max
depth stays 0. There is nothing for a stack bound to catch here.
So the cap-regression DETECTION lives in the Windows in-process branch
(real ~1 MB native stack, where #913 actually manifested — exercised by
the windows CI leg). The POSIX branch is, and always was, a bounded
no-crash smoke of the pathological parse; the 10s alarm makes it
deterministic (12-15s across three runs, was 2-218s machine-dependent)
without any detection loss, because there was no POSIX detection to
lose. The experimental pthread machinery is dropped; the test comment
now carries the falsification data so no future reader mistakes the
POSIX branch for a cap guard.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
perl_glr_deep_parse_recursion_capped wandered between 2s and 218s for the
IDENTICAL input: past the merge cap the ambiguity-exploded GLR parse grinds
at an environment-dependent rate (allocator/scheduler state in the forked
child), and the test paid for however long that grind ran. It was the
critical path of every parallel run and the dominant chunk of the
sequential one.
The guard's actual signal is the CRASH, not the grind: when
CBM_TS_STACK_MERGE_MAX_DEPTH regresses, the recursive stack-merge
overflows within the first second at this depth (same profile as the
sibling pre-guard probe, rc=139 in under a second). The forked child now
arms a 10s alarm whose handler exits cleanly — a real SIGSEGV/SIGBUS
still terminates the child by signal first and stays visible to
WIFSIGNALED, so the RED path is untouched; only the pointless remainder
of the grind is skipped. Windows (in-process branch) is unchanged.
Suite b: 13s across three consecutive runs (was 2-218s). Full parallel
run: 97s wall, 6,374 passed / 0 failed / 1 skipped, union guard clean —
vs 351s sequential on the same machine (3.6x).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The runner executed all 104 suites sequentially (~10 min locally, the bulk
of every CI test leg). The suites are process-isolated already (per-process
mkdtemp HOME sentinel; two full parallel runs produced zero cross-suite
failures), so the serialization was pure convention.
- test-runner --list-suites: prints every registered suite, one per line,
emitted by the SAME macro table that executes suites — the list cannot
drift from the run set by construction.
- scripts/run-tests-parallel.sh: runs each suite as its own process
(jobs = CPU count; CBM_TEST_PAR_JOBS overrides) under a ZERO-LOSS
CONTRACT: a union guard fails the gate if the set of suites that
produced a result differs from --list-suites (nothing can be silently
dropped, a newly added suite is picked up automatically); per-suite
pass/fail/skip are summed into the sequential runner's exact summary
format; any suite crash, failure, or omission exits nonzero. Per-suite
wall times are printed for balance tracking.
- make test-par: the parallel target. make test (sequential) is unchanged
and remains the escape hatch (CBM_TEST_SEQUENTIAL=1 in test.sh).
- scripts/test.sh: builds, then routes through test-par — every CI test
leg gets the speedup with zero workflow-topology change (same jobs,
same gates, same billing; the legs just finish sooner).
- The stack_overflow suite is split into a/b/c (7+6+7 of its 20 tests,
pure re-registration): as one suite it was the wall-clock critical path
of any parallel run — every other suite finished underneath its ~4
minutes.
Measured locally (Apple Silicon, ASan runner): sequential ~10 min vs
parallel 232 s, totals identical (6,361 passed / 0 failed / 1 skipped),
union guard clean. The TSan leg keeps its dedicated subset runner
(sequential) — out of scope here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The PR CI caught two consumers the format migration missed — both parsed
tool output by its old shape and failed silently or with a wrong count:
- hook-augment built its PreToolUse additionalContext by reading the
legacy 'results' object-array from search_graph format:json; after the
json-tree reshape that key no longer exists, so the hook emitted nothing
(caught end-to-end by tests/windows/test_hook_augment.py). The parser
now walks groups -> qn_prefix/file + column-ordered rows. A new local
test feeds the parser the LIVE server envelope, so any future drift
between response shape and hook parser fails in the normal suite on
every platform — not only in the Windows CI guard. Reproduced RED
(ctx NULL, the exact CI failure) before the fix, GREEN after.
- smoke-test.sh carried six old-shape parsers (callers[N], rows[N]{,
clusters[N], semantic[N]{, and the assert_toon_table helper); the tree
headers never matched, so trace verification read 0 callers. All six
updated to the tree contract; the full smoke script passes end-to-end
locally.
cli suite 222 passed; lint clean; scripted smoke ALL PASSED.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fresh-eyes review of the phase-2/3 work; five defects, each against our own
stated principles (no silent caps, exact counts, same model in both
encodings):
- trace callees_total/callers_total counted test-file rows even when
include_tests=false hides them from the table — the scalar disagreed with
the enumerable set (a field-eval agent read callers_total=175 against a
handful of visible rows and stopped trusting the tool). Totals now apply
the same filter as the emitters; description documents the transitive
(within-depth) semantics. Guard test: filtered=1 vs include_tests=2.
- cycles clamped the SCC list at 100 silently, losing the true count.
cycles_total is now always exact and cycles_omitted is emitted when the
display clamp engages (both encodings).
- detect_changes accepted a 'limit' param (referenced by its own omitted-
hint) that the schema never declared — undiscoverable. Declared.
- detect_changes format:json lacked the impacted_modules rollup the tree
encoding carries — model parity restored via a shared rollup helper.
- tree search extra-field cells were appended raw: a signature with spaces
shifted every following column. Cells now route through the shared
emitters (space-triggered quoting, '-' placeholder); the properties test
pins the quoted form.
Full suite 6,373 passed / 0 failed; lint clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase 3 of the graph-analysis overhaul. Two new capabilities on a shared
multi-source traversal substrate, plus a context-bomb guard.
detect_changes — rebuilt into real blast-radius analysis:
The old handler parsed and echoed depth but never traversed: impacted was
just the symbols DEFINED in changed files. It now resolves the git diff to
those symbols, then runs ONE multi-source BFS to the transitive impact set.
- default direction inbound = the blast radius (transitive callers that may
need review); outbound/both available.
- seeds are excluded from the result, and a changed file reached from
another changed file is not counted as extra impact (that is not
downstream fallout).
- merge_base SHA pins the exact diff base for reproducibility.
- impacted_modules: a 2-segment (src/mcp, tests/...) rollup — a quotient
view of where the radius lands, always complete (overflow bucketed, never
silently dropped). Per-symbol rows cap at limit (default 200, nearest
hops first) with impacted_total exact and impacted_omitted surfaced.
On a 7-file core diff: 1469 impacted symbols, 64KB unclipped -> 10.7KB with
the rollup + cap. format:json returns the same model as structured JSON.
cbm_store_bfs_multi (store): one recursive CTE anchored on ALL seeds via a
temp table — never seed_count separate walks over overlapping hubs.
Seed-exclusion NOT IN, MIN(hop) across the set, canonical (hop,id) order,
a memory-safety ceiling that reports *truncated instead of silently
capping. Reproduce-first: seed-exclusion + min-hop, and truncation.
get_architecture aspects:[cycles] — SCC condensation:
Iterative Tarjan (heap stacks, no recursion overflow) over the CALLS graph
reports circular dependencies (strongly-connected components of size > 1).
cbm_store_fetch_call_edges feeds it with a 400k-edge scan budget that
reports partial coverage rather than truncating silently. Opt-in ONLY —
never via a default or 'all' call, since it scans the whole call graph.
On this repo it surfaces 26 real cycles (e.g. the C-LSP type-evaluation
recursion) in ~5KB. Reproduce-first: a 3-node cycle is found, an acyclic
chain is not, and a default call does not run the scan.
get_code_snippet: cap the returned line span at 500 (a structural Module/File
node spans its whole file — an unclipped read returned ~400KB in one call).
source_clipped flags it; start_line/end_line still allow a targeted re-read.
Full suite 6,372 passed / 0 failed; lint clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
- cbm_toon_* renamed to cbm_tree_* (the emitters have emitted tree syntax
since the format flip; the name now matches).
- format enums reduced to [tree, json], default tree; dead tree/toon flags
removed; tool RESPONSE descriptions rewritten for the grouped-tree
contract (group prefix printed once, full qn = prefix + '.' + name).
- format:'json' is now the json-stringified tree: the SAME grouped model
serialized as JSON — {total, cols, groups:[{qn_prefix, file,
rows:[[column-ordered cells]]}], has_more} for search, per-leg
{cols, groups} for trace (risk/args columns follow the flags),
query_graph's columns+rows shape already complied. Per-row key envelopes
and the legacy property blob are gone (blob-never-spills pinned by test);
requested add columns in both encodings.
- include_connected keeps the per-node-object shape (nested neighbors) —
the one remaining legacy consumer, conversion tracked.
- legacy verbose emitters deleted (bfs_to_json_array).
Full suite 6,367 passed / 0 failed; lint clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Round-2 overhaul of the tool-output surface, built eval-first (byte harness +
no-context field-eval agents, always vs the v0.9.0 release binary):
Format — tree syntax everywhere:
- compact_out emits tree tables: 'key: N (cols: a b c)' headers, indented
space-delimited rows, '-' placeholders, space-aware quoting. Every table
in every tool converts mechanically.
- Grouped prefix-factored output is the DEFAULT for search_graph and
trace_path: rows grouped under their shared (qn-prefix, file) printed
once, row qn = group prefix + '.' + name (stated in-band). Measured:
192-hit search 2,929B (was 4,811 TOON / 238,466 v0.9.0), 82-row trace
2,256B (was 4,234 / 8,094). Field eval: 5/5 answers correct, QN
reconstruction verified, no comprehension loss.
- detail:'ids' tier: bare-qn enumeration for wide sweeps (-44%).
Pagination — exactly-once, stateless:
- store_meta generation (per-DB random uid + mutation counter bumped in
cbm_store_upsert_project — every index run's choke point): cursors can
never silently validate against a rebuilt DB.
- trace cursors: opaque token (leg, generation, params-hash, (hop,id)
watermark); pages drain callees then callers with exact
callees_total/callers_total on every page; teaching errors for
params-mismatch and staleness. Guard: 12 callees paged 5+5+2, every row
on exactly one page.
- deterministic ordering: BFS ORDER BY hop,id; canonical (hop,id) sort +
min-hop dedup across same-name seeds (hop fed risk_labels wrong before);
search (name,id)/(rank,id) tie-breaks make offset pages stable.
Lean defaults + teaching errors:
- get_graph_schema no longer advertises blocked internal fields (fp/sp/bt).
- index_repository coverage lists: counts + 5 examples + one-line pointer
notes (full record was always in the logfile); 6.0KB -> 1.5KB here,
53KB observed on a large repo. index_status gates the git context block
behind verbose:true.
- trace limit param (<=5000) + truncated:true — never a silent truncation;
invalid direction now errors with the valid values; requesting core
columns via fields hints instead of emitting empty cells; cypher
resolves f.file/f.qn/f.path aliases.
Trace-correctness fixes (reproduce-first, RED verified):
- bfs_collect_edges 4KB id-string overflow: past ~1000 visited nodes the
edge SQL failed to prepare and EVERY trace edge silently vanished; now a
per-connection temp-table join (star guard: 1200/1200 edges).
- bfs_union_same_name: min-hop across seeds + edge dedup.
Guards: gating tool-output regression suite (context-explosion detector:
blocked-field leaks, repeated-key envelopes, oversized notes + calibrated
byte ceilings + semantic floors). Full suite 6,367 passed / 0 failed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On-disk write connections set no journal_size_limit, so the -wal file is never
physically shrunk: all our checkpoints are PASSIVE (TRUNCATE is deliberately
avoided because its ftruncate(fd,0) can SIGBUS a sibling mmap'd process on
macOS), and PASSIVE never ftruncates. Under checkpoint starvation the WAL then
grows without bound (reported: 115 GB, filled the drive) and even a later
successful checkpoint leaves the file at its peak size (#1083).
Set journal_size_limit = 256 MiB on write connections so SQLite truncates the
-wal back to that bound whenever a checkpoint can finally reset the log —
reclaiming disk opportunistically during a session, not only on clean close.
256 MiB is far above the healthy WAL (~4 MiB under the default 1000-page
autocheckpoint), so normal indexing never triggers truncate/regrow churn; the
limit only fires after abnormal growth. No TRUNCATE checkpoint is introduced.
Scope: this is the *reclaim* half of #1083. It does not prevent the peak under
continuous reader contention (a checkpoint can't reset the log while a reader
holds a mark) — the write-amplification driver (concurrent full re-indexes of
one repo) is serialized by the cross-process pipeline lock in #892.
Test: a file-backed write store now reports journal_size_limit = 256 MiB
(RED on main: -1 / unlimited).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel