Why: the activation rendezvous is per OS account, so an install or update
run under another HOME / CBM_CACHE_DIR (a sandbox, a second profile) meets
the account's live daemon at the same endpoint. The daemon's
activation-shutdown handler verifies only the requester's build, never its
cache, so the CLI's scope decision is the only thing that tells "my daemon"
from "someone else's". That decision drained the cohort unless the active
cache was positively DIFFERENT: a busy session group (another activation or
a session start held it, so the active identity was never read) and an
unreadable active cache both fell through to "ours", printed "Stopping
active CBM sessions" and shut the daemon and every session behind it down.
Fix (user decision 2026-09-28, "unknown = foreign, keep it"): only a
positive match may drain. cli_activation_resolve_scope returns
CLI_ACTIVATION_SCOPE_ACTIVE only for an OK admission (exact identity, cache
included, or an empty group) or for a conflict whose cache fingerprint
equals this activation's. BUSY and a blank active cache now return the new
CLI_ACTIVATION_SCOPE_UNCONFIRMED, handled like a foreign cohort: no lock is
taken, no drain request is sent, the mutation proceeds, and the audit log
records "quiesce_skipped: active cohort ownership unconfirmed (...)". A
read-only status probe names the daemon that was left (pid, version) and
the notice says how to stop it: close its sessions, then run
`codebase-memory-mcp daemon stop`. The confirmed paths are unchanged: a
normal self-upgrade still drains its own daemon, a clearly foreign cache is
still left alone.
Proof (product binaries; private CBM_RUNTIME_DIR; an own permanent test
daemon started from the same build under a host HOME/cache; a sandbox
HOME/cache on the same runtime dir; the cohort maintenance gate held by a
second process, i.e. a busy session group):
- origin/main: "Stopping active CBM sessions and operations for
install...", the test daemon was stopped (daemon status: not running),
and the install still failed (rc=1 after 23 s, "could not be stopped
safely").
- this change: "Leaving the running CBM daemon untouched (pid 76713,
version dev): this install could not confirm that it serves the cache
this install targets (the session group was busy, so its cache identity
could not be read)." The test daemon kept running with the same pid and
the install completed (rc=0, 10 s).
A lifetime record whose cache field is zeroed is refused (UNSAFE) by both
builds with the daemon kept; that path never drained and is unchanged.
Tests (cli suite; fixture daemon in a private runtime dir; sandbox HOME):
- cli_install_foreign_home_unreadable_cohort_cache_keeps_host_daemon
(the new seam cbm_cli_set_activation_scope_cache_unreadable_for_test
blanks the observed active cache fingerprint)
- cli_install_foreign_home_busy_cohort_keeps_host_daemon (the maintenance
gate is held through the product's own lock-directory handle)
Both were RED 3x on origin/main for the right reason (the host daemon was
drained: ASSERT(host_serving)), are green with the fix, and turn RED again
when the fix is reverted. Controls stay green:
cli_install_into_host_namespace_still_drains_host_cohort (confirmed same
cache: drained, as before) and
cli_install_{skip_binary,binary}_into_foreign_home_never_drains_host_cohort
(clearly different cache: kept, as before).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The parallel-scheduler contract failed twice on test-windows CLANG64 while
running fake suites only:
- PR #2394 (run 36364479687): "Windows timeout race refused without a
surviving descendant to refuse over". The refusal came 38s after the
forced leader exit (a cold powershell/CIM descendant probe, once in the
wave loop and again in the cleanup pass), while the fixture's descendant
was a `time.sleep(30)` that had already exited on its own.
- PR #2345 (run 36199749518): "suite 'hang_after_summary' taskkill could
not prove process-tree cleanup". The scheduler bounded taskkill.exe with
--kill-grace, which the contract passes as 1s, so a slow cold start of
the tool was reported as a failed cleanup.
Both verdicts were a race between a clock and runner latency. An audit of
the fixture found more of the same shape:
- Every hanging fixture ended on its own after 30s. A starved scheduler
could therefore see a "hung" leader exit 0, and a descendant the
scheduler failed to kill could vanish before the leak check looked.
- The 1s --timeout could kill a slow leader before it had printed its
summary (asserted pass=1), spawned its descendant or written the
descendant pid. That pid was then read back with int() from a file
written create-then-write, so a reader could see no file or an empty one.
The empty-file read recorded earlier was the scheduler's ready file,
which 3b9052c9 already publishes atomically; the fixture's own writer
had the same shape.
- The descendant installed its SIGTERM-ignore during its own start-up. A
SIGTERM that won that race skipped the SIGKILL escalation the POSIX leg
exists to cover, silently and with the verdict unchanged.
- On Windows, the stubborn-tree leak check was a single look right after
taskkill, but TerminateProcess returns before its target has exited.
Fix it by construction, not by budget:
- A gate. Every process that must outlive the scheduler's decision blocks
reading stdin. That is a pipe whose only write end the contract holds,
handed down through the scheduler, and the read returns only at EOF. The
contract closes the pipe once the verdict is recorded. If the contract
dies, EOF releases everything, so nothing is orphaned and no fixture
ends on a clock.
- The three hanging scenarios run under the scheduler's existing
pre-terminate barrier. They are released only once the fixture has
published `<suite>.established`, which happens after the summary is
flushed, the descendant has confirmed it is armed, and its pid is known.
The marker is written to a temp file and moved into place with
os.replace, so a reader sees either no file or the whole file.
- The Windows leak check waits on the descendant's process handle. That
wait is exact, not a race: the descendant is held by the gate, so the
scheduler's kill is the only way it can end.
- run-test-wave.py runs taskkill through windows_taskkill_tree() with its
own stable-state budget (WINDOWS_TASKKILL_SECONDS = 15), as 132e8fc3 did
for the descendant probe. --kill-grace keeps bounding what its name
says. Real runs already pass --kill-grace 15, so they are unchanged.
The contract pins this structurally, the same way it pins the probe:
exactly one taskkill site, no kill-grace there, and a helper bounded by
the constant that fails closed on a timeout, a start failure or a
non-zero exit.
Every assertion keeps its contract:
- A leader that hangs after a green summary is recorded as rc=124 pass=1.
- A zero-test child is recorded as rc=97.
- The wave continues after a bounded failure.
- A SIGTERM-resistant tree is killed in full.
- A leader that exits between the timeout decision and termination is
still cleaned up on POSIX. On Windows the scheduler refuses (rc=2,
naming the cleanup failure) over a descendant that is really alive.
A new self-check fails loudly if a held leader or descendant is not live
while the barrier holds, that is, if the fixture no longer exercises what
those assertions claim.
Proof, on macOS arm64 and a Linux arm64 container. The Windows branch was
reasoned through; the VM is down.
- Torture hook (proof only) that delays each hanging leader 2s, past the
1s timeout. The old fixture is RED 3/3 on every scenario (pass=0,
FileNotFoundError on descendant.pid, "lost its bounded result"). The new
one is green, including at 8s, and 10/10 under the hook plus CPU load.
- The #2394 mechanism: after an injected 31s refusal latency the old
descendant is gone and the new one is live. It exits 0.02s after the
gate closes.
- The taskkill pin is RED on the old scheduler and green on the new one.
- Stress, 20 runs each under CPU load. The old full contract passed 20/20
on macOS and 20/20 on Linux, so these hosts do not reproduce the CI
timing. The rebuilt held scenarios passed 20/20 on macOS and 20/20 on
Linux. The full contract after the change passes a single run on macOS
and on Linux, and 20/20 on each leg under CPU load (4 load workers;
the Linux container is capped at 4 CPUs).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
daemon_ipc_windows_startup_retries_transient_rendezvous_reader failed on
a windows-latest CLANG64 runner (ASSERT(startup_observed); the whole
suite took 7 s) on a PR that touched only CLI files.
ipc_test_startup_gate_wait_acquired polled 50000 x cbm_usleep(200) and
returned false once the budget ran out. On POSIX that is ~10 s, but on
Windows cbm_usleep(us) is Sleep(us / 1000), so cbm_usleep(200) is
Sleep(0): the budget collapsed to 50000 bare yields, which a loaded
runner spends in milliseconds before the startup thread reaches the
gate. A timeout decided the verdict.
The helper now waits until the startup thread has either parked in the
gate or returned without reaching it (a new `finished` flag, set by
ipc_test_win_startup_call after its call returns). Both flags only go
false -> true and one of them is always set: before the gate the startup
path only try-acquires, and a thread parked in the gate cannot return
until the test releases it. No bound is left to decide the verdict. A
thread that returns without reaching the gate fails at once, and the
test reports its try-acquire status as the reason.
The same shape is removed from two cross-platform tests:
- daemon_ipc_wait_forever_is_interruptible waited for the server's
forever-wait under a 2 s wall-clock deadline. Past the deadline the
test skipped the interrupt, so the join then blocked on a server
parked in WAIT_FOREVER: a hang, not a failure. It now waits for
`waiting` or for a new `finished` flag set when the server thread
returns.
- daemon_ipc_connect_waits_for_delayed_listener capped its wait for
`delay_started` at 2000 polls. That store is the thread's first
action, so the wait needs no attempt bound.
No production change and no new sleep: the existing per-poll yields
stay.
Proof. A standalone harness mirrors the gate, both helpers and the
thread wrapper, with cbm_usleep modelled as on Windows
(Sleep(us / 1000), so cbm_usleep(200) is a bare yield):
- startup thread arriving only after 50000 polls: old helper 20/20 FAIL
(startup_observed=false with call.result=1, the CI signature), new
helper 20/20 PASS;
- startup thread descheduled 500 ms: the old helper gives up after 9 ms,
10/10 FAIL; new helper 10/10 PASS; control, old helper with POSIX
usleep: 10/10 PASS;
- thread returning without reaching the gate: the new helper fails
within 11 ms with call.result=-1 (the old one took 16 s under POSIX).
Real daemon_ipc suite on macOS: with the server and listener threads
held 4 s and 6 s (beyond both old bounds) the two cross-platform tests
pass. The old 2000-poll loop under the 6 s hold fails
ASSERT(delay_observed). The old 2 s deadline under the 4 s hold hangs in
the join until killed. A server that skips accept makes the new loop
fail ASSERT(reached_wait) with no added wait.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
watcher_poll_only_watched_projects failed on the macOS LSan job of PR
#2295, whose diff was Java-only, with
`FAIL tests/test_watcher.c:2971: index_call_count == 0, expected 1`
(8211 passed, 1 failed). The LSan report that followed points into the
aborted test body (watcher and store never freed after the early FAIL):
a consequence, not a separate leak.
The assert expected ONE touch+poll after a dirty change to reindex
exactly once. That needs every git probe of the poll to succeed. A poll
whose git spawn is refused under runner load is a fail-closed no-op by
design (#937: the baseline stays uncommitted and the next poll retries),
and that path returns without a log line. This is the same test race
that #2376 fixed in watcher_unwatch_drains_pending_free and
watcher_sustained_failure_logs_once_issue2015, not a production defect.
Holding one exhausted spawn with
cbm_subprocess_force_spawn_eagain_for_testing at that poll reproduces
the CI output exactly: watch, baseline, then
`FAIL tests/test_watcher.c:2971: index_call_count == 0, expected 1 == 1`,
with no watcher.changed and no watcher.git.failed.
The old test was also partly inert. The Go original (TestPollAllOnlyWatched,
from the #49 fix that replaced scan-all with an explicit watch list)
registered BOTH projects in the store and asserted that the unwatched
one gets no watcher state. The C port dropped the store registration, so
nothing let the watcher learn about B, and the B half could not fail.
The test now:
- registers A and B in the store, and watches only A;
- asserts that the baseline poll indexes nothing and does not adopt B;
- makes both worktrees dirty (a dirty change, not a commit, so a late
baseline still sees it);
- polls until A's reindex has happened, asserting after EVERY poll that B
has no watcher state (cbm_watcher_index_failure_count == -1), was never
reindexed, and that the watch count is still 1;
- asserts that A was reindexed exactly once.
No timer or deadline decides the verdict. A watcher that never reindexes
A hangs, which the runner reports as a failure, never a pass.
Evidence (macOS arm64, ASan/UBSan test-runner, CBM_SANITIZED, so one
exhausted spawn = 10 injected refusals), with temporary injection that
has since been removed:
- CI-shaped (baseline OK, first change poll refused): consumed=10 at
loop poll 1, watcher.changed at loop poll 2, PASS.
- Heavy (baseline poll plus loop polls 1-3 refused): loop poll 4 takes
the baseline, poll 5 reindexes A; loop polls=5 watched=1 other=0, PASS.
- Mutation, the pre-#49 scan-all (poll_once adopts every stored project):
`FAIL tests/test_watcher.c:3001: cbm_watcher_index_failure_count(w,
"projB-ow") == 0, expected -1 == -1`, both unforced and with the heavy
refusals (A's and B's baseline spawns both refused). The other 82
watcher tests stay green under that mutation, so this is the only
test guarding #49.
Final tree: build/c/test-runner watcher, 5/5 runs, 83 passed each.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The #2184 parallel test named a local `near`. Windows headers define
`near` (and `far`) as empty legacy pointer qualifiers, so under MinGW
`cbm_gbuf_t *near = ...` loses its name and test_parallel.c fails to
compile (test-windows CLANG64 2/2: "expected identifier or '('").
Rename it to `gb_near`. A MinGW syntax-only check of tests/test_parallel.c
reported 4 errors before this change and none after.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Why: the test runner isolated HOME and CBM_CACHE_DIR but not
CBM_RUNTIME_DIR, and neither of the first two moves the daemon
rendezvous. Its key is a constant product hash (cbm_daemon_rendezvous_key,
"c888acc3ae367a1e"), and with no explicit parent and CBM_RUNTIME_DIR unset
cbm_daemon_bootstrap_endpoint_new falls back to the account-wide default
parent (/private/tmp on macOS, /tmp on Linux, LocalAppData on Windows).
Every endpoint a test resolved without an explicit parent was therefore
byte-identical to the developer's live daemon socket
(/private/tmp/cbm-daemon-<uid>/cbm-c888acc3ae367a1e.sock), whatever HOME
said.
The only in-runner product caller is CLI activation
(cli_activation_production_context_init, reached from cbm_cmd_install,
cbm_cmd_uninstall and update). Only a process-global seam kept it off the
default: SUITE(cli) sets that seam on entry and clears it on exit. On the
default endpoint, activation admits into the live daemon's cohort. When
the cohort is ours or busy, or the active cache fingerprint is unreadable,
activation prints "Stopping active CBM sessions" and drains them. That is
the same class of failure as an old binary's install, which stopped a
user daemon and six MCP sessions. One test outside SUITE(cli), or one
fixture that drops the variable, was enough to reach it.
daemon_bootstrap_runtime_dir_env_relocates_rendezvous did exactly that:
it restored by unsetting CBM_RUNTIME_DIR.
Fix:
- tests/test_main.c: tf_setup_cache_sentinel creates a fresh per-run
runtime parent (th_secure_runtime_parent_new "run", under /tmp on POSIX
and LocalAppData on Windows) and sets CBM_RUNTIME_DIR to it for the
whole run. It replaces any inherited value, the rule
scripts/test-runtime.sh applies to the shell harness. The existing
atexit cleanup removes the directory.
- src/daemon/bootstrap.c, CBM_ENABLE_TEST_SEAMS only: once a test runner
arms the guard, an endpoint request with no explicit parent and no
CBM_RUNTIME_DIR is refused with a stderr diagnostic instead of
resolving the default. No product binary arms it.
- tests/repro/repro_main.c arms the same guard. No reproduction needs the
rendezvous.
- tests/test_daemon_bootstrap.c: the relocation test restores the value
it found instead of unsetting it.
- tests/test_helpers.h: th_secure_runtime_base() names the anchor that
th_secure_runtime_parent_new already used.
Proof:
- Before, on origin/main 1f9b4db1: runner_isolation is RED 3/3 at its
first assertion (CBM_RUNTIME_DIR is NULL during the run).
- After: green 3/3. The endpoint resolved through the product resolver
lies inside /private/tmp/cbm-runtime-run-XXXXXX/, never at
/private/tmp/cbm-daemon-<uid>. All 43 activation-guard scopes logged
by the suites below are fixture or cli-suite directories, and the
guard never fired. The per-run directory is gone after exit, and the
live daemon and its runtime directory are unchanged.
- Revert: dropping the setenv gives RED (value is NULL). Disarming the
guard gives RED (refused_without).
Tests:
- runner_isolation: runner_isolation_daemon_runtime_dir_is_private,
runner_isolation_refuses_default_daemon_runtime
- Green on macOS: cli (328), mcp + mcp_mutation_guard (344),
daemon_runtime + daemon_application (106), daemon + daemon_version +
daemon_frontend + daemon_bootstrap + daemon_ipc (117), runner_isolation
+ activation_transaction + version_cohort + project_lock +
index_supervisor + private_file_lock + lock_registry (73). All skips
are existing Windows- or Linux-only platform skips.
- make -f Makefile.cbm lint-ci passes. test-repro-runner builds.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two CI failures, both in the tests this branch added:
1. Windows (MinGW clang64) did not compile the test runner:
test_parse_coverage.c and test_pipeline.c called POSIX setenv/unsetenv,
which the Windows CRT does not declare. They now use the repo's
cbm_setenv/cbm_unsetenv from foundation/compat.h, as the other suites do.
2. sql_dump_of_many_megabytes_is_indexed_not_timed_out_issue1735 failed on
the ubuntu-24.04-arm ASan leg with "parse timeout". It extracted a 26 MB
dump under CBM_EXTRACT_BUDGET, which is CPU time, so the verdict
depended on runner speed. It passed on the fast legs and tripped on the
slow one, and the early FAIL return leaked the result (the LSan report
in that log). The property behind "indexed, not timed out" is a count:
parse work is set by the statements, not the rows. The test now extracts
unbudgeted, asserts the outcome (the view and the subquery usage are in
the graph), and bounds the tree by the same 330 statements with 4 rows
each. Results are freed before any assertion.
Binds to the fix: forcing the full parse for the big dump through the
CBM_TEST_SQL_FULL_PARSE_ON seam fails the bound (5,158,222 tree nodes
vs a bound of 52,872); with the row exclusion it passes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel