Both Windows runner soak legs died right after 'server running' while
the daemon's own log shows a healthy diagnostics.start with valid
paths — the wait polled 10s for a snapshot whose first WRITE lands
one 5s interval after start and can exceed 10s on a cold 4-vCPU
runner mid-initial-index (the VM's 18 cores never miss it; the x64
leg's SIGPIPE was the same failure path dying inside the pipeline).
Budget doctrine: the wait sits above the worst case.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Hosted Windows runners reset the readiness poll's full-archive GET
mid-body (WinError 10054 on the server, ~99 aborted transfers per
job) while the identical stack passes on the VM; every windows smoke
job died at 'fixture server did not serve' with a healthy server. A
HEAD proves the artifact routes without transferring the body per
poll; the download phases still own full-body transport and report
phase-precise if that is ever broken.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every ui-variant smoke job failed on every platform in dry run
30226483380 — and it turns out a ui-variant Phase 15 had never
legitimately passed anywhere: the old probe (a) never passed --ui=true,
but the HTTP UI is a persisted DEFAULT-OFF setting, so a fresh profile
(every runner, every smoke HOME) can never serve; (b) fed stdin an
empty file, but the UI does not pin the process — stdio EOF ends it
cleanly (rc=0) before any poll sees it serve; (c) gave a 10s window to
a server measured to bind ~6s after launch on a FAST host; (d) POSTed
an MCP initialize at /rpc, which speaks the UI's own narrow query
protocol, not MCP.
The rebuilt probe mirrors the drive-listing guard's proven invocation
(--ui=true --port=N equals-form, held-open stdin), waits 30s, and
verifies the API surface with the stateless GET /api/ui-config.
Validated red->green against the release-shaped artifact: the full
darwin ui artifact lane now passes 15a/15b legitimately.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The unified smoke-local.sh readiness-polls the fixture server with
curl; the Alpine image never had it, so every poll iteration failed
command-not-found and the portable smoke leg reported 'fixture server
did not serve' with a perfectly healthy server. Local-only image (CI
portable smokes run on ubuntu runners, curl preinstalled).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On Windows the fixture server's atomic port publish (os.replace) — or
a first-touch AV scan of the fresh file, with Defender ACTIVE on the
VM — briefly denies the read: is_file() is true, read_text() raises
PermissionError, and the poll loop crashed the whole contract with a
bare traceback (observed on the VM once the 0e/0h needle fixes let
the chain reach this self-test at all). Treat an unreadable-yet port
file as not-ready and keep polling inside the existing deadline.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
BSD mktemp substitutes only TRAILING X's, so the template
cbm-soak-leg-XXXXXX.log creates a near-literal file on macOS and the
second leg of the same job collides with the first leg's leftover
('File exists'). First runs mask it — which is why the quick leg
passed and query-leak failed locally, and why CI's macos soak jobs
(BSD mktemp, two legs per job) would fail the same way.
Drop the suffix; the path is only a log handle.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The manifest content (leg, slice, list hash) is fully determined at
slicing time, so write it up front: a red run's manifest is exactly as
load-bearing for the cross-shard union proof as a green one's.
With that, a missing manifest at upload time can only mean the job
died before the harness started (contract step / build failure) — that
failure is already the job's red, so the upload warns instead of
stacking a second error on top (the recurring 'No files were found
with the provided path: build/c/test-logs/shard-manifest.txt'
annotation). The shard-completeness job still gates the union.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Local soak legs write soak-results*/ into the worktree; a bulk add
swept one run's logs into the previous commit. Remove them from
tracking and ignore the output directories (CI uploads them as
artifacts; they are never repository content).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dry run 30225685743 failed at the ensure-defender step on every
Windows job: the hosted images policy-lock real-time protection OFF
(WinDefend starts, Set-MpPreference accepts, RTP stays off —
untamperable from the job).
Revert to the pre-gate runner behavior: no ensure-defender step in
runner jobs, the release Defender scan is on-demand and fail-soft
again. The local VM preflight KEEPS the fail-closed Defender-ON gate
(same canonical script) — AV-interaction coverage is a deliberate
local superset where the platform allows it.
The venue-parity contract now enforces the INVERSE for runners
(ensure-defender must not gate hosted jobs) and still requires the VM
preflight gate; VM README documents the verified posture delta.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The venue unification moved the VM soak sequence and its completion
guard into scripts/soak-legs.sh; the smoke-fixture contract still
pinned the old direct soak-test.sh call in vm-run-tests.sh, failing
Step 0h — and with it every test leg on every venue of dry run
30224804737 (the test wave's 20 reds share this one cause).
Masked until now: the contract chain runs sequentially and Step 0e
failed first on the pre-fix tree, so 0h was never reached; post-fix
verification used --suites mode, which skips the chain. All contract
steps 0a-0j now verified green standalone in bulk.
Invariant preserved, anchor moved: the VM wrapper must route --soak
through the canonical entry under its protected temp root, and the
canonical entry must completion-guard every leg on the soak summary
(grep for '=== soak-test: PASSED ===' lives in soak-legs.sh:101).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
LSan (Linux amd64 leg): 200 bytes — the runtime client (192) plus its
IPC connection (8) — leaked whenever an earlier stage of
daemon_runtime_disconnect_cancels_blocked_non_index_child... failed:
both close blocks were gated on the request thread having completed,
so a failed run skipped the close entirely.
Teardown now closes unconditionally: close_begin interrupts the
transport (forcing the blocked call to complete), the join becomes
safe, and close_finish frees the client on every path.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dry run 30206811293: macos-15-intel release smoke failed B3 with a
visibly correct 'semantic: 50' table — BSD grep returns no-match for
the ENTIRE output when any line carries a NUL, control byte, or
invalid UTF-8 (verified: printf poisoned input -> grep -qE fails), so
one bad byte in one row silently unmatches every anchored check.
Close the class at the emission choke point: control bytes and
invalid UTF-8 force the quoted form; quoting escapes controls as
\u00XX and replaces invalid sequences with U+FFFD (RFC 3629
validation, NUL-safe). Tool output is valid UTF-8 by construction.
Unit test proven RED-on-revert (raw bytes emitted) and green with the
fix. The smoke's B3 failure path now also dumps od -c bytes so a
future red names the exact byte.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dry run 30206811293: windows-11-arm (both variants) and windows-2025
smoke failed in the update/uninstall E2E phases with 'private staged
copy failed secure open (error 32)' — ERROR_SHARING_VIOLATION. On the
newer hosted images a first-touch scanner transiently holds the
just-staged executable without FILE_SHARE_DELETE, so the launcher's
GENERIC_READ|DELETE retirement open collides on cold images.
Absorb the scan window with the same bounded backoff the staged
RENAME path already uses (<=~2.5s, exponential); every retry re-runs
the full secure-open validation, so no TOCTOU surface changes.
No deterministic local repro exists (the VM passes this phase with
Defender active; the collision is image-specific) — the next dry run
on the same three jobs is the arbiter.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
git-apply of the staged unification recorded the new canonical entries
as 100644; workflows execute them directly (run: scripts/ci/...), so a
fresh CI checkout would fail at ci-ok/shard-union/soak/package steps
with permission denied — the same class the parity worktree's own
preflight just tripped over locally.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes the closable local-vs-remote venue gaps so that, outside arch
legs / YAML glue / release plumbing, every local red is a remote red
and vice versa on shared legs:
- scripts/package-release.sh: THE canonical archive step (names +
five-file Windows bundle layout defined once); _build.yml's eight
inline archive blocks become calls to it.
- scripts/ci/smoke-artifact.sh: the artifact-flow smoke lane — build,
package, extract, then the canonical wrapper in artifact mode; wired
as run.sh smoke-artifact (compose service), win.sh smoke-artifact,
and directly runnable on macOS. Archive-layout bugs now surface
locally instead of in a release dry run.
- glibc-floor leg (Dockerfile.glibc22 + compose + run.sh): portable
binary smokes on ubuntu-22.04/glibc 2.35; the dynamic binary must
refuse there (2.38+ floor by design).
- Defender-ON parity (user directive): scripts/ci/ensure-defender.ps1
enables + VERIFIES real-time protection, fail-closed; runs in every
Windows CI job (_test x2, _soak x3, _smoke, pr.yml) AND in the VM
preflight; _smoke.yml's scan engine-failure soft-skip becomes a red
gate. Expected cost: slower Windows jobs (AV scanning during
install/build/test I/O); the next dry run proves the runner side.
- Contracts: launcher-bundle five-file check retargeted onto
package-release.sh + per-archive canonical-call association;
venue-parity contract requires the new lanes, counts one
ensure-defender step per Windows job, adds --help probes for the new
entries. Extended contract fails on the pre-change tree (verified:
20 violations on HEAD).
- VM README: ephemerality/Defender posture documented — utmctl has no
snapshot verb, so per-run revert stays a manual qcow2 option; the
sweep preflight remains the standing mechanism.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The venue unification moved release-smoke staging/execution out of
_smoke.yml into the canonical wrappers (vm-smoke.sh / smoke-local.sh),
so Step 0e's needle checks against inline YAML failed on every venue.
Re-anchor each pinned invariant where the behavior now lives:
- canonical-exe execution + profile-rooted fixtures + update fixture
identification: asserted in vm-smoke.sh / smoke-local.sh, plus the
workflow must call the wrapper on the extracted artifact
- version/install-audit preflight needles updated to the artifact-dir
text shape (still workflow provisioning)
- RUNNER_TEMP: absolute ban replaced by a provision/scan allowlist
walker + an explicit no-exe-under-RUNNER_TEMP backstop
No pinned security property is dropped; the retargeted contract fails
on the pre-unification architecture (verified against HEAD).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The staged venue-unification bundle: _smoke.yml onto the shared
wrappers with an extracted-artifact input (CBM_SMOKE_ARTIFACT_DIR),
soak-legs.sh as the one canonical soak entry (quick + query-leak legs
across all venues), protected per-user TEMP roots shared across
venues, clean-disk preflights before every VM/Docker run, win.sh and
run.sh routed through the canonical scripts, soak.yml retired, and
the venue-parity contract (whitelist walker: workflows may provision
or call canonical entries, nothing else).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This reverts commit f843db8, which removed the #581 fix on a false
attribution. The arm64 shards were crashing because of an unaligned SQLite
page-cache slab (fixed in #1274, arm64 shard 2/2 green since); removing this
callback never changed those crashes, which is what exonerated it. The Windows
soak then failed for the obvious reason: the leak was back.
Static MinGW links have no DllMain and register no TLS callback, so
mi_thread_done never runs and every thread leaks its heap -- 607 heaps after 300
requests holding 170 MiB against a ~300 KiB live set. With the callback the
10-minute Windows soak measured 1.1x against the original 53x.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The slab was a static char array handed to SQLITE_CONFIG_PAGECACHE. A static
char array carries no alignment guarantee and SQLite requires 8-byte-aligned
page-cache memory: x86_64 tolerates the unaligned access, arm64 faults. Every
suite that opens a store died instantly on windows-11-arm with
STATUS_ILLEGAL_INSTRUCTION (pass=0, secs=0) while pure-logic suites in the same
wave passed, and both arm64 shards were green on the commit before the merge.
It was also never the fix it was added for: measurement showed the slab
installed correctly and #581's growth unchanged, because the cause was thread
heaps never being released. Removing it rather than aligning it -- an unverified
optimisation has no claim on a release gate.
Note PR CI does not build the windows-11-arm leg, so this class of defect can
only surface in the dry run.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The refusal notes changed the error text the Windows launcher guards match
against, and test-windows-guards went red on this PR after they landed. They were
investigation instrumentation and they succeeded -- they turned 'staging
transaction open failed (status -3, os 0)' into ERROR_DISK_FULL and named the
exact site -- but a diagnostic that shifts a matched error string does not belong
in a release-blocking change.
Worth redoing deliberately on its own, with the guards' expectations updated in
the same commit.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The hosted windows-11-arm shards crash at process load (STATUS_ILLEGAL_INSTRUCTION,
pass=0, secs=0) while main passes them, and removing the TLS detach callback did
not help -- so the remaining candidate is this layer, which touches every
allocation on Windows through the interposer. It declares a _Thread_local guard
read on every malloc, and thread-local storage in a static MinGW aarch64 link is
exactly the fragile case; nothing in it needs to ship.
The profiler did its job: it produced the attribution that identified #581's
mechanism, and that analysis is recorded. Sources stay in the tree, unbuilt, so
a future investigation can re-enable them deliberately.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reverts the #581 fix mechanism, not the diagnosis.
main's dry run passes both windows-11-arm shards and test-windows-guards; this
branch fails all three. The only change here that affects process LOAD is the
.CRT$XLB detach callback, and the arm64 failures are load-time crashes
(STATUS_ILLEGAL_INSTRUCTION, pass=0, secs=0, five suites in the same second) --
while the guards exercise many launcher and payload spawns. The same binary
loads fine on a UTM arm64 VM, so the placement is tolerated by some loaders and
not others, which is not something to ship on a supported platform.
#581's root cause stands and is documented: a static MinGW link has no DllMain
and registers no TLS callback, so mi_thread_done never runs and every thread
leaks its heap (607 heaps after 300 requests, 170 MiB against ~300 KiB live).
The delivery mechanism needs to be one the loader accepts everywhere -- an FLS
callback registered at runtime is the obvious candidate, with no linker section
placement involved.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The copy loop's failure was reported as 'os 0' because the cleanup and close
calls after the loop clobber the thread's last error. Captures it at the failure
point, and separates a failed read from a failed write -- copying a payload that
another process still holds is a different problem from a full disk.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The copy loop's five distinct failures -- short read/write, an empty source, a
failed fsync, a failed close, and a failed directory sync -- shared one branch
and one unnamed IO status. Exhaustive labelling identified this as the path the
Windows launcher guard's immediate reinstall takes; this names which of the five
actually fires.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fourteen paths returned CBM_ACTIVATION_TRANSACTION_IO without setting a refusal
note, so the caller could only ever print 'status -3, os 0' -- an I/O error with
no cause. Identifying which one fires cost a ten-minute Windows guard run per
hypothesis, and two targeted guesses missed.
Labels every site so a single run names the path. Temporary labels; once the
guard's path is known they become descriptive.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
activation_transaction_prepare refused an insecure target directory and
returned IO with no note, while destroy() on the way out clobbered
GetLastError -- so the caller printed 'staging transaction open failed (status
-3, os 0)': an I/O error with neither a cause nor an OS code.
That is the path the Windows launcher guard's immediate reinstall takes, and the
missing note is why each hypothesis about it cost a ten-minute guard run. A
target directory failing the exact-owner rule is a different bug from an
unreadable source, and now says so.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three distinct refusals -- an insecure source directory, a failed snapshot, and
a missing source file -- collapsed into one boolean, and the frees on the way
out clobber GetLastError, so the caller reported 'staging transaction open
failed (status -3, os 0)': an I/O error with no cause and no refusal note.
The Windows launcher guard's immediate-reinstall step fails through exactly this
path, and with no cause reported each hypothesis cost a full ten-minute guard
run. Captures the OS error before the frees and names the predicate that
refused.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The guard reported only a version banner, so a silent nonzero exit was
indistinguishable from a rejected precondition. The install command works in
isolation and across a plain uninstall-then-reinstall, so the exit code is the
remaining unknown -- a crash code would say so directly.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Neither the fresh portable pair nor the ownership stamp fixed the immediate
managed reinstall, so both were guesses and do not belong on a release-blocking
PR. The guard's failure is pre-existing on main -- byte-identical output there,
and unchanged with the #581 fix disabled -- so this branch carries no regression
and no speculative test edits either.
What the investigation did establish: the reinstall's install command fails with
'managed install requires a verified adjacent codebase-memory-mcp.exe launcher'
when run directly, and emits no diagnostic on the stream the guard captures,
which is why the guard reports only a version banner.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The adjacent launcher was present but refused as unverified: copied files
inherit Administrators ownership on runner images, and the launcher's
exact-owner validators require the current user, which is what
stamp_fixture_owner_current exists to simulate. Without it the guard failed on
the product's ownership check rather than on the immediate-reinstall contract.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The immediate-reinstall guard reused the portable pair from
work/portable-install, whose adjacent launcher the earlier managed installs
consume. Managed install then refuses correctly -- 'requires a verified adjacent
codebase-memory-mcp.exe launcher' -- so the guard was failing on the product
behaving as designed rather than on the contract it protects. Reproduced by
running the install command directly (exit 1, that exact error) and confirmed
identical on origin/main, so this was never a regression from this branch.
Stages its own pair, leaving the assertions untouched.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Measurement retired the theory it was built on: the slab installs correctly now
(config_failed=0) and the leak was unchanged, because the cause was thread
heaps never being released rather than page-cache fragmentation.
It also runs at the top of main() for every process, including the launcher,
and the managed reinstall guard went red. A change that fixes nothing it
claimed does not get to stay on a release-blocking PR while a Windows guard is
failing. cbm_store_configure_pagecache remains available for a future attempt
wired ahead of sqlite3_initialize deliberately.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
These were added under the theory that abandoned thread-heap pages needed
collecting by hand. The TLS detach callback now releases each thread's heap
properly, so an explicit mi_collect at every job and connection exit buys
nothing and adds allocator teardown work on paths that were crashing a parallel
Windows ARM64 shard (daemon_application, test-par exit 2 with the suite summary
otherwise clean).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reverts 3cd1d33. It was built on the theory that per-connection thread-heap
churn drove #581, and the measurements later falsified that: growth scaled with
requests rather than threads, and the real cause was thread heaps never being
released at all, fixed separately by the TLS detach callback.
It also broke a Windows regression guard -- test_daemon_stability
section_cold_storm, 'a fixed Windows bug is broken again' -- which is what
concurrent clients racing to spawn a daemon do to reworked reaping and shutdown
ordering. A concurrency rewrite that fixes nothing it claimed and reopens a
closed bug does not belong in a release-blocking PR.
The allocator thread-heap fix is untouched.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reading the allocator's per-thread bookkeeping races a thread exiting
concurrently: even mi_theap_get_default() reads state that _mi_thread_done
rewrites via _mi_theap_default_set, and TSan paired those two directly with the
read coming from the periodic diagnostics thread.
Splits the collector. cbm_mem_map_collect_os() fills OS totals only and is what
the periodic snapshot uses; the walking variant stays for tests and explicitly
requested diagnostics, where no other thread is exiting. What the OS-only form
cannot see is carried by the residual, as mem.h promises.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Calling mi_heap_main() is itself the race: it reads the allocator's main-heap
pointer while an exiting thread rewrites it via _mi_thread_done ->
_mi_theap_default_set. TSan paired those two directly on macOS, with the read
coming from the diagnostics thread. Removing the aggregate walk was not enough
because the abandoned-page walk still passed mi_heap_main() as its argument.
The map now touches only this thread's own theap. Everything it can no longer
see is carried by the residual, which is what mem.h promises.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Walking mi_heap_main() reads the process-wide aggregate heap while other
threads allocate into it, which is a data race by construction; TSan reported
it on macOS as init.c:452 in mi_heap_main. A diagnostic must not introduce a
race into the code it measures.
Now walks only what this thread may safely read: its own theap, plus abandoned
pages, which have no owning thread left to race with. The lost coverage is
carried by the residual, which is what it is for -- an unmeasured map must
never read as an empty one.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The shim exists only so the allocation profiler has an observation point on
Linux. Applying its --wrap flags to the test and TSan link redirected malloc
into mimalloc underneath ASan's own interception, putting two allocators on the
same pointers. Production keeps the shim; sanitized builds do not. Windows is
unchanged -- there the wrap flags are what make mimalloc own allocations.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three CI failures, one cause each:
- Vendored integrity flagged a MISMATCH on the Windows mimalloc primitives.
The decommit counters added there were diagnostic and have served their
purpose -- 16k calls, zero failures, which retired the theory that purge was
silently failing. Restored to pristine, and the census fields they fed are
gone with them.
- GCC rejected an anonymous enum declared inside a struct ('declaration does
not declare anything'); clang accepted it, so it only broke the Ubuntu and
Windows smoke builds. Hoisted to file scope.
- clang-format across the files touched this session.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The probe assumed the walk would attribute over half of a 12 MB allocation.
mimalloc v3 exposes only the main heap, abandoned pages and the calling
thread's theap -- there is no API to enumerate every theap -- so on Windows the
probe's blocks are unreachable through all three and the walk saw ~190 KB.
Asserts the triple documented in mem.h instead: what the walk cannot see, the
residual must carry. Either attribution covers the probe or the committed total
grew by at least as much, so memory can never go missing from the map
unnoticed. Bucket attribution is checked where the walk did reach the probe.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel