cppcheck blocked the release run, correctly:
src/main.c:283: Condition '!main_test_worker_project_lock_marker(mutation)' is always false
src/main.c:2049: Condition '!worker_start_watchdog_test_descendant()' is always false
Compiling the seams out left behind stubs that returned true unconditionally, so
negating them was provably constant. Refactored rather than suppressed (repo
rule): the stubs are deleted and the CALL SITES are guarded, so a release build
does not contain the call at all -- which is what "the seam is not in the
artifact" should have meant in the first place.
The worker path needed care. The probe was one term in a five-term `||` chain,
and its POSITION is load-bearing: it must fork AFTER the isolated process group
exists (it inherits that group) and BEFORE the parent-death watchdog thread
starts, because forking a multithreaded process is the bug that ordering avoids.
Hoisting the call out of the chain would compile, pass tests, and silently break
that invariant. So the chain is split into three ordered steps -- process group,
then (test builds only) the probe, then the watchdog -- sharing one
worker_containment_unavailable() failure path so every step still fails
identically, with write()/_exit() rather than stdio and atexit handlers.
Verified in BOTH directions, since a guard that is always off is as wrong as one
always on: a TEST_SEAMS=1 build still contains the probe and
tests/test_worker_watchdog.sh passes against it, while the release build carries
no seam string. `make -f Makefile.cbm lint-ci` now passes locally -- the step
skipped before the last push, and the reason CI found this instead of me.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The update handoff moved out of the product on every platform, but two loose
ends were left behind.
README was actively WRONG, not merely incomplete. It still documented in-place
updates for macOS/Linux, and it promised "The MCP server also checks for updates
on startup and notifies on the first tool call" -- a claim that died with the MCP
update thread and then the daemon's curl check. The section now states the one
contract that holds everywhere (`update` validates flags and prints the install
script command), why it works that way (on Windows a running image cannot
replace itself; on POSIX a deliberate choice, because an in-process updater is
structurally a downloader), and the affirmative property that replaced it: cbm
makes no network request of its own accord and the archives carry no download
URLs at all.
`uninstall` now REPORTS the install script it finds beside the binary, printing
the path and the `rm` command, and says "would leave" under --dry-run. It does
not delete it. install.sh places itself there so `update` has something to hand
off to, but that file may equally be the user's own copy, a symlink into a
checkout, or package-manager managed -- an uninstaller must not delete a file it
cannot prove it owns, and one printed line leaves the decision where it belongs.
Windows checks install.ps1 too and tolerates either path separator.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Microsoft's ML flagged the rc.1 release binaries. The decisive evidence is that
the SAME sha256 went from 0/62 clean to Microsoft-detected in about an hour with
no byte change, so the verdict lives partly in scanner-side state and no code
change can promise a clean result. What code CAN do is stop shipping things that
have no business in a release artifact, which is worth doing on its own merits
and incidentally widens the classifier margin. Every claim below is verified
against a built binary by the new gate, not by reading source.
Executable stack (the worst of the findings). vendored/nomic/code_vectors_blob.S
is the only assembly in the build and carried no .note.GNU-stack. An unannotated
object makes ld assume the worst for the whole link, so EVERY Linux release we
have ever shipped had GNU_STACK RWE. Adds the note (cause) plus ELF-only
-Wl,-z,noexecstack (outcome); the gate fails the release if it returns.
Test seams are now opt-in, never opt-out. TEST_SEAMS=1 defines
CBM_ENABLE_TEST_SEAMS; without it the crash-orphan probe -- which forks a child
that ignores SIGTERM and loops forever, then writes its pid to a caller-supplied
path -- and the lease-ownership marker compile to trivial stubs, so call sites are
untouched and the binary holds no fork, no signal handler and no env-var string.
Opt-IN is the point: forgetting the flag yields a clean binary rather than a leaky
one. scripts/test.sh requests it in the leg that consumes it, and
tests/test_worker_watchdog.sh now asserts the capability up front instead of
dying later with an opaque "Killed: 9".
The daemon's background version check is gone. It spawned curl against
api.github.com/repos/.../releases/latest on the first eligible session of every
run to say "a newer version exists" -- a release URL and an outbound request in
every shipped binary, for something the install scripts already report. The
INJECTABLE SEAM survives: update_ops is still honoured, the fakes in
tests/test_daemon_application.c still cover notice/ownership/cancellation/replay,
and with no provider application_update_subscribe_locked returns early so no
generation ever starts. "No network request by default" is now structural.
Dead capability out of release builds. The tar.gz/zip extraction block
(gzip_decompress through cbm_extract_binary_from_zip, plus its cli.h
declarations) moves under CBM_CLI_ENABLE_TEST_API -- verified self-contained, zero
uses of any helper outside it, only callers the excluded updater and
tests/test_cli.c. Downloading an archive, decompressing it, picking an executable
out of it and marking it executable is the canonical dropper composite; it is now
absent rather than merely unreachable. SQLite is built with
-DSQLITE_OMIT_LOAD_EXTENSION (no caller of load_extension anywhere in src/ or
internal/), removing that API surface and part of the dlopen/dlsym surface.
Temp files and environment scanning (S2/S3). Predictable paths in mcp.c,
artifact.c and diagnostics.c are created privately and exclusively and written
through the returned descriptor; pass_envscan.c no longer descends symlinked
directories out of the project root, and its fixed 512-byte path buffers no
longer truncate into pointer arithmetic that could land outside the buffer.
Build-time entropy. mimalloc's version banner baked __DATE__/__TIME__ into every
binary, so two builds of identical source seconds apart could never share a hash
and no release could inherit a false-positive determination made about its
predecessor. Local patch removes it (marked to survive refreshes), -Wdate-time
makes any future use a build error, and -Wl,--no-insert-timestamp stops the PE
header carrying the link clock.
scripts/ci/check-binary-composition.sh is the proof that each removal stays
removed, wired into package-release.sh after strip so the local artifact-flow
smoke enforces exactly what CI does. It asserts absences plus a CANARY string, so
handing it a compressed, stubbed or empty file fails instead of passing
vacuously, and a missing tool is a hard error -- a skipped assertion must never
look like a satisfied one.
Two build-system traps found by that gate, both of which had silently defeated a
fix: the product binary is compiled in one shot from sources, so a flag flip did
not rebuild it (now tracked by a .build-config stamp that also removes the
binary, making it independent of mtime granularity); and prod_sqlite3.o /
prod_mimalloc.o depended on a single named source, so SQLITE_OMIT_LOAD_EXTENSION
and the mimalloc patch BOTH compiled to nothing on the first incremental build.
Source review would have called them done.
Deliberately NOT changed. Three seams stay in release artifacts because
scripts/smoke-test.sh runs against the real artifact and needs them:
CBM_TEST_CRASH_ON and CBM_TEST_HANG_ON inject the faults that prove supervisor
recovery, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID is what keeps the PATH smoke from
writing the tester's actual PATH. The gate treats those as an allowlist, so a
NOVEL seam still fails. The true no-UI standard build is deferred rather than
rushed: src/ui/* is in PROD_SRCS and four files outside src/ui reference UI
symbols, including the daemon that serves the UI, so that assertion reports
instead of failing until the split lands -- a gate everyone knows is red teaches
people to ignore gates.
No grammar is removed. ObjectScript accounts for essentially all binary growth
since the last provably-clean release (+21.4MB rodata, +1.1MB text from two
four-line shims), which made it the obvious ablation candidate, but a dry run
performed twelve real Defender endpoint scans across standard/UI and amd64/arm64
with ObjectScript, the daemon and the expanded hooks all present and every scan
was clean. Nothing there is a deterministic trigger, so cutting a
community-contributed language would spend a real feature on unproven margin.
Lean is not a candidate either: at 99.6MB of source it is by far the largest
grammar, but it shipped in v0.9.0 which scanned 20/20 clean, so removing it would
produce a novel unscanned profile instead of restoring a known-good one.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the move started on Windows. `update` prints the install script's
command and exits 0 on every platform; the download/extract/chmod/exec sequence
is compiled out of release builds entirely, and the MCP background thread that
ran cbm_popen("curl ... api.github.com ...") on first tool call is gone.
The installers now place themselves beside the binary, which closes the last
gap: `update` referenced a raw.githubusercontent URL purely because an
install.sh-installed user had no local copy to point at. Two details make that
safe.
The source is the install script from the archive we JUST checksum-verified,
never "$0" -- which does not exist under `curl | bash`, and would pin the user
to the OLD installer forever. Since the script ships inside the verified
archive, it inherits that verification instead of needing its own.
And it is published by atomic rename, never written over the live path. Bash
reads a script incrementally by byte offset, so overwriting the file it is
executing continues reading NEW bytes at the OLD offset: silent, bizarre
corruption. cp to a temp name then mv -f swaps the directory entry while the
running shell keeps its original inode. Windows follows the same rule via
Copy-Item + Move-Item even though PowerShell parses up front.
Deliberately NOT done: spawning the new installer to delete and replace its
predecessor. Fetch remote code -> drop to disk -> spawn -> self-delete is a
textbook stager, and self-deletion is among the most heavily weighted
heuristics there is. It also gains nothing -- `"$DLBIN" install` already runs
the NEW release's install logic, because the binary owns the install.
Net: zero install or download URLs remain in the shipped binary, and the
allow-list entry is retired with it. Both scripts scan CLEAN (0/61) with the
self-placement code in them.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The `update` handoff prints a curl one-liner when install.sh is not beside the
binary, and the security audit blocked the release on it:
BLOCKED: src/cli/cli.c: URL not on allow-list: https://raw.githubusercontent.com/DeusData/
Two things were wrong. The URL was not declared, and it was split across string
literal continuations, so the audit extracted only the first fragment -- an
allow-list entry for the real URL could never have matched it. Hoisted to a
single CBM_INSTALL_SH_URL token so the declaration means what it says.
It is the same URL as the documented one-liner install, and it is printed for
the user to paste: the binary no longer downloads anything.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Windows already handed updating to install.ps1, because a running .exe cannot
replace its own image. The remaining platforms now do the same, for a different
and better reason.
An in-process updater is structurally a downloader: it fetches a remote archive,
verifies it, extracts it, marks the result executable and runs it. That composite
lived in every shipped artifact -- along with the release URLs it fetched -- to
serve a command most users run a handful of times. The install scripts already
do all of it, are idempotent (so re-running one IS the update), and run while cbm
is not running.
`update` now prints the exact command for the running platform and exits 0:
unix bash "<dir>/install.sh" (or the curl one-liner when not adjacent)
windows powershell -ExecutionPolicy Bypass -File "<dir>\install.ps1"
Both scripts ship inside their release archive, so the printed path resolves
next to the binary. Flags are still parsed and validated, so `update --dry-run
--ui` keeps rejecting typos rather than silently accepting them.
The updater itself is EXCLUDED from release builds rather than left to dead-code
elimination: build_update_url, download_verify_install, extract_and_install_binary,
checksum fetch, detect_os/detect_arch and the cli_download_* helpers now compile
only under CBM_CLI_ENABLE_TEST_API, which release builds do not define. The C
suite still covers the flow through the activation test seam. Verified against
the release binary: `releases/latest/download` is gone entirely.
Also removes the MCP background update check. It ran on the first tool call and
did:
cbm_popen("curl -sf --max-time 5 -H 'Accept: application/vnd.github+json' "
"'https://api.github.com/repos/.../releases/latest' 2>/dev/null")
An embedded shell command plus a startup network callout, in every session, to
tell users a newer version exists. It shelled out, depended on curl being
installed, and phoned home from every agent session. The install scripts report
versions; this did not need to be in the server.
Smoke follows the same collapse. Phases 6c and 14 asserted a Windows-specific
handoff and a POSIX in-process replacement; they now assert one platform-neutral
contract selected by UPDATE_SCRIPT, and 14a's byte-identity check compares the
driver against ITSELF before and after -- comparing against "$BINARY" passed on
Windows only because Windows has no signing step, while the POSIX fixture
ad-hoc re-signs its copy and so differed before `update` ever ran.
The Linux glibc guard is RELOCATED, not retired. The standard linux asset links
glibc 2.38+ and breaks Debian 11, RHEL 8 and Ubuntu 20.04, so the installer must
fetch the static "-portable" build. That constraint moved from the binary to
install.sh, and the smoke now guards it where the behaviour lives.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Release run 30385988475 lost three Windows smoke jobs to one phenomenon on two
different surfaces. A Windows file that was just written or just executed can
refuse deletion and rename for a moment while a scanner reads it or while the OS
finishes reaping a child. Both conditions clear on their own within moments.
It reads as nondeterminism because it is timing: windows-2025 passed both amd64
variants while windows-latest -- the SAME image, windows-2025-vs2026 -- failed
both, and windows-11-arm failed too.
PRODUCT: uninstall abandoned a live installation.
With one binary per platform, uninstall must remove the very image it is
executing, and on Windows a running image can only be renamed, never
overwritten. That rename got a single attempt, so a transient sharing violation
surfaced to the user as
error: failed to remove ...; completed configuration/index cleanup may remain
with the install left half-torn-down. install.ps1 already retries this exact
operation ten times for this exact reason; the C path now does the same, and
only for sharing/access/lock violations, so a file that is genuinely held still
fails closed instead of spinning.
Proven with a failure-injection seam rather than by racing a real scanner:
3 injected failures must still succeed, 64 must still fail AND leave the target
in place. Deterministic, so it does not need CI to learn whether it works.
HARNESS: the smoke killed itself silently.
Phases that install a binary then `rm -rf` the directory hit the same lock. Under
`set -euo pipefail` that returns non-zero and ends the run WITHOUT printing
anything -- the logs jump straight from "OK 13h" to job cleanup with no FAIL
line, which is what made this expensive to read.
None of those 16 removals is an assertion; they are fixture cleanup, and
_smoke.yml already states the principle ("a temp dir on an ephemeral runner must
never fail the job") that smoke-test.sh did not honour. They now go through
smoke_rmtree, which retries so the disk is actually reclaimed -- these fixtures
hold ~300 MB binaries and runners are disk-tight -- then warns and continues.
Verified: Windows VM full smoke ALL PASSED (previously died after 13h), macOS
full smoke ALL PASSED, activation_transaction + cli 259 passed on Windows, all
10 shell contracts pass, lint-ci clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two Windows tests needed the startup thread to be observed HOLDING the startup
lock, and looked for it by polling the lock for BUSY up to 200 times. That state
has no lower bound: whenever the rendezvous handoff does not block, acquire ->
handoff -> release completes in microseconds, so the window can close between
two polls and startup_observed asserts on a coin flip. This blocked release run
30374135923 on test-windows (windows-2025, CLANG64, x86_64).
Budget-tuning the poll was tried and rejected -- a wider budget only lowers the
failure rate, and a transient window is never a fixed test. Instead the
production path exposes a gate that fires once the startup lock is held and
before the handoff runs, matching the hook pattern already used for POSIX
publication. The test parks the startup thread there and releases it when it is
ready, so the interleaving is pinned by construction. The acquired flag is
monotonic, so the observation cannot be missed; the remaining bound only catches
a thread that never started at all. The gate is inert unless a test installs it.
Also fixes an unrelated use-after-free this exposed. ipc_forever_wait_server
closed the connection and NULLed it from the server thread, while the test was
about to call cbm_daemon_ipc_connection_interrupt on the pointer it had already
loaded -- and receive_frame can return on its own before that call lands. It
crashed roughly one Windows run in five, independent of the change above.
Ownership now stays with the test, which closes after joining, so interrupt can
only ever see a live connection.
Verified on real Windows: 5 consecutive daemon_ipc runs, no failure and no
crash. macOS 46 passed x5 under ASan/UBSan; Linux 6606 passed, 0 failed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Windows smoke guards that an invalid PATH seam must fail closed rather than
silently falling back to the live HKCU\Environment\Path. That guard went green
in a way that meant nothing: the seam still refused, and install still printed
"PATH configuration failed", but the process exited 0.
Windows release builds used to compile a separate managed-install path. With one
binary per platform they compile the shared cbm_cmd_install, which since 30b5f12
downgraded ANY failed dry-run plan check to CLI_OK. That commit was fixing a real
complaint -- the dry-run summary was being skipped entirely on Windows -- but it
bundled the exit status into the fix. Keeping the summary is right; reporting
success is not. --dry-run exists to answer "would this install work?", so a
caller that only sees the exit code could not tell a refused PATH probe from a
clean plan.
The summary and the triage note still print; only the status changes. A real
install already failed non-zero here, so the fail-closed behaviour itself was
never at risk -- just its observability.
The smoke also still encoded the launcher-era portable-vs-managed split, which
does not survive one-binary-per-platform:
* 6b expected uninstall to be REFUSED. That refusal existed because a portable
extracted bundle was a different artifact from the launcher-managed install
it would have torn down. The extracted binary now IS the installed one.
* 6c expected update to be REFUSED, contradicting the contract Phase 14a
asserts against a real update: Windows hands off to install.ps1, exits 0 and
prints the command. 6c now pins that same handoff.
* 14a compared binaries with cmp, which is absent from the Windows MSYS shell,
so it reported "different" for two copies of one file. Hashes instead, the
way Phase 12 already verifies the release archive.
* 14f seeded the MCP entry at the RETIRED binary. The old launcher-managed
update rewrote that entry in-process; the handoff does not, so uninstall
correctly declined to remove an entry owned by a different installation and
the phase demanded the one thing it must never do. It now seeds the
installed binary, which is what install.ps1 leaves a real user holding.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dry run 30206811293: macos-15-intel release smoke failed B3 with a
visibly correct 'semantic: 50' table — BSD grep returns no-match for
the ENTIRE output when any line carries a NUL, control byte, or
invalid UTF-8 (verified: printf poisoned input -> grep -qE fails), so
one bad byte in one row silently unmatches every anchored check.
Close the class at the emission choke point: control bytes and
invalid UTF-8 force the quoted form; quoting escapes controls as
\u00XX and replaces invalid sequences with U+FFFD (RFC 3629
validation, NUL-safe). Tool output is valid UTF-8 by construction.
Unit test proven RED-on-revert (raw bytes emitted) and green with the
fix. The smoke's B3 failure path now also dumps od -c bytes so a
future red names the exact byte.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dry run 30206811293: windows-11-arm (both variants) and windows-2025
smoke failed in the update/uninstall E2E phases with 'private staged
copy failed secure open (error 32)' — ERROR_SHARING_VIOLATION. On the
newer hosted images a first-touch scanner transiently holds the
just-staged executable without FILE_SHARE_DELETE, so the launcher's
GENERIC_READ|DELETE retirement open collides on cold images.
Absorb the scan window with the same bounded backoff the staged
RENAME path already uses (<=~2.5s, exponential); every retry re-runs
the full secure-open validation, so no TOCTOU surface changes.
No deterministic local repro exists (the VM passes this phase with
Defender active; the collision is image-specific) — the next dry run
on the same three jobs is the arbiter.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This reverts commit f843db8, which removed the #581 fix on a false
attribution. The arm64 shards were crashing because of an unaligned SQLite
page-cache slab (fixed in #1274, arm64 shard 2/2 green since); removing this
callback never changed those crashes, which is what exonerated it. The Windows
soak then failed for the obvious reason: the leak was back.
Static MinGW links have no DllMain and register no TLS callback, so
mi_thread_done never runs and every thread leaks its heap -- 607 heaps after 300
requests holding 170 MiB against a ~300 KiB live set. With the callback the
10-minute Windows soak measured 1.1x against the original 53x.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The slab was a static char array handed to SQLITE_CONFIG_PAGECACHE. A static
char array carries no alignment guarantee and SQLite requires 8-byte-aligned
page-cache memory: x86_64 tolerates the unaligned access, arm64 faults. Every
suite that opens a store died instantly on windows-11-arm with
STATUS_ILLEGAL_INSTRUCTION (pass=0, secs=0) while pure-logic suites in the same
wave passed, and both arm64 shards were green on the commit before the merge.
It was also never the fix it was added for: measurement showed the slab
installed correctly and #581's growth unchanged, because the cause was thread
heaps never being released. Removing it rather than aligning it -- an unverified
optimisation has no claim on a release gate.
Note PR CI does not build the windows-11-arm leg, so this class of defect can
only surface in the dry run.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The refusal notes changed the error text the Windows launcher guards match
against, and test-windows-guards went red on this PR after they landed. They were
investigation instrumentation and they succeeded -- they turned 'staging
transaction open failed (status -3, os 0)' into ERROR_DISK_FULL and named the
exact site -- but a diagnostic that shifts a matched error string does not belong
in a release-blocking change.
Worth redoing deliberately on its own, with the guards' expectations updated in
the same commit.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The hosted windows-11-arm shards crash at process load (STATUS_ILLEGAL_INSTRUCTION,
pass=0, secs=0) while main passes them, and removing the TLS detach callback did
not help -- so the remaining candidate is this layer, which touches every
allocation on Windows through the interposer. It declares a _Thread_local guard
read on every malloc, and thread-local storage in a static MinGW aarch64 link is
exactly the fragile case; nothing in it needs to ship.
The profiler did its job: it produced the attribution that identified #581's
mechanism, and that analysis is recorded. Sources stay in the tree, unbuilt, so
a future investigation can re-enable them deliberately.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reverts the #581 fix mechanism, not the diagnosis.
main's dry run passes both windows-11-arm shards and test-windows-guards; this
branch fails all three. The only change here that affects process LOAD is the
.CRT$XLB detach callback, and the arm64 failures are load-time crashes
(STATUS_ILLEGAL_INSTRUCTION, pass=0, secs=0, five suites in the same second) --
while the guards exercise many launcher and payload spawns. The same binary
loads fine on a UTM arm64 VM, so the placement is tolerated by some loaders and
not others, which is not something to ship on a supported platform.
#581's root cause stands and is documented: a static MinGW link has no DllMain
and registers no TLS callback, so mi_thread_done never runs and every thread
leaks its heap (607 heaps after 300 requests, 170 MiB against ~300 KiB live).
The delivery mechanism needs to be one the loader accepts everywhere -- an FLS
callback registered at runtime is the obvious candidate, with no linker section
placement involved.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The copy loop's failure was reported as 'os 0' because the cleanup and close
calls after the loop clobber the thread's last error. Captures it at the failure
point, and separates a failed read from a failed write -- copying a payload that
another process still holds is a different problem from a full disk.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The copy loop's five distinct failures -- short read/write, an empty source, a
failed fsync, a failed close, and a failed directory sync -- shared one branch
and one unnamed IO status. Exhaustive labelling identified this as the path the
Windows launcher guard's immediate reinstall takes; this names which of the five
actually fires.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fourteen paths returned CBM_ACTIVATION_TRANSACTION_IO without setting a refusal
note, so the caller could only ever print 'status -3, os 0' -- an I/O error with
no cause. Identifying which one fires cost a ten-minute Windows guard run per
hypothesis, and two targeted guesses missed.
Labels every site so a single run names the path. Temporary labels; once the
guard's path is known they become descriptive.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
activation_transaction_prepare refused an insecure target directory and
returned IO with no note, while destroy() on the way out clobbered
GetLastError -- so the caller printed 'staging transaction open failed (status
-3, os 0)': an I/O error with neither a cause nor an OS code.
That is the path the Windows launcher guard's immediate reinstall takes, and the
missing note is why each hypothesis about it cost a ten-minute guard run. A
target directory failing the exact-owner rule is a different bug from an
unreadable source, and now says so.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three distinct refusals -- an insecure source directory, a failed snapshot, and
a missing source file -- collapsed into one boolean, and the frees on the way
out clobber GetLastError, so the caller reported 'staging transaction open
failed (status -3, os 0)': an I/O error with no cause and no refusal note.
The Windows launcher guard's immediate-reinstall step fails through exactly this
path, and with no cause reported each hypothesis cost a full ten-minute guard
run. Captures the OS error before the frees and names the predicate that
refused.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Measurement retired the theory it was built on: the slab installs correctly now
(config_failed=0) and the leak was unchanged, because the cause was thread
heaps never being released rather than page-cache fragmentation.
It also runs at the top of main() for every process, including the launcher,
and the managed reinstall guard went red. A change that fixes nothing it
claimed does not get to stay on a release-blocking PR while a Windows guard is
failing. cbm_store_configure_pagecache remains available for a future attempt
wired ahead of sqlite3_initialize deliberately.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
These were added under the theory that abandoned thread-heap pages needed
collecting by hand. The TLS detach callback now releases each thread's heap
properly, so an explicit mi_collect at every job and connection exit buys
nothing and adds allocator teardown work on paths that were crashing a parallel
Windows ARM64 shard (daemon_application, test-par exit 2 with the suite summary
otherwise clean).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reverts 3cd1d33. It was built on the theory that per-connection thread-heap
churn drove #581, and the measurements later falsified that: growth scaled with
requests rather than threads, and the real cause was thread heaps never being
released at all, fixed separately by the TLS detach callback.
It also broke a Windows regression guard -- test_daemon_stability
section_cold_storm, 'a fixed Windows bug is broken again' -- which is what
concurrent clients racing to spawn a daemon do to reworked reaping and shutdown
ordering. A concurrency rewrite that fixes nothing it claimed and reopens a
closed bug does not belong in a release-blocking PR.
The allocator thread-heap fix is untouched.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reading the allocator's per-thread bookkeeping races a thread exiting
concurrently: even mi_theap_get_default() reads state that _mi_thread_done
rewrites via _mi_theap_default_set, and TSan paired those two directly with the
read coming from the periodic diagnostics thread.
Splits the collector. cbm_mem_map_collect_os() fills OS totals only and is what
the periodic snapshot uses; the walking variant stays for tests and explicitly
requested diagnostics, where no other thread is exiting. What the OS-only form
cannot see is carried by the residual, as mem.h promises.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Calling mi_heap_main() is itself the race: it reads the allocator's main-heap
pointer while an exiting thread rewrites it via _mi_thread_done ->
_mi_theap_default_set. TSan paired those two directly on macOS, with the read
coming from the diagnostics thread. Removing the aggregate walk was not enough
because the abandoned-page walk still passed mi_heap_main() as its argument.
The map now touches only this thread's own theap. Everything it can no longer
see is carried by the residual, which is what mem.h promises.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Walking mi_heap_main() reads the process-wide aggregate heap while other
threads allocate into it, which is a data race by construction; TSan reported
it on macOS as init.c:452 in mi_heap_main. A diagnostic must not introduce a
race into the code it measures.
Now walks only what this thread may safely read: its own theap, plus abandoned
pages, which have no owning thread left to race with. The lost coverage is
carried by the residual, which is what it is for -- an unmeasured map must
never read as an empty one.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three CI failures, one cause each:
- Vendored integrity flagged a MISMATCH on the Windows mimalloc primitives.
The decommit counters added there were diagnostic and have served their
purpose -- 16k calls, zero failures, which retired the theory that purge was
silently failing. Restored to pristine, and the census fields they fed are
gone with them.
- GCC rejected an anonymous enum declared inside a struct ('declaration does
not declare anything'); clang accepted it, so it only broke the Ubuntu and
Windows smoke builds. Hoisted to file scope.
- clang-format across the files touched this session.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A static MinGW link has no DllMain and registers no TLS callback, so nothing
ran at thread exit and every thread left its mimalloc thread-heap behind for
the life of the process. After 300 MCP requests the allocator reported 607
thread-heaps, peak equal to total equal to current -- none ever released --
holding 170.5 MiB against a live set of roughly 300 KiB, growing linearly and
without bound (#581).
POSIX is unaffected because a pthread TSD destructor already calls this, which
is precisely why POSIX stayed flat under the identical workload. The heaps were
invisible to mi_heap_visit, which only walks the main and calling heaps, so the
growth appeared to belong to no allocator at all and survived every allocator
option that was tried against it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Growth is invariant to eleven allocator configurations, so the memory is not
the allocator's. The size profile of what accumulates distinguishes a swarm of
small commits from thread-stack-sized reservations.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The ARM64 fallback captured a return address inside the capture helper, so
every allocation collapsed onto the handful of hook addresses and the site
column named the profiler instead of the caller. Each hook now supplies its own
return address. The census also reports allocation volume, which separates
'the profiler sees nothing' from 'it sees traffic that is all freed'.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The hooks run inside malloc/free, so reading the env gate — which allocates —
re-entered them and recursed until the stack died. The process then produced no
stdout, no stderr and no log, which reads as the server refusing to start rather
than as a profiler defect. A thread-local guard makes the profiler invisible to
itself; the non-recursive mutex would also have deadlocked here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fixed corpus, fixed request count, one process, no daemon handshake and no
reindex or crash-recovery phases, so two runs differ only where the code does.
Requests go through a request/response driver rather than a batched pipe:
closing stdin makes the server treat it as a client disconnect and shut down
before answering, which yields a census of a process that never worked.
Linux gains a measurement-only --wrap shim so the profiler can observe
allocations the way the Windows interposer already does; macOS has no --wrap
and stays census-only, which the driver reports rather than passing off as
zero.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The census could say which POOL held memory but never which allocation put it
there, so each hypothesis needed a code change plus an eight-minute soak to
test and the investigation could only ask one yes/no question at a time.
Records allocations at or above a threshold (default 32 KiB, since arena
fragmentation is driven by large transient runs) against their captured stack,
into fixed tables whose exhaustion is counted and reported rather than silently
dropped. Emitted alongside every census sample, so pool totals and attribution
always describe the same instant, plus an explicit unattributed remainder so
incomplete coverage announces itself.
scripts/memlab-report.py ranks sites by retained-byte growth using a linear fit
and can diff two platforms' profiles against each other.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
SQLITE_CONFIG_PAGECACHE is only accepted before sqlite3_initialize, so calling
it from the store open paths was always refused (rc=misuse) and every
request-scoped store fell back to allocating its page cache from the general
allocator. Each request therefore took a fresh multi-megabyte run of arena
slices and left the previous one fragmented behind mimalloc's commit frontier,
which is the per-request retention behind #581.
Hoisted to process init, immediately after the allocator binding and before
anything can open a store. The existing call sites stay as an idempotent
backstop.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Each connection created its own OS thread, so the allocator minted a fresh
thread-heap every time. mimalloc then took new slices at its arena commit
frontier for each heap and never walked back to the purged ones behind it, so
Windows commit charge and RSS ratcheted upward for the life of the daemon
(#581). POSIX never showed it because madvise drops purged pages from RSS
regardless of where the frontier sits.
The worker slots were already pooled and bounded by max_clients; only the
threads were not. Each slot now starts its thread once and parks it on a
condition variable between connections, which keeps the set of allocator
thread-heaps constant. Slots are still released by the reaper as soon as a
connection finishes; threads are joined once, at shutdown.
Adds cbm_cond_t (pthread_cond_t / CONDITION_VARIABLE) since the pool needs a
wait primitive the codebase did not have.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel