The copy loop's failure was reported as 'os 0' because the cleanup and close
calls after the loop clobber the thread's last error. Captures it at the failure
point, and separates a failed read from a failed write -- copying a payload that
another process still holds is a different problem from a full disk.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The copy loop's five distinct failures -- short read/write, an empty source, a
failed fsync, a failed close, and a failed directory sync -- shared one branch
and one unnamed IO status. Exhaustive labelling identified this as the path the
Windows launcher guard's immediate reinstall takes; this names which of the five
actually fires.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fourteen paths returned CBM_ACTIVATION_TRANSACTION_IO without setting a refusal
note, so the caller could only ever print 'status -3, os 0' -- an I/O error with
no cause. Identifying which one fires cost a ten-minute Windows guard run per
hypothesis, and two targeted guesses missed.
Labels every site so a single run names the path. Temporary labels; once the
guard's path is known they become descriptive.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
activation_transaction_prepare refused an insecure target directory and
returned IO with no note, while destroy() on the way out clobbered
GetLastError -- so the caller printed 'staging transaction open failed (status
-3, os 0)': an I/O error with neither a cause nor an OS code.
That is the path the Windows launcher guard's immediate reinstall takes, and the
missing note is why each hypothesis about it cost a ten-minute guard run. A
target directory failing the exact-owner rule is a different bug from an
unreadable source, and now says so.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three distinct refusals -- an insecure source directory, a failed snapshot, and
a missing source file -- collapsed into one boolean, and the frees on the way
out clobber GetLastError, so the caller reported 'staging transaction open
failed (status -3, os 0)': an I/O error with no cause and no refusal note.
The Windows launcher guard's immediate-reinstall step fails through exactly this
path, and with no cause reported each hypothesis cost a full ten-minute guard
run. Captures the OS error before the frees and names the predicate that
refused.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The guard reported only a version banner, so a silent nonzero exit was
indistinguishable from a rejected precondition. The install command works in
isolation and across a plain uninstall-then-reinstall, so the exit code is the
remaining unknown -- a crash code would say so directly.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Neither the fresh portable pair nor the ownership stamp fixed the immediate
managed reinstall, so both were guesses and do not belong on a release-blocking
PR. The guard's failure is pre-existing on main -- byte-identical output there,
and unchanged with the #581 fix disabled -- so this branch carries no regression
and no speculative test edits either.
What the investigation did establish: the reinstall's install command fails with
'managed install requires a verified adjacent codebase-memory-mcp.exe launcher'
when run directly, and emits no diagnostic on the stream the guard captures,
which is why the guard reports only a version banner.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The adjacent launcher was present but refused as unverified: copied files
inherit Administrators ownership on runner images, and the launcher's
exact-owner validators require the current user, which is what
stamp_fixture_owner_current exists to simulate. Without it the guard failed on
the product's ownership check rather than on the immediate-reinstall contract.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The immediate-reinstall guard reused the portable pair from
work/portable-install, whose adjacent launcher the earlier managed installs
consume. Managed install then refuses correctly -- 'requires a verified adjacent
codebase-memory-mcp.exe launcher' -- so the guard was failing on the product
behaving as designed rather than on the contract it protects. Reproduced by
running the install command directly (exit 1, that exact error) and confirmed
identical on origin/main, so this was never a regression from this branch.
Stages its own pair, leaving the assertions untouched.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Measurement retired the theory it was built on: the slab installs correctly now
(config_failed=0) and the leak was unchanged, because the cause was thread
heaps never being released rather than page-cache fragmentation.
It also runs at the top of main() for every process, including the launcher,
and the managed reinstall guard went red. A change that fixes nothing it
claimed does not get to stay on a release-blocking PR while a Windows guard is
failing. cbm_store_configure_pagecache remains available for a future attempt
wired ahead of sqlite3_initialize deliberately.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
These were added under the theory that abandoned thread-heap pages needed
collecting by hand. The TLS detach callback now releases each thread's heap
properly, so an explicit mi_collect at every job and connection exit buys
nothing and adds allocator teardown work on paths that were crashing a parallel
Windows ARM64 shard (daemon_application, test-par exit 2 with the suite summary
otherwise clean).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reverts 3cd1d33. It was built on the theory that per-connection thread-heap
churn drove #581, and the measurements later falsified that: growth scaled with
requests rather than threads, and the real cause was thread heaps never being
released at all, fixed separately by the TLS detach callback.
It also broke a Windows regression guard -- test_daemon_stability
section_cold_storm, 'a fixed Windows bug is broken again' -- which is what
concurrent clients racing to spawn a daemon do to reworked reaping and shutdown
ordering. A concurrency rewrite that fixes nothing it claimed and reopens a
closed bug does not belong in a release-blocking PR.
The allocator thread-heap fix is untouched.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reading the allocator's per-thread bookkeeping races a thread exiting
concurrently: even mi_theap_get_default() reads state that _mi_thread_done
rewrites via _mi_theap_default_set, and TSan paired those two directly with the
read coming from the periodic diagnostics thread.
Splits the collector. cbm_mem_map_collect_os() fills OS totals only and is what
the periodic snapshot uses; the walking variant stays for tests and explicitly
requested diagnostics, where no other thread is exiting. What the OS-only form
cannot see is carried by the residual, as mem.h promises.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Calling mi_heap_main() is itself the race: it reads the allocator's main-heap
pointer while an exiting thread rewrites it via _mi_thread_done ->
_mi_theap_default_set. TSan paired those two directly on macOS, with the read
coming from the diagnostics thread. Removing the aggregate walk was not enough
because the abandoned-page walk still passed mi_heap_main() as its argument.
The map now touches only this thread's own theap. Everything it can no longer
see is carried by the residual, which is what mem.h promises.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Walking mi_heap_main() reads the process-wide aggregate heap while other
threads allocate into it, which is a data race by construction; TSan reported
it on macOS as init.c:452 in mi_heap_main. A diagnostic must not introduce a
race into the code it measures.
Now walks only what this thread may safely read: its own theap, plus abandoned
pages, which have no owning thread left to race with. The lost coverage is
carried by the residual, which is what it is for -- an unmeasured map must
never read as an empty one.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The shim exists only so the allocation profiler has an observation point on
Linux. Applying its --wrap flags to the test and TSan link redirected malloc
into mimalloc underneath ASan's own interception, putting two allocators on the
same pointers. Production keeps the shim; sanitized builds do not. Windows is
unchanged -- there the wrap flags are what make mimalloc own allocations.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three CI failures, one cause each:
- Vendored integrity flagged a MISMATCH on the Windows mimalloc primitives.
The decommit counters added there were diagnostic and have served their
purpose -- 16k calls, zero failures, which retired the theory that purge was
silently failing. Restored to pristine, and the census fields they fed are
gone with them.
- GCC rejected an anonymous enum declared inside a struct ('declaration does
not declare anything'); clang accepted it, so it only broke the Ubuntu and
Windows smoke builds. Hoisted to file scope.
- clang-format across the files touched this session.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The probe assumed the walk would attribute over half of a 12 MB allocation.
mimalloc v3 exposes only the main heap, abandoned pages and the calling
thread's theap -- there is no API to enumerate every theap -- so on Windows the
probe's blocks are unreachable through all three and the walk saw ~190 KB.
Asserts the triple documented in mem.h instead: what the walk cannot see, the
residual must carry. Either attribution covers the probe or the committed total
grew by at least as much, so memory can never go missing from the map
unnoticed. Bucket attribution is checked where the walk did reach the probe.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A static MinGW link has no DllMain and registers no TLS callback, so nothing
ran at thread exit and every thread left its mimalloc thread-heap behind for
the life of the process. After 300 MCP requests the allocator reported 607
thread-heaps, peak equal to total equal to current -- none ever released --
holding 170.5 MiB against a live set of roughly 300 KiB, growing linearly and
without bound (#581).
POSIX is unaffected because a pthread TSD destructor already calls this, which
is precisely why POSIX stayed flat under the identical workload. The heaps were
invisible to mi_heap_visit, which only walks the main and calling heaps, so the
growth appeared to belong to no allocator at all and survived every allocator
option that was tried against it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Growth is invariant to eleven allocator configurations, so the memory is not
the allocator's. The size profile of what accumulates distinguishes a swarm of
small commits from thread-stack-sized reservations.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The ARM64 fallback captured a return address inside the capture helper, so
every allocation collapsed onto the handful of hook addresses and the site
column named the profiler instead of the caller. Each hook now supplies its own
return address. The census also reports allocation volume, which separates
'the profiler sees nothing' from 'it sees traffic that is all freed'.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
SQLite and tree-sitter are bound directly to mi_malloc, so they never pass the
malloc interposer and the profile could not see them — which excluded the two
subsystems that allocate the most per request.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The hooks run inside malloc/free, so reading the env gate — which allocates —
re-entered them and recursed until the stack died. The process then produced no
stdout, no stderr and no log, which reads as the server refusing to start rather
than as a profiler defect. A thread-local guard makes the profiler invisible to
itself; the non-recursive mutex would also have deadlocked here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fixed corpus, fixed request count, one process, no daemon handshake and no
reindex or crash-recovery phases, so two runs differ only where the code does.
Requests go through a request/response driver rather than a batched pipe:
closing stdin makes the server treat it as a client disconnect and shut down
before answering, which yields a census of a process that never worked.
Linux gains a measurement-only --wrap shim so the profiler can observe
allocations the way the Windows interposer already does; macOS has no --wrap
and stays census-only, which the driver reports rather than passing off as
zero.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The census could say which POOL held memory but never which allocation put it
there, so each hypothesis needed a code change plus an eight-minute soak to
test and the investigation could only ask one yes/no question at a time.
Records allocations at or above a threshold (default 32 KiB, since arena
fragmentation is driven by large transient runs) against their captured stack,
into fixed tables whose exhaustion is counted and reported rather than silently
dropped. Emitted alongside every census sample, so pool totals and attribution
always describe the same instant, plus an explicit unattributed remainder so
incomplete coverage announces itself.
scripts/memlab-report.py ranks sites by retained-byte growth using a linear fit
and can diff two platforms' profiles against each other.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
SQLITE_CONFIG_PAGECACHE is only accepted before sqlite3_initialize, so calling
it from the store open paths was always refused (rc=misuse) and every
request-scoped store fell back to allocating its page cache from the general
allocator. Each request therefore took a fresh multi-megabyte run of arena
slices and left the previous one fragmented behind mimalloc's commit frontier,
which is the per-request retention behind #581.
Hoisted to process init, immediately after the allocator binding and before
anything can open a store. The existing call sites stay as an idempotent
backstop.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Each connection created its own OS thread, so the allocator minted a fresh
thread-heap every time. mimalloc then took new slices at its arena commit
frontier for each heap and never walked back to the purged ones behind it, so
Windows commit charge and RSS ratcheted upward for the life of the daemon
(#581). POSIX never showed it because madvise drops purged pages from RSS
regardless of where the frontier sits.
The worker slots were already pooled and bounded by max_clients; only the
threads were not. Each slot now starts its thread once and parks it on a
condition variable between connections, which keeps the set of allocator
thread-heaps constant. Slots are still released by the reaper as soon as a
connection finishes; threads are joined once, at shutdown.
Adds cbm_cond_t (pthread_cond_t / CONDITION_VARIABLE) since the pool needs a
wait primitive the codebase did not have.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel