Reverts 3cd1d33. It was built on the theory that per-connection thread-heap
churn drove #581, and the measurements later falsified that: growth scaled with
requests rather than threads, and the real cause was thread heaps never being
released at all, fixed separately by the TLS detach callback.
It also broke a Windows regression guard -- test_daemon_stability
section_cold_storm, 'a fixed Windows bug is broken again' -- which is what
concurrent clients racing to spawn a daemon do to reworked reaping and shutdown
ordering. A concurrency rewrite that fixes nothing it claimed and reopens a
closed bug does not belong in a release-blocking PR.
The allocator thread-heap fix is untouched.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reading the allocator's per-thread bookkeeping races a thread exiting
concurrently: even mi_theap_get_default() reads state that _mi_thread_done
rewrites via _mi_theap_default_set, and TSan paired those two directly with the
read coming from the periodic diagnostics thread.
Splits the collector. cbm_mem_map_collect_os() fills OS totals only and is what
the periodic snapshot uses; the walking variant stays for tests and explicitly
requested diagnostics, where no other thread is exiting. What the OS-only form
cannot see is carried by the residual, as mem.h promises.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Calling mi_heap_main() is itself the race: it reads the allocator's main-heap
pointer while an exiting thread rewrites it via _mi_thread_done ->
_mi_theap_default_set. TSan paired those two directly on macOS, with the read
coming from the diagnostics thread. Removing the aggregate walk was not enough
because the abandoned-page walk still passed mi_heap_main() as its argument.
The map now touches only this thread's own theap. Everything it can no longer
see is carried by the residual, which is what mem.h promises.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Walking mi_heap_main() reads the process-wide aggregate heap while other
threads allocate into it, which is a data race by construction; TSan reported
it on macOS as init.c:452 in mi_heap_main. A diagnostic must not introduce a
race into the code it measures.
Now walks only what this thread may safely read: its own theap, plus abandoned
pages, which have no owning thread left to race with. The lost coverage is
carried by the residual, which is what it is for -- an unmeasured map must
never read as an empty one.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The shim exists only so the allocation profiler has an observation point on
Linux. Applying its --wrap flags to the test and TSan link redirected malloc
into mimalloc underneath ASan's own interception, putting two allocators on the
same pointers. Production keeps the shim; sanitized builds do not. Windows is
unchanged -- there the wrap flags are what make mimalloc own allocations.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three CI failures, one cause each:
- Vendored integrity flagged a MISMATCH on the Windows mimalloc primitives.
The decommit counters added there were diagnostic and have served their
purpose -- 16k calls, zero failures, which retired the theory that purge was
silently failing. Restored to pristine, and the census fields they fed are
gone with them.
- GCC rejected an anonymous enum declared inside a struct ('declaration does
not declare anything'); clang accepted it, so it only broke the Ubuntu and
Windows smoke builds. Hoisted to file scope.
- clang-format across the files touched this session.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The probe assumed the walk would attribute over half of a 12 MB allocation.
mimalloc v3 exposes only the main heap, abandoned pages and the calling
thread's theap -- there is no API to enumerate every theap -- so on Windows the
probe's blocks are unreachable through all three and the walk saw ~190 KB.
Asserts the triple documented in mem.h instead: what the walk cannot see, the
residual must carry. Either attribution covers the probe or the committed total
grew by at least as much, so memory can never go missing from the map
unnoticed. Bucket attribution is checked where the walk did reach the probe.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A static MinGW link has no DllMain and registers no TLS callback, so nothing
ran at thread exit and every thread left its mimalloc thread-heap behind for
the life of the process. After 300 MCP requests the allocator reported 607
thread-heaps, peak equal to total equal to current -- none ever released --
holding 170.5 MiB against a live set of roughly 300 KiB, growing linearly and
without bound (#581).
POSIX is unaffected because a pthread TSD destructor already calls this, which
is precisely why POSIX stayed flat under the identical workload. The heaps were
invisible to mi_heap_visit, which only walks the main and calling heaps, so the
growth appeared to belong to no allocator at all and survived every allocator
option that was tried against it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Growth is invariant to eleven allocator configurations, so the memory is not
the allocator's. The size profile of what accumulates distinguishes a swarm of
small commits from thread-stack-sized reservations.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The ARM64 fallback captured a return address inside the capture helper, so
every allocation collapsed onto the handful of hook addresses and the site
column named the profiler instead of the caller. Each hook now supplies its own
return address. The census also reports allocation volume, which separates
'the profiler sees nothing' from 'it sees traffic that is all freed'.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
SQLite and tree-sitter are bound directly to mi_malloc, so they never pass the
malloc interposer and the profile could not see them — which excluded the two
subsystems that allocate the most per request.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The hooks run inside malloc/free, so reading the env gate — which allocates —
re-entered them and recursed until the stack died. The process then produced no
stdout, no stderr and no log, which reads as the server refusing to start rather
than as a profiler defect. A thread-local guard makes the profiler invisible to
itself; the non-recursive mutex would also have deadlocked here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fixed corpus, fixed request count, one process, no daemon handshake and no
reindex or crash-recovery phases, so two runs differ only where the code does.
Requests go through a request/response driver rather than a batched pipe:
closing stdin makes the server treat it as a client disconnect and shut down
before answering, which yields a census of a process that never worked.
Linux gains a measurement-only --wrap shim so the profiler can observe
allocations the way the Windows interposer already does; macOS has no --wrap
and stays census-only, which the driver reports rather than passing off as
zero.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The census could say which POOL held memory but never which allocation put it
there, so each hypothesis needed a code change plus an eight-minute soak to
test and the investigation could only ask one yes/no question at a time.
Records allocations at or above a threshold (default 32 KiB, since arena
fragmentation is driven by large transient runs) against their captured stack,
into fixed tables whose exhaustion is counted and reported rather than silently
dropped. Emitted alongside every census sample, so pool totals and attribution
always describe the same instant, plus an explicit unattributed remainder so
incomplete coverage announces itself.
scripts/memlab-report.py ranks sites by retained-byte growth using a linear fit
and can diff two platforms' profiles against each other.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
SQLITE_CONFIG_PAGECACHE is only accepted before sqlite3_initialize, so calling
it from the store open paths was always refused (rc=misuse) and every
request-scoped store fell back to allocating its page cache from the general
allocator. Each request therefore took a fresh multi-megabyte run of arena
slices and left the previous one fragmented behind mimalloc's commit frontier,
which is the per-request retention behind #581.
Hoisted to process init, immediately after the allocator binding and before
anything can open a store. The existing call sites stay as an idempotent
backstop.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Each connection created its own OS thread, so the allocator minted a fresh
thread-heap every time. mimalloc then took new slices at its arena commit
frontier for each heap and never walked back to the purged ones behind it, so
Windows commit charge and RSS ratcheted upward for the life of the daemon
(#581). POSIX never showed it because madvise drops purged pages from RSS
regardless of where the frontier sits.
The worker slots were already pooled and bounded by max_clients; only the
threads were not. Each slot now starts its thread once and parks it on a
condition variable between connections, which keeps the set of allocator
thread-heaps constant. Slots are still released by the reaper as soon as a
connection finishes; threads are joined once, at shutdown.
Adds cbm_cond_t (pthread_cond_t / CONDITION_VARIABLE) since the pool needs a
wait primitive the codebase did not have.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Adds mem_malloc_owned to every diagnostics snapshot: a real malloc/free pair
probed with mi_is_in_heap_region rather than an inference from build flags,
because the Windows static-CRT override is defined at compile time yet can
fail to take effect at link time -- which is exactly what happens here.
This single bit distinguishes 'nothing is live' from 'the allocator does not
own this build', which are opposite conclusions drawn from the same zero
live_bytes. On Windows it reads false, which means every allocator tuning in
cbm_mem_init -- immediate purge, purge-decommits, page reclaim, arena policy
-- is inert for ordinary allocations there, while POSIX gets all of it. That
is the parity gap behind #581, and reading this bit first would have replaced
a long elimination hunt with one measurement.
Also reverts the interim _heapmin trim explored while localising #581. It cut
the ten-minute Windows soak from 1168 MB to 232 MB but never reached the
13-to-15 MB flatness POSIX achieves, and it cost roughly four times the query
latency because it walks the whole heap on the request path. Trimming a heap
the allocator should have owned treats the symptom; the override itself is
the fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The allocator walk could say how much memory was live and in what size class,
but not who was holding it -- and when growth does not route through the
allocator at all, as on Windows, it could say nothing useful. Phase marks
close that gap using the one metric that does see such growth: OS committed
bytes. Each mark attributes the committed delta since the previous mark to the
previous label, so over many passes the label whose total climbs is where
memory stays.
The design is deliberately coarse and says so. Committed bytes are
process-wide, so a busy concurrent thread lands noise in whichever label is
open; signal comes from accumulation over many passes, never one delta. Marks
must bracket a path with no unlabelled gap, or growth hides in the gap -- so
the request path labels its tail 'idle' explicitly, which is also what makes
request-path retention distinguishable from background growth. Off unless
CBM_MEM_PHASES=1.
Pointed at #581 it localised the growth in two steps. At request granularity
dispatch_tool held +102 MB over 27 requests while scope begin/end and idle
were flat, which independently killed the theory that the growth was
time-based. Splitting the handler then put +88.4 MB of it in resolve_store
against +14.1 MB for the query body, with the matching release returning only
-35.9 MB. So each request resolves a store, commits ~3.4 MB, and gets ~1.4 MB
back: a ~2 MB net ratchet per request that never returns. That accounts for
the observed growth, for its scaling with corpus size, for why every tool
leaks identically, and for why only Windows shows it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel