Answers a question the existing tooling could not: of everything we
allocate, how much is never used?
Lanes, all driven by scripts/memwaste.sh:
event every allocation observed through the memory core — size, age,
churn, and whether the bytes were ever written
access SanitizerCoverage load/store instrumentation, so "allocated and
never touched" is measured rather than inferred
scaling k vs 2k replicas through clang profile counters, to catch
super-linear work per file
padding allocator slack per size class
ratchet the numbers recorded per venue so a regression is visible
The event lane needed a foreign-free counter: on Windows the override sees
frees for blocks it never allocated (they come from the CRT), and counting
those as untracked tripped the soundness gate. mem_events now separates
them (14 daemon / 5 worker on the Go corpus).
Two portability fixes the lanes surfaced, both real:
- the enable decision must not be cached before `environ` exists. Under
-fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our
wrapper before glibc sets up environ, so the lane silently disabled
itself and produced no dump at all on Linux.
- the access map needs 48 bits: arm64 Linux hands out addresses above
2^47, which a 47-bit map dropped on the floor.
mem_profile is retired into this: it measured a strict subset and had its
own sampling story.
Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on
all three platforms report-only, and pr.yml/release.yml carry the wiring.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The census could say which POOL held memory but never which allocation put it
there, so each hypothesis needed a code change plus an eight-minute soak to
test and the investigation could only ask one yes/no question at a time.
Records allocations at or above a threshold (default 32 KiB, since arena
fragmentation is driven by large transient runs) against their captured stack,
into fixed tables whose exhaustion is counted and reported rather than silently
dropped. Emitted alongside every census sample, so pool totals and attribution
always describe the same instant, plus an explicit unattributed remainder so
incomplete coverage announces itself.
scripts/memlab-report.py ranks sites by retained-byte growth using a linear fit
and can diff two platforms' profiles against each other.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel