Answers a question the existing tooling could not: of everything we
allocate, how much is never used?
Lanes, all driven by scripts/memwaste.sh:
event every allocation observed through the memory core — size, age,
churn, and whether the bytes were ever written
access SanitizerCoverage load/store instrumentation, so "allocated and
never touched" is measured rather than inferred
scaling k vs 2k replicas through clang profile counters, to catch
super-linear work per file
padding allocator slack per size class
ratchet the numbers recorded per venue so a regression is visible
The event lane needed a foreign-free counter: on Windows the override sees
frees for blocks it never allocated (they come from the CRT), and counting
those as untracked tripped the soundness gate. mem_events now separates
them (14 daemon / 5 worker on the Go corpus).
Two portability fixes the lanes surfaced, both real:
- the enable decision must not be cached before `environ` exists. Under
-fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our
wrapper before glibc sets up environ, so the lane silently disabled
itself and produced no dump at all on Linux.
- the access map needs 48 bits: arm64 Linux hands out addresses above
2^47, which a 47-bit map dropped on the floor.
mem_profile is retired into this: it measured a strict subset and had its
own sampling story.
Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on
all three platforms report-only, and pr.yml/release.yml carry the wiring.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/memlab.sh gave the profiled run a private CBM_CACHE_DIR only, so
the process joined the operator's account daemon and cleanup removed the
cache without stopping any daemon. Source scripts/test-runtime.sh for a
harness-owned runtime and cache, and stop the private daemon before the
work directory is removed.
Add tests/test_memlab_runtime_isolation_contract.sh, which fails before
this change, and wire it into scripts/test.sh.
Part of #1696.
Signed-off-by: Anton Standrik <astandrik@yandex-team.ru>
The local Windows leg caught this within hours of merging #1532: on a machine
that is not domain-joined, USERDOMAIN is the literal string WORKGROUP, and
`WORKGROUP\test` is not a resolvable principal —
WARN: build-dir DACL stamp (pre-wave) failed (user=WORKGROUP\test ...):
WORKGROUP\test: No mapping between account names and security IDs was done.
so the grant fails outright and the tree keeps its inherited
`Authenticated Users:(M)`. The guard then correctly reports that cross-account
mutation is still possible, and the whole leg stops. Since most Windows
machines are not domain-joined, that is the common case, not the exotic one.
#1532 was fixing a real bug in the other direction (a bare name resolves
against the machine first, so a host named like its user grants to an empty
principal). Both failures are name RESOLUTION failures, so this stops resolving
names: the account is identified by SID, which is unambiguous everywhere. The
SYSTEM and Administrators grants in these same commands have always used that
form — the current user was the odd one out. Name lookup survives only as a
fallback where PowerShell is unavailable.
Applied to all three scripts carrying the stamp (run-tests-parallel, soak-test,
memlab), since they share the failure and would otherwise drift apart.
Co-Authored-By: Kiborgik <Kiborgik@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Windows DACL stamps in run-tests-parallel.sh, soak-test.sh and
memlab.sh pass the current user to icacls by bare name. Git Bash resolves
whoami to coreutils, which prints an unqualified name, and icacls resolves
an unqualified name against the machine before the user. On a host whose
name equals the user's (COMPUTERNAME=BUILD with user build) the grant
lands on an empty principal:
BUILD\:(OI)(CI)(F)
Combined with the /inheritance:r on the same invocation, the directory
ends up granting no usable account. run-tests-parallel.sh then cannot
write its own log directory and the wave dies with:
build/c/test-logs/results.txt: Permission denied
FAIL: test-runner --list-suites exited nonzero
Prefix the name with USERDOMAIN when it is set, which makes the account
unambiguous, and fall back to the bare name otherwise. SYSTEM and
Administrators in the same invocations already use SID form and are
unaffected.
Signed-off-by: PETRO YAKOVYSHYN <petroyakovyshyn@gmail.com>
Fixed corpus, fixed request count, one process, no daemon handshake and no
reindex or crash-recovery phases, so two runs differ only where the code does.
Requests go through a request/response driver rather than a batched pipe:
closing stdin makes the server treat it as a client disconnect and shut down
before answering, which yields a census of a process that never worked.
Linux gains a measurement-only --wrap shim so the profiler can observe
allocations the way the Windows interposer already does; macOS has no --wrap
and stays census-only, which the driver reports rather than passing off as
zero.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel