mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-03 08:03:38 +08:00
6f290e11d0f46c065b052db2ff70a7bbceede5dd
3050
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
6f290e11d0 |
build: make each grammar object depend on the sources it includes
A grammar unity object, grammar_<lang>.c, #includes its own vendored parser.c and, where the grammar has one, scanner.c. Those files were prerequisites of nothing. Editing a vendored scanner therefore left the object untouched and the edit silently absent from the binary. This is not hypothetical: the properties scanner fix in the previous commit appeared not to work at all. An A/B said 3 stalls against 2 - no effect - and the reason was that prod_grammar_properties.o predated the edit by five minutes and `nm` still showed the old `_reached_eof` symbol sitting in it. The measured "no effect" was a stale object, not a bad fix. The build already treats its other two unity objects this way and says why: LSP_UNITY_DEPS and TS_RUNTIME_DEPS both list the sources and headers they pull in, because this Makefile deliberately does not use compiler auto-dependency (-MMD) generation. Grammars were the gap, and MANIFEST.md's "Local source patches" table means patching a vendored scanner is an expected thing to do - so the trap was reachable by design. Secondary expansion is what lets one pattern rule name each grammar's own directory ($$* is the stem, grammar_properties -> properties). Applied to all three grammar object rules: production, test and tsan. Non-grammar objects match no directory and gain no prerequisite. Verified by touching internal/cbm/vendored/grammars/properties/scanner.c and watching exactly one object rebuild. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
2f33999044 |
test(vm): let the log decide the Windows leg, not the exit status
On 2026-09-18 a Windows leg printed "7925 passed, 0 failed" and "=== All tests passed ===" and still exited 1, while the identical work run through win.sh's other entry exited 0. The status travels ssh -> cmd.exe -> msys2_shell.cmd and that chain loses it. A channel that can turn 0 into 1 can turn 1 into 0, and that direction is a false green: a red Windows leg reported as passing, which is the one thing the local ladder exists to prevent. So the two channels are now split by what each can actually know. The log decides the test outcome — it is written by the runner on the VM and cannot be mangled in transit — and rc only decides what the log cannot see. Green needs zero reported failures AND the completion marker scripts/test.sh prints last, so a leg that stopped early has a summary but no marker and is red. A lost exit status is reported, not obeyed. The logic is a function, vm_verdict(), reachable as `--verdict <log> <rc>`, so tests/test_vm_verdict_contract.sh drives the REAL code with synthetic logs instead of a copy that could drift from it: lost exit status, failures under rc=0, a marker that must not launder failures, failures in a later suite, partial runs, and no summary at all (which keeps its own exit code 90, so "never ran" stays distinguishable from "ran and failed"). Eight cases, wired into scripts/test.sh as step 0y. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
11cb476dff |
fix(semantic): inject callee tokens in a fixed order
inject_calls_pattern_tokens read a node's CALLS edges straight out of the graph buffer, which holds them in insertion order — and insertion order is the order parallel workers happened to merge. It then injected the callee names in that order and truncated at max_tokens, so which callees survived depended on worker scheduling. Co-occurrence enrichment reads a window around each token position, so the surviving vectors differed between runs while the document frequencies (and therefore idf) stayed identical, which is why this showed up as a handful of SEMANTICALLY_RELATED edges flickering and nothing else. It now collects the neighbours through collect_sorted_call_neighbors(), the same canonical ordering the rest of the pass already uses. Three runs of the Go corpus now agree exactly — 334535 nodes / 1795409 edges with vec=0e45744ff1042b51 idf=7e994a1d864a7b07 toks=2518a08a0b6af0c7 — and so do php, kotlin and rust. The edge counts move slightly against the previous build (php 57182 -> 57198, kotlin +17, rust +3): sorting changes which tokens survive the truncation, so the semantic edges change with it. That is a deliberate output change, not a regression. Also adds the checksums that found it, all off unless their env var is set: CBM_SEM_STATE_DUMP writes one hash per scoring input (codes, tf-idf, struct profile, minhash) and one per corpus (ri vectors, idf, and the token SEQUENCE, which is what differed here), and CBM_SEM_PAIR_DUMP writes every candidate pair that passed the threshold with its score, in canonical order. This subsystem has now produced three order-dependence bugs; a checksum at the seam is what turns "the graph differs" into a localised answer in minutes, and it costs nothing when the variable is unset. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
1d58385b51 |
refactor(mem): route this branch's allocations through the memory core
The memory-core linter is a ratchet: raw malloc/calloc/realloc/free/strdup sites may not grow, because allocations belong on one accounted path (src/foundation/mem_core.h) rather than spread across hundreds of call sites. This branch's own work had added 35 raw sites across nine files -- the parallel k8s prep, the configlink key matching, the worker pool's counted path, the Go shared stdlib registry, the spill namespace copy, the route service array and the namespace-name arrays. All of them now use cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/ cbm_mem_strdup. One was a latent bug rather than a style point: pass_k8s freed a buffer that k8s_read_file had allocated with raw malloc, so converting only the free would have mismatched the allocator -- the read function and every free of its result are converted together. The Windows free-space path widens its argument through the memory core instead of cbm_utf8_to_wide(), which allocates with raw malloc. Two files improved enough to tighten their baselines, which the ratchet requires so the gain cannot be quietly given back: sqlite_writer.c 92->74 (the index-cell arena), pass_k8s.c 13->6 and registry.c 19->17. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
6b48097471 |
fix(routes): make Route extraction independent of worker merge order
Two indexes of the same corpus with the same binary produced different
graphs. On the go corpus the difference was 9 nodes and 22 edges, all of
it Route nodes and their HANDLES edges.
Two order-dependent choices, both reading arrays that are in graph-buffer
insertion order -- i.e. in whatever order parallel workers happened to
merge:
- create_grpc_routes() collected proto services into a fixed
services[64] and stopped there, so WHICH services got Routes depended
on scheduling. It was also a silent quality cut: every service past
the 64th got no Routes at all, and kubernetes has far more than 64.
All services are collected now, which is why the go corpus gains ~48
Route nodes that were previously dropped.
- match_one_infra_route() linked an infra Route to the FIRST matching
handler in array order. It now picks the best candidate under a total
order on qualified name, so the choice cannot depend on scheduling.
Two runs of the go corpus after this: node sets identical, edge sets
differing by a single SEMANTICALLY_RELATED edge -- down from 9 nodes and
22 edges. Node counts are identical across four consecutive runs.
The remaining flicker is one near-threshold semantic edge. Ruled out by
reading rather than guessing: funcs[] is already canonicalised by
qualified name, the CALLS inputs are byte-identical between runs, LSH
bucket fill is sequential, candidate truncation follows from that,
admission is applied sequentially in canonical order, corpus batches are
sequential and co-occurrence accumulates per target in fixed order. It
needs instrumented tracing of the pair that flips, and is recorded rather
than claimed fixed.
The new test indexes 70 proto services -- more than the old cap -- and
asserts every one gets its Route, so the cap cannot come back whatever
order the workers produce.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
9a2bdc8c1d |
fix(pipeline): spilling must not change the graph
The same repository indexed with results parked on disk produced a DIFFERENT graph than one indexed in memory. Measured on the php corpus, same binary, reproducible 3/3 each way: 16,561 nodes / 57,182 edges in memory against 16,562 / 59,379 while spilling. Not jitter, and not one-directional -- 1,458 CALLS pairs appeared only under spill and 840 only without it. Present on main, so it ships in v0.11.0 today, and it bites exactly the large-repo users who need spilling. Cause: pass_parallel built the namespace map -- what `use` / `using` / package imports resolve against -- from the in-memory result cache, where a PARKED result is NULL. Every spilled file therefore contributed no namespace, and a file missing from that map does not fail to resolve, it resolves through the looser fallback. That is why edge counts moved in both directions instead of simply dropping. The fix keeps each parked file's namespace IN MEMORY while its result is on disk (one short string per file, nothing against the result it replaces), and builds the map from names so the memory and disk paths agree. Non-spill runs take the identical code path and are unchanged. Across ten languages and four memory settings (default, 2 GB, 700 MB, forced spill), php/java/csharp/kotlin now produce identical graphs where they differed by up to 40,341 edges before. The existing spill-parity contract in test_parallel passed throughout, because its fixture is Go+Java and the compact parity harness omits the pass that builds the map -- neither reaches the namespace path. The new test runs the real pipeline over more than MIN_FILES_FOR_PARALLEL namespaced PHP files, with and without spilling, and fails without this fix. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c3714f1133 |
fix(mem): size the budget from free memory, and relieve on real scarcity
Indexing the linux kernel with NO ceiling set took the machine down: on a
48 GB host the default budget is 50% of TOTAL ram (24 GB), the run used
essentially all of it (24.5 GB peak RSS), and with a 12 GiB VM already
running there was nothing left. The budget never fired -- it read 22 GB
charged against 24 GB -- while the host was out of memory.
Total RAM is the CEILING; what is free right now is the CONSTRAINT.
- cbm_mem_clamp_to_available() reduces a fraction-derived budget to fit
what is actually free, reserving a quarter of it (held between 1 and
8 GB) for the OS and everything else, never below a 512 MB floor. An
explicit CBM_MEM_BUDGET_MB still wins -- it is the user's deliberate
choice -- but is warned about when it exceeds free memory, because
the failure it buys reads like a product bug rather than a setting.
- cbm_mem_should_relieve() = our budget exceeded OR the machine short.
Deliberately separate from cbm_mem_over_budget(): system pressure
must RELIEVE (spill, reclaim), never abort, since the spike may not
even be ours. The extraction loop now enters spill on real scarcity,
and the dump phase -- where the kernel run actually died, with our
charge comfortably under budget -- keys its free-heavy decision on it
too. cbm_mem_system_under_pressure() existed and had no callers.
Same scenario after: budget clamped to 18,947 MB, peak RSS 19.5 GB,
status indexed in 185.7 s, machine fine.
Spill also gets limits and cleanup, because relief that fills the disk or
leaves gigabytes behind is not relief:
- a free-space preflight (never start spilling below 10 GB free) and a
ceiling at half of what is free; past it results stay in memory and
back-pressure takes over. cbm_fs_free_bytes() is new (statvfs
f_bavail / GetDiskFreeSpaceExW quota).
- POSIX unlinks each spill file while holding it open, so the space
returns on ANY exit including SIGKILL -- a killed run cannot leave
spill behind because there is no name left to leave it under.
Windows keeps close-path removal plus a sweep where an open file
refuses deletion, which is exactly the liveness test.
- the staging-database sweep now reclaims orphans of EVERY project in
the cache directory, not just the one being indexed. A kernel index
killed by the OOM killer used to park ~15 GB until someone re-indexed
the kernel, and indefinitely if nobody did. The per-stage lock probe
still keeps a live writer's stage, whichever project owns it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
92e7e6a913 |
fix: keep large thread-locals off the static TLS block
Indexing broke on x86-64 Linux: the index worker refused to start and
SIGKILLed its own process group, so every venue reported nothing but
"index worker ended with killed (exit=-1, signal=9)".
glibc places a thread's static TLS block inside the stack allocation it
makes for that thread. This branch's new thread-local caches -- a
512-slot field cache (~24 KB), per-depth tree cursors (~4 KB) and two
parked CBMArenas (~8 KB; an arena carries blocks[256] + block_sizes[256])
-- took static TLS from 47,905 to 93,513 bytes. The 64 KB parent-death
watchdog thread then could not be created at all, and a worker without
process-tree containment correctly refuses to index.
Four layers, because there were four things wrong:
- The big thread-locals move to the heap behind a thread-local POINTER,
allocated once per thread and reused, each with a correct fallback if
allocation fails (the field cache and cursor pool are caches; a miss
is slower, never wrong). Static TLS is back to 52,233 bytes.
- PARENT_WATCHDOG_STACK_SIZE 64 KB -> 256 KB. "The watchdog only polls,
a tiny stack suffices" was true when written and false once TLS
reached ~50 KB, because the TLS block comes out of that same
allocation.
- cbm_thread_create retries with the default stack on EINVAL. A stack
size is a hint about what a thread needs; the platform refusing the
hint is not a reason to fail to create the thread.
- tests/test_thread_stack_tls_contract.sh (Step 5a of scripts/test.sh)
keeps static TLS a quarter clear of the smallest requested stack,
reading that smallest size from the sources so a new small stack
tightens the gate automatically.
Two diagnostics that made this findable at all, and that stay:
smoke-test.sh prints the worker log its failure message names (CI deletes
that sandbox with the job, so the one artifact naming the cause was
unreadable), and worker_containment_unavailable() now reports WHICH of
its conditions failed, with pid/pgrp/ppid.
Verified in the amd64 container, the venue nothing on the local ladder
covers: baseline main indexes 9039 nodes/15067 edges, this branch before
the fix refuses to start the worker, and after it indexes 9039/15067.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
f0a86ef2c2 |
feat(ci): seeded fuzz leg, and register both new legs as canonical entries
scripts/fuzz.sh drives the libFuzzer targets in two shapes from one entry: a fixed --runs from a fixed --seed, so a CI smoke is a function of the code and not a lottery (O9), and a time-boxed --max-total-time for exploration. Checked-in seeds are copied into a working corpus under build/fuzz/, so a run never rewrites the repository; a crash leaves its reproducer behind and the script exits non-zero. Wired as a smoke into dry-run.yml and as exploration into nightly-soak.yml. The venue-parity contract rightly refused both workflow steps: a venue may provision or plumb, but only a CANONICAL leg entry may exercise the product, and neither script was registered. Registering them — rather than working around the contract — is the fix, since both own their leg end to end (build, corpus or binary, artifact paths, exit code) exactly as soak-legs.sh owns the soak. _memwaste.yml joins the policed venue walk for the same reason, and both scripts join the layer-5 interface probes. Those probes immediately earned their keep, finding two real defects: `fuzz.sh --help` exited 2 because the positional target consumed the flag, and memwaste.sh read a typo'd flag as the repository path — which would have indexed nothing and reported an empty lane as a clean one. test_shell_line_endings.sh now degrades where the checkout has no reachable repository metadata: the Linux container bind-mounts the working tree, and a git worktree's .git is a pointer to the host repository, so `git ls-files` cannot run there and the contract failed with "matched no files — the glob set is broken", killing the leg before it compiled. It is a repository property, so it stays gating in every venue that has the metadata (macOS host, the Windows VM's real checkout at C:\cbm with 117 files, every hosted-CI checkout). test_version_metadata_contract.sh already guards the same way for the same reason; the alternatives considered and rejected are recorded at the case. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
959f4de50c |
perf: cut the allocation churn and repeated work the sanitizer found
Every item here was measured on a real corpus before and after, not
reasoned about. Go corpus, event lane, first run -> now: 245.5 M
allocations -> 134.8 M, 190 GB cumulative -> 29.3 GB, 158 GB never
written -> 9.2 GB, 132 M churn pairs -> 35.7 M, 93 GB of allocator byte
work -> 0.84 GB. Wall clock: Go 40.7 s -> 19.3 s, linux kernel 239.6 s ->
194.6 s, and all ten LSP languages faster with none slower.
Graph output is unchanged, proven rather than assumed: per-class multiset
digests over the full kernel index (8,563,471 nodes identical, 18/20 node
classes and 18/24 edge classes byte-identical, the rest differing only in
the classes a same-binary re-run also perturbs through worker merge order).
The changes, each with its own reason:
arena an exact arena that is appended to later restarts growth at
8 KB instead of the 64 KB default — the cross-file pass adds a
few resolved calls to a compacted result, so the default block
was ~60 KB untouched per result, 0.5 GB of the Go worker peak.
cbm_arena_init_lazy defers the first block for arenas that are
usually never used at all.
scratch per-thread reuse instead of allocate-and-free per unit of
work: result_compact (8 MB kept), the cross-file pass (4 MB),
extract_defs' walk stack, and the sqlite index cells (a 1 MB
arena replacing 12.1 M malloc/free pairs on Go). Walk frames
moving into scratch is why pass_k8s now opens the keep window
explicitly: without it every YAML file opened a 512 KB scratch
on the main thread, +3 GB.
sqlite index cells are written in place — building the record and
copying it into the cell was a second allocation per entry,
16 M record buffers of pure churn on Go.
registry is_test_qn and common_prefix_len in one pass, per-bucket test
flags computed once instead of per candidate comparison, and
the reachability keys arena allocated lazily.
minhash the unique-trigram set is sized from the trigram count and
skips repeats, and LSH queries reuse one seen-set per worker
instead of allocating per query.
parallel lsp_surface rows and the k8s manifest prep now run across
workers (graph writes stay sequential and ordered), the file
node is hoisted out of register_and_link_def, and configlink
prepares dependency imports once with a length prefilter.
go_lsp one sealed stdlib registry shared by all workers, published
by CAS, instead of one per worker.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
632c19d904 |
fix(mem): charge the larger of tracked-live and allocator commit
cbm_mem_charged() trusted mimalloc's committed counter. In a release build that counter never decreases on purge: prim.c only tracks decommit under `#if !MI_DEBUG && MI_SECURE<=2`, so after a large spill the budget still believed memory it had already returned was held. On the linux kernel corpus with a 15 GB budget that read 19,787 MB charged against 15,000 allowed — the run spilled and swept repeatedly while the OS had the memory back. Charging max(OS charge, tracked live bytes) gives 14,379 MB peak charged, sweep down from 14,990 MB to 5,067 MB, and no budget-pressure warnings. The reasoning and the measured numbers are at the call site, because the next reader will otherwise re-derive "the allocator says X" from scratch. Also adds cbm_mem_allocator_stats_log() behind CBM_MEM_ALLOCATOR_STATS: mimalloc's own MIMALLOC_SHOW_STATS never prints for the worker, which exits through _Exit, and mi_stats_merge does not exist in the vendored v3. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
81264814f1 |
feat(qa): memory + CPU waste sanitizer over the memory core
Answers a question the existing tooling could not: of everything we
allocate, how much is never used?
Lanes, all driven by scripts/memwaste.sh:
event every allocation observed through the memory core — size, age,
churn, and whether the bytes were ever written
access SanitizerCoverage load/store instrumentation, so "allocated and
never touched" is measured rather than inferred
scaling k vs 2k replicas through clang profile counters, to catch
super-linear work per file
padding allocator slack per size class
ratchet the numbers recorded per venue so a regression is visible
The event lane needed a foreign-free counter: on Windows the override sees
frees for blocks it never allocated (they come from the CRT), and counting
those as untracked tripped the soundness gate. mem_events now separates
them (14 daemon / 5 worker on the Go corpus).
Two portability fixes the lanes surfaced, both real:
- the enable decision must not be cached before `environ` exists. Under
-fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our
wrapper before glibc sets up environ, so the lane silently disabled
itself and produced no dump at all on Linux.
- the access map needs 48 bits: arm64 Linux hands out addresses above
2^47, which a 47-bit map dropped on the floor.
mem_profile is retired into this: it measured a strict subset and had its
own sampling story.
Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on
all three platforms report-only, and pr.yml/release.yml carry the wiring.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
59a05eb1bf |
Merge pull request #2220 from DeusData/chore/sync-version-0.11.0
chore(pkg): sync packaging metadata to v0.11.0 |
||
|
Martin Vogel
|
ce0b4b214e |
Merge pull request #2221 from DeusData/docs/readme-test-count-0.11.0
docs(readme): current test count — 8,060 tests across 141 suites |
||
|
Martin Vogel
|
22ce867efd |
docs(readme): current test count — 8,060 tests across 141 suites
Badge 6768 -> 8050 passing, the macOS full-suite count on v0.11.0's main (8,050 passed, 10 platform-skipped, 141 suites, run 35003197455); the build-from-source section states the suite total and suite count. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
62fd80b842 |
chore(pkg): sync packaging metadata to v0.11.0
Every packaging surface states the newest v-prefixed tag, as the version-metadata contract requires: the Chocolatey nuspec and installer (checksum re-pinned to the released windows-amd64 archive), the Go installer, the Nix flake, server.json, the npm and PyPI wrappers. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
8972ea69c6 | Break-glass merge PR #2212 v0.11.0 | ||
|
Martin Vogel
|
172787de9c |
fix(mem): no glibc malloc_trim; the static Linux release links again
The v0.10.9 release run (34948714902) failed in both Linux portable builds: the static link found malloc, free, realloc, calloc, memalign, valloc, pvalloc and posix_memalign defined twice, once by mimalloc's override and once by libc.a(malloc.o). Nothing in the build flags had changed since v0.10.8; what changed was one call. cbm_mem_release_to_os called glibc's malloc_trim(0), and that symbol lives only in malloc.o, so the reference pulled the whole object into the link next to mimalloc's definitions. Dynamic builds never noticed: interposition resolves the duplicates at load time, which is why the whole test matrix was green. The call was pointless where it broke things: on Linux and Windows the override routes every malloc to mimalloc, so glibc's heap holds nothing to trim and mi_collect(true) is the release. The branch and its include are gone; macOS keeps its pressure-relief call, since the system heap still serves everything outside the core there. Verification: the static build (scripts/build.sh CC=gcc CXX=g++ STATIC=1) in the local arm64 container reproduced the CI failure exactly before the change and links after it, the binary reporting its version; mem and parallel suites green on the host. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
2058d49a04 |
Merge pull request #2202 from DeusData/feat/memory-core
feat(mem): a central allocation core, the memory migrated onto it, and results spilled to disk under budget |
||
|
Martin Vogel
|
d70b794179 |
fix(mem): the MSan guard in the spill store goes through sanitized.h
cppcheck cannot evaluate `#if __has_feature(memory_sanitizer)` even
behind `#if defined(__has_feature)` ("undefined function-like macro
invocation", the lint lane on
|
||
|
Martin Vogel
|
d89ccabef0 |
fix(mem): the parked result image is declared defined for MemorySanitizer
The MSan lane on
|
||
|
Martin Vogel
|
9e69cb7d9b |
fix(mem): free the thread locale at exit; a settling gate keeps its latch
Two CI-only lanes went red on
|
||
|
Martin Vogel
|
9724d90304 |
fix(mem): dedicated SQLite heap only in the worker; guard the committed stat
The daemon leaked 180 KB per query. The Linux soak's quick leg went from 11 to 144 MB of RSS in ten minutes (13.1x, 628 MB/hr), mimalloc's committed bytes tracking RSS almost byte for byte and the growth linear in the query count, file descriptors flat -- where the same leg had been flat at 15 MB in July. The cause is the previous commit's SQLite heap: one mimalloc heap per thread, created on first use. The daemon runs a thread per connection, so every query ran on a fresh thread with a fresh heap, and every block the shared connection keeps beyond the request -- page cache pages, the statement cache, schema objects -- was allocated in that heap's pages. When the thread exited, its heap was deleted and those pages were left behind holding a few live blocks each: committed for as long as the connection lives, a few pages per query, never reused by anyone. The dedicated heap exists for one reason: the index worker's default heap holds the graph (40M+ blocks), and SQLite churn on that heap paid a page walk per allocation during the coverage publish (132 s vs 9.7 s on the kernel). That process has long-lived threads and exits when the index is done. So the dedicated heap is now a switch, off by default, that the index worker turns on at its entry (main.c, --index-worker); everywhere else SQLite allocates from the calling thread's default heap exactly as before the previous commit. The switch exists in every build and changes nothing where the allocator binds are compiled out. Linux soak, quick leg, before -> after: RSS 11 -> 144 MB in ten minutes (13.1x, 628 MB/hr) -> 10.6 -> 11.5 MB and flat from the first minute over 564 queries with reindexes every two minutes; query-leak leg: 11.6 MB flat over 1,376 queries; committed heap 6.8 MB flat. The same soak showed a second thing. From its second sample on, the query-leak leg reported a committed heap of 2^64 - 121 MB: mimalloc's committed statistic is a signed counter merged per thread at thread exit, and a process whose long-lived thread commits what its short-lived threads free reads it negative -- cast to size_t on the way out. That figure is one of the two the budget takes the larger of. cbm_mem_allocator_committed now reports 0 for a reading above SIZE_MAX/2, so the charge falls back to the OS number instead of a 16 EB budget breach, and the diagnostics report the committed pair through it (the peak never undercuts the current value and never carries a wrapped one), the way they already did for RSS. Detector: the soak's quick and query-leak legs (scripts/soak-legs.sh), the release-gating sequence in every venue -- RED on the previous tree (13.1x), GREEN on this one. A unit test cannot see this class: the test build keeps SQLite on libc so the sanitizers see every block. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a2f29b3fe7 |
fix(mem): the 15 GB budget is met by its own metric, end to end
The 15 GB kernel run completed with every phase mark under budget, but
the budget metric itself -- the charged footprint the extract gate reads
on every file pull -- reached 16.8 GB. Two defects, found with a probe at
the gate (one line per 256 MB of charge after the spill latch, with the
class table; mem.charge.probe, CBM_MEM_PHASES=1, stays in).
Worker graph buffers kept a dense id array over the global id space. The
charge climbed gradually from the 14.06 GB latch to 14.96 GB, then jumped
about 1 GB between two consecutive gate reads, and the only class that
moved in that step was gbuf_index. Worker buffers draw ids from the shared
counter, so each of the 18 kept an id -> node array spanning the whole
global id space -- 8.5M ids x 8 B, next power of two -- almost entirely
empty, and when the global id crossed 2^23 all eighteen doubled in the
same instant. Nothing asks a worker buffer by id before the merge (every
cbm_gbuf_find_by_id caller runs on the main buffer); the worker
constructor now drops the array the way it already drops the five
secondary indexes, and the main buffer answers by id after the merge.
Spill mode did not park what was already cached. The sweeps ran only on an
over-budget observation; a run that latched early (near_budget) and then
stayed under budget through extraction -- which is exactly what the first
fix produced: 14,949 MB at the extract mark -- carried the 44,797 results
cached before the latch (8 GB) into registry build and resolve, which
cannot park, and resolve ran at a charged 21.8 GB against the 15 GB
budget with nothing to stop it. The earlier runs only fitted because the
id-array burst tripped the sweeps by accident. Extraction now ends with a
final sweep whenever spill mode is on: results belong on disk before the
phases that cannot park inherit them.
Kernel, CBM_MEM_BUDGET_MB=15000, before -> after: charged high-water
16,771-16,818 MB -> 14,582 and 14,732 MB in two runs (97-98% of the
budget); the final sweep parks ~45k results and takes the charge from
14.6 to 4.3 GB before resolve, which peaks at 11.5 GB committed; wall
234 -> 224-230 s; nodes identical (8,529,729), edges inside the
run-to-run band. RSS high-water 18.7 GB is
pages already purged and not yet reclaimed: the footprint never exceeds
7.3 GB at a mark.
peak_charged_mb (
|
||
|
Martin Vogel
|
7b77fd4834 |
feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.
WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)
The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.
WHAT CHANGED
Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.
Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.
Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.
The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.
The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.
Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.
Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).
The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.
The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.
Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.
SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).
Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.
Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.
The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.
Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.
PROOF
Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
jitter (gRPC Route nodes: two identical runs differ by 21).
Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
surface rows, nodes/edges inside the Route jitter.
Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
inside the +-400 run-to-run band the kernel shows between identical
runs.
Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
outside what spill can move). Nodes identical; edges 15,773,598.
Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
every phase mark under 15,000 MB by every metric, but the process
RSS high-water mark reached 16.9 GB inside the semantic pass (class
peak 4.9 GB against 0.8 GB live).
Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
stays under 15,000 MB through the whole semantic pass (14,018 ->
14,381 MB across its sub-phases, 6 batches of 125,644 functions);
every phase mark is under budget by every metric (max footprint
12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
15.65 GB and is set in extraction, at the moment the gate latches
(15,002 MB charged) and before the first sweep lands -- a 4%
overshoot of in-flight work, no longer the semantic pass. Nodes
identical, edges 15,772,913, zero foreign blocks.
Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
enters at 14,062 MB (near_budget); the charged footprint never
exceeds 11.8 GB at any mark; the process RSS high-water mark is
15.1 GB in extraction and 15.3 GB during the final write-out --
1-2% over, the in-flight window and the dump transient. Nodes
identical, edges 15,772,543, zero foreign blocks.
Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
peak 998 MB RSS, graph identical to the baseline.
BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)
corpus wall s peak RSS GB nodes / edges / CALLS
perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3
php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8%
rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369
c 5.9 -> 5.7 0.16 -> 0.19 identical
kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1%
django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4%
go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2%
java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6%
csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6%
typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130
kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1%
No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (
|
||
|
Martin Vogel
|
b5f10c2214 |
fix(mcp): give the over-budget error a retry value, and say the peak is not the requirement
v0.10.9 introduces Decision A: an over-budget index attempt fails whole with a named error instead of thrashing. The error told the caller to "Raise CBM_MEM_BUDGET_MB" but not to what, so the obvious next move is to read peak_rss_mb and retry just above it. That move fails, every time, by construction. The abort fires WHEN resident memory crosses the budget, so peak_rss_mb is pinned just above the budget -- it is where indexing was stopped, not what the repository needs. Measured on the linux kernel (94521 files, 43.1M LOC) while benching v0.10.9: default budget 24576 MB -> aborted, peak_rss_mb 25622 completing the same index actually took 31.75 GB So the reported peak understates the real requirement by 24%, and 1.32x the budget was needed. A caller retrying at peak+10% would have burned another 30 seconds to fail again with a nearly identical message. The response now carries suggested_budget_mb (1.5x the current budget, which would have cleared the kernel case) and the hint says plainly that peak_rss_mb is a floor, not a target. Computed as (3*b+1)/2 rather than b + b/2: integer division makes the latter degenerate to b at b == 1, so the suggestion would have repeated the budget that just failed -- the extended test caught exactly that. Not changed: the abort itself, the backpressure ladder, and worker cleanup. A clean test confirmed workers do NOT survive the abort (process count 2 before, 2 after, both idle at 0% CPU), so there was nothing to fix there. tests/test_mcp.c now pins both halves -- a concrete value strictly greater than the budget, and the hint stating the peak is where indexing STOPPED. Verified against the real pre-fix output captured during the bench, which contains neither. mcp suite 317 passed, 0 failed, 4 skipped; lint-ci clean. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7fedfe85c6 |
lint(mem): scan all of src/ and internal/, exempt vendored/ by path segment
The memory-core linter scanned src/ and internal/cbm/. Widen it to src/ and every subtree under internal/, so a new tree added there is covered without anyone remembering to list it, and the count is for the whole project -- cli, mcp, daemon, store, pipeline, the extraction engine. Vendored code is now excluded wherever a vendored/ segment appears in the path rather than by one hardcoded prefix, so that exemption cannot drift either. internal/ holds only cbm today, so the regenerated baseline is byte-identical: 92 files, 4086 raw sites. Widening lost no coverage and vendored stays out (0 sites listed). The red/green proof still holds under the new scope, and the failure message now names the NEW site -- one appended malloc reports grew by 2 (83 -> 85) at the appended line, instead of listing six pre-existing ones. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
da3258f3c7 |
feat(mem): a central allocation core, a linter that enforces it, and phase attribution in the index pipeline
Memory in this project is now allocated through ONE core. This lands the
core, the gate that keeps new code on it, and the instrumentation that lets
an index run say which class of memory grew in which pass -- the three things
the 2026-09-13 audit found missing.
WHAT THE AUDIT FOUND
There was no centralized allocation layer. foundation/mem.h was policy and
measurement only (budget, RSS, over_budget, memory map, phase marks) with no
alloc/free wrapper, so the budget could be observed after the fact and never
enforced. Raw allocator calls in the tree: 4086 across 92 files. The
extraction engine had already adopted arenas (1301 call sites); the graph
buffer -- which holds the 35 GB peak of a kernel index -- used none,
allocating every 64 B node, 48 B edge and every name/qualified_name/
properties_json individually.
Two further facts made a self-accounting core the only workable design:
1. cbm_mem_map_collect() walks THIS thread's mimalloc heap only. Walking
the process-wide heap is a data race TSan caught on macOS, so it is
deliberately not done, and an 18-worker index attributes almost
nothing -- the rest lands in residual.
2. The mimalloc global override is ON for Linux/MinGW and permanently OFF
for macOS (two-level namespace: this binary's free becomes mi_free while
system libraries keep allocating from the system zone, and a pointer
crossing that boundary aborts). On macOS ordinary malloc is served by
the system allocator; the startup audit reports owned_classes=0/6. Any
accounting that assumes mimalloc owns the pointer is blind on an entire
platform.
THE CORE (src/foundation/mem_core.{h,c})
cbm_alloc / cbm_calloc / cbm_realloc / cbm_mem_strdup / cbm_free, each tagged
with an allocation class (gbuf_node, gbuf_edge, gbuf_string, gbuf_index,
extract, semantic, dump, store, other). Per-class live bytes, live blocks and
peak, kept in atomics -- thread-safe by construction, identical on every
platform, independent of which allocator serves malloc. libc semantics are
preserved exactly (alloc(0) is freeable, free(NULL) is a no-op, realloc(NULL)
allocates, a failed realloc leaves the block intact) so adoption is a
mechanical rename and never a behaviour change.
No per-allocation header: a {class,size} prefix costs 16 bytes on ~100M blocks
at kernel scale, 1.6 GB of overhead to measure a memory problem. Sizes come
from the platform usable-size query (malloc_size / malloc_usable_size /
_msize), which is correct under either allocator and free. Where no query
exists (BSD) the request size is charged, which understates -- the safe
direction for a diagnostic. A mismatched class on free clamps at zero rather
than wrapping, so a small drift can never masquerade as a colossal leak.
Arena-backed allocators report in bulk through cbm_mem_class_add_external /
remove_external so their memory appears in the same table; the extraction
engine must not be rewritten to per-object allocation, that would undo the
batching that keeps its allocation count low.
THE LINTER (scripts/lint-memory-core.py, wired into make lint-ci)
Raw malloc/calloc/realloc/free/strdup/strndup outside the core is a defect.
4086 sites cannot migrate at once, so the gate is a ratchet: a checked-in
baseline (scripts/memory-core-baseline.txt) records each file's count and
the build goes red when any file grows, or a new file appears with any.
Files only ever go down; the baseline line is lowered in the change that
migrates the file, and --strict fails a stale baseline. Comments and string
literals are stripped first, so prose that mentions malloc( cannot trip it --
the security audit already bit on exactly that with fork(. Proven both ways:
one appended malloc turns it red naming the file (83 -> 85), restoring turns
it green. Scans src/ and internal/cbm/: cli, mcp, daemon, store, pipeline.
PHASE ATTRIBUTION (src/pipeline/pipeline.c)
cbm_mem_phase_mark and the new cbm_mem_class_log now fire at pipeline.begin
and at every pass.timing site: parallel_extract, the six sequential passes,
the eight predump passes, tests and dump_and_persist, with peaks reset per
index. Both instruments existed in foundation/ and were wired only into MCP
request handling, never into the index pipeline -- which is where the memory
is. Locating the kernel peak this week required an external RSS sampler
because nothing in-process could say which phase it was in.
PRESSURE PRIMITIVES (system_info.c, mem.{c,h}, platform.h)
cbm_system_available_ram() -- macOS host_statistics64 (free + inactive +
purgeable), Linux MemAvailable, Windows GlobalMemoryStatusEx; 0 when unknown,
never cached -- and cbm_mem_system_under_pressure(), true below 12.5% of RAM
available and false when unknown so nothing ever aborts on a guess. Landed
UNWIRED: an earlier draft used them to let the over-budget latch press on
whenever the system was not under pressure, and that permitted the very
overshoot this work exists to end (Decision A's tests at
test_pipeline.c:13481 and :13583 went red under it, correctly). They are the
backstop for the next step, admission control, which keeps a run under budget
before an allocation rather than measuring it after.
mem suite 10 new tests green; pipeline + mem 339 passed 0 failed; lint-ci
clean including the new gate.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
339b3f4097 |
Merge pull request #2195 from DeusData/fix/userns-smoke-spawn
fix(daemon,test,ci): make the #1830 userns guard actually run, and delete the cache behind it |
||
|
Martin Vogel
|
ac8f5b8a81 |
ci(test): report whether unprivileged userns is actually available
The sysctl that lifts the AppArmor restriction is best-effort, and the test runner prints per-shard aggregates only -- 'N passed, M skipped', never individual test names. So if the sysctl stopped working, the #1830 namespace test would quietly return to skipping and the log would stay green and look identical. That is the same false assurance that let a security relaxation ship behind a test which never executed. Probe it the way the test does (unprivileged, no sudo) and print the answer, so there is one greppable line of evidence per ubuntu leg. Deliberately not a gate: a runner that cannot lift the restriction should still run everything else. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
085017ab21 |
fix(daemon): reword the overflow-uid note so the security audit stops blocking it
scripts/security-audit.sh greps src/ for the fork call literal and blocks any file that is not on the allow-list. The note explaining why the overflow uid is no longer cached quoted that literal while describing the hazard, so the audit refused src/daemon/ipc.c -- on a COMMENT, in a file that performs no such call. Reworded rather than allow-listed. An allow-list entry would have silenced the check for the whole file, permanently, so that one sentence could keep its punctuation; the scanner is right to be blunt here and the comment is the cheaper thing to change. The reason is recorded inline so the next person does not reintroduce it. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
d21e6cc762 |
fix(daemon,ci): stop caching the overflow uid, and run the userns guard on the core matrix
Three maintainer decisions from the challenger read on the #1830 unit. 1. DELETE THE CACHE (not just fix its comment). posix_ancestor_overflow_uid() memoised via pthread_once behind a comment claiming "immutable /proc state". Both halves were false: unshare(CLONE_NEWUSER) rewrites /proc/self/uid_map, and pthread_once state survives fork() already marked done. A process that forked and then entered a namespace kept the parent verdict and refused a directory it should have accepted. It was harmless only because every caller today runs in a freshly exec'd process -- the daemon double-fork execs, workers go through cbm_subprocess_spawn's fork+exec, worker_pool.c never forks, and src/ contains no setns or unshare at all. That made a security decision depend on an invariant nothing enforced. The next fork-without-exec caller would have inherited a stale verdict silently. Now derived fresh per call: two small /proc reads against an openat + fstat + fchmod + ACL check per path component in the same walk. With a test that would have caught the original bug and needs no namespace. daemon_ipc_posix_overflow_uid_is_never_cached_issue1830 counts real derivations through the production entry point and asserts the count rises on every ancestor check. Re-introducing a cache fails it by name on every platform. Proven both ways on linux/arm64 as uid 1001: cache re-introduced 49 passed 1 failed; removed 50 passed 0 skipped. Worth knowing why that test is not redundant: with the re-exec probe in place, the userns smoke test PASSES even with the cache restored, because the probe is a fresh process and immune to it by construction. The two tests are complementary -- the smoke test binds the namespace wiring, this one binds the absence of the cache. 2. RUN THE GUARD ON THE CORE MATRIX. Ubuntu 23.10+ ships kernel.apparmor_restrict_unprivileged_userns=1, so the #1830 smoke test skipped on every 24.04 leg and its only real-namespace coverage sat on the two ubuntu-22.04 legs. Those are broad-matrix only, so the test ran in NO pull request, and GitHub retires those images on 2027-04-17 -- after which it would have become a permanent silent skip, and a skip is not a red. A deliberate relaxation of a security check was guarded by a test that ran nowhere a change would be reviewed, on a clock. _test.yml now lifts the restriction with one sysctl on every ubuntu leg (hosted runners have passwordless sudo), best-effort so a runner that refuses simply returns to skipping. The O10 skip-whitelist comment no longer presents the AppArmor restriction as immovable. 3. KEEP THE TOLERANCE, CORRECT THE SAFETY ARGUMENT. The comment claimed that in a single-uid map "nobody reachable can have created it or can mutate it". The overflow uid is what EVERY unmapped host uid maps to, not only host root (user_namespaces(7)), so overflow-owned does not prove root-created: on a shared host another local user's sticky directory is indistinguishable from root-owned /tmp from inside the namespace. The tolerance stays, because there is no in-namespace discriminator that could tell them apart -- and narrowing it would refuse the legitimate root-owned 01777 /tmp that #1830 exists to accept, i.e. it would revert the fix in practice. What changes is the claim: the real guarantee is that no principal reachable from inside the namespace can create or mutate such an ancestor, and the leaf is never tolerated as overflow (0700, euid-rechecked), so a hostile host-side ancestor owner is bounded to denial of service and socket-path control. macOS daemon_ipc 53 passed 2 skipped; linux/arm64 50 passed 0 skipped; lint-ci clean. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
f9038b84eb |
fix(test,daemon): make the userns probe revert-sensitive and stop a false invariant shipping
Follow-up to the re-exec fix, from the challenger read the frozen scope requires for security-boundary units. Three findings, all addressed here; two direction questions go to the maintainer separately. 1. src/daemon/ipc.c stated an invariant that is FALSE. The comment above posix_ancestor_overflow_uid() read "Derived once from immutable /proc state". /proc/self/uid_map is not immutable -- unshare(CLONE_NEWUSER) is exactly what rewrites it -- and pthread_once state survives fork() already marked done. The hazard was written up in the tests, which is the one place an ipc.c maintainer will never look, while production shipped the opposite claim. The comment now states the real guarantee: the cache is safe only because every caller runs in a freshly exec'd process, and it names what would break it. An audit of every process-creation site in src/ backs that: the daemon double-fork execs (daemon/bootstrap.c), cbm_fork_with_retry execs (foundation/subprocess.c), index workers go through cbm_subprocess_spawn, worker_pool.c never forks, and src/ contains no setns or unshare at all. No production path forks without exec and then reaches this check. 2. The probe now _exit()s instead of returning. The runner is built with -fsanitize=address, so returning runs the atexit chain, and a future leak anywhere in the prologue would exit 23 -- which the parent would have read as a verdict on a security test that has none. 3. The parent now treats 0/1/2/3 as the closed protocol and names anything else with the code it actually saw. A signal or a sanitizer exit is a broken harness, not a refusal, and must not be able to look like one. The probe guard also now matches its caller (linux AND test seams) instead of compiling dead code in a non-seams build. VERIFICATION. The challenger was right that the earlier proof was incomplete: it showed the new test goes green, never that the test still binds the #1830 relaxation. Stubbing posix_compute_ancestor_overflow_uid() to POSIX_NO_OVERFLOW_UID -- the relaxation removed -- and running as uid 1001: tolerance off, new probe 48 passed, 1 failed (refused, exit 1) tolerance on, new probe 49 passed, 0 skipped so the smoke test is revert-sensitive to the production decision, not merely to the test change. Worth recording: both deterministic #1830 tests PASSED with the tolerance stubbed off, because they drive the test-seam override rather than the real derivation. They bind the decision table; only this smoke test binds the wiring. macOS daemon_ipc 53 passed 1 skipped; linux/arm64 49 passed 0 skipped; lint-ci clean. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
469c3dd996 |
fix(test): exec the #1830 userns probe instead of forking a primed process
The single-uid userns smoke test forked a child, unshared CLONE_NEWUSER, wrote a 1:1 uid_map, then called cbm_daemon_ipc_private_directory_secure() directly in that child. The call cannot give the right answer there. posix_ancestor_overflow_uid() in src/daemon/ipc.c derives the overflow uid once per process under pthread_once, and the comment above it calls the /proc state "immutable". unshare(CLONE_NEWUSER) is precisely what changes /proc/self/uid_map, and pthread_once state survives fork(). Seven earlier call sites in this suite run the derivation first, so by the time the test forks, the child has already inherited the HOST answer -- no overflow uid -- and refuses the directory no matter what its namespace says. The test therefore asserted a product behaviour it never executed, and failed as a product refusal (exit 1) while the product was correct: production daemons start inside their namespace and derive the right answer on first use. Re-exec /proc/self/exe into a new --userns-secure-probe mode so the decision is made by a process that STARTED in the namespace, which is also the only shape production ever takes. A failed exec gets its own exit code 3 and a named FAIL, so a broken harness can never be misread as a quiet refusal. Verified both directions on linux/arm64 in the container leg as uid 1001: reverted, 48 passed / 1 failed on "expected WEXITSTATUS(status) == 1"; with the fix, 49 passed, 0 skipped. The non-root half of that recipe is now written down at the test, because it is the part that fooled me first: as root the mapped range covers uid 0, so root-owned /tmp stays 1:1 inside the namespace and never turns overflow. The first run of this reproduction passed with the fix REVERTED for exactly that reason. Only a non-root uid reaches the branch the test exists to cover. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0b51555402 |
Merge pull request #2194 from DeusData/fix/security-caller-permissions
fix(ci): grant the security permissions in dry-run.yml, and derive the caller list |
||
|
Martin Vogel
|
09c0e88a64 |
fix(ci): grant the security permissions in dry-run.yml, and derive the caller list
#2189 gave codeql-gate the security-events:read it needs to see code-scanning
alerts, and granted it at the two callers I knew about: pr.yml and release.yml.
There is a third. dry-run.yml calls the same reusable workflow and was left
granting only the workflow default, so every dispatch of it now fails at
startup -- before a single job runs, with no job output to explain why.
Observed immediately: run 34725401182 on
|
||
|
Martin Vogel
|
7d74337efd |
Merge pull request #2076 from DeusData/fix/index-worker-session-scope
fix(worker): scope the index worker to the request's repository, not the daemon's environment |
||
|
Martin Vogel
|
2b80b31306 | Merge remote-tracking branch 'origin/main' into HEAD | ||
|
Martin Vogel
|
93486bca22 |
Merge pull request #2189 from DeusData/fix/codeql-gate-fail-closed
fix(ci): make the code-scanning gate able to see alerts, and fail closed |
||
|
Martin Vogel
|
8d1f4b4c5e |
Merge pull request #2190 from DeusData/fix/vitest-4-1-11
fix(graph-ui): bump vitest past the @vitest/mocker path traversal (GHSA-82fw-gwwq-j7x9) |
||
|
Martin Vogel
|
044fef7957 | Merge remote-tracking branch 'origin/main' into fix/vitest-4-1-11 | ||
|
Martin Vogel
|
1ad52f5fb1 |
fix(ci): make the code-scanning gate able to see alerts, and fail closed
The codeql-gate has never checked anything. It declared no permissions block,
so it inherited the workflow's contents:read and the code-scanning alert API
answered 403. gh writes the API error BODY to stdout, so
ALERTS=$(gh api '.../code-scanning/alerts?state=open' --jq 'length' 2>/dev/null || echo "0")
left ALERTS holding
{"message":"Resource not accessible by integration",...,"status":"403"}0
every [ ... -gt ... ] then failed as a non-integer comparison, the if took its
false branch, and the step printed "CodeQL gate passed (0 alerts)" and exited 0.
Observed in the log of a GREEN run (34700514102, job 103571322660) while the
repository had an open high-severity alert. The `|| echo "0"` reads as defensive
and is the exact opposite: it converts "I cannot see the alerts" into "there are
no alerts", which is the one direction a security gate must never fail in.
Grant the job security-events:read (and actions:read, which the CodeQL-run poll
needs for the same reason), and replace the fallback with a helper that treats a
non-zero gh exit or a non-numeric body as a hard failure. The helper returns
rather than exits because it runs inside a command substitution, where an exit
would end only the subshell and let the caller continue with an empty count --
the same silent pass in a new costume; both call sites check the status.
tests/test_security_gate_fail_closed.sh pins both halves and fails with six
assertions against the previous workflow.
Also bumps graph-ui's vitest to ^4.1.11 (GHSA-82fw-gwwq-j7x9, path traversal via
@vitest/mocker, CVSS 5.9). Development scope, and graph-ui ships in no release
artifact, so nothing released was ever exposed -- but it was the sole open alert
and the gate above is worthless while it stands unresolved. Raising the floor
past the vulnerable range also stops a fresh install resolving back into it.
graph-ui: npm ci clean, 47 tests in 12 files pass on 4.1.11.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
9c6f45bda7 |
Merge pull request #2030 from experticket-admin/fix/windows-worker-job-memory-limit
fix(windows): enforce index worker memory budgets |
||
|
Martin Vogel
|
8c2692c1ee |
fix(graph-ui): bump vitest past the @vitest/mocker path traversal
GHSA-82fw-gwwq-j7x9: path traversal / arbitrary file read via the @vitest/mocker redirect mock, CVSS 5.9, affecting >= 2.1.0 < 4.1.11. graph-ui resolved 4.1.8. Development scope, and graph-ui appears nowhere in scripts/package-release.sh, so no released artifact was ever exposed. It is fixed rather than reasoned around because it is the repository's only open code-scanning alert, and the gate that is supposed to act on it cannot do its job while it stands. The manifest floor moves ^4.1.0 -> ^4.1.11 as well as the lockfile, so a fresh npm install cannot resolve back into the vulnerable range. graph-ui: npm ci clean, 47 tests across 12 files pass on 4.1.11. Split from the code-scanning gate fix deliberately. That change makes the gate able to see open alerts and fail closed; merging it while this alert is still open would block it on the very condition it exists to detect. Landing the bump first lets Scorecard rescan main (it runs on push) and clear the alert, after which the gate fix merges against a clean slate. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
9da4d66df2 |
Merge pull request #2115 from DeusData/fix/activation-guard-home-scope
fix(cli): scope the activation guard to the install-target HOME so a sandbox install can't drain the host daemon |
||
|
Martin Vogel
|
042a58bb1f |
Merge pull request #2188 from DeusData/fix/1856-win10-tls-and-stock-au-ace
fix(install): unblock Windows 10 downloads, and make the activation refusal actionable (#1856) |
||
|
Martin Vogel
|
fa99ff64eb |
test(cli): stop a negative probe from spending 15s proving silence
The macos-15-intel leg killed the cli suite at its 900s wall clock while reporting 7537 passed, 0 failed. Two separate things were wrong. The suite was misclassified. cli spends 497s of the 900s default on macos-14, the FASTEST macOS runner, while macos-15-intel in the same matrix is 2.4-3.6x slower on comparable suites (daemon_runtime 842s vs 349s, stack_overflow_b 277s vs 76s). 497s at that ratio cannot fit. daemon_runtime, at 842s on that same runner, passes only because it was already in SLOW_SUITES. cli belongs there too -- it is deterministic and simply large, which is the only thing that list is allowed to mean. The drain test was also waiting on a clock instead of on the system. Profiling the suite per test (315 tests, 300s wall / 166s cpu) put cli_install_into_host_namespace_still_drains_host_cohort at 42s wall for 19s of cpu -- 23s idle, the largest single idle block, where the next worst was 11s. The existing cli_install_force_quiesces_active_cohort_before_replacing_binary drains a cohort too and idles 0.5s, which is what pointed at the difference. The cost was the host_serving probe. Its generous budget exists for the POSITIVE question -- is this daemon still up? -- where a slow reply on a loaded runner must not be misread as drained; that fixed a real flake and is kept. Asked in the negative it inverts: no reply is coming, so the full 15s is spent establishing silence and ASSERT_FALSE is decided by the timeout expiring rather than by the daemon's behaviour. The drain is already proven positively in that test -- install returns 0 only after the activation completed, and the host child exits CLI_SCOPE_HOST_DRAINED -- so the probe only has to confirm it. Also shortens the drained child's teardown budget, which retried service_free / lease_release / manager_free against 10s on a path where the activation had already torn the service down. Behaviour at the deadline is unchanged; it is reached sooner when nothing is wedged. Measured at only -3s on its own, kept because it is correct and free. Local, same machine, ASan+UBSan: the drain test 42.0s -> 25.8s (-38%), the suite 300s -> 279s, 315 passed before and after -- no test removed, no assertion weakened. PR #2188, which runs the same leg without these two tests, passed macos-15-intel in 37m34s, so the suite was not already over budget on main. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
734906379f |
fix(cli): name the check that refused, and a remedy that can remove it
A reporter brought a full ProcMon trace to #1856 and still could not say which check refused their install. That is not their failing: the binary discards the answer before printing it. cli_activation_production_context_init() validates the cache, rendezvous and log directories -- it is the emitter MOST likely to hold a useful detail -- and it printed CLI_ACTIVATION_REFUSED_MESSAGE bare, through the raw sink rather than cli_activation_diagnostic(). So the reader got "Check the errors above" with nothing above. That is the exact dead end #1416 and #1537 already fixed; the property was repaired on the paths that had tests and left broken on this sibling, which had none. Route it through the attributing helper: the refusal now names the transaction refusal note or cbm_daemon_ipc_validation_detail(). The remedy was also wrong for the reported shape. `icacls <dir> /remove:g <sid>` cannot remove an INHERITED ACE, and the stock C:\ grant for Authenticated Users reaches every new child directory exactly that way -- so a reporter following our advice watches the command succeed and the refusal persist. The refusal now says whether the ACE was inherited, and the advice names `icacls <dir> /inheritance:r /grant:r "%USERNAME%":(OI)(CI)F` for that case. No safety check is relaxed. An install directory writable by other accounts is still refused; it is now refused in terms the reader can act on. The new contract test guards the CLASS rather than this one call site: any emitter of a refusal constant that bypasses the attributing helper fails it, which is what would have caught this. It fails on the previous tree naming src/cli/cli.c:750, plus the missing inherited-ACE report and the missing /inheritance:r remedy. Local: cli 314 passed; activation_transaction daemon_ipc daemon_bootstrap daemon_version 110 passed, 1 skipped (user namespaces are Linux-only, runs on the Linux leg); make -f Makefile.cbm lint-ci clean. Reported by zaferavci1. Refs #1856 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
aacf96a20e |
fix(install): stop naming TLS 1.3 where schannel cannot negotiate it
install.ps1 set its protocol bitmask to Tls12 -bor Tls13 unconditionally. Windows 10's schannel has no TLS 1.3 -- it arrives with Windows 11 and Server 2022 (build 20348) -- but .NET Framework 4.8 still DEFINES the enum value, so the assignment succeeds and nothing warns. The first HTTPS request then dies with "The request was aborted: Could not create SSL/TLS secure channel", because an unsupported flag in this bitmask is a hard failure rather than a downgrade. Every Windows 10 user was blocked before a single byte downloaded, on the documented `irm ... | iex` path, with an error that points at the network rather than at the installer. Compute the set instead: start at Tls12 and add Tls13 only on build 20348+. The enum-name probe alongside the build gate keeps the script parsing on .NET Framework 4.7, where the member does not exist at all. Verified on the Windows VM (build 26200) with the shipped parser: PARSE_OK, pure ASCII preserved, the guard computes and assigns "Tls12, Tls13" there, forcing build 19045 through the same expression yields "Tls12" alone, and an HTTPS HEAD to github.com under the assigned set returns 200. The contract test fails on the previous installer with all four assertions -- unconditional bit, missing build gate, missing enum probe, literal bitmask -- and passes with this change. Reported with a full ProcMon trace and a decoded SID by zaferavci1. Refs #1856 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
9a460b389c |
fix(sha256): keep the message schedule out of the transform's frame
sha256_transform held its 64-word message schedule as a 256-byte local. That is large enough for ASan's use-after-return fake stack, so an instrumented build heap-allocated it through __asan_stack_malloc on every 64-byte block -- millions of allocations to hash one large file. Under the sanitized CI settings (detect_stack_use_after_return=1) fingerprinting a freshly installed binary therefore took minutes, and the cli suite blew its 900s wall-clock budget and was killed as hung on the Linux arm64 leg. The suite passes on macOS, where the fake stack is not engaged the same way, which is why this only ever surfaced on one leg. Move the schedule into cbm_sha256_ctx so it is allocated once per hash instead of once per block. The computed values and the resulting digest are identical; no sanitizer option is relaxed and use-after-return detection stays enabled everywhere. Verified in the Linux arm64 sanitized container under the exact CI ASAN_OPTIONS (detect_stack_use_after_return=1:strict_string_checks=1: detect_stack_use_after_scope=1): cli 316 passed in 356s, previously killed at the 900s budget; daemon_version 10 passed, covering the build-fingerprint digests that would break first on a wrong hash. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |