mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-09 09:11:00 +08:00
8af291deb48acda7568300577b670da415da978e
291
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
6bf6e34e43 |
perf(store): evict idle adjacency layers, budget them, own memory class
The daemon keeps adjacency layers (store/graph_mem.h) resident across
requests -- 237 MB for the kernel -- with no end of life but LRU pressure.
- Idle eviction: a layer nobody has held for CBM_GRAPH_MEM_IDLE_EVICT_MS
(10 minutes, measured from its last release) is evicted on the next
request: cbm_mcp_handle_tool sweeps after releasing the request store, and
every acquire sweeps too. No timer thread. Ten minutes outlasts the gaps
between an agent's graph calls within one task, while a session that moved
on stops pinning the memory; a wrong guess costs one rebuild (1.2-1.5 s on
the 17 GB kernel index, tens of ms on ordinary repos). The clock is a test
seam (cbm_graph_mem_set_clock_for_testing), so tests move time instead of
waiting.
- Worker budget: application_worker_memory_slice_locked hands index workers
the aggregate minus the resident layer bytes (cbm_graph_mem_cached_bytes),
at least one byte per active job so a slice never reads as "no cap".
- Memory class: layers are charged to the new CBM_MEM_CLASS_GRAPH_CACHE
("graph_cache" in census and class logs) instead of STORE.
Responses stay byte-identical to the baseline on django and redis (tools
and Cypher) and on the kernel trace/hub and extra BFS/degree cases.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
3c951326c2 |
fix(cypher): walk operator chains in the early WHERE pass in a loop
eval_expr_partial, the three-valued evaluator that execute_single runs on every seed row before relationship expansion, recursed into the right child of every AND / OR / XOR node. With operator chains linked along the right child, that is one stack frame per operand: the 50,000-operand OR and AND chain tests overflowed the 256 KiB query worker stack in this early pass, before the already iterative eval_expr ever ran. Walk a chain the way eval_expr does: recurse into NOT and into each operand, and follow the links in a loop. The operands are read in the same order and with the same short-circuits as the nested form (AND stops at the first definite false, OR at the first definite true, XOR reads every operand), and the running value is combined three-valued, so an UNKNOWN operand anywhere in the chain keeps the seed for the full evaluation after expansion. A new test runs three-operand OR / XOR / AND chains whose UNKNOWN operand sits on a chain link and not only on the chain's head. A loop that lets a later operand overwrite the UNKNOWN prunes the seed and fails it. eval_expr_partial moves into the whitelist's chain-walker group with eval_expr and cypher_expr_selectivity: it still recurses, but only as deep as the parse-depth cap allows. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
df6f674ecb |
fix(cypher): walk operator chains and UNION branches in loops
A WHERE chain such as `a OR b OR c ...` was folded into a left-deep tree, so every walker over the expression tree recursed once per operand: the evaluator, the seed-selectivity pass of the planner and, every 128 nodes, the free routine. UNION branches were parsed by a nested cbm_parse call per branch, so the parser recursed once per branch as well. Stack use therefore followed the length of a query, not its nesting, and the daemon evaluates queries on a worker thread with a 256 KiB stack. A long but flat query, such as a few thousand `OR name = ...` terms, needs to work there. Chains of one operator are now linked along the right child at parse time, OP(a, OP(b, OP(c, ...))), and eval_expr and cypher_expr_selectivity follow the links in a loop, recursing only into the operands, whose nesting the existing parse-depth cap bounds. Operand order and the left-to-right short-circuit of AND and OR are the same as in the nested form, XOR keeps its running parity, and a missing value takes part in a chain exactly as before. expr_free already walks with an explicit stack and needs no change for a right-linked chain. UNION branches are read by one parser in a loop in cbm_parse and linked as they come; parse_post_where no longer parses the branch after the keyword, and the ALL flag still sits on the branch before it. Tests: 50,000-operand OR and AND chains and 20,000 UNION ALL branches run on a thread with the worker stack size inside a forked child, with the expected rows computed by the test; a chain with operands over a missing property is compared with its parenthesised forms in both associations; 50 UNION and UNION ALL branches return the right rows. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a67a3f417a |
fix(sha256): build the ARMv8 SHA2 path with GCC 11
GCC 11's arm_neon.h declares the SHA2 intrinsics under
`#pragma GCC target ("+nothing+crypto")`. sha256_blocks_arm was marked
`target("+sha2")`, a narrower target, so GCC 11 refused to inline
vsha256hq_u32 and friends ("target specific option mismatch") and the
build stopped. GCC 13 accepts "+sha2", which is why PR CI (ubuntu-24.04-arm)
stayed green; the dry run's ubuntu-22.04-arm leg (GCC 11) failed on all
three shards (run 38186491924).
GCC now gets "+crypto", which implies "+sha2" on every GCC. The function
still uses only SHA2 instructions, and the runtime HWCAP check gates it.
clang keeps "sha2".
Checked in arm64 containers:
- sha256.c with CI's ASan/UBSan flags compiles on GCC 11, 12, 13 and 14;
GCC 11 with "+sha2" fails with the CI errors;
- a known-answer harness ("abc", "", a 1 MiB pattern) matches hashlib on
the arm-sha2 backend and the forced portable backend, GCC 11 and 14,
-O0 and -O2.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
a19d7a45fb |
Use memory core for Lens response ownership
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
08e0474a36 |
Harden CBM Lens integration against current main
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0392068d49 |
Merge main into CBM Lens Beta
Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
afeaca0217 |
Complete child-local environments and ObjectScript parse deadlines
Merge validated environment overrides into a child-owned environment on POSIX and Windows, preserving caller PATH lookup for executable names. Forward each ObjectScript export parse timeout to the extractor. Cover inherited and replaced values, empty values, duplicate overrides, parent isolation, invalid names, executable lookup, and timeout forwarding. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3eeefa2d2a |
perf(sha256): compress blocks on ARMv8 SHA2 / x86 SHA-NI (#2441)
Every process start fingerprints its own executable for the daemon build
identity, and the hook path pays for that before it can answer. With the
portable transform that was ~1.2 s of CPU for the ~300 MB binary, so on a
slower machine hook-augment hit its 2000 ms deadline and printed nothing
(#2441).
Two changes in src/foundation/sha256.c:
- cbm_sha256_update compresses whole blocks straight from the input instead
of copying every byte through the context buffer.
- Block compression runs on the CPU's SHA-256 instructions when it has them:
ARMv8 SHA2 (sysctl on macOS, HWCAP on Linux, IsProcessorFeaturePresent on
Windows) or x86 SHA-NI (CPUID). The backend is probed once; the portable
transform stays the fallback.
Measured on an Apple arm64 Mac (macOS 26.6), release builds of main
|
||
|
Martin Vogel
|
1d7c807136 |
Merge main into fix/workspace-manifest-resolved-entries for the release batch
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e775c7f511 |
fix(subprocess): a child that is exiting at the group check is a finished run
macOS answers ESRCH to setpgid and getpgid for our own unreaped child from
the moment it starts to exit, before waitid can report it.
|
||
|
Martin Vogel
|
a5c7ae9789 |
fix(subprocess): a child that already exited is a finished run, not a failed spawn
After posix_spawn the parent re-asserts the child's process group. A short command can exit before the parent gets there; macOS then answers ESRCH for the exited, unreaped child to both setpgid and getpgid, and the run was reported as SPAWN_FAILED (errno 3). macOS CI read `git write-tree` that way under load (macos-15-intel and the LeakSanitizer leg). The group was already set before exec, by the spawn attributes or by the child itself, so an exited child is accepted when waitid(WNOWAIT) shows it waiting to be reaped; the supervision then collects its real exit status. A test seam holds the parent until the child has exited, which makes that order deterministic: subprocess_run_child_exited_before_group_check is red without the fix (SPAWN_FAILED) and green with it, on macOS and Linux. The test file is also reflowed to the project's clang-format, as on main. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
867e5f883d |
Merge origin/main into the smart-CI branch
Ten files changed on both sides; every conflict was a union of independent additions: - workflows (_lint.yml, _test.yml): keep the branch's lane `if:` and main's pinned ubuntu-24.04 runner with its reason. - internal/cbm/cbm.h: main's callable-identity and module-doc fields plus the branch's configured-test fields. - Makefile.cbm: both new test files; prod_lsp_all.o keeps main's .build-config dependency and the branch's coverage flags. - scripts/test.sh: main's setup-scripts contract becomes Step 0y3, ahead of the branch's 0z series and the contracts-only exit. - src/foundation/subprocess.c: main's cbm_posix_close_fds_from (it closes the error descriptor too). - src/mcp/mcp.c: main's repository default base for detect_changes, with the early returns released through the branch's detect_release. - src/pipeline/lsp_resolve.h, tests/test_main.c: both includes, suites and start-up steps (main's git environment reset first). - scripts/memory-core-baseline.txt: recounted on the merged tree and lowered (mcp.c 773, config_toml_edit.c 73). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c5abb64943 |
feat(test-impact): detect_changes scope:"tests" end to end
One request pins the change (merge base .. HEAD), builds the graph and the runnable test registry of exactly the HEAD snapshot, seeds the graph from the change, walks what depends on it and answers which tests to run (cbm.test_impact). Missing or doubtful evidence never narrows: it becomes a run-all answer that names why. Snapshot (classified ledger -> frozen graph -> registry): - one shared language classifier for discovery and pinned blobs (raw bytes on every platform); the classified ledger feeds the frozen build, which indexes exactly the listed files and records partial parses as per-file PARSE_GAP instead of refusing the graph; - the pinned tree skips symlinks and submodule links (counted), as discovery never indexes them; - the test model scopes uncertainty to the suites of its file (they run whole), reads line splices where they occur, treats a mid-file `#ifndef X` + `#define X` as an ordinary condition, and no longer reads a plain identifier named like a macro as an invocation. On this repository the registry is now narrowable and matches a text census exactly (8,365 cases, 141 runner suites). Seeding and reach (port of the measured replay, rule B): - hunk -> overlapping definitions, comment-only hunks dropped, pure deletions -> enclosing definition, no-definition hunks -> whole file, deleted names found by text, header changes propagated through the include closure and the headers' macros (pastes included), file-level escalation for unclassifiable header lines; - inbound fixpoint walk over CALLS, ASYNC_CALLS, USAGE, CALL_REFERENCE, READS (+ routes); cases, suites and the runner's entry points are sinks; a reached runner main runs everything (RUNNER_REACHED); - a parse gap whose text names a reached definition reaches the definition around it (or its file); - macro-registered tests run their suites whole; cases and suites map to graph nodes by their start line. The line patterns reproduce the reference regexes exactly on 1,383,850 lines of this repository and 37,167,926 kernel lines; on 24 replayed changes the engine selects at least what the reference selected (13 identical, the rest only conservative supersets on the same graph). Also: - configured test declarations that cannot be mapped degrade their file (no configured roles, a test_declarations diagnostic) instead of aborting the whole index; - the selector keeps other runners' suites out of the selection, and the result codec judges the model by narrowability like the selector; - a change to .codebase-memory.json runs everything (built-in rule); - .codebase-memory.json with the c-cbm preset and this repository's rules. Gate fixes (suites, lint-ci, Windows cross-compile): - the request runs on its own thread with a default-size stack (daemon tool calls run on 256 KiB threads, where the in-process build died); - a request-supplied config_path is no longer followed by an unlink of an uninitialized path buffer; - the inventory's frame-local reader no longer stays reachable from the caller's source after the filter call; - FILE_ID_INFO needs _WIN32_WINNT >= 0x0602; the pinned-tree header pins it before any system header (toolchains defaulting lower failed); - memory-core ratchet: the win_utf8 converters gain core-backed `_in(cls, ...)` forms over one implementation (the legacy names keep the C library for unmigrated callers), the Git facts module uses them, and the Windows spawn path releases everything through one failure exit; pipeline, extraction and config allocations that this work touched moved to the core in matched pairs; baselines tightened; - the branch's C files formatted with the project's clang-format. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
8ab555ee38 |
Merge pull request #2253 from DeusData/distill/1245-cypher-partial-where
fix(cypher): stop the early WHERE pruning rows it cannot judge yet |
||
|
Martin Vogel
|
c442028010 |
fix(workspace): classify manifest entries in their resolved form
A project's .cbmpathwhitelist lists extra roots it would like indexed. When such a manifest was approved, and again each time it was consulted, every entry was classified exactly as written in the file, while the directory that would actually be indexed is the one the entry resolves to. An entry that reached the home directory, a credential directory or the volume root through a link or through `..` segments was judged by its spelling and passed as an ordinary path; only the containment check resolved it. An entry naming a directory that did not exist was approved as well. Both cbm_workspace_manifest_approve and cbm_workspace_manifest_allows now resolve each entry with cbm_canonical_path, the resolver every other caller of the policy applies, and classify that form. Approval refuses an entry that is relative, does not exist or is not a directory, with a message that names the entry and, when the resolved directory differs from what was written, shows both. Approval stays bound to the manifest text; where an entry leads is read again when it is used, so a link that is retargeted after approval is judged by its new target. At use time an entry that no longer resolves matches nothing, and the other entries of the manifest are consulted as before. An ordinary, existing, absolute entry keeps its result. A manifest that names a directory which does not exist yet is now refused at approval and has to be approved once the directory is there. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7bb1de0ed5 |
Merge pull request #2479 from DeusData/fix/workspace-resolved-home
fix(workspace): compare the home directory in its canonical form |
||
|
Martin Vogel
|
f40b1a564b |
Merge pull request #2475 from DeusData/fix/regex-compile-size-budget
fix(regex): size a pattern before compiling and refuse oversized ones |
||
|
Martin Vogel
|
a245618a64 |
wip(ci): checkpoint hybrid test-impact implementation
Preserve the current selector, frozen-input inventory, result codec, coverage collection, runner filtering, and isolated Windows venue work. Component-level regression proofs are recorded in the PR handover. The live classified provider, public execution integration, artifact admission, fresh oracle/shadow evidence, and full platform gates remain incomplete. This checkpoint does not enable or claim verified narrowing. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
91ba8a3aa9 |
fix(subprocess): close inherited descriptors with kernel range operation
Use the Linux close_range syscall or BSD closefrom before exec, retaining the bounded close loop when the range operation is unavailable. Add coverage of descriptor closure, strategy selection, and forced fallback. Refs #1484 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
2e6dc0877c |
fix(workspace): compare the home directory in its canonical form
cbm_workspace_home_dir() returned the raw HOME (or USERPROFILE) string, while cbm_workspace_classify_root() compares it against a canonical, link-resolved path. Where the home directory is reached through a link (/home kept on another volume, a macOS firmlink, a HOME that is itself a link) the two spellings differ and the home rule never fired, so the home directory was not classified as the home directory. The helper now resolves the value through cbm_canonical_path, the same resolver every caller applies to the path it classifies, and falls back to the raw value when the directory cannot be resolved. The result lives in a per-thread buffer recomputed on each call, as cbm_resolve_cache_dir does, so a changed environment is seen at once. The environment is read with cbm_safe_getenv so a non-ASCII USERPROFILE arrives as UTF-8. The ancestor comparison now treats "/" and "\" as the same separator: on Windows the resolver returns backslashes while some callers normalize to forward slashes, and both spell one directory. Two tests cover the helper: one with a plain temp directory (already a firmlink on macOS), one with a home reached through a symbolic link (POSIX; creating a link needs a privilege on Windows). The existing explicit-approval test in the mcp suite now presents its session root in resolved form, as the daemon does; with the raw spelling it would have passed as an ordinary root and stopped exercising the approval path. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
9db7de7bae |
fix(regex): size a pattern before compiling and refuse oversized ones
cbm_regcomp() handed every pattern straight to the platform compiler (libc regcomp on POSIX, the vendored TRE on Windows). Both expand bounded repetition at compile time, so the compiled size of a pattern is not proportional to its length. Measured on macOS libc (TRE-derived) with REG_EXTENDED|REG_NOSUB, the compiled form costs about 0.6 KB per expanded atom (1.1 KB with submatch tracking, 1.7 KB with REG_ICASE), a K-way alternation adds about 70 B x K^2 for its position sets, and an alternation under an interval pays that for every copy. The wrapper now sizes a pattern in one linear pass before compiling it: an atom costs 1, a group costs its content, an interval multiplies the element it applies to by its upper bound, and a K-way alternation is charged K^2/8 on top of its branches. Arithmetic saturates and group nesting is bounded. A pattern longer than 32768 bytes, or whose estimate exceeds 32768 units, is refused with CBM_REG_ETOOBIG before anything is allocated. The heaviest accepted compile is a repeated alternation at about 30 MB with REG_NOSUB (about 90 MB under the ICASE-plus-submatch flag set no call site uses). Ordinary intervals, the pipeline's fixed patterns and an alternation of 400 twenty-character identifiers remain accepted. The refusal reads as a normal compile error at every call site. search_code now reports an unusable path_filter as a tool error instead of silently dropping the filter, search_graph reports a failed row scan instead of returning an empty result, and query_graph carries the reason in the result's warning field; all three name it. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
48acb559e6 |
Merge pull request #2405 from DeusData/fix/internal-db-enumeration
fix(cli): never treat internal cache stores as project indexes |
||
|
Martin Vogel
|
d0ec2313d9 |
Merge pull request #2394 from DeusData/fix/issue-2184-post-extraction-budget
fix(pipeline): spill before the post-extraction phases when they would cross the budget (#2184) |
||
|
Martin Vogel
|
94097f0727 |
Merge pull request #2341 from DeusData/fix/git-spawn-local-env
fix(git): clear git's repository-local env for cbm's own git children (#2003) |
||
|
Martin Vogel
|
ea931e7647 |
feat(ui): refine graph exploration and add local agent configuration
Add focused galaxy tracing, edge filters, organic cluster layouts and cached graph exploration with compact architecture and impact summaries. Make the browser-local agent explain the current selection proactively, and add a Config panel with persisted daemon overrides, shared browser preferences and a reference for external configuration inputs. Validation: 182 focused frontend tests and TypeScript passed. Reused the recorded production build and 485 passing native checks on unchanged source. The pre-existing capped-layout loading-note test failure remains documented separately; the full local CI ladder was waived. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin VogelandAndrew Hundt
|
dc5293d71e |
fix(cypher): stop the early WHERE pruning rows it cannot judge yet
Distilled from #1245 by Andrew Hundt (upstream
|
||
|
Martin Vogel
|
460153204b |
fix(cli): never treat internal cache stores as project indexes
The cache directory holds two internal SQLite stores next to the project
indexes: the user's settings (_config.db) and the cross-repo store
(_cross_repo.db). cbm_list_indexes, cbm_remove_indexes and
count_db_indexes matched every "*.db" there. As a result:
- install --reset-indexes and uninstall (when index deletion is confirmed)
deleted the user's config (every `config set` value silently reverted
to its default) and the cross-repo store along with the indexes;
- the "Found N index(es)" listing and count over-reported by up to two.
pass_cross_repo.c already excluded both stores by exact filename, but in
its own copy of the check. Centralise the rule in foundation/str_util:
cbm_is_internal_cache_db() (exact names, so a real project whose name
starts with "_" or contains "config" is still an index) and
cbm_is_project_index_db(). Every enumeration of the cache directory's
.db files (list, count, remove, cross-repo target scan) now calls it, and
cbm_config_open() builds its path from the same CBM_CONFIG_DB_FILENAME
constant.
Proof on a real isolated cache (one indexed Go repo, `config set
auto_index true`, a _cross_repo.db present):
before: "Found 3 index(es)", "Removed 3 index(es)", cache left empty,
`config get auto_index` -> false
after: "Found 1 index(es)", "Removed 1 index(es)", _config.db and
_cross_repo.db kept, `config get auto_index` -> true
Test: cli::cli_index_enumeration_skips_internal_cache_dbs (RED on main,
green with the fix, RED again on revert).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
26cbccd998 |
fix(pipeline): spill before the post-extraction phases when they would cross the budget (#2184)
Spill mode was entered only DURING extraction, when the charge crossed
budget - budget/16, and the final sweep at extraction end ran only if that
latch had already fired. A run that ended extraction just under the latch
kept every result in memory, and the phases that cannot spill -- registry
build, cross-LSP prepare (all_defs, per-language registries, surface rows,
module index) and resolve (edges into the graph buffer, cross-LSP appends
into the cached results) -- then grew the process well past the budget.
openclaw at 8 workers with a 4079 MB budget: 3678 MB charged at extraction
end, 4076 after registry build, 4933 after cross-LSP prepare, 5829 in
resolve (+43 %). The same repo at 18 workers latched during extraction and
stayed under budget.
Extraction end now projects the post-extraction growth from the result
counts (defs, calls + usages, files) and enters spill mode when charged +
growth would cross the latch, so the existing final sweep parks every
cached result before registry build. The per-unit costs (1280 B/def,
160 B/ref, 8 KiB/file) are fitted on the measured charge growth from the
parallel_extract to the parallel_resolve mark on five local corpora
(go, django, kotlin, rust, php) and cover openclaw; the model under-reads by
at most 7 % (inside the budget/16 margin) and over-reads by up to 46 %, the
safe side: a spill never changes graph content. The decision is a pure
function of counts (no RSS-window sampling) and is logged as
mem.post_extract.projection.
A charge test seam (cbm_mem_set_charged_for_tests) pins cbm_mem_charged(),
which otherwise reads the process footprint, so the "ended just under the
latch" case is testable deterministically.
Proof (release build, M5 Pro, CBM_WORKERS=8, peak charged = the budget
metric's high-water mark):
go, CBM_MEM_BUDGET_MB=2000 before: extract 1348 -> registry 1497 ->
lsp_prepare 2168 -> resolve 2660 MB (+33 % over budget, no spill)
after: projection charged 1335 + growth 1909 > line 1875 -> spill,
21,882 results parked; extract 413 -> registry 558 -> lsp_prepare 1151
-> resolve 1297, semantic 1746 MB peak: every phase under budget.
Node/edge dumps byte-identical (335,191 nodes / 1,813,857 edges); wall
29.2 s vs 29.0 s.
go, 40000 MB (generous): decision=keep, no spill, identical dumps, wall
27.4 s vs 27.5 s.
django, 400 MB: resolve mark 455 -> 277 MB after the spill, dumps
identical; the remaining peak (476 MB) is the semantic pass's retained
set on top of the graph floor, which this change does not address.
Refs #2184
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
News
|
ed991154f2 |
fix(windows): preserve mkstemp errors through cleanup
Signed-off-by: News <knewstimek@users.noreply.github.com> |
||
|
Martin Vogel
|
40dad610ee |
fix(git): clear git's repository-local env for cbm's own git children (#2003)
Git exports repository-local variables (GIT_DIR, GIT_WORK_TREE,
GIT_INDEX_FILE, GIT_COMMON_DIR, ...) into hooks, and some editors export
them into their terminals. They take precedence over `git -C <dir>`, and
every git child cbm spawned inherited them. A cbm started from such an
environment therefore resolved every project against the CALLER's
repository: git_context reported a plain directory as is_git=true with
the caller's worktree root, branch and HEAD, the history pass attributed
the caller's commits to it, and the watcher / detect_changes /
artifact git calls read (and `config merge.ours.driver` could write) the
wrong repository.
Fix: one production helper, src/foundation/git_env.{c,h}, holds the
15-variable list `git rev-parse --local-env-vars` prints and builds the
child environment with exactly those removed -- a POSIX envp array or a
Windows CREATE_UNICODE_ENVIRONMENT block (case-insensitive names). It is
applied per spawn; cbm's own environment is never modified (no unsetenv
in the parent, which would race other threads):
- cbm_popen_git(cmd): read-mode popen for git command lines. POSIX spawns
/bin/sh -c via posix_spawn with the scrubbed envp (stdin /dev/null,
pipe ends close-on-exec, reaped by cbm_pclose); Windows passes the block
to the existing handle-isolated CreateProcessW spawn. Fails closed.
- cbm_proc_opts_t.strip_git_repo_env: cbm_subprocess_spawn passes the
scrubbed envp to posix_spawnp / the fork+exec child, or the block to
CreateProcessW. Fails closed.
- Every git spawn uses them: git_context, pass_githistory, artifact
(4 sites), the watcher's git runner, and detect_changes' git commands.
search_code's grep scan keeps the inherited environment.
Tests (RED before, green after, RED on revert of the helper and of the
call sites):
- git_context::git_context_ignores_inherited_repo_env -- GIT_DIR and the
hook set pointing at a decoy repo: a plain dir is is_git=false, a
fixture repo reports its own HEAD and branch, the decoy is untouched.
- git_context::githistory_ignores_inherited_repo_env -- the history pass
on a plain dir sees 0 commits (was the decoy's 4).
- subprocess::subprocess_strip_git_repo_env_is_per_child and
popen_git_strips_repo_env_and_reports_exit_status -- all 15 variables
are gone in the child, unrelated variables pass through, the parent
environment is unchanged, and cbm_pclose keeps pclose() status
semantics.
Proof (macOS arm64, GIT_DIR/GIT_WORK_TREE/GIT_INDEX_FILE/GIT_PREFIX
exported at a decoy clone, `cli index_status verbose=true`): a plain dir
went from is_git=true, worktree_root=<decoy>, branch=<decoy's> (v0.11.0)
to is_git=false; a fixture repo went from the decoy's root/branch/HEAD
to its own root and branch.
Refs #2003
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
News
|
1d2bbc4259 |
fix(windows): preserve worker recovery errors
Signed-off-by: News <knewstimek@users.noreply.github.com> |
||
|
Martin Vogel
|
1a3425536b |
fix(graph): quality round after the v0.11.0 probe — deterministic resolution, findability, coverage, names
Nine findings from the 2026-09-16 graph-quality probe over the eleven bench corpora, each with a
test that fails without its fix and a before/after measurement on the real corpus.
Deterministic resolution (O9). Two indexes of one kernel tree differed in 5,821 IMPORTS and 632
CALLS edges. Three order/load dependencies carried it: the `#include` target was the first by-name
hit whose path ends with the include path — a bucket in worker merge order (include/linux/device.h
vs tools/virtio/linux/device.h); the registry broke suffix_match score ties by bucket position; the
LSP budget-share rule measured wall time. Include targets now prefer the File node, the least nested
path, then the lexicographic path; registry ties prefer the least nested QN, then the smaller QN; the
budget share is measured in thread CPU time like the budget itself. Two kernel indexes are now
identical (8,529,723 nodes, 1,974,940 CALLS, zero churn), as are two elasticsearch indexes.
Findability. search_graph's BM25 path honours the `label` filter and ranks an exact-name definition
above test methods that repeat the token (15/15 probe queries).
Coverage. The parse-coverage gap tree subtracts definitions the walker recovered inside an error
region (multi-line only); kernel/sched/core.c reports 476 flagged lines instead of 7,674.
Precision. Config-language callers (Makefile, CMake, YAML) no longer bind unique_name into code
(kernel Makefile -> C CALLS 195 -> 0). JS-family token names ({}, 1, (x: T) => U) are not
definitions; computed, quoted and escaped member keys stay (TypeScript 204 junk -> 0, 1,782 real
names kept). Names are cut at their first line (elasticsearch YAML 8,306 -> 0); a comment inside an
argument list is not an argument, so no Route node is named by a Java block comment; the wildcard
route `/*` is still a route. HTML/Vue/Svelte/Astro embedded member calls stay unbound.
Recall. A Python call through a self- or cls-rooted attribute keeps a specific unique_name member
target (Django +751, elasticsearch +3,288 CALLS, no new callee names).
Robustness. Edge properties never carry a split UTF-8 sequence (the 120-byte expression cut and the
JSON escaper both trim a partial code point). CLI array flags accept a JSON-array literal. Cypher
seeds a single-hop pattern from its selective end (kernel `MATCH (a)-[:CALLS]->(b) WHERE b.name =
'_printk'`: execution limit -> 7.9 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
News
|
25d08258e5 |
fix(windows): keep aligned allocations on mimalloc
Signed-off-by: News <knewstimek@users.noreply.github.com> |
||
|
Martin Vogel
|
92abefa3f5 |
Merge pull request #1723 from liuchong/feat/index-resource-discovery
feat(index): add opt-in discovery resource limits |
||
|
Martin Vogel
|
9db483402e |
fix(mem,spill): close two data races the worker threads exposed
TSan failed on all three platforms in PR CI. Both races are in code this branch
added, and both were invisible to the local ladder because it has no TSan leg —
that lane is CI-only, so "macOS and Linux green" said nothing about either.
1. system_info.c: the cached system info
cbm_system_info() published its cache with a plain flag:
if (!info_cached) { cached_info = detect_...(); info_cached = 1; }
Fine while only the main thread asked. The memory-relief work started calling
cbm_mem_system_under_pressure() from the per-file extract loop, so several
workers now race on the flag AND on the struct — and a struct assignment is
not atomic, so a reader could see half of one. Reported at
system_info.c:284 against :274, from extract_worker, on macOS, ubuntu-arm and
ubuntu-latest alike.
Now a three-state atomic: whichever thread wins the compare-exchange is the
only writer of cached_info, a loser returns the copy it detected itself (the
same answer — detection only reads the OS), and later readers see the
published struct through the acquire/release pair. No mutex, so nothing has
to be initialised before first use.
2. pass_parallel.c: the spill pointer
pp_spill_active() tested ec->pctx->spill BEFORE its atomic spill_mode gate,
so it read an 8-byte pointer with no synchronisation while pp_spill_enter was
publishing that pointer under spill_mu. pp_spill_enter assigns the pointer
and only then release-stores spill_mode, so gating on the atomic first makes
the pointer read ordered by that release. Short-circuit order is load bearing
in that expression now, and the comment says so.
Found by running the TSan lane locally with halt_on_error=0 rather than CI's
halt_on_error=1: the first setting stops at the first race, so each fix would
have revealed the next one a CI round trip later. Enumerating instead showed
the second race was the last one in that suite.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
dae1e7dff2 |
style: format slow_move's attribute so both clang-format versions agree
PR CI's lint job failed on src/foundation/mem_override_libc.c:143-144 while the
local lint lane passed on the same file. The two run different formatters: CI
installs LLVM 20, the local lane uses the Homebrew build, which is now 22. A
wrapped attribute is one of the few things they lay out differently, so the
wrapped form was stable locally and a violation in CI.
__attribute__((no_builtin("memcpy", "memmove",
"memset"))) static void *slow_move(...)
Written on one line the declaration is 117 columns, over the 100-column limit,
so it has to wrap and the two versions disagree about how. Spelling the
attribute as a macro brings the declaration to 73 columns, which leaves no
wrapping decision for either version to have an opinion about. The three
siblings below it already fit inline and are left alone.
Also repins scripts/vendored-checksums.txt for MANIFEST.md itself: the security
job's vendored-integrity layer checksums the manifest too, so editing it to
record the properties scanner patch invalidated its own entry. Produced by
scripts/security-vendored.sh --update; the one changed line is the manifest
hash, which is also the check that the earlier manual scanner.c repin was right.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
1d58385b51 |
refactor(mem): route this branch's allocations through the memory core
The memory-core linter is a ratchet: raw malloc/calloc/realloc/free/strdup sites may not grow, because allocations belong on one accounted path (src/foundation/mem_core.h) rather than spread across hundreds of call sites. This branch's own work had added 35 raw sites across nine files -- the parallel k8s prep, the configlink key matching, the worker pool's counted path, the Go shared stdlib registry, the spill namespace copy, the route service array and the namespace-name arrays. All of them now use cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/ cbm_mem_strdup. One was a latent bug rather than a style point: pass_k8s freed a buffer that k8s_read_file had allocated with raw malloc, so converting only the free would have mismatched the allocator -- the read function and every free of its result are converted together. The Windows free-space path widens its argument through the memory core instead of cbm_utf8_to_wide(), which allocates with raw malloc. Two files improved enough to tighten their baselines, which the ratchet requires so the gain cannot be quietly given back: sqlite_writer.c 92->74 (the index-cell arena), pass_k8s.c 13->6 and registry.c 19->17. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c3714f1133 |
fix(mem): size the budget from free memory, and relieve on real scarcity
Indexing the linux kernel with NO ceiling set took the machine down: on a
48 GB host the default budget is 50% of TOTAL ram (24 GB), the run used
essentially all of it (24.5 GB peak RSS), and with a 12 GiB VM already
running there was nothing left. The budget never fired -- it read 22 GB
charged against 24 GB -- while the host was out of memory.
Total RAM is the CEILING; what is free right now is the CONSTRAINT.
- cbm_mem_clamp_to_available() reduces a fraction-derived budget to fit
what is actually free, reserving a quarter of it (held between 1 and
8 GB) for the OS and everything else, never below a 512 MB floor. An
explicit CBM_MEM_BUDGET_MB still wins -- it is the user's deliberate
choice -- but is warned about when it exceeds free memory, because
the failure it buys reads like a product bug rather than a setting.
- cbm_mem_should_relieve() = our budget exceeded OR the machine short.
Deliberately separate from cbm_mem_over_budget(): system pressure
must RELIEVE (spill, reclaim), never abort, since the spike may not
even be ours. The extraction loop now enters spill on real scarcity,
and the dump phase -- where the kernel run actually died, with our
charge comfortably under budget -- keys its free-heavy decision on it
too. cbm_mem_system_under_pressure() existed and had no callers.
Same scenario after: budget clamped to 18,947 MB, peak RSS 19.5 GB,
status indexed in 185.7 s, machine fine.
Spill also gets limits and cleanup, because relief that fills the disk or
leaves gigabytes behind is not relief:
- a free-space preflight (never start spilling below 10 GB free) and a
ceiling at half of what is free; past it results stay in memory and
back-pressure takes over. cbm_fs_free_bytes() is new (statvfs
f_bavail / GetDiskFreeSpaceExW quota).
- POSIX unlinks each spill file while holding it open, so the space
returns on ANY exit including SIGKILL -- a killed run cannot leave
spill behind because there is no name left to leave it under.
Windows keeps close-path removal plus a sweep where an open file
refuses deletion, which is exactly the liveness test.
- the staging-database sweep now reclaims orphans of EVERY project in
the cache directory, not just the one being indexed. A kernel index
killed by the OOM killer used to park ~15 GB until someone re-indexed
the kernel, and indefinitely if nobody did. The per-stage lock probe
still keeps a live writer's stage, whichever project owns it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
92e7e6a913 |
fix: keep large thread-locals off the static TLS block
Indexing broke on x86-64 Linux: the index worker refused to start and
SIGKILLed its own process group, so every venue reported nothing but
"index worker ended with killed (exit=-1, signal=9)".
glibc places a thread's static TLS block inside the stack allocation it
makes for that thread. This branch's new thread-local caches -- a
512-slot field cache (~24 KB), per-depth tree cursors (~4 KB) and two
parked CBMArenas (~8 KB; an arena carries blocks[256] + block_sizes[256])
-- took static TLS from 47,905 to 93,513 bytes. The 64 KB parent-death
watchdog thread then could not be created at all, and a worker without
process-tree containment correctly refuses to index.
Four layers, because there were four things wrong:
- The big thread-locals move to the heap behind a thread-local POINTER,
allocated once per thread and reused, each with a correct fallback if
allocation fails (the field cache and cursor pool are caches; a miss
is slower, never wrong). Static TLS is back to 52,233 bytes.
- PARENT_WATCHDOG_STACK_SIZE 64 KB -> 256 KB. "The watchdog only polls,
a tiny stack suffices" was true when written and false once TLS
reached ~50 KB, because the TLS block comes out of that same
allocation.
- cbm_thread_create retries with the default stack on EINVAL. A stack
size is a hint about what a thread needs; the platform refusing the
hint is not a reason to fail to create the thread.
- tests/test_thread_stack_tls_contract.sh (Step 5a of scripts/test.sh)
keeps static TLS a quarter clear of the smallest requested stack,
reading that smallest size from the sources so a new small stack
tightens the gate automatically.
Two diagnostics that made this findable at all, and that stay:
smoke-test.sh prints the worker log its failure message names (CI deletes
that sandbox with the job, so the one artifact naming the cause was
unreadable), and worker_containment_unavailable() now reports WHICH of
its conditions failed, with pid/pgrp/ppid.
Verified in the amd64 container, the venue nothing on the local ladder
covers: baseline main indexes 9039 nodes/15067 edges, this branch before
the fix refuses to start the worker, and after it indexes 9039/15067.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
959f4de50c |
perf: cut the allocation churn and repeated work the sanitizer found
Every item here was measured on a real corpus before and after, not
reasoned about. Go corpus, event lane, first run -> now: 245.5 M
allocations -> 134.8 M, 190 GB cumulative -> 29.3 GB, 158 GB never
written -> 9.2 GB, 132 M churn pairs -> 35.7 M, 93 GB of allocator byte
work -> 0.84 GB. Wall clock: Go 40.7 s -> 19.3 s, linux kernel 239.6 s ->
194.6 s, and all ten LSP languages faster with none slower.
Graph output is unchanged, proven rather than assumed: per-class multiset
digests over the full kernel index (8,563,471 nodes identical, 18/20 node
classes and 18/24 edge classes byte-identical, the rest differing only in
the classes a same-binary re-run also perturbs through worker merge order).
The changes, each with its own reason:
arena an exact arena that is appended to later restarts growth at
8 KB instead of the 64 KB default — the cross-file pass adds a
few resolved calls to a compacted result, so the default block
was ~60 KB untouched per result, 0.5 GB of the Go worker peak.
cbm_arena_init_lazy defers the first block for arenas that are
usually never used at all.
scratch per-thread reuse instead of allocate-and-free per unit of
work: result_compact (8 MB kept), the cross-file pass (4 MB),
extract_defs' walk stack, and the sqlite index cells (a 1 MB
arena replacing 12.1 M malloc/free pairs on Go). Walk frames
moving into scratch is why pass_k8s now opens the keep window
explicitly: without it every YAML file opened a 512 KB scratch
on the main thread, +3 GB.
sqlite index cells are written in place — building the record and
copying it into the cell was a second allocation per entry,
16 M record buffers of pure churn on Go.
registry is_test_qn and common_prefix_len in one pass, per-bucket test
flags computed once instead of per candidate comparison, and
the reachability keys arena allocated lazily.
minhash the unique-trigram set is sized from the trigram count and
skips repeats, and LSH queries reuse one seen-set per worker
instead of allocating per query.
parallel lsp_surface rows and the k8s manifest prep now run across
workers (graph writes stay sequential and ordered), the file
node is hoisted out of register_and_link_def, and configlink
prepares dependency imports once with a length prefilter.
go_lsp one sealed stdlib registry shared by all workers, published
by CAS, instead of one per worker.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
632c19d904 |
fix(mem): charge the larger of tracked-live and allocator commit
cbm_mem_charged() trusted mimalloc's committed counter. In a release build that counter never decreases on purge: prim.c only tracks decommit under `#if !MI_DEBUG && MI_SECURE<=2`, so after a large spill the budget still believed memory it had already returned was held. On the linux kernel corpus with a 15 GB budget that read 19,787 MB charged against 15,000 allowed — the run spilled and swept repeatedly while the OS had the memory back. Charging max(OS charge, tracked live bytes) gives 14,379 MB peak charged, sweep down from 14,990 MB to 5,067 MB, and no budget-pressure warnings. The reasoning and the measured numbers are at the call site, because the next reader will otherwise re-derive "the allocator says X" from scratch. Also adds cbm_mem_allocator_stats_log() behind CBM_MEM_ALLOCATOR_STATS: mimalloc's own MIMALLOC_SHOW_STATS never prints for the worker, which exits through _Exit, and mi_stats_merge does not exist in the vendored v3. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
81264814f1 |
feat(qa): memory + CPU waste sanitizer over the memory core
Answers a question the existing tooling could not: of everything we
allocate, how much is never used?
Lanes, all driven by scripts/memwaste.sh:
event every allocation observed through the memory core — size, age,
churn, and whether the bytes were ever written
access SanitizerCoverage load/store instrumentation, so "allocated and
never touched" is measured rather than inferred
scaling k vs 2k replicas through clang profile counters, to catch
super-linear work per file
padding allocator slack per size class
ratchet the numbers recorded per venue so a regression is visible
The event lane needed a foreign-free counter: on Windows the override sees
frees for blocks it never allocated (they come from the CRT), and counting
those as untracked tripped the soundness gate. mem_events now separates
them (14 daemon / 5 worker on the Go corpus).
Two portability fixes the lanes surfaced, both real:
- the enable decision must not be cached before `environ` exists. Under
-fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our
wrapper before glibc sets up environ, so the lane silently disabled
itself and produced no dump at all on Linux.
- the access map needs 48 bits: arm64 Linux hands out addresses above
2^47, which a 47-bit map dropped on the floor.
mem_profile is retired into this: it measured a strict subset and had its
own sampling story.
Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on
all three platforms report-only, and pr.yml/release.yml carry the wiring.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
172787de9c |
fix(mem): no glibc malloc_trim; the static Linux release links again
The v0.10.9 release run (34948714902) failed in both Linux portable builds: the static link found malloc, free, realloc, calloc, memalign, valloc, pvalloc and posix_memalign defined twice, once by mimalloc's override and once by libc.a(malloc.o). Nothing in the build flags had changed since v0.10.8; what changed was one call. cbm_mem_release_to_os called glibc's malloc_trim(0), and that symbol lives only in malloc.o, so the reference pulled the whole object into the link next to mimalloc's definitions. Dynamic builds never noticed: interposition resolves the duplicates at load time, which is why the whole test matrix was green. The call was pointless where it broke things: on Linux and Windows the override routes every malloc to mimalloc, so glibc's heap holds nothing to trim and mi_collect(true) is the release. The branch and its include are gone; macOS keeps its pressure-relief call, since the system heap still serves everything outside the core there. Verification: the static build (scripts/build.sh CC=gcc CXX=g++ STATIC=1) in the local arm64 container reproduced the CI failure exactly before the change and links after it, the binary reporting its version; mem and parallel suites green on the host. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
d89ccabef0 |
fix(mem): the parked result image is declared defined for MemorySanitizer
The MSan lane on
|
||
|
Martin Vogel
|
9e69cb7d9b |
fix(mem): free the thread locale at exit; a settling gate keeps its latch
Two CI-only lanes went red on
|
||
|
Martin Vogel
|
9724d90304 |
fix(mem): dedicated SQLite heap only in the worker; guard the committed stat
The daemon leaked 180 KB per query. The Linux soak's quick leg went from 11 to 144 MB of RSS in ten minutes (13.1x, 628 MB/hr), mimalloc's committed bytes tracking RSS almost byte for byte and the growth linear in the query count, file descriptors flat -- where the same leg had been flat at 15 MB in July. The cause is the previous commit's SQLite heap: one mimalloc heap per thread, created on first use. The daemon runs a thread per connection, so every query ran on a fresh thread with a fresh heap, and every block the shared connection keeps beyond the request -- page cache pages, the statement cache, schema objects -- was allocated in that heap's pages. When the thread exited, its heap was deleted and those pages were left behind holding a few live blocks each: committed for as long as the connection lives, a few pages per query, never reused by anyone. The dedicated heap exists for one reason: the index worker's default heap holds the graph (40M+ blocks), and SQLite churn on that heap paid a page walk per allocation during the coverage publish (132 s vs 9.7 s on the kernel). That process has long-lived threads and exits when the index is done. So the dedicated heap is now a switch, off by default, that the index worker turns on at its entry (main.c, --index-worker); everywhere else SQLite allocates from the calling thread's default heap exactly as before the previous commit. The switch exists in every build and changes nothing where the allocator binds are compiled out. Linux soak, quick leg, before -> after: RSS 11 -> 144 MB in ten minutes (13.1x, 628 MB/hr) -> 10.6 -> 11.5 MB and flat from the first minute over 564 queries with reindexes every two minutes; query-leak leg: 11.6 MB flat over 1,376 queries; committed heap 6.8 MB flat. The same soak showed a second thing. From its second sample on, the query-leak leg reported a committed heap of 2^64 - 121 MB: mimalloc's committed statistic is a signed counter merged per thread at thread exit, and a process whose long-lived thread commits what its short-lived threads free reads it negative -- cast to size_t on the way out. That figure is one of the two the budget takes the larger of. cbm_mem_allocator_committed now reports 0 for a reading above SIZE_MAX/2, so the charge falls back to the OS number instead of a 16 EB budget breach, and the diagnostics report the committed pair through it (the peak never undercuts the current value and never carries a wrapped one), the way they already did for RSS. Detector: the soak's quick and query-leak legs (scripts/soak-legs.sh), the release-gating sequence in every venue -- RED on the previous tree (13.1x), GREEN on this one. A unit test cannot see this class: the test build keeps SQLite on libc so the sanitizers see every block. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7b77fd4834 |
feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.
WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)
The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.
WHAT CHANGED
Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.
Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.
Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.
The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.
The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.
Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.
Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).
The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.
The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.
Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.
SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).
Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.
Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.
The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.
Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.
PROOF
Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
jitter (gRPC Route nodes: two identical runs differ by 21).
Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
surface rows, nodes/edges inside the Route jitter.
Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
inside the +-400 run-to-run band the kernel shows between identical
runs.
Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
outside what spill can move). Nodes identical; edges 15,773,598.
Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
every phase mark under 15,000 MB by every metric, but the process
RSS high-water mark reached 16.9 GB inside the semantic pass (class
peak 4.9 GB against 0.8 GB live).
Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
stays under 15,000 MB through the whole semantic pass (14,018 ->
14,381 MB across its sub-phases, 6 batches of 125,644 functions);
every phase mark is under budget by every metric (max footprint
12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
15.65 GB and is set in extraction, at the moment the gate latches
(15,002 MB charged) and before the first sweep lands -- a 4%
overshoot of in-flight work, no longer the semantic pass. Nodes
identical, edges 15,772,913, zero foreign blocks.
Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
enters at 14,062 MB (near_budget); the charged footprint never
exceeds 11.8 GB at any mark; the process RSS high-water mark is
15.1 GB in extraction and 15.3 GB during the final write-out --
1-2% over, the in-flight window and the dump transient. Nodes
identical, edges 15,772,543, zero foreign blocks.
Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
peak 998 MB RSS, graph identical to the baseline.
BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)
corpus wall s peak RSS GB nodes / edges / CALLS
perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3
php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8%
rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369
c 5.9 -> 5.7 0.16 -> 0.19 identical
kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1%
django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4%
go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2%
java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6%
csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6%
typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130
kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1%
No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (
|
||
|
Martin Vogel
|
da3258f3c7 |
feat(mem): a central allocation core, a linter that enforces it, and phase attribution in the index pipeline
Memory in this project is now allocated through ONE core. This lands the
core, the gate that keeps new code on it, and the instrumentation that lets
an index run say which class of memory grew in which pass -- the three things
the 2026-09-13 audit found missing.
WHAT THE AUDIT FOUND
There was no centralized allocation layer. foundation/mem.h was policy and
measurement only (budget, RSS, over_budget, memory map, phase marks) with no
alloc/free wrapper, so the budget could be observed after the fact and never
enforced. Raw allocator calls in the tree: 4086 across 92 files. The
extraction engine had already adopted arenas (1301 call sites); the graph
buffer -- which holds the 35 GB peak of a kernel index -- used none,
allocating every 64 B node, 48 B edge and every name/qualified_name/
properties_json individually.
Two further facts made a self-accounting core the only workable design:
1. cbm_mem_map_collect() walks THIS thread's mimalloc heap only. Walking
the process-wide heap is a data race TSan caught on macOS, so it is
deliberately not done, and an 18-worker index attributes almost
nothing -- the rest lands in residual.
2. The mimalloc global override is ON for Linux/MinGW and permanently OFF
for macOS (two-level namespace: this binary's free becomes mi_free while
system libraries keep allocating from the system zone, and a pointer
crossing that boundary aborts). On macOS ordinary malloc is served by
the system allocator; the startup audit reports owned_classes=0/6. Any
accounting that assumes mimalloc owns the pointer is blind on an entire
platform.
THE CORE (src/foundation/mem_core.{h,c})
cbm_alloc / cbm_calloc / cbm_realloc / cbm_mem_strdup / cbm_free, each tagged
with an allocation class (gbuf_node, gbuf_edge, gbuf_string, gbuf_index,
extract, semantic, dump, store, other). Per-class live bytes, live blocks and
peak, kept in atomics -- thread-safe by construction, identical on every
platform, independent of which allocator serves malloc. libc semantics are
preserved exactly (alloc(0) is freeable, free(NULL) is a no-op, realloc(NULL)
allocates, a failed realloc leaves the block intact) so adoption is a
mechanical rename and never a behaviour change.
No per-allocation header: a {class,size} prefix costs 16 bytes on ~100M blocks
at kernel scale, 1.6 GB of overhead to measure a memory problem. Sizes come
from the platform usable-size query (malloc_size / malloc_usable_size /
_msize), which is correct under either allocator and free. Where no query
exists (BSD) the request size is charged, which understates -- the safe
direction for a diagnostic. A mismatched class on free clamps at zero rather
than wrapping, so a small drift can never masquerade as a colossal leak.
Arena-backed allocators report in bulk through cbm_mem_class_add_external /
remove_external so their memory appears in the same table; the extraction
engine must not be rewritten to per-object allocation, that would undo the
batching that keeps its allocation count low.
THE LINTER (scripts/lint-memory-core.py, wired into make lint-ci)
Raw malloc/calloc/realloc/free/strdup/strndup outside the core is a defect.
4086 sites cannot migrate at once, so the gate is a ratchet: a checked-in
baseline (scripts/memory-core-baseline.txt) records each file's count and
the build goes red when any file grows, or a new file appears with any.
Files only ever go down; the baseline line is lowered in the change that
migrates the file, and --strict fails a stale baseline. Comments and string
literals are stripped first, so prose that mentions malloc( cannot trip it --
the security audit already bit on exactly that with fork(. Proven both ways:
one appended malloc turns it red naming the file (83 -> 85), restoring turns
it green. Scans src/ and internal/cbm/: cli, mcp, daemon, store, pipeline.
PHASE ATTRIBUTION (src/pipeline/pipeline.c)
cbm_mem_phase_mark and the new cbm_mem_class_log now fire at pipeline.begin
and at every pass.timing site: parallel_extract, the six sequential passes,
the eight predump passes, tests and dump_and_persist, with peaks reset per
index. Both instruments existed in foundation/ and were wired only into MCP
request handling, never into the index pipeline -- which is where the memory
is. Locating the kernel peak this week required an external RSS sampler
because nothing in-process could say which phase it was in.
PRESSURE PRIMITIVES (system_info.c, mem.{c,h}, platform.h)
cbm_system_available_ram() -- macOS host_statistics64 (free + inactive +
purgeable), Linux MemAvailable, Windows GlobalMemoryStatusEx; 0 when unknown,
never cached -- and cbm_mem_system_under_pressure(), true below 12.5% of RAM
available and false when unknown so nothing ever aborts on a guess. Landed
UNWIRED: an earlier draft used them to let the over-budget latch press on
whenever the system was not under pressure, and that permitted the very
overshoot this work exists to end (Decision A's tests at
test_pipeline.c:13481 and :13583 went red under it, correctly). They are the
backstop for the next step, admission control, which keeps a run under budget
before an allocation rather than measuring it after.
mem suite 10 new tests green; pipeline + mem 339 passed 0 failed; lint-ci
clean including the new gate.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
9c6f45bda7 |
Merge pull request #2030 from experticket-admin/fix/windows-worker-job-memory-limit
fix(windows): enforce index worker memory budgets |