trending/codebase-memory-mcp
1
0
Fork 0
mirror of https://github.com/DeusData/codebase-memory-mcp.git synced 2026-11-11 09:25:35 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,454 Commits 631 Branches 56 Tags
867e5f883df4f33db168d74d9ffca73a3f542edf
Commit Graph
277 Commits
Author SHA1 Message Date
Martin Vogel 867e5f883d Merge origin/main into the smart-CI branch 
Ten files changed on both sides; every conflict was a union of independent
additions:

- workflows (_lint.yml, _test.yml): keep the branch's lane `if:` and main's
  pinned ubuntu-24.04 runner with its reason.
- internal/cbm/cbm.h: main's callable-identity and module-doc fields plus the
  branch's configured-test fields.
- Makefile.cbm: both new test files; prod_lsp_all.o keeps main's
  .build-config dependency and the branch's coverage flags.
- scripts/test.sh: main's setup-scripts contract becomes Step 0y3, ahead of
  the branch's 0z series and the contracts-only exit.
- src/foundation/subprocess.c: main's cbm_posix_close_fds_from (it closes the
  error descriptor too).
- src/mcp/mcp.c: main's repository default base for detect_changes, with the
  early returns released through the branch's detect_release.
- src/pipeline/lsp_resolve.h, tests/test_main.c: both includes, suites and
  start-up steps (main's git environment reset first).
- scripts/memory-core-baseline.txt: recounted on the merged tree and lowered
  (mcp.c 773, config_toml_edit.c 73).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-10-05 18:53:37 +02:00
Martin Vogel c5abb64943 feat(test-impact): detect_changes scope:"tests" end to end 
One request pins the change (merge base .. HEAD), builds the graph and the
runnable test registry of exactly the HEAD snapshot, seeds the graph from
the change, walks what depends on it and answers which tests to run
(cbm.test_impact). Missing or doubtful evidence never narrows: it becomes a
run-all answer that names why.

Snapshot (classified ledger -> frozen graph -> registry):
- one shared language classifier for discovery and pinned blobs (raw bytes
  on every platform); the classified ledger feeds the frozen build, which
  indexes exactly the listed files and records partial parses as per-file
  PARSE_GAP instead of refusing the graph;
- the pinned tree skips symlinks and submodule links (counted), as
  discovery never indexes them;
- the test model scopes uncertainty to the suites of its file (they run
  whole), reads line splices where they occur, treats a mid-file
  `#ifndef X` + `#define X` as an ordinary condition, and no longer reads a
  plain identifier named like a macro as an invocation. On this repository
  the registry is now narrowable and matches a text census exactly
  (8,365 cases, 141 runner suites).

Seeding and reach (port of the measured replay, rule B):
- hunk -> overlapping definitions, comment-only hunks dropped, pure
  deletions -> enclosing definition, no-definition hunks -> whole file,
  deleted names found by text, header changes propagated through the
  include closure and the headers' macros (pastes included), file-level
  escalation for unclassifiable header lines;
- inbound fixpoint walk over CALLS, ASYNC_CALLS, USAGE, CALL_REFERENCE,
  READS (+ routes); cases, suites and the runner's entry points are sinks;
  a reached runner main runs everything (RUNNER_REACHED);
- a parse gap whose text names a reached definition reaches the
  definition around it (or its file);
- macro-registered tests run their suites whole; cases and suites map to
  graph nodes by their start line.
The line patterns reproduce the reference regexes exactly on 1,383,850
lines of this repository and 37,167,926 kernel lines; on 24 replayed
changes the engine selects at least what the reference selected (13
identical, the rest only conservative supersets on the same graph).

Also:
- configured test declarations that cannot be mapped degrade their file
  (no configured roles, a test_declarations diagnostic) instead of
  aborting the whole index;
- the selector keeps other runners' suites out of the selection, and the
  result codec judges the model by narrowability like the selector;
- a change to .codebase-memory.json runs everything (built-in rule);
- .codebase-memory.json with the c-cbm preset and this repository's rules.

Gate fixes (suites, lint-ci, Windows cross-compile):
- the request runs on its own thread with a default-size stack (daemon
  tool calls run on 256 KiB threads, where the in-process build died);
- a request-supplied config_path is no longer followed by an unlink of an
  uninitialized path buffer;
- the inventory's frame-local reader no longer stays reachable from the
  caller's source after the filter call;
- FILE_ID_INFO needs _WIN32_WINNT >= 0x0602; the pinned-tree header pins
  it before any system header (toolchains defaulting lower failed);
- memory-core ratchet: the win_utf8 converters gain core-backed
  `_in(cls, ...)` forms over one implementation (the legacy names keep
  the C library for unmigrated callers), the Git facts module uses them,
  and the Windows spawn path releases everything through one failure
  exit; pipeline, extraction and config allocations that this work
  touched moved to the core in matched pairs; baselines tightened;
- the branch's C files formatted with the project's clang-format.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-10-05 02:15:08 +02:00
Martin Vogel 8ab555ee38 Merge pull request #2253 from DeusData/distill/1245-cypher-partial-where 
fix(cypher): stop the early WHERE pruning rows it cannot judge yet
 2026-10-04 21:01:04 +02:00
Martin Vogel 7bb1de0ed5 Merge pull request #2479 from DeusData/fix/workspace-resolved-home 
fix(workspace): compare the home directory in its canonical form
 2026-10-04 18:03:21 +02:00
Martin Vogel f40b1a564b Merge pull request #2475 from DeusData/fix/regex-compile-size-budget 
fix(regex): size a pattern before compiling and refuse oversized ones
 2026-10-04 18:03:16 +02:00
Martin Vogel a245618a64 wip(ci): checkpoint hybrid test-impact implementation 
Preserve the current selector, frozen-input inventory, result codec,
coverage collection, runner filtering, and isolated Windows venue work.
Component-level regression proofs are recorded in the PR handover.

The live classified provider, public execution integration, artifact
admission, fresh oracle/shadow evidence, and full platform gates remain
incomplete. This checkpoint does not enable or claim verified narrowing.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-10-03 23:55:18 +02:00
Martin Vogel 91ba8a3aa9 fix(subprocess): close inherited descriptors with kernel range operation 
Use the Linux close_range syscall or BSD closefrom before exec, retaining
the bounded close loop when the range operation is unavailable. Add
coverage of descriptor closure, strategy selection, and forced fallback.

Refs #1484

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-10-03 18:09:22 +02:00
Martin Vogel 2e6dc0877c fix(workspace): compare the home directory in its canonical form 
cbm_workspace_home_dir() returned the raw HOME (or USERPROFILE) string,
while cbm_workspace_classify_root() compares it against a canonical,
link-resolved path. Where the home directory is reached through a link
(/home kept on another volume, a macOS firmlink, a HOME that is itself a
link) the two spellings differ and the home rule never fired, so the home
directory was not classified as the home directory.

The helper now resolves the value through cbm_canonical_path, the same
resolver every caller applies to the path it classifies, and falls back
to the raw value when the directory cannot be resolved. The result lives
in a per-thread buffer recomputed on each call, as cbm_resolve_cache_dir
does, so a changed environment is seen at once. The environment is read
with cbm_safe_getenv so a non-ASCII USERPROFILE arrives as UTF-8.

The ancestor comparison now treats "/" and "\" as the same separator: on
Windows the resolver returns backslashes while some callers normalize to
forward slashes, and both spell one directory.

Two tests cover the helper: one with a plain temp directory (already a
firmlink on macOS), one with a home reached through a symbolic link
(POSIX; creating a link needs a privilege on Windows). The existing
explicit-approval test in the mcp suite now presents its session root in
resolved form, as the daemon does; with the raw spelling it would have
passed as an ordinary root and stopped exercising the approval path.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-10-02 05:34:41 +02:00
Martin Vogel 9db7de7bae fix(regex): size a pattern before compiling and refuse oversized ones 
cbm_regcomp() handed every pattern straight to the platform compiler
(libc regcomp on POSIX, the vendored TRE on Windows). Both expand bounded
repetition at compile time, so the compiled size of a pattern is not
proportional to its length. Measured on macOS libc (TRE-derived) with
REG_EXTENDED|REG_NOSUB, the compiled form costs about 0.6 KB per expanded
atom (1.1 KB with submatch tracking, 1.7 KB with REG_ICASE), a K-way
alternation adds about 70 B x K^2 for its position sets, and an
alternation under an interval pays that for every copy.

The wrapper now sizes a pattern in one linear pass before compiling it:
an atom costs 1, a group costs its content, an interval multiplies the
element it applies to by its upper bound, and a K-way alternation is
charged K^2/8 on top of its branches. Arithmetic saturates and group
nesting is bounded. A pattern longer than 32768 bytes, or whose estimate
exceeds 32768 units, is refused with CBM_REG_ETOOBIG before anything is
allocated. The heaviest accepted compile is a repeated alternation at
about 30 MB with REG_NOSUB (about 90 MB under the ICASE-plus-submatch
flag set no call site uses). Ordinary intervals, the pipeline's fixed
patterns and an alternation of 400 twenty-character identifiers remain
accepted.

The refusal reads as a normal compile error at every call site.
search_code now reports an unusable path_filter as a tool error instead
of silently dropping the filter, search_graph reports a failed row scan
instead of returning an empty result, and query_graph carries the reason
in the result's warning field; all three name it.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-10-02 01:55:34 +02:00
Martin Vogel 48acb559e6 Merge pull request #2405 from DeusData/fix/internal-db-enumeration 
fix(cli): never treat internal cache stores as project indexes
 2026-09-30 22:21:11 +02:00
Martin Vogel d0ec2313d9 Merge pull request #2394 from DeusData/fix/issue-2184-post-extraction-budget 
fix(pipeline): spill before the post-extraction phases when they would cross the budget (#2184)
 2026-09-30 22:20:56 +02:00
Martin Vogel 94097f0727 Merge pull request #2341 from DeusData/fix/git-spawn-local-env 
fix(git): clear git's repository-local env for cbm's own git children (#2003)
 2026-09-30 22:19:32 +02:00
Martin VogelandAndrew Hundt dc5293d71e fix(cypher): stop the early WHERE pruning rows it cannot judge yet 
Distilled from #1245 by Andrew Hundt (upstream b835b5e8), re-derived on
main's cypher.c rather than cherry-picked -- the file has moved ~9k lines
since the branch point.

The WHERE clause runs twice: once early, on seed rows where only the
first alias is bound, to prune; and once late, when every alias is bound,
to decide. The early pass used the same two-valued evaluator as the late
one. That evaluator answers "true" for a comparison on an alias it cannot
see -- the permissive default that makes the late pass work -- but NOT and
XOR invert whatever they are given, so

    MATCH (a)-[:CALLS]->(b) WHERE NOT b.name CONTAINS "x"

evaluated NOT(true) on every seed and pruned every row. Zero results, no
error, for a query that is correct.

The early pass now uses a three-valued evaluator. A condition on an alias
that is not yet bound is UNKNOWN; NOT of UNKNOWN stays UNKNOWN; AND and OR
short-circuit only on a definite operand; XOR with any UNKNOWN operand is
UNKNOWN. Only a definite FALSE prunes. The late pass is unchanged and
still gives the final two-valued answer, so nothing that was correct
before changes.

Three deliberate differences from the upstream diff:

  - the unbound-alias rule lives in one leaf, eval_condition_partial, so
    the expression-tree path and main's legacy flat AND/OR path cannot
    drift apart;
  - that leaf guards c->variable == NULL. On main a condition whose left
    side is a function of literals, coalesce(0, 1) >= 2, carries no
    variable at all; the upstream version would have passed NULL to
    strcmp;
  - the flat path keeps main's rule that an empty condition list is true.

RED before the fix, all four with row_count == 0, expected 1:
cypher_exec_where_not_on_relationship_target,
cypher_exec_where_mixed_alias_and, ..._or, ..._xor. GREEN after: cypher
208 passed; cypher + mcp 526 passed, 4 skipped (the Windows-only skips).

eval_expr_partial recurses and is added to CBM_RECURSION_WHITELIST.

Co-authored-by: Andrew Hundt <ATHundt@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-28 00:18:31 +02:00
Martin Vogel 460153204b fix(cli): never treat internal cache stores as project indexes 
The cache directory holds two internal SQLite stores next to the project
indexes: the user's settings (_config.db) and the cross-repo store
(_cross_repo.db). cbm_list_indexes, cbm_remove_indexes and
count_db_indexes matched every "*.db" there. As a result:

- install --reset-indexes and uninstall (when index deletion is confirmed)
  deleted the user's config (every `config set` value silently reverted
  to its default) and the cross-repo store along with the indexes;
- the "Found N index(es)" listing and count over-reported by up to two.

pass_cross_repo.c already excluded both stores by exact filename, but in
its own copy of the check. Centralise the rule in foundation/str_util:
cbm_is_internal_cache_db() (exact names, so a real project whose name
starts with "_" or contains "config" is still an index) and
cbm_is_project_index_db(). Every enumeration of the cache directory's
.db files (list, count, remove, cross-repo target scan) now calls it, and
cbm_config_open() builds its path from the same CBM_CONFIG_DB_FILENAME
constant.

Proof on a real isolated cache (one indexed Go repo, `config set
auto_index true`, a _cross_repo.db present):
  before: "Found 3 index(es)", "Removed 3 index(es)", cache left empty,
          `config get auto_index` -> false
  after:  "Found 1 index(es)", "Removed 1 index(es)", _config.db and
          _cross_repo.db kept, `config get auto_index` -> true

Test: cli::cli_index_enumeration_skips_internal_cache_dbs (RED on main,
green with the fix, RED again on revert).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-28 00:02:09 +02:00
Martin Vogel 26cbccd998 fix(pipeline): spill before the post-extraction phases when they would cross the budget (#2184) 
Spill mode was entered only DURING extraction, when the charge crossed
budget - budget/16, and the final sweep at extraction end ran only if that
latch had already fired. A run that ended extraction just under the latch
kept every result in memory, and the phases that cannot spill -- registry
build, cross-LSP prepare (all_defs, per-language registries, surface rows,
module index) and resolve (edges into the graph buffer, cross-LSP appends
into the cached results) -- then grew the process well past the budget.
openclaw at 8 workers with a 4079 MB budget: 3678 MB charged at extraction
end, 4076 after registry build, 4933 after cross-LSP prepare, 5829 in
resolve (+43 %). The same repo at 18 workers latched during extraction and
stayed under budget.

Extraction end now projects the post-extraction growth from the result
counts (defs, calls + usages, files) and enters spill mode when charged +
growth would cross the latch, so the existing final sweep parks every
cached result before registry build. The per-unit costs (1280 B/def,
160 B/ref, 8 KiB/file) are fitted on the measured charge growth from the
parallel_extract to the parallel_resolve mark on five local corpora
(go, django, kotlin, rust, php) and cover openclaw; the model under-reads by
at most 7 % (inside the budget/16 margin) and over-reads by up to 46 %, the
safe side: a spill never changes graph content. The decision is a pure
function of counts (no RSS-window sampling) and is logged as
mem.post_extract.projection.

A charge test seam (cbm_mem_set_charged_for_tests) pins cbm_mem_charged(),
which otherwise reads the process footprint, so the "ended just under the
latch" case is testable deterministically.

Proof (release build, M5 Pro, CBM_WORKERS=8, peak charged = the budget
metric's high-water mark):
  go, CBM_MEM_BUDGET_MB=2000   before: extract 1348 -> registry 1497 ->
    lsp_prepare 2168 -> resolve 2660 MB (+33 % over budget, no spill)
    after: projection charged 1335 + growth 1909 > line 1875 -> spill,
    21,882 results parked; extract 413 -> registry 558 -> lsp_prepare 1151
    -> resolve 1297, semantic 1746 MB peak: every phase under budget.
    Node/edge dumps byte-identical (335,191 nodes / 1,813,857 edges); wall
    29.2 s vs 29.0 s.
  go, 40000 MB (generous): decision=keep, no spill, identical dumps, wall
    27.4 s vs 27.5 s.
  django, 400 MB: resolve mark 455 -> 277 MB after the spill, dumps
    identical; the remaining peak (476 MB) is the semantic pass's retained
    set on top of the graph floor, which this change does not address.

Refs #2184

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-28 00:02:01 +02:00
News ed991154f2 fix(windows): preserve mkstemp errors through cleanup 
Signed-off-by: News <knewstimek@users.noreply.github.com>
 2026-09-26 20:30:02 +09:00
Martin Vogel 40dad610ee fix(git): clear git's repository-local env for cbm's own git children (#2003) 
Git exports repository-local variables (GIT_DIR, GIT_WORK_TREE,
GIT_INDEX_FILE, GIT_COMMON_DIR, ...) into hooks, and some editors export
them into their terminals. They take precedence over `git -C <dir>`, and
every git child cbm spawned inherited them. A cbm started from such an
environment therefore resolved every project against the CALLER's
repository: git_context reported a plain directory as is_git=true with
the caller's worktree root, branch and HEAD, the history pass attributed
the caller's commits to it, and the watcher / detect_changes /
artifact git calls read (and `config merge.ours.driver` could write) the
wrong repository.

Fix: one production helper, src/foundation/git_env.{c,h}, holds the
15-variable list `git rev-parse --local-env-vars` prints and builds the
child environment with exactly those removed -- a POSIX envp array or a
Windows CREATE_UNICODE_ENVIRONMENT block (case-insensitive names). It is
applied per spawn; cbm's own environment is never modified (no unsetenv
in the parent, which would race other threads):

- cbm_popen_git(cmd): read-mode popen for git command lines. POSIX spawns
  /bin/sh -c via posix_spawn with the scrubbed envp (stdin /dev/null,
  pipe ends close-on-exec, reaped by cbm_pclose); Windows passes the block
  to the existing handle-isolated CreateProcessW spawn. Fails closed.
- cbm_proc_opts_t.strip_git_repo_env: cbm_subprocess_spawn passes the
  scrubbed envp to posix_spawnp / the fork+exec child, or the block to
  CreateProcessW. Fails closed.
- Every git spawn uses them: git_context, pass_githistory, artifact
  (4 sites), the watcher's git runner, and detect_changes' git commands.
  search_code's grep scan keeps the inherited environment.

Tests (RED before, green after, RED on revert of the helper and of the
call sites):
- git_context::git_context_ignores_inherited_repo_env -- GIT_DIR and the
  hook set pointing at a decoy repo: a plain dir is is_git=false, a
  fixture repo reports its own HEAD and branch, the decoy is untouched.
- git_context::githistory_ignores_inherited_repo_env -- the history pass
  on a plain dir sees 0 commits (was the decoy's 4).
- subprocess::subprocess_strip_git_repo_env_is_per_child and
  popen_git_strips_repo_env_and_reports_exit_status -- all 15 variables
  are gone in the child, unrelated variables pass through, the parent
  environment is unchanged, and cbm_pclose keeps pclose() status
  semantics.

Proof (macOS arm64, GIT_DIR/GIT_WORK_TREE/GIT_INDEX_FILE/GIT_PREFIX
exported at a decoy clone, `cli index_status verbose=true`): a plain dir
went from is_git=true, worktree_root=<decoy>, branch=<decoy's> (v0.11.0)
to is_git=false; a fixture repo went from the decoy's root/branch/HEAD
to its own root and branch.

Refs #2003

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-25 18:37:16 +02:00
News 1d2bbc4259 fix(windows): preserve worker recovery errors 
Signed-off-by: News <knewstimek@users.noreply.github.com>
 2026-09-22 22:34:11 +09:00
Martin Vogel 1a3425536b fix(graph): quality round after the v0.11.0 probe — deterministic resolution, findability, coverage, names 
Nine findings from the 2026-09-16 graph-quality probe over the eleven bench corpora, each with a
test that fails without its fix and a before/after measurement on the real corpus.

Deterministic resolution (O9). Two indexes of one kernel tree differed in 5,821 IMPORTS and 632
CALLS edges. Three order/load dependencies carried it: the `#include` target was the first by-name
hit whose path ends with the include path — a bucket in worker merge order (include/linux/device.h
vs tools/virtio/linux/device.h); the registry broke suffix_match score ties by bucket position; the
LSP budget-share rule measured wall time. Include targets now prefer the File node, the least nested
path, then the lexicographic path; registry ties prefer the least nested QN, then the smaller QN; the
budget share is measured in thread CPU time like the budget itself. Two kernel indexes are now
identical (8,529,723 nodes, 1,974,940 CALLS, zero churn), as are two elasticsearch indexes.

Findability. search_graph's BM25 path honours the `label` filter and ranks an exact-name definition
above test methods that repeat the token (15/15 probe queries).

Coverage. The parse-coverage gap tree subtracts definitions the walker recovered inside an error
region (multi-line only); kernel/sched/core.c reports 476 flagged lines instead of 7,674.

Precision. Config-language callers (Makefile, CMake, YAML) no longer bind unique_name into code
(kernel Makefile -> C CALLS 195 -> 0). JS-family token names ({}, 1, (x: T) => U) are not
definitions; computed, quoted and escaped member keys stay (TypeScript 204 junk -> 0, 1,782 real
names kept). Names are cut at their first line (elasticsearch YAML 8,306 -> 0); a comment inside an
argument list is not an argument, so no Route node is named by a Java block comment; the wildcard
route `/*` is still a route. HTML/Vue/Svelte/Astro embedded member calls stay unbound.

Recall. A Python call through a self- or cls-rooted attribute keeps a specific unique_name member
target (Django +751, elasticsearch +3,288 CALLS, no new callee names).

Robustness. Edge properties never carry a split UTF-8 sequence (the 120-byte expression cut and the
JSON escaper both trim a partial code point). CLI array flags accept a JSON-array literal. Cypher
seeds a single-hop pattern from its selective end (kernel `MATCH (a)-[:CALLS]->(b) WHERE b.name =
'_printk'`: execution limit -> 7.9 s).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-21 20:29:27 +02:00
News 25d08258e5 fix(windows): keep aligned allocations on mimalloc 
Signed-off-by: News <knewstimek@users.noreply.github.com>
 2026-09-21 01:37:51 +09:00
Martin Vogel 92abefa3f5 Merge pull request #1723 from liuchong/feat/index-resource-discovery 
feat(index): add opt-in discovery resource limits
 2026-09-20 17:17:47 +02:00
Martin Vogel 9db483402e fix(mem,spill): close two data races the worker threads exposed 
TSan failed on all three platforms in PR CI. Both races are in code this branch
added, and both were invisible to the local ladder because it has no TSan leg —
that lane is CI-only, so "macOS and Linux green" said nothing about either.

1. system_info.c: the cached system info

   cbm_system_info() published its cache with a plain flag:

       if (!info_cached) { cached_info = detect_...(); info_cached = 1; }

   Fine while only the main thread asked. The memory-relief work started calling
   cbm_mem_system_under_pressure() from the per-file extract loop, so several
   workers now race on the flag AND on the struct — and a struct assignment is
   not atomic, so a reader could see half of one. Reported at
   system_info.c:284 against :274, from extract_worker, on macOS, ubuntu-arm and
   ubuntu-latest alike.

   Now a three-state atomic: whichever thread wins the compare-exchange is the
   only writer of cached_info, a loser returns the copy it detected itself (the
   same answer — detection only reads the OS), and later readers see the
   published struct through the acquire/release pair. No mutex, so nothing has
   to be initialised before first use.

2. pass_parallel.c: the spill pointer

   pp_spill_active() tested ec->pctx->spill BEFORE its atomic spill_mode gate,
   so it read an 8-byte pointer with no synchronisation while pp_spill_enter was
   publishing that pointer under spill_mu. pp_spill_enter assigns the pointer
   and only then release-stores spill_mode, so gating on the atomic first makes
   the pointer read ordered by that release. Short-circuit order is load bearing
   in that expression now, and the comment says so.

Found by running the TSan lane locally with halt_on_error=0 rather than CI's
halt_on_error=1: the first setting stops at the first race, so each fix would
have revealed the next one a CI round trip later. Enumerating instead showed
the second race was the last one in that suite.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-19 22:24:11 +02:00
Martin Vogel dae1e7dff2 style: format slow_move's attribute so both clang-format versions agree 
PR CI's lint job failed on src/foundation/mem_override_libc.c:143-144 while the
local lint lane passed on the same file. The two run different formatters: CI
installs LLVM 20, the local lane uses the Homebrew build, which is now 22. A
wrapped attribute is one of the few things they lay out differently, so the
wrapped form was stable locally and a violation in CI.

    __attribute__((no_builtin("memcpy", "memmove",
                              "memset"))) static void *slow_move(...)

Written on one line the declaration is 117 columns, over the 100-column limit,
so it has to wrap and the two versions disagree about how. Spelling the
attribute as a macro brings the declaration to 73 columns, which leaves no
wrapping decision for either version to have an opinion about. The three
siblings below it already fit inline and are left alone.

Also repins scripts/vendored-checksums.txt for MANIFEST.md itself: the security
job's vendored-integrity layer checksums the manifest too, so editing it to
record the properties scanner patch invalidated its own entry. Produced by
scripts/security-vendored.sh --update; the one changed line is the manifest
hash, which is also the check that the earlier manual scanner.c repin was right.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-19 16:27:56 +02:00
Martin Vogel 1d58385b51 refactor(mem): route this branch's allocations through the memory core 
The memory-core linter is a ratchet: raw malloc/calloc/realloc/free/strdup
sites may not grow, because allocations belong on one accounted path
(src/foundation/mem_core.h) rather than spread across hundreds of call
sites. This branch's own work had added 35 raw sites across nine files --
the parallel k8s prep, the configlink key matching, the worker pool's
counted path, the Go shared stdlib registry, the spill namespace copy,
the route service array and the namespace-name arrays.

All of them now use cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/
cbm_mem_strdup. One was a latent bug rather than a style point:
pass_k8s freed a buffer that k8s_read_file had allocated with raw malloc,
so converting only the free would have mismatched the allocator -- the
read function and every free of its result are converted together.

The Windows free-space path widens its argument through the memory core
instead of cbm_utf8_to_wide(), which allocates with raw malloc.

Two files improved enough to tighten their baselines, which the ratchet
requires so the gain cannot be quietly given back: sqlite_writer.c 92->74
(the index-cell arena), pass_k8s.c 13->6 and registry.c 19->17.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-18 23:27:23 +02:00
Martin Vogel c3714f1133 fix(mem): size the budget from free memory, and relieve on real scarcity 
Indexing the linux kernel with NO ceiling set took the machine down: on a
48 GB host the default budget is 50% of TOTAL ram (24 GB), the run used
essentially all of it (24.5 GB peak RSS), and with a 12 GiB VM already
running there was nothing left. The budget never fired -- it read 22 GB
charged against 24 GB -- while the host was out of memory.

Total RAM is the CEILING; what is free right now is the CONSTRAINT.

  - cbm_mem_clamp_to_available() reduces a fraction-derived budget to fit
    what is actually free, reserving a quarter of it (held between 1 and
    8 GB) for the OS and everything else, never below a 512 MB floor. An
    explicit CBM_MEM_BUDGET_MB still wins -- it is the user's deliberate
    choice -- but is warned about when it exceeds free memory, because
    the failure it buys reads like a product bug rather than a setting.
  - cbm_mem_should_relieve() = our budget exceeded OR the machine short.
    Deliberately separate from cbm_mem_over_budget(): system pressure
    must RELIEVE (spill, reclaim), never abort, since the spike may not
    even be ours. The extraction loop now enters spill on real scarcity,
    and the dump phase -- where the kernel run actually died, with our
    charge comfortably under budget -- keys its free-heavy decision on it
    too. cbm_mem_system_under_pressure() existed and had no callers.

Same scenario after: budget clamped to 18,947 MB, peak RSS 19.5 GB,
status indexed in 185.7 s, machine fine.

Spill also gets limits and cleanup, because relief that fills the disk or
leaves gigabytes behind is not relief:

  - a free-space preflight (never start spilling below 10 GB free) and a
    ceiling at half of what is free; past it results stay in memory and
    back-pressure takes over. cbm_fs_free_bytes() is new (statvfs
    f_bavail / GetDiskFreeSpaceExW quota).
  - POSIX unlinks each spill file while holding it open, so the space
    returns on ANY exit including SIGKILL -- a killed run cannot leave
    spill behind because there is no name left to leave it under.
    Windows keeps close-path removal plus a sweep where an open file
    refuses deletion, which is exactly the liveness test.
  - the staging-database sweep now reclaims orphans of EVERY project in
    the cache directory, not just the one being indexed. A kernel index
    killed by the OOM killer used to park ~15 GB until someone re-indexed
    the kernel, and indefinitely if nobody did. The per-stage lock probe
    still keeps a live writer's stage, whichever project owns it.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-18 23:26:28 +02:00
Martin Vogel 92e7e6a913 fix: keep large thread-locals off the static TLS block 
Indexing broke on x86-64 Linux: the index worker refused to start and
SIGKILLed its own process group, so every venue reported nothing but
"index worker ended with killed (exit=-1, signal=9)".

glibc places a thread's static TLS block inside the stack allocation it
makes for that thread. This branch's new thread-local caches -- a
512-slot field cache (~24 KB), per-depth tree cursors (~4 KB) and two
parked CBMArenas (~8 KB; an arena carries blocks[256] + block_sizes[256])
-- took static TLS from 47,905 to 93,513 bytes. The 64 KB parent-death
watchdog thread then could not be created at all, and a worker without
process-tree containment correctly refuses to index.

Four layers, because there were four things wrong:

  - The big thread-locals move to the heap behind a thread-local POINTER,
    allocated once per thread and reused, each with a correct fallback if
    allocation fails (the field cache and cursor pool are caches; a miss
    is slower, never wrong). Static TLS is back to 52,233 bytes.
  - PARENT_WATCHDOG_STACK_SIZE 64 KB -> 256 KB. "The watchdog only polls,
    a tiny stack suffices" was true when written and false once TLS
    reached ~50 KB, because the TLS block comes out of that same
    allocation.
  - cbm_thread_create retries with the default stack on EINVAL. A stack
    size is a hint about what a thread needs; the platform refusing the
    hint is not a reason to fail to create the thread.
  - tests/test_thread_stack_tls_contract.sh (Step 5a of scripts/test.sh)
    keeps static TLS a quarter clear of the smallest requested stack,
    reading that smallest size from the sources so a new small stack
    tightens the gate automatically.

Two diagnostics that made this findable at all, and that stay:
smoke-test.sh prints the worker log its failure message names (CI deletes
that sandbox with the job, so the one artifact naming the cause was
unreadable), and worker_containment_unavailable() now reports WHICH of
its conditions failed, with pid/pgrp/ppid.

Verified in the amd64 container, the venue nothing on the local ladder
covers: baseline main indexes 9039 nodes/15067 edges, this branch before
the fix refuses to start the worker, and after it indexes 9039/15067.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-18 23:26:05 +02:00
Martin Vogel 959f4de50c perf: cut the allocation churn and repeated work the sanitizer found 
Every item here was measured on a real corpus before and after, not
reasoned about. Go corpus, event lane, first run -> now: 245.5 M
allocations -> 134.8 M, 190 GB cumulative -> 29.3 GB, 158 GB never
written -> 9.2 GB, 132 M churn pairs -> 35.7 M, 93 GB of allocator byte
work -> 0.84 GB. Wall clock: Go 40.7 s -> 19.3 s, linux kernel 239.6 s ->
194.6 s, and all ten LSP languages faster with none slower.

Graph output is unchanged, proven rather than assumed: per-class multiset
digests over the full kernel index (8,563,471 nodes identical, 18/20 node
classes and 18/24 edge classes byte-identical, the rest differing only in
the classes a same-binary re-run also perturbs through worker merge order).

The changes, each with its own reason:

  arena      an exact arena that is appended to later restarts growth at
             8 KB instead of the 64 KB default — the cross-file pass adds a
             few resolved calls to a compacted result, so the default block
             was ~60 KB untouched per result, 0.5 GB of the Go worker peak.
             cbm_arena_init_lazy defers the first block for arenas that are
             usually never used at all.
  scratch    per-thread reuse instead of allocate-and-free per unit of
             work: result_compact (8 MB kept), the cross-file pass (4 MB),
             extract_defs' walk stack, and the sqlite index cells (a 1 MB
             arena replacing 12.1 M malloc/free pairs on Go). Walk frames
             moving into scratch is why pass_k8s now opens the keep window
             explicitly: without it every YAML file opened a 512 KB scratch
             on the main thread, +3 GB.
  sqlite     index cells are written in place — building the record and
             copying it into the cell was a second allocation per entry,
             16 M record buffers of pure churn on Go.
  registry   is_test_qn and common_prefix_len in one pass, per-bucket test
             flags computed once instead of per candidate comparison, and
             the reachability keys arena allocated lazily.
  minhash    the unique-trigram set is sized from the trigram count and
             skips repeats, and LSH queries reuse one seen-set per worker
             instead of allocating per query.
  parallel   lsp_surface rows and the k8s manifest prep now run across
             workers (graph writes stay sequential and ordered), the file
             node is hoisted out of register_and_link_def, and configlink
             prepares dependency imports once with a length prefilter.
  go_lsp     one sealed stdlib registry shared by all workers, published
             by CAS, instead of one per worker.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-18 03:31:22 +02:00
Martin Vogel 632c19d904 fix(mem): charge the larger of tracked-live and allocator commit 
cbm_mem_charged() trusted mimalloc's committed counter. In a release build
that counter never decreases on purge: prim.c only tracks decommit under
`#if !MI_DEBUG && MI_SECURE<=2`, so after a large spill the budget still
believed memory it had already returned was held.

On the linux kernel corpus with a 15 GB budget that read 19,787 MB charged
against 15,000 allowed — the run spilled and swept repeatedly while the OS
had the memory back. Charging max(OS charge, tracked live bytes) gives
14,379 MB peak charged, sweep down from 14,990 MB to 5,067 MB, and no
budget-pressure warnings.

The reasoning and the measured numbers are at the call site, because the
next reader will otherwise re-derive "the allocator says X" from scratch.

Also adds cbm_mem_allocator_stats_log() behind CBM_MEM_ALLOCATOR_STATS:
mimalloc's own MIMALLOC_SHOW_STATS never prints for the worker, which
exits through _Exit, and mi_stats_merge does not exist in the vendored v3.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-18 03:30:54 +02:00
Martin Vogel 81264814f1 feat(qa): memory + CPU waste sanitizer over the memory core 
Answers a question the existing tooling could not: of everything we
allocate, how much is never used?

Lanes, all driven by scripts/memwaste.sh:
  event    every allocation observed through the memory core — size, age,
           churn, and whether the bytes were ever written
  access   SanitizerCoverage load/store instrumentation, so "allocated and
           never touched" is measured rather than inferred
  scaling  k vs 2k replicas through clang profile counters, to catch
           super-linear work per file
  padding  allocator slack per size class
  ratchet  the numbers recorded per venue so a regression is visible

The event lane needed a foreign-free counter: on Windows the override sees
frees for blocks it never allocated (they come from the CRT), and counting
those as untracked tripped the soundness gate. mem_events now separates
them (14 daemon / 5 worker on the Go corpus).

Two portability fixes the lanes surfaced, both real:
  - the enable decision must not be cached before `environ` exists. Under
    -fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our
    wrapper before glibc sets up environ, so the lane silently disabled
    itself and produced no dump at all on Linux.
  - the access map needs 48 bits: arm64 Linux hands out addresses above
    2^47, which a 47-bit map dropped on the floor.

mem_profile is retired into this: it measured a strict subset and had its
own sampling story.

Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on
all three platforms report-only, and pr.yml/release.yml carry the wiring.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-18 03:30:39 +02:00
Martin Vogel 172787de9c fix(mem): no glibc malloc_trim; the static Linux release links again 
The v0.10.9 release run (34948714902) failed in both Linux portable
builds: the static link found malloc, free, realloc, calloc, memalign,
valloc, pvalloc and posix_memalign defined twice, once by mimalloc's
override and once by libc.a(malloc.o). Nothing in the build flags had
changed since v0.10.8; what changed was one call. cbm_mem_release_to_os
called glibc's malloc_trim(0), and that symbol lives only in malloc.o, so
the reference pulled the whole object into the link next to mimalloc's
definitions. Dynamic builds never noticed: interposition resolves the
duplicates at load time, which is why the whole test matrix was green.

The call was pointless where it broke things: on Linux and Windows the
override routes every malloc to mimalloc, so glibc's heap holds nothing
to trim and mi_collect(true) is the release. The branch and its include
are gone; macOS keeps its pressure-relief call, since the system heap
still serves everything outside the core there.

Verification: the static build (scripts/build.sh CC=gcc CXX=g++
STATIC=1) in the local arm64 container reproduced the CI failure exactly
before the change and links after it, the binary reporting its version;
mem and parallel suites green on the host.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-15 18:08:36 +02:00
Martin Vogel d89ccabef0 fix(mem): the parked result image is declared defined for MemorySanitizer 
The MSan lane on 9724d903 stopped in the extraction suite's spill
round-trip test: uninitialized bytes in fwrite at offset 4087 of a
6,952-byte region, the parked result's arena block (the stack overflow
that followed is MemorySanitizer's own report path failing on the nested
bug). The park writes a result as an opaque image: the record header, a
struct copy, and the compacted arena block. Both carry padding bytes no
code ever wrote, inside structs and between objects, and MSan tracks that
mark through the compaction's memcpy, so zeroing the destination block
only moved the report (offset 6714 in the local MSan container). The
image is read back whole and never interpreted byte by byte; the padding
content is irrelevant on disk.

Under MSan the header and the block are declared defined right before
the write (__msan_unpoison, compiled in only when the build is
instrumented); every other build compiles the annotation to nothing. The
exact block is also zeroed once when it is created, so the gaps between
objects are deterministic in the spill file.

Verification: extraction, arena and parallel suites under ASan on the
fixed tree; the extraction suite in the local MSan container image
passes 348 / 0 with no sanitizer report (it reproduced the CI report exactly before the fix).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-14 21:26:06 +02:00
Martin Vogel 9e69cb7d9b fix(mem): free the thread locale at exit; a settling gate keeps its latch 
Two CI-only lanes went red on 9724d903, both this branch's own doing.

The macOS leak lane: every test passed and LeakSanitizer reported 16 to
101 objects of 1,472 bytes per test process, allocated from
cbm_arena_sprintf. The formatter creates one C locale per thread on macOS
(the vsnprintf locale lock fix) and kept it in a thread-local pointer
only, so every worker thread that ever formatted a name leaked its locale
at exit. A pthread key destructor now frees it when the thread exits; the
main thread keeps its locale until process exit.

The TSan lane on Ubuntu arm: pipeline_backpressure_futile_nap_disengages
counted 8 nap cycles against a bound of 7, the "gate re-paid per pull"
shape the test exists for. The spill shortcut reports "not over" while a
sweep is still bringing memory down, and that reading re-armed the
futility latch, so the next over-budget pull paid a full nap cycle again.
The latch now re-arms only on a genuine under-budget reading; a reading
the shortcut produced is not one.

Verification: the local macOS leak lane (make -f Makefile.cbm test-lsan)
reports zero leaks, 8050 passed / 10 skipped (the CI lane had 16-101
leaked objects per process); the local TSan lane (scripts/test.sh --tsan)
passes 1157 / 9 skipped with the nap-cycle test green (the CI lane had it
at 8 cycles); pipeline, parallel and arena suites twice and lint-ci on the
fixed tree.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-14 21:04:16 +02:00
Martin Vogel 9724d90304 fix(mem): dedicated SQLite heap only in the worker; guard the committed stat 
The daemon leaked 180 KB per query. The Linux soak's quick leg went from
11 to 144 MB of RSS in ten minutes (13.1x, 628 MB/hr), mimalloc's
committed bytes tracking RSS almost byte for byte and the growth linear
in the query count, file descriptors flat -- where the same leg had been
flat at 15 MB in July.

The cause is the previous commit's SQLite heap: one mimalloc heap per
thread, created on first use. The daemon runs a thread per connection,
so every query ran on a fresh thread with a fresh heap, and every block
the shared connection keeps beyond the request -- page cache pages, the
statement cache, schema objects -- was allocated in that heap's pages.
When the thread exited, its heap was deleted and those pages were left
behind holding a few live blocks each: committed for as long as the
connection lives, a few pages per query, never reused by anyone.

The dedicated heap exists for one reason: the index worker's default
heap holds the graph (40M+ blocks), and SQLite churn on that heap paid a
page walk per allocation during the coverage publish (132 s vs 9.7 s on
the kernel). That process has long-lived threads and exits when the
index is done. So the dedicated heap is now a switch, off by default,
that the index worker turns on at its entry (main.c, --index-worker);
everywhere else SQLite allocates from the calling thread's default heap
exactly as before the previous commit. The switch exists in every build
and changes nothing where the allocator binds are compiled out.

Linux soak, quick leg, before -> after: RSS 11 -> 144 MB in ten minutes
(13.1x, 628 MB/hr) -> 10.6 -> 11.5 MB and flat from the first minute
over 564 queries with reindexes every two minutes; query-leak leg:
11.6 MB flat over 1,376 queries; committed heap 6.8 MB flat.

The same soak showed a second thing. From its second sample on, the
query-leak leg reported a committed heap of 2^64 - 121 MB: mimalloc's
committed statistic is a signed counter merged per thread at thread
exit, and a process whose long-lived thread commits what its
short-lived threads free reads it negative -- cast to size_t on the
way out. That figure is one of the two the budget takes the larger of.
cbm_mem_allocator_committed now reports 0 for a reading above
SIZE_MAX/2, so the charge falls back to the OS number instead of a
16 EB budget breach, and the diagnostics report the committed pair
through it (the peak never undercuts the current value and never
carries a wrapped one), the way they already did for RSS.

Detector: the soak's quick and query-leak legs (scripts/soak-legs.sh),
the release-gating sequence in every venue -- RED on the previous tree
(13.1x), GREEN on this one. A unit test cannot see this class: the test
build keeps SQLite on libc so the sanitizers see every block.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-14 19:17:40 +02:00
Martin Vogel 7b77fd4834 feat(mem): memory on the core, waste cut, results spilled under budget 
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.

WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)

The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.

WHAT CHANGED

Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.

Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.

Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.

The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.

The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.

Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.

Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).

The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.

The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.

Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.

SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).

Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.

Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.

The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.

Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.

PROOF

  Go     peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
         MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
         jitter (gRPC Route nodes: two identical runs differ by 21).
  Go     CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
         on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
         surface rows, nodes/edges inside the Route jitter.
  Go     CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
         took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
  Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
         spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
         committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
         GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
         inside the +-400 run-to-run band the kernel shows between identical
         runs.
  Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
         spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
         GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
         committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
         GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
         outside what spill can move). Nodes identical; edges 15,773,598.
  Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
         every phase mark under 15,000 MB by every metric, but the process
         RSS high-water mark reached 16.9 GB inside the semantic pass (class
         peak 4.9 GB against 0.8 GB live).
  Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
         stays under 15,000 MB through the whole semantic pass (14,018 ->
         14,381 MB across its sub-phases, 6 batches of 125,644 functions);
         every phase mark is under budget by every metric (max footprint
         12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
         15.65 GB and is set in extraction, at the moment the gate latches
         (15,002 MB charged) and before the first sweep lands -- a 4%
         overshoot of in-flight work, no longer the semantic pass. Nodes
         identical, edges 15,772,913, zero foreign blocks.
  Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
         enters at 14,062 MB (near_budget); the charged footprint never
         exceeds 11.8 GB at any mark; the process RSS high-water mark is
         15.1 GB in extraction and 15.3 GB during the final write-out --
         1-2% over, the in-flight window and the dump transient. Nodes
         identical, edges 15,772,543, zero foreign blocks.
  Go     CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
         peak 998 MB RSS, graph identical to the baseline.

BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)

  corpus      wall s          peak RSS GB     nodes / edges / CALLS
  perl        5.7 -> 5.3      0.09 -> 0.05    -3 / -130 / -3
  php         6.3 -> 5.8      0.66 -> 0.20    +21.9% / +1.4% / +23.8%
  rust        6.8 -> 6.3      1.15 -> 0.67    +1 / +56 / -369
  c           5.9 -> 5.7      0.16 -> 0.19    identical
  kotlin      8.9 -> 9.4      1.55 -> 0.96    +33.5% / -22.8% / +12.1%
  django      10.9 -> 10.2    3.13 -> 1.12    +6 / -14.5% / -28.4%
  go          52.6 -> 42.9    17.2 -> 3.9     +14.2% / -10.3% / +2.2%
  java        79.8 -> 73.8    27.6 -> 11.0    -121 / -1.9% / -0.6%
  csharp      463.3 -> 128.1  27.0 -> 11.2    +299 / -0.9% / -0.6%
  typescript  33.4 -> 33.6    12.0 -> 2.5     -186 / -3.5% / +130
  kernel      294.8 -> 233.3  27.5 -> 25.1    -171 / -1.3% / -0.1%

No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (339b3f40), not this PR: a build of the base commit run
through the same driver reproduces the final graphs exactly on django and
kotlin, within the pre-existing gRPC Route jitter on go (30 nodes, 27
edges, CALLS identical), and on C# with the same node count. This PR's
own C# difference against the base is 5 CALLS (the five truncated JIT
stress files) plus one ijwhost swap and 563 edges of 5.79 M; the base
takes 448 s / 29.7 GB for that corpus.

Diagnostics that made this measurable stay in: the extract.arenas and
extract.census lines (behind CBM_MEM_PHASES=1), the CBM_MEM_RELEASE=1
release-to-OS probe, and peak_charged_mb on every mem.phase line -- the
high-water mark of the budget metric itself (cbm_mem_peak_charged), next
to the RSS peak that counts pages already purged but not yet reclaimed.

Tests: extract_compact_* and extract_spill_round_trip_keeps_every_field
(extraction), parallel_spill_mode_builds_the_same_graph (parallel: every
file parked, node count and every edge type equal to the in-memory run),
pipeline_semantic_batched_matches_unbatched (pipeline: one repo indexed
unbatched and with CBM_SEM_BATCH=5 -- byte-identical node vectors, token
vectors and SEMANTICALLY_RELATED edges),
extract_lsp_skipped_when_parse_used_ its_budget_share and
extract_walk_truncated_at_its_cpu_budget (extraction: the seams mark a
file / stop the walk, the LSP walk and the dispatcher skip it, the defs
stay), registry_overlay_chain_iterates_ and_copies_on_write (c_lsp), arena
exact-init/growth, mem charged/footprint/ pressure.

Not in this increment: run-to-run edge jitter (+-400 on the kernel, Route
class on Go) predates this work and has its own item; the incremental route
spilling; a tighter result encoding (def ids instead of QN strings in
resolved calls) is the next lever on the 9 GB the kernel's results still
take.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-14 06:11:03 +02:00
Martin Vogel da3258f3c7 feat(mem): a central allocation core, a linter that enforces it, and phase attribution in the index pipeline 
Memory in this project is now allocated through ONE core. This lands the
core, the gate that keeps new code on it, and the instrumentation that lets
an index run say which class of memory grew in which pass -- the three things
the 2026-09-13 audit found missing.

WHAT THE AUDIT FOUND

There was no centralized allocation layer. foundation/mem.h was policy and
measurement only (budget, RSS, over_budget, memory map, phase marks) with no
alloc/free wrapper, so the budget could be observed after the fact and never
enforced. Raw allocator calls in the tree: 4086 across 92 files. The
extraction engine had already adopted arenas (1301 call sites); the graph
buffer -- which holds the 35 GB peak of a kernel index -- used none,
allocating every 64 B node, 48 B edge and every name/qualified_name/
properties_json individually.

Two further facts made a self-accounting core the only workable design:

  1. cbm_mem_map_collect() walks THIS thread's mimalloc heap only. Walking
     the process-wide heap is a data race TSan caught on macOS, so it is
     deliberately not done, and an 18-worker index attributes almost
     nothing -- the rest lands in residual.
  2. The mimalloc global override is ON for Linux/MinGW and permanently OFF
     for macOS (two-level namespace: this binary's free becomes mi_free while
     system libraries keep allocating from the system zone, and a pointer
     crossing that boundary aborts). On macOS ordinary malloc is served by
     the system allocator; the startup audit reports owned_classes=0/6. Any
     accounting that assumes mimalloc owns the pointer is blind on an entire
     platform.

THE CORE (src/foundation/mem_core.{h,c})

cbm_alloc / cbm_calloc / cbm_realloc / cbm_mem_strdup / cbm_free, each tagged
with an allocation class (gbuf_node, gbuf_edge, gbuf_string, gbuf_index,
extract, semantic, dump, store, other). Per-class live bytes, live blocks and
peak, kept in atomics -- thread-safe by construction, identical on every
platform, independent of which allocator serves malloc. libc semantics are
preserved exactly (alloc(0) is freeable, free(NULL) is a no-op, realloc(NULL)
allocates, a failed realloc leaves the block intact) so adoption is a
mechanical rename and never a behaviour change.

No per-allocation header: a {class,size} prefix costs 16 bytes on ~100M blocks
at kernel scale, 1.6 GB of overhead to measure a memory problem. Sizes come
from the platform usable-size query (malloc_size / malloc_usable_size /
_msize), which is correct under either allocator and free. Where no query
exists (BSD) the request size is charged, which understates -- the safe
direction for a diagnostic. A mismatched class on free clamps at zero rather
than wrapping, so a small drift can never masquerade as a colossal leak.

Arena-backed allocators report in bulk through cbm_mem_class_add_external /
remove_external so their memory appears in the same table; the extraction
engine must not be rewritten to per-object allocation, that would undo the
batching that keeps its allocation count low.

THE LINTER (scripts/lint-memory-core.py, wired into make lint-ci)

Raw malloc/calloc/realloc/free/strdup/strndup outside the core is a defect.
4086 sites cannot migrate at once, so the gate is a ratchet: a checked-in
baseline (scripts/memory-core-baseline.txt) records each file's count and
the build goes red when any file grows, or a new file appears with any.
Files only ever go down; the baseline line is lowered in the change that
migrates the file, and --strict fails a stale baseline. Comments and string
literals are stripped first, so prose that mentions malloc( cannot trip it --
the security audit already bit on exactly that with fork(. Proven both ways:
one appended malloc turns it red naming the file (83 -> 85), restoring turns
it green. Scans src/ and internal/cbm/: cli, mcp, daemon, store, pipeline.

PHASE ATTRIBUTION (src/pipeline/pipeline.c)

cbm_mem_phase_mark and the new cbm_mem_class_log now fire at pipeline.begin
and at every pass.timing site: parallel_extract, the six sequential passes,
the eight predump passes, tests and dump_and_persist, with peaks reset per
index. Both instruments existed in foundation/ and were wired only into MCP
request handling, never into the index pipeline -- which is where the memory
is. Locating the kernel peak this week required an external RSS sampler
because nothing in-process could say which phase it was in.

PRESSURE PRIMITIVES (system_info.c, mem.{c,h}, platform.h)

cbm_system_available_ram() -- macOS host_statistics64 (free + inactive +
purgeable), Linux MemAvailable, Windows GlobalMemoryStatusEx; 0 when unknown,
never cached -- and cbm_mem_system_under_pressure(), true below 12.5% of RAM
available and false when unknown so nothing ever aborts on a guess. Landed
UNWIRED: an earlier draft used them to let the over-budget latch press on
whenever the system was not under pressure, and that permitted the very
overshoot this work exists to end (Decision A's tests at
test_pipeline.c:13481 and :13583 went red under it, correctly). They are the
backstop for the next step, admission control, which keeps a run under budget
before an allocation rather than measuring it after.

mem suite 10 new tests green; pipeline + mem 339 passed 0 failed; lint-ci
clean including the new gate.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-13 16:45:33 +02:00
Martin Vogel 9c6f45bda7 Merge pull request #2030 from experticket-admin/fix/windows-worker-job-memory-limit 
fix(windows): enforce index worker memory budgets
 2026-09-12 19:40:34 +02:00
Martin Vogel 9da4d66df2 Merge pull request #2115 from DeusData/fix/activation-guard-home-scope 
fix(cli): scope the activation guard to the install-target HOME so a sandbox install can't drain the host daemon
 2026-09-12 19:20:41 +02:00
Martin Vogel 9a460b389c fix(sha256): keep the message schedule out of the transform's frame 
sha256_transform held its 64-word message schedule as a 256-byte local.
That is large enough for ASan's use-after-return fake stack, so an
instrumented build heap-allocated it through __asan_stack_malloc on every
64-byte block -- millions of allocations to hash one large file. Under the
sanitized CI settings (detect_stack_use_after_return=1) fingerprinting a
freshly installed binary therefore took minutes, and the cli suite blew its
900s wall-clock budget and was killed as hung on the Linux arm64 leg. The
suite passes on macOS, where the fake stack is not engaged the same way,
which is why this only ever surfaced on one leg.

Move the schedule into cbm_sha256_ctx so it is allocated once per hash
instead of once per block. The computed values and the resulting digest are
identical; no sanitizer option is relaxed and use-after-return detection
stays enabled everywhere.

Verified in the Linux arm64 sanitized container under the exact CI
ASAN_OPTIONS (detect_stack_use_after_return=1:strict_string_checks=1:
detect_stack_use_after_scope=1): cli 316 passed in 356s, previously killed
at the 900s budget; daemon_version 10 passed, covering the build-fingerprint
digests that would break first on a wrong hash.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-12 15:38:20 +02:00
Martin Vogel dd454b2346 Merge pull request #2111 from DeusData/fix/1839-1864-stage-orphans 
fix(pipeline): sweep orphaned/nested .db.stage files and stop the false invalid_existing_db route (#1839, #1864)
 2026-09-12 12:45:14 +02:00
Martin Vogel 951b9ab022 Merge branch 'main' into fix/1839-1864-stage-orphans 2026-09-11 23:26:38 +02:00
Martin Vogel ca86fdb057 Merge branch 'main' into fix/1828-enospc-daemon-start-diagnostics 2026-09-11 23:26:30 +02:00
Martin Vogel 527016258e fix(extract): budget tree-sitter parses by thread CPU time, not wall clock 
The per-file tree-sitter parse was bounded by a 5s WALL-CLOCK budget. Under CI
contention a worker thread can be descheduled long enough for 5s of wall time
to elapse against near-zero CPU, so a small, perfectly parseable file was
abandoned with a false "parse timeout" and yielded ZERO defs. When that file
defined a symbol other files resolve against (e.g. celery.app.Celery.send_task
backing an ASYNC_CALLS edge, or a userpb gRPC client backing GRPC_CALLS), the
cross-file service edge then failed to classify -- surfacing as the flaky
convergence_probe / edge_types_probe service-edge assertions (count 0), a
verdict decided by wall-clock scheduling rather than the input (violates O9,
"no timeout decides a test").

Gate the parse on per-thread CPU time instead, with a generous wall ceiling as
a backstop:
- New portable helper cbm_thread_cpu_time_ns() (compat.c/.h): POSIX/macOS
  clock_gettime(CLOCK_THREAD_CPUTIME_ID), Windows GetThreadTimes (kernel+user);
  returns 0 on clock failure, degrading safely to the wall ceiling. It is NOT
  cbm_clock_gettime, which is wall on every platform (QueryPerformanceCounter
  on Windows).
- The progress callback's payload becomes {cpu_deadline_ns, wall_ceiling_ns};
  it trips when this thread's CPU time passes the budget OR wall time passes
  t0 + budget * 12 (~60s for the 5s budget). Descheduling burns wall, not CPU,
  so a starved-but-parseable file survives; a CPU-spinning parse still trips
  the CPU deadline; a genuinely stuck parse still trips the wall ceiling
  (bounded, no hang).

RED-repro (deterministic, CBM_ENABLE_TEST_SEAMS only): a thread-local wall-clock
offset armed by CBM_TEST_WALL_STALL_ON emulates a descheduled worker (wall
reads budget + 1s ahead while CPU is untouched -- above the old 1x wall budget,
below the ceiling). async_calls_celery_wall_stall_seam arms it for the Celery
DEFINITION file and asserts ASYNC_CALLS >= 1. With the fix the parse completes
and the edge resolves; with the budget reverted to wall-only + a 1x ceiling the
def file is dropped and the edge count is 0.

Verified (macOS, sanitized): edge_types_probe + convergence_probe 107 passed /
0 failed with the fix; 58 passed / 1 failed on revert (only the seam test, at
the ASYNC_CALLS assertion); 107 restored. Regression smoke (extraction
parse_coverage pipeline lang_contract grammar_regression) 698 passed / 0
failed. lint-ci clean (cppcheck + clang-format Homebrew LLVM + NOLINT); the
Windows GetThreadTimes path is compile-checked by CI (not runnable on macOS).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-11 21:40:54 +02:00
Subgurim 93111777ad merge: actualiza la base de PR2030 con main 
Signed-off-by: Subgurim <subgurim@gmail.com>
 2026-09-11 13:26:16 +02:00
Subgurim 262ab01217 fix(windows): inicializa diagnósticos y verifica el límite con control sin techo 
Signed-off-by: Subgurim <subgurim@gmail.com>
 2026-09-11 13:25:58 +02:00
Martin Vogel cc6743e9f3 Merge pull request #2060 from TheStranjer/fix/follow-own-symlinked-agent-roots 
fix(cli): follow agent-root symlinks owned by the invoking user
 2026-09-09 23:37:45 +02:00
Subgurim 2b8bd0c065 fix(windows): add headroom and diagnostics to worker memory limits 
Signed-off-by: Subgurim <subgurim@gmail.com>
 2026-09-09 16:35:05 +02:00
刘冲 21ae02154a feat(index): add opt-in discovery resource limits 
Indexing accepts whatever a repository contains. A tree carrying a vendored
monorepo, a generated dump, or a runaway build directory is discovered in
full, and the first sign of trouble is a host under memory pressure with
nothing that attributes it to indexing.

Add two opt-in limits evaluated during discovery against accepted source
files only: index_max_files and index_max_source_mb. Both default to off, so
nothing changes until an operator sets one. Crossing a limit fails the whole
attempt with a structured resource_limit_exceeded result naming the resource,
the observed value and the limit; no partial graph is published, and an
existing serving index keeps answering.

Limits are read from the CLI-managed _config.db and are not MCP request
arguments. A supervised parent replaces any caller-supplied policy before
spawning its worker, and the worker rejects a missing or incomplete contract,
so the CLI, the daemon and the supervised worker all enforce the same
decision.

Both keys reach an operator through the existing config get/set/list/reset
with no new subcommand. `set` suppresses its own generic message for them
because the policy writer names the precise reason -- so that writer speaks
on every failure it can return, including a validated value whose write then
fails on a database that cannot be written. Exiting non-zero in silence is
not an acceptable answer from a CLI.

The two shell regressions that hand-roll the supervisor's worker argv carry
that contract as well. Without it the worker exits before either guard can
observe anything, and the guard would go quietly vacuous.

Signed-off-by: 刘冲 <mail@liuchong.dev>
 2026-09-09 16:06:59 +08:00
TheStranjer 3c854ae217 fix(cli): bind a symlink's ownership judgement to the inode that is followed 
Review of the previous head reproduced a check-then-use window in the
parent-chain walk: the link was judged by name (fstatat) and then opened
by name (openat), with nothing tying the two to the same entry. An
LD_PRELOAD shim that swaps the link for one pointing elsewhere the moment
the judgement returns made both earlier heads follow the swapped link,
including one owned by a foreign uid under root; the post-follow target
check only bounds where such a swap can land.

The link is now never opened by name after it has been judged. On Linux
an O_PATH|O_NOFOLLOW descriptor pins the inode, fstat on that descriptor
makes the ownership decision and readlinkat(fd, "") reads the target text
from the same inode; on other systems the link is stat'ed by name before
and after the readlinkat and both must be the same inode with the same
owner. The walk then opens the text from the parent directory, so a
relative text resolves against the link's own directory exactly as the
kernel resolves it, and the target check runs on what that opens. The
readlinkat form is used rather than /proc/self/fd so the walk does not
depend on /proc being mounted. What is still done by path is resolving
the text itself, which the target check bounds; the comment says so.

Test: relative, ../, absolute and chained link texts resolve from the
link's directory with the process CWD moved elsewhere. The race itself
is exercised out of tree with the harness from the review thread.

Refs #1722

Signed-off-by: TheStranjer <791672+TheStranjer@users.noreply.github.com>
 2026-09-08 00:33:36 -06:00
TheStranjer 8622d6a0ed fix(cli): make following user-owned symlinks an opt-in per call site 
Review of the previous head found the global rule unsound for one caller:
cbm_mkdir_p also creates <repo_path>/.codebase-memory inside the checkout
being indexed, and git creates symlinks owned by whoever cloned, so a
repository shipping `.codebase-memory -> ~/.ssh` would have been followed
by the artifact writer. "User-owned" says nothing about "user-intended"
once the path root is a checkout.

cbm_mkdir_p keeps the strict walk (root-owned links only). The new
cbm_mkdir_p_ex(path, mode, policy) with CBM_MKDIR_FOLLOW_OWNED also
follows a symlink owned by the invoking user, and only callers whose
paths are rooted in the user's own configuration opt in: the installer's
agent roots, skills, instruction files and hook directories, the
JSON-like config editor's parent creation, the cache root, <cache>/logs,
the UI config directory and the DB parent. The artifact directory inside
the checkout and the ~/.local/bin creations stay strict; the activation
transaction re-walks the latter with its own no-follow policy anyway.

What a follow lands on is now checked too: the opened target must be a
directory owned by root or the invoking user and not world-writable
unless sticky, so a link swapped between the fstatat and the openat
cannot steer the walk into a directory an attacker controls or one where
anyone can pre-plant entries. This applies to root-owned links as well;
macOS /private/tmp is root-owned and sticky and still passes. Refusal
restores the first openat's errno.

Tests: the plain walk refuses an own link that the opted-in walk
follows; targets that are world-writable without the sticky bit are
refused; as root, a foreign-owned target and a user-owned link are both
refused, each followed by restoring ownership and walking again; the
identical `.codebase-memory -> <user-owned dir>` link is refused by the
plain walk and followed once opted in; cbm_artifact_export refuses a
checked-in artifact-dir link and leaves its target empty; the installer
test gains the euid-0 leg with home, link and target owned by a non-root
uid. Under euid 0 the tests demote their links to a non-root owner, since
root-owned links are trusted by both walks.

Refs #1722

Signed-off-by: TheStranjer <791672+TheStranjer@users.noreply.github.com>
 2026-09-07 23:17:50 -06:00
Martin VogelandClaude Fable 5.1 0c4831a364 fix(pipeline): own every staging file through an exclusive sidecar lock 
A stage was recognisable only by its name: the mkstemp descriptor was
closed at once and nothing marked who was writing it. A worker killed
mid-run -- the daemon cancels with SIGTERM and SIGKILLs after one second
of grace, which a gigabyte backup or delta clone never finishes inside --
left its full-size stage behind for good, and no later run could tell a
dead stage from a live one, so none tried (#1839).

Ownership is now an exclusive kernel lock on the sidecar "<stage>.lock",
taken when the stage is minted and dropped by the same helpers that
discard the stage or rename it into place. The kernel releases it on any
death, so "can I take this lock?" is exactly "is this stage dead?" -- no
pid, no mtime, no grace period. The lock lives on a sidecar rather than
the stage file itself because on macOS an flock conflicts with SQLite's
fcntl byte locks on the same file (verified: flock(LOCK_EX) then
fcntl(F_WRLCK) on another descriptor fails with EAGAIN, both directions).

The primitive is cbm_lockfile_open: POSIX open(O_CLOEXEC|O_NOFOLLOW) plus
flock(LOCK_EX|LOCK_NB); Windows _wsopen with _SH_DENYRW and _O_NOINHERIT.
Neither is inherited by spawned children. A backup that fails keeps the
outer stage NAME owned: the rebuilt generation is renamed over it by the
inner finalize and published from it, so its lock is held to the end.
The sweep that uses this ownership follows separately.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-09-08 01:37:44 +02:00