mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-26 06:43:15 +08:00
6f290e11d0f46c065b052db2ff70a7bbceede5dd
391
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
2f33999044 |
test(vm): let the log decide the Windows leg, not the exit status
On 2026-09-18 a Windows leg printed "7925 passed, 0 failed" and "=== All tests passed ===" and still exited 1, while the identical work run through win.sh's other entry exited 0. The status travels ssh -> cmd.exe -> msys2_shell.cmd and that chain loses it. A channel that can turn 0 into 1 can turn 1 into 0, and that direction is a false green: a red Windows leg reported as passing, which is the one thing the local ladder exists to prevent. So the two channels are now split by what each can actually know. The log decides the test outcome — it is written by the runner on the VM and cannot be mangled in transit — and rc only decides what the log cannot see. Green needs zero reported failures AND the completion marker scripts/test.sh prints last, so a leg that stopped early has a summary but no marker and is red. A lost exit status is reported, not obeyed. The logic is a function, vm_verdict(), reachable as `--verdict <log> <rc>`, so tests/test_vm_verdict_contract.sh drives the REAL code with synthetic logs instead of a copy that could drift from it: lost exit status, failures under rc=0, a marker that must not launder failures, failures in a later suite, partial runs, and no summary at all (which keeps its own exit code 90, so "never ran" stays distinguishable from "ran and failed"). Eight cases, wired into scripts/test.sh as step 0y. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
1d58385b51 |
refactor(mem): route this branch's allocations through the memory core
The memory-core linter is a ratchet: raw malloc/calloc/realloc/free/strdup sites may not grow, because allocations belong on one accounted path (src/foundation/mem_core.h) rather than spread across hundreds of call sites. This branch's own work had added 35 raw sites across nine files -- the parallel k8s prep, the configlink key matching, the worker pool's counted path, the Go shared stdlib registry, the spill namespace copy, the route service array and the namespace-name arrays. All of them now use cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/ cbm_mem_strdup. One was a latent bug rather than a style point: pass_k8s freed a buffer that k8s_read_file had allocated with raw malloc, so converting only the free would have mismatched the allocator -- the read function and every free of its result are converted together. The Windows free-space path widens its argument through the memory core instead of cbm_utf8_to_wide(), which allocates with raw malloc. Two files improved enough to tighten their baselines, which the ratchet requires so the gain cannot be quietly given back: sqlite_writer.c 92->74 (the index-cell arena), pass_k8s.c 13->6 and registry.c 19->17. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
92e7e6a913 |
fix: keep large thread-locals off the static TLS block
Indexing broke on x86-64 Linux: the index worker refused to start and
SIGKILLed its own process group, so every venue reported nothing but
"index worker ended with killed (exit=-1, signal=9)".
glibc places a thread's static TLS block inside the stack allocation it
makes for that thread. This branch's new thread-local caches -- a
512-slot field cache (~24 KB), per-depth tree cursors (~4 KB) and two
parked CBMArenas (~8 KB; an arena carries blocks[256] + block_sizes[256])
-- took static TLS from 47,905 to 93,513 bytes. The 64 KB parent-death
watchdog thread then could not be created at all, and a worker without
process-tree containment correctly refuses to index.
Four layers, because there were four things wrong:
- The big thread-locals move to the heap behind a thread-local POINTER,
allocated once per thread and reused, each with a correct fallback if
allocation fails (the field cache and cursor pool are caches; a miss
is slower, never wrong). Static TLS is back to 52,233 bytes.
- PARENT_WATCHDOG_STACK_SIZE 64 KB -> 256 KB. "The watchdog only polls,
a tiny stack suffices" was true when written and false once TLS
reached ~50 KB, because the TLS block comes out of that same
allocation.
- cbm_thread_create retries with the default stack on EINVAL. A stack
size is a hint about what a thread needs; the platform refusing the
hint is not a reason to fail to create the thread.
- tests/test_thread_stack_tls_contract.sh (Step 5a of scripts/test.sh)
keeps static TLS a quarter clear of the smallest requested stack,
reading that smallest size from the sources so a new small stack
tightens the gate automatically.
Two diagnostics that made this findable at all, and that stay:
smoke-test.sh prints the worker log its failure message names (CI deletes
that sandbox with the job, so the one artifact naming the cause was
unreadable), and worker_containment_unavailable() now reports WHICH of
its conditions failed, with pid/pgrp/ppid.
Verified in the amd64 container, the venue nothing on the local ladder
covers: baseline main indexes 9039 nodes/15067 edges, this branch before
the fix refuses to start the worker, and after it indexes 9039/15067.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
f0a86ef2c2 |
feat(ci): seeded fuzz leg, and register both new legs as canonical entries
scripts/fuzz.sh drives the libFuzzer targets in two shapes from one entry: a fixed --runs from a fixed --seed, so a CI smoke is a function of the code and not a lottery (O9), and a time-boxed --max-total-time for exploration. Checked-in seeds are copied into a working corpus under build/fuzz/, so a run never rewrites the repository; a crash leaves its reproducer behind and the script exits non-zero. Wired as a smoke into dry-run.yml and as exploration into nightly-soak.yml. The venue-parity contract rightly refused both workflow steps: a venue may provision or plumb, but only a CANONICAL leg entry may exercise the product, and neither script was registered. Registering them — rather than working around the contract — is the fix, since both own their leg end to end (build, corpus or binary, artifact paths, exit code) exactly as soak-legs.sh owns the soak. _memwaste.yml joins the policed venue walk for the same reason, and both scripts join the layer-5 interface probes. Those probes immediately earned their keep, finding two real defects: `fuzz.sh --help` exited 2 because the positional target consumed the flag, and memwaste.sh read a typo'd flag as the repository path — which would have indexed nothing and reported an empty lane as a clean one. test_shell_line_endings.sh now degrades where the checkout has no reachable repository metadata: the Linux container bind-mounts the working tree, and a git worktree's .git is a pointer to the host repository, so `git ls-files` cannot run there and the contract failed with "matched no files — the glob set is broken", killing the leg before it compiled. It is a repository property, so it stays gating in every venue that has the metadata (macOS host, the Windows VM's real checkout at C:\cbm with 117 files, every hosted-CI checkout). test_version_metadata_contract.sh already guards the same way for the same reason; the alternatives considered and rejected are recorded at the case. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
81264814f1 |
feat(qa): memory + CPU waste sanitizer over the memory core
Answers a question the existing tooling could not: of everything we
allocate, how much is never used?
Lanes, all driven by scripts/memwaste.sh:
event every allocation observed through the memory core — size, age,
churn, and whether the bytes were ever written
access SanitizerCoverage load/store instrumentation, so "allocated and
never touched" is measured rather than inferred
scaling k vs 2k replicas through clang profile counters, to catch
super-linear work per file
padding allocator slack per size class
ratchet the numbers recorded per venue so a regression is visible
The event lane needed a foreign-free counter: on Windows the override sees
frees for blocks it never allocated (they come from the CRT), and counting
those as untracked tripped the soundness gate. mem_events now separates
them (14 daemon / 5 worker on the Go corpus).
Two portability fixes the lanes surfaced, both real:
- the enable decision must not be cached before `environ` exists. Under
-fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our
wrapper before glibc sets up environ, so the lane silently disabled
itself and produced no dump at all on Linux.
- the access map needs 48 bits: arm64 Linux hands out addresses above
2^47, which a 47-bit map dropped on the floor.
mem_profile is retired into this: it measured a strict subset and had its
own sampling story.
Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on
all three platforms report-only, and pr.yml/release.yml carry the wiring.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
7b77fd4834 |
feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.
WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)
The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.
WHAT CHANGED
Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.
Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.
Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.
The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.
The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.
Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.
Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).
The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.
The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.
Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.
SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).
Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.
Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.
The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.
Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.
PROOF
Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
jitter (gRPC Route nodes: two identical runs differ by 21).
Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
surface rows, nodes/edges inside the Route jitter.
Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
inside the +-400 run-to-run band the kernel shows between identical
runs.
Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
outside what spill can move). Nodes identical; edges 15,773,598.
Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
every phase mark under 15,000 MB by every metric, but the process
RSS high-water mark reached 16.9 GB inside the semantic pass (class
peak 4.9 GB against 0.8 GB live).
Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
stays under 15,000 MB through the whole semantic pass (14,018 ->
14,381 MB across its sub-phases, 6 batches of 125,644 functions);
every phase mark is under budget by every metric (max footprint
12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
15.65 GB and is set in extraction, at the moment the gate latches
(15,002 MB charged) and before the first sweep lands -- a 4%
overshoot of in-flight work, no longer the semantic pass. Nodes
identical, edges 15,772,913, zero foreign blocks.
Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
enters at 14,062 MB (near_budget); the charged footprint never
exceeds 11.8 GB at any mark; the process RSS high-water mark is
15.1 GB in extraction and 15.3 GB during the final write-out --
1-2% over, the in-flight window and the dump transient. Nodes
identical, edges 15,772,543, zero foreign blocks.
Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
peak 998 MB RSS, graph identical to the baseline.
BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)
corpus wall s peak RSS GB nodes / edges / CALLS
perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3
php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8%
rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369
c 5.9 -> 5.7 0.16 -> 0.19 identical
kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1%
django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4%
go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2%
java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6%
csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6%
typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130
kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1%
No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (
|
||
|
Martin Vogel
|
7fedfe85c6 |
lint(mem): scan all of src/ and internal/, exempt vendored/ by path segment
The memory-core linter scanned src/ and internal/cbm/. Widen it to src/ and every subtree under internal/, so a new tree added there is covered without anyone remembering to list it, and the count is for the whole project -- cli, mcp, daemon, store, pipeline, the extraction engine. Vendored code is now excluded wherever a vendored/ segment appears in the path rather than by one hardcoded prefix, so that exemption cannot drift either. internal/ holds only cbm today, so the regenerated baseline is byte-identical: 92 files, 4086 raw sites. Widening lost no coverage and vendored stays out (0 sites listed). The red/green proof still holds under the new scope, and the failure message now names the NEW site -- one appended malloc reports grew by 2 (83 -> 85) at the appended line, instead of listing six pre-existing ones. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
da3258f3c7 |
feat(mem): a central allocation core, a linter that enforces it, and phase attribution in the index pipeline
Memory in this project is now allocated through ONE core. This lands the
core, the gate that keeps new code on it, and the instrumentation that lets
an index run say which class of memory grew in which pass -- the three things
the 2026-09-13 audit found missing.
WHAT THE AUDIT FOUND
There was no centralized allocation layer. foundation/mem.h was policy and
measurement only (budget, RSS, over_budget, memory map, phase marks) with no
alloc/free wrapper, so the budget could be observed after the fact and never
enforced. Raw allocator calls in the tree: 4086 across 92 files. The
extraction engine had already adopted arenas (1301 call sites); the graph
buffer -- which holds the 35 GB peak of a kernel index -- used none,
allocating every 64 B node, 48 B edge and every name/qualified_name/
properties_json individually.
Two further facts made a self-accounting core the only workable design:
1. cbm_mem_map_collect() walks THIS thread's mimalloc heap only. Walking
the process-wide heap is a data race TSan caught on macOS, so it is
deliberately not done, and an 18-worker index attributes almost
nothing -- the rest lands in residual.
2. The mimalloc global override is ON for Linux/MinGW and permanently OFF
for macOS (two-level namespace: this binary's free becomes mi_free while
system libraries keep allocating from the system zone, and a pointer
crossing that boundary aborts). On macOS ordinary malloc is served by
the system allocator; the startup audit reports owned_classes=0/6. Any
accounting that assumes mimalloc owns the pointer is blind on an entire
platform.
THE CORE (src/foundation/mem_core.{h,c})
cbm_alloc / cbm_calloc / cbm_realloc / cbm_mem_strdup / cbm_free, each tagged
with an allocation class (gbuf_node, gbuf_edge, gbuf_string, gbuf_index,
extract, semantic, dump, store, other). Per-class live bytes, live blocks and
peak, kept in atomics -- thread-safe by construction, identical on every
platform, independent of which allocator serves malloc. libc semantics are
preserved exactly (alloc(0) is freeable, free(NULL) is a no-op, realloc(NULL)
allocates, a failed realloc leaves the block intact) so adoption is a
mechanical rename and never a behaviour change.
No per-allocation header: a {class,size} prefix costs 16 bytes on ~100M blocks
at kernel scale, 1.6 GB of overhead to measure a memory problem. Sizes come
from the platform usable-size query (malloc_size / malloc_usable_size /
_msize), which is correct under either allocator and free. Where no query
exists (BSD) the request size is charged, which understates -- the safe
direction for a diagnostic. A mismatched class on free clamps at zero rather
than wrapping, so a small drift can never masquerade as a colossal leak.
Arena-backed allocators report in bulk through cbm_mem_class_add_external /
remove_external so their memory appears in the same table; the extraction
engine must not be rewritten to per-object allocation, that would undo the
batching that keeps its allocation count low.
THE LINTER (scripts/lint-memory-core.py, wired into make lint-ci)
Raw malloc/calloc/realloc/free/strdup/strndup outside the core is a defect.
4086 sites cannot migrate at once, so the gate is a ratchet: a checked-in
baseline (scripts/memory-core-baseline.txt) records each file's count and
the build goes red when any file grows, or a new file appears with any.
Files only ever go down; the baseline line is lowered in the change that
migrates the file, and --strict fails a stale baseline. Comments and string
literals are stripped first, so prose that mentions malloc( cannot trip it --
the security audit already bit on exactly that with fork(. Proven both ways:
one appended malloc turns it red naming the file (83 -> 85), restoring turns
it green. Scans src/ and internal/cbm/: cli, mcp, daemon, store, pipeline.
PHASE ATTRIBUTION (src/pipeline/pipeline.c)
cbm_mem_phase_mark and the new cbm_mem_class_log now fire at pipeline.begin
and at every pass.timing site: parallel_extract, the six sequential passes,
the eight predump passes, tests and dump_and_persist, with peaks reset per
index. Both instruments existed in foundation/ and were wired only into MCP
request handling, never into the index pipeline -- which is where the memory
is. Locating the kernel peak this week required an external RSS sampler
because nothing in-process could say which phase it was in.
PRESSURE PRIMITIVES (system_info.c, mem.{c,h}, platform.h)
cbm_system_available_ram() -- macOS host_statistics64 (free + inactive +
purgeable), Linux MemAvailable, Windows GlobalMemoryStatusEx; 0 when unknown,
never cached -- and cbm_mem_system_under_pressure(), true below 12.5% of RAM
available and false when unknown so nothing ever aborts on a guess. Landed
UNWIRED: an earlier draft used them to let the over-budget latch press on
whenever the system was not under pressure, and that permitted the very
overshoot this work exists to end (Decision A's tests at
test_pipeline.c:13481 and :13583 went red under it, correctly). They are the
backstop for the next step, admission control, which keeps a run under budget
before an allocation rather than measuring it after.
mem suite 10 new tests green; pipeline + mem 339 passed 0 failed; lint-ci
clean including the new gate.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
2b80b31306 | Merge remote-tracking branch 'origin/main' into HEAD | ||
|
Martin Vogel
|
1ad52f5fb1 |
fix(ci): make the code-scanning gate able to see alerts, and fail closed
The codeql-gate has never checked anything. It declared no permissions block,
so it inherited the workflow's contents:read and the code-scanning alert API
answered 403. gh writes the API error BODY to stdout, so
ALERTS=$(gh api '.../code-scanning/alerts?state=open' --jq 'length' 2>/dev/null || echo "0")
left ALERTS holding
{"message":"Resource not accessible by integration",...,"status":"403"}0
every [ ... -gt ... ] then failed as a non-integer comparison, the if took its
false branch, and the step printed "CodeQL gate passed (0 alerts)" and exited 0.
Observed in the log of a GREEN run (34700514102, job 103571322660) while the
repository had an open high-severity alert. The `|| echo "0"` reads as defensive
and is the exact opposite: it converts "I cannot see the alerts" into "there are
no alerts", which is the one direction a security gate must never fail in.
Grant the job security-events:read (and actions:read, which the CodeQL-run poll
needs for the same reason), and replace the fallback with a helper that treats a
non-zero gh exit or a non-numeric body as a hard failure. The helper returns
rather than exits because it runs inside a command substitution, where an exit
would end only the subshell and let the caller continue with an empty count --
the same silent pass in a new costume; both call sites check the status.
tests/test_security_gate_fail_closed.sh pins both halves and fails with six
assertions against the previous workflow.
Also bumps graph-ui's vitest to ^4.1.11 (GHSA-82fw-gwwq-j7x9, path traversal via
@vitest/mocker, CVSS 5.9). Development scope, and graph-ui ships in no release
artifact, so nothing released was ever exposed -- but it was the sole open alert
and the gate above is worthless while it stands unresolved. Raising the floor
past the vulnerable range also stops a fresh install resolving back into it.
graph-ui: npm ci clean, 47 tests in 12 files pass on 4.1.11.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
9da4d66df2 |
Merge pull request #2115 from DeusData/fix/activation-guard-home-scope
fix(cli): scope the activation guard to the install-target HOME so a sandbox install can't drain the host daemon |
||
|
Martin Vogel
|
fa99ff64eb |
test(cli): stop a negative probe from spending 15s proving silence
The macos-15-intel leg killed the cli suite at its 900s wall clock while reporting 7537 passed, 0 failed. Two separate things were wrong. The suite was misclassified. cli spends 497s of the 900s default on macos-14, the FASTEST macOS runner, while macos-15-intel in the same matrix is 2.4-3.6x slower on comparable suites (daemon_runtime 842s vs 349s, stack_overflow_b 277s vs 76s). 497s at that ratio cannot fit. daemon_runtime, at 842s on that same runner, passes only because it was already in SLOW_SUITES. cli belongs there too -- it is deterministic and simply large, which is the only thing that list is allowed to mean. The drain test was also waiting on a clock instead of on the system. Profiling the suite per test (315 tests, 300s wall / 166s cpu) put cli_install_into_host_namespace_still_drains_host_cohort at 42s wall for 19s of cpu -- 23s idle, the largest single idle block, where the next worst was 11s. The existing cli_install_force_quiesces_active_cohort_before_replacing_binary drains a cohort too and idles 0.5s, which is what pointed at the difference. The cost was the host_serving probe. Its generous budget exists for the POSITIVE question -- is this daemon still up? -- where a slow reply on a loaded runner must not be misread as drained; that fixed a real flake and is kept. Asked in the negative it inverts: no reply is coming, so the full 15s is spent establishing silence and ASSERT_FALSE is decided by the timeout expiring rather than by the daemon's behaviour. The drain is already proven positively in that test -- install returns 0 only after the activation completed, and the host child exits CLI_SCOPE_HOST_DRAINED -- so the probe only has to confirm it. Also shortens the drained child's teardown budget, which retried service_free / lease_release / manager_free against 10s on a path where the activation had already torn the service down. Behaviour at the deadline is unchanged; it is reached sooner when nothing is wedged. Measured at only -3s on its own, kept because it is correct and free. Local, same machine, ASan+UBSan: the drain test 42.0s -> 25.8s (-38%), the suite 300s -> 279s, 315 passed before and after -- no test removed, no assertion weakened. PR #2188, which runs the same leg without these two tests, passed macos-15-intel in 37m34s, so the suite was not already over budget on main. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
734906379f |
fix(cli): name the check that refused, and a remedy that can remove it
A reporter brought a full ProcMon trace to #1856 and still could not say which check refused their install. That is not their failing: the binary discards the answer before printing it. cli_activation_production_context_init() validates the cache, rendezvous and log directories -- it is the emitter MOST likely to hold a useful detail -- and it printed CLI_ACTIVATION_REFUSED_MESSAGE bare, through the raw sink rather than cli_activation_diagnostic(). So the reader got "Check the errors above" with nothing above. That is the exact dead end #1416 and #1537 already fixed; the property was repaired on the paths that had tests and left broken on this sibling, which had none. Route it through the attributing helper: the refusal now names the transaction refusal note or cbm_daemon_ipc_validation_detail(). The remedy was also wrong for the reported shape. `icacls <dir> /remove:g <sid>` cannot remove an INHERITED ACE, and the stock C:\ grant for Authenticated Users reaches every new child directory exactly that way -- so a reporter following our advice watches the command succeed and the refusal persist. The refusal now says whether the ACE was inherited, and the advice names `icacls <dir> /inheritance:r /grant:r "%USERNAME%":(OI)(CI)F` for that case. No safety check is relaxed. An install directory writable by other accounts is still refused; it is now refused in terms the reader can act on. The new contract test guards the CLASS rather than this one call site: any emitter of a refusal constant that bypasses the attributing helper fails it, which is what would have caught this. It fails on the previous tree naming src/cli/cli.c:750, plus the missing inherited-ACE report and the missing /inheritance:r remedy. Local: cli 314 passed; activation_transaction daemon_ipc daemon_bootstrap daemon_version 110 passed, 1 skipped (user namespaces are Linux-only, runs on the Linux leg); make -f Makefile.cbm lint-ci clean. Reported by zaferavci1. Refs #1856 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin VogelandZhiyu
|
a08b9ecbbd |
fix(worker): scope the index worker to the request's repository, not the daemon's environment
The daemon-spawned index worker inherited the daemon process environment and
re-ran the allowed-root check without a session policy. The daemon's session
policy is installed only in application.c (the sole production caller of
cbm_mcp_server_set_session_context); after the daemon admits an
index_repository request, index_supervisor.c builds the worker argv
(`cli --index-worker --index-worker-build <fp> index_repository <args_json>
--response-out ...`, no root anywhere) and subprocess.c spawns it with the
daemon's environ. The worker then creates its own server with
cbm_mcp_server_new(NULL), so handle_index_repository runs with no session
policy and falls back to getenv("CBM_ALLOWED_ROOT") - the environment of
whoever STARTED the daemon (bootstrap.c spawns it with the starter's environ).
Observed: a daemon started with CBM_ALLOWED_ROOT=<rootA>; a client session for
<rootB>/tiny is admitted by the daemon (index.supervisor.reap outcome=clean
exit_code=0) and then refused by the worker with "<rootB>/tiny is outside the
allowed root. To allow it, run: codebase-memory-mcp allow-root ...". Any daemon
whose starter had CBM_ALLOWED_ROOT set broke every other session outside it.
Fix: in the worker arm of run_cli, right after the worker's server is created
and before any tool runs, read repo_path from the request args (both daemon
spawn paths rewrite the worker args with the canonical repo_path), canonicalize
it and install it explicitly as both session root and allowed root. The daemon
already authorized the request; the worker only executes it. A repository that
cannot be canonicalized does not exist and can only have been admitted by a
session with no declared boundary (containment needs a real path), so the
worker mirrors that with an explicit unrestricted policy - still never the
environment fallback - and the pipeline keeps reporting the missing repository
as the tool error it always was. A request without repo_path fails closed with
"request workspace scope invalid". No argv or IPC change, no process
environment mutation.
Regression test: tests/test_worker_session_scope.sh drives the real binary as a
supervised worker with CBM_ALLOWED_ROOT=<rootA> and repo_path=<rootB>/tiny and
requires an indexed response with no boundary refusal; a worker without
repo_path must exit nonzero on the scope check. Wired as Step 5f of
scripts/test.sh next to the other real-binary worker contracts. RED on main
(the exact "outside the allowed root" refusal above), GREEN with this change.
Distilled from #1925 with co-author credit.
Verification: test-runner suites index_supervisor, daemon_application, daemon,
mcp, cli green; tests/test_worker_error_response.sh, test_worker_watchdog.sh
and test_parent_watchdog.sh green against the fixed binary; make lint-ci and
scripts/check-no-test-skips.sh clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: Zhiyu <zhiyuzhang001@gmail.com>
|
||
|
Martin Vogel
|
53f55ebe72 | Merge branch 'main' into codex/fix-1689-codex-agents-cleanup | ||
|
Martin VogelandKody
|
3b9052c955 |
test: write scheduler barrier files atomically
Path.write_text is open-then-write: it creates and truncates the file first and the content lands afterwards. The parallel harness contract polls for `<suite>.ready` to appear and then parses its content as the leader pid, so a reader that wins that window sees an empty file and fails with `ValueError: invalid literal for int() with base 10: ''` -- a spurious FAIL of the harness contract with a scheduler-side cause. The same shape applied to `<suite>.leader-exited`. Publish the barrier files through a helper that writes a same-directory temp file and os.replace()s it onto the destination, which is atomic on POSIX and Windows: a poller now sees either no file or complete content. All three barrier publications go through it. Pin the contract structurally (no timing): the scheduler may not write a barrier file in place, and it must rename into place instead. The pin is RED on the previous scheduler. Distilled from #1188 with co-author credit. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> Co-authored-by: Kody <kaidi.shi.1121@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b51411753e | Merge branch 'main' into codex/fix-1689-codex-agents-cleanup | ||
|
Martin Vogel
|
619c683480 |
fix(mcp): settle the lean-output branch on current main
Fix-forward after rebasing onto main (545 commits, 26 conflicted-file resolutions). Where main had shipped a behaviour in the meantime, main wins; the PR keeps its output contract on top of it: - search_code: the scan runs under main's supervised, deadline- and output-bounded runner and feeds the branch's streaming classifier from the runner's output file; the POSIX wrappers map grep's no-match to 0, so every non-zero exit fails closed (no exit-1 special case). - trace_path: strategy/confidence precede args in header and rows (#1542), in both the tree table and the json legs. - cypher: relationship expansion is never capped (a cap falsified aggregates) and unlabeled MATCH scans every candidate; `truncated` is raised only by a real ceiling — a clamped hop range is probed one hop beyond the cap and reports truncated only when a candidate exists there. The plain BFS and the trail BFS both probe one row past max_results so saturation is observable instead of silent. - search_code full mode and the final json use main's U+FFFD sanitizer instead of '?' replacement. - search_graph json keeps an empty `groups` array for semantic-only calls; list_projects json echoes offset/limit next to next_offset and accepts main's include_details spelling for the stats projection. - discovery budget: main added get_file_outline, compare_graphs, manage_adr set_sections and the debug/include_details parameters; their descriptions are trimmed to the lean style and the ceiling moves from 15 to 18 KiB. Tests that asserted a JSON default now request format:"json"; the fail-closed search test uses an unreadable regular file (a missing or non-regular indexed path is deliberately not a scan operand on main); the non-regular fixture gives its real sources File nodes, as every indexed project has; list_projects total counts listable projects (the 0-byte ghost is never a row, so it is not a page either). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3909e5a0e1 |
feat(mcp): make tool output lean and lossless
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b3d898e12f |
Merge pull request #1914 from DeusData/fix/elf-hardening-relro-now
fix(build): put the whole GOT inside the RELRO window on ELF targets |
||
|
Martin Vogel
|
b6e4beb9b1 |
Merge pull request #1849 from DeusData/fix/issue-1733-windows-hook-wrapper-resume
fix: register Claude hooks without a shell |
||
|
Martin Vogel
|
39c74bdd5c |
test: converge hook tests and smoke on the exec-form registration
The rebase onto main brought the Bash-enabled PreToolUse matcher (Grep|Glob|Bash), which the exec-form tests still counted as Grep|Glob. The Windows lifecycle test seeded the .cmd shell spelling as the converged identity; the shell-free registration converges every owned shell spelling onto one exec-form entry per matcher instead. The test now asserts that, plus their removal on uninstall, and hands cbm_install_agent_configs the '/'-spelled binary path cbm_cmd_install registers (cbm_mkdtemp returns a native '\' temp root on Windows, which uninstall never derives). Smoke 8aq proved Claude's gate registration by grepping the shim name out of settings.json; the exec form carries no shim name, so it now checks the binary-plus-'hook-augment' PreToolUse hook. 9d's shim-name marker would otherwise pass vacuously after uninstall; the 'hook-augment' argument is an owned marker there too. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7b92d2a621 |
ci(composition): scope A1d-bind-now to release links, declare fixtures by name
Step 0o (MCPB bundle contract) and Step 0q (release candidate derivation) hand the composition gate a stub compiled straight from a .c file with the compiler's defaults. That stub exists to exercise packaging, format detection and the needle scans; it is not linked with -z now and never ships. A1d measured eager binding on it anyway, so on a toolchain that binds lazily by default (test-diag: clang-22) Step 0o died deterministically with A1d-bind-now stage/codebase-memory-mcp: GOT section(s) .got.plt fall outside PT_GNU_RELRO [15824,16384) — they stay WRITABLE after relocation The property A1d protects is a property of the release link (-z now in Makefile.cbm), so the assertion is scoped to it: a caller that feeds the gate a packaging fixture declares so with CBM_COMPOSITION_FIXTURE=1 and A1d is reported n/a — printed on every run as a named exemption, never a silent skip. Every other assertion still runs on the fixture unchanged, and release derivation never sets the variable. Verified in the Linux arm64 container (Ubuntu 24.04, gcc 13.3): a stub linked -z lazy reproduces the CI failure text on the gate without the variable and reports n/a with it; both contracts pass under a CC wrapper that links lazily (the shape that was red), and unchanged on macOS. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
132e8fc323 |
test: give the Windows descendant probe its own budget
When a suite leader has already exited, taskkill /T cannot walk its tree, so run-test-wave.py proves cleanup by asking Windows whether anything is still parented to the dead PID. That probe was timed with --kill-grace, which the parallel gate passes as 1s. A cold powershell.exe + CIM start routinely costs more than that on a runner, so TimeoutExpired -- a fact about interpreter start-up, not about the process tree -- was read as "assume the worst" and failed an already-clean shard with exit 2. Give the probe its own stable-state budget (15s) plus one retry, keeping the fail-closed semantics: an exhausted or failed probe still counts as "cannot prove absence". The refusal now names which of the two happened, an unfinished probe or a counted set of live descendants, instead of reporting both as a leaked tree. The contract test pins this structurally -- no sleeps, no timing thresholds: the probe must be bounded by the module constant rather than by the kill-grace argument, must retry a timed-out probe, must never read an unusable probe as absence, and must report the two failures distinctly. Its Windows leader-exit case now waits on the scheduler's refusal rather than on a deadline shorter than the new budget. Refs #1741 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
be66a2d2c6 |
smoke(8al-node): load the Pi extension with the modules the Pi host provides
The 8al-node probe imports the generated Pi extension in bare Node with an empty node_modules. That was a sound stand-in for the Pi host only while the extension imported nothing but node: builtins; the first package import (`typebox`, Pi's schema library for tool parameters — a real Pi dependency, verified against pi-coding-agent 0.84.4 on #1809) can never resolve there, so the probe fails on a correct extension and would pass a wrong one just as readily. Give the probe what the host gives the extension: a Proxy-backed `typebox` stub in the probe's node_modules (any Type.X(...) returns a descriptor, so the probe never tracks which helpers the emitter uses — it only exercises the tool lifecycle), plus an explicit allowlist: every import specifier in the generated file must be a node: builtin or a package on PI_HOST_PACKAGES, otherwise the probe fails naming the specifier. That turns the guarantee the empty node_modules gave by accident (no surprise dependency) into an asserted contract. Verified locally by running only the 8al-node block against extensions generated by two binaries: main@5802ccdd extension, old probe: OK; new probe: OK #1809 extension, old probe: ERR_MODULE_NOT_FOUND typebox (the CI red) #1809 extension, new probe: OK #1809 extension + an injected 'lodash' import, new probe: FAIL 8al-node: ... packages the Pi host does not provide: lodash Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
astandrik
|
4c1b23477b |
fix(codex): keep global activation pointer
Signed-off-by: astandrik <astandrik@yandex-team.ru> |
||
|
astandrik
|
4cafe8eb22 |
fix(codex): stop installing global AGENTS guidance
Signed-off-by: astandrik <astandrik@yandex-team.ru> |
||
|
Joshua Richter
|
e820409233 |
docs(vendored): record the Swift scanner shift patch in MANIFEST.md
The vendoring table claims our copy of a grammar matches the pinned upstream commit. A local patch makes that claim false unless the patch is written down, so a future re-vendor would drop the fix without anyone noticing. The row covers both changed lines, because both are the same defect in the same file: OP_SYMBOL_SUPPRESSOR at line 131 and eat_operators at line 514. The row also names the two upstream commits that already carry these changes, and the table intro gains one clause for the case. Upstream fixed eat_operators in fb63a7004f07 on 2026-04-06 (their issue #558) and the OP_SYMBOL_SUPPRESSOR entry in 6ab8d1d74ebd on 2026-08-10. Our pin 8abb3e8b3325 is from 2026-03-20 and predates both, so these are backports rather than local inventions. That distinction changes what a re-vendor should do. The table's instruction is "must re-apply these", which is right for the other four rows -- upstream never took those. For this row it would mean hand-applying a change the newer file already has. The row now says to delete it instead, and the intro allows for that case. MANIFEST.md is itself listed in scripts/vendored-checksums.txt, so its own recorded checksum moves with the edit. Refs #1892 Signed-off-by: Joshua Richter <jrichter5781@gmail.com> |
||
|
Joshua Richter
|
a2ea711c29 |
fix(grammar): stop the Swift try-bang entry shifting past its own type
scanner.c line 131 shifts a literal 1UL by 32 to build the entry that suppresses the bang of a try!. unsigned long is 64 bits on Linux and macOS but 32 bits on Windows, so there the shift count equals the width of the type, which is undefined. 1ULL is 64 bits on every target this project builds for. Nothing reports it today. The expression is a compile-time constant, so no sanitizer sees it run, and Makefile.cbm:719 builds vendored grammars with -w, which switches off -Wshift-count-overflow. Upstream tree-sitter-swift made the same change in 6ab8d1d74ebd, after the commit this grammar is pinned to. Refs #1892 Signed-off-by: Joshua Richter <jrichter5781@gmail.com> |
||
|
Joshua Richter
|
3f831936b5 |
fix(grammar): stop the Swift scanner shifting past the width of an int
The Swift scanner keeps a 64-bit mask of the symbols that suppress a
match -- the rule that stops `try!` emitting its `!` as a token of its
own. It tests one bit per candidate:
uint64_t suppressing_symbols = OP_SYMBOL_SUPPRESSOR[full_match];
for (uint64_t suppressor = 0; suppressor < TOKEN_COUNT; suppressor++) {
if (!(suppressing_symbols & 1 << suppressor)) {
The mask is uint64_t but the literal `1` is an int, so the shift is an
int shift. TOKEN_COUNT is larger than 32, so once suppressor reaches 31
the shift runs past the width of the type. That is undefined behavior,
and every bit above 31 is tested against a value the standard does not
define.
Nothing caught it because nothing in the tree reached the suppressor
path. Any Swift force-unwrap does: `cached!` is enough.
UBSan reports it as:
scanner.c:514:47: runtime error: left shift of 1 by 31 places
cannot be represented in type 'int'
`1ULL` makes the literal as wide as the mask it is tested against.
The new test in tests/test_extraction.c cannot go red on its own. The
normal test build prints the UBSan message and carries on, which is why
this survived. The Windows CLANGARM64 leg runs UBSan in trap mode, and
there the same shift is an illegal-instruction crash -- so the test
exists to make sure that leg keeps parsing a force-unwrap at all.
scripts/vendored-checksums.txt records the new hash for the one changed
file, as scripts/security-vendored.sh --update writes it. Layer 8 of the
security gate compares vendored content against that manifest, so the
edit and its recorded hash belong in the same commit.
Found while adding Swift URL extraction in #1892 / #1976, and split out
of that PR so the vendored change can be reviewed on its own.
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
|
||
|
Martin Vogel
|
17b5a43263 |
fix(perl): update grammar for format declarations
Refresh the complete generated Perl grammar bundle from upstream v1.2.1 at c3e17b31179bf8f658c9f37c7a3ea6a202212d5a, raising it from ABI 14 to supported ABI 15. Preserve the existing license and record the source and per-file provenance. Add deterministic coverage for a valid format followed by a named sub, while retaining a malformed-source control. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a27266309c |
Merge pull request #1888 from DeusData/feat/919-chialisp-distill
feat(lang): add Chialisp support with a first-party tree-sitter grammar |
||
|
Martin VogelandPR9000
|
fece72b049 |
fix(build): put the whole GOT inside the RELRO window on ELF targets
Adds -Wl,-z,relro and -Wl,-z,now to ELF_HARDENING_FLAGS, and asserts the
resulting property on the produced artifact rather than on the compiler's
willingness to accept a flag.
The shipped Linux binaries measurably lacked the property. Built the way
.github/workflows/_build.yml builds them (STATIC=1, gcc 13.3 / GNU ld 2.42,
Ubuntu 24.04 aarch64), PT_GNU_RELRO ends at 0x11DA0000 in both cases:
without -z now: .got 0x11d9ec20+0x13c8, .got.plt 0x11d9ffe8+0x40
-> .got.plt ends at 0x11DA0028, 40 bytes PAST the window.
Those GOT slots stayed writable for the whole process
lifetime, in every Linux binary shipped to date.
with -z now: .got.plt folds into .got, the whole GOT sits inside the
window and is re-mapped read-only after startup.
Both flags are free at runtime for a static binary -- everything resolves at
link time, so there is no lazy binding left to pay for. -z relro is already
this toolchain's default and changes nothing here; it is named anyway so the
property stops depending on one distro's spec file, since the musl/portable
and glibc-floor images are different toolchains and a default is not a
guarantee.
A1c/A1d in scripts/ci/check-binary-composition.sh assert the OUTCOME on the
binary via readelf, for the same reason A1 exists beside the .note.GNU-stack
annotation: a flag the linker accepts is not evidence the artifact gained
anything. PR #1138 demonstrated that directly -- its -pie was silently
discarded under -static, and its -D_FORTIFY_SOURCE=2 silently DOWNGRADED
Ubuntu's default of 3, both while every compile probe passed.
The readers use awk rather than a short-circuiting reader on purpose: a
reader that exits at the first match leaves the upstream taking EPIPE, and
under `set -o pipefail` the satisfied case is reported as the failing one.
Deliberately NOT taken from #1138: -D_FORTIFY_SOURCE=2 (a downgrade on the
release platform), -fPIE/-pie (inert under STATIC=1), -fstack-protector-strong
(real but unmeasured cost, and #1138 injected it into SQLite, tree-sitter and
mimalloc -- the indexing hot loop), and the $(shell) probe machinery (~10
compiler spawns per Makefile parse, making the binary a function of the
builder's toolchain).
Co-authored-by: PR9000 <119280965+PR9000@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
5821078ee5 |
fix(pkg): correct stale packaging versions to v0.10.8 + gate the drift
Seven packaging surfaces declared 0.8.1 while v0.10.8 was the shipped release — nine releases of drift in metadata published to package registries, and the third version-metadata drift this month. pkg/go was not cosmetic: it hardcodes its version into the release URL it downloads from, so `go install` fetched v0.8.1 binaries. Chocolatey had to move as a unit. The nuspec is version-only but chocolateyInstall.ps1 pins the asset sha256, so bumping the nuspec alone would ship a package declaring 0.10.8 while installing 0.8.1. The new checksum is the v0.10.8 windows-amd64.zip digest, verified by downloading the published asset and hashing it, not merely copied from checksums.txt. The gate (tests/test_version_metadata_contract.sh, scripts/test.sh Step 0x) checks against the newest stable v-prefixed tag, because nothing in the tree declares the version: CBM_VERSION is injected at build time and the release is dispatched with a version input. That absence is exactly why these files drifted. Mutual consistency across the surfaces needs no git and always runs, so a shallow clone degrades the gate loudly instead of passing silently. Homebrew (0.10.3), Scoop and AUR (0.8.1) are left pinned: each embeds per-release asset checksums, so bumping them is a distribution chore rather than a metadata edit. They are still gated — internally consistent, and never naming a release that does not exist — and each carries a recorded reason for being frozen. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
51c48e2093 |
Merge pull request #1858 from DeusData/fix/issue-1834-pi-stdin-transport
fix(pi): send tool arguments over stdin |
||
|
Martin Vogel
|
ee0a2e6dc6 |
docs(grammars): recount the manifest's ABI distribution from the tree
The line read `7x ABI-13, 86x ABI-14, 65x ABI-15`, which sums to 158 against
161 vendored grammars. Counting `#define LANGUAGE_VERSION` in each parser.c
gives 9 / 79 / 73 = 161, so the line was already wrong before Chialisp; bumping
the ABI-14 bucket by one, as the Chialisp change did, would have carried a
four-off error forward under a fresh timestamp.
Recount with Chialisp included: 9 / 80 / 73 = 162, which matches both the
grammar-directory count and the language-count contract. The regeneration
command is now recorded next to the numbers so the next person does not have to
rediscover it:
grep -h '#define LANGUAGE_VERSION' internal/cbm/vendored/grammars/*/parser.c \
| sort | uniq -c
Vendored-integrity gate and the language-count contract both re-run green.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogelanddarkverbito
|
706bb2ce81 |
feat(lang): add Chialisp support with a first-party tree-sitter grammar
Chialisp (.clsp/.clib/.clinc) is the Chia smart-coin language. The only
public grammar, Quexington/tree-sitter-chialisp, cannot parse it: on the five
reference files from chia-blockchain@main it produced 3-34 ERROR nodes each.
Three defects account for all of them, and each desynchronised the rest of the
file rather than losing one node:
* comments were `/;.*\r\n/` -- CRLF-only, while real files are LF;
* `(include foo.clib)` was rejected because of the dot in the filename;
* `(defconstant NAME <expr>)` accepted only a primitive after the name.
So the grammar is written here instead, in tools/tree-sitter-chialisp/, and
files as first-party: its vendored LICENSE is the project's own, byte-identical
to the repository root, carrying no third-party copyright.
It is deliberately GENERIC -- source_file / list / symbol / string / number /
hex / dot / comment, modelling the clvm_tools reader rather than the Chialisp
form vocabulary. `mod`, `defun`, `defconstant` and `include` are ordinary head
symbols; which lists are definitions is decided in extract_defs.c. A dialect
that adds a form (cl-21 -> cl-23 -> cl-26 each added several) therefore needs
no regenerated parser. Acceptance: all five reference files parse with zero
ERROR and zero MISSING nodes; the three defects above are pinned as tests.
Extraction, and the four decisions in it:
* `Constant` is admitted to the cross-file name registry
(cbm_label_is_registry_symbol). Chialisp's condition codes live as
`(defconstant ...)` in a shared .clib and are referenced by name from every
puzzle; labelled but unregistered they are seeded nowhere, and roughly half
of a Chialisp graph would be present as nodes yet never the target of an
edge. All three registry-seeding sites route through that one predicate, so
they cannot diverge. No other language emits `Constant`, so the blast radius
is exactly Chialisp. Pinned by a two-file graph test that goes red when the
label is removed from the predicate.
* `export` / `namespace` are NOT definition heads. `(export foo)` names a
function already defined in the same file, so treating it as a def head
mints a second node for one symbol. They stay in the not-a-call filter.
* A parameter list is a `list` too, but it binds rather than invokes. Without
that gate `(defun check_conditions (HEIGHTLOCK conditions) ...)` mints a
phantom call to HEIGHTLOCK, and real Chialisp is mostly such definitions.
* `(include *standard-cl-26*)` is a dialect directive, not a file dependency;
recording it would invent a dependency on a file that does not exist.
Three helpers that the defs, calls and unified walks all need -- the quote-data
gate, comment-skipping named-child access, and the Chialisp def-head set -- live
in helpers.c rather than being copied per translation unit. They previously
existed as byte-identical duplicates that had already drifted: extract_defs.c
skipped comments when reading a def head while its mirror in extract_unified.c
did not, so a comment between a head and its name desynchronised defs from call
scope. Both now use the same accessor.
Ledger: 161 -> 162 grammars across the seven published surfaces, ABI-14 86 -> 87,
first-party 13 -> 14, CBM_LANG_COUNT 164 -> 165.
Distilled from PR #919 by darkverbito, whose extraction logic -- the CLVM
not-a-call vocabulary, the quoted-data gate, the filename-derived `mod` name and
the include/embed import handling -- is kept and is the substance of this change.
The grammar that PR vendored is not used.
Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
d706c33ac2 |
Merge pull request #1337 from abendrothj/bash-tool-hook-augment
feat(hook-augment): augment Bash tool searches via graph |
||
|
Martin Vogel
|
10a92a0ebd |
Merge remote-tracking branch 'origin/main' into pr1229-work
# Conflicts: # README.md # docs/index.html # docs/llms.txt # pkg/npm/README.md # src/cli/agent_profiles.c # src/cli/agent_profiles.h # tests/test_agent_profiles.c |
||
|
Martin Vogel
|
4ec0f5f09d |
Merge pull request #1859 from DeusData/chore/language-count-contract
docs: correct the published language count and gate it against the vendored grammars |
||
|
Martin Vogel
|
b0a3ca2861 |
Merge pull request #1105 from nguiaSoren/feat/335-watcher-enabled-config
feat(config): add watcher_enabled config key to disable the background watcher (#335) |
||
|
Martin Vogel
|
6c74ae382a |
docs: correct the published language count to 161 on every surface
Seven surfaces disagreed with reality and with each other: README.md, docs/index.html, docs/llms.txt and pkg/npm/README.md said 158, while server.json, the chocolatey nuspec and scripts/package-release.sh said 159. 161 grammars are vendored. 21 individual claims corrected. The count is now gated by tests/test_language_count_contract.sh, so the next language to land cannot ship a stale figure. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
4518c9ed67 |
test(contract): gate the published language count against the vendored grammars
The count was hand-maintained on seven separate surfaces and had drifted apart from reality AND from itself: README, docs/index.html, docs/llms.txt and the npm README said 158, while server.json, the chocolatey nuspec and package-release.sh said 159. Derived from the vendored grammar directories rather than CBM_LANG_COUNT, because the registry contains entries with no parser at all: CBM_LANG_NIM (grammar removed 2026-06-12) and CBM_LANG_OBJECTSCRIPT_EXPORT have no grammar directory, no tree_sitter function and no MANIFEST row, yet files still route to them by extension. Publishing CBM_LANG_COUNT would claim languages we do not parse. A vendored grammar directory demonstrably exists and is checksum-gated. Accepted trade-off, stated in the test: this UNDERCOUNTS languages that share one grammar (TypeScript/TSX, objectscript_udl/_routine). We publish the number we can prove rather than the larger number we cannot. The contract binds four things so it cannot go quietly vacuous: - the grammar directory count is sanity-bounded, so a layout change fails loudly instead of making every check below it trivially true - MANIFEST.md's declared figure must already agree with the directory count - every registered surface must state exactly that number, and a registered surface that states NO number fails (the wording changed, or it should be deregistered) - any UNREGISTERED file stating a language count fails, so an eighth surface cannot reintroduce the drift this exists to prevent Files that legitimately say "<N> languages" about something else — benchmark corpus subsets, probe scopes, the version-pinned winget manifests — are listed with what their number actually counts, so no exemption is unexplained. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
095acc5cee |
Merge pull request #1854 from DeusData/feat/plsql-language
feat(lang): add Oracle PL/SQL support (distilled from #1033) |
||
|
Martin Vogel
|
ee32816e3c |
fix(pi): send tool arguments over stdin
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
DavidHLP
|
e8d3c809df | Merge branch 'main' into feat/649-omp-client-surface | ||
|
Martin VogelandOğuz
|
97ecfe7fb0 |
feat(lang): add Oracle PL/SQL support (distilled from #1033)
Vendor AndreasMaierDe/tree-sitter-plsql @ 28aebef209be (MIT, ABI 14, no external scanner) and wire CBM_LANG_PLSQL end to end: - extraction onto existing labels only: package spec/body -> Class, procedure/function (create/definition/declaration) -> Function, file -> Module; callee extraction for ref_call incl. the package-qualified pkg.name form; identifier usage vocabulary with the upstream 'parameter' kind (a ref_call ARGUMENT wrapper) carved out of the whole-binding rule - discover: .pks/.pkb/.pck/.pls/.plb/.plsql/.fnc/.trg/.bdy/.tps/.tpb (.sql stays SQL and .prc stays FORM, both asserted negatively) - ledgers: CALL_WITHOUT_REFERENCE_VOCAB(PLSQL), call-node manifest ref_call row (historical total 219 -> 220), call-argument matrix B row - tests: extraction triple (package body + package-qualified call, standalone function, honest negative pin of the upstream CREATE TYPE ... AS OBJECT ERROR-node limitation), grammar-regression case, label golden Class:1,Function:1,Module:1, MUST_EXTRACT_DEFS, extension mapping, bug-repro board battery row - MANIFEST.md (community-niche verdict, local include-quote patch row), THIRD_PARTY.md section, refreshed vendored checksums PL/SQL support was contributed by Oguz (@ouzsrcm) in #1033; re-vendored from upstream per vendoring policy with his language wiring distilled on top. Co-authored-by: Oğuz <oguzhan.saricam@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
72ee88054f |
feat(cli): add Grok Build as an install target
Grok Build (xAI's grok CLI/TUI) already reached codebase-memory-mcp through its Claude/Cursor compat layer, but a Grok-only user got nothing. Detect $GROK_HOME (default ~/.grok) or a grok CLI and install: - an owned [mcp_servers.codebase-memory-mcp] managed block in config.toml; a same-name foreign table is left byte-identical because a second header would make Grok reject the whole file, while the known owned shape written by `grok mcp add` is adopted. No env forwarding: grok 1.0.5 passes the full parent environment to stdio servers (verified), unlike Codex (#1562). - an owned rules file under $GROK_HOME/rules/ (always scanned, all projects) - the shared skill under $GROK_HOME/skills/ - Scout/Verify/Auditor agents with named-server mcpInheritance and the tier allowlist spelled out as the server__tool ids Grok's search_tool/use_tool dispatcher accepts (inheritance filters per server, never per tool) - hooks withheld: Grok's passive events discard stdout and PreToolUse honors only deny/rewrite decisions, so there is no context-output contract. Verified against grok 1.0.5: inspect discovers the server, agents, skill and rules; mcp doctor handshakes and discovers 15 tools. --clients=grok token, README/npm/llms/landing-page matrices (43 -> 44 surfaces), and smoke coverage. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
2b51b7dc1a |
chore(vendor): register arkts in the vendored integrity baseline
Layer 8 (scripts/security-vendored.sh) failed because the seven new arkts files were untracked in the checksum baseline. Regenerated with --update after reviewing the diff: the only additions are the arkts grammar's own files, plus MANIFEST.md's hash changing because the preceding commit edited it. Nothing else was re-blessed. Verified independently rather than taken on trust: arkts/_common_scanner.h sha256 == typescript/_common_scanner.h sha256 (da66ef2bd14a...), i.e. the external scanner body is the already-reviewed typescript one, and arkts/scanner.c is a 21-line symbol-rename trampoline that allocates nothing and delegates scan straight through. All three security-static layers now pass locally: static allow-list, UI audit, and vendored integrity -- the last also reporting no subprocess, network, or dangerous calls anywhere in vendored grammar code. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
032ebf5327 |
fix(vendor): ship arkts upstream MIT byte-identical and register the fork
The vendored LICENSE carried tree-sitter-typescript's MIT text plus an appended derivative notice. That is not what MIT asks of a derivative -- it asks that upstream's notice travel with the code -- and the appended text meant the provenance audit could not byte-verify the file against upstream, so arkts landed as NO-MANIFEST-ENTRY and failed license-gate. The vendored LICENSE is now byte-identical to upstream's, and arkts is registered in the audit's FORKS map, which is defined for exactly this case: a self-maintained fork whose vendored LICENSE must match its upstream. That is a routing entry, not an exemption -- check_upstream still byte-verifies against the real upstream repository, so the fact is checked rather than asserted. Audit now reports arkts as IDENTICAL. Our copyright for the ArkTS additions is not lost, it moves to where our SOURCE is: the grammar.js header (which now states all three copyrights explicitly) and THIRD_PARTY.md, which ships inside the release archives. The edit to scripts/audit-license-provenance.py is a maintainer-only enforcement-surface change, made deliberately and stated here rather than slipped in; a contribution may never supply its own audit entry. No generated parser or source changes -- comments, licence text, manifest prose and one routing entry only. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |