mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-05 08:23:53 +08:00
a7f24eac472acfe0093719073f895325ef86ffb2
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
3b93db06b2 |
feat(infra): a script's curl / wget request HTTP_CALLS its Route
Shell scripts, Make recipes and CI steps download tools and call services
with curl and wget; the graph had no edge for any of them, so neither what
a repository fetches nor the scripts that call its own services were
visible.
Such a command now HTTP_CALLS the Route of each URL it states, with
{via: curl | wget, method, url_path, host, line}:
- the method from the flags: -X / --request / --method, a body (-d, -F,
--data*, --json, --form*, wget --post-data / --post-file) POST, -T PUT,
-I HEAD, -G back to GET;
- the URL from constant words (VER=1.2.3 ... v${VER}), unknown parts `{}`
as in a Route path -- also in an unquoted word tree-sitter-bash splits
around its expansions; a URL whose host is not literal, or the value of a
proxy / referer option, is no request;
- never the user info, the query or the fragment: a token in a URL does not
reach the graph;
- the Route: path-only (__route__POST__/v1/items) when the host is a
Kubernetes Service of the repository in a service DNS form, where that
service's handler meets it as a code client's request does; any other
host -- a download, localhost, a compose service -- keeps it
(__route__GET__https://dl.k8s.io/release/{}/bin/linux/amd64/kubectl,
props {method, host, external}), and route_nodes makes no path-only twin
of it.
Incremental indexing now drops a Route without a file that no edge touches
any more -- its last caller's file changed or went. A full index never
holds one; before, the Route of a deleted code client stayed behind too.
Measured on 17 infrastructure repositories: 92 edges, every one correct by
hand; an independent reading of the scripts and Makefiles finds 82
requests, 81 of them in the graph (the last is lost in a file
tree-sitter-bash cannot parse past `[ "$v" \< "1.33" ]`). No property of
any other node or edge changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
2cd0956560 | Break-glass merge PR #2597 | ||
|
Martin Vogel
|
cfab5d75f6 |
fix(graph): the id counter is never lowered after a parallel phase
After a parallel extract or resolve, the main buffer's counter was set to the shared allocator's value. Merged worker nodes keep their shared ids, and merged edges take fresh ids from the main counter after the highest of them, so the main counter can end above the shared one. Setting it down handed the merged edges' ids out a second time. All four sync points (full route and closure repair, after extract and after resolve) now go through cbm_pipeline_sync_next_id, which only raises the counter. The delta preseed still pins it with cbm_gbuf_set_next_id, as before. On the 17 infra corpora the old sync left 94,239 duplicate edge ids in the buffer (11 corpora; deckhouse 44,959), and the fix leaves none. The persisted graph does not change, because the dump numbers edges itself. The duplicates matter to anything that finds buffer edges by id: cascade_delete_edges unindexes by id and then frees the edge, so a duplicate can leave a freed edge in an index. Tests: a parallel full index with the counters pinned by a test seam fails with a duplicate edge id without the fix. Both full-route and both closure-repair syncs are checked to run through the helper. gbuf_raise_next_id_never_lowers covers the new buffer call. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
231bcbf6f5 |
feat(docker): a repository built per component splits into component Images
An image is its repository, tag and digest on the edges -- until one
repository is built from several Dockerfiles, one per component tag:
opentelemetry-demo builds every service as
ghcr.io/open-telemetry/demo:<version>-<service>, so its one Image was
built from 25 Dockerfiles and a walk from any service through the image
reached all of them.
Such a repository is now split by component: its build tags without
their common prefix / suffix (cut at separators -- latest-frontend,
latest-cart -> frontend, cart). A component built from exactly one
Dockerfile, with no unknown part, becomes an Image of its own,
`<repo>:*-<component>` {repository, component, tag_pattern}; its build
edges leave from it, and every reference whose tag names a component (a
K8s container, a compose service, a FROM) uses it -- whatever its
version, the longest component a tag holds as whole parts, none on a
tie. Everything else stays on the repository Image, which goes when no
edge is left (now on the full route too). Recomputed whole each run.
A closure repair cannot re-point the references of files it does not
repair, so when the run's component Images differ from the previous
ones it hands over to a full rebuild.
Edges are deleted by pointer (cbm_gbuf_delete_edges): after the
parallel resolve merge, cbm_gbuf_set_next_id can hand an edge id out
twice, and a delete by id took unrelated edges with it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
cbe2b420d6 |
feat(docker): an image a skaffold or compose file builds DEPENDS_ON its Dockerfile
The image a workload runs and the Dockerfile that builds it were not
joined: Service -> Deployment -> Image stopped there. Build
configurations now say which Dockerfile builds which image:
- skaffold: build.artifacts (top level and profiles) with the docker
builder -- image, context (default the file's directory),
docker.dockerfile (default Dockerfile, relative to the context);
ko / jib / bazel / buildpacks / custom / kaniko artifacts are none.
- compose: services with build (a context, or context / dockerfile)
and image; ${X}, ${X:-d}, ${X-d}, $X from the project's .env.
Each becomes Image -DEPENDS_ON {via: build, tool, file, image, line}->
the Dockerfile's Module (pass_ci.c reads the YAML, pass_docker.c
links). On microservices-demo the chain is now whole:
Service/frontend -> Deployment/frontend -> Image frontend ->
src/frontend/Dockerfile -> main (src/frontend/main.go).
These edges belong to the build configuration declaring them, not to
either end, so every run recomputes them whole from all build files
(read through the File nodes, on every route): the incremental routes
drop the old set first (the staging graph before the delta patch, the
full-load buffer before the inbound-edge capture), and the delta patch
writes them even between nodes that both pre-existed.
cbm_gbuf_delete_edges_from_label / cbm_delta_drop_edges_from_label
delete a type's edges by source label.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
2ec0185b9a |
feat(docker): a Dockerfile DEPENDS_ON the images it builds on
Dockerfiles were Module + ENV/ARG Variables only: the 261 FROM and
114 COPY --from lines of the infra corpora joined nothing, and build
files named for a variant were not indexed at all.
- pass_docker (new, end of the K8s pass): FROM <image> and
COPY --from=<image> -> DEPENDS_ON {via: from | copy-from, image,
stage, line} from the Dockerfile's Module to an Image node. ARG
defaults are expanded (global ARGs in FROM, a stage's ARG / ENV in
COPY); a stage (by name or index), scratch and an ARG without a value
are no image.
- Image (new label, as approved): one node per repository, normalised
(docker.io for no registry host, library/ for an official image, a
registry host and port kept); tag and digest stay on the edge.
- Detection: Dockerfile.<x>, Dockerfile-<x>, Dockerfile_<x>,
Containerfile[.<x>] and <x>.Dockerfile are Dockerfiles, unless the
suffix is a language extension of its own (Dockerfile.j2 stays a
Jinja template) or a BuildKit ignore file.
- QNs: a Dockerfile.<variant> keeps its suffix (fqn.c and helpers.c,
which must agree); dropping it gave the variant the plain
Dockerfile's QN, merging both Modules. Covered by this branch's
index-format bump.
- Incremental: an Image no DEPENDS_ON / USAGE edge touches any more is
dropped on both routes, like a Process without SPAWNS; the
unreferenced-node GC takes a list of edge types.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
cb497a8218 |
fix(incremental): drop Process nodes no SPAWNS reaches any more
A Process node exists only as the target of a SPAWNS edge (a call or a shell command starting a program) and belongs to no file. When the last file starting the program was deleted or edited, an incremental re-index removed the edge but kept the node, which a full index never makes: the two diverged (gardener: deleting hack/tools.mk left install-promtool.sh and install-protoc.sh behind). Both incremental routes now drop such nodes: the full-load route on its in-memory graph before the post-passes (new cbm_gbuf_delete_unreferenced), the closure-repair route on the staging database right after the patch, the first point where that route sees the whole graph (its in-memory buffer holds no edges). A full index is unchanged on 17 repositories (all graphs identical). Deleting a makefile included by another, a script CI jobs run, or a makefile a recipe makes into now leaves the graph a fresh index builds, apart from SEMANTICALLY_RELATED edges, which differ as before. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
dd3a294737 |
feat(k8s): Go operators link to the kinds they reconcile
A Kubernetes operator written with controller-runtime ties its Go code
to cluster objects in ways the type graph does not carry: the
controller builder names the kinds it reconciles, a CRD's group and
kind are backed by a Go struct, and the manifests ship inside the
binary through //go:embed. None of it was in the graph -- on the
kubermatic, gardener, cluster-api and deckhouse operators, "which
controller reconciles Seeds" or "which Go type is this CRD" had no edge
to walk.
A new operator pass (after USAGE resolution, reading each Go file once)
adds:
- HANDLES {via: controller, role: for|owns|watches} from the function
that builds a controller -- `For(&v1.X{})`, `Owns(...)`,
`Watches(...)`, `source.Kind(cache, &v1.X{})` -- to the Go type,
resolved as Go resolves it: the alias is an import of the file, and
the import path maps to a directory through the repository's go.mod
module roots (kubermatic's `k8c.io/kubermatic/sdk/v2/apis/...` lives
in `sdk/apis/...`); unqualified types are the file's own package.
gofmt chains with the '.' on an earlier line, comment lines between
links, and calls inside comments are handled;
- USAGE {ref: crd-type} from a CRD Resource (now carrying crd_group and
crd_kind) to `type <Kind> struct` in the package marked
`+groupName=<group>` or declaring `GroupName = "<group>"`;
- REFERENCES_FILE {via: go:embed} from a Go file to every file its
//go:embed patterns match (Go's path.Match per segment, hidden files
only under all:).
Go method QNs carry no receiver, so every reconciler's SetupWithManager
in one package is a single node standing for several definitions. The
graph buffer gains cbm_gbuf_foreach_span, which visits every definition
span a node stands for (collision spans not yet settled, else its
variants, else its own), so a call is placed in the right definition
whichever file it is in.
Measured on 17 infrastructure repositories against an independent
reading of the Go sources: controller HANDLES recall 343/343, precision
335/335 (kubermatic 125, gardener 79, deckhouse 80, cluster-api 59
facts); kubermatic acceptance: reconciler -> kind 13/13, CRD -> Go type
32/32, go:embed files 115/115. Compared with the previous build only
the new edges and the two CRD properties differ; three runs build
identical graphs.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
59ff283e5f |
fix(graph): a node's variant spans no longer depend on worker order
Two runs of the same binary over deckhouse disagreed on the `variants` property of QN-colliding nodes. Two causes: - A cap: at most 64 spans per node. Generated code exceeds it (protobuf `String()`/`Reset()`, `DeepCopy` across a package, 7,586 junit `testcase` elements in one file), and which 64 survived followed the merge order of the parallel workers. - A label gate: spans were recorded only between definitions of the same label. A Function arriving between same-QN Methods (Go method QNs carry no receiver: pkg/log has func Error and three Error methods) replaced the node's list, so logger.go's span came and went with scheduling. The node now carries every definition span that collapsed into it, whatever the labels and however many: the list is the union of the arrivals, a function of the set, never of its order. That is also what the list is for -- a change in any of those bodies lands on this node. Lifting the cap made rewriting the JSON list on every collision quadratic in the collisions of one QN (deckhouse +60% wall time), so the spans are now gathered in a side table and written once, when the buffer is exported: dump, flush, store merge and the incremental delta patch. A worker buffer hands its spans to the main buffer on merge instead of writing them, so every list is written at the same point of the pipeline whichever worker held the colliding definitions, and the rewrite keeps the number text other passes printed. On the 17 infrastructure corpora: three runs per corpus are identical node for node and edge for edge (before: deckhouse differed on 5-8 nodes per pair of runs); against the previous build every edge and node identity is unchanged, 380 lists gained the spans the cap or the label gate dropped, none lost one, 6,391 only moved the key to the end. deckhouse wall time 10 s, as before. Tests: cross-label lists in eight arrival orders and 100 spans in two orders fail on the old code; the worker-merge hand-over and the write-once text are covered too. Two merge tests that pinned the cross-label exclusion now expect both definitions' spans. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7b77fd4834 |
feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.
WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)
The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.
WHAT CHANGED
Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.
Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.
Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.
The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.
The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.
Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.
Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).
The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.
The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.
Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.
SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).
Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.
Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.
The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.
Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.
PROOF
Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
jitter (gRPC Route nodes: two identical runs differ by 21).
Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
surface rows, nodes/edges inside the Route jitter.
Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
inside the +-400 run-to-run band the kernel shows between identical
runs.
Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
outside what spill can move). Nodes identical; edges 15,773,598.
Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
every phase mark under 15,000 MB by every metric, but the process
RSS high-water mark reached 16.9 GB inside the semantic pass (class
peak 4.9 GB against 0.8 GB live).
Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
stays under 15,000 MB through the whole semantic pass (14,018 ->
14,381 MB across its sub-phases, 6 batches of 125,644 functions);
every phase mark is under budget by every metric (max footprint
12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
15.65 GB and is set in extraction, at the moment the gate latches
(15,002 MB charged) and before the first sweep lands -- a 4%
overshoot of in-flight work, no longer the semantic pass. Nodes
identical, edges 15,772,913, zero foreign blocks.
Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
enters at 14,062 MB (near_budget); the charged footprint never
exceeds 11.8 GB at any mark; the process RSS high-water mark is
15.1 GB in extraction and 15.3 GB during the final write-out --
1-2% over, the in-flight window and the dump transient. Nodes
identical, edges 15,772,543, zero foreign blocks.
Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
peak 998 MB RSS, graph identical to the baseline.
BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)
corpus wall s peak RSS GB nodes / edges / CALLS
perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3
php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8%
rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369
c 5.9 -> 5.7 0.16 -> 0.19 identical
kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1%
django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4%
go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2%
java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6%
csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6%
typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130
kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1%
No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (
|
||
|
Martin Vogel
|
1d7912d2df |
perf(gbuf): release indexes before dump transients; dense by_id; drop dead field
Three memory quick-wins from the gbuf compaction research (graph output verified identical; all delete/incremental paths suite-covered): 1. Release ALL lookup indexes right after build_dump_nodes instead of after edge building: nothing later in the dump reads them, and at kernel scale they are ~3.8 GB (hash buckets + key strings + pointer arrays) that previously coexisted with every dump-side transient — the dump-phase RSS peak. At 2x-kernel scale this is ~7.5 GB at exactly the OOM point. 2. node_by_id was a hash table keyed on STRDUP'D DECIMAL STRINGS of the int64 id — ~0.44 GB of buckets and key strings, plus a snprintf + strdup + hash on every hot find_by_id call. Ids are dense and sequential; a plain grow-on-demand pointer array replaces all of it with one bounds-checked load. 3. The per-node and per-edge `project` field was write-only (every reader uses the store structs or gb->project directly; -Werror confirms) — 8 bytes x 24M instances ~= 192 MB of dead pointers, deleted. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e81a796fe6 |
Embed Nomic nomic-embed-code pretrained vectors (40K tokens × 768d) for semantic search quality
Major quality improvement: replace random index vectors with pre-trained Nomic nomic-embed-code token embeddings. Tokens like 'error' and 'exception' now start with similar vectors (learned from millions of code repos) instead of arbitrary random projections. Co-occurrence enrichment adds project-specific context on top. Architecture: - vendored/nomic/code_vectors.bin: 37.7MB raw int8 vectors - vendored/nomic/code_vectors_blob.S: assembler .incbin (instant build) - vendored/nomic/code_vectors.h: extern declarations + pretrained_vec_at() - vendored/nomic/code_tokens.h: 40856 token strings (575KB) - semantic.c: cbm_sem_random_index() now looks up pretrained vectors first, falls back to sparse random for unknown tokens - CBM_SEM_DIM raised from 256 to 768 to match Nomic nomic-embed-code Also: RRI (Reflective Random Indexing), code pattern vocabulary injection, 120+ abbreviation expansions, callee/caller/body token enrichment, label filter (Function/Method/Class only) in vector search SQL. Binary size: 136MB → 169MB (+33MB from embedded vectors). Search quality: keyword queries return relevant error-handling functions. Domain-specific keyword queries return the expected functions. |
||
|
Martin Vogel
|
7a9b7db830 |
Add semantic_query to search_graph: vector search via cbm_cosine_i8 SQL function
Vector search pipeline (end-to-end): - store.c: cbm_cosine_i8 custom SQLite function (int8 dot product with sqrt magnitude), registered at store open time - store.c: cbm_store_vector_search() builds merged query vector from keywords via base RI vectors (xxHash), int8-quantizes, runs SQL query with JOIN nodes ON node_id = id for metadata - mcp.c: search_graph gains semantic_query param (array of strings), returns semantic_results alongside regular results - sqlite_writer.c: fixed build_vector_record to include node_id in record body (matches build_node_record pattern) - mcp.c: use yyjson_mut_obj_add_strcpy (not add_str) to avoid use-after-free when vector results are freed before JSON serialization Quality note: base RI vectors at query time don't capture co-occurrence enrichment — results are moderate quality. Storing enriched token vectors would improve this significantly (future work). |
||
|
Martin Vogel
|
80680ea367 |
Cross-service communication discovery + RAM-first incremental indexing
AST-based detection of HTTP calls, async dispatch (Pub/Sub, Cloud Tasks, Kafka, SQS, etc.), and config accesses via resolved qualified names. Route nodes as cross-service rendezvous points with infra→handler matching. Constant propagation for module-level string assignments. YAML infrastructure URL extraction from Cloud Scheduler configs. RAM-first incremental pipeline: load DB into graph buffer, purge changed file nodes, extract directly into existing buffer (resolver sees all nodes), dump back to disk. Zero edge gap on kubernetes/django/meilisearch/neovim. - service_patterns.c: ~170 library patterns (90 HTTP, 50 async, 30 config) - pass_route_nodes.c: Route node creation + infra URL matching - extract_unified.c: string constant collection + string ref classification - extract_calls.c: first_string_arg + keyword argument extraction - pipeline_incremental.c: RAM-first load→purge→extract→resolve→dump - graph_buffer.c: load_from_db, delete_by_file, foreach visitors - C++ LSP crash fix: NULL guard in cbm_type_substitute |
||
|
Martin Vogel
|
d6a5f8984e |
Add incremental reindex + ADR hints in index/schema/adr responses
Incremental pipeline (pipeline_incremental.c): - Disk-based delta re-indexing: classifies files by mtime+size vs stored hashes, deletes changed nodes (edges cascade), re-parses only changed files, merges into existing DB via cbm_gbuf_merge_into_store - pipeline.c auto-routes: existing DB with hashes -> incremental, else full - Full pipeline persists file hashes after dump for next incremental run - No-op when nothing changed (watcher polls every 5-60s, 99% skip) - 4 tests: noop, changed file, deleted file, new file added ADR hints (ported from Go): - index_repository: adr_present + adr_hint when no ADR exists - get_graph_schema: adr_present + adr_hint per project - manage_adr GET: adr_hint with creation instructions when no ADR |
||
|
Martin Vogel
|
94a9a92f83 |
Add C-native pipeline with blocking clang-tidy, cppcheck, clang-format linting
Complete C implementation of the indexing pipeline (src/), parallel worker pool, graph buffer with merge support, SQLite writer, and 1893-test suite. Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT suppressions), cppcheck, and clang-format. All issues fixed at source — proper headers for external linkage, GROW_ARRAY macro restructured to eliminate type parameter, null-deref paths guarded, named intermediates for suspicious-argument checks. |