mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-21 05:59:35 +08:00
docs/cli-daemon-behavior
147
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
0bee4c1b79 |
Merge pull request #2406 from DeusData/fix/spill-test-disk-determinism
test(spill): make spill test verdicts independent of host free disk |
||
|
Martin Vogel
|
1ff6299976 |
Merge pull request #2354 from DeusData/fix/issue-2235
fix(extract): read the url of a JS/TS request-config object (#2235) |
||
|
Martin Vogel
|
5910c47a97 |
Merge pull request #2332 from DeusData/fix/issue-1916
fix(pipeline): compose axios.create baseURL into HTTP_CALLS (#1916) |
||
|
Martin Vogel
|
f5ecb5f739 |
Merge pull request #2321 from DeusData/fix/issue-1457
fix(dart): index extension declarations and their members (#1457) |
||
|
Martin Vogel
|
17718126ea |
Merge pull request #2375 from DeusData/fix/complexity-walker-no-silent-cap
fix(extract): AST walker stacks grow instead of silently capping content |
||
|
Martin Vogel
|
d82e1e6014 |
Merge pull request #2326 from DeusData/fix/issue-1735-sql-cost
fix(sql,coverage): make large SQL dumps cheap to parse (#1735) |
||
|
Martin Vogel
|
d17f97f431 |
Merge pull request #2352 from DeusData/fix/issue-2176
fix(rescript): stop binary .res resources hanging the indexer (#2176) |
||
|
Martin Vogel
|
64cf275ffa |
Merge pull request #2367 from isc-tdyar/fix/iris-cls-dispatch
fix(python): resolve iris.cls("Pkg.X").Method() to the named class
|
||
|
Martin Vogel
|
a1b7c05685 |
test(spill): make spill test verdicts independent of host free disk
extraction::extract_spill_round_trip_keeps_every_field and parallel::parallel_spill_mode_builds_the_same_graph failed on any machine with less than 10 GB free: cbm_result_spill_open refuses to spill onto a nearly full disk (mem.spill.refused reason=low_disk), so the store never opened. Three pipeline tests took the same refused-store path there and passed without spilling at all (backpressure_futile_nap_disengages, over_budget_after_futility_fails_whole_and_preserves_db, spill_resolves_namespace_imports_like_memory). A verdict must be a pure function of code, test, platform and seed, not of how full the disk is. Add a free-space pin to result_spill that exists only in test-seam builds (CBM_ENABLE_TEST_SEAMS). Production still asks the disk, and its 10 GB floor and half-of-free cap are unchanged. The test runner pins 64 GiB at the top of main, before worker re-exec dispatch, so every suite and re-exec'd index worker sees the same disk. A new test, extraction::extract_spill_refuses_below_the_free_disk_floor, pins the guard itself: one byte under the floor refuses, the floor opens. Verified with cbm_fs_free_bytes temporarily forced to 9 GB: both tests RED 3/3 before, green 3/3 after, RED again with the runner pin removed; the refusal test goes RED with the production guard disabled. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7707c9d791 |
fix(extract): AST walker stacks grow instead of silently capping content
Three iterative AST walkers kept their pending nodes in fixed-size stacks
and silently stopped pushing children once full:
- cbm_compute_complexity and count_branching_iter (helpers.c,
BRANCHING_STACK_CAP 4096) -> cyclomatic, cognitive, loop and access
metrics
- extract_body_ident_tokens (extract_defs.c, BT_STACK 512) -> the
body-token sample that feeds semantic search / SEMANTICALLY_RELATED
Children are pushed last-to-first, so a full stack dropped a wide body's
FIRST children. The metrics undercounted, and the token sampler read a
mid-body window instead of the first identifiers. That let a work cap decide
graph content.
Real input: tree-sitter-llvm's generated ts_lex (a 4654-case switch)
reported cyclomatic 11133 / cognitive 29303. It really has 8776 if + 4654
case + 1 default + 1 switch = 13432 branches, which is what it reports now
(cognitive 35639). On the Linux kernel no function reaches the old 4096 cap,
so complexity is byte-identical for every function. Three functions with
bodies over 512 pending nodes now sample their tokens from the start of the
body (for example rimob_panel_on), which moves 11 SEMANTICALLY_RELATED edges.
Index time is unchanged within noise (main 214.4/207.9 s, fix 197.6/192.2 s).
Fix: cbm_walk_stack_reserve()/cbm_walk_stack_release() in helpers.c. Each
walker keeps a small on-stack first chunk (512 frames) and spills to the heap
through the memory core (cbm_alloc/cbm_realloc/cbm_free, class EXTRACT) only
for deep or wide trees. Traversal order is unchanged. If the stack cannot
grow, the failure is logged ("extract.walk_stack_alloc_failed") and the
walker returns a documented sentinel instead of a truncated result:
CBM_WALK_METRIC_UNAVAILABLE (-1) for the metrics, or no token sample.
The body-token sampler's caps of 128 identifiers and a 2048-byte buffer are
its sampling contract and stay. The 512-entry stack was never part of that
contract, because it broke the "first identifiers in source order" property.
Tests (extraction suite): a C function with 5000 top-level if/while
statements must report exact cyclomatic/cognitive/loop counts through both
the extractor and cbm_count_branching, and a 600-call body must sample from
f0 on. All three are RED on main (4095 / 4095 / sample starting at f91) and
GREEN with the fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
f4654a7c77 |
Merge pull request #2373 from DeusData/fix/defs-walk-no-cap
fix(extract): definitions walk stack grows without a ceiling |
||
|
Thomas Dyar
|
075f40e9d4 |
fix(python): make iris.cls() callee primary, drop the extra bare call
When the receiver is iris.cls("<literal>"), the class-qualified name now
replaces the primary call's callee instead of being pushed as a second
call. Keeping the bare name let it weak-match an unrelated same-named
method.
Tests assert exactly one Run call per iris.cls() site, and the pipeline
fixture adds two unrelated Run definitions that must receive no edges.
Refs #1260
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
|
||
|
Martin Vogel
|
77ab4bea87 |
fix(extract): definitions walk stack grows without a ceiling
walk_defs keeps its pending frames in a growable stack that was bounded by an 8M-frame ceiling (env CBM_WALK_DEFS_MAX). At the ceiling wd_push logged "extract.walk_defs_capped" once and then stopped pushing. Children are pushed last-to-first, so a file wider than the ceiling lost its FIRST definitions: a work cap decided graph content. Fix: no ceiling. The stack doubles on demand, heap frames now come from the memory core (cbm_realloc/cbm_free, class EXTRACT) instead of safe_realloc and raw free, and the doubling is overflow-guarded (int capacity and size_t byte count). The only thing that can stop the walk is an allocation failure, and that is loud: logged at error level as "extract.walk_stack_alloc_failed" (walker=walk_defs, pending, path) -- the same event the complexity walkers use -- and the file's result is marked has_error with "definitions walk: stack allocation failed", which the definitions passes already report as a per-file extract error. The arena-backed path (ctx->scratch) gets the same failure handling. Traversal order is unchanged. CBM_WALK_DEFS_MAX is removed. It was read only in wd_stack_max(); it was not documented anywhere (docs, --help, config, tests), so nothing else changes. extract_defs.c loses its raw free() call, so the memory-core baseline for the file drops from 5 to 4. Test (extraction suite): walk_defs_wide_file_extracts_every_definition extracts a C file with 1000 top-level functions with CBM_WALK_DEFS_MAX=256 set, and asserts every wd0..wd999 definition is present. RED on main (extract.walk_defs_capped limit=256; wd0 missing), GREEN with the fix, RED again with the production change reverted. The extraction (379), complexity (5), ast_profile (10) and semantic (35) suites pass. Real input: Linux kernel, index-only, fresh cache and runtime dirs, same machine. main and fix both produce 8,530,336 nodes and 15,764,128 edges, SEMANTICALLY_RELATED 607 each, and the full node set (label, QN, lines) and edge set (type, source QN, target QN) hash identically -- no kernel file approaches the old ceiling. Index wall time 210.0 s (main) vs 194.4 s (fix), within run-to-run noise. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Thomas Dyar
|
8f803f7262 |
fix(python): resolve iris.cls("Pkg.X").Method() to the named class
In IRIS embedded Python, iris.cls("Pkg.X").Method(...) names its target
class in the receiver's string literal. There was no receiver path for it
next to the classMethod* string dispatch, so after the weak-member-call
suppression in
|
||
|
Martin Vogel
|
3ef7d8cdc2 |
fix(extract): read the url of a JS/TS request-config object (#2235)
A generated API client (Orval and similar) passes its request as a config
object to a local fetch wrapper:
http<Customer>({url: `/api/customers/${id}`, method: 'GET', signal})
The template literal itself already canonicalizes to "/api/customers/{}"
(#1006). What was lost is the object wrapper: the call's arguments only
carried a value for bare strings, template literals, constants and URL
builders, so the object argument had no URL. The arg-url heuristic
therefore minted no HTTP_CALLS edge, and cross-repo matching had nothing to
join against the server's GET /api/customers/{id} Route. The same held for
HTTP clients called with a config object (`axios({url})`): no URL, no edge.
The extractor now reads the `url` property of an object argument in
JS/TS/TSX/ArkTS -- both as that argument's value (for the arg-url
heuristic) and as the call's URL (for HTTP-client classification). Only
the `url` key counts. Route registrations (Fastify `route({url, handler})`)
and axios `getUri(config)`, which formats a URL and sends nothing, are
excluded.
orval-labs/orval @ 08d7fcf1 (parallel resolver): HTTP_CALLS 1383 -> 1533
(+150 arg_url edges from the generated mutators such as customInstance,
responseType and customClient; 0 lost), Route 302 -> 308, CALLS unchanged
(19190). Without the getUri exclusion there would have been 546 URL-builder edges.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
8093d10b81 |
fix(rescript): stop binary .res resources hanging the indexer (#2176)
A 4.7 MB binary Godot resource (piko_walk_mesh.res) never finished indexing: `.res` maps to ReScript, so the resource was parsed as ReScript source and the definitions pass spun. Three defects, each fixed: 1. Quadratic usage walk (the hang). The ReScript `let` binding check (is_first_named_part_of, shared with Julia, Typst, Elm, PureScript and Nickel) climbed from every named leaf with ts_node_parent, which descends from the root and scans each level's children. A binary file parses into a flat error tree (~170k root children per MB), so each climb cost O(root children) and the file O(n^2): 1 MB spent 46 s in cbm_extract_unified, 4.7 MB never returned. The check now climbs the unified walk's own cursor (O(1) per hop), like the other occurrence classifiers; the node-based climb stays only for callers outside the walk. 1 MB: 46 s -> 0.03 s; 4.7 MB: 2.2 s end to end. 2. Quadratic lexing in error recovery. tree-sitter retries the lexer at every byte of an unparseable stretch with every external token valid; the ReScript scanner's template-string loop then ran to the next backtick/$/backslash/NUL (end of file if none) and discarded the result. That cost O(n^2) inside lexing, where the parse budget's progress callback never runs, so such files were dropped by the 5 s CPU budget instead of parsed -- plain text too (100 KB of non-ReScript text: 15 s -> 0.18 s). Local scanner patch: skip the template branch when NEWLINE is also valid, which only the ERROR state allows. Valid files parse to identical trees. Recorded in vendored/grammars/MANIFEST.md. 3. Binary content parsed as source. `.res` is also Godot's binary resource format and the Windows compiled-resource format. Following the .m/.cls/.frm/.inc/.cfc precedent, cbm_disambiguate_res reads the first 4 KB and reports a file holding a NUL byte as unsupported; ReScript text never contains one. Tests (deterministic work counters, no clocks): - extraction::extract_rescript_let_bindings_use_the_walk_cursor -- zero root-descending parent hops at 128 and 1024 statements (before: 515 / 4099). - complexity::complexity_rescript_error_recovery_lexing_is_linear -- lexer bytes pulled through a chunked TSInput, 4 KB vs 8 KB, ratio must be linear (before 3.99, after 2.00). - language::lang_res_binary_resource_unsupported and lang_res_rescript_stays_rescript. Fixes #2176 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e4780c2086 |
fix(coverage): stop walking the source per error region in the #1071 check (#1735)
cbm_subtract_macro_invocation_regions asked "is this region a call to a file-defined macro?" before "is it inside a function?". The first question (cbm_span_is_macro_invocation) walks the source from byte 0 to find the region's lines, so every file with parse-error regions paid regions x file bytes, whatever its language and whether or not it has a single function. On a multi-MB SQL dump that is up to 256 walks over the whole file. Both predicates are pure, so asking the cheap one first changes no answer: a region no function encloses never touches the source, and the regions that do share one cbm_line_offsets table built on first use. The table gives exactly the span the walk found; the walk stays as the fallback when the table cannot be allocated. #1071 behaviour is unchanged: same regions, same verdicts. Indexing this repository's src/ before and after gives identical graphs (7,339 nodes, 28,924 edges) and the same 46 parse_partial files with byte-identical ranges. Tests count the bytes read, never time them (test-seam counter cbm_test_macro_line_scan_bytes): - 40 top-level junk regions: 39,870 bytes read -> 0, still 40 regions - 60 in-body real errors: 85,725 bytes read -> 2,801 (one table = file size), still 60 regions Refs #1735 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c6ccdd9395 |
fix(dart): index extension declarations and their members (#1457)
The Dart language spec listed class_definition, enum_declaration,
mixin_declaration and type_alias as container kinds, but not
extension_declaration. The def walk never treated `extension X on Y { }`
as a container, and extract_func_def could not name the bare
`method_signature` wrappers in the extension body. So the extension and
every member were dropped: only the Module, File, the extended class and
top-level functions made it into the graph.
- lang_specs.c: add extension_declaration to dart_class_types. A named
extension becomes a Class node, and its members become Methods with
DEFINES_METHOD edges from it. In-body calls get the method as caller.
- extract_defs.c: an unnamed `extension on T { }` has no name to attach
its members to. extract_class_def would return before
extract_class_methods and drop them. Walk such an extension's members
as file-level Functions instead, unwrapping method_signature to its
named function_signature.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
b9ad9ed144 |
fix(pipeline): compose axios.create baseURL into HTTP_CALLS (#1916)
The Vue-admin wrapper pattern -- `const api = axios.create({ baseURL:
'/api' }); export default api` in one module, `api.get('/orders')` in
another -- produced no HTTP_CALLS edge, so cross-repo-intelligence found
nothing to link to the backend's GET /api/orders.
Root cause: nothing recorded that `api` is an axios instance. The call
`api.get('/orders')` fell through to the route-registration suffix
fallback (`.get` + a '/'-leading argument), which treats it as an Express
`app.get` REGISTRATION: a CALLS edge to a phantom server-side Route
`/orders` (via=route_registration), and never an HTTP_CALLS edge. The
base path was unknowable anyway.
Fix, reusing HTTP_CALLS/Route (no new labels):
- extraction (JS/TS/TSX/ArkTS): a module-level binding initialised by
`axios.create(...)` records http_client="axios" and the literal
`baseURL` (NULL when absent or non-literal, never guessed); the Module
def carries the same pair when its default export is such a client
(`export default api` / `export default axios.create(...)`). ES default
imports are flagged (CBMImport.is_default).
- both call resolvers (pass_calls.c, pass_parallel.c), before the #523
callee-spelling check and the route-registration fallback: a
`<recv>.<verb>(path)` whose receiver binds -- same file, named import, or
default import -- to a client emits HTTP_CALLS to base + path (axios
combineURLs: trailing '/' trimmed, absolute URLs kept as-is, path kept
when the base is unknown). Shared helper
cbm_pipeline_http_client_call_url keeps the two paths identical.
- incremental: the client pair joins the file's LSP surface (only for files
that have one, so every other surface sha is unchanged), so a baseURL
edit pulls importers into the closure; delta preseed proxies carry the
two keys so a caller re-resolved alone still sees its unchanged wrapper.
- result_compact relocates the two new strings (ASan caught the dangling
worker-arena pointer).
Look-alikes (`factory.create({ baseURL })`, a plain object's `.get`) are
unchanged.
Test: pipeline_axios_wrapper_baseurl_composes_http_calls_issue1916 --
default/named/same-file/dynamic-base/look-alike cases, sequential vs
parallel resolver equality, and same-DB closure-repair reindex (wrapper
edit, then caller-only edit) converging with fresh indexes. RED on origin/
main (seq.wrapper_http == 0); each of the parallel, surface and preseed
pieces reverted alone turns its own assertion RED.
Fixes #1916
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
cf7eecd7d0 | Merge branch 'main' into fix/jaxrs-relative-path | ||
|
Martin Vogel
|
1160fa3591 |
Merge pull request #2274 from DeusData/distill/1245-c-pointer-return-types
fix(extract): a C-family function's return type keeps its pointer and qualifiers |
||
|
Hubert Tarnacki
|
246e4190c3 |
fix(routes): keep @Path("") unset instead of yielding an empty route path
Accepting relative JAX-RS templates made route_path_from_string_node return
"" for @Path(""), which is non-NULL and so replaced the "/" fallback with an
empty method path. @Path("") is legal JAX-RS and means "the class path
itself"; on main the empty value was rejected and the method fell back to
"/" like a method without @Path. Return NULL for the empty case so that
behaviour is unchanged.
The regression covers both shapes: with a class-level @Path the method
composes to the class path, and without one the route stays "/" (with the
empty string it was dropped entirely, because join_route_paths returns the
NULL prefix for an empty path).
Signed-off-by: Hubert Tarnacki <hubert.tarnacki@gmail.com>
|
||
|
Martin Vogel
|
9b7d63f579 |
Merge pull request #1931 from ilyabrykau-orca/fix/go-cgo-pseudo-import
fix(extract): drop cgo's `import "C"` pseudo-package |
||
|
Martin Vogel
|
28b4093c99 |
Merge pull request #1976 from CaptainMittens/fix/swift-nested-url-constructor
fix(extraction): reach a Swift URL built by a constructor |
||
|
Martin VogelandAndrew Hundt
|
343d7dd5e4 |
fix(extract): a C-family function's return type keeps its pointer and qualifiers
Distilled from #1245 by Andrew Hundt (
|
||
|
Martin Vogel
|
61cfb09b52 | Merge branch 'main' into fix/jaxrs-relative-path | ||
|
Martin Vogel
|
1a3425536b |
fix(graph): quality round after the v0.11.0 probe — deterministic resolution, findability, coverage, names
Nine findings from the 2026-09-16 graph-quality probe over the eleven bench corpora, each with a
test that fails without its fix and a before/after measurement on the real corpus.
Deterministic resolution (O9). Two indexes of one kernel tree differed in 5,821 IMPORTS and 632
CALLS edges. Three order/load dependencies carried it: the `#include` target was the first by-name
hit whose path ends with the include path — a bucket in worker merge order (include/linux/device.h
vs tools/virtio/linux/device.h); the registry broke suffix_match score ties by bucket position; the
LSP budget-share rule measured wall time. Include targets now prefer the File node, the least nested
path, then the lexicographic path; registry ties prefer the least nested QN, then the smaller QN; the
budget share is measured in thread CPU time like the budget itself. Two kernel indexes are now
identical (8,529,723 nodes, 1,974,940 CALLS, zero churn), as are two elasticsearch indexes.
Findability. search_graph's BM25 path honours the `label` filter and ranks an exact-name definition
above test methods that repeat the token (15/15 probe queries).
Coverage. The parse-coverage gap tree subtracts definitions the walker recovered inside an error
region (multi-line only); kernel/sched/core.c reports 476 flagged lines instead of 7,674.
Precision. Config-language callers (Makefile, CMake, YAML) no longer bind unique_name into code
(kernel Makefile -> C CALLS 195 -> 0). JS-family token names ({}, 1, (x: T) => U) are not
definitions; computed, quoted and escaped member keys stay (TypeScript 204 junk -> 0, 1,782 real
names kept). Names are cut at their first line (elasticsearch YAML 8,306 -> 0); a comment inside an
argument list is not an argument, so no Route node is named by a Java block comment; the wildcard
route `/*` is still a route. HTML/Vue/Svelte/Astro embedded member calls stay unbound.
Recall. A Python call through a self- or cls-rooted attribute keeps a specific unique_name member
target (Django +751, elasticsearch +3,288 CALLS, no new callee names).
Robustness. Edge properties never carry a split UTF-8 sequence (the 120-byte expression cut and the
JSON escaper both trim a partial code point). CLI array flags accept a JSON-array literal. Cypher
seeds a single-hop pattern from its selective end (kernel `MATCH (a)-[:CALLS]->(b) WHERE b.name =
'_printk'`: execution limit -> 7.9 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Hubert Tarnacki
|
452485f73e |
fix(routes): accept relative JAX-RS @Path templates
JAX-RS allows @Path values without a leading slash, both on the class
and on the method. The route extractor rejected such literals, so
resources such as `@Path("widgets")` + `@Path("{id}")` produced no Route
nodes and no HANDLES edges at all. Accept relative templates and
normalize the class/method composition so the resulting path is always
rooted.
Existing JAX-RS Java probe tests are updated to use relative templates
so they guard the regression.
Signed-off-by: Hubert Tarnacki <hubert.tarnacki@gmail.com>
|
||
|
XIYBHK
|
fcf451c6a6 |
fix(parsing): recover export-macro-hidden type definitions (#1989)
Collect a bounded set of conventional export macro candidates and inject empty definitions into the existing C/C++ preprocessing second pass without overriding explicit caller definitions.
Conservatively reconcile remapped definitions to recover hidden classes, structs, enums, free functions, and inline methods while suppressing matching base-class phantom callables.
Add focused regression coverage for supported suffixes, ordinary all-caps negative controls, candidate limits, comments, strings, raw strings, overlong names, explicit define priority, and C/C++ extraction.
Rebase onto main
|
||
|
Martin Vogel
|
60c307e8d5 |
Merge pull request #2242 from lorenzozanee/fix/typescript-awaited-generic-calls
fix(typescript): preserve awaited generic call edges |
||
|
lorenzozanee
|
e8a46c16e2 |
fix(typescript): preserve awaited generic call edges
Signed-off-by: lorenzozanee <wyz0707@proton.me> |
||
|
Martin Vogel
|
8c1a9d61c2 |
fix(extract): no clock decides what ends up in the graph
Indexing the same C# corpus twice on the same machine produced 5,794,336 edges
and then 5,794,328. Every other edge type matched exactly, every semantic
checksum matched (447,064 functions, 81,534 tokens, identical vec/idf/toks),
and the difference was ten WRITES edges out of one generated file.
Three places in extraction decided graph content from a clock:
* the unified AST walk stopped on a thread-CPU deadline;
* a file whose parse spent more than half its budget was refused the
per-file and cross-file LSP walks;
* so was a file whose parse plus walk exceeded the budget.
The first is the one that bit. Any file large enough to reach the deadline is
cut at whatever node the clock lands on, and five generated .NET JIT tests
reached it on every run: hugeSimpleExpr1.cs stopped at node 2,660,352 in one
run and 2,574,336 in the next, out of 2,774,794. That file's tail is field
assignments, so the two runs disagreed by exactly ten WRITES edges. The graph
was a function of how fast the machine happened to be running, and nothing said
so: result->walk_truncated was set and no code anywhere read it.
The deadline existed because those files cost 18-48 us per visited node, and
that turns out to be one fixable thing. csharp_callable_value_site() was the
only language branch in call_reference_candidate_site() still climbing with
ts_node_parent, which tree-sitter answers by descending from the ROOT, scanning
children at every level; every other language already climbs on the walk
cursor. On expressions megabytes deep that was ~85% of the entire unified walk
(sampled: stamp_usage_site -> ts_node_parent -> ts_node_child_with_descendant).
It now takes the cursor and the parent the caller already resolved, and
is_direct_argument_value_cursor() is split so a caller that has stepped the
cursor onto the parent can continue from there rather than paying for it twice.
With that fixed, the budgets are not needed to make indexing finish, so they
are gone rather than made deterministic. A cutoff in nodes would at least be
reproducible, but it still answers "what is in this repo?" with "as much as we
felt like reading". CBM_WALK_MAX_NODES reinstates one for an operator who wants
it; the default is no budget.
Removing the two LSP-skip rules was checked against the file and the path they
were added for (a 23 MB single-expression JIT test that once cost 354 s in the
walk and then crashed the cross-file resolve): the whole 12k-file C# corpus
twice with cross-file resolve, that file alone, and synthetic C# nested 10,000
levels deep — no crash, and past roughly that depth tree-sitter stops producing
a usable tree, so a tree deep enough to endanger a recursive walk never reaches
one.
C# corpus, before -> after:
edges 5,794,318 / 5,794,328 / 5,794,336 -> 5,794,873 every run
truncated 5 files, unreported -> 0
time ~115 s -> 97-101 s
Deterministic, about 550 edges more complete (the tails nobody was walking),
and faster than the build that was cutting them off. All ten bench corpora now
return identical nodes and edges across two runs.
A truncated walk and a skipped LSP walk are now reported in the coverage report
with how far the walk got ("1673216/4133946 nodes walked"), like a partial
parse. A file the walk abandoned halfway used to be reported as fully indexed,
which is a graph quietly lying about its coverage.
Tests: extract_csharp_argument_values_use_the_walk_cursor asserts no occurrence
falls back to the root-descending parent lookup — 128 fallbacks without the fix
and 0 with it. Its fixture parenthesises the argument on purpose: a bare
Sink(Target) needs no climb at all, so it cannot tell the two walks apart and
passed either way while the bug was in place.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
959f4de50c |
perf: cut the allocation churn and repeated work the sanitizer found
Every item here was measured on a real corpus before and after, not
reasoned about. Go corpus, event lane, first run -> now: 245.5 M
allocations -> 134.8 M, 190 GB cumulative -> 29.3 GB, 158 GB never
written -> 9.2 GB, 132 M churn pairs -> 35.7 M, 93 GB of allocator byte
work -> 0.84 GB. Wall clock: Go 40.7 s -> 19.3 s, linux kernel 239.6 s ->
194.6 s, and all ten LSP languages faster with none slower.
Graph output is unchanged, proven rather than assumed: per-class multiset
digests over the full kernel index (8,563,471 nodes identical, 18/20 node
classes and 18/24 edge classes byte-identical, the rest differing only in
the classes a same-binary re-run also perturbs through worker merge order).
The changes, each with its own reason:
arena an exact arena that is appended to later restarts growth at
8 KB instead of the 64 KB default — the cross-file pass adds a
few resolved calls to a compacted result, so the default block
was ~60 KB untouched per result, 0.5 GB of the Go worker peak.
cbm_arena_init_lazy defers the first block for arenas that are
usually never used at all.
scratch per-thread reuse instead of allocate-and-free per unit of
work: result_compact (8 MB kept), the cross-file pass (4 MB),
extract_defs' walk stack, and the sqlite index cells (a 1 MB
arena replacing 12.1 M malloc/free pairs on Go). Walk frames
moving into scratch is why pass_k8s now opens the keep window
explicitly: without it every YAML file opened a 512 KB scratch
on the main thread, +3 GB.
sqlite index cells are written in place — building the record and
copying it into the cell was a second allocation per entry,
16 M record buffers of pure churn on Go.
registry is_test_qn and common_prefix_len in one pass, per-bucket test
flags computed once instead of per candidate comparison, and
the reachability keys arena allocated lazily.
minhash the unique-trigram set is sized from the trigram count and
skips repeats, and LSH queries reuse one seen-set per worker
instead of allocating per query.
parallel lsp_surface rows and the k8s manifest prep now run across
workers (graph writes stay sequential and ordered), the file
node is hoisted out of register_and_link_def, and configlink
prepares dependency imports once with a length prefilter.
go_lsp one sealed stdlib registry shared by all workers, published
by CAS, instead of one per worker.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
7b77fd4834 |
feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.
WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)
The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.
WHAT CHANGED
Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.
Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.
Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.
The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.
The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.
Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.
Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).
The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.
The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.
Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.
SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).
Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.
Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.
The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.
Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.
PROOF
Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
jitter (gRPC Route nodes: two identical runs differ by 21).
Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
surface rows, nodes/edges inside the Route jitter.
Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
inside the +-400 run-to-run band the kernel shows between identical
runs.
Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
outside what spill can move). Nodes identical; edges 15,773,598.
Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
every phase mark under 15,000 MB by every metric, but the process
RSS high-water mark reached 16.9 GB inside the semantic pass (class
peak 4.9 GB against 0.8 GB live).
Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
stays under 15,000 MB through the whole semantic pass (14,018 ->
14,381 MB across its sub-phases, 6 batches of 125,644 functions);
every phase mark is under budget by every metric (max footprint
12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
15.65 GB and is set in extraction, at the moment the gate latches
(15,002 MB charged) and before the first sweep lands -- a 4%
overshoot of in-flight work, no longer the semantic pass. Nodes
identical, edges 15,772,913, zero foreign blocks.
Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
enters at 14,062 MB (near_budget); the charged footprint never
exceeds 11.8 GB at any mark; the process RSS high-water mark is
15.1 GB in extraction and 15.3 GB during the final write-out --
1-2% over, the in-flight window and the dump transient. Nodes
identical, edges 15,772,543, zero foreign blocks.
Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
peak 998 MB RSS, graph identical to the baseline.
BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)
corpus wall s peak RSS GB nodes / edges / CALLS
perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3
php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8%
rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369
c 5.9 -> 5.7 0.16 -> 0.19 identical
kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1%
django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4%
go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2%
java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6%
csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6%
typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130
kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1%
No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (
|
||
|
Joshua Richter
|
eb80a4ecff |
fix(extraction): reach a Swift URL built by a constructor
Swift has no URL literal, so almost no real code passes a bare string to a request. It writes URL(string: "https://…")! instead, and the literal then sits two levels below the argument list: past the trailing "!", which the grammar models as a postfix_expression, and inside the constructor's own value_arguments. extract_url_or_topic_arg saw only the outer node and gave up, so the URL never reached the service-pattern table and no Route node formed. That is the shape issue #1892 reported from a real project — reaching a bare string argument was only the layer underneath it. swift_unwrap_url_constructor() steps past both wrappers. It unwraps only URL, URLComponents and URLRequest, so any other constructor keeps its own meaning and the outer call does not borrow the inner call's string. A non-literal argument such as URL(string: base + path) falls through to the ordinary handling unchanged. The value_argument unwrap added for the bare-string case is now swift_argument_value(), because the nested argument list needs the same step and the code was identical. Refs #1892 Signed-off-by: Joshua Richter <jrichter5781@gmail.com> |
||
|
Martin Vogel
|
4f82a1e56f | Merge branch 'main' into fix/svelte-html-astro-embedded-structure | ||
|
Martin Vogel
|
dc703dad4a | Merge branch 'main' into upstream-pr/elixir-call-arguments | ||
|
Brian McNaboe
|
24f54c0eb6 |
fix(extract): extract structure from Svelte/HTML/Astro inline scripts
The Vue change (#1852) built the shared embedded-script seam — the included-ranges parse over the original buffer and the lang=/src= resolver — but gated structural extraction on the host being Vue. A Svelte component, an HTML page or an Astro file therefore still yielded only File + Module nodes and IMPORTS edges from its <script> blocks: no Function definitions, no CALLS, and a false dead-code verdict for anything used only from a component. Widen the seam to every host that declares an embedded spec: - Structural extraction (definitions + calls) runs for every embedded block; the per-host boolean is gone. - The Vue attribute resolver becomes script_embedded_language() and runs for all four markup hosts. It starts from the spec row's language instead of hard-coding JavaScript, and learns HTML's type= attribute: absent or empty, module, text/javascript and application/javascript pass; application/json, importmap, text/x-template and the like bail, so data blocks cannot leak garbage definitions into the graph. src= and unsupported lang= values bail as before. CFML's <cfscript> carries no attributes and keeps its spec-fixed language. - Astro's frontmatter fence and <script> bodies default to TypeScript: Astro type-checks both as TS with no lang= attribute to say so, and the fence cannot carry one. The TypeScript grammar parses untyped JavaScript unchanged. Tests: the #1852 control that pinned Svelte/HTML/Astro at zero functions and zero calls now asserts the real symbols; stay-at-zero controls cover type="application/json" and <script src> in every host; both Svelte blocks (context="module", and Svelte 5's bare module) contribute in host-file coordinates; HTML's accepted type= forms and Astro's TS frontmatter are covered; an end-to-end CALLS edge inside a .svelte file is asserted. The Svelte and Astro label goldens gain the Variable their fixtures always declared. Verified on a 213-component Vue corpus: node counts per label and per host language, edge counts and trace_path output are identical before and after. Fixes #1807 Signed-off-by: Brian McNaboe <bmcnaboe@gmail.com> |
||
|
Martin Vogel
|
380b1c5af1 | Merge branch 'main' into feat/cshtml-razor-pages | ||
|
Martin Vogel
|
5802ccdd77 |
Merge pull request #1896 from CaptainMittens/fix/swift-call-arguments
fix(extraction): reach Swift call arguments through call_suffix |
||
|
Martin Vogel
|
1770d68a2c |
feat(extract): discover Razor Pages / MVC views (.cshtml) and route their @page
Follow-up to #1824, which mapped Blazor `.razor` to the C# host and extracted `@page` routes from it. `.cshtml` is the same Razor syntax with the same C# host and was equally unmapped: an ASP.NET Core MVC or Razor Pages application produced no nodes at all for its entire view tree. `@page` matters more here than it does for components. On a Blazor component the directive is an optional annotation; in Razor Pages it is the directive that *defines* a page, so the routable surface of a Razor Pages app lives exclusively in this file type. That is why the extension mapping alone is not enough. `cbm_path_is_razor()` hardcoded a single `.razor` suffix, so a `.cshtml` file would have been discovered and parsed but skipped by the directive scan -- present in the graph, with none of its routes. Widened to a suffix table covering both. Scope is deliberately the Razor family and nothing else: - `.vbhtml` is excluded because there is no VB.NET language in this repo, so there is no host to map it to. - `.aspx` / `.ascx` are excluded because Web Forms is a different templating syntax (`<%@ %>`, `runat="server"`) with no `@page` directive, so neither the C# error recovery nor the directive scan applies. Best-effort on the same terms as `.razor`: the C# grammar recovers the `@{ }` and `@functions` blocks, and the surrounding markup lands in ERROR regions reported through parse_partial. Tests: the extension mapping; a Razor Page asserting the extracted route path and method; and a negative case for a layout with no `@page`, since most `.cshtml` files are layouts and partials and a spurious Route node per view would be worse than the missing ones this fixes. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
cc3e2638da |
Merge pull request #1912 from DeusData/feat/python-bare-local-binding
fix(python): suppress bare calls shadowed by an enclosing parameter |
||
|
Martin Vogel
|
db2f104a1f |
Merge pull request #2013 from shafty023/fix/1997-extract-scratch-arena
fix(extract): AST traversal stacks no longer outlive the file they walk |
||
|
Joshua Richter
|
c4e902849f |
fix(extraction): reach Swift call arguments through call_suffix
handle_calls reads a call's arguments through one tree-sitter field lookup, ts_node_child_by_field_name(node, "arguments"). The vendored Swift grammar declares no "arguments" field at all -- its own ts_field_names[] table has zero occurrences, against one in Go and two in TypeScript. Swift models a call as a target expression plus a call_suffix, and the arguments hang off the suffix as value_arguments. So args was null for every Swift call ever parsed, first_string_arg was never populated, and no Swift HTTP call could raise an HTTP_CALLS edge or a Route node. Alamofire, Moya and URLSession have been in the service-pattern table the whole time and match the callee text correctly; the URL simply never arrived. Three changes, all in extract_calls.c: - swift_call_args() reaches the argument list through call_suffix, in the same shape as the existing objectscript_call_args() fallback and used from the same place. A trailing closure has no value_arguments, so it returns a null node and that call behaves as before. - extract_url_or_topic_arg() unwraps Swift's per-argument value_argument node, stepping past a leading value_argument_label. Without it, dataTask(with: "/api/v1/widgets") yields the label "with" rather than the path. PHP and C# already had the same unwrap for their own "argument" node. - is_string_like() gains line_string_literal, which is what Swift calls an ordinary "..." literal. The list already held raw_string_literal, so only Swift's common case was missing. This is the layer underneath #1892 rather than the whole of it. A literal URL argument now arrives; a literal nested inside a constructor, as in URLSession.shared.data(from: URL(string: "...")!), still does not, because extract_positional_url reads a literal, a template string, a concatenation or a named constant and that shape is none of them. Reproduce-first: all three tests fail without the fix, two on a null first_string_arg and one on HTTP_CALLS being 0. Fixes #1892 Signed-off-by: Joshua Richter <jrichter5781@gmail.com> |
||
|
Martin Vogel
|
01bf5f23e3 |
Merge pull request #2009 from CaptainMittens/fix/route-handler-last-argument
fix(routes): take a route's handler from the last argument, not the first |
||
|
Martin Vogel
|
e6a34fcea0 |
Merge pull request #1824 from The1nk/feat/blazor-razor-routes
feat(extract): discover Blazor .razor components and emit Route nodes for @page |
||
|
Daniel Ochoa
|
40f2722d61 |
fix(extract): AST traversal stacks no longer outlive the file they walk
Fixes #2010. Refs #1997. Indexing a 14k-file TypeScript repository holds 13.3 GB resident at post_extract on Linux aarch64, and about 3.1 GB of it is AST traversal scratch that no CBMFileResult points at. ts_nstack_init cut its stacks from ctx->arena, which is result->arena. In the parallel path that arena is stored into ec->result_cache[file_idx] (pass_parallel.c:963) and freed only after parallel_resolve and the infra passes (pipeline.c:1324), so a structure written for a one-file lifetime is in fact held for the whole index, once per file, concurrently. The header claimed otherwise and was wrong: "freed when the arena is destroyed at end of file extraction". The largest instance is channels. CHAN_STACK_CAP is 4096 and sizeof(TSNode) is 32, so scan_string_consts_js (extract_channels.c:104) and extract_channels_js (:376) take 128 KB each, per JavaScript or TypeScript file. Across 12,673 such files that is 3,168 MB, and it produced 23 Channel nodes out of 164,684. CBMExtractCtx gains a scratch arena, created and destroyed by the cbm_extract_file_ex call that builds the context. Nothing else changes about where anything lives: ctx->arena still owns every string a CBMFileResult points at, and ctx->scratch owns only traversal stacks, which nothing points at. Nothing a CBMFileResult points at moved. A context built without a scratch, and a file whose scratch arena fails to allocate, both fall back to ctx->arena, which is exactly the behaviour that shipped before this change, so no path can be handed a NULL arena and silently lose nodes. The signature change is what makes the mistake unrepresentable. ts_nstack_init takes the context rather than an arena, so handing it ctx->arena is an incompatible pointer type; push and push_children read the arena the stack recorded, so a stale argument is an arity error. Both fail under -Werror rather than silently retaining memory. The 55 init sites are NOT one uniform sed. 29 pass ctx->arena and 26 pass a local alias, and that alias is shared with result allocations: in parse_zig_imports (extract_imports.c:1093) the same `a` feeds cbm_node_text, strip_quotes, path_last and cbm_imports_push (extract_imports.c:1101, 1106, 1108, 1109). Rebinding it would have moved import names and module paths into memory that dies with the call while the result kept pointing at them. Each site was edited on its own and no alias was rebound. Re-derive the set with: grep -rn 'ts_nstack_init' internal/cbm/extract_*.c grep -rn 'ts_nstack_push' internal/cbm/extract_*.c parse_lisp_imports used its alias only for the stack, so the alias goes too. push_nested_class_nodes and push_class_body_children took an arena purely to reach the stack, so they take the context now; the one call site passes ctx. Two measurements, from two different builds, and they are not the same number. A macOS census build, which instruments every cbm_arena_alloc and sums the blocks behind every live CBMFileResult arena, reports the retained per-file arena set dropping from 11,226 MB mapped to 5,680 MB, and blocks from 48,045 to 34,186. Those are mapped bytes, not resident: that build logs mem.allocator.bound_populations_only, so untouched block tails never commit. The resident figures come from Linux aarch64, gcc 13.3, static build, where the build logs mem.allocator.owned classes=all and mapped equals resident. Five runs per side on the full 13,999-file tree: mem.collect post_extract rss_mb 13293 -> 6752 (-49.2%), mem.phase peak_mb 14840 -> 8565 (-42.3%). nodes 164684, parse_partial 280 and registry entries/defines/imports 787954/804999/100027 identical on all ten runs; edges 835869-835874 against 835867-835881, overlapping ranges, so run to run noise. Wall clock means 35651 ms against 30502 ms, but the ranges overlap and the host had other load, so the only supportable claim is no regression. ObjectScript Studio Export on a small fixture: nodes 18, edges 25, both sides. Scratch block size is 512 KB, chosen from measurement. arena_grow fires on one file in 12,000 at both 512 KB and 1 MB, and constantly at 256 KB, where the two channel stacks alone are exactly 262144 bytes. 512 KB therefore costs half the resident block per worker for the same growth behaviour, and stays under MI_LARGE_MAX_OBJ_SIZE in the vendored mimalloc, which is MI_LARGE_PAGE_SIZE/8, exactly 524288 bytes (types.h:426, with MI_ENABLE_LARGE_PAGES defaulting to 1 at :115 and not overridden by this build), so a 512 KB block is still bin-allocated from a 4 MiB large page. Growth is not free at that size for the same reason: arena_grow doubles to 1 MiB, above that bound and therefore a singleton OS allocation. One file in twelve thousand pays it, which is why it is accepted. The largest single file's scratch high water mark on this corpus is 33.5 MB, now transient rather than retained. Tests. Nothing in the suite asserted on result->arena size, block count or total_alloc after cbm_extract_file, which is why this went unnoticed for the life of extract_node_stack.h. traversal_stack_not_in_result_arena_issue2010 pins the byte budget: extracting "export const x = 1;" as TypeScript charged 365984 bytes to the result arena before and charges 87456 after, a difference of 278528, exactly the two 4096-entry channel walks plus the 512-entry ES import walk at 32 bytes per TSNode. Of the 87456 that remain, 7680 is the defs item array at GROW_ARRAY's starting capacity of 32 times sizeof(CBMDefinition) 240 and the other 79776 is everything else the extraction interns, none of it traversal scratch; the 128 KB bound is therefore above the new figure with room and a factor of four below the old one. extract_traversal_stacks_come_from_ctx_scratch_issue2010 in test_mem.c pins the lifetime, since a byte budget alone would also be satisfied by shrinking CHAN_STACK_CAP: it builds the extraction context directly over two arenas it owns, runs cbm_extract_channels, and asserts the scratch took at least the 262144 bytes of the two walks while the result arena did not. extract_c_macro_hidden_call_survives_preprocessed_pass_issue2010 is not a scratch test. pp_ctx carries ctx->scratch so every context in the file is uniform, but nothing reads it there: pp_ctx reaches only cbm_extract_unified and cbm_run_c_lsp, and neither extract_unified.c nor anything under internal/cbm/lsp/ includes extract_node_stack.h, so no traversal stack is built on that path today. The test guards the macro-expansion path itself, which had no assertion on a call that exists only after expansion. ObjectScript composite extraction is already covered by pipeline_objectscript_export_preserves_calls_sequential_parallel and iris_export_xml_multi_class, so no new test is added there; a per-call arena is correct on that path by construction, since each call owns and destroys its own. Local: scripts/test.sh, 7558 passed, 0 failed, 8 skipped, 139 suites. Alternatives considered Thread-local scratch arena, reset per file, with a depth guard for nesting. Rejected on three counts. This codebase never uses thread-local storage for owned bulk memory; every existing _Thread_local here is a small cache, a guard or a log buffer, and every per-worker resource is a field in a context struct. It would have needed cbm_arena_reset, which has no production caller and a documented subtlety with grown block sizes (arena.c:213-217). And extraction threads are created per cbm_parallel_for call and joined (worker_pool.c:71,88) with no thread-exit hook in this project's own code, so the arena would have leaked once per thread per dispatch and turned the leak lane red. Save and restore mark and release on the arena. Rejected because a per-call arena already makes nesting correct by construction, and a mark type plus two functions would have to be added to both arena headers. Per-walk arenas, one per traversal rather than one per file. Rejected on cost: 55 sites would each need an explicit span, for the same bound. Right-sizing the initial capacities alone. CHAN_STACK_CAP at 256 rather than 4096 would cut those two sites from 3,168 MB to about 198 MB, which is real, but it tunes the constant while the lifetime defect remains at all 55 sites, and growth is not free: 256 doubling to 4096 abandons 122,880 bytes, and one file in this corpus took 33.5 MB of stacks. The cap and the scratch block size are coupled, which is a further reason to change one at a time. Worth doing separately. Routing to a scratch inside extract_node_stack.h while keeping the arena parameters. Rejected because a parameter that is ignored lies about the API, and the alias hazard above would stay a review obligation instead of a compile error. Freeing the per-file result arena earlier. Rejected: the results are read by parallel_resolve and the infra passes. That is the retention half #1925 addresses, and this change is independent of it. Signed-off-by: Daniel Ochoa <daniel.ochoa@closedloop.ai> |
||
|
Martin Vogel
|
97517a4652 |
fix(python): carry bare-call parameter bindings in the walk
Replaces the cursor ascent from
|
||
|
Ilya Brykau
|
5cb90f64ad |
fix(extract): drop cgo's import "C" pseudo-package
`import "C"` is not an import: "C" is reserved by go/build and no package with that path can exist — it only tells the toolchain to compile the preceding comment as C. cbm emitted it as an ordinary import, so the import resolver fell through to its symbol-name fallback (cbm_pipeline_resolve_import_node Strategy 3, pass_pkgmap.c), matched the literal name "C" against every project node called C, and picked the lexicographically smallest one. Measured on a real Go+C repo: all 27 files carrying `import "C"` pointed their IMPORTS edge at the same unrelated `C` member of a test helper — 27 false edges, and 27 files whose import map named a package that does not exist. Skip the spec in parse_go_import_spec. Node counts are unchanged and the file's real imports are untouched; only the false edge disappears (a 3-file fixture goes 22 edges -> 21, 14 nodes -> 14; the real repo goes 27 -> 0 such edges). Reproduce-first: go_cgo_pseudo_import_dropped is RED without the extractor change (the "C" import is still in imports[]) and GREEN with it, while still asserting the file's real `fmt` import survives. Signed-off-by: Ilya Brykau <ilya.brykau@orca.security> |
||
|
Joshua Richter
|
592894a438 |
fix(routes): take a route's handler from the last argument, not the first
Express, Fastify, gin and Laravel all put middleware between the route
path and the handler, and the handler comes last:
routerGet("/users", requireAuth, rateLimit, listUsers);
extract_handler_arg returned the FIRST argument that looked like a
function reference, so requireAuth won and the HANDLES edge pointed at
the middleware instead of listUsers. That is a wrong edge, not a
missing one, and it misleads anyone tracing a request through the
graph.
The scan bound hid a second fault. An arrow function matches none of
the accepted node kinds, so three inline middlewares pushed the real
handler past MAX_HANDLER_SCAN and no handler came back at all.
The loop now examines every argument and keeps the last eligible one.
Nothing that follows a handler matches the accepted kinds either, since
an options argument is an object node, so the last match is the
handler. HANDLER_START_IDX stays, because argument 0 really is the
path. MAX_HANDLER_SCAN had no other use and is gone.
A two-argument route has one eligible argument, so first and last agree
and the existing Express, Fastify, gin and Laravel route tests are
unchanged.
Both tests fail without the change, the first reporting "requireAuth"
where "listUsers" belongs and the second reporting no handler at all.
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
|