mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-11-14 16:52:09 +08:00
063708ba535653659560e7e75bc511b9dc485068
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Colby MchenryandClaude Opus 5.5
|
3a4cd3bf65 |
fix(ruby): calls to !/? methods and calls on what .new made link their method (#3049)
Two kinds of Ruby call linked nothing. A call to a method whose name ends
in `!` or `?` on a receiver (`writer.flush!`, `writer.ready?` after
`writer = CacheWriter.new`, `CacheWriter.create!(1)`) failed the method-call
pattern, whose method part took no `!` or `?`. A method called straight on
a new object (`CacheWriter.new(key).write_cache`, `Billing::Ledger.new.charge!`)
was recorded by both extractors with the receiver's raw text, arguments and
blocks included, which no strategy matched.
Both extractors now record a call on `Const.new(...)`, `A::B.new` or
`::Foo.new` as `new Const().m`, the constructed-receiver shape TS/JS, Java
and C# already use; the wasm extractor and the kernel's ruby.rs emit the
same refs. ruby-constant-paths resolves that shape before the name
pre-filter: the class by Ruby's constant lookup from the namespaces around
the call, the method as the class's own instance method or the first its
ancestry declares, and nothing when either is missing, so rack-protection's
`Request.new(env)` (rack's Rack::Request) links nothing. The Ruby dot pattern
in matchMethodCall takes a trailing `!` or `?`; the receiver-word scoring
among several project methods of the name stays off for such names on a
value receiver, where it was wrong on 55 of 74 new edges (`options.key?`
on a Hash -> InheritableOptions#key?). rubyAncestry no longer reads a
module a call returns (`include RouteSet.new.tap {...}.url_helpers`) as a
mixin of the constant the call starts at.
Validation, kernel engine, main
|
||
|
Colby MchenryandClaude Opus 5.5
|
16a8e77c46 |
fix(ruby): a call reaches the class method or the instance method its receiver calls (#2834)
A Ruby class method (`def self.build`, `def Widget.build`, or a def inside
`class << self`) was indexed exactly like an instance method: is_static 0,
under the same qualified name as an instance method of that name
(Widget::build twice). So every call took whichever of the two the class
declared first. Widget.build could reach `def build`, and widget.build on an
instance could reach `def self.build`. In the repro for issue 2503,
`card = Billing::ChargeCard.new(...); card.call` linked to `def self.call`
rather than `def call`.
Fix:
- Extraction, both engines: the Ruby isStatic hook (isRubyClassMethod in
languages/ruby.ts, is_class_method in codegraph-kernel/src/ruby.rs) marks a
singleton_method, or a method whose parent past one body_statement is a
singleton_class, as static. Every other def is marked not static, the same
way on both, so the graph only changes for class methods.
- Resolution, Ruby only, where the call says which kind it is:
rubyMethodsOfKindFirst orders a call's candidates with the wanted kind
first. A call on a constant (Strategy 1, its inherited walk, and the
namespaced path in ruby-constant-paths.ts) wants the class method. A
receiver typed by `x = Widget.new` (resolveMethodOnType) wants the
instance method. A bare call splits only a real pair, one qualified name
among its candidates as both kinds, keeping the caller's kind: a block
may run on another object, so nothing else moves. A receiver known only
by its name (Strategies 2 and 3) says nothing of its kind and is left as
it was: rubocop's `cop.cop_name` runs over cop classes. With only one
kind present, every path picks what it did before.
Validation:
- New __tests__/ruby-singleton-methods.test.ts (6 cases, each on the
kernel and on wasm) and a parity case in kernel-ruby-parity.test.ts. On
main 9 of the 18 tests in the two files fail, every one that checks the
new behaviour; the guards and the old parity cases pass. All pass with
the fix.
- A/B against main
|
||
|
Colby MchenryandClaude Opus 5.5
|
dcb985f1d2 |
fix(ruby): a call on a namespaced class links its caller to the class and its method (#2827)
A Ruby call on a constant written with its namespace, like Billing::ChargeCard.call(...) or ::Foo.call, recorded no edge. Both extractors (the kernel's ruby.rs and the wasm one) emitted the receiver's references ref only for a bare `constant` receiver, so a scope_resolution receiver got none. The call ref itself, Billing::ChargeCard.call, matched no resolver strategy either: the method-call patterns reject a `::` in the receiver. A class used only through such calls showed no callers and no dependents. Fix: - Both engines treat a constant path (a constant, or a scope_resolution of constants, a leading `::` included) as a constant receiver and emit the references ref under the receiver as written. A path through a value (obj::Thing, self.class::LIMIT) still emits none, and .new stays an instantiation. - A new resolver step, src/resolution/ruby-constant-paths.ts, resolves a Ruby references ref written as such a path, and a calls ref on one, the way Ruby looks the constant up: the class or module around the reference, then each namespace outside it, then the top level; a `::`-rooted path at the top level only. A call reaches the class's or module's own method, else the first one its ancestry declares (rubyAncestry, now exported). The step runs before the pre-filter, which dropped `::`-rooted names, and it is exclusive: a path the project doesn't declare (ActiveRecord::Base) or a class without the method links nothing, never a namesake by its last name, and production code never reaches a class only a test declares. Validation: - New __tests__/ruby-namespaced-receiver.test.ts (8 cases, each on the kernel and on wasm, plus a sync case) and a parity case in kernel-ruby-parity.test.ts. On main 16 of the 22 tests in the two files fail (the rest are guards and the old parity cases); all pass with the fix. Reverting only the resolver step, or only the wasm extractor, turns 13 and 10 of them red. - A/B against main (kernel path), every edge triaged: sinatra +23 edges, rubocop +168, mastodon's Ruby tree (4,363 files) +676; 0 removed, nodes identical. 866 of 867 are right: references by the path as written or under an enclosing namespace (each such case checked against the call site's own nesting), calls to a class method, a module_function method, or an inherited class method. 1 is wrong: Paperclip::Transcoder.make reaches the instance method `make`, since a singleton method and an instance method share one qualified name today; a bare Transcoder.make does the same on main. Controls express and flask are byte-identical. - kernel-parity.mjs --lang ruby: sinatra 147/147, rubocop 1799/1801 (the same 2 deferred on main), mastodon 3333/3333, 0 diffs. The fix arm's kernel and wasm graphs are byte-identical on sinatra and rubocop. - Index time, median of 3 interleaved runs: +0.4% to +1.7%, the same as the controls (+0.8%, +1.3%). Fixes #2503 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Opus 5.5
|
86708bf519 |
fix(extraction): preserve symbols with colliding same-line IDs (#1349) (#2030)
Same-kind, same-name declarations on one line shared an ID and silently overwrote each other in SQLite. Use per-extraction collision tracking to append a UTF-16 column only to later distinct declarations. Apply the same rule across wasm, Rust language walkers, Liquid, and CFML while preserving file IDs and edge endpoints. Add persistence, call-edge, identity-vector, and Unicode parity regressions. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby Mchenry
|
838006c947 |
fix(kernel): guard the native walkers against stack overflow and defer deep files to wasm (#1581) (#1600)
Fixes #1581.
## What was wrong
`codegraph init` / `codegraph index` died with `Segmentation fault` — the whole CLI
process, not a parse worker — on a C/C++ file with very deep brace nesting (llvm's
`clang/test/Parser/parser_overflow.c`, 16,384 nested `{`). The reporter's diagnosis is
exactly right: tree-sitter's parser is iterative, so the file parses fine, and then the
native kernel's **recursive walker** (`visit_node` → `visit_for_calls_and_structure` → …,
one frame per AST level) overflowed the thread's stack. A native overflow can't be caught
the way a wasm abort can, and a parse worker is a thread of the `codegraph` process, so
the SIGSEGV took the entire indexer down — no message, no per-file fallback, no partial
index.
Two things made "just give the worker a bigger stack" the wrong fix:
- it only moves the cliff — reproduced here: the reporter's 16,384-deep file kills a
default 4 MiB worker (rc=132 on macOS / 139 on Linux), and a 100k-deep file kills the
8 MiB **main** thread too;
- the walkers are shared by every kernel-routed language (20 of them), and each has
several recursion points with different frame sizes, so no single stack size is a
provable bound.
Meanwhile the wasm path already handles this shape gracefully: its JS walker catches its
own `RangeError` per file and stores a partial result with a `parse_error`. The kernel
just needed a way to get there instead of dying.
## What this does
**The kernel guards its own recursion against the calling thread's real stack bounds and
defers a too-deep file to wasm** — the same `defer:` routing signal it already uses for
files with parse errors, which `src/extraction/kernel/index.ts` treats as "take the wasm
path for this file", silently.
- `codegraph-kernel/src/stack.rs`: per-thread stack bounds from the OS, computed once per
thread and cached — glibc/musl `pthread_getattr_np` + `pthread_attr_getstack`, macOS
`pthread_get_stackaddr_np` + `pthread_get_stacksize_np`, Win32
`GetCurrentThreadStackLimits` (a hand-declared `kernel32` extern; no `windows-sys`).
`exhausted()` is one thread-local load and one compare: true once the stack pointer is
within a 256 KiB red zone of the limit, and it latches a flag. Where the OS can't report
bounds it falls back to a fixed 1 MiB descent budget measured from the entry stack
pointer — safe on anything from Node's 4 MiB worker default up. So the guard is exact on
the 4 MiB worker, the 8 MiB main thread, and any `resourceLimits.stackSizeMb` alike.
- `stack_guard!()` (defined in `lib.rs`) is the first statement of every recursive walker
function — all **150** self-recursive or on-cycle functions across the 15 walker modules,
found by script (every cycle in the call graph, not just direct self-calls). It returns
`Default::default()` (`()`, `false`, `None`, `""`) so an exhausted walk simply stops
descending; a hook returning `false` sends its caller down the generic child walk, whose
own guard returns at once.
- `extract_file` runs the whole walk under `stack::run_guarded`: if the flag is set
afterwards the (truncated) result is discarded and replaced by
`defer: nesting too deep for the native walker — wasm recovery handles it`.
- `parse-pool.ts`: a comment at `new Worker(scriptPath)` records why there is deliberately
no `resourceLimits.stackSizeMb` bump.
- No new crates beyond `libc` as a direct unix dependency (already in `Cargo.lock`
transitively). No wire/ABI change.
Net effect for the reporter's repo: `deep.c` goes to the wasm path, lands as
`function foo` plus a recorded parse warning, and the other 31,607 files index normally.
`CODEGRAPH_KERNEL=0` and the `exclude` workaround are no longer needed.
## Tests
**Rust unit tests** (`cargo test`, 21 passed — 7 new in `stack.rs`): the walkers for
C, C++, Rust, TypeScript and Python are driven on a **1 MiB** thread (a quarter of Node's
worker default) with 30k-deep nesting and must return `defer:` instead of crashing;
shallow files are untouched; the latch resets between runs; the OS bounds are sane on the
main thread and describe a small thread's own stack.
**`__tests__/kernel-deep-nesting.test.ts`** (new, 8 tests — skips without a staged `.node`,
fails under `CODEGRAPH_KERNEL_EXPECT=1` if the kernel is missing, like the other kernel
suites):
- every default-routed language (all 20) survives a 60k-deep expression on the main thread
— clean result or the wasm fallback's partial result, never a crash;
- the reporter's exact 16,384-brace C file is indexed (partial) on the main thread;
- 200-deep expressions in every language still take the kernel path clean (the guard never
trips on normal code);
- inside a **default-sized 4 MiB `worker_threads` Worker** through `dist/`: the reporter's
`deep.c` and a 60k-deep expression in every language come back `deferred` with exit 0,
and a normal file still extracts natively;
- end-to-end through the built CLI: `codegraph init` on a repo holding `deep.c` + `ok.c`
exits 0 and records both files, with both functions.
**Existing kernel suites**: all 15 (`kernel-*-parity`, `kernel-scaffold`,
`kernel-retry-materialize`, `kernel-grammar-parity`) pass unchanged, 147 tests — the guard
never fires on the parity fixtures.
**Reporter's probes** (`one.js` from the issue, default 4 MiB worker, this build):
`deep.c` → `deferred`, exitCode=0 (was rc=132/139); `deep100k.c` → `deferred`, exitCode=0.
Main thread: `deep.c` / `deep100k.c` → wasm partial with
`Parse error: Maximum call stack size exceeded`; a 6,000-term binary expression and a
3,000-branch `else if` chain stay on the kernel path with clean results.
**Perf** (same `dist/`, only the `.node` swapped via `CODEGRAPH_KERNEL_PATH`; interleaved
main/new ×3, `codegraph init`, macOS arm64):
| repo | main (median) | guarded (median) | nodes / edges |
|---|---|---|---|
| express (141 files) | 0.60 s (0.58–0.65) | 0.61 s (0.58–0.61) | 1,084 / identical |
| redis (786 C/H files) | 4.44 s (4.39–4.66) | 4.49 s (4.41–4.70) | 19,942 / 76,446 identical |
Within run-to-run noise, as expected for one TLS load + compare per recursion entry.
**Linux (Docker, `node:22-bookworm`, kernel built in-container, `docker run --rm --init`)** —
the reporter's platform and the glibc `pthread_getattr_np` bounds path:
```
=== platform ===
Linux efe3cc86947b 6.12.54-linuxkit #1 SMP Tue Nov 4 21:21:47 UTC 2025 aarch64 GNU/Linux
v22.22.3
-rwxr-xr-x 1 root root 35332288 Aug 22 18:02 codegraph-kernel/prebuilds/linux-arm64/codegraph-kernel.node
=== reporter repro (issue #1581): 16,384-brace deep.c, codegraph init ===
│
└ Done
init exit code: 0
file: deep.c
file: deep100k.c
file: ok.c
function: add
function: bar
function: foo
=== worker probe: kernel raw extract in a default 4 MiB worker ===
deep.c: deferred
deep.c: worker exitCode=0
deep100k.c: deferred
deep100k.c: worker exitCode=0
ok.c: kernel nodes=2
ok.c: worker exitCode=0
=== cargo test stack:: (glibc pthread_getattr_np bounds path) ===
test stack::tests::os_bounds_are_sane_on_this_platform ... ok
test stack::tests::small_stack_reports_its_own_bounds ... ok
test stack::tests::normal_files_are_untouched_by_the_guard ... ok
test stack::tests::deep_braces_c_defer_instead_of_crashing ... ok
test stack::tests::latch_resets_between_runs ... ok
test stack::tests::deep_parens_cpp_rust_ts_python_defer_instead_of_crashing ... ok
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 15 filtered out; finished in 0.23s
=== vitest: kernel-deep-nesting + kernel-scaffold ===
✓ __tests__/kernel-scaffold.test.ts (10 tests) 30ms
✓ __tests__/kernel-deep-nesting.test.ts (8 tests) 36989ms
Test Files 2 passed (2)
Tests 18 passed (18)
```
(The pre-fix crash was reproduced on macOS — rc=132 in a default worker, rc=139 on the main thread at 100k depth — not re-run inside this container; the reporter's Linux x86_64 trace is the SIGSEGV form of the same overflow.)
**Windows (Parallels ARM64 VM, MSVC 14.44, `cargo 1.97`, kernel built on the VM,
`GetCurrentThreadStackLimits` path)**:
```
head:
|
||
|
Colby MchenryandClaude Fable 5
|
1909931238 |
feat(kernel): R7b Ruby walker — ruby module, tree-sitter-ruby 0.23.1 bump, ref-flag wire slot, ruby default-routed (#1379)
Third R7b port, checklist-first recipe (docs/design/ruby-kernel-port-checklist.md). Grammar bump first, validated standalone (the rust pattern): tree-sitter-ruby ^0.20.1 (tree-sitter-wasms, 2024-02) → v0.23.1 — crate pinned =0.23.1, wasm built from tag 71bd32f's checked-in parser.c/scanner.c (both sha-matched against the crates.io tarball; content bump, ABI stays 14). Old-vs-new full-init dumps: sinatra/jekyll byte-identical; rails = exactly the one classified hunk (the `recv&.!=` safe-nav operator misparse fix, `table_name.!` → `table_name.!=`, precision-positive). Walker (python.rs chassis + the six ruby divergences) preserves bug-for-bug: the importTypes:['call'] funnel (class-body DSL — attr_accessor, has_many, define_method incl. its block, sinatra route blocks — emits NOTHING at non-body scope), hook-handled module multiply-capture (nested modules re-scan their subtree per level after popping — `this.hooked` fn-refs from class AND module AND file), the sibling-scan visibility trio (bare `private` invisible; `private :sym`/`private def` poison all later defs; the inner def stays public), bare-call statements (do…end body_statement emits, brace-block block_body doesn't), `.new` instantiates with last-`::`-segment names, constant-receiver references refs, require/require_relative path refs (posix-normalized, `.rb`-suffixed, `Kernel.require` and interpolated-path quirks included), `=begin` docstring marker survival, and the reverse-order value-ref DFS. Wire v2: the hook's mixin `implements` refs carry `filePath: ctx.filePath` — the ONE extraction-ref denormalized field (php's trait-use refs share the shape). RefRow's first pad byte becomes a flags slot (REF_FLAG_FILE_PATH); decode re-attaches its own filePath parameter; KERNEL_ABI_VERSION 1→2 on both sides (mismatched dist/.node pairs degrade to wasm, as designed). Gates: sweeps 0-diff sinatra 147/147, jekyll 164/164, rails 3452/3452 (3,763 files, 0 deferrals — ruby error incidence 0.00%, any deferral = walker bug); full-init dumps byte-identical ×3 (7.2k/9.4k/375.6k lines); kernel-ruby-parity suite (torture + CRLF + wire-flag pin + defer) + ruby grammar-parity row; full suite 2,613 green ×2 under CODEGRAPH_KERNEL_EXPECT=1 (one unrelated mcp-initialize timing flake under parallel load, passes solo 3/3 ×3). DEFAULT_ROUTED += ruby (12 langs). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |