extract_docstring() took only the single comment node directly before a
definition. Multi-line `//` and `///` docs therefore kept just their last
line (in C# usually `/// </summary>`); docs above attributes, exports and
wrapper declarations were lost; fields, variables and modules never got a
doc; Perl POD was never stored; and every doc was cut at 500 bytes.
Capture the full documentation comment instead:
- contiguous same-style comment runs; Go comment groups with go/ast
directive lines (`//go:`, `//line`, `//export`, ...) stripped
- docs above Rust attributes, JS/TS export statements and wrapper
declarations (arrow and function expressions, class fields,
assignments, ambient declarations) and C# top-level statements
- docs for fields, variables, constants and enum members, found with a
memoised linear walk
- Go package comments and Rust `//!` docs on the File node
- Perl POD sections that name a sub (`=head`/`=item <name>`)
- Rust impl methods, Kotlin KDoc folded into package_header or sibling
gaps, C `typedef struct/enum/union` declarations
- a doc comment wins over an intervening plain `//`; runs detached by a
blank line and runs made only of divider lines are not docs
- no 500-byte cut; the properties buffer grows with the docstring, so a
long doc no longer pushes other properties out
Measured on nine real-world corpora (Go, C#, Java, TypeScript, Rust, C,
PHP, Perl, Kotlin): full-doc capture Go 35% -> 99.999%, C# 10% -> 99.8%,
Java 74% -> 99.99%, Rust 7% -> 99.9%, C 67% -> 99.2%, Kotlin 70% ->
99.96%, TypeScript sources 42% -> 100%, named-sub Perl POD 0% -> 100%.
Node sets and all non-semantic edges are unchanged; index time stays
within run-to-run noise.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
dab9187ca5