Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
rst_body cut a reST section's text at 500 bytes by setting a stop value
past the end, then backing off from out[RST_BODY_MAX] while that byte
looked like a UTF-8 continuation byte. That byte was never written, so
where the text ended (499 or 500 bytes) depended on what the arena memory
held, and when a multibyte character did not fit, bytes never written
could end up in the text.
Only whole characters are written now, and the text ends at the last one
written. The same django tree indexed twice gave doc_link_candidates rows
that differed in about 250 of 11,300 (through the section text's TF-IDF);
with this change two runs are identical, and so are the 6,594 sections.
The new test pins the cut at the boundary: a word that fills the 500 bytes,
a two-byte character that would cross them, a word cut at a character.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolves the definition walks next to each other: main's decl_namespace (#2120) and this branch's extra_props are both compacted and both counted in the result census.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolves the index_status tool description: main's watch state (#2167) and this branch's doc_links both stay.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Java class may declare a field and a method of one name (`executable`
and `executable(Path)`), and the graph keeps one node per qualified name,
so one of the two was lost: the field in 90 and the method in 17 of the
107 such pairs in junit-framework. With fields named correctly (the
previous commit) every initialized field joined these collisions.
When the same class body declares a method of that name, the field now
takes the qualified name `<Owner>.<name>#field`, the `base#suffix`
fence of `#macro` and the Rust cfg twins; every other field keeps its
name. The registry already files a fenced symbol under its plain name,
so name lookups still reach it. A read, a write or a plain value
reference that resolves to such a method goes to its field twin instead
(cbm_registry_value_target); calls and callable references keep the
method.
junit-framework (development corpus), before -> after: Field 2,808 ->
2,898, Method 14,124 -> 14,141 (the 107 pairs, both nodes kept); WRITES
into these methods 66 -> 0 and into their fields 1 -> 96; USAGE into
these methods 97 -> 0 and into their fields 2 -> 212 (accessors reading
their own field were self-edges before); CALLS into the methods 1,158 +
24 (then landing on the field) -> 1,215.
A C# partial class split across files can still collide: the check sees
one class body.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Sphinx object directives (`.. py:class::`, `.. method::`,
`.. function::`, ...) bind the definition they document. The family
was held back until its held-out audit: a census of 105 links in
repositories never used for development, judged blind in two rounds,
found all 105 correct (Wilson 95% lower bound 0.965).
On django's documentation (a development corpus) the 2,046 held rows
become 1,895 edges: 1,640 new section-to-definition pairs, and 255
pairs that a role or code span already linked now carry the directive.
No other edge or row changes.
include and kernel-doc stay held. rst_ship_gate tests the production
gate itself, after the suite's forced families are reset.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A code span naming `OliveTin.exe` was read as member `exe` of the
module stem of OliveTin.exe.manifest. Extensions of built and shipped
files (exe, msi, deb, rpm, dmg, tar, xz, bz2, 7z, apk, ipa, manifest,
csv, tsv) now make a dotted token a file name; none of them is a
common member name.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Java, C# and Solidity list the same node kind as a field and as a
variable, so every class field was emitted twice: as a Field of its
class and as a Variable carrying the module's qualified name
(`demo.query.filters` for field Query.filters). The Variable collided
with package paths, and usages bound to it instead of the Field.
extract_class_variables now skips a declaration that extract_class_fields
already made a Field (walked alongside the body's Fields, in order).
Fields with an initializer were named by the whole declarator text
(`PROP = "v"`), so the duplicate Variable had been the only findable
node for them; resolve_field_name_node now takes a variable_declarator's
own name.
Measured on junit-framework: Variables 2,899 -> 685; of 9,901 usages
into the removed duplicates, 6,938 now reach the Field. Swift and
GDScript (Variables only) and D/Dart (no duplicates) are unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A fresh held-out census of ADR supersedes statements (the phrase opens its
line or field and names a record directly) passed: 80 of 81 correct (point
0.988, Wilson 95 % lower bound 0.933; 27 ADR repositories from the same
objective code-search walk, at most 25 links per repository, blind readers,
second-reader agreement 1.0). The family ships; the same words in running
prose (`supersedes_prose`) stay rows.
Its one error was a paragraph wrapped before the phrase:
`[ADR-0240](0240-x.md)` / `supersedes ADR-0134 and ...`, the subject on the
line before. A statement may no longer continue a paragraph: a line that no
list, quote or table mark opens, after a line of running text (not blank, a
heading or a `Key: value` field), is prose unless the phrase is itself a
field (`Supersedes:`). On the census this moves exactly that link to prose;
the 80 correct ones stay statements and none is added. Tests cover the
wrapped paragraph, consecutive field lines and a field under running text,
each failing with its rule reverted; the ship gate checks a statement edge
and a prose row in production.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Supplementary held-out audits, each registered before its queries and run on
repositories chosen by an objective code-search walk, disjoint from every
earlier audit and development corpus (blind readers, a second reader on
every link not judged correct and on a seeded tenth of the rest; bar: Wilson
95 % lower bound >= 0.90 and point >= 0.95). A fix is judged on a NEW sample,
never on the one that found the error.
Ship now:
- Markdown code_path, refined to code spans with two or more path segments:
97 of 100 (lower bound 0.916). A file or one directory named alone in a
code span is its own family, bare_path, held: across the judged samples it
was 28 of 41 correct (`config.yaml` the reader's own file, `doc/` another
project's directory). A span without a word character (`../`, `./`) is no
path at all.
- reST literalinclude: 100 of 100; an open range (`:lines: 5-`, `-5`) now
reads the included file for its end and binds like a closed one.
- reST code_path: 99 of 100; its one error, a bare `../`, is fixed above.
- PDF qualified names: 97 of 99 after the owner rule: a definition whose
qualified name lacks its owner (a Go method under its package) matches
only owner-qualified text, so `time.Now` no longer binds a receiver's Now.
Fixed, still held:
- reST object directives: a function or class directive never binds a
Module, Folder or File (an example script's module took them); a new
sample is due.
- ADR supersedes: three censuses failed (37/38, 144/163, 82/87). Fixed from
them: a combined template label (`Supersedes / Depends on:`), a value of
"none" or "n/a" followed by records it extends, another record as the
subject (`ADR 0239 supersedes ...`, a table cell `0259 (supersedes ...)`),
a full stop after a number ending the target sentence while a
`[text](destination)` link stays one reference (adr-tools'
`Supersedes [3. Title](0003-...md)`). The family is split: a statement that
opens its line or field and names a record directly stays `supersedes`
(44 of 45 on the last census); the same words in running prose are
`supersedes_prose` (38 of 42), held. Both still held in this commit; a
fresh census of statements passed afterwards and is acted on next.
- An ADR log in a repository without code definitions lost every supersedes
link by file: the ADR table was filled after an early return. It is now
filled first.
Every fix has a test that fails with the fix reverted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Linux arm64 production build (gcc -O2 -Werror) stopped on
maybe-uninitialized in the TrueType cmap readers and the cross-reference
stream rows: ttf_fmt4 and ttf_fmt12 ignored the results of rd16/rd32 after a
bounds check that makes every read succeed, and xref_rows read pairs[] that
only the default /Index branch fills. The ASan test build did not warn.
The reads now fail the map on a failed read (the prototype's short-slice
rule; never taken, the bounds check stands before them), and pairs[] starts
at zero where it is declared. Text output is unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The pre-registered held-out audit of the document families: nine
repositories chosen by an objective rule and disjoint from every
development corpus (three each for Markdown, reStructuredText and
AsciiDoc), 782 sampled links judged blind against the source, a second
reader on every link not judged correct and on a seeded tenth of the
rest (102; agreement 100 %). Bar per family: Wilson 95 % lower bound
>= 0.90 and point >= 0.95.
ships markdown link 100/100, path 50/50 (census), code_name 98/100;
rst role 100/100, autodoc 98/98 (census), code_name 99/100;
asciidoc include 92/92 (census)
rows markdown code_path 93/100 (a file name in a code span that
means the reader's own file, not the repository's);
asciidoc code_name 15/25 (Java overloads, a config key and a
package read as members, a binary name read as a manifest);
asciidoc code_path 12/13; rst code_path 3/3 and supersedes 1/1
(censuses too small for the bar)
rows not audited, no held-out occurrences: rst object, include,
literalinclude, kernel_doc; asciidoc attribute
A family that is not shipped still resolves; a resolved reference is a
doc_link_unresolved row with reason below_bar_tier. The suites of these
families test their resolution with the families forced on (the test
seam), and doc_links_md_ship_gate tests the production gate (red with
one held family switched back on).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The pre-registered held-out check of the PDF families: the C pipeline was
run on the eighteen repositories of the field test's two held-out PDF
audits (its links equal the frozen prototype pipeline's on the same
graph: 5,217 of 5,217, none only in one), and the audited prototype
links it makes take their labels (255 of 260 audited links). Bar per
family: Wilson 95 % lower bound >= 0.90 and point >= 0.95.
pdf_path 61 of 61 correct point 1.000 low 0.941 ships
pdf_file 110 of 114 point 0.965 low 0.913 ships
pdf_qn 77 of 80 point 0.963 low 0.896 does not ship
pdf_qn misses the bar: its references now become below_bar_tier rows
(the family's ships flag), as the pre-registration says, with no rule
tuned on the held-out data. Its three errors, all in one repository, are
names of an external API matched to a same-named local definition. The
pipeline test now
asserts the rows, and back navigation through the file a page names.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A PDF page's Section carried the whole page text as its docstring, which
is also what get_code_snippet returns for it: a crafted text layer
stored megabytes per node and handed them to an agent whole (D6 of the
document layer security review). Markdown, reStructuredText and AsciiDoc
sections keep a short docstring already.
A page's docstring is now its text up to 256 KiB, cut on a character
boundary and followed by a line that says where it was cut and how long
the text is. Pages hold a few kilobytes; the mentions are still read
from the whole text.
Test: pdf_page_doc_bounded: a 315 KB page keeps 256 KiB plus the note,
and a path written at its end is still a mention (the whole text is
stored with the bound reverted).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
For each line break that joins a name, the scanner listed the
page's candidates overlapping the joined span by reading every candidate of
the page: (joins x candidates), quadratic in a page's text, and the
geometry deciding the line breaks is the file's (D5 of the document
layer security review).
A join's span covers the end of one line and the start of the next, so
the emitted candidates are indexed by line once per page and a join
merges the two lines' lists: the same candidates in the same order, so
the same tokens.
Test: pdf_scan_join_linear: a page twice as long costs at most 2.5 x the
join work (seam counter; 717,600 vs 2,875,200 with the full scan
restored). The scanner's expected-token test is unchanged, and the PDF
links on three dev corpora (Polly, mediastore3, dask) equal the frozen
field-test pipeline's as before (0 links only in C, 0 only in Python).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Form XObject that drew the next one many times, level under level,
was read (draws ^ depth) times: a few kilobytes asked for an unbounded
reading, and the page text grew with every leaf (D2 of the document
layer security review). The forms' text cannot be read faithfully; it is
exponential in the file.
The first reading of each form in a document stays whole. Reading one
again (the same content drawn elsewhere: a logo, a header on every page)
is counted in decoded bytes; beyond 256 MiB of such re-readings the
document reads no form again, and each skip is counted
(form_rereads_cut), as a decoded stream above 512 MiB already is. Real
documents re-read small forms: the 749-PDF dev corpus extracts
byte-identically as before (720 of 720 documents, 1,492 of 1,492 pages;
flattened copies too). Reading each form only once per page was tried
first and changed one real page, so it was not taken.
Test: pdf_extract_form_ladder_bounded: seven levels of forms drawing the
next four times each, 64 KiB per form: re-readings are cut and the page
keeps its glyphs bounded (with the bound reverted nothing is cut).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Objects whose value began a token that never ended (a string without its
closing parenthesis) were each read to the end of the file, and the
cross-reference rebuild reads every object: n such objects cost n times
the file in time and in arena memory (D4 of the document layer security
review). The same held for scanned trailers and for the members of an
object stream.
The objects of a file do not overlap, so an indirect object's value is
now read up to the next object header (the header scan, built once per
document), a scanned trailer up to the next "trailer", and an object
stream member up to the next member. A stream's bytes are still placed by
its /Length. The total work is linear in the file; nothing is cut that a
well-formed file holds.
Test: pdf_extract_open_strings_linear: a thousand objects with open
strings and no cross-reference grow the arena and extraction classes by
at most 16 x the file + 1 MiB (67 MB for a 112 KB file with the window
reverted). The 749-PDF dev corpus extracts byte-identically as before
(720 of 720 documents, 1,492 of 1,492 pages), and its 720 flattened
copies give the same output before and after.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every classic section that named an /XRefStm parsed and decoded that
stream afresh, and each decoded copy was kept until the document closed:
thousands of tiny sections naming one highly compressible stream asked
for thousands of copies (D3 of the document layer security review).
A target already merged is skipped: a later (older) section naming it
again adds nothing, since every number it holds is in the table already.
A cross-reference stream's decoded bytes are freed once its rows are
read (a stream parsed there is no cached object), so distinct streams
cost one at a time. The filter pipeline of pdf_stream_decode is split
out for that (decode_filters); cached streams decode as before.
Test: pdf_extract_xrefstm_once: a thousand sections naming one stream
decode it once (xref_streams == 1; 1,000 with the skip reverted). The
749-PDF dev corpus extracts byte-identically as before (720 of 720
documents, 1,492 of 1,492 pages).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Loading a stream whose /Length referred to another stream loaded that
stream first, and its /Length the next one: one nested call per object.
Loading an object of an object stream whose /Length (or filter) lay in
another object stream opened that one inside the first, again one call
per stream. A crafted chain overflowed the stack (D1 of the document
layer security review).
A /Length is now read as a value only: an object that is a stream there
gives no length (the stream is then read up to its endstream, as for any
wrong length) and is marked, so a second reference costs nothing. The
objects that open an object stream are no objects of an object stream
(ISO 32000-1, 7.5.7); one that is, is refused and counted
(objstm_nested). Both rules leave the nesting constant; nothing is cut
by a limit.
Test: pdf_extract_object_chains extracts a 20,000-deep /Length chain and
a 3,000-deep object-stream chain on a 256 KiB thread stack (stack
overflow before; each half fails alone when its rule is reverted). The
749-PDF dev corpus extracts byte-identically to the prototype as before
(720 of 720 documents, 1,492 of 1,492 pages).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The PDF text layer carries data tables (D3). Their sources and notices are
now in THIRD_PARTY.md (shipped in every archive through
THIRD_PARTY_NOTICES.md) and in the SBOM:
- the Adobe Glyph List subset (BSD-3-Clause, Copyright 2002-2019 Adobe),
- the printable-ASCII advance widths of Helvetica and Times-Roman from
Adobe's core-14 font metrics (the AFM files are not distributed),
- the Unicode Character Database data of pdf_unicode.c (Unicode
License v3, full text included).
scripts/gen-pdf-unicode.py (stdlib CPython only) regenerates
pdf_unicode.c byte for byte with CPython 3.14.6 (Unicode 16.0.0);
pdf_tables.c is maintained data from here on. License texts were taken
from their sources (adobe-type-tools/agl-aglfn, unicode.org).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
AsciiDoc (.adoc, .asciidoc) is now indexed, without a grammar like PDF: a
Section per heading (`=` .. `======`, or `#`), the text up to the next
heading as its docstring, delimited blocks respected (listing, literal,
comment, passthrough, example, sidebar, quote, table, fences). Its
references become MENTIONS edges, via "asciidoc", from the section they are
written in:
include include::target[...] -- Asciidoctor's preprocessor directive,
read in every block but a comment block; the page's own
attributes, then the Antora component's and the playbook's,
substituted
attribute an {attribute} whose value is an Antora javadoc link
({javadoc-root}/<module>/<package path>/<Type>.html[#member])
binds the Java or Kotlin type, or its one member, that the
repository's main sources declare under that path
code_path, monospace spans naming a path or a qualified name (the
code_name Markdown code-span rules)
The rules are the field test's (H8 Asciidoctor/Antora harvester, audited
~100 % precise): an Antora resource ID ([component:][module:]family$path)
names <component>/modules/<module>/<family dir>/<path>, or the file a
collector scan contributes there (a scan of build output is generated: no
row); any other target is relative to the including file. `lines=` and
`tag=`/`tags=` regions bind by H8's rule: edge blank and comment lines
trimmed, the innermost enclosing definition, unless that is a type (or there
is none) and the definitions inside cover every other line (an annotation
right above one counts) -- then that definition; tag regions come from the
included file's own `tag::x[]` / `end::x[]` comments, read from that indexed
file; a tag the file lacks binds the file. One reference makes one edge: a
region holding several top-level definitions binds the file.
Antora settings travel as YAML scope blobs (antora.yml: component name,
asciidoc attributes, collector scans; antora-playbook*.yml: attributes),
read as text, so an incremental run resolves as a full one; a change to them
is GLOBAL. AsciiDoc joins the documents bound by lines: a body edit of an
included file re-resolves its pages.
Also: the Markdown resolver's module index no longer takes AsciiDoc or PDF
files for code modules.
Proof on junit-team/junit-framework at 6c8ca116 (H8's corpus and commit),
C against H8's recorded links: javadoc attributes 362/362 identical;
includes 292 identical, 60 bind the type enclosing the several definitions
H8 names, 7 regions with several top-level definitions bind the file. 317
sections, 668 edges, no include rows (9 generated partials, 1 file type the
indexer does not read); monospace spans 30/30 correct by hand. 8.3 s for the
whole index.
Tests: suite doc_links_adoc (structure and tokens, the Antora blob, the
pipeline: resource IDs through a collector scan, tag regions, a region with
its imports binding its class, lines=, javadoc attributes, spans; incremental
== full: a moved tag region, a changed component attribute). Revert check:
without H8's region rule and without AsciiDoc among the line-bound documents
exactly their assertions fail (observed).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A reST document now gets a Section per title (the text up to the next title
as its docstring, its lines from the title to the next one) and MENTIONS
edges, via "rst", from each section to the code it names:
role :class:`pkg.Model`, :meth:`~x.Y.z`, :func:`.f`, :c:func:`f`,
and extlink roles whose URL is a repository blob path
(:source:`django/db/models/base.py`)
object .. py:class:: / .. method:: / .. function:: ... and their
C-domain twins
autodoc .. automodule:: / .. autoclass:: ...
include, .. include:: / .. kernel-include:: of a code file,
literalinclude .. literalinclude:: (its :lines: bind the innermost
definition holding them, kept as target_lines; its
:pyobject: the named definition)
kernel_doc .. kernel-doc:: path and its :identifiers: / :functions:
code_path, inline literals naming a path or a qualified name (the
code_name Markdown code-span rules)
Structure is the field test's line model (H3): titles by adornment style,
literal blocks after `::` and in code directives, comments, directive
arguments and options, Sphinx's module / currentmodule / class contexts,
CPython's str whitespace and \w. Text before the first title belongs to the
file. A Sphinx documentation set's conf.py is read as text, never run: its
source_suffix makes discovery index e.g. Django's `.txt` documents as reST in
that set only, its primary_domain decides undomained roles, its extlinks and
intersphinx names are known to the resolver.
Python names resolve the way Sphinx imports them: the longest prefix that is
a module's import name (named from its topmost package directory), the rest
by qualified name, a package's re-exports (`from .base import Model` in its
__init__.py) followed. The re-exports and the conf.py settings travel as
Python scope blobs (doclink_py.c), so an incremental run resolves as a full
one; one resolver serves reST and Python. Forms in Sphinx's order (T, C.T,
M.T, M.C.T; a leading `.` reverses it). EXACT when a form names a node,
otherwise UNIQUE by name with the field test's fixes: a bare member
(:meth:`save`, :attr:`x`) is never searched (37 of 40 such results were
wrong), an object directive's target lies in its declared module, a module
never answers a member's role (`:meth:`dispatch`` is not tests/dispatch),
test code is no target of a search but binds when the document writes its
name in full (`django.test.TestCase`). C identifiers are whole names: the one
C/C++ definition of that name and kind. Unresolved: missing when the name or
path is the repository's, ambiguous, test_only_target; another project's
name (stdlib, intersphinx) is no row.
reST architecture decision records: the ADR detector reads reST records
like Markdown ones (titles underlined with = or -): course-discovery's 34
decision records become ADR nodes.
Proof on Django's documentation (the field test's corpus and commit), C
against the frozen H3 harvester over the same graph, per (document,
section, target): 6,695 of H3's 7,238 links identical; H3-only 543 = links
C attributes to the document because a section title repeats in it (the
Markdown rule; the same document -> code link exists), deliberate refusals
(bare members, directive searches without a module) and H3's own audited
error class (router protocol methods -> ConnectionRouter, forms Field ->
model Field); C-only = the same attribution mirrored, plus 1,053 inline
literal links (sample of 30: 30 correct). 6,574 reST sections; 12 s for the
whole index.
Tests: suite doc_links_rst (structure and tokens, the scope blobs, the
pipeline's rules, source_suffix in discovery, reST ADRs, incremental ==
full). Revert check: without the module-kind guard and without the
source_suffix mapping exactly their assertions fail (observed). Two
expectations of the old behaviour now name the new one: the reST grammar
golden and probe have the document's Sections (they recorded "reST titles
are no Sections", the gap this closes).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A PDF is now indexed: its text layer is read by a dependency-free extractor
(internal/cbm/pdf, zlib only) and every page becomes a Section node ("page N",
the page's text as its docstring, property page), so search finds what a page
says and get_code_snippet shows a page's text, never the file's bytes. The code
a page names structurally becomes MENTIONS edges from that page, via "pdf":
pdf_path a repository path ("pkg/server/handler.go", "./x/y")
pdf_file a file name with a code extension ("models.py", "Pool.sol")
pdf_qn a qualified name ("app.models.User.save", "a::b", "x->y")
Bare identifiers never become edges (field test E6c: a type-target rule failed
its pre-registered bar), and only EXACT or UNIQUE resolutions link: the field
test's tiers (path-exact, component-aligned path-suffix, qn-exact, qn-suffix,
the latter also through a node's owner -- a Go method's QN has no receiver --
and with the same one-entity collapses). A mention split across a line break is
proposed as one reference and replaces its fragments only when it resolves.
Hygiene as measured: no links from PDFs under test or fixture directories (R1),
none to data files (R2), none for keyword arguments (R3); a test, mock or
fixture target is a test_only_target row (R4, with the test directories the
field test found missing). A path or name whose first component is not of this
repository is no row at all.
The extractor is a port of the field-tested prototype: xref tables and
streams, /Prev chains, repair near a wrong offset and reconstruction from the
object headers, object streams, Flate (with partial recovery), LZW, ASCII85,
ASCIIHex, RunLength and PNG/TIFF predictors, simple fonts (base encodings,
/Differences, built-in Type1 encodings, standard-14 widths, Type3), ToUnicode
and encoding CMaps, Type0/CID fonts with the embedded TrueType cmap read
backwards, Form XObjects, inline images skipped, word and line breaks from
glyph geometry, right-to-left runs reversed. Its geometry is bit-exact on every
platform (no FMA contraction; CPython's hypot). Untrusted input: every read is
bounds-checked, recursion is depth-limited, integers from the file saturate, a
decoded stream is cut at 512 MiB, and every lookup that could rescan the file
per object or per glyph is answered from an index built once. Not read:
encrypted documents (reported as such); scanned pages have no text layer.
The mention scanner is the field test's (idents.py) with CPython's semantics:
NFKC per code point plus canonical composition and the str predicates, from
generated Unicode 16.0 tables (pdf_unicode.c), and re's leftmost-first
matching.
Pipeline: CBM_LANG_PDF (no grammar; the registry ledger gets a DOCUMENT
capability, the call-node manifest exempts it like the other grammar-free
classification), extraction before any grammar lookup, a resolver hook
file_prepare (a file's state from all its references before any resolves).
Incremental: a proxy node carries parent_class where owner + name differs
from its QN (Go methods, class-body variables), so a re-resolved document binds
a Go method as a full build does.
Proof against the field-test prototype on its corpus (749 PDFs from 21
repositories, restored at the recorded commits, sha256-verified): text
byte-identical on all 1,492 pages of the 720 comparable PDFs (0.73 s vs 14.7 s);
mention tokens identical on all 960 page-text sets of E6, E6b and E6c (26,435
tokens); links identical to the frozen Python pipeline run over the same graph
on semantic-kernel 5/5, cosmos-sdk 30/30, Polly 11/11, mediastore3 3/3, dask
4/4. 44,070 seeded mutants (truncations, flips, nesting and keyword bombs, on
the originals and on uncompressed copies) under ASan and UBSan: 0 findings
after the first campaign's integer overflows were fixed as a class.
Tests: suite doc_links_pdf (layout, fonts and CMaps, object and xref streams,
xref repair and reconstruction, Form XObjects incl. a self-referencing one,
inline images, filters, hostile input, the scanner against the field test's
output, the pipeline's tiers, hygiene and rows, back navigation, the snippet,
incremental == full).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Markdown file that is an architecture decision record now also defines an
ADR node (File DEFINES ADR): its name is the record's canonical id ("ADR-12";
another prefix as written, "DEC-5"; a dated record keeps its file stem), its
qualified name "<module>.__adr__", and its properties carry the record's
facts: adr_id, title, status (normalized: proposed, accepted, implemented,
superseded, deprecated, rejected, abandoned, draft, unknown), status_text,
date, deciders, superseded_by, and the decision itself as the docstring, so
search finds a record by what it decided. Its sections link to code like any
document's (MENTIONS, previous commit).
Detection is per file, with the field tests' rules: a file in a conventional
ADR directory (adr, adrs, decisions, decision-records, ...) whose name carries
an id or whose text has ADR structure; any adr-NNN file; a numbered or dated
file with ADR structure. Never README, index or template files without an id,
never test fixtures. Facts come from front matter, then header fields
(bullets, bold, tables), then the Status section; the date also from a
"Last updated" field, the change log's first date and the file name.
An ADR's own statement that it supersedes another ("Supersedes [ADR-3](...)",
"Supersedes ADR-3"; "Replaces ADR-3" where it opens a line, never "replaces"
as a verb of prose) becomes a SUPERSEDES edge between the two ADR nodes, by
link or by id. "Superseded by" is written in the old record about another
file; an edge from the other node would be owned by a file whose text does
not hold it, so it stays a fact of the old record (status, superseded_by).
The link layer gains a per-family edge type (doclink.h: the family list's
"edge" column), so a family can produce SUPERSEDES instead of MENTIONS; edges
are grouped by (type, source, target). Definitions gain a tail field for a
document's facts (extra_props: key, value pairs), carried through result
compaction and written by both property builders.
Measured against the field-test harvester on five repositories: detection
182 of 182 records, the same files; title 182/182; status 179/182 (cosmos
ADR-050 and its annexes say "Status: ARCHIVED", kept as status_text with
status unknown); date 174/182 (semantic-kernel's "date: {2024-05-02}" is read
as a date where the harvester dropped it). SUPERSEDES: semantic-kernel
0038 -> 0015, log4brains 20201016 -> 20200926, each checked against its
sentence. Markdown MENTIONS unchanged on fastapi, curl, dapr, cosmos-sdk.
Not covered: ADRs in reStructuredText, and ADR-log configuration files
(.adr-dir, log4brains) as a detection source.
Tests: doc_links_md_adr (detection and exclusions, facts from front matter
and from a Status section, SUPERSEDES by link and by id, no edge from prose,
the record's MENTIONS, incremental == full for a status edit and for an
added record named by id).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Markdown document is documentation in itself: its sections now link to
the code they reference explicitly, as MENTIONS edges (via "markdown") from
the Section node (the File node for text before the first heading, and for a
heading whose title repeats in the file, whose Section node it does not
own). An agent can go from a document to the code and back through the
graph: trace_path inbound with edge_types ["MENTIONS"] lists the sections
that name a definition.
Families (the "syntax" property; each with its ship gate):
- link: the destination of an inline link or a link reference definition
(no URL, no in-page anchor, no image);
- path: a bare path in prose (a slash and a file extension);
- code_path: a code span or HTML <code> span holding a path, a file name or
a directory, with an optional line range (#L3-L9, :3-9) or member
(path::Name);
- code_name: a code span holding a qualified name (a.b, a::b, A#b, A\B,
mod:attr).
Resolution is guess-free, with the rules the documentation field tests
measured:
- a path is read relative to the document and, when it has a directory
part, to the repository root; a file name alone only next to the document;
- a line range binds the innermost code definition holding the whole range,
else the file with "target_lines" on the edge; path::member the one
definition of that name in the file;
- a qualified name binds the one code definition whose qualifier chain ends
with the written qualifier, else the one module or package so named; a
match as written outranks one that needs case folding; a field binds only
through its type, never through an instance; module:attr only in Python;
product documentation never binds test code;
- a bare name is never an edge.
Unresolved references are rows in doc_link_unresolved: missing (the
directory or the repository's own root is there, the name is not),
ambiguous, test_only_target. A URL, an image, a URL route (/docs), a file of
a type the indexer does not read, and code of other projects are no
references to the repository and leave no row.
Extraction scans the document's bytes once (doclink_md.c); every scan of a
line is linear, or n log n where it sorts, using position lists that grow
with the bytes they track, not with the line. Front matter, fenced code,
HTML comments and MDX import/export lines are not scanned.
The incremental route's node proxies now carry their line numbers
(pipeline_delta.c), so a re-resolved document binds a line range exactly as
a full build does; the Markdown resolver reads definitions from its own
by-file list of the run's nodes, never from DEFINES edges, which an
incremental run holds for re-extracted files only.
Measured on four repositories against the main branch (which links file to
file only, REFERENCES_FILE): fastapi 2,899 MENTIONS edges (2,225
REFERENCES_FILE before), cosmos-sdk 1,286 with 205 distinct definitions
reached and 188 edges from 41 ADRs into code, dapr 184, curl 52.
Link pairs agree with the field-test harvester (cosmos-sdk 408 of 409; the
difference is a link of a document to itself, skipped on purpose). A hand
check of 45 sampled qualified-name links found one wrong, fixed by the
exact-case rule above.
Tests: doc_links_md (extraction, repeated headings, the span classifier
table, resolution with every reason, back navigation, incremental == full
for a code edit, a document edit and an added name).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Conflict in internal/cbm/result_compact.c: main (#2543) added the variants copy and this branch the decl_namespace copy at the same place in cr_walk_def; both kept.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The project-file scan wrote the "could not be read" blob for a *.csproj
that is no readable MSBuild <Project>, and for a *.props or *.targets file
only once its <Project> root was seen. A *.props or *.targets file that is
malformed before its root, or ends before one, got no blob at all: an
import of it counted as outside the repository with a closed scope, and as
the nearest Directory.Build.* it was passed over for the one above, whose
usings were then applied although MSBuild does not read that file.
Such a file now gets the "could not be read" blob too, so the projects that
evaluate it have an open scope. A *.props or *.targets file whose root
element is not <Project> is still no MSBuild file and has no blob.
Test: msbuild_malformed_props_opens (a malformed nearest
Directory.Build.props under a usable one: a name found nowhere is external,
and a type only the upper file's using reaches is not bound; decoy: a
project without a nearer file binds through the upper file and keeps a
closed scope).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_doclink_note_doc_line returned silently when the map (or its first
block) could not be allocated. The doc it was noting then keeps its
definition's line, and a doc shared by the declarators of one field
declaration is no longer marked as taken, so its references are parsed
once per declarator -- while the layer reports ok.
Both allocation failures now set the file's doc_links.failed, as every
other lost piece of doc-link data does; the layer then says error.
A test seam, CBM_DOCLINK_ALLOC_DOC_LINE, fails the map's growth.
Test: alloc_failure_status gets the doc-line map as one more point (one
error row).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A C# scope blob written in a run that the reader refuses costs only its own
file in that run (a8c5708d). The blob itself was stored, though, and the next
incremental run read it back as a stored scope the reader refuses, which
fails the layer for the whole run.
The surface row now stores, in place of such a blob, the C# rejected marker
(the tag and one `!` record). Whether the reader takes a blob is decided
where the row is made, by the reader's own record checks on an index of the
blob's own (what makes a blob refused depends on the blob alone); the
doc-link layer asks the language through two new resolver fields,
scope_accepted and rejected_scope (cbm_doclinks_storable_scope). A later run
that reads the marker back treats the file as the run that wrote it: it
declares nothing, its references are graph gaps, it is counted with the
rejected files. A stored scope that does not parse for any other reason
still fails the run. The scope delta of a file whose blob is the marker on
either side is LOCAL only when both are.
The test seam that spoils a file's scope now does it where the scope is
written (internal/cbm/doclink_cs.c), so what is stored carries it too.
Test: cs_rejected_scope_across_runs (a full run with the spoiled file, then
a body-only edit elsewhere and one in the spoiled file: each incremental
step equals a full run, and the stored scope is the marker).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The review names two scans whose failure returned no scope: the C# scope
scan and the project-file scan. Both record the failure on the file's doc
links since e775dcd9, but only the C# scan had a failure seam. The
project-file scan gets one (CBM_DOCLINK_ALLOC_PROJECT), and the status test
gets a seventh point: a project file whose scan ran out of memory fails the
layer (one error row), like the other six.
Test: alloc_failure_status (its repository now has a project file).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
S8: a C# scope blob written in this run that the resolver cannot parse no
longer fails the layer for the whole repository. What the file had put into
the shared index is taken back (the namespaces it created and their keys,
the declared and production marks it set, the quarantine keys it added), the
file is kept as an empty scope marked rejected, and each of its references
becomes a graph_gap row. The rejected files and references are counted
(stats, and the log line doc_links.cs.rejected_scopes); the layer status
stays ok. A STORED scope that does not parse still fails the run. The
scanner's output passes the reader's record checks for the shapes the review
names: a namespace name with an empty dotted segment is a namespace the scan
cannot name. Seams: spoil the scope of one file in the run, and parse a blob
with the resolver's reader.
S10: every byte a scan writes into a scope blob is well-formed UTF-8, where
it is written. In a C# file a name that is not UTF-8 is not placed (an
identifier, a namespace or type name) and a text that is not is written as
"?" (a kept text, a signature); in a project file every byte that starts no
well-formed sequence is written as U+FFFD, in element names, attribute
values and text, and a numeric reference to a surrogate code point is
U+FFFD too. The stored bytes stay a function of the file alone, and the
surface writer no longer fails the run on such a file.
Tests: cs_rejected_scope_contained, cs_scanner_output_reads_back,
cs_utf8_surfaces (the former observation test now requires the writer to
succeed and the blob to be UTF-8).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
S4: the declarators of one C# field declaration share one doc comment. Its
references are taken once, from the first declarator (the Field definition
that carries them, or the first variable where there is none); the other
declarators take nothing from it, so tokens, resolutions and rows grow with
the references, not with references x declarators. A shared doc that could
not be collected whole is no doc of any declarator and is not looked up again
per declarator.
S17: memory that runs out while a file's doc-link data is extracted -- a
doc span or doc text, a reference value, a token, the scope scan -- is
recorded on the file's doc links (CBMDocLinkArray.failed), and the resolving
half fails the layer for the run (status error). A MENTIONS edge that could
not be stored fails the layer too and is not counted. Seams: the scope scan
(CBM_DOCLINK_ALLOC_SCOPE) and the edge insert.
Tests: cs_shared_doc_once, alloc_failure_status; the shared-doc tests of the
checkpoint follow the once rule.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
- S11: a namespace read back from a stored scope is held to the nesting the
scanner writes (64 segments of its full name); past it the scope is
refused like any other stored scope this code did not write.
- S12: index_status lists only the reasons the layer writes; any other
reason text in the table is counted under one fixed key
(unrecognized_reason) and the status is error.
- S13: the per-file scratch table of the index build is made anew when a far
larger file grew it, so emptying it costs about the file's own size.
- S14: the portable copy of a scope keeps an empty line field empty and is
never longer than the scope it is made from.
- S15: a reference whose brackets do not pair, or that goes on after its
parameter list or its type arguments, is unparseable instead of being
resolved by the part that can be read.
- S16: a project file the scan did not read (malformed, or larger than a
project file is) opens the scope of the projects that evaluate it: what it
holds is unknown, not absent. It is counted once.
- S18: an incremental repair matches the names a changed file no longer
declares by the resolver's own identifier rule (bytes of multi-byte
characters included) and without a length cut.
Tests: cs_portable_scope_bound, cs_unbalanced_brackets,
cs_stored_scope_nesting, index_status_unknown_reason,
msbuild_unread_project_opens, cs_scratch_table_work,
incremental_non_ascii_name; msbuild_gate and index_status_preview_bounds
follow the new S16 and S12 outcomes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A reference written in a doc comment is extracted with the definition it
documents (internal/cbm/doclink.c; the C# scanner doclink_cs.c) and
resolved per file in the pipeline (src/pipeline/doc_links.c,
doc_links_cs.c, doc_links_msbuild.c). A reference that names a definition
of the repository with certainty becomes a MENTIONS edge from the
documented definition (the File node for a file-level comment) with the
properties via, syntax, tier, line and count. Any other reference becomes
a row of the new doc_link_unresolved table with its reason: missing,
ambiguous, external, test_only_target, graph_gap, unparseable or
below_bar_tier. Nothing is guessed. index_status reports the layer in a
doc_links block.
C# binds a cref by the compiler's lookup rule: the first scope with the
name decides, a cref never binds an inherited member, System is outside
the repository. Assemblies come from project files, and MSBuild global
usings (<Using>, ImplicitUsings, Directory.Build.props/.targets, imports,
conditions) are evaluated from the project files' content as data the
indexer has already read; the layer opens no file. Two declarations of a
name in different assemblies make the reference ambiguous.
Each file's scope is stored line-free with its LSP surface ("dl"), so an
incremental run re-resolves the files a change can reach and falls back
to a full run for a change that can re-route references anywhere (a
project file, a global using, a deleted file).
The scanner and the resolver treat the repository as untrusted input:
bounded nesting, no input-driven recursion, work that grows linearly
(held by test seams, never by a clock), no silent limits, and a damaged
or foreign stored scope fails closed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Merging main brought #2340's virtual_newline field in CBMStringInput. cbm_rescue_defs_from_projection still built its input with the two-field initializer, a missing-field-initializers error under -Werror. The projection keeps the raw source's length and line structure, so it now goes through cbm_parse_source like the raw parse and the C++ branch views: the same virtual newline, and the same edit that removes it again.
extraction pipeline graph_buffer store_search mcp registry edge_types_probe parse_coverage c_lsp test_impact_engine conditional_variants on the merge result: 2277 passed, 4 skipped.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Main's graph_buffer.c ("Definition variants") records a definition's
variant spans as [{"file_path","start_line","end_line"}], and the
test-impact engine reads that schema to find every TEST(...) branch of one
test. This branch's extractor wrote [{"start","end"}]. On the merge with
main, graph_buffer's same-QN merge found a non-empty variants list it could
not read and dropped every span, the definition's own included. The engine
then no longer found the second #if branch of TEST(alpha_uses_lib), so
test_impact_engine_variant_tests_are_one_test reported suite alpha
UNMAPPED, in CI on macOS and Windows and locally.
The extractor now writes the shared schema, with the file path
JSON-escaped. The C-preprocessor-only gate and the one-carrier rule are
unchanged, and the variants assertions use the shared format.
test_impact_engine, extraction, pipeline, graph_buffer and
conditional_variants: 857 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Conflict in tests/test_parse_coverage.c: this branch's #2078 tests and main's #1967 line-count test (#2336) were both appended before the suite; both kept, every TEST still registered.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Main (#2458) named the anonymous struct/union/enum of `typedef struct { ... } Name;` in extract_class_def from the typedef's declarator. This branch's extract_c_typedef already emits that aggregate as Name from the type_definition, so on the merged tree Name was emitted twice under one label: its members twice and a false variants entry. extract_c_typedef stays the one place that names it; the extract_class_def case is removed. Both typedef test sets pass on the result: #2458's extract_c_anonymous_typedef_aggregate_is_named_by_its_typedef and member-access tests, and this branch's C1 typedef/variant tests (extraction, pipeline, registry, graph_buffer, store_search, mcp: 1,336 passed, 4 skipped).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Merging main brought extract_cpp_branch_views (conditional variants), which built its CBMStringInput with the two-field initializer. With this branch's virtual_newline field that is a missing initializer, an error under GCC's -Wextra -Werror (the shadow job's product build). A branch view keeps the raw source's length and line structure, so it now goes through cbm_parse_source like the raw parse: the same virtual newline, and the same edit that removes it again.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Conflict in internal/cbm/extract_calls.c with main's #1146 slashless Laravel routes (#2471): both helper sets kept (this branch's route_prefix_compose/route_path_with_prefix, main's php_plain_string_value/php_first_arg_value/php_group_array_prefix). At the call site main's slashless-URI step runs first, then this branch's shared route_path_with_group_prefix replaces main's inline #952 composition (identical composition and guard), then main's string-action handler rule.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Conflict in tests/test_edge_types_probe.c: both sides added a self-contained helper + tests before the Rails test (this branch: et_handles_exact_routes and the class-handler tests; main/#2471: et_route_set_exact and the slashless-route tests) and their RUN_TEST lines; both blocks kept whole, this branch's first.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Conflicts: internal/cbm/extract_defs.c includes (union of both sides); src/pipeline/registry.c keeps main's lang-aware index_under_name call, and the fence comment combines both facts (Rust cfg twins are variants of one QN; a C macro's #macro QN never reaches the registry).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Conflict in internal/cbm/cbm.h resolved by keeping both added CBMDefinition fields: decl_namespace (this branch) and the test-definition role/spans (main).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel