Normalize Windows cache paths when reporting runtime settings and resolve
Git executables before spawning history queries. Keep executable lookup
results per thread and cover the ownership contract deterministically.
Support Git 2.43 while denying history-query network transports, and close
the activity database on daemon startup failure so Windows can clean up.
Validation: 463 native macOS tests, 421 Windows tests, 44 MSan UI tests,
and focused canonical lint passed. Production-only revert checks confirm
the repaired failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Merge validated environment overrides into a child-owned environment on POSIX and Windows, preserving caller PATH lookup for executable names. Forward each ObjectScript export parse timeout to the extractor.
Cover inherited and replaced values, empty values, duplicate overrides, parent isolation, invalid names, executable lookup, and timeout forwarding.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Follow-up to #2589.
No timeout decides a cli_scope test any more:
- The status probe and the parent's client connect repeat an attempt that
reaches a live but mute endpoint (a loaded runner answering slowly). A
reply or a definite refusal decides. Each attempt keeps the API's finite
5 s budget. This replaces the 15 s status budget and the single 5 s
connect.
- The host child is reaped without a deadline (it used to get SIGKILL after
60 s).
- The host child joins its version cohort with UINT64_MAX, which waits for
locks but answers a conflicting holder at once (it used to have a 45 s
deadline).
A wedged child is left to the harness's per-suite ceiling.
cli_scope_fixture_finish restored HOME, CBM_CACHE_DIR and SHELL even when
start had failed before saving them, which unset them for every later test
in the suite. It now restores only what start saved.
New tests:
- cli_scope_fixture_finish_keeps_env_after_an_early_failure fails on main
at ASSERT(home_kept);
- cli_scope_retries_a_mute_endpoint_until_it_answers checks the retry rule
on scripted attempt outcomes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cli_install_skip_binary_unchanged_in_host_namespace_quiesces_nothing
failed once on 2026-10-09 at ASSERT(ready), with nothing else in the log.
cli_scope_fixture_start returned false silently from any of its setup
steps, and a 90 s deadline decided the wait for the host child's
readiness.
The fixture now records and prints the step that failed: the fixture
directory, the host directories, the runner's own image fingerprint, the
pipes, the fork, the host's start, or the client connect, with the OS error
where one applies. The host child prints which of its own steps failed
before it answers 'E'.
The readiness wait has no deadline any more, because no timeout may decide
a test. The child answers 'R' or 'E', or the pipe reaches EOF when it dies.
A child that wedges is left to the harness's per-suite ceiling.
New test: cli_scope_fixture_names_its_failing_setup_step.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On macOS the signed-candidate preparation folded every step into one
boolean, so each failure printed the same line: "signed update candidate
preparation failed". A full /tmp read exactly like a broken codesign. That
is what a cli test hit on 2026-10-09, while a multi-GB index was filling a
nearly full disk.
Each step now records why it failed, and the update prints, for example:
error: signed update candidate preparation failed while writing the
unsigned candidate: No space left on device
The staging functions in activation_transaction.c keep the errno of the
failing write (or sync, close, read) through their cleanup, so a caller can
name it. The non-macOS path adds the OS error to its existing "failed to
stage verified update" line (POSIX only; Windows reports through
GetLastError). A failed fingerprint of the staged file now gets its own
message instead of "...: activation transaction completed".
A test seam makes the next staging write fail with a chosen errno. The new
test cli_update_names_the_failing_staging_step_and_os_error fails without
the cli.c change and passes with it, and checks that the old executable
stays in place. Also checked on real input: with /tmp on a small volume
that fills between two updates, the update prints the line above.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
rst_body cut a reST section's text at 500 bytes by setting a stop value
past the end, then backing off from out[RST_BODY_MAX] while that byte
looked like a UTF-8 continuation byte. That byte was never written, so
where the text ended (499 or 500 bytes) depended on what the arena memory
held, and when a multibyte character did not fit, bytes never written
could end up in the text.
Only whole characters are written now, and the text ends at the last one
written. The same django tree indexed twice gave doc_link_candidates rows
that differed in about 250 of 11,300 (through the section text's TF-IDF);
with this change two runs are identical, and so are the 6,594 sections.
The new test pins the cut at the boundary: a word that fills the 500 bytes,
a two-byte character that would cross them, a word cut at a character.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1a342553 made suffix_match ties deterministic by preferring the least
nested QN, then the smaller QN. Determinism was the point, and it stays.
But depth says nothing about which same-named definition a call means.
An audit of main against v0.11.0 (2026-10-10) traced three resolution
regressions to that one rule:
- django: about 5,600 `Model.objects.X(...)` calls tie between
`QuerySet.X` and same-named methods elsewhere. `Library.filter`,
`EngineHandler.all` and `HttpResponseBase.get` are each one segment
shallower, so the QuerySet share fell from 57 % to 24 %.
- Kotlin / Java: a call went to a property or field that merely shares
the callee's name. Kotlin `where(...)` went to `var where`, and Java
`esr.indexMode()` to a field whose QN lacks its class segment.
Kotlin CALLS edges on Variable/Field targets went 785 -> 1,648.
- C#: .NET reference assemblies (`src/libraries/<Asm>/ref/<Asm>.cs`,
`throw null` bodies) are shallower than the implementation, and "ref"
sorts before "src". 14.1 % of dotnet/runtime's CALLS landed on such
stubs (4.9 % in v0.11.0).
A tie now goes to:
1. the more plainly callable candidate: Function/Method, then any other
definition, then an unknown label, then Variable/Field;
2. for C# only, an implementation over a reference-assembly stub;
3. the lexicographically smaller QN.
This is still a pure function of the candidate set (O9). The kernel
cases behind 1a342553 keep their answers: `dev_name` is the only Function
among the fields, and include/linux sorts before tools/virtio.
Measured on the bench corpora (macOS arm64, main cb8b641f vs this
change, one cold full index each):
- django: `objects.X` -> QuerySet 23.7 % -> 41.4 %; CALLS -> Variable
1,022 -> 741
- Kotlin (Exposed): CALLS -> Variable 1,648 -> 480
- Java (elasticsearch): CALLS -> Variable/Field 62,313 -> 42,957
- C# (dotnet/runtime): CALLS -> ref/ stubs 143,799 -> 52,000;
CALLS -> Variable/Field 33,779 -> 28,662
- TypeScript: CALLS -> Variable 7,978 -> 7,341
The rest of the django gap needs Manager -> QuerySet typing, which no
tie order can supply.
Tests:
- registry_tie_prefers_callable_over_property
- registry_tie_ignores_nesting_depth
- registry_tie_prefers_csharp_implementation_over_ref_stub
All three are RED on main and RED again with the registry change
reverted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
CI's cppcheck 2.20 failed the lint job on rr_c_role: after the last dot was
cut to '\0', a second strrchr looked for the owner's dot, and cppcheck took
that result for the first one ("Condition 'prev' is always true",
knownConditionTrueFalse). The owner is now the segment found by walking back
from the last dot to the dot before it or to the start of the name: the same
owner, without a second search (Homebrew's 2.21 did not report it).
rst_c_member_owner covers the member path, which no test reached: the owner
running back to the start (point.x), to the dot before it (shapes.box.x),
through "struct" and -> (struct point->y), and a field taken for a struct
(box.corner.y, missing). It passes on the old and the new code alike, and
fails when the walk ignores the dot before the owner.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every process start fingerprints its own executable for the daemon build
identity, and the hook path pays for that before it can answer. With the
portable transform that was ~1.2 s of CPU for the ~300 MB binary, so on a
slower machine hook-augment hit its 2000 ms deadline and printed nothing
(#2441).
Two changes in src/foundation/sha256.c:
- cbm_sha256_update compresses whole blocks straight from the input instead
of copying every byte through the context buffer.
- Block compression runs on the CPU's SHA-256 instructions when it has them:
ARMv8 SHA2 (sysctl on macOS, HWCAP on Linux, IsProcessorFeaturePresent on
Windows) or x86 SHA-NI (CPUID). The backend is probed once; the portable
transform stays the fallback.
Measured on an Apple arm64 Mac (macOS 26.6), release builds of main 72a2c0bc
and of this change, hook-augment SessionStart in fresh isolated dirs, 9
interleaved runs, medians:
- hook-augment: wall 1.448 s -> 0.376 s, user CPU 1.162 s -> 0.097 s
- SHA-256 of the 303 MB binary: 1.18 s -> 0.095 s CPU per pass (the
block-direct update alone, portable rounds: 0.74 s)
- with CBM_HOOK_DEADLINE_MS=1000 (a slower machine): before, exit 0 with
empty stdout after 1.01 s; after, the full 188-byte answer in 0.37 s
Digests are unchanged: the build id the new binary logs equals the file's
SHA-256, and the hook output is byte-identical. A new test compares the
hardware and portable backends at every length 0..299 and at block-boundary
sizes up to 1 MiB under eight update chunkings, plus the NIST million-'a'
vector; it fails if the ARM schedule is broken or if Apple arm64 falls back
to portable. The x86 path was checked under qemu-x86_64 -cpu max (2,436
comparisons equal, million-'a' correct) and falls back to portable on a CPU
without SHA-NI. On a Windows arm64 VM the probe picks the hardware path:
0.094 s against 0.82 s portable for the 303 MB binary, digest equal to
sha256sum's.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
index_status now ends with a watch object (#2167) whose hint names index_repository. The checks searched from doc_links: to the end of the text, so the watch hint failed the no-hint check and could have satisfied the hint and index_repository checks of the error cases. dm_doc_links_block cuts the text at the next top-level key.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolves the definition walks next to each other: main's decl_namespace (#2120) and this branch's extra_props are both compacted and both counted in the result census.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolves the index_status tool description: main's watch state (#2167) and this branch's doc_links both stay.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Doc -> code candidates now carry a p judged for their kind, their document
format and where the document sits.
Format (sample 6, 234 blind-judged candidates): reStructuredText, AsciiDoc and
PDF sections each get their own curve (reST stored from tfidf 0.30, AsciiDoc
from 0.20, PDF not measured and not stored). A candidate that repeats an
exact MENTIONS link of the same section is not stored.
Position (sample 7, 490 blind-judged candidates from four monorepos): every
document gets a home folder -- its folder; a docs folder documents its
parent; a folder without code hands over to the nearest one with code; the
repository root means the whole project. Markdown candidates outside the
document's home were right 6 of 90 and are never stored; inside it they are
stored from tfidf 0.30 (0.33 / 0.53); for documents of the whole project
samples 4 and 7 pooled give 0.22 / 0.51 from 0.30.
New kinds, all in doc_link_candidates (evidence gains "kind" and
"position"; no new edge type, no schema change):
- local: up to two home functions ranked below the top five (0.32);
- file: a whole file, its vector the sum of its functions' terms and its
definitions' names, config keys included, cut to 64 terms (0.30, from
tfidf 0.40);
- folder: a README or index document -> its home folder (0.87).
get_code_snippet on a section lists every kind with its label;
get_file_outline shows the sections about the file or a folder holding it
(cbm_store_doc_candidates_for_path).
cbm_sem_p_2dp rounded 0.325 and 0.525 down (their float values sit 1.2e-6
below the half); its epsilon is 1e-4 on the x100 scale.
Replayed on the sample repositories, the build stores exactly the rows the
judged curves select (950 rows, 0 differences).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
pipeline_doc_candidates_published_and_kept_by_delta appended its
delta-route edit to "user_config.go" while the fixture writes
"User_config.go". On a case-insensitive filesystem (macOS, Windows)
that is the same file; on Linux it is a new file, closure repair
declines new files (correctly), the run falls back to a full rebuild,
the route assertion fails and the early return leaks the pipeline
(LeakSanitizer on the arm64 GCC leg). The edit now names the fixture
file exactly.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A documentation section that explains a function rarely names it, so
the deterministic doc links cannot connect them. The semantic pass now
ranks, for every Markdown section that can be about code, the functions
whose TF-IDF terms best match the section's heading and body, over the
functions' own corpus (function-pair scores are untouched):
- gate: no candidates for release-number or project-meta headings
(changelog, licence, contributing, links, contents, install, ...),
bodies under 20 words, or bodies that are mostly links;
- targets: functions and methods, never tests (by path or name);
- stored: a section's best five with tfidf >= 0.20, each with p, the
probability that the section is about the function, from 274
blind-judged candidates in five held-out repositories (three reading
rounds): 0.36 below 0.30, 0.42 below 0.40, else 0.55. Below 0.20 the
judged share was 0.06, so nothing is stored there.
These are candidates for an agent to verify, not graph edges: at best
about half are right, far below the bar for a link. They live in a side
table, doc_link_candidates (section and function qualified names, rank,
score, p, evidence), created when a generation is published, so older
indexes read as "no candidates". Full and legacy-partial reindexes
replace a project's rows; the delta route keeps the previous rows (its
proxy buffer has no section text or function tokens to recompute
them), and a row whose node is gone is skipped when read.
get_code_snippet shows them: on a Section, "possibly_about" (its
candidate functions); on a function or method, "possibly_described_in"
(the sections), at most five, highest p then score first, with a
candidates_note saying what they are.
Measured on fastapi (6,144 gated sections): 24,293 rows, identical to
the generator's dump; index time within noise; the database grows by
the table (37 MB there, with a 148-character project prefix on every
qualified name).
Tests: semantic:sem_doc_calibrated_p_bands,
store_arch:doc_candidates_replace_and_get (RED when a replace keeps old
rows), pipeline:pipeline_doc_candidates_published_and_kept_by_delta
(RED with test functions admitted, the heading gate off, the publish
write skipped, or the delta route replacing rows),
mcp:snippet_doc_candidates_both_directions (RED with stale rows shown).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The semantic pass read every string property of a function (body text,
signature, return type, decorators, docstring) with a reader that
stopped at the first '"' and kept escapes raw: a value holding an
escaped quote lost everything after it, and "\nreturn" became the token
"nreturn". cbm_sem_json_str honours the escapes: \" \\ \/ keep their
character, \n \t \r \b \f and \uXXXX become a separator.
The tokens change, so admissions change: on 13 repositories 1,340 ->
1,387 admitted SEMANTICALLY_RELATED pairs, 614 of them unchanged. Of
the dropped pairs that earlier samples had judged, 52 of 89 were
related (0.58); of the kept ones, 73 of 92 (0.79). A pre-registered
blind sample of 100 newly admitted pairs from 11 held-out repositories
(three reading rounds) judged 65 related (0.65, Wilson 0.55-0.74), so
the admitted set is 0.72 precise against 0.69 before. The same index
run twice gives identical pairs and scores.
"p" is refit on the new admitted population (bands at the quartiles of
the admitted scores, pooled where a higher band judged lower; added and
kept pairs weighted by their share): score < 0.773: 0.63, else 0.76
(was 0.56 / 0.64 / 0.73 / 0.86 in four bands).
Tests: sem_json_str_honours_escapes (RED with escapes ignored),
sem_calibrated_p_bands.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
When a function has more candidate partners than its budget of ten
SEMANTICALLY_RELATED edges, admission took the pairs in discovery order:
the first ten won, whatever their scores. Admission is now best-first
(cbm_sem_admit_best_first: descending score, ties in the canonical
order, still a pure function of the sorted inputs). On a blind judged
sample from held-out repositories, the pairs first-come admitted over
budget were 0 of 17 related; the pairs best-first admits instead, 7 of
20. Where no budget binds nothing changes (django 66 edges identical;
fastapi 426 -> 430 edges, 222 changed).
Every SEMANTICALLY_RELATED edge now carries "p": the probability that a
reader judges the pair related, by score band, from 120 blind-judged
admitted pairs (score < 0.778: 0.56, < 0.806: 0.64, < 0.852: 0.73,
else 0.86). p is computed from the score as stored (three decimals),
so one shown score always has one p. Scores and the 0.75 threshold are
unchanged; a scoring change (TF-IDF alone, or weights fitted on the
first sample) was judged on the same sample and did not beat the
shipped score by the pre-registered margin.
Tests: sem_admit_best_first_keeps_the_best, sem_calibrated_p_bands,
pipeline_semantic_edges_carry_p; each is RED with its behavior
reverted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The TF-IDF signal keyed each term by the token's POSITION in the
function, so any two functions of similar length shared their first
terms whatever their words: 0.20 of every SEMANTICALLY_RELATED score
measured length, not vocabulary. Terms are now keyed by corpus token
(cbm_sem_tfidf_terms: one term per distinct word, weight term
frequency x idf, ascending token index, which the sparse cosine
merges on).
Measured on a blind judged sample: 222 function pairs from six
held-out repositories, stratified by score band, read in three rounds.
The share of admitted SEMANTICALLY_RELATED edges a reader judges
related rises from 0.713 to 0.742, with about 19% fewer edges (fastapi
507 -> 426, django 64 -> 66). The same sample shows that no score
threshold reaches 0.90 precision, so the documented "~95% at 0.75" did
not hold before this change either.
The harness that measured it ships with the fix, off by default:
CBM_SEM_PAIR_SIGNALS=<path> writes one line per scored pair (both
qualified names, the score, every signal value, whether the pair was
admitted, both locations); CBM_SEM_PAIR_SIGNALS_FLOOR also records
pairs below the threshold, never admitting them. cbm_sem_combined_score
is now cbm_sem_signal_values + cbm_sem_combine (same arithmetic, same
order). CBM_SEMANTIC_INDEX_VERSION 3 -> 4: indexes built before score
pairs differently.
Tests: sem_tfidf_terms_compare_vocabulary_not_positions,
sem_combine_weighs_signals,
pipeline_semantic_pair_signals_never_change_the_graph; each is RED with
its behavior reverted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A second held-out census of AsciiDoc code names (61 of 83) failed on
one class: 21 of its 22 errors bound a Java package folder to a dotted
name that is no code at all. Debezium's `snapshot.mode` and
`debezium.sink.type` are configuration keys, OpenNMS's
`org.opennms.features.amqp.eventreceiver` is an OSGi configuration id,
`org.apache.cassandra.auth` there is a JMX domain, `sk.ainet.vendors` a
Maven groupId; each spells the path of a real package directory.
A code name now never binds a folder whose first code definition is in
a Java, Kotlin, Scala or Groovy file. A Python package is what its
dotted name says and keeps binding. Like the four rules before it, this
one only takes links away.
Proof: development corpora: fastapi, dask and django unchanged;
junit-framework loses 2 links to package folders (the docs name
`org.junit.jupiter.api.condition` as a package, so those two were
right: a recall cost of the rule). Judged held-out replays: Markdown
and reST code names keep all 195 links judged correct and lose none;
the failed AsciiDoc census loses its 21 judged-false folder links and
its 3 judged-correct ones. The AsciiDoc family stays off until a new
census.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
AsciiDoc code names (resolved like Markdown's) failed their held-out
census at 43 of 80. Four of its error classes have a rule that holds
for every format:
- a name written from the root `std` never binds a C/C++ definition or
header of the repository: `std::complex` is the standard library's,
not Boost.Typeof's `typeof/std/complex.hpp`;
- a name written from a JVM root package (`com`, `org`, `java`, ...)
binds a JVM definition only where it starts the package (after a
source-root piece such as `src/main/java`), never as the tail of a
relocated copy (`...hbase.shaded.com.google.protobuf.Message`);
- a capitalized type of a Java, Kotlin, Scala, Groovy or C# definition
matches in its own case only: `debug.log` is a log file, not the
method `log` of class `Debug`;
- a definition in another copy of the document's project never answers
the document: below the directory where the two paths part, the
document's own branch holds the same rest of the path, at least two
directories deep, as a file with definitions (snapshots side by side;
a shared `__init__.py` is no copy).
Each rule only takes a link away: a refused candidate still counts, so
two candidates stay ambiguous, and it is never the one linked.
Proof: six development corpora (junit-framework, fastapi, dask,
pydantic, curl, django) keep all 1,730 code_name links. Replayed on the
judged held-out samples: Markdown and reST code names keep all 195 links
judged correct and add none; on the failed AsciiDoc census the rules
remove judged-false links only (7 in the vendored-Boost repository,
which loses 127 code_name links and gains none). The first version of the copy rule dropped
one (a package's `__init__.py` taken for a copy) and turned ambiguity
between copies into new links; both are fixed here. The AsciiDoc family
stays off until a new census.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Java class may declare a field and a method of one name (`executable`
and `executable(Path)`), and the graph keeps one node per qualified name,
so one of the two was lost: the field in 90 and the method in 17 of the
107 such pairs in junit-framework. With fields named correctly (the
previous commit) every initialized field joined these collisions.
When the same class body declares a method of that name, the field now
takes the qualified name `<Owner>.<name>#field`, the `base#suffix`
fence of `#macro` and the Rust cfg twins; every other field keeps its
name. The registry already files a fenced symbol under its plain name,
so name lookups still reach it. A read, a write or a plain value
reference that resolves to such a method goes to its field twin instead
(cbm_registry_value_target); calls and callable references keep the
method.
junit-framework (development corpus), before -> after: Field 2,808 ->
2,898, Method 14,124 -> 14,141 (the 107 pairs, both nodes kept); WRITES
into these methods 66 -> 0 and into their fields 1 -> 96; USAGE into
these methods 97 -> 0 and into their fields 2 -> 212 (accessors reading
their own field were self-edges before); CALLS into the methods 1,158 +
24 (then landing on the field) -> 1,215.
A C# partial class split across files can still collide: the check sees
one class body.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Sphinx object directives (`.. py:class::`, `.. method::`,
`.. function::`, ...) bind the definition they document. The family
was held back until its held-out audit: a census of 105 links in
repositories never used for development, judged blind in two rounds,
found all 105 correct (Wilson 95% lower bound 0.965).
On django's documentation (a development corpus) the 2,046 held rows
become 1,895 edges: 1,640 new section-to-definition pairs, and 255
pairs that a role or code span already linked now carry the directive.
No other edge or row changes.
include and kernel-doc stay held. rst_ship_gate tests the production
gate itself, after the suite's forced families are reset.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A code span naming `OliveTin.exe` was read as member `exe` of the
module stem of OliveTin.exe.manifest. Extensions of built and shipped
files (exe, msi, deb, rpm, dmg, tar, xz, bz2, 7z, apk, ipa, manifest,
csv, tsv) now make a dotted token a file name; none of them is a
common member name.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Java, C# and Solidity list the same node kind as a field and as a
variable, so every class field was emitted twice: as a Field of its
class and as a Variable carrying the module's qualified name
(`demo.query.filters` for field Query.filters). The Variable collided
with package paths, and usages bound to it instead of the Field.
extract_class_variables now skips a declaration that extract_class_fields
already made a Field (walked alongside the body's Fields, in order).
Fields with an initializer were named by the whole declarator text
(`PROP = "v"`), so the duplicate Variable had been the only findable
node for them; resolve_field_name_node now takes a variable_declarator's
own name.
Measured on junit-framework: Variables 2,899 -> 685; of 9,901 usages
into the removed duplicates, 6,938 now reach the Field. Swift and
GDScript (Variables only) and D/Dart (no duplicates) are unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A fresh held-out census of ADR supersedes statements (the phrase opens its
line or field and names a record directly) passed: 80 of 81 correct (point
0.988, Wilson 95 % lower bound 0.933; 27 ADR repositories from the same
objective code-search walk, at most 25 links per repository, blind readers,
second-reader agreement 1.0). The family ships; the same words in running
prose (`supersedes_prose`) stay rows.
Its one error was a paragraph wrapped before the phrase:
`[ADR-0240](0240-x.md)` / `supersedes ADR-0134 and ...`, the subject on the
line before. A statement may no longer continue a paragraph: a line that no
list, quote or table mark opens, after a line of running text (not blank, a
heading or a `Key: value` field), is prose unless the phrase is itself a
field (`Supersedes:`). On the census this moves exactly that link to prose;
the 80 correct ones stay statements and none is added. Tests cover the
wrapped paragraph, consecutive field lines and a field under running text,
each failing with its rule reverted; the ship gate checks a statement edge
and a prose row in production.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Supplementary held-out audits, each registered before its queries and run on
repositories chosen by an objective code-search walk, disjoint from every
earlier audit and development corpus (blind readers, a second reader on
every link not judged correct and on a seeded tenth of the rest; bar: Wilson
95 % lower bound >= 0.90 and point >= 0.95). A fix is judged on a NEW sample,
never on the one that found the error.
Ship now:
- Markdown code_path, refined to code spans with two or more path segments:
97 of 100 (lower bound 0.916). A file or one directory named alone in a
code span is its own family, bare_path, held: across the judged samples it
was 28 of 41 correct (`config.yaml` the reader's own file, `doc/` another
project's directory). A span without a word character (`../`, `./`) is no
path at all.
- reST literalinclude: 100 of 100; an open range (`:lines: 5-`, `-5`) now
reads the included file for its end and binds like a closed one.
- reST code_path: 99 of 100; its one error, a bare `../`, is fixed above.
- PDF qualified names: 97 of 99 after the owner rule: a definition whose
qualified name lacks its owner (a Go method under its package) matches
only owner-qualified text, so `time.Now` no longer binds a receiver's Now.
Fixed, still held:
- reST object directives: a function or class directive never binds a
Module, Folder or File (an example script's module took them); a new
sample is due.
- ADR supersedes: three censuses failed (37/38, 144/163, 82/87). Fixed from
them: a combined template label (`Supersedes / Depends on:`), a value of
"none" or "n/a" followed by records it extends, another record as the
subject (`ADR 0239 supersedes ...`, a table cell `0259 (supersedes ...)`),
a full stop after a number ending the target sentence while a
`[text](destination)` link stays one reference (adr-tools'
`Supersedes [3. Title](0003-...md)`). The family is split: a statement that
opens its line or field and names a record directly stays `supersedes`
(44 of 45 on the last census); the same words in running prose are
`supersedes_prose` (38 of 42), held. Both still held in this commit; a
fresh census of statements passed afterwards and is acted on next.
- An ADR log in a repository without code definitions lost every supersedes
link by file: the ADR table was filled after an early return. It is now
filled first.
Every fix has a test that fails with the fix reverted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The pre-registered held-out audit of the document families: nine
repositories chosen by an objective rule and disjoint from every
development corpus (three each for Markdown, reStructuredText and
AsciiDoc), 782 sampled links judged blind against the source, a second
reader on every link not judged correct and on a seeded tenth of the
rest (102; agreement 100 %). Bar per family: Wilson 95 % lower bound
>= 0.90 and point >= 0.95.
ships markdown link 100/100, path 50/50 (census), code_name 98/100;
rst role 100/100, autodoc 98/98 (census), code_name 99/100;
asciidoc include 92/92 (census)
rows markdown code_path 93/100 (a file name in a code span that
means the reader's own file, not the repository's);
asciidoc code_name 15/25 (Java overloads, a config key and a
package read as members, a binary name read as a manifest);
asciidoc code_path 12/13; rst code_path 3/3 and supersedes 1/1
(censuses too small for the bar)
rows not audited, no held-out occurrences: rst object, include,
literalinclude, kernel_doc; asciidoc attribute
A family that is not shipped still resolves; a resolved reference is a
doc_link_unresolved row with reason below_bar_tier. The suites of these
families test their resolution with the families forced on (the test
seam), and doc_links_md_ship_gate tests the production gate (red with
one held family switched back on).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The pre-registered held-out check of the PDF families: the C pipeline was
run on the eighteen repositories of the field test's two held-out PDF
audits (its links equal the frozen prototype pipeline's on the same
graph: 5,217 of 5,217, none only in one), and the audited prototype
links it makes take their labels (255 of 260 audited links). Bar per
family: Wilson 95 % lower bound >= 0.90 and point >= 0.95.
pdf_path 61 of 61 correct point 1.000 low 0.941 ships
pdf_file 110 of 114 point 0.965 low 0.913 ships
pdf_qn 77 of 80 point 0.963 low 0.896 does not ship
pdf_qn misses the bar: its references now become below_bar_tier rows
(the family's ships flag), as the pre-registration says, with no rule
tuned on the held-out data. Its three errors, all in one repository, are
names of an external API matched to a same-named local definition. The
pipeline test now
asserts the rows, and back navigation through the file a page names.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A PDF page's Section carried the whole page text as its docstring, which
is also what get_code_snippet returns for it: a crafted text layer
stored megabytes per node and handed them to an agent whole (D6 of the
document layer security review). Markdown, reStructuredText and AsciiDoc
sections keep a short docstring already.
A page's docstring is now its text up to 256 KiB, cut on a character
boundary and followed by a line that says where it was cut and how long
the text is. Pages hold a few kilobytes; the mentions are still read
from the whole text.
Test: pdf_page_doc_bounded: a 315 KB page keeps 256 KiB plus the note,
and a path written at its end is still a mention (the whole text is
stored with the bound reverted).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
For each line break that joins a name, the scanner listed the
page's candidates overlapping the joined span by reading every candidate of
the page: (joins x candidates), quadratic in a page's text, and the
geometry deciding the line breaks is the file's (D5 of the document
layer security review).
A join's span covers the end of one line and the start of the next, so
the emitted candidates are indexed by line once per page and a join
merges the two lines' lists: the same candidates in the same order, so
the same tokens.
Test: pdf_scan_join_linear: a page twice as long costs at most 2.5 x the
join work (seam counter; 717,600 vs 2,875,200 with the full scan
restored). The scanner's expected-token test is unchanged, and the PDF
links on three dev corpora (Polly, mediastore3, dask) equal the frozen
field-test pipeline's as before (0 links only in C, 0 only in Python).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Form XObject that drew the next one many times, level under level,
was read (draws ^ depth) times: a few kilobytes asked for an unbounded
reading, and the page text grew with every leaf (D2 of the document
layer security review). The forms' text cannot be read faithfully; it is
exponential in the file.
The first reading of each form in a document stays whole. Reading one
again (the same content drawn elsewhere: a logo, a header on every page)
is counted in decoded bytes; beyond 256 MiB of such re-readings the
document reads no form again, and each skip is counted
(form_rereads_cut), as a decoded stream above 512 MiB already is. Real
documents re-read small forms: the 749-PDF dev corpus extracts
byte-identically as before (720 of 720 documents, 1,492 of 1,492 pages;
flattened copies too). Reading each form only once per page was tried
first and changed one real page, so it was not taken.
Test: pdf_extract_form_ladder_bounded: seven levels of forms drawing the
next four times each, 64 KiB per form: re-readings are cut and the page
keeps its glyphs bounded (with the bound reverted nothing is cut).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Objects whose value began a token that never ended (a string without its
closing parenthesis) were each read to the end of the file, and the
cross-reference rebuild reads every object: n such objects cost n times
the file in time and in arena memory (D4 of the document layer security
review). The same held for scanned trailers and for the members of an
object stream.
The objects of a file do not overlap, so an indirect object's value is
now read up to the next object header (the header scan, built once per
document), a scanned trailer up to the next "trailer", and an object
stream member up to the next member. A stream's bytes are still placed by
its /Length. The total work is linear in the file; nothing is cut that a
well-formed file holds.
Test: pdf_extract_open_strings_linear: a thousand objects with open
strings and no cross-reference grow the arena and extraction classes by
at most 16 x the file + 1 MiB (67 MB for a 112 KB file with the window
reverted). The 749-PDF dev corpus extracts byte-identically as before
(720 of 720 documents, 1,492 of 1,492 pages), and its 720 flattened
copies give the same output before and after.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every classic section that named an /XRefStm parsed and decoded that
stream afresh, and each decoded copy was kept until the document closed:
thousands of tiny sections naming one highly compressible stream asked
for thousands of copies (D3 of the document layer security review).
A target already merged is skipped: a later (older) section naming it
again adds nothing, since every number it holds is in the table already.
A cross-reference stream's decoded bytes are freed once its rows are
read (a stream parsed there is no cached object), so distinct streams
cost one at a time. The filter pipeline of pdf_stream_decode is split
out for that (decode_filters); cached streams decode as before.
Test: pdf_extract_xrefstm_once: a thousand sections naming one stream
decode it once (xref_streams == 1; 1,000 with the skip reverted). The
749-PDF dev corpus extracts byte-identically as before (720 of 720
documents, 1,492 of 1,492 pages).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Loading a stream whose /Length referred to another stream loaded that
stream first, and its /Length the next one: one nested call per object.
Loading an object of an object stream whose /Length (or filter) lay in
another object stream opened that one inside the first, again one call
per stream. A crafted chain overflowed the stack (D1 of the document
layer security review).
A /Length is now read as a value only: an object that is a stream there
gives no length (the stream is then read up to its endstream, as for any
wrong length) and is marked, so a second reference costs nothing. The
objects that open an object stream are no objects of an object stream
(ISO 32000-1, 7.5.7); one that is, is refused and counted
(objstm_nested). Both rules leave the nesting constant; nothing is cut
by a limit.
Test: pdf_extract_object_chains extracts a 20,000-deep /Length chain and
a 3,000-deep object-stream chain on a 256 KiB thread stack (stack
overflow before; each half fails alone when its rule is reverted). The
749-PDF dev corpus extracts byte-identically to the prototype as before
(720 of 720 documents, 1,492 of 1,492 pages).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel