The READS/WRITES resolvers and the USAGE registry fallback hand bare
reference text to the short-name registry, which contains Field nodes -
so once Go struct fields exist (#1935), every Go local err := ... binds
whichever struct field is named err, project-wide: 21308 USAGE and 5191
WRITES onto Go fields on the measured repo, top target a test struct's
field T collecting 3013 edges. In Go that binding is impossible by
construction: a field is only reachable through a selector expression
(x.f), and selector references resolve on the LSP path - every
Field-targeted reference edge in the census carried dot-less text.
Add cbm_go_suppress_bare_field_ref() next to the #1928 predicate and
consult it at the same four sites (both READS/WRITES resolvers, both
USAGE registry fallbacks): drop the bind when the file is Go, the
target label is Field, and the reference text has no '.'. Go-gated
because C#/Java/C++/Python method bodies legitimately reference their
own members bare (cp_reads_writes_cs_static_field pins that shape).
Field-validated on the #1940 stack: USAGE onto Go fields 21308 -> 0,
WRITES 5191 -> 0; the only remaining field-targeted edges are 2466
CALLS, which are #1906/#1907's selector-guard territory. Reproduce-
first pipeline probes (sequential + parallel twins) were RED on the
stack without this commit.
Fixes#1942
Signed-off-by: Ilya Brykau <ilya.brykau@orca.security>
USAGE, WRITES and READS edges resolve through the same short-name
registry as CALLS but never consulted the #725 cross-language guard.
On a Go tree with eBPF C probes every Go identifier spelled like a C
one produced a reference edge into the C file: 31.5% of all WRITES on
the originally measured repo crossed the Go->C boundary, led by dozens
of Go test locals named event writing a C probe's automatic variable.
Add cbm_suppress_cross_language_ref() - the reference-edge analog of
cbm_suppress_cross_language_suffix_match - and consult it on BOTH
resolvers of each edge type: the sequential pass (pass_usages.c
resolve_usage_edges registry-fallback branch, resolve_rw_edges) and
their parallel twins (pass_parallel.c resolve_file_usages,
resolve_file_rw). The sequential-only version of this change left 344
Go->C WRITES alive on a ~1150-file repo because large repos resolve
through pass_parallel.c - the field census caught it, and the
parallel-twin test now pins it.
Unlike the CALLS guard the predicate takes no strategy parameter: a
reference edge carries no import-closure evidence, so every registry
strategy is a bare-name guess across a boundary. LSP-backed semantic
references resolve before the fallback and are unaffected. JS/TS stay
one family, and C/C++ count as one family too (.h maps to
CBM_LANG_CPP, so a .c file referencing its own header is not a
boundary).
Field-validated on the ~1150-file Go+C repo: Go->C/C++ WRITES 835->0,
USAGE 1545->0; C->Go 15/90->0; 6432 reference edges dropped in total,
every one cross-language (Go->.json 3034, Go->.hpp/.h 1612, Go->.c
746, Go->.sh/.yaml/.yml 443, ...) and none same-language. CALLS and
IMPORTS totals are byte-identical to main.
Fixes#1928
Signed-off-by: Ilya Brykau <ilya.brykau@orca.security>
A Go import path names a package - never a function, method, field or
another language's build target. Strategy 1 resolves every correct Go
import (module path -> the package's Folder node); when it misses the
import is external and the correct result is NO edge. Two name-guess
fallbacks violated that:
- Strategy 3's language-agnostic symbol-name fallback matched the
path's last segment against any project definition of the same
simple name and returned the lexicographically smallest survivor:
import "os/exec" bound a test harness's exec() method (52 edges on
the measured repo), two imports bound a Function extracted from a
Makefile. 89 of 2295 Go IMPORTS edges (3.9%) were false; 27 of those
are the import "C" case (#1926/#1931), the remaining 62 this.
- Strategy 1b's sibling-file resolution admits symbol labels through
import_targetable_label, re-creating the same bug one directory
closer: with Strategy 3 gated, the field census still bound one
os/exec import to a same-package exec() method.
Gate both on the importing file's language through a new pure
predicate, cbm_import_symbol_fallback_allowed(): false for Go, true
for everything else - member-importing languages (Python, Java, Rust
use crate::ops::helper) legitimately need the symbol fallback, and
build/markup grammars (SCSS partials, Meson subdir, Pony use) the
sibling one. Strategies 1, 2 and 4 are untouched.
Reproduce-first test ei_go_import_never_binds_symbol asserts the
IMPORTS relation EXACTLY (an internal package import stays; neither
the cross-package nor the same-package decoy may be bound): RED
count=2 expected==1 before each gate, GREEN after. Adds
ei_edge_count_is, the exact-count sibling of ei_edge_present, since a
floor cannot catch a fabricated extra edge.
Field-validated on a real ~1150-file Go+C repo: Go IMPORTS by target
label went Folder 2206 / Method 85 / Function 4 on main to
Folder 2206 / Method 0 / Function 0 with the fix; all 2206 correct
package-Folder edges survive.
Fixes#1934
Signed-off-by: Ilya Brykau <ilya.brykau@orca.security>
A RETURN could name a variable the WITH before it dropped. The query ran,
exited clean, printed a column for that name, and filled it with empty
strings. Nothing said anything was wrong.
That silence is the harm. A reader sees a column of nothing and reads it as
"the graph holds no such data". The true answer is "your query named
something that is out of scope". For a code graph, a false "nothing calls X"
is the worst answer it can give.
The check runs on the parsed query rather than on the run-time bindings, and
that split is the whole point. It asks whether the query DECLARED the name,
which is a different question from whether a row happened to bind it. So the
existing convention is untouched: an OPTIONAL MATCH target that matched
nothing is still declared, still legal, and still projects "".
Two placeholders are skipped, because neither names anything the query
declared: count(*) stores "*", and a CASE expression stores "CASE".
A query with more names than the guard can model skips the check rather than
guessing. A wrong refusal costs the caller a working query, which is worse
than the silence this removes.
Same failure shape as #373, and the same answer: say so out loud.
Fixes#1919
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
Fixes the 2026-08-29 daemon zombie class: a client disconnected while its
application request was in flight, the request never observed cancellation,
and runtime_worker_finish joined the request thread with no deadline. The
disconnect path wedged forever, the generation stayed formally RUNNING, and
a dead process held the endpoint pipes and UI port for nine hours while
every new client timed out with no diagnostic.
Four changes:
- runtime: the disconnect-path reap of an in-flight application request now
waits at most RUNTIME_ABANDONED_REQUEST_JOIN_TIMEOUT_MS (30 s), then logs
daemon.application_request_unresponsive (peer pid, request token) and
fail-stops the process. The kernel releases every native claim and the
next client starts a fresh generation. Test seams expose the ceiling and
replace the terminal stop with a recordable hook.
- diagnostics: a transport connect that reaches a live process but gets no
valid answer now resolves the holder's kernel pid
(GetNamedPipeServerProcessId / SO_PEERCRED). `daemon start` classifies a
mute holder as RESERVED (no doomed competitor spawns) and names the pid
in its timeout message; `daemon status` prints "not responding (endpoint
held by pid N)" with recovery guidance instead of "not running".
- frontend: the maintenance monitor's quiet-state poll drops from 10 ms to
250 ms. Each probe revalidates the whole ancestor directory chain on
Windows; at 100 Hz that burned ~20% of a core for the life of every MCP
session (7762 CPU-seconds on one 9.5 h session). Detection latency spends
0.25 s of the 15 s activation drain budget.
- version_cohort: the healthy fresh-claim record is renamed from
"claimed_unheld" (misread as a stale-claim anomaly during the incident
diagnosis) to "claimed_fresh" with prior_holder=none.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: OhOkThisIsFine <102485413+OhOkThisIsFine@users.noreply.github.com>
The reconcile_basis marker was never written on Windows, so all six
build_trusted_artifact_repo() tests plus artifact_export_marks_clean_basis
failed there (16 passed, 7 failed) while every Linux and macOS leg was green.
Root cause is a stamp-encoding mismatch, not shell quoting. file_hashes rows
are stamped by the semantic manifest via cbm_path_info_utf8(), which on Windows
derives mtime_ns from ftLastWriteTime -- 100-nanosecond FILETIME ticks.
db_hashes_match_disk() re-stat()ed those same files through _wstat64, whose
st_mtime carries WHOLE SECONDS, and compared the two. The values can only agree
when a file's write time lands exactly on a second boundary, so the trust check
rejected the very first row on every export, on Windows and only on Windows.
On POSIX both paths read st_mtim nanoseconds and agree, which is why the split
was clean along platform lines.
The disk side now reads through cbm_path_info_utf8 -- the same function that
wrote the rows -- so there is no encoding left to keep in sync. Its
is_regular/is_symlink guard preserves exactly the symlink refusal
reconcile_stat_no_symlink provided.
reconcile_restamp_rows deliberately keeps art_stat_mtime_ns: a restamped row is
read by the incremental classifier and by check_index_coverage, both of which
stat() and encode whole seconds on Windows. The asymmetry is now pinned with a
comment at both sites so neither is "unified" into breaking the other.
Diagnostics, which is why this took two wrong diagnoses to find:
- Export's four clean-basis preconditions were one short-circuiting && chain
collapsing to a single bool. They are now evaluated by reconcile_basis_trusted,
which records the first failing gate (head_unresolved / head_not_hex_oid /
tree_not_clean / db_hashes_differ_from_disk, the last carrying the offending
path and BOTH stamps) and logs it as artifact.reconcile_basis_omitted. The
reason is recorded rather than recomputed on demand: export's own
ensure_gitattributes leaves an untracked .gitattributes behind, so any later
evaluation answers tree_not_clean whatever the real cause was.
- build_trusted_artifact_repo() had six `return NULL` exits and reported none of
them; CI could not tell a failed `git commit` from a missing marker. Every exit
now names its step and carries the number that decided it -- git exit status,
pipeline rc, or export's recorded blocker -- surfaced through
BUILD_TRUSTED_REPO_OR_FAIL at the call site so file:line still points at the
failing test.
Verified locally on macOS by emulating the Windows encoding: truncating the
disk-side mtime to whole seconds reproduces the CI signature exactly (16 passed,
7 failed, same seven sites), and the new diagnostic names it as
`db_hashes_differ_from_disk path=a.rs reason=mtime_differs stored_mtime_ns=...
disk_mtime_ns=...`. Restored, the artifact/pipeline/incremental/mcp/index_format/
daemon_bootstrap suites are green and make lint-ci passes.
Co-authored-by: moofone <35233167+moofone@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every ADR fixture in the tree used \n, so nothing exercised a CRLF-stored
document — and CI could not have found this, because the fixtures are LF
whichever platform runs them. A red would only have appeared once a real
user's CRLF ADR met this code.
The exposure is specific to splicing. The old rebuild normalised everything
on the way out, so line-ending confusion was invisible; a byte-span
replacement can miscount a separator run, and byte-identity is the property
the splice design exists to provide.
Two of the three paths were already correct, and now say so in tests:
- Replacing a section preserves CRLF exactly. The walk-back over a trailing
break run steps across both '\r' and '\n', so it cannot cut mid-pair.
- Heading matching is ending-agnostic: "## PURPOSE\r\n" locates the same as
"## PURPOSE\n", because the scanner trims '\r' from the name. Trimming and
matching were different claims and only trimming was pinned.
Appending a new section was wrong. It counted trailing '\n' characters, so a
document ending "...Foo\r\n" was read as having one break but then had a
BARE '\n' appended, and the "## NAME" line it wrote used '\n' too — leaving a
mixed document that neither the author nor the tool asked for. A CRLF file
ending in a blank line also gained a spurious extra break.
Appends now count a "\r\n" pair as one break, and write the separator and
heading line using the document's own ending.
The contract, now that a document can be mixed:
- Bytes that already exist are never rewritten, so each line keeps whatever
ending it had. This follows from splicing and is what the acceptance
property already guaranteed.
- Text this code writes itself — the separator and the "## NAME" line —
follows the document's majority ending, LF on a tie.
- A caller's body is inserted verbatim. Rewriting bytes a caller supplied
would be the same silent modification this whole change set removes.
Co-authored-by: andis777 <24672074+andis777@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`CBMDefinition.base_classes` carries the SOURCE SPELLING of each base
("Base", "django.db.Model"): extraction strips keywords and generic
arguments, but it cannot know where the name is declared. The Python and
TS cross-file registrars, however, consume `CBMLSPDef.embedded_types` as
fully-qualified names — py_lookup_attribute and ts_lookup_member feed each
entry straight into cbm_registry_lookup_type.
An unqualified spelling therefore matched nothing declared in ANOTHER
file. `class Child(Base)` in child.py never saw Base in base.py, so a
call to an inherited method through a typed receiver found no member and
fell through to the weak textual cascade, where the receiver-aware guard
(#592/#606) correctly kills it. The member-lookup walk over
embedded_types was already there in both languages; only the names it
walked were unusable across files.
cbm_pxc_collect_all_defs now resolves each base spelling to a project QN
using exactly the inputs pass_semantic uses to draw its INHERITS edge:
the project registry, the declaring module, and the file's import map.
Two properties follow. The LSP's inheritance view is the same relation
the graph records, so the two cannot diverge. And the binding is import-
or same-module-backed rather than a short-name guess, so the CALLS edge
it enables is a supported fact that the weak-member guard keeps.
Weak registry strategies (suffix_match / unique_name / field_type_hint /
fuzzy) are rejected for bases via an explicit drop-list: a base bound
because some project type happens to share its name is precisely the
fabricated relation #606 removed, and inheritance multiplies it — every
inherited member of the wrong base would become a callable target. An
unresolved base keeps its raw spelling and the behaviour that predates
this change, so stdlib and third-party bases are unaffected.
Scoped to the languages whose registrars read embedded_types as QNs
(Python, JS/TS/TSX). Go, JVM, C#, C++ and Rust already qualify their own
embedded types and keep the raw join.
Cost is O(defs) hash lookups plus ONE import map per file: no per-call-
site hierarchy walk, no registry scan, no per-file registry rebuild. The
complexity guard stays linear (nodes 1.93, edges 2.07, per-file defs
2.00 against a files x corpus coupling of ~4).
Tests:
- ts/S6 flips to its own documented condition: was `calls == 0` as a
tripwire recording the gap, now asserts calls >= 1 AND an lsp_ts_*
strategy. Measured: strategy=lsp_ts_method, confidence 0.95.
- python/S6b is new and covers what python/S6 structurally cannot: S6's
`def run(c)` parameter is un-annotated, so no receiver type exists to
inherit through and any binding there is a guess about a name (it
resolves via unique_name at 0.75). S6b supplies the receiver three ways
— annotated parameter, `self` in the subclass, constructor result — and
asserts INHERITS >= 1 plus an lsp_* strategy. Revert-check: all three
fall back to unique_name at 0.75 without this change, lsp_method at
0.90 with it.
- Both rows assert the STRATEGY, not just the edge count: a short-name
guess and a resolved inherited member produce the same count in a
two-file fixture, and only the strategy tells them apart.
- Corrects python/S6's stale comment citing a base_classes extraction
bug; extraction_inheritance is green for Python and TS, and
base_classes now holds clean names.
macOS full suite 7690 passed / 0 failed / 8 skipped (140 suites);
make lint-ci clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An imported artifact carries the EXPORTER's mtime/size in every file_hashes
row, so the first incremental run after a bootstrap sees every file as
changed and re-parses ~the whole repository — most of the time the artifact
was supposed to save.
Reconcile the rows instead. After a successful import, ask the local git
which files are unchanged between the artifact's commit and the working
tree, and re-stamp exactly those rows with local stat() values. Everything
git cannot vouch for stays foreign and is re-parsed.
Export writes an optional "reconcile_basis":"git-clean-head" marker into the
existing artifact.json, and only when the tree is provably clean outside
.codebase-memory/ AND every hash row still matches disk. Unknown fields are
ignored by older binaries, so no schema_version bump.
Eligibility is deliberately narrow, and the polarity is skip-on-doubt:
a row is restamped only if it is tracked at the artifact commit (git
diff/ls-files are blind to ignored files — a .cbmignore negation can index
a gitignored file, #500) and absent from the changed set. Any uncertainty —
no git, missing/non-hex commit, shallow clone, truncated -z output,
allocation failure — skips reconciliation entirely and falls back to
today's behavior. Reconciliation can never fail the import.
Windows correctness: every git argument uses DOUBLE quotes, per the rule
already stated in artifact.c. cbm_popen runs cmd.exe /c on Windows, which
does not honor single quotes, so a single-quoted ":(exclude)" pathspec made
the clean-tree probe report dirty and the marker was never written there.
The commit-existence probe uses `cat-file -t <oid>` rather than
`cat-file -e <oid>^{commit}` because ^ is cmd.exe's escape character.
Fail closed on allocation failure: both cbm_ht_create() results are checked,
and every insertion is verified to have landed (cbm_ht_set discards the
underlying insert's status, so OOM drops entries silently). A dropped entry
in the changed set would make a genuinely modified file read as unchanged
and be restamped — a silently stale graph. This is the one place where the
membership-set primitive has unsafe polarity; classify_files uses it where a
dropped entry only means "re-parse".
Symlinks are refused rather than followed when stamping: git tracks a
symlink's link text, not its target's content, so "unchanged" from git says
nothing about the bytes the indexer parses. Uses the established
lstat + S_ISLNK / reparse-point idiom (discover.c, pass_pkgmap.c,
pass_envscan.c), which on Windows also keeps non-ASCII paths off the ANSI
CRT.
The trust trade-off is documented at both the import site and in
artifact.h: foreign mtimes currently force a re-parse that auto-scrubs a
poisoned artifact at an unpredictable clone time, so today's exposure is
transient and self-healing. Restamping makes it durable and gates it on a
producer-written string — acceptable only because the marker alone never
suffices, as every restamped row must additionally be proven unchanged by
the local git against a commit present in this clone.
Also skip synthetic file_hashes rows. The pipeline stores semantic-input
digests under .codebase-memory/.semantic-input/ that have no file on disk;
stat()ing them made the export-side check fail unconditionally, so the
marker was never written on ANY platform and the whole feature was inert.
find_deleted_files skips the same class. Both halves of the trust check now
ignore the .codebase-memory/ subtree, matching the clean-tree pathspec.
Distilled from #868 by moofone; #869 and #870 are not included.
Co-authored-by: moofone <35233167+moofone@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
manage_adr mode='set_sections' rebuilt the ADR from cbm_adr_parse_sections()
and cbm_adr_render(). That model is lossy, so the rebuild silently rewrote
text nobody asked to change. Measured on the previous commit:
- Everything before the first recognised heading was dropped.
- A mis-cased '## Purpose' was not a heading, so when it came first its
whole block was dropped with the preamble — an ordinary typo cost the
author a section.
- Sections stored out of canonical order were reordered.
mode='update' never had this exposure because it writes the caller's bytes
verbatim; the section merge introduced it.
The merge now locates the target heading's byte span and replaces only that
span, appending a block when the heading is absent. Bytes outside the span
are never reconstructed, so a preamble, a code fence, an unrecognised
heading and the author's section order all survive byte-for-byte. That
property is the acceptance test: every document in
adr_splice_preserves_untouched_bytes is a case the rebuild corrupted.
Arbitrary headings now work, which is what an "add an entry" mode needs.
The canonical six become a convention rather than a privilege: they are
still what ADR_EMPTY_HINT suggests, but they get no ordering priority and
none is required. Under splicing, ordering is whatever the document says,
and any privileged reordering would reintroduce the data loss above.
Names match exactly, including case, so '## Purpose' and '## PURPOSE' are
different sections. Folding them would silently merge two blocks the
author chose to keep apart — the same class of loss this commit removes.
cbm_adr_validate_section_keys() no longer enforces the canonical set. It
now rejects only names that could not round-trip through a '## NAME' line
(empty, '#'-leading, newline-bearing, edge-whitespace, over-long), because
such a name scans back as a different heading or as none, and a second
identical write would append a duplicate instead of being a no-op.
A heading inside a fenced code block is a code sample, not a section. An
unterminated fence hides every heading after it, so a write would append a
duplicate rather than replace: that is refused explicitly, and mode='sections'
reports the same ambiguity instead of answering with a partial list.
manage_adr mode='sections' now uses cbm_adr_scan_headings(), the same
classifier the write path splices with. It used to list any '#'-prefixed
line, so a '## Foo' in prose or inside a fence was reported as a section no
write could target. Two components disagreeing about what a section is was
how a section write came to be able to destroy one.
cbm_adr_parse_sections() is unchanged, so documents parse exactly as before.
TEST(adr_parse_sections_non_canonical) still passes untouched.
Co-authored-by: andis777 <24672074+andis777@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
manage_adr could only replace: mode='update' overwrites the stored
document in full, so adding one entry costs a re-send of the whole ADR.
That is a data-integrity problem before it is a cost one — the caller has
to reproduce every byte it did not intend to change, so the unchanged
prefix survives only as well as the round-trip that carried it.
mode='set_sections' rewrites only the sections named in `section_updates`
and leaves the rest of the stored document untouched, so the stored copy
stays the authority for everything the caller did not name. Unlike a
whole-document append it is idempotent: a client that loses a response
and retries re-sets the same section to the same body and the document is
byte-identical, where an append would silently duplicate the chunk.
Only the six canonical section names are writable. That is a correctness
constraint rather than a style rule: adr_try_section_header() parses ONLY
canonical headers, so a non-canonical '## FOO' written here would be read
back as body text of the section above it, and a second identical write
would append a duplicate — destroying the idempotence the mode exists for.
Three things the new mode needed that were not there:
- cbm_store_adr_update_sections() now wraps its read-modify-write in
BEGIN IMMEDIATE. Three writers replace this row wholesale — the indexing
pipeline, the UI POST /api/adr handler, and mode='update' — so the
unguarded get/merge/store lost whichever of them committed between the
read and the UPSERT. mode='update' is a single atomic UPSERT and never
had that window; a section merge introduces it.
- set_sections joins the write_request classification. A mode missing from
it takes no per-project mutation lease, resolves the store query-only,
and never reaches open_adr_store_for_write — its write would be
attempted through a read-only handle while an index runs.
- The write path reads the legacy <root>/.codebase-memory/adr.md itself.
The existing migration runs on the read path only because it must not
block on the lease; without this a section write would merge onto an
empty document and discard an ADR still present on disk.
CBM_ADR_MAX_LENGTH now applies to an MCP write path: it is enforced inside
cbm_store_adr_update_sections, which mode='update' does not go through. An
empty section body, an unknown section name and a missing section_updates
are all rejected before any store is opened, so a caller that meant to
write never receives a success-shaped read.
Section-level update was chosen over the whole-document append proposed in
PR #1243; the analysis that established the problem is from that PR.
Co-authored-by: andis777 <24672074+andis777@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Python attribute calls were not classified as member calls, so weak
short-name strategies (suffix_match / unique_name / field_type_hint)
could bind `obj.method()` to an arbitrary same-named project symbol —
`accelerator.print()` resolving to MockAccelerator.print.
Flag a Python attribute call is_method when its receiver is unresolved,
and extend the existing TS/JS weak-member guard to Python. Receivers that
ARE known stay exempt: self/cls, super(), and any identifier bound by one
of the file's imports (including the root of an attribute chain), so
Python's canonical `module.function()` cross-file call still resolves.
The guard helper is renamed cbm_tsjs_suppress_weak_method_match ->
cbm_suppress_weak_member_match. Its drop-list is unchanged, and the
per-language gate stays at the two call sites, which enumerate the same
language set in pass_calls.c and pass_parallel.c so the sequential and
parallel resolvers cannot diverge. ArkTS remains in that set.
python/S6/inherited_method in the resolution probe now asserts calls == 0,
recording the same trade TypeScript took when the receiver-aware guard
landed: the edge it used to produce came from a unique_name registry
fallback (measured on main: strategy=unique_name, cands=1, conf=0.7500)
where a weak short-name guess happened to be right in a 2-file fixture.
It flips back to >= 1 once py_lsp_cross resolves inheritance, matching the
flip-back note already carried on the TypeScript S6 case.
Fixes#1276
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: Enferlain <15861396+Enferlain@users.noreply.github.com>
Closes#518 (Section nodes don't index body text) and #519 (META.yaml /
frontmatter description values not indexed). A question asked in words
could not reach the nodes that hold the words: Section carried only its
heading, a config Module only its path, and both labels were filtered out
of BM25 results outright.
nodes_fts gains a fifth column, `body`, backfilled from each node's
EXISTING properties.docstring. No new column on `nodes`, no new table, and
CBM_INDEX_FORMAT_VERSION deliberately does NOT move -- nobody is forced to
reindex. A legacy four-column table survives untouched under CREATE VIRTUAL
TABLE IF NOT EXISTS: the backfill probes for the column before naming it,
and bm25()'s surplus fifth weight is inert on a table that has no fifth
column (FTS5 reads a weight only for a column an instance landed in). Such
a database keeps opening and searching, just without prose.
Two node kinds gain the docstring they lacked:
- Markdown section bodies (#518) -- the blocks following a heading, up to
the next heading or subsection, collapsed to single spaces and capped at
MAX_COMMENT_LEN (500 B) with a UTF-8 backoff that never splits a
multi-byte sequence.
- Top-level YAML/JSON description / summary / purpose (#519), promoted
onto the file's Module node.
Section and Module are no longer excluded from BM25 results. Both the
ranked query and the mirrored count query were changed together; removing
the filter from only one desynchronises results from counts. Column weights
are bm25(nodes_fts, 1.0, 1.0, 1.0, 1.0, 0.3): the four identifier columns
keep parity with each other and prose sits well below them, so a prose-only
hit surfaces but never outranks an identifier match.
The FTS backfill is consolidated into one writer, cbm_store_fts_rebuild(),
used by the wholesale rebuild AND the delta merge. That second site matters:
it had its own hand-written four-column INSERT, which with a fifth column
would leave body NULL for every node arriving by delta -- silently, on the
warm path most users hit, while a full reindex looked perfect. There is now
exactly one place the column list is decided, and a test that fails if the
delta path stops writing prose.
The backfill guards json_extract with json_valid: malformed properties JSON
exists in pre-fix databases and would otherwise abort the whole rebuild.
Tests: markdown section bodies (own body, stops at the next heading, UTF-8
safe cap, empty body stays NULL), YAML/JSON description promotion (block
scalars, synonym fallback, top-level only, config-only), FTS body indexing
and its column isolation, malformed-JSON tolerance, legacy four-column
degrade, incremental watermark, BM25 finding Section and Module by prose,
results/total consistency, identifier-outranks-prose, and the delta-merge
regression.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: ShauryaaSharma <196896785+ShauryaaSharma@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Chialisp (.clsp/.clib/.clinc) is the Chia smart-coin language. The only
public grammar, Quexington/tree-sitter-chialisp, cannot parse it: on the five
reference files from chia-blockchain@main it produced 3-34 ERROR nodes each.
Three defects account for all of them, and each desynchronised the rest of the
file rather than losing one node:
* comments were `/;.*\r\n/` -- CRLF-only, while real files are LF;
* `(include foo.clib)` was rejected because of the dot in the filename;
* `(defconstant NAME <expr>)` accepted only a primitive after the name.
So the grammar is written here instead, in tools/tree-sitter-chialisp/, and
files as first-party: its vendored LICENSE is the project's own, byte-identical
to the repository root, carrying no third-party copyright.
It is deliberately GENERIC -- source_file / list / symbol / string / number /
hex / dot / comment, modelling the clvm_tools reader rather than the Chialisp
form vocabulary. `mod`, `defun`, `defconstant` and `include` are ordinary head
symbols; which lists are definitions is decided in extract_defs.c. A dialect
that adds a form (cl-21 -> cl-23 -> cl-26 each added several) therefore needs
no regenerated parser. Acceptance: all five reference files parse with zero
ERROR and zero MISSING nodes; the three defects above are pinned as tests.
Extraction, and the four decisions in it:
* `Constant` is admitted to the cross-file name registry
(cbm_label_is_registry_symbol). Chialisp's condition codes live as
`(defconstant ...)` in a shared .clib and are referenced by name from every
puzzle; labelled but unregistered they are seeded nowhere, and roughly half
of a Chialisp graph would be present as nodes yet never the target of an
edge. All three registry-seeding sites route through that one predicate, so
they cannot diverge. No other language emits `Constant`, so the blast radius
is exactly Chialisp. Pinned by a two-file graph test that goes red when the
label is removed from the predicate.
* `export` / `namespace` are NOT definition heads. `(export foo)` names a
function already defined in the same file, so treating it as a def head
mints a second node for one symbol. They stay in the not-a-call filter.
* A parameter list is a `list` too, but it binds rather than invokes. Without
that gate `(defun check_conditions (HEIGHTLOCK conditions) ...)` mints a
phantom call to HEIGHTLOCK, and real Chialisp is mostly such definitions.
* `(include *standard-cl-26*)` is a dialect directive, not a file dependency;
recording it would invent a dependency on a file that does not exist.
Three helpers that the defs, calls and unified walks all need -- the quote-data
gate, comment-skipping named-child access, and the Chialisp def-head set -- live
in helpers.c rather than being copied per translation unit. They previously
existed as byte-identical duplicates that had already drifted: extract_defs.c
skipped comments when reading a def head while its mirror in extract_unified.c
did not, so a comment between a head and its name desynchronised defs from call
scope. Both now use the same accessor.
Ledger: 161 -> 162 grammars across the seven published surfaces, ABI-14 86 -> 87,
first-party 13 -> 14, CBM_LANG_COUNT 164 -> 165.
Distilled from PR #919 by darkverbito, whose extraction logic -- the CLVM
not-a-call vocabulary, the quoted-data gate, the filename-derived `mod` name and
the include/embed import handling -- is kept and is the substance of this change.
The grammar that PR vendored is not used.
Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The closure-delta incremental route scored importance off a graph that does
not contain the project's edges, and persisted the result.
cbm_delta_preseed fills that route's gbuf with PROXY nodes -- id, label, name,
qualified name, file path, and nothing else. Only the re-extracted files carry
real edges; every inbound edge from an unchanged file is snapshotted and
re-linked later, inside cbm_delta_patch. Scoring in run_postpasses therefore
read an in-degree of zero for symbols with hundreds of real callers, and
because the repaired files' rows are purged and re-inserted, cbm_delta_patch
wrote those zeros to disk. Measured on a fixture whose `helper` has 25
callers: full index 5.000000, after one closure-delta re-index 0.000000.
closure_try_plan runs first, so this is the preferred warm path, not an edge
case. The comment at the call site claimed a rehydrated whole-project buffer;
that is true only of the legacy-partial route, and it is now stated correctly.
The fix rescores in SQL over the staging store, immediately after
cbm_delta_patch has inserted the new nodes and re-linked the inbound edges --
the first moment on this route where the complete graph is queryable. Loading
the full graph into RAM instead would have given back the point of the delta
route.
Rescoring is project-wide, not limited to the changed files. Distinct-files-
per-name and in-degree are both GLOBAL inputs: a body-only edit in one file
can change the score of a symbol defined in a file that did not change. A
scoped rescore would leave those stale.
TWO ROUTES, ONE RULE. The scoring rule and the JSON write-back are not
reimplemented in SQL. cbm_pipeline_importance_score() and
cbm_pipeline_importance_set_prop() are now single definitions that both routes
call -- the SQL side reaches them through two SQLite user-defined functions,
so the same C code runs either way and the rule cannot drift. SQL contributes
only the three aggregates, each an indexed GROUP BY. What remains free is the
input GATHERING, and tests bind that directly: a full index and a closure-delta
incremental index converging on the same tree must produce identical scores for
every symbol. Both new tests assert the route was actually taken, so neither
can pass by silently running some other path.
Cost, measured on a 666 MB Java corpus: cold full index unchanged at 83 s
(the recompute does not run on that path); warm closure-delta re-index 20 s
without the recompute, 26 s with it. The delta route keeps its advantage over
a full reindex (26 s vs 83 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Score every Function/Method/Class node during indexing and persist the
result as a numeric "importance" key inside the node's existing
properties_json:
importance = sqrt(num_refs) * priv * generic * distinct * test_penalty
num_refs is the incoming CALLS + USAGE degree. The multipliers demote
private (leading-underscore) and generic names (defined in >= 5 distinct
files), promote distinctive identifiers (snake_case or camelCase, len >=
8), and demote test scaffolding -- symbols in a test file per the graph's
own cbm_is_test_path() classifier, or targets of an incoming TESTS edge.
Weighted degree only; a transitive (PageRank) refinement is deliberately
not built here until it can be shown to beat this on a fixed judgment
set.
No schema change and no CBM_INDEX_FORMAT_VERSION bump: the score lives in
the existing properties_json TEXT column, so no index is forced to
rebuild and indexes written by older builds simply lack the key.
Three failure modes this shape invites, all of them silent, are addressed
directly:
* Cost. The generic-name multiplier needs |{files a name is defined in}|.
Computing that per NODE with a pairwise file-path scan is O(k^3) in a
same-name group of size k -- ruinous on real corpora, where a single
Java name group reaches k > 4000. Here the count is computed once per
distinct NAME and memoized, and file paths are deduplicated through a
hash set, so total distinct-file work is linear in the graph. Measured
cold full index of a 666 MB Java corpus, same -O2 binary: 85 s with the
pass disabled, 80 s with this implementation, 481 s with the per-node
shape.
* Registration. PREDUMP_PASS_COUNT is now derived from the pass table
with sizeof instead of being hand-written. A hand-written count that
lags an appended entry silently skips the last-registered pass with
every test still green.
* Incremental. The incremental post-pass sequence runs the pass too, over
a graph rehydrated from the store whose nodes already carry the key.
The write-back overwrites an existing key in place instead of appending
a second one, which would otherwise corrupt properties_json into
{"importance":1.0,...,"importance":2.0}.
tests/test_importance.c covers the formula and binds all three failure
modes; the complexity suite gains a GATED counter asserting the pass's
same-name-group work stays linear (~2x, not ~4x) under a corpus doubling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: petercoxphoto <20230727+petercoxphoto@users.noreply.github.com>
Maintainer mechanics on top of the accepted feature: merge current main and
format the 24 violations in the edited region of client_adapter.c — the
only red CI leg. No content change; the generated plugin is byte-identical
apart from whitespace in the C literals' layout.
Co-authored-by: jc <jcvi007@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #1412 by @allanoepping. Fixes a real recall hole: tag-dialect
.cfc components were routed to the cfscript grammar (extracting nothing),
and <cfscript> bodies inside tag-dialect files are opaque cf_script_content
to the cfml grammar — whole legacy components produced zero functions.
Fix 1 — his .cfc dialect sniffer, nearly verbatim: cbm_disambiguate_cfc()
reads a 16K head via cbm_fopen; <cfcomponent>/<cffunction> anywhere means
tag dialect; a leading <cfscript (after whitespace and <!--- ---> comments)
or no markup at all keeps the old CFSCRIPT routing, so script-dialect files
and read failures behave exactly as before. Wired in detect_file_language
beside the existing _m/_cls/_inc sniffers, with his 8 direct tests.
Fix 2 — his embedded-defs capability, re-based onto the shared machinery
that superseded his mechanism mid-flight (#1852, merged after his last
rebase): a cfml_embedded_imports spec row (cf_script_tag/cf_script_content
-> CFSCRIPT), CFML added to the embedded-imports dispatch, and the
extract_structure gate extended to CFML. Included ranges on the original
buffer give absolute coordinates for free, so his line-number assertions
hold verbatim, and calls inside blocks become extractable as a bonus. His
~90-line slice+shift implementation is deliberately NOT carried — it
mirrored the sanctioned shape of its day, which #1852 replaced.
One correctness guard the port needed: the Vue attribute resolver is now
gated to Vue — cf_script_tag carries no lang=/src= attributes, and running
the resolver would have silently rewritten CFML's embedded language from
CFSCRIPT to the resolver's JavaScript default.
His embedded extraction test (line remap + <cfsetting> ERROR-cascade case)
carries over verbatim and binds: reverting the spec/gate wiring fails it at
ASSERT(has_def(r, "Function", "greet")). Suites: extraction, language,
edge_imports, lang_contract, grammar_labels, grammar_regression — 637/0.
Co-authored-by: Allan Oepping <allan@intellipay.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Maintainer commit on top of the Bash-matcher work, closing the cost
question the 2026-08-03 review left open before default-on shipped.
Measured on a production binary (2026-08-28): every hook invocation paid
~1.1-1.2 s of user CPU BEFORE any gating — not in cbm_mcp_server_new as
first suspected, but in main()'s executable-identity hash for cohort
admission, which the hook-client path runs ahead of any input inspection.
With Bash in the installed matcher, nearly every agent command (`ls`,
`make`, `git`, ...) would have paid that for nothing: agents issue far
more Bash calls than Grep/Glob, and almost none are searches.
The fix: main() now reads stdin on the hook-client path and exits before
identity hashing when the event can never produce output — an un-forced
PreToolUse Bash event whose command yields no search pattern or no
queryable token. A no-op touches no shared state, so it needs no
identity. The gate is a pure exported function using the exact parser
pair ha_process uses, so gate and augmenter cannot disagree; anything
uncertain (lifecycle events, other tools, coverage adapters, oversized
stdin, forced events) pays full fare unchanged. Consumed stdin is handed
back through a prefetch so downstream readers see it exactly once.
Measured after: no-op Bash calls ~10 ms wall / 0.00 s user CPU (was
~1.2 s); real Bash searches and Grep/Glob unchanged at full fare.
cli 259/0, mcp 179/0.
Co-authored-by: Jake Abendroth <contact@jakea.net>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #1298 by @aoright — the subset that survived review and
main's drift, per the 2026-08-17 ruling.
- pass_lsp_cross.h: the header still documented and declared
all_defs/imp_keys/imp_vals while the implementation renamed them to
defs/imp_names/imp_qns — doc-only drift, now synced; plus the braced
guard in cbm_pxc_registry_for_lang.
- runtime.h: semantic_version[12] gains a named size
(CBM_DAEMON_SEMVER_SIZE), the "defensible: a real size" case from the
review.
- httpd.h: request-field buffers take the CBM_SZ_* names for their actual
sizes; the constants.h include sits in the top include block rather than
mid-file.
Deliberately NOT carried, per the same ruling: the store.c calloc hunk
(the claimed uninitialized read does not exist), the generic-constant
renames (semantic laundering), the watcher.c split (its target function
gained a semantic fix on 2026-08-20 and the refactor must be re-derived),
and the lsp_resolve.h extractions (their target loop was rebuilt on main).
Co-authored-by: aoright <102943475+aoright@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel