The READS/WRITES resolvers and the USAGE registry fallback hand bare
reference text to the short-name registry, which contains Field nodes -
so once Go struct fields exist (#1935), every Go local err := ... binds
whichever struct field is named err, project-wide: 21308 USAGE and 5191
WRITES onto Go fields on the measured repo, top target a test struct's
field T collecting 3013 edges. In Go that binding is impossible by
construction: a field is only reachable through a selector expression
(x.f), and selector references resolve on the LSP path - every
Field-targeted reference edge in the census carried dot-less text.
Add cbm_go_suppress_bare_field_ref() next to the #1928 predicate and
consult it at the same four sites (both READS/WRITES resolvers, both
USAGE registry fallbacks): drop the bind when the file is Go, the
target label is Field, and the reference text has no '.'. Go-gated
because C#/Java/C++/Python method bodies legitimately reference their
own members bare (cp_reads_writes_cs_static_field pins that shape).
Field-validated on the #1940 stack: USAGE onto Go fields 21308 -> 0,
WRITES 5191 -> 0; the only remaining field-targeted edges are 2466
CALLS, which are #1906/#1907's selector-guard territory. Reproduce-
first pipeline probes (sequential + parallel twins) were RED on the
stack without this commit.
Fixes#1942
Signed-off-by: Ilya Brykau <ilya.brykau@orca.security>
Reproduce-first probe for the two fixes on this branch: RED without the
descend fix (count_defs_with_label(r, "Field") == 0, expected 3), GREEN
with it. Asserts the three named fields with their declared types in
return_type, the absence of the blank identifier, and that interface
members keep extracting exactly as before.
Signed-off-by: Ilya Brykau <ilya.brykau@orca.security>
USAGE, WRITES and READS edges resolve through the same short-name
registry as CALLS but never consulted the #725 cross-language guard.
On a Go tree with eBPF C probes every Go identifier spelled like a C
one produced a reference edge into the C file: 31.5% of all WRITES on
the originally measured repo crossed the Go->C boundary, led by dozens
of Go test locals named event writing a C probe's automatic variable.
Add cbm_suppress_cross_language_ref() - the reference-edge analog of
cbm_suppress_cross_language_suffix_match - and consult it on BOTH
resolvers of each edge type: the sequential pass (pass_usages.c
resolve_usage_edges registry-fallback branch, resolve_rw_edges) and
their parallel twins (pass_parallel.c resolve_file_usages,
resolve_file_rw). The sequential-only version of this change left 344
Go->C WRITES alive on a ~1150-file repo because large repos resolve
through pass_parallel.c - the field census caught it, and the
parallel-twin test now pins it.
Unlike the CALLS guard the predicate takes no strategy parameter: a
reference edge carries no import-closure evidence, so every registry
strategy is a bare-name guess across a boundary. LSP-backed semantic
references resolve before the fallback and are unaffected. JS/TS stay
one family, and C/C++ count as one family too (.h maps to
CBM_LANG_CPP, so a .c file referencing its own header is not a
boundary).
Field-validated on the ~1150-file Go+C repo: Go->C/C++ WRITES 835->0,
USAGE 1545->0; C->Go 15/90->0; 6432 reference edges dropped in total,
every one cross-language (Go->.json 3034, Go->.hpp/.h 1612, Go->.c
746, Go->.sh/.yaml/.yml 443, ...) and none same-language. CALLS and
IMPORTS totals are byte-identical to main.
Fixes#1928
Signed-off-by: Ilya Brykau <ilya.brykau@orca.security>
The reconcile_basis marker was never written on Windows, so all six
build_trusted_artifact_repo() tests plus artifact_export_marks_clean_basis
failed there (16 passed, 7 failed) while every Linux and macOS leg was green.
Root cause is a stamp-encoding mismatch, not shell quoting. file_hashes rows
are stamped by the semantic manifest via cbm_path_info_utf8(), which on Windows
derives mtime_ns from ftLastWriteTime -- 100-nanosecond FILETIME ticks.
db_hashes_match_disk() re-stat()ed those same files through _wstat64, whose
st_mtime carries WHOLE SECONDS, and compared the two. The values can only agree
when a file's write time lands exactly on a second boundary, so the trust check
rejected the very first row on every export, on Windows and only on Windows.
On POSIX both paths read st_mtim nanoseconds and agree, which is why the split
was clean along platform lines.
The disk side now reads through cbm_path_info_utf8 -- the same function that
wrote the rows -- so there is no encoding left to keep in sync. Its
is_regular/is_symlink guard preserves exactly the symlink refusal
reconcile_stat_no_symlink provided.
reconcile_restamp_rows deliberately keeps art_stat_mtime_ns: a restamped row is
read by the incremental classifier and by check_index_coverage, both of which
stat() and encode whole seconds on Windows. The asymmetry is now pinned with a
comment at both sites so neither is "unified" into breaking the other.
Diagnostics, which is why this took two wrong diagnoses to find:
- Export's four clean-basis preconditions were one short-circuiting && chain
collapsing to a single bool. They are now evaluated by reconcile_basis_trusted,
which records the first failing gate (head_unresolved / head_not_hex_oid /
tree_not_clean / db_hashes_differ_from_disk, the last carrying the offending
path and BOTH stamps) and logs it as artifact.reconcile_basis_omitted. The
reason is recorded rather than recomputed on demand: export's own
ensure_gitattributes leaves an untracked .gitattributes behind, so any later
evaluation answers tree_not_clean whatever the real cause was.
- build_trusted_artifact_repo() had six `return NULL` exits and reported none of
them; CI could not tell a failed `git commit` from a missing marker. Every exit
now names its step and carries the number that decided it -- git exit status,
pipeline rc, or export's recorded blocker -- surfaced through
BUILD_TRUSTED_REPO_OR_FAIL at the call site so file:line still points at the
failing test.
Verified locally on macOS by emulating the Windows encoding: truncating the
disk-side mtime to whole seconds reproduces the CI signature exactly (16 passed,
7 failed, same seven sites), and the new diagnostic names it as
`db_hashes_differ_from_disk path=a.rs reason=mtime_differs stored_mtime_ns=...
disk_mtime_ns=...`. Restored, the artifact/pipeline/incremental/mcp/index_format/
daemon_bootstrap suites are green and make lint-ci passes.
Co-authored-by: moofone <35233167+moofone@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
clone_to_b() cloned the source repo but never configured user.email or
user.name on the result. A clone does not inherit the source repo's LOCAL
config, and CI runners have no global identity, so every commit made in the
clone exited 128 -- surfacing as:
FAIL tests/test_artifact.c:734:
runf("git -C ... add -A && git -C ... commit -qm edits") == 32768, expected 0
32768 is 128 << 8. It passed locally only because developer machines carry a
global git identity; the Linux CI shards do not.
git_init() already configures identity for the source repo (:574-575); the
clone needs the same treatment. This is the defect originally diagnosed on
#868 as "git identity in clone_to_b" -- correct for the POSIX legs, and
separate from the cmd.exe quoting bug that caused the Windows failure.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every ADR fixture in the tree used \n, so nothing exercised a CRLF-stored
document — and CI could not have found this, because the fixtures are LF
whichever platform runs them. A red would only have appeared once a real
user's CRLF ADR met this code.
The exposure is specific to splicing. The old rebuild normalised everything
on the way out, so line-ending confusion was invisible; a byte-span
replacement can miscount a separator run, and byte-identity is the property
the splice design exists to provide.
Two of the three paths were already correct, and now say so in tests:
- Replacing a section preserves CRLF exactly. The walk-back over a trailing
break run steps across both '\r' and '\n', so it cannot cut mid-pair.
- Heading matching is ending-agnostic: "## PURPOSE\r\n" locates the same as
"## PURPOSE\n", because the scanner trims '\r' from the name. Trimming and
matching were different claims and only trimming was pinned.
Appending a new section was wrong. It counted trailing '\n' characters, so a
document ending "...Foo\r\n" was read as having one break but then had a
BARE '\n' appended, and the "## NAME" line it wrote used '\n' too — leaving a
mixed document that neither the author nor the tool asked for. A CRLF file
ending in a blank line also gained a spurious extra break.
Appends now count a "\r\n" pair as one break, and write the separator and
heading line using the document's own ending.
The contract, now that a document can be mixed:
- Bytes that already exist are never rewritten, so each line keeps whatever
ending it had. This follows from splicing and is what the acceptance
property already guaranteed.
- Text this code writes itself — the separator and the "## NAME" line —
follows the document's majority ending, LF on a tie.
- A caller's body is inserted verbatim. Rewriting bytes a caller supplied
would be the same silent modification this whole change set removes.
Co-authored-by: andis777 <24672074+andis777@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`CBMDefinition.base_classes` carries the SOURCE SPELLING of each base
("Base", "django.db.Model"): extraction strips keywords and generic
arguments, but it cannot know where the name is declared. The Python and
TS cross-file registrars, however, consume `CBMLSPDef.embedded_types` as
fully-qualified names — py_lookup_attribute and ts_lookup_member feed each
entry straight into cbm_registry_lookup_type.
An unqualified spelling therefore matched nothing declared in ANOTHER
file. `class Child(Base)` in child.py never saw Base in base.py, so a
call to an inherited method through a typed receiver found no member and
fell through to the weak textual cascade, where the receiver-aware guard
(#592/#606) correctly kills it. The member-lookup walk over
embedded_types was already there in both languages; only the names it
walked were unusable across files.
cbm_pxc_collect_all_defs now resolves each base spelling to a project QN
using exactly the inputs pass_semantic uses to draw its INHERITS edge:
the project registry, the declaring module, and the file's import map.
Two properties follow. The LSP's inheritance view is the same relation
the graph records, so the two cannot diverge. And the binding is import-
or same-module-backed rather than a short-name guess, so the CALLS edge
it enables is a supported fact that the weak-member guard keeps.
Weak registry strategies (suffix_match / unique_name / field_type_hint /
fuzzy) are rejected for bases via an explicit drop-list: a base bound
because some project type happens to share its name is precisely the
fabricated relation #606 removed, and inheritance multiplies it — every
inherited member of the wrong base would become a callable target. An
unresolved base keeps its raw spelling and the behaviour that predates
this change, so stdlib and third-party bases are unaffected.
Scoped to the languages whose registrars read embedded_types as QNs
(Python, JS/TS/TSX). Go, JVM, C#, C++ and Rust already qualify their own
embedded types and keep the raw join.
Cost is O(defs) hash lookups plus ONE import map per file: no per-call-
site hierarchy walk, no registry scan, no per-file registry rebuild. The
complexity guard stays linear (nodes 1.93, edges 2.07, per-file defs
2.00 against a files x corpus coupling of ~4).
Tests:
- ts/S6 flips to its own documented condition: was `calls == 0` as a
tripwire recording the gap, now asserts calls >= 1 AND an lsp_ts_*
strategy. Measured: strategy=lsp_ts_method, confidence 0.95.
- python/S6b is new and covers what python/S6 structurally cannot: S6's
`def run(c)` parameter is un-annotated, so no receiver type exists to
inherit through and any binding there is a guess about a name (it
resolves via unique_name at 0.75). S6b supplies the receiver three ways
— annotated parameter, `self` in the subclass, constructor result — and
asserts INHERITS >= 1 plus an lsp_* strategy. Revert-check: all three
fall back to unique_name at 0.75 without this change, lsp_method at
0.90 with it.
- Both rows assert the STRATEGY, not just the edge count: a short-name
guess and a resolved inherited member produce the same count in a
two-file fixture, and only the strategy tells them apart.
- Corrects python/S6's stale comment citing a base_classes extraction
bug; extraction_inheritance is green for Python and TS, and
base_classes now holds clean names.
macOS full suite 7690 passed / 0 failed / 8 skipped (140 suites);
make lint-ci clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An imported artifact carries the EXPORTER's mtime/size in every file_hashes
row, so the first incremental run after a bootstrap sees every file as
changed and re-parses ~the whole repository — most of the time the artifact
was supposed to save.
Reconcile the rows instead. After a successful import, ask the local git
which files are unchanged between the artifact's commit and the working
tree, and re-stamp exactly those rows with local stat() values. Everything
git cannot vouch for stays foreign and is re-parsed.
Export writes an optional "reconcile_basis":"git-clean-head" marker into the
existing artifact.json, and only when the tree is provably clean outside
.codebase-memory/ AND every hash row still matches disk. Unknown fields are
ignored by older binaries, so no schema_version bump.
Eligibility is deliberately narrow, and the polarity is skip-on-doubt:
a row is restamped only if it is tracked at the artifact commit (git
diff/ls-files are blind to ignored files — a .cbmignore negation can index
a gitignored file, #500) and absent from the changed set. Any uncertainty —
no git, missing/non-hex commit, shallow clone, truncated -z output,
allocation failure — skips reconciliation entirely and falls back to
today's behavior. Reconciliation can never fail the import.
Windows correctness: every git argument uses DOUBLE quotes, per the rule
already stated in artifact.c. cbm_popen runs cmd.exe /c on Windows, which
does not honor single quotes, so a single-quoted ":(exclude)" pathspec made
the clean-tree probe report dirty and the marker was never written there.
The commit-existence probe uses `cat-file -t <oid>` rather than
`cat-file -e <oid>^{commit}` because ^ is cmd.exe's escape character.
Fail closed on allocation failure: both cbm_ht_create() results are checked,
and every insertion is verified to have landed (cbm_ht_set discards the
underlying insert's status, so OOM drops entries silently). A dropped entry
in the changed set would make a genuinely modified file read as unchanged
and be restamped — a silently stale graph. This is the one place where the
membership-set primitive has unsafe polarity; classify_files uses it where a
dropped entry only means "re-parse".
Symlinks are refused rather than followed when stamping: git tracks a
symlink's link text, not its target's content, so "unchanged" from git says
nothing about the bytes the indexer parses. Uses the established
lstat + S_ISLNK / reparse-point idiom (discover.c, pass_pkgmap.c,
pass_envscan.c), which on Windows also keeps non-ASCII paths off the ANSI
CRT.
The trust trade-off is documented at both the import site and in
artifact.h: foreign mtimes currently force a re-parse that auto-scrubs a
poisoned artifact at an unpredictable clone time, so today's exposure is
transient and self-healing. Restamping makes it durable and gates it on a
producer-written string — acceptable only because the marker alone never
suffices, as every restamped row must additionally be proven unchanged by
the local git against a commit present in this clone.
Also skip synthetic file_hashes rows. The pipeline stores semantic-input
digests under .codebase-memory/.semantic-input/ that have no file on disk;
stat()ing them made the export-side check fail unconditionally, so the
marker was never written on ANY platform and the whole feature was inert.
find_deleted_files skips the same class. Both halves of the trust check now
ignore the .codebase-memory/ subtree, matching the clean-tree pathspec.
Distilled from #868 by moofone; #869 and #870 are not included.
Co-authored-by: moofone <35233167+moofone@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
parallel_lsp_index_exact_ambiguity_does_not_fall_through_to_legacy
asserted alpha_edges == 1, which the Python weak-member guard suppresses.
That assertion is a side effect, not the test's subject. Its subject —
that an ambiguous exact LSP match must not fall through to the
lower-ranked legacy semantic row — is unchanged: beta_edges == 0,
legacy_edges == 0, and all four probe assertions (ambiguity.injected,
legacy_injected, carrier_allows_fallback, shared_matcher_failed_closed)
still hold. Verified before changing the number, not after.
The harness deliberately injects an ambiguous exact match and forces the
shared matcher to fail closed, so value.render() drops past the LSP into
the registry with three same-named candidates (Alpha/Beta/Legacy.render)
and bound by suffix_match -- measured: strategy=suffix_match, cands=3,
conf=0.5500. That is a 1-in-3 guess that happened to land on Alpha:
exactly the weak member-call class this change suppresses, and the same
accepted trade recorded on python/S6 in test_lsp_resolution_probe.c.
Not over-suppression: sibling tests in this suite index the same
value.render() source and resolve it via lsp_method (cands=1,
conf=0.9000), a strategy the guard keeps -- they still assert 1. Only the
sabotaged-LSP path degrades to a weak textual guess. Flips back to 1 once
py_lsp_cross resolves the receiver on this path.
Refs #1276
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
manage_adr mode='set_sections' rebuilt the ADR from cbm_adr_parse_sections()
and cbm_adr_render(). That model is lossy, so the rebuild silently rewrote
text nobody asked to change. Measured on the previous commit:
- Everything before the first recognised heading was dropped.
- A mis-cased '## Purpose' was not a heading, so when it came first its
whole block was dropped with the preamble — an ordinary typo cost the
author a section.
- Sections stored out of canonical order were reordered.
mode='update' never had this exposure because it writes the caller's bytes
verbatim; the section merge introduced it.
The merge now locates the target heading's byte span and replaces only that
span, appending a block when the heading is absent. Bytes outside the span
are never reconstructed, so a preamble, a code fence, an unrecognised
heading and the author's section order all survive byte-for-byte. That
property is the acceptance test: every document in
adr_splice_preserves_untouched_bytes is a case the rebuild corrupted.
Arbitrary headings now work, which is what an "add an entry" mode needs.
The canonical six become a convention rather than a privilege: they are
still what ADR_EMPTY_HINT suggests, but they get no ordering priority and
none is required. Under splicing, ordering is whatever the document says,
and any privileged reordering would reintroduce the data loss above.
Names match exactly, including case, so '## Purpose' and '## PURPOSE' are
different sections. Folding them would silently merge two blocks the
author chose to keep apart — the same class of loss this commit removes.
cbm_adr_validate_section_keys() no longer enforces the canonical set. It
now rejects only names that could not round-trip through a '## NAME' line
(empty, '#'-leading, newline-bearing, edge-whitespace, over-long), because
such a name scans back as a different heading or as none, and a second
identical write would append a duplicate instead of being a no-op.
A heading inside a fenced code block is a code sample, not a section. An
unterminated fence hides every heading after it, so a write would append a
duplicate rather than replace: that is refused explicitly, and mode='sections'
reports the same ambiguity instead of answering with a partial list.
manage_adr mode='sections' now uses cbm_adr_scan_headings(), the same
classifier the write path splices with. It used to list any '#'-prefixed
line, so a '## Foo' in prose or inside a fence was reported as a section no
write could target. Two components disagreeing about what a section is was
how a section write came to be able to destroy one.
cbm_adr_parse_sections() is unchanged, so documents parse exactly as before.
TEST(adr_parse_sections_non_canonical) still passes untouched.
Co-authored-by: andis777 <24672074+andis777@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
manage_adr could only replace: mode='update' overwrites the stored
document in full, so adding one entry costs a re-send of the whole ADR.
That is a data-integrity problem before it is a cost one — the caller has
to reproduce every byte it did not intend to change, so the unchanged
prefix survives only as well as the round-trip that carried it.
mode='set_sections' rewrites only the sections named in `section_updates`
and leaves the rest of the stored document untouched, so the stored copy
stays the authority for everything the caller did not name. Unlike a
whole-document append it is idempotent: a client that loses a response
and retries re-sets the same section to the same body and the document is
byte-identical, where an append would silently duplicate the chunk.
Only the six canonical section names are writable. That is a correctness
constraint rather than a style rule: adr_try_section_header() parses ONLY
canonical headers, so a non-canonical '## FOO' written here would be read
back as body text of the section above it, and a second identical write
would append a duplicate — destroying the idempotence the mode exists for.
Three things the new mode needed that were not there:
- cbm_store_adr_update_sections() now wraps its read-modify-write in
BEGIN IMMEDIATE. Three writers replace this row wholesale — the indexing
pipeline, the UI POST /api/adr handler, and mode='update' — so the
unguarded get/merge/store lost whichever of them committed between the
read and the UPSERT. mode='update' is a single atomic UPSERT and never
had that window; a section merge introduces it.
- set_sections joins the write_request classification. A mode missing from
it takes no per-project mutation lease, resolves the store query-only,
and never reaches open_adr_store_for_write — its write would be
attempted through a read-only handle while an index runs.
- The write path reads the legacy <root>/.codebase-memory/adr.md itself.
The existing migration runs on the read path only because it must not
block on the lease; without this a section write would merge onto an
empty document and discard an ADR still present on disk.
CBM_ADR_MAX_LENGTH now applies to an MCP write path: it is enforced inside
cbm_store_adr_update_sections, which mode='update' does not go through. An
empty section body, an unknown section name and a missing section_updates
are all rejected before any store is opened, so a caller that meant to
write never receives a success-shaped read.
Section-level update was chosen over the whole-document append proposed in
PR #1243; the analysis that established the problem is from that PR.
Co-authored-by: andis777 <24672074+andis777@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Python attribute calls were not classified as member calls, so weak
short-name strategies (suffix_match / unique_name / field_type_hint)
could bind `obj.method()` to an arbitrary same-named project symbol —
`accelerator.print()` resolving to MockAccelerator.print.
Flag a Python attribute call is_method when its receiver is unresolved,
and extend the existing TS/JS weak-member guard to Python. Receivers that
ARE known stay exempt: self/cls, super(), and any identifier bound by one
of the file's imports (including the root of an attribute chain), so
Python's canonical `module.function()` cross-file call still resolves.
The guard helper is renamed cbm_tsjs_suppress_weak_method_match ->
cbm_suppress_weak_member_match. Its drop-list is unchanged, and the
per-language gate stays at the two call sites, which enumerate the same
language set in pass_calls.c and pass_parallel.c so the sequential and
parallel resolvers cannot diverge. ArkTS remains in that set.
python/S6/inherited_method in the resolution probe now asserts calls == 0,
recording the same trade TypeScript took when the receiver-aware guard
landed: the edge it used to produce came from a unique_name registry
fallback (measured on main: strategy=unique_name, cands=1, conf=0.7500)
where a weak short-name guess happened to be right in a 2-file fixture.
It flips back to >= 1 once py_lsp_cross resolves inheritance, matching the
flip-back note already carried on the TypeScript S6 case.
Fixes#1276
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: Enferlain <15861396+Enferlain@users.noreply.github.com>
The cross-file Constant resolution test walks USAGE edges and calls
cbm_store_find_node_by_id() for each source, which populates the caller's
cbm_node_t with heap_strdup'd fields (store.c:195 via scan_node). It freed
the arrays it owned -- consts and edges -- but never released the per-edge
node, leaking one set of fields per iteration.
LeakSanitizer caught it in every sanitizer-enabled lane while the plain
test shards and both Windows legs stayed green, which is why the local
test-runner could not see it:
test-lsan-macos 417 bytes in 6 allocations
test-unix (ubuntu-latest, 2/3) 401 bytes in 6 allocations
test-unix (ubuntu-24.04-arm, 2/3) same signature
test-diag same signature
Use cbm_node_free_fields(), not cbm_store_free_nodes(). The latter frees
the array pointer itself after releasing each element's fields, so calling
it on a stack-allocated node is a bad free -- ASan aborts with
"bad-free test_lang_contract.c" if you try. cbm_node_free_fields() releases
the strdup'd members and leaves the struct alone, which is the idiom the
neighbouring probes already use (test_edge_types_probe.c:245).
The three other find_node_by_id call sites in this file predate this branch
and already handle their own cleanup; only the new Chialisp contract leaked.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The tag lookup and one mutual-consistency call ended in pipelines whose
grep legitimately matches nothing — no v* tags in a --no-tags CI
checkout, no bare-return form in server.json. Under `set -euo pipefail`
that exit status killed the contract mid-run: it exited 1 having printed
nothing, which reads like a failure that forgot to explain itself, and in
the tagless case it died before it could report that its truth source was
unavailable.
Found by running the contract in a real `git clone --no-tags --depth 1`,
not by inspection. With the guards, that clone now passes while
announcing the degradation, and still fails on a drifted surface through
mutual consistency alone.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Seven packaging surfaces declared 0.8.1 while v0.10.8 was the shipped
release — nine releases of drift in metadata published to package
registries, and the third version-metadata drift this month.
pkg/go was not cosmetic: it hardcodes its version into the release URL
it downloads from, so `go install` fetched v0.8.1 binaries.
Chocolatey had to move as a unit. The nuspec is version-only but
chocolateyInstall.ps1 pins the asset sha256, so bumping the nuspec alone
would ship a package declaring 0.10.8 while installing 0.8.1. The new
checksum is the v0.10.8 windows-amd64.zip digest, verified by downloading
the published asset and hashing it, not merely copied from checksums.txt.
The gate (tests/test_version_metadata_contract.sh, scripts/test.sh Step
0x) checks against the newest stable v-prefixed tag, because nothing in
the tree declares the version: CBM_VERSION is injected at build time and
the release is dispatched with a version input. That absence is exactly
why these files drifted. Mutual consistency across the surfaces needs no
git and always runs, so a shallow clone degrades the gate loudly instead
of passing silently.
Homebrew (0.10.3), Scoop and AUR (0.8.1) are left pinned: each embeds
per-release asset checksums, so bumping them is a distribution chore
rather than a metadata edit. They are still gated — internally
consistent, and never naming a release that does not exist — and each
carries a recorded reason for being frozen.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Only `name` goes through cbm_camel_split; qualified_name, label, file_path
and now `body` are indexed verbatim. `body` is natural-language prose, so
splitting it would mangle the very text the column exists to make findable,
and the unicode61 tokenizer already handles prose correctly.
That invariant lived only in the shape of one format string, where the
splitter is substituted into the name position alone. Nothing failed if a
later edit wrapped the body expression too.
cbm_camel_split("getUserById") yields "getUserById get User By Id", so a
split body would additionally match the fragment "User" -- seeding a
camelCase identifier into the prose and asserting the fragment does NOT
match turns the invariant into a test. The mirror assertion, that `name`
DOES match its fragment, pins the other half so the guard cannot be
satisfied by simply indexing nothing.
Verified binding: wrapping the body expression in cbm_camel_split makes
body:User match 1 instead of 0 and the test fails.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The #518/#519 tests so far each cover one layer: extraction produces the
docstring, the store indexes it into nodes_fts.body, the delta merge writes
the same columns, and the query returns Section and Module. Nothing yet
proved the layers connect.
This indexes a real temporary repository -- a README.md whose section body
runs well past the 500-byte cap, plus a META.yaml with a top-level
description -- through the full pipeline, then asserts both prose words are
matchable in the `body` column.
It is also the guard on the size budget. build_def_props drops an oversized
field ATOMICALLY, so a section body that did not fit the 2 KB properties
buffer would vanish without a trace and every narrower test would still
pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#518 (Section nodes don't index body text) and #519 (META.yaml /
frontmatter description values not indexed). A question asked in words
could not reach the nodes that hold the words: Section carried only its
heading, a config Module only its path, and both labels were filtered out
of BM25 results outright.
nodes_fts gains a fifth column, `body`, backfilled from each node's
EXISTING properties.docstring. No new column on `nodes`, no new table, and
CBM_INDEX_FORMAT_VERSION deliberately does NOT move -- nobody is forced to
reindex. A legacy four-column table survives untouched under CREATE VIRTUAL
TABLE IF NOT EXISTS: the backfill probes for the column before naming it,
and bm25()'s surplus fifth weight is inert on a table that has no fifth
column (FTS5 reads a weight only for a column an instance landed in). Such
a database keeps opening and searching, just without prose.
Two node kinds gain the docstring they lacked:
- Markdown section bodies (#518) -- the blocks following a heading, up to
the next heading or subsection, collapsed to single spaces and capped at
MAX_COMMENT_LEN (500 B) with a UTF-8 backoff that never splits a
multi-byte sequence.
- Top-level YAML/JSON description / summary / purpose (#519), promoted
onto the file's Module node.
Section and Module are no longer excluded from BM25 results. Both the
ranked query and the mirrored count query were changed together; removing
the filter from only one desynchronises results from counts. Column weights
are bm25(nodes_fts, 1.0, 1.0, 1.0, 1.0, 0.3): the four identifier columns
keep parity with each other and prose sits well below them, so a prose-only
hit surfaces but never outranks an identifier match.
The FTS backfill is consolidated into one writer, cbm_store_fts_rebuild(),
used by the wholesale rebuild AND the delta merge. That second site matters:
it had its own hand-written four-column INSERT, which with a fifth column
would leave body NULL for every node arriving by delta -- silently, on the
warm path most users hit, while a full reindex looked perfect. There is now
exactly one place the column list is decided, and a test that fails if the
delta path stops writing prose.
The backfill guards json_extract with json_valid: malformed properties JSON
exists in pre-fix databases and would otherwise abort the whole rebuild.
Tests: markdown section bodies (own body, stops at the next heading, UTF-8
safe cap, empty body stays NULL), YAML/JSON description promotion (block
scalars, synonym fallback, top-level only, config-only), FTS body indexing
and its column isolation, malformed-JSON tolerance, legacy four-column
degrade, incremental watermark, BM25 finding Section and Module by prose,
results/total consistency, identifier-outranks-prose, and the delta-merge
regression.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: ShauryaaSharma <196896785+ShauryaaSharma@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The line read `7x ABI-13, 86x ABI-14, 65x ABI-15`, which sums to 158 against
161 vendored grammars. Counting `#define LANGUAGE_VERSION` in each parser.c
gives 9 / 79 / 73 = 161, so the line was already wrong before Chialisp; bumping
the ABI-14 bucket by one, as the Chialisp change did, would have carried a
four-off error forward under a fresh timestamp.
Recount with Chialisp included: 9 / 80 / 73 = 162, which matches both the
grammar-directory count and the language-count contract. The regeneration
command is now recorded next to the numbers so the next person does not have to
rediscover it:
grep -h '#define LANGUAGE_VERSION' internal/cbm/vendored/grammars/*/parser.c \
| sort | uniq -c
Vendored-integrity gate and the language-count contract both re-run green.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Chialisp commit was formatted with the pinned Homebrew clang-format, which
is right for the production sources but wrong for the three test files it also
touched: tests/ is not in LINT_SRCS, so those files have never been under the
format gate and were already format-dirty on main. Running the formatter over
them rewrote code that had nothing to do with Chialisp — 478 whitespace-only
lines in repro_invariant_breadth.c alone, plus 50 in test_extraction.c and 14 in
test_lang_contract.c — which is a review burden and hides the actual change.
Restore those three files to main's bytes and re-apply only the Chialisp
additions, in each file's own house style. The result is token-identical to the
version that was tested, apart from one em-dash written as `--` in a comment.
Re-verified after the rewrite: the six chialisp extraction tests, the cross-file
Constant graph test, the matrix-B row, the breadth case
(`[INV-BREADTH] PASS chialisp calls=1 callable=1 module=0`), and `make lint-ci`.
The production sources keep their formatting: they ARE in LINT_SRCS, and the
only whitespace they carry is the trailing-comment realignment in cbm.h that
adding a longer enumerator forces, plus the re-indent in extract_defs.c from
moving the name-resolution block into an else branch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Chialisp (.clsp/.clib/.clinc) is the Chia smart-coin language. The only
public grammar, Quexington/tree-sitter-chialisp, cannot parse it: on the five
reference files from chia-blockchain@main it produced 3-34 ERROR nodes each.
Three defects account for all of them, and each desynchronised the rest of the
file rather than losing one node:
* comments were `/;.*\r\n/` -- CRLF-only, while real files are LF;
* `(include foo.clib)` was rejected because of the dot in the filename;
* `(defconstant NAME <expr>)` accepted only a primitive after the name.
So the grammar is written here instead, in tools/tree-sitter-chialisp/, and
files as first-party: its vendored LICENSE is the project's own, byte-identical
to the repository root, carrying no third-party copyright.
It is deliberately GENERIC -- source_file / list / symbol / string / number /
hex / dot / comment, modelling the clvm_tools reader rather than the Chialisp
form vocabulary. `mod`, `defun`, `defconstant` and `include` are ordinary head
symbols; which lists are definitions is decided in extract_defs.c. A dialect
that adds a form (cl-21 -> cl-23 -> cl-26 each added several) therefore needs
no regenerated parser. Acceptance: all five reference files parse with zero
ERROR and zero MISSING nodes; the three defects above are pinned as tests.
Extraction, and the four decisions in it:
* `Constant` is admitted to the cross-file name registry
(cbm_label_is_registry_symbol). Chialisp's condition codes live as
`(defconstant ...)` in a shared .clib and are referenced by name from every
puzzle; labelled but unregistered they are seeded nowhere, and roughly half
of a Chialisp graph would be present as nodes yet never the target of an
edge. All three registry-seeding sites route through that one predicate, so
they cannot diverge. No other language emits `Constant`, so the blast radius
is exactly Chialisp. Pinned by a two-file graph test that goes red when the
label is removed from the predicate.
* `export` / `namespace` are NOT definition heads. `(export foo)` names a
function already defined in the same file, so treating it as a def head
mints a second node for one symbol. They stay in the not-a-call filter.
* A parameter list is a `list` too, but it binds rather than invokes. Without
that gate `(defun check_conditions (HEIGHTLOCK conditions) ...)` mints a
phantom call to HEIGHTLOCK, and real Chialisp is mostly such definitions.
* `(include *standard-cl-26*)` is a dialect directive, not a file dependency;
recording it would invent a dependency on a file that does not exist.
Three helpers that the defs, calls and unified walks all need -- the quote-data
gate, comment-skipping named-child access, and the Chialisp def-head set -- live
in helpers.c rather than being copied per translation unit. They previously
existed as byte-identical duplicates that had already drifted: extract_defs.c
skipped comments when reading a def head while its mirror in extract_unified.c
did not, so a comment between a head and its name desynchronised defs from call
scope. Both now use the same accessor.
Ledger: 161 -> 162 grammars across the seven published surfaces, ABI-14 86 -> 87,
first-party 13 -> 14, CBM_LANG_COUNT 164 -> 165.
Distilled from PR #919 by darkverbito, whose extraction logic -- the CLVM
not-a-call vocabulary, the quoted-data gate, the filename-derived `mod` name and
the include/embed import handling -- is kept and is the substance of this change.
The grammar that PR vendored is not used.
Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The closure-delta incremental route scored importance off a graph that does
not contain the project's edges, and persisted the result.
cbm_delta_preseed fills that route's gbuf with PROXY nodes -- id, label, name,
qualified name, file path, and nothing else. Only the re-extracted files carry
real edges; every inbound edge from an unchanged file is snapshotted and
re-linked later, inside cbm_delta_patch. Scoring in run_postpasses therefore
read an in-degree of zero for symbols with hundreds of real callers, and
because the repaired files' rows are purged and re-inserted, cbm_delta_patch
wrote those zeros to disk. Measured on a fixture whose `helper` has 25
callers: full index 5.000000, after one closure-delta re-index 0.000000.
closure_try_plan runs first, so this is the preferred warm path, not an edge
case. The comment at the call site claimed a rehydrated whole-project buffer;
that is true only of the legacy-partial route, and it is now stated correctly.
The fix rescores in SQL over the staging store, immediately after
cbm_delta_patch has inserted the new nodes and re-linked the inbound edges --
the first moment on this route where the complete graph is queryable. Loading
the full graph into RAM instead would have given back the point of the delta
route.
Rescoring is project-wide, not limited to the changed files. Distinct-files-
per-name and in-degree are both GLOBAL inputs: a body-only edit in one file
can change the score of a symbol defined in a file that did not change. A
scoped rescore would leave those stale.
TWO ROUTES, ONE RULE. The scoring rule and the JSON write-back are not
reimplemented in SQL. cbm_pipeline_importance_score() and
cbm_pipeline_importance_set_prop() are now single definitions that both routes
call -- the SQL side reaches them through two SQLite user-defined functions,
so the same C code runs either way and the rule cannot drift. SQL contributes
only the three aggregates, each an indexed GROUP BY. What remains free is the
input GATHERING, and tests bind that directly: a full index and a closure-delta
incremental index converging on the same tree must produce identical scores for
every symbol. Both new tests assert the route was actually taken, so neither
can pass by silently running some other path.
Cost, measured on a 666 MB Java corpus: cold full index unchanged at 83 s
(the recompute does not run on that path); warm closure-delta re-index 20 s
without the recompute, 26 s with it. The delta route keeps its advantage over
a full reindex (26 s vs 83 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Score every Function/Method/Class node during indexing and persist the
result as a numeric "importance" key inside the node's existing
properties_json:
importance = sqrt(num_refs) * priv * generic * distinct * test_penalty
num_refs is the incoming CALLS + USAGE degree. The multipliers demote
private (leading-underscore) and generic names (defined in >= 5 distinct
files), promote distinctive identifiers (snake_case or camelCase, len >=
8), and demote test scaffolding -- symbols in a test file per the graph's
own cbm_is_test_path() classifier, or targets of an incoming TESTS edge.
Weighted degree only; a transitive (PageRank) refinement is deliberately
not built here until it can be shown to beat this on a fixed judgment
set.
No schema change and no CBM_INDEX_FORMAT_VERSION bump: the score lives in
the existing properties_json TEXT column, so no index is forced to
rebuild and indexes written by older builds simply lack the key.
Three failure modes this shape invites, all of them silent, are addressed
directly:
* Cost. The generic-name multiplier needs |{files a name is defined in}|.
Computing that per NODE with a pairwise file-path scan is O(k^3) in a
same-name group of size k -- ruinous on real corpora, where a single
Java name group reaches k > 4000. Here the count is computed once per
distinct NAME and memoized, and file paths are deduplicated through a
hash set, so total distinct-file work is linear in the graph. Measured
cold full index of a 666 MB Java corpus, same -O2 binary: 85 s with the
pass disabled, 80 s with this implementation, 481 s with the per-node
shape.
* Registration. PREDUMP_PASS_COUNT is now derived from the pass table
with sizeof instead of being hand-written. A hand-written count that
lags an appended entry silently skips the last-registered pass with
every test still green.
* Incremental. The incremental post-pass sequence runs the pass too, over
a graph rehydrated from the store whose nodes already carry the key.
The write-back overwrites an existing key in place instead of appending
a second one, which would otherwise corrupt properties_json into
{"importance":1.0,...,"importance":2.0}.
tests/test_importance.c covers the formula and binds all three failure
modes; the complexity suite gains a GATED counter asserting the pass's
same-name-group work stays linear (~2x, not ~4x) under a corpus doubling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: petercoxphoto <20230727+petercoxphoto@users.noreply.github.com>
The two layer-5 interface probes tested for a substring by piping the
captured output into `grep -q`. Under `set -o pipefail` (line 32) that
turns a SATISFIED contract into a probe failure whenever the payload is
larger than the pipe buffer: `grep -q` exits at the first match, and the
writer takes EPIPE on the remainder, so the pipeline reports the writer's
non-zero status even though the match succeeded.
Observed on macos-15-intel:
test_venue_parity_contract.sh: line 404: printf: write error: Broken pipe
INTERFACE CONTRACT: scripts/ci/smoke-artifact.sh --help must exit 0 and
print a Usage: block (rc=0)
VENUE PARITY CONTRACT VIOLATED — interface probes failed (layer 5)
rc=0 and the Usage: block was present — the match is exactly what made
grep exit early. `smoke-artifact.sh --help` is 698 bytes against a
512-byte PIPE_BUF, and prints `Usage:` on its first line, so the writer
is killed at the earliest possible moment. Whether the remaining write
lands before grep exits is a scheduling race, which is why this only
ever showed up on one runner.
Replace both pipelines with in-shell `case` matching: no subprocess, no
pipe, no race, and a strictly literal match (the second probe's "--help."
was previously a regex whose `.` matched any character).
Verified: the contract passes (18 --help entries, 8 strict-flag entries),
and the matcher still returns 0 for output with no Usage: block, so the
probe can still fail.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Ilya Brykau