Only `name` goes through cbm_camel_split; qualified_name, label, file_path
and now `body` are indexed verbatim. `body` is natural-language prose, so
splitting it would mangle the very text the column exists to make findable,
and the unicode61 tokenizer already handles prose correctly.
That invariant lived only in the shape of one format string, where the
splitter is substituted into the name position alone. Nothing failed if a
later edit wrapped the body expression too.
cbm_camel_split("getUserById") yields "getUserById get User By Id", so a
split body would additionally match the fragment "User" -- seeding a
camelCase identifier into the prose and asserting the fragment does NOT
match turns the invariant into a test. The mirror assertion, that `name`
DOES match its fragment, pins the other half so the guard cannot be
satisfied by simply indexing nothing.
Verified binding: wrapping the body expression in cbm_camel_split makes
body:User match 1 instead of 0 and the test fails.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The #518/#519 tests so far each cover one layer: extraction produces the
docstring, the store indexes it into nodes_fts.body, the delta merge writes
the same columns, and the query returns Section and Module. Nothing yet
proved the layers connect.
This indexes a real temporary repository -- a README.md whose section body
runs well past the 500-byte cap, plus a META.yaml with a top-level
description -- through the full pipeline, then asserts both prose words are
matchable in the `body` column.
It is also the guard on the size budget. build_def_props drops an oversized
field ATOMICALLY, so a section body that did not fit the 2 KB properties
buffer would vanish without a trace and every narrower test would still
pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#518 (Section nodes don't index body text) and #519 (META.yaml /
frontmatter description values not indexed). A question asked in words
could not reach the nodes that hold the words: Section carried only its
heading, a config Module only its path, and both labels were filtered out
of BM25 results outright.
nodes_fts gains a fifth column, `body`, backfilled from each node's
EXISTING properties.docstring. No new column on `nodes`, no new table, and
CBM_INDEX_FORMAT_VERSION deliberately does NOT move -- nobody is forced to
reindex. A legacy four-column table survives untouched under CREATE VIRTUAL
TABLE IF NOT EXISTS: the backfill probes for the column before naming it,
and bm25()'s surplus fifth weight is inert on a table that has no fifth
column (FTS5 reads a weight only for a column an instance landed in). Such
a database keeps opening and searching, just without prose.
Two node kinds gain the docstring they lacked:
- Markdown section bodies (#518) -- the blocks following a heading, up to
the next heading or subsection, collapsed to single spaces and capped at
MAX_COMMENT_LEN (500 B) with a UTF-8 backoff that never splits a
multi-byte sequence.
- Top-level YAML/JSON description / summary / purpose (#519), promoted
onto the file's Module node.
Section and Module are no longer excluded from BM25 results. Both the
ranked query and the mirrored count query were changed together; removing
the filter from only one desynchronises results from counts. Column weights
are bm25(nodes_fts, 1.0, 1.0, 1.0, 1.0, 0.3): the four identifier columns
keep parity with each other and prose sits well below them, so a prose-only
hit surfaces but never outranks an identifier match.
The FTS backfill is consolidated into one writer, cbm_store_fts_rebuild(),
used by the wholesale rebuild AND the delta merge. That second site matters:
it had its own hand-written four-column INSERT, which with a fifth column
would leave body NULL for every node arriving by delta -- silently, on the
warm path most users hit, while a full reindex looked perfect. There is now
exactly one place the column list is decided, and a test that fails if the
delta path stops writing prose.
The backfill guards json_extract with json_valid: malformed properties JSON
exists in pre-fix databases and would otherwise abort the whole rebuild.
Tests: markdown section bodies (own body, stops at the next heading, UTF-8
safe cap, empty body stays NULL), YAML/JSON description promotion (block
scalars, synonym fallback, top-level only, config-only), FTS body indexing
and its column isolation, malformed-JSON tolerance, legacy four-column
degrade, incremental watermark, BM25 finding Section and Module by prose,
results/total consistency, identifier-outranks-prose, and the delta-merge
regression.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: ShauryaaSharma <196896785+ShauryaaSharma@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The closure-delta incremental route scored importance off a graph that does
not contain the project's edges, and persisted the result.
cbm_delta_preseed fills that route's gbuf with PROXY nodes -- id, label, name,
qualified name, file path, and nothing else. Only the re-extracted files carry
real edges; every inbound edge from an unchanged file is snapshotted and
re-linked later, inside cbm_delta_patch. Scoring in run_postpasses therefore
read an in-degree of zero for symbols with hundreds of real callers, and
because the repaired files' rows are purged and re-inserted, cbm_delta_patch
wrote those zeros to disk. Measured on a fixture whose `helper` has 25
callers: full index 5.000000, after one closure-delta re-index 0.000000.
closure_try_plan runs first, so this is the preferred warm path, not an edge
case. The comment at the call site claimed a rehydrated whole-project buffer;
that is true only of the legacy-partial route, and it is now stated correctly.
The fix rescores in SQL over the staging store, immediately after
cbm_delta_patch has inserted the new nodes and re-linked the inbound edges --
the first moment on this route where the complete graph is queryable. Loading
the full graph into RAM instead would have given back the point of the delta
route.
Rescoring is project-wide, not limited to the changed files. Distinct-files-
per-name and in-degree are both GLOBAL inputs: a body-only edit in one file
can change the score of a symbol defined in a file that did not change. A
scoped rescore would leave those stale.
TWO ROUTES, ONE RULE. The scoring rule and the JSON write-back are not
reimplemented in SQL. cbm_pipeline_importance_score() and
cbm_pipeline_importance_set_prop() are now single definitions that both routes
call -- the SQL side reaches them through two SQLite user-defined functions,
so the same C code runs either way and the rule cannot drift. SQL contributes
only the three aggregates, each an indexed GROUP BY. What remains free is the
input GATHERING, and tests bind that directly: a full index and a closure-delta
incremental index converging on the same tree must produce identical scores for
every symbol. Both new tests assert the route was actually taken, so neither
can pass by silently running some other path.
Cost, measured on a 666 MB Java corpus: cold full index unchanged at 83 s
(the recompute does not run on that path); warm closure-delta re-index 20 s
without the recompute, 26 s with it. The delta route keeps its advantage over
a full reindex (26 s vs 83 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Score every Function/Method/Class node during indexing and persist the
result as a numeric "importance" key inside the node's existing
properties_json:
importance = sqrt(num_refs) * priv * generic * distinct * test_penalty
num_refs is the incoming CALLS + USAGE degree. The multipliers demote
private (leading-underscore) and generic names (defined in >= 5 distinct
files), promote distinctive identifiers (snake_case or camelCase, len >=
8), and demote test scaffolding -- symbols in a test file per the graph's
own cbm_is_test_path() classifier, or targets of an incoming TESTS edge.
Weighted degree only; a transitive (PageRank) refinement is deliberately
not built here until it can be shown to beat this on a fixed judgment
set.
No schema change and no CBM_INDEX_FORMAT_VERSION bump: the score lives in
the existing properties_json TEXT column, so no index is forced to
rebuild and indexes written by older builds simply lack the key.
Three failure modes this shape invites, all of them silent, are addressed
directly:
* Cost. The generic-name multiplier needs |{files a name is defined in}|.
Computing that per NODE with a pairwise file-path scan is O(k^3) in a
same-name group of size k -- ruinous on real corpora, where a single
Java name group reaches k > 4000. Here the count is computed once per
distinct NAME and memoized, and file paths are deduplicated through a
hash set, so total distinct-file work is linear in the graph. Measured
cold full index of a 666 MB Java corpus, same -O2 binary: 85 s with the
pass disabled, 80 s with this implementation, 481 s with the per-node
shape.
* Registration. PREDUMP_PASS_COUNT is now derived from the pass table
with sizeof instead of being hand-written. A hand-written count that
lags an appended entry silently skips the last-registered pass with
every test still green.
* Incremental. The incremental post-pass sequence runs the pass too, over
a graph rehydrated from the store whose nodes already carry the key.
The write-back overwrites an existing key in place instead of appending
a second one, which would otherwise corrupt properties_json into
{"importance":1.0,...,"importance":2.0}.
tests/test_importance.c covers the formula and binds all three failure
modes; the complexity suite gains a GATED counter asserting the pass's
same-name-group work stays linear (~2x, not ~4x) under a corpus doubling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: petercoxphoto <20230727+petercoxphoto@users.noreply.github.com>
The two layer-5 interface probes tested for a substring by piping the
captured output into `grep -q`. Under `set -o pipefail` (line 32) that
turns a SATISFIED contract into a probe failure whenever the payload is
larger than the pipe buffer: `grep -q` exits at the first match, and the
writer takes EPIPE on the remainder, so the pipeline reports the writer's
non-zero status even though the match succeeded.
Observed on macos-15-intel:
test_venue_parity_contract.sh: line 404: printf: write error: Broken pipe
INTERFACE CONTRACT: scripts/ci/smoke-artifact.sh --help must exit 0 and
print a Usage: block (rc=0)
VENUE PARITY CONTRACT VIOLATED — interface probes failed (layer 5)
rc=0 and the Usage: block was present — the match is exactly what made
grep exit early. `smoke-artifact.sh --help` is 698 bytes against a
512-byte PIPE_BUF, and prints `Usage:` on its first line, so the writer
is killed at the earliest possible moment. Whether the remaining write
lands before grep exits is a scheduling race, which is why this only
ever showed up on one runner.
Replace both pipelines with in-shell `case` matching: no subprocess, no
pipe, no race, and a strictly literal match (the second probe's "--help."
was previously a regex whose `.` matched any character).
Verified: the contract passes (18 --help entries, 8 strict-flag entries),
and the matcher still returns 0 for output with no Usage: block, so the
probe can still fail.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #1412 by @allanoepping. Fixes a real recall hole: tag-dialect
.cfc components were routed to the cfscript grammar (extracting nothing),
and <cfscript> bodies inside tag-dialect files are opaque cf_script_content
to the cfml grammar — whole legacy components produced zero functions.
Fix 1 — his .cfc dialect sniffer, nearly verbatim: cbm_disambiguate_cfc()
reads a 16K head via cbm_fopen; <cfcomponent>/<cffunction> anywhere means
tag dialect; a leading <cfscript (after whitespace and <!--- ---> comments)
or no markup at all keeps the old CFSCRIPT routing, so script-dialect files
and read failures behave exactly as before. Wired in detect_file_language
beside the existing _m/_cls/_inc sniffers, with his 8 direct tests.
Fix 2 — his embedded-defs capability, re-based onto the shared machinery
that superseded his mechanism mid-flight (#1852, merged after his last
rebase): a cfml_embedded_imports spec row (cf_script_tag/cf_script_content
-> CFSCRIPT), CFML added to the embedded-imports dispatch, and the
extract_structure gate extended to CFML. Included ranges on the original
buffer give absolute coordinates for free, so his line-number assertions
hold verbatim, and calls inside blocks become extractable as a bonus. His
~90-line slice+shift implementation is deliberately NOT carried — it
mirrored the sanctioned shape of its day, which #1852 replaced.
One correctness guard the port needed: the Vue attribute resolver is now
gated to Vue — cf_script_tag carries no lang=/src= attributes, and running
the resolver would have silently rewritten CFML's embedded language from
CFSCRIPT to the resolver's JavaScript default.
His embedded extraction test (line remap + <cfsetting> ERROR-cascade case)
carries over verbatim and binds: reverting the spec/gate wiring fails it at
ASSERT(has_def(r, "Function", "greet")). Suites: extraction, language,
edge_imports, lang_contract, grammar_labels, grammar_regression — 637/0.
Co-authored-by: Allan Oepping <allan@intellipay.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #1255 by @MasterFede5, rebased onto the single-binary
installer and completed with the two review conditions.
PowerShell 5.1 wraps redirected native stderr into ErrorRecords, so under
the script's global ErrorActionPreference=Stop a HEALTHY binary that
prints a single warning to stderr while exiting 0 aborts both
`--version 2>&1` probes with "downloaded binary failed to run". Both
probes now relax EAP to Continue for the probe call only, with an
exception-safe restore; failure detection stays on $LASTEXITCODE.
The review-required pre-seed ($global:LASTEXITCODE = 1) closes the window
the relaxation would otherwise open: a binary that fails to START under
Continue may leave $LASTEXITCODE untouched, and the installed-binary probe
runs after earlier successful native calls, so a stale 0 would have turned
a corrupt install into "Installed: ...". Seeding 1 makes start-failure
read as failure on both probes.
The bundle contract pins the guard's two load-bearing parts (the scoped
relaxation with restore, and the pre-seed), verified failing when either
is stripped. The change adds zero non-ASCII bytes; the file remains pure
ASCII per the PS 5.1 ANSI-decode rule.
Co-authored-by: MasterFede5 <friedrickms@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Maintainer follow-up on top of the contributor's rebased commit,
implementing the Half-B strategy decision from the 2026-08-21 review:
- Linux: when the self-resolved path fails the executable check after an
atomic rename-over, hand back the /proc/self/exe magic link. The link
still executes the in-memory OLD build, which is the only spawn the
worker's build-fingerprint gate accepts. Deliberately NOT the saved
launch path: after a replacement that path holds a DIFFERENT build, so
preferring it swaps an ENOENT for a fingerprint refusal.
- macOS: no magic link exists; fail closed. The supervisor logs
index.supervisor.no_self_path and indexing resumes on daemon restart
instead of spawning a missing binary. A rename-over that leaves a VALID
binary still resolves (and the fingerprint gate refuses downstream, with
its own coverage).
- The prefer-saved-launch-path branch from the original head is dropped
per the same ruling.
Also fixes both Half-A review items:
- Aliasing in cbm_http_server_set_binary_path: `path` may alias
g_binary_path on a repeated call. The overlapping snprintf cannot
actually fire today — resolve's leading out[0]='\0' zeroes the shared
buffer first — but that same reset means the call silently DISCARDS the
stored path it was asked to re-resolve, and the in-place write is one
reset-reorder away from real UB. Resolve into a local first: correct
input, no aliasing trap.
- Coverage asymmetry: the probe and driver ran only on Linux while the
production edit covered all of POSIX. Both now run on macOS too, and the
Linux probe first proves the "(deleted)" state via readlink so its
assertions cannot pass vacuously on an intact image.
Revert-check: with main's http_server.c restored, the driver fails at
WEXITSTATUS 44 (resolver returned a dead path) on macOS; with the guard it
passes 63/63.
Co-authored-by: sunmingqiang <sunmingqiang@bytedance.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Prefer the saved executable launch path when it still points to an
executable, and reject a deleted /proc/self/exe path before spawning
index workers.
Add a Linux integration regression that forks, atomically replaces the
running test binary on disk, and verifies that subprocess resolution
uses the saved launch path instead of the now-deleted self path.
Signed-off-by: David Musk <davidmusk93@users.noreply.github.com>
Signed-off-by: sunmingqiang <sunmingqiang@bytedance.com>
The language-count contract shipped this morning in #1859 had a hole: it
matched only the prose form "<N> languages", so README's shields badge
("languages-158-orange") never matched and the gate reported success while
the badge sat three behind the prose in the same file.
The cause is worth recording. The surfaces were enumerated with the same
regex the gate then used, so the verification could only ever confirm what
the enumeration had already seen. A gate and its own completeness check must
not share a pattern.
Both claim forms are now matched, in the per-surface check and in the
unregistered-file sweep, so a new file stating a count in either form still
fails. README's badge is corrected to 161.
Also corrects a stale range comment left by the PL/SQL merge: matrix B's
header said "CBMLanguage 80..163" when PL/SQL had made it 164. That is the
comment variant of the additive-counter trap — the numeric constants were
audited across that rebase, the prose describing them was not. The two
sibling range descriptors (GO..ARKTS = 68 rows, RACKET..PLSQL = 47 rows)
were checked and are correct.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Seven surfaces disagreed with reality and with each other: README.md,
docs/index.html, docs/llms.txt and pkg/npm/README.md said 158, while
server.json, the chocolatey nuspec and scripts/package-release.sh said 159.
161 grammars are vendored.
21 individual claims corrected. The count is now gated by
tests/test_language_count_contract.sh, so the next language to land cannot
ship a stale figure.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The count was hand-maintained on seven separate surfaces and had drifted apart
from reality AND from itself: README, docs/index.html, docs/llms.txt and the npm
README said 158, while server.json, the chocolatey nuspec and package-release.sh
said 159.
Derived from the vendored grammar directories rather than CBM_LANG_COUNT,
because the registry contains entries with no parser at all: CBM_LANG_NIM
(grammar removed 2026-06-12) and CBM_LANG_OBJECTSCRIPT_EXPORT have no grammar
directory, no tree_sitter function and no MANIFEST row, yet files still route to
them by extension. Publishing CBM_LANG_COUNT would claim languages we do not
parse. A vendored grammar directory demonstrably exists and is checksum-gated.
Accepted trade-off, stated in the test: this UNDERCOUNTS languages that share
one grammar (TypeScript/TSX, objectscript_udl/_routine). We publish the number
we can prove rather than the larger number we cannot.
The contract binds four things so it cannot go quietly vacuous:
- the grammar directory count is sanity-bounded, so a layout change fails loudly
instead of making every check below it trivially true
- MANIFEST.md's declared figure must already agree with the directory count
- every registered surface must state exactly that number, and a registered
surface that states NO number fails (the wording changed, or it should be
deregistered)
- any UNREGISTERED file stating a language count fails, so an eighth surface
cannot reintroduce the drift this exists to prevent
Files that legitimately say "<N> languages" about something else — benchmark
corpus subsets, probe scopes, the version-pinned winget manifests — are listed
with what their number actually counts, so no exemption is unexplained.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Assert positively that the fixture's tree contains ERROR/MISSING nodes
(parse_incomplete), so a future grammar upgrade that fixes AS OBJECT
turns this RED and prompts claiming the Class def instead of pinning its
absence vacuously.
Co-authored-by: Oğuz <oguzhan.saricam@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Vendor AndreasMaierDe/tree-sitter-plsql @ 28aebef209be (MIT, ABI 14, no
external scanner) and wire CBM_LANG_PLSQL end to end:
- extraction onto existing labels only: package spec/body -> Class,
procedure/function (create/definition/declaration) -> Function,
file -> Module; callee extraction for ref_call incl. the
package-qualified pkg.name form; identifier usage vocabulary with the
upstream 'parameter' kind (a ref_call ARGUMENT wrapper) carved out of
the whole-binding rule
- discover: .pks/.pkb/.pck/.pls/.plb/.plsql/.fnc/.trg/.bdy/.tps/.tpb
(.sql stays SQL and .prc stays FORM, both asserted negatively)
- ledgers: CALL_WITHOUT_REFERENCE_VOCAB(PLSQL), call-node manifest
ref_call row (historical total 219 -> 220), call-argument matrix B row
- tests: extraction triple (package body + package-qualified call,
standalone function, honest negative pin of the upstream
CREATE TYPE ... AS OBJECT ERROR-node limitation), grammar-regression
case, label golden Class:1,Function:1,Module:1, MUST_EXTRACT_DEFS,
extension mapping, bug-repro board battery row
- MANIFEST.md (community-niche verdict, local include-quote patch row),
THIRD_PARTY.md section, refreshed vendored checksums
PL/SQL support was contributed by Oguz (@ouzsrcm) in #1033; re-vendored
from upstream per vendoring policy with his language wiring distilled on
top.
Co-authored-by: Oğuz <oguzhan.saricam@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Maintainer fixups pushed to the PR branch, each with a test that fails
without it.
- The TargetConfigNames/TargetConfigName path emitted nothing. Route nodes
are keyed __route__ensemble__<prod>.<item> (as emit_async_call and
emit_handles both do), but the settings loop looked the source item up by
its bare <prod>.<item> qn. cbm_gbuf_find_by_qn is an exact hashtable
lookup (graph_buffer.c:769), so item_node was always NULL and every
declarative routing edge was dropped by `continue`. An incomplete rename
from the EnsembleItem version. Proof: the new settings test reports
async_calls == 0 against the previous code.
- Heap-buffer-overflow on malformed XML. With an <Item that has no closing
</Item>, item_end became the buffer's NUL terminator and `p = item_end + 7`
then pointed 7 bytes past the allocation, which the loop dereferenced
before strstr scanned unbounded heap. Now the scan stops at an
unterminated item. Proof: ASan reports heap-buffer-overflow at
pass_ensemble_routing.c:544 against the previous code.
- extract_xml_attr scanned forward from the tag with no bound, so a tag
omitting an attribute silently inherited it from a LATER tag, mispairing
ClassName/Enabled across items. The search is now bounded at the tag's
closing '>'. Proof: an item with no ClassName produced a second, bogus
Route node against the previous code.
- read_file ignored the fread result, leaving an uninitialised tail on a
short read for the string scans to walk. Terminate at the byte count
actually read.
Three tests added: a production whose ONLY routing information is a
TargetConfigNames setting (so any ASYNC_CALLS edge must come from that
path), an unterminated <Item> in an otherwise well-formed file, and an item
missing ClassName ahead of a well-formed one.
Full local suite: 7576 passed, 0 failed, 8 skipped (139 suites).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Grok Build (xAI's grok CLI/TUI) already reached codebase-memory-mcp through
its Claude/Cursor compat layer, but a Grok-only user got nothing. Detect
$GROK_HOME (default ~/.grok) or a grok CLI and install:
- an owned [mcp_servers.codebase-memory-mcp] managed block in config.toml;
a same-name foreign table is left byte-identical because a second header
would make Grok reject the whole file, while the known owned shape written
by `grok mcp add` is adopted. No env forwarding: grok 1.0.5 passes the
full parent environment to stdio servers (verified), unlike Codex (#1562).
- an owned rules file under $GROK_HOME/rules/ (always scanned, all projects)
- the shared skill under $GROK_HOME/skills/
- Scout/Verify/Auditor agents with named-server mcpInheritance and the tier
allowlist spelled out as the server__tool ids Grok's search_tool/use_tool
dispatcher accepts (inheritance filters per server, never per tool)
- hooks withheld: Grok's passive events discard stdout and PreToolUse honors
only deny/rewrite decisions, so there is no context-output contract.
Verified against grok 1.0.5: inspect discovers the server, agents, skill and
rules; mcp doctor handshakes and discovers 15 tools. --clients=grok token,
README/npm/llms/landing-page matrices (43 -> 44 surfaces), and smoke coverage.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel