index_repository rendered every non-zero pipeline code as "Pipeline
failed. Check repo_path exists and contains source files", which sent
people debugging a path that was fine (#2020). Three outcomes now say
what actually happened:
- CBM_PIPELINE_ABORT_OVER_BUDGET (#1997, #832): status "error" with
reason "over_memory_budget", previous_index "preserved", the budget and
peak RSS in MB, and a hint naming CBM_MEM_BUDGET_MB / CBM_WORKERS /
excludes. The handler logs index.abort reason=over_memory_budget. The
CLI prints the same JSON and exits 1; a supervised worker still exits 0
with the response written, so the daemon passes it through verbatim
instead of entering skip-and-continue recovery.
- CBM_PIPELINE_ABORT_PRESERVE_DB: status "aborted_previous_preserved"
(retry; the previous index still serves).
- CBM_PIPELINE_PERSIST_FAILED: status "persist_failed" (disk/permissions).
The incremental test helper goes fail-closed on error responses so a
transient pre-publication abort surfaces once at the source instead of
cascading into dozens of downstream count failures.
Test: an in-process index over 64 files at a 1 MiB budget seam returns
the named error, leaves the project DB the same size on disk, and
index_status still reports the previous node count.
Refs #2020#1997#832
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase 5. Four test groups, each checked RED before it was kept.
- The Studio Export range join puts ONE ",+<N>" marker at the end with the
summed drop count. A marker left mid-string makes every reader stop there
and silently lose the ranges after it. Reaching the join through the
pipeline needs an export file with 256+ error regions across two <Class>
elements, so it goes through a test seam, following the pattern already in
this repo (CBM_COVERAGE_MARKER_TEST_API).
- check_index_coverage emits every range in front of a marker, never turns
the marker's digits into a range, and reports "truncated" from BOTH caps —
the producer's and its own 256 limit.
- test_index_resilience now has a ceiling beside its floor: exactly one of
the two fixture files is flagged, the clean neighbour is absent, and the
range does not cover the whole file.
- The three _Thread_local forms are pinned as measured. Only the array form
fails today; the plan's Phase 0 also listed the pointer form, and that is
wrong on the grammar shipped now.
Also fixes 13 clang-format violations the earlier commits on this branch left
in cbm.c, mcp.c and pass_definitions.c. `make -f Makefile.cbm lint-format`
would have failed CI. The changes are whitespace only — the two reflowed tool
descriptions concatenate byte-identically, so no output moved.
Full suite: 7735 passed, 28 failed, 7 skipped. The 28 are the pre-existing
cli install/uninstall failures, identical at clean HEAD.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
Land the four residual hardening points from #938 on the current
semantic-search design (semantic_limit/offset, the certified top-K
fetch loop):
- extract_semantic_keywords rejects a semantic_query array that holds a
non-string element instead of silently running the string subset; the
caller gets the same isError as for a bare-string semantic_query.
- run_semantic_query_core fails closed when the vector scan itself
fails ("semantic search failed", isError) instead of rendering a store
error as zero matches plus the broaden-your-keywords hint.
cbm_store_vector_search now tells the two states apart: a store with
no node_vectors table (lean index) is CBM_STORE_NOT_FOUND and keeps
the empty-page + moderate/full-index hint that main relied on by
swallowing the "no such table" prepare failure; a failing scan is
CBM_STORE_ERR.
- vs_min_cosine_score hoists the keyword norms (once per search) and
the node norm (once per node) out of the per-(node, keyword) loop.
The integer sums are exact, so every score stays bit-identical.
- vs_append_result checks its strdup results, frees the partial row on
failure and counts a row only once every copy succeeded, so the
caller's free path stays valid.
Two reproduce-first tests in the mcp suite fail on main and pass with
the change: non-string elements produce the type error, and a forced
cbm_cosine_i8 failure produces the fail-closed error for semantic-only
and combined calls while a missing node_vectors table still yields the
lean-index hint.
Refs #915
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: Dustin Persek <dustin.persek@protonmail.com>
A grep record longer than the read buffer used to be split across two
reads, and the continuation was parsed as a fresh file:line:content
record — a fragment of matched content became a file path and the text
after the next delimiter became a line number. An agent reading that
result called get_code_snippet on a path that was never in the
repository. The same split could also lose a real match, when the tail
held fewer than two delimiters and the record was dropped instead.
The production fix landed via #1597 (3c7427e): search_code now reads each
record with cbm_getline into a growable buffer, so a record can no longer
split. That closed#2011 as a side effect, and nothing on main guards the
shape — the buffer could be reintroduced by a future rewrite of the read
loop with no test to catch it.
Add the regression test the fix never got. It writes a >2 KiB colon-laden
line plus a normal one, calls search_code through cbm_mcp_server_handle,
and asserts the repository's two real matches are reported as two — not
three — and that no row carries the line 0 the fabricated record used to
produce.
Signed-off-by: kavish-19 <63698788+kavish-19@users.noreply.github.com>
The unreadable-operand scenario used mode-0 permission bits, which
Windows cannot express and root ignores (the Colima legs run as root), so
the test skipped on both — and bare SKIP() is forbidden. The scoped file
list deliberately drops every missing or non-regular indexed path before
grep sees it, so a refusal can only come from the scan itself. The scan
seam now delivers one plausible hit and then grep's error status, on every
platform, and the test asserts that neither that hit nor zero matches is
ever reported as a complete page.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
- test_daemon_lifecycle.py: daemon coordination is account-wide and
CBM_CACHE_DIR does not move the rendezvous (main's CBM_RUNTIME_DIR
relocation), so a private CBM_RUNTIME_DIR per cache is what makes the
--verbose call cold and lets it observe the spawn hint.
- test_hook_augment.py: the CLI answers in the lean tree form by default;
the guard parses the JSON projection, so it asks for it.
- test_mcp.c: the fail-closed search test skipped on Windows past the
teardown that names its POSIX-only response.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fix-forward after rebasing onto main (545 commits, 26 conflicted-file
resolutions). Where main had shipped a behaviour in the meantime, main
wins; the PR keeps its output contract on top of it:
- search_code: the scan runs under main's supervised, deadline- and
output-bounded runner and feeds the branch's streaming classifier from
the runner's output file; the POSIX wrappers map grep's no-match to 0,
so every non-zero exit fails closed (no exit-1 special case).
- trace_path: strategy/confidence precede args in header and rows (#1542),
in both the tree table and the json legs.
- cypher: relationship expansion is never capped (a cap falsified
aggregates) and unlabeled MATCH scans every candidate; `truncated` is
raised only by a real ceiling — a clamped hop range is probed one hop
beyond the cap and reports truncated only when a candidate exists there.
The plain BFS and the trail BFS both probe one row past max_results so
saturation is observable instead of silent.
- search_code full mode and the final json use main's U+FFFD sanitizer
instead of '?' replacement.
- search_graph json keeps an empty `groups` array for semantic-only calls;
list_projects json echoes offset/limit next to next_offset and accepts
main's include_details spelling for the stats projection.
- discovery budget: main added get_file_outline, compare_graphs, manage_adr
set_sections and the debug/include_details parameters; their descriptions
are trimmed to the lean style and the ceiling moves from 15 to 18 KiB.
Tests that asserted a JSON default now request format:"json"; the
fail-closed search test uses an unreadable regular file (a missing or
non-regular indexed path is deliberately not a scan operand on main); the
non-regular fixture gives its real sources File nodes, as every indexed
project has; list_projects total counts listable projects (the 0-byte
ghost is never a row, so it is not a page either).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
CI's tsan and lsan-macos runs failed the mutation-guard suite on
ASSERT(cbm_file_exists(db_path)) at the guard-balance test, while the
behavior suite passed everywhere. The assertion ran after the test's
own teardown: cleanup_project_db() unlinks the same db path before the
assertion block, and the manage_adr branch above replants and
quarantines that very path, so the late file check observed the
test's cleanup, not the query-only resolve it was meant to pin.
Whether it tripped depended on unlink/visibility timing, which is why
it surfaced only under some schedulers and never in the single-suite
local runs.
Capture cbm_file_exists() into query_db_left_in_place right after the
query branch completes, before anything replants or removes the file,
and assert on that snapshot.
Signed-off-by: 周文瑄 <zhouwx1997@126.com>
search_graph, query_graph, trace_path, get_code_snippet, get_graph_schema,
get_architecture, search_code, index_status, check_index_coverage and
detect_changes were annotated readOnlyHint=false / destructiveHint=true
because resolve_store()'s corrupt-store recovery quarantines database
files - a mutation. That conservatism locked every plan-mode client out
of the read-only analysis surface (#1100 calls the same tools "eleven
read-only analysis tools"), since spec-compliant plan gates refuse tools
that are not declared read-only.
This closes the gap the annotation comment pointed at. The query-only
resolve now classifies a failed integrity check without mutating: no
mutation lease, no quarantine. A confirmed-corrupt database is reported
(a new STORE_RECOVERY_CORRUPT status; the tool reply names the corruption
and points at index_repository instead of a misleading "project not
found") and left in place for a write-side open to quarantine and
rebuild, exactly as before. Transient verdicts stay retryable. Write-side
opens (index_repository, manage_adr updates) keep the existing blocking
recovery, quarantine, and generation re-check semantics unchanged.
The ten query tools' annotations flip to readOnlyHint=true /
destructiveHint=false, making them callable under plan mode in Claude
Code, Codex, ZCode, and every other client that gates on the hints.
Tests: the annotations expectation table is flipped; the query branch of
the guard-balance and guard-denial tests now assert zero leases taken and
byte-identical files; the ghost-db test (#704) expects the corruption
message; the four write-side recovery tests (generation re-check after
lease wait, unique backup naming, publish failure, published-WAL
completeness) are driven through manage_adr update - the write path where
that recovery now lives.
Signed-off-by: 周文瑄 <zhouwx1997@126.com>
strtod answers 0.0 for text it cannot read, and 0.0 is a real confidence
in both places that call it. So a property blob carrying a malformed
value, such as "confidence":null, came back as a recorded confidence of
zero. Two things went wrong with that.
In src/graph_buffer/graph_buffer.c, edge_props_confidence answers
CBM_EDGE_CONF_ABSENT (-1) when an edge carries no confidence, so any real
confidence outranks it. A malformed value answered 0.0, which beats -1 in
the merge comparison, so the malformed blob displaced a clean stored one.
The function's own comment already promised that "absent/unparseable
reads as -1". Only the absent half was true.
In src/mcp/mcp.c, bfs_edge_evidence_for_hop sets the confidence to -1 and
the emitter publishes any value of 0 or more as a recorded number. A
malformed value printed as 0.00, which reads as "the resolver was certain
this is wrong" rather than "nobody wrote a number here".
Both sites now pass an end pointer to strtod and keep the absent sentinel
when the pointer never moved, which is the shape src/store/store.c:402
already uses.
Two tests come with the change, and both were seen failing before the fix
and passing after:
gbuf_edge_props_unreadable_confidence_does_not_displace_absent
tool_trace_path_unreadable_confidence_reports_not_recorded
Red: 258 passed, 2 failed. Green: 260 passed, 0 failed.
The full suite reports 7633 passed, 2 failed. Both failures are in
tests/test_cli.c (lines 1749 and 6725) and reproduce on a clean tree
without this change. They depend on the coding agents installed on the
machine, not on this change. make -f Makefile.cbm lint-ci passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
The rebase resolution hand-aligned the analysis_tools array in both
src/mcp/mcp.c and tests/test_mcp.c after adding get_file_outline beside
compare_graphs, and the column widths did not match what clang-format
produces. That was the only red leg.
Whitespace only: with all whitespace stripped, both files hash identically
before and after (mcp.c d9b9bf785ac9, test_mcp.c 5a7fbf137492), so no
rebuild was required to re-validate the suites.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Registering get_file_outline makes 17 tools. The pagination test passed the
literal cursor "8" and asserted no nextCursor came back, which silently
encoded "there are at most MCP_TOOLS_PAGE_SIZE * 2 tools". At 17 the tail
from offset 8 legitimately overflows one page, so the gate failed for a
reason that had nothing to do with pagination being wrong.
Production behaviour is correct and unchanged: the no-cursor path returns
every tool, and a cursored page emits nextCursor exactly while tools remain
after it. Only the test's hard-coded count assumption was wrong.
The offset is now derived from the live count, so the final page is asserted
wherever it falls and adding tool 18 will not break it. A positive control
asserts that page 0 DOES advertise a cursor, so the no-cursor assertion
cannot pass vacuously if paging stopped emitting altogether.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
manage_adr mode='set_sections' rebuilt the ADR from cbm_adr_parse_sections()
and cbm_adr_render(). That model is lossy, so the rebuild silently rewrote
text nobody asked to change. Measured on the previous commit:
- Everything before the first recognised heading was dropped.
- A mis-cased '## Purpose' was not a heading, so when it came first its
whole block was dropped with the preamble — an ordinary typo cost the
author a section.
- Sections stored out of canonical order were reordered.
mode='update' never had this exposure because it writes the caller's bytes
verbatim; the section merge introduced it.
The merge now locates the target heading's byte span and replaces only that
span, appending a block when the heading is absent. Bytes outside the span
are never reconstructed, so a preamble, a code fence, an unrecognised
heading and the author's section order all survive byte-for-byte. That
property is the acceptance test: every document in
adr_splice_preserves_untouched_bytes is a case the rebuild corrupted.
Arbitrary headings now work, which is what an "add an entry" mode needs.
The canonical six become a convention rather than a privilege: they are
still what ADR_EMPTY_HINT suggests, but they get no ordering priority and
none is required. Under splicing, ordering is whatever the document says,
and any privileged reordering would reintroduce the data loss above.
Names match exactly, including case, so '## Purpose' and '## PURPOSE' are
different sections. Folding them would silently merge two blocks the
author chose to keep apart — the same class of loss this commit removes.
cbm_adr_validate_section_keys() no longer enforces the canonical set. It
now rejects only names that could not round-trip through a '## NAME' line
(empty, '#'-leading, newline-bearing, edge-whitespace, over-long), because
such a name scans back as a different heading or as none, and a second
identical write would append a duplicate instead of being a no-op.
A heading inside a fenced code block is a code sample, not a section. An
unterminated fence hides every heading after it, so a write would append a
duplicate rather than replace: that is refused explicitly, and mode='sections'
reports the same ambiguity instead of answering with a partial list.
manage_adr mode='sections' now uses cbm_adr_scan_headings(), the same
classifier the write path splices with. It used to list any '#'-prefixed
line, so a '## Foo' in prose or inside a fence was reported as a section no
write could target. Two components disagreeing about what a section is was
how a section write came to be able to destroy one.
cbm_adr_parse_sections() is unchanged, so documents parse exactly as before.
TEST(adr_parse_sections_non_canonical) still passes untouched.
Co-authored-by: andis777 <24672074+andis777@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
manage_adr could only replace: mode='update' overwrites the stored
document in full, so adding one entry costs a re-send of the whole ADR.
That is a data-integrity problem before it is a cost one — the caller has
to reproduce every byte it did not intend to change, so the unchanged
prefix survives only as well as the round-trip that carried it.
mode='set_sections' rewrites only the sections named in `section_updates`
and leaves the rest of the stored document untouched, so the stored copy
stays the authority for everything the caller did not name. Unlike a
whole-document append it is idempotent: a client that loses a response
and retries re-sets the same section to the same body and the document is
byte-identical, where an append would silently duplicate the chunk.
Only the six canonical section names are writable. That is a correctness
constraint rather than a style rule: adr_try_section_header() parses ONLY
canonical headers, so a non-canonical '## FOO' written here would be read
back as body text of the section above it, and a second identical write
would append a duplicate — destroying the idempotence the mode exists for.
Three things the new mode needed that were not there:
- cbm_store_adr_update_sections() now wraps its read-modify-write in
BEGIN IMMEDIATE. Three writers replace this row wholesale — the indexing
pipeline, the UI POST /api/adr handler, and mode='update' — so the
unguarded get/merge/store lost whichever of them committed between the
read and the UPSERT. mode='update' is a single atomic UPSERT and never
had that window; a section merge introduces it.
- set_sections joins the write_request classification. A mode missing from
it takes no per-project mutation lease, resolves the store query-only,
and never reaches open_adr_store_for_write — its write would be
attempted through a read-only handle while an index runs.
- The write path reads the legacy <root>/.codebase-memory/adr.md itself.
The existing migration runs on the read path only because it must not
block on the lease; without this a section write would merge onto an
empty document and discard an ADR still present on disk.
CBM_ADR_MAX_LENGTH now applies to an MCP write path: it is enforced inside
cbm_store_adr_update_sections, which mode='update' does not go through. An
empty section body, an unknown section name and a missing section_updates
are all rejected before any store is opened, so a caller that meant to
write never receives a success-shaped read.
Section-level update was chosen over the whole-document append proposed in
PR #1243; the analysis that established the problem is from that PR.
Co-authored-by: andis777 <24672074+andis777@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#518 (Section nodes don't index body text) and #519 (META.yaml /
frontmatter description values not indexed). A question asked in words
could not reach the nodes that hold the words: Section carried only its
heading, a config Module only its path, and both labels were filtered out
of BM25 results outright.
nodes_fts gains a fifth column, `body`, backfilled from each node's
EXISTING properties.docstring. No new column on `nodes`, no new table, and
CBM_INDEX_FORMAT_VERSION deliberately does NOT move -- nobody is forced to
reindex. A legacy four-column table survives untouched under CREATE VIRTUAL
TABLE IF NOT EXISTS: the backfill probes for the column before naming it,
and bm25()'s surplus fifth weight is inert on a table that has no fifth
column (FTS5 reads a weight only for a column an instance landed in). Such
a database keeps opening and searching, just without prose.
Two node kinds gain the docstring they lacked:
- Markdown section bodies (#518) -- the blocks following a heading, up to
the next heading or subsection, collapsed to single spaces and capped at
MAX_COMMENT_LEN (500 B) with a UTF-8 backoff that never splits a
multi-byte sequence.
- Top-level YAML/JSON description / summary / purpose (#519), promoted
onto the file's Module node.
Section and Module are no longer excluded from BM25 results. Both the
ranked query and the mirrored count query were changed together; removing
the filter from only one desynchronises results from counts. Column weights
are bm25(nodes_fts, 1.0, 1.0, 1.0, 1.0, 0.3): the four identifier columns
keep parity with each other and prose sits well below them, so a prose-only
hit surfaces but never outranks an identifier match.
The FTS backfill is consolidated into one writer, cbm_store_fts_rebuild(),
used by the wholesale rebuild AND the delta merge. That second site matters:
it had its own hand-written four-column INSERT, which with a fifth column
would leave body NULL for every node arriving by delta -- silently, on the
warm path most users hit, while a full reindex looked perfect. There is now
exactly one place the column list is decided, and a test that fails if the
delta path stops writing prose.
The backfill guards json_extract with json_valid: malformed properties JSON
exists in pre-fix databases and would otherwise abort the whole rebuild.
Tests: markdown section bodies (own body, stops at the next heading, UTF-8
safe cap, empty body stays NULL), YAML/JSON description promotion (block
scalars, synonym fallback, top-level only, config-only), FTS body indexing
and its column isolation, malformed-JSON tolerance, legacy four-column
degrade, incremental watermark, BM25 finding Section and Module by prose,
results/total consistency, identifier-outranks-prose, and the delta-merge
regression.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: ShauryaaSharma <196896785+ShauryaaSharma@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
