mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-19 05:40:03 +08:00
docs/cli-daemon-behavior
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
8093d10b81 |
fix(rescript): stop binary .res resources hanging the indexer (#2176)
A 4.7 MB binary Godot resource (piko_walk_mesh.res) never finished indexing: `.res` maps to ReScript, so the resource was parsed as ReScript source and the definitions pass spun. Three defects, each fixed: 1. Quadratic usage walk (the hang). The ReScript `let` binding check (is_first_named_part_of, shared with Julia, Typst, Elm, PureScript and Nickel) climbed from every named leaf with ts_node_parent, which descends from the root and scans each level's children. A binary file parses into a flat error tree (~170k root children per MB), so each climb cost O(root children) and the file O(n^2): 1 MB spent 46 s in cbm_extract_unified, 4.7 MB never returned. The check now climbs the unified walk's own cursor (O(1) per hop), like the other occurrence classifiers; the node-based climb stays only for callers outside the walk. 1 MB: 46 s -> 0.03 s; 4.7 MB: 2.2 s end to end. 2. Quadratic lexing in error recovery. tree-sitter retries the lexer at every byte of an unparseable stretch with every external token valid; the ReScript scanner's template-string loop then ran to the next backtick/$/backslash/NUL (end of file if none) and discarded the result. That cost O(n^2) inside lexing, where the parse budget's progress callback never runs, so such files were dropped by the 5 s CPU budget instead of parsed -- plain text too (100 KB of non-ReScript text: 15 s -> 0.18 s). Local scanner patch: skip the template branch when NEWLINE is also valid, which only the ERROR state allows. Valid files parse to identical trees. Recorded in vendored/grammars/MANIFEST.md. 3. Binary content parsed as source. `.res` is also Godot's binary resource format and the Windows compiled-resource format. Following the .m/.cls/.frm/.inc/.cfc precedent, cbm_disambiguate_res reads the first 4 KB and reports a file holding a NUL byte as unsupported; ReScript text never contains one. Tests (deterministic work counters, no clocks): - extraction::extract_rescript_let_bindings_use_the_walk_cursor -- zero root-descending parent hops at 128 and 1024 statements (before: 515 / 4099). - complexity::complexity_rescript_error_recovery_lexing_is_linear -- lexer bytes pulled through a chunked TSInput, 4 KB vs 8 KB, ratio must be linear (before 3.99, after 2.00). - language::lang_res_binary_resource_unsupported and lang_res_rescript_stays_rescript. Fixes #2176 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogelandpetercoxphoto
|
e5f3aab02e |
feat(pipeline): index-time importance scoring (weighted degree)
Score every Function/Method/Class node during indexing and persist the
result as a numeric "importance" key inside the node's existing
properties_json:
importance = sqrt(num_refs) * priv * generic * distinct * test_penalty
num_refs is the incoming CALLS + USAGE degree. The multipliers demote
private (leading-underscore) and generic names (defined in >= 5 distinct
files), promote distinctive identifiers (snake_case or camelCase, len >=
8), and demote test scaffolding -- symbols in a test file per the graph's
own cbm_is_test_path() classifier, or targets of an incoming TESTS edge.
Weighted degree only; a transitive (PageRank) refinement is deliberately
not built here until it can be shown to beat this on a fixed judgment
set.
No schema change and no CBM_INDEX_FORMAT_VERSION bump: the score lives in
the existing properties_json TEXT column, so no index is forced to
rebuild and indexes written by older builds simply lack the key.
Three failure modes this shape invites, all of them silent, are addressed
directly:
* Cost. The generic-name multiplier needs |{files a name is defined in}|.
Computing that per NODE with a pairwise file-path scan is O(k^3) in a
same-name group of size k -- ruinous on real corpora, where a single
Java name group reaches k > 4000. Here the count is computed once per
distinct NAME and memoized, and file paths are deduplicated through a
hash set, so total distinct-file work is linear in the graph. Measured
cold full index of a 666 MB Java corpus, same -O2 binary: 85 s with the
pass disabled, 80 s with this implementation, 481 s with the per-node
shape.
* Registration. PREDUMP_PASS_COUNT is now derived from the pass table
with sizeof instead of being hand-written. A hand-written count that
lags an appended entry silently skips the last-registered pass with
every test still green.
* Incremental. The incremental post-pass sequence runs the pass too, over
a graph rehydrated from the store whose nodes already carry the key.
The write-back overwrites an existing key in place instead of appending
a second one, which would otherwise corrupt properties_json into
{"importance":1.0,...,"importance":2.0}.
tests/test_importance.c covers the formula and binds all three failure
modes; the complexity suite gains a GATED counter asserting the pass's
same-name-group work stays linear (~2x, not ~4x) under a corpus doubling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: petercoxphoto <20230727+petercoxphoto@users.noreply.github.com>
|
||
|
Martin Vogel
|
ec16764fee |
fix(test): convert forbidden SKIP()s in the complexity suite to policy forms
The no-skips lint gate (scripts/check-no-test-skips.sh) rightly rejected the throughput-report test's two SKIP() calls: - CBM_SKIP_PERF=1 is deliberate operator configuration, not a hidden environment failure: reporting is off by request, so the test PASSes with a stderr note instead of skipping. - an uncreatable report dir IS an environment failure and now FAILs with the remedy in the message (set CBM_COMPLEXITY_REPORT_DIR), per the policy text. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e700b6215f |
test(complexity): deterministic O(n^2) guard on work-counter ratios
Finding #1669 took an 11-corpus A/B across two release binaries. This suite makes that bug class fail a unit test in seconds, on every platform, from tiny corpora. Method: build k and 2k REPLICATED module copies, run the full in-process pipeline on both, assert counter RATIOS. Independent copies mean every extensive quantity — nodes, edges, Σ per-file registry defs — must grow linearly (ratio ~2). A files x corpus coupling makes per-file work itself grow with k and lands at ratio ~4. Ratios expose the exponent regardless of absolute scale, so 60-120 files suffice. Verdicts are pure functions of (code, input): gates ride ONLY on deterministic work counters and data-product counts, never on wall time. Throughput (nodes/s, edges/s) is information-only, written to private/benchmarks/complexity-<ts>.json (local, gitignored; skipped under CBM_SKIP_PERF where rates are meaningless). Two corpus shapes, both needed: - Independent modules (java/py/go/ts templates): catches cross-module contamination and dedup breakage. The #1669 bug is GREEN here — fully closed modules filter perfectly, which is exactly why it survived. - The growing shared package (bigpkg): one Java package whose file count scales with k — the real-repo shape (files concentrate in large packages). The JVM namespace filter branch makes per-file work track package size, so this corpus is the honest #1669 reproducer: ratio 4.00 RED on the pre-fix tree, 4.00 RED for a module-scoped overlay, 2.00 GREEN for the own-file overlay. It discriminated the correct fix design before the fix was written. Both legs of every pair exceed MIN_FILES_FOR_PARALLEL(50): below it the sequential path runs, which builds no shared registries and would be the wrong code path to gate (its per-file cost is bounded by the 50-file ceiling). Recorded but deliberately NOT gated, with reasons at the case: tail_candidates and fallback_rows are legitimately superlinear under replication until those scans are bounded, and measured ~1 ns/unit. Every ratio gate carries a non-vacuousness floor on the base counter so broken counter wiring fails loudly instead of green-washing (cbm_pxc_count_perfile_defs feeds the overlay path into the same counter the fallback path already used; wired for Java, extend to the TS overlay when touching ts_lsp). Dynamic coverage: languages iterate CBM_LANG_COUNT; embedded templates cover the LSP-hybrid languages, and tests/fixtures/complexity/<lang>/ dirs are auto-discovered so a new language joins the guard by dropping fixtures. Uncovered languages are listed in the report with the reason. The local report additionally carries per-language node/edge counts with ratios and a per-pass elapsed_ms table per run (captured via a TEE log sink during the in-process pipeline runs) — trend data for humans, still never a gate. Refs #1669. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |