The #979 direct-hit early return in resolve_import_map fires before
suffix handling, so any imported_symbol.method() call whose base symbol
is itself an indexed node resolved to the base node instead of
base.method — the exact mis-resolution the neighboring comment warns
about. On django this degraded the graph by ~11K edges (CALLS
74,685->72,175, TESTS 38,032->29,236); auth.login => Signal.send became
an edge onto the user_logged_in Variable node.
Take the direct hit only for suffix-less callees — precisely the
aliased direct-symbol import case (#875) the early return was added
for. Verified on django: CALLS 74,708 (baseline + the 23 genuine #979
wins), TESTS 38,035; the #979 bare-alias test stays green.
Reproduce-first: resolve_import_map_alias_with_suffix_hits_method is
RED on the unguarded code (returns the bare symbol QN) and GREEN with
the guard.
Closes#1000
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
- Add method_call_expression to perl_call_types in lang_specs.c so the
structural tier emits a method-call edge for the LSP bridge to refine
(parity with PHP member_call_expression). callee_name is the bare method
via the field-based extractor's `method` branch.
- Normalize the textual callee in cbm_pipeline_find_lsp_resolution to its
last "::"-separated segment so qualified static Pkg::sub() calls match the
resolved sub's dotted short-name (parity with PHP scoped_call_expression).
- Zero-edge guarantee preserved: untyped receivers still emit no edge.
Resolves DEVN-04 from plan 22-03. Verified on a Base/Derived/main fixture:
run_typed->{greet,describe}, run_static->helper, run_classcall->greet,
describe->greet (inherited), run_untyped->none.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Follow-up to #787/#828, disclosed during #828's review: the same clobber
class also affects two call sites #828 deliberately left out of scope.
create_env_configures_for_file (pass_definitions.c) and resolve_file_throws
(pass_parallel.c) both resolved their edge source via a direct
cbm_gbuf_find_by_qn(enclosing_func_qn) lookup with no guard against the
lookup landing on the shared Folder/Project node for directory-module
languages (Java/Go) -- the exact #787 root cause, just unguarded at two
more sites. A class-level construct (no enclosing function) has its
enclosing_func_qn fall back to the MODULE QN, which for a directory-based
module collides with the package's Folder node, conflating every
same-package file's CONFIGURES/THROWS edges onto one arbitrary source.
- create_env_configures_for_file: added the same
cbm_pipeline_node_is_dir_container guard already used by the three
#828-fixed finders, falling back to the per-file File node on a
directory-container hit.
- resolve_file_throws: was missing the `rel` parameter its sibling
resolve_file_calls/usages/rw already carry, so it inlined a raw lookup
instead of calling the shared, already-guarded find_source_node. Added
`rel` and switched to find_source_node, matching its siblings exactly.
Regression guard: tests/repro/repro_issue842.c, reproduce-first per the
issue's request for "an #828-style multi-order red fixture":
- CONFIGURES reproduces on a small 5-file sequential fixture (Go, since
Java's System.getenv detection is separately broken -- extract_env_key_from_call
reads a "function" field Java's method_invocation node doesn't have; noted
in the test, out of scope here).
- THROWS only reproduces past MIN_FILES_FOR_PARALLEL=50 (the worker-merge
path) -- confirmed empirically that the small sequential fixture already
attributes correctly even on unfixed HEAD, mirroring #787's own two-faced
(sequential/parallel) bug. Padded with 60 filler classes, following
repro_issue787.c's build_parallel_fixture precedent.
RED on unfixed HEAD (both tests): 0 passed, 2 failed. GREEN with this fix:
2 passed, 0 failed. Full scripts/test.sh (5958 cases) passes clean with
this change; scripts/lint.sh's two pre-existing failures (pass_lsp_cross.h
parameter-name mismatch, extract_defs.c cppcheck AST parse) are unrelated
and present identically on unfixed main.
Fixes#842
Signed-off-by: Dhruvy0804 <dhruvgarg553@gmail.com>
Completes the #963 coverage taxonomy: alongside the best-effort FAILURE
signals (parse_partial, skipped), the report now lists what was
deliberately NOT indexed — a deterministic, by-design class, never mixed
into the failure surfaces.
- discover: file skips are classified by reason (gitignore / cbmignore /
skip-list / ignored-suffix / fast-pattern / size-cap) and collected via
the new cbm_discover_ex2 (per-file entries capped at 2000 with the
uncapped total reported — truncation is explicit, never silent; whole
excluded subtrees stay exhaustive as before). Unsupported-extension
files are deliberately NOT recorded: no grammar for an extension is
not an ignore decision, and listing every README/asset would drown
the signal
- pipeline: ignored files captured per run (full + incremental) and
persisted into the index_coverage table as not_indexed_file rows,
excluded dirs as not_indexed_dir; incremental runs rewrite the
by-design set fresh from discovery instead of carrying old rows
- store: the deleted-file prune exempts not_indexed_* (deliberately
unindexed paths never have hash rows); the missed shadow graph keeps
showing FAILURES only — the UI's report-an-edge-case callout can
never fire for gitignored paths, and a project whose only coverage
rows are by-design gets no miss graph at all
- mcp: index_repository response gains not_indexed_files {files, count,
truncated, note} (dirs stay under excluded); index_status's coverage
report gains a not_indexed section (dirs + files + note) and no longer
lumps by-design rows into skipped; tool descriptions updated — the
by-design class is presented as deliberate and deterministic (change
the ignore rules and re-index), with no best-effort hedging
- tests: e2e guard (gitignored file + dir reported with reasons on both
surfaces, failure surfaces stay clean, missed graph excludes them,
rows survive an incremental re-index, ignored constructs genuinely
absent from the graph); store unit test covers the prune exemption
and shadow-graph exclusion
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The UI security audit (security-ui.sh, layer A1) forbids hardcoded
external URLs in graph-ui source — the callout's GitHub issues link
tripped it on all pr-smoke legs. Follow the established pattern for
external targets (/api/repo-info deep-links): /api/ui-config now
carries upstream_issues_url and the callout consumes it, rendering the
issue button only when the backend provides an https URL. The protocol
check uses a regex literal on purpose: a bare protocol string in source
also aborts the audit's URL extraction.
Verified: security-ui.sh passes locally; /api/ui-config serves the URL;
the callout opens with the prefilled link in the browser; frontend
build + tests green.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Per review: no dedicated coverage tool.
- index_status now carries the coverage report (parse_partial + skipped
lists from the separate index_coverage table, plus the best-effort
coverage_note); get_index_coverage is removed. Tool description
updated so agents learn the coverage semantics from index_status;
index_repository and the response note now point there
- the read hook resolves coverage through index_status (same envelope
fields, unchanged note wording); verified end-to-end against a real
index (flagged file -> note with exact ranges, clean file -> silence)
- get_code_snippet responses from a parse_partial file now carry a
correlated coverage_note ("line range(s) X could not be parsed ...
the source above is ground truth") — the result names its file, so
the warning is precisely anchored. Entirely-skipped files cannot
appear here (no nodes), so no uncorrelated noise is possible; clean
files carry no note (asserted both ways in the e2e test)
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Render the miss graph side by side with the code graph instead of
swapping layouts:
- /api/layout (code graph) now also attaches "missed_graph": {nodes,
edges, offset} — the shadow-project layout placed below the primary
cluster (same satellite pattern as linked_projects; -Y slot so
cross-repo satellites collide last). graph=missed stays the isolated
view for API users
- the UI paints the skeleton white and ghostly beside the galaxy,
auto-frames BOTH clusters on load, and navigates naturally: clicking
the skeleton flies the camera into it (file labels + detail panel),
clicking a code node flies back to the code side, and clicking empty
space while the skeleton has focus returns to the overview (the
galaxy can be entirely off-screen at that point)
- the sidebar toggle now shows/hides the skeleton (with a missed-file
count) rather than swapping the layout
Verified in the browser: overview composition, skeleton focus with
split.c labeled, and the empty-space return flight; ui/httpd suites and
frontend build + tests green.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Four refinements to the coverage signal:
- extraction: subtract DEFINITE recovery before flagging. Tree-sitter
error recovery plus the ERROR-descending def walker often re-extract
constructs inside a failed region (verified: `def broken(:` comes
back as a def); a region whose every line is covered by definitions
that START inside it is not a miss, and a fully recovered file is not
flagged at all. Container defs (Module/Package) don't count as
evidence, and partially covered regions stay flagged — the
#ifdef-split case keeps its flag because the first branch's function
is genuinely lost
- naming: the query_graph option is graph="missed" (the graph shows
ONLY misses — "coverage" was misleading); shadow project renamed to
"<name>::missed"; tool descriptions updated so agents discover both
the option and its semantics
- hook: the CLI-installed PreToolUse augmenter now also matches Read
and injects a coverage note when the file being read is listed as not
fully indexed ("line ranges X-Y could not be parsed — the file
content you are reading is ground truth"). Safe against the old
issue-362 hazard: the augmenter is structurally non-blocking (always
exit 0, additionalContext only), mirroring the Gemini matcher that
already includes read_file; matcher upgrade bookkeeping updated
- ui: "Missed files" toggle in the graph sidebar renders the miss
graph as a second graph option — /api/layout gains graph=missed
(same db file, shadow-project scoping; base project name validated
as before)
Tests: recovery-subtraction cases (recovered def unflagged, garbage
region flagged, trailing recovered defs keep the flag), CLI matcher
tests updated, e2e resilience fixtures switched to an unrecovered
miss; verified end-to-end: hook emits the note for a flagged file and
stays silent otherwise, /api/layout?graph=missed serves the miss graph
and the code graph is unchanged, frontend build + tests green.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Persist the parse-coverage signal outside the graph and make the misses
queryable both as a list and as a graph.
- store: new index_coverage table (project, rel_path, kind, detail) —
coverage is metadata ABOUT the graph and never mixes into the graph
tables; replace/get/free APIs with a deleted-file prune keyed off
file_hashes (the live-file set)
- store: materialize a derived miss graph under the shadow project
"<project>::coverage" (Project -> Folder chain -> File{kind,detail},
CONTAINS_FOLDER/CONTAINS_FILE edges), rebuilt from the table inside
the same transaction on every replace; invisible to list_projects
(which scans the cache directory, not the projects table)
- pipeline: coverage is written in the same persist step as every other
post-dump SQL write (hashes, ADR, FTS). Full runs replace the whole
set; incremental runs merge rows surviving from files not re-extracted
with this run's fresh entries, so a fixed file's flag clears and
deleted files prune automatically
- mcp: new get_index_coverage tool (per-project parse_partial + skipped
lists with the best-effort note); query_graph gains graph="coverage"
to run the same cypher against the miss graph; the File-node property
marker from the previous commit is dropped in favor of the separate
table — the code graph gains no coverage rows
- wording: constructs in flagged ranges MAY be missing — tree-sitter
error recovery still salvages some (verified: a function inside a
flagged #ifdef region and even a broken def were still extracted)
- tests: store round-trip/prune/shadow-graph unit test; e2e asserts the
response fields, the advertised coverage-graph query, code-graph
purity, and that the flag clears after the file is fixed (incremental
route)
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Files whose parse tree contains tree-sitter ERROR/MISSING regions were
silently indexed as if complete — constructs inside those regions are
absent from the graph with no signal anywhere (classic trigger: the
preprocessor-blind #ifdef-split-brace pattern in C).
- cbm_extract_file: set parse_incomplete and record the 1-based line
ranges of the top-most error regions (bounded at 64), serialized as
"start-end,start-end" into the result arena
- pipeline: record such files under the new phase "parse_partial"
(distinct from skips — the file IS indexed) in both the parallel and
sequential extraction paths; stamp the File node with
{parse_incomplete:true, error_ranges} on full and incremental runs
via a new cbm_gbuf_set_node_props (upsert survivor rules bypass)
- mcp: report parse_partial {files, count, truncated, note} separately
from skipped[] (skipped_count semantics unchanged); the per-run
logfile lists parse_partial rows; the index_repository tool
description documents the signal and the query_graph pattern to find
flagged files
- tests: new parse_coverage suite (RED-first, 7 cases incl. false-
positive guards and the region cap) plus an end-to-end resilience
guard asserting response fields, logfile, File-node marker, and the
advertised query_graph query
The signal is explicitly best-effort: a flag means constructs in the
listed ranges were dropped (prefer grep there); the absence of a flag
is NOT a completeness guarantee.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_path_within_root used an #ifdef _WIN32/#else that opened the if(...) {
brace in each branch and shared a single close brace. That is legal C, but it
splits the function body's braces across preprocessor branches, so a
source-level parser running without the preprocessor (e.g. the tree-sitter
code indexer) sees unbalanced braces, emits an error node, and never creates a
Function node for it — the definition went missing from the graph while its
callers stayed linked to an unresolved name.
Extract a small per-OS resolve_canonical_path() helper so the function has one
unconditional control flow. Behavior is unchanged (the path-containment
security tests still pass); it now indexes correctly.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fold the inline CBM_MEM_BUDGET_MB override logic in cbm_mem_init into a pure,
testable cbm_mem_resolve_budget() that returns a result struct
(budget/source/clamped/invalid), so the parse + clamp lives in exactly one
place and cbm_mem_init only surfaces the outcome as log lines.
Parsing now matches the strict src/foundation/limits.c convention (errno/ERANGE,
reject trailing garbage, positive-only): a fat-fingered value like "8GB" or a
20-digit typo becomes a warning + safe ram_fraction fallback instead of a
silently wrong budget. A valid-but-huge value clamps to detected total RAM
(logged mem.budget.clamped) rather than overflowing the MiB->bytes multiply and
wrapping to a near-zero budget (which would pin cbm_mem_over_budget() true and
stall indexing). mem.init now logs source= on both the override and fraction
paths, and the README gains the CBM_MEM_BUDGET_MB row it lacked.
Hardening of the existing #363 knob.
Signed-off-by: Sam Li <yangsec888@gmail.com>
cbm_cli_sha256_file shelled out to sha256sum/shasum, which is unportable:
the tool may be absent, and under cmd.exe the single-quoted path is not
dequoted so the hash of a downloaded release could never be computed on
Windows. Add a small in-process SHA-256 (FIPS 180-4, src/foundation/sha256.c)
and use it directly — no external tool, no per-OS branching, no popen. The
checksum test now covers three NIST vectors (empty, "abc", and a 56-byte
input that forces padding into a second block).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The layout endpoint treated the default render cap (2000) as a ceiling:
an explicit max_nodes above it was silently clamped back down, so the UI
could never show more than 2000 nodes regardless of the request. Make the
default a default — an explicit request is now honored up to a hard
ceiling of 10M nodes; CBM_UI_MAX_RENDER_NODES still adjusts the ceiling
for constrained deployments. The default budget rises to 5000, and the
layout refinement pass scales its iteration count down past 100k/500k
bodies to keep very large budgets responsive. Edges need no changes: the
layout already returns every edge between loaded nodes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The policy was only attached to embedded-asset responses, but browsers
enforce a CSP from the document response; on subresources it is inert.
Attach the same header where index.html is served.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The UI makes no external network calls, but nothing enforced that at the
browser. Emit a CSP whose directives name no external host, so a future
dependency or injected content cannot load or connect off-origin;
connect-src 'self' confines fetch/XHR/WebSocket to the local server. The
'self'/data:/blob:/inline-style/wasm allowances cover the bundled app's own
needs (React inline styles, three.js textures/workers). Header emission only,
no rendering change expected — confirm the 3D view in a browser before
release.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
index_repository canonicalizes repo_path but imposes no workspace boundary,
so a caller-supplied path can resolve anywhere on disk. Add an opt-in
CBM_ALLOWED_ROOT: when set, a repo_path that resolves outside it is refused
(reusing the same containment check the read sinks use). Unset by default, so
the standard index-the-given-path behaviour is unchanged; deployments where
repo_path may be influenced by an untrusted caller can set a boundary.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The UI handlers accumulate JSON with `pos += snprintf(buf + pos, size - pos,
...)`. snprintf returns the length it WOULD have written, so on truncation
pos ran past the end of the buffer and the next call's `size - pos` wrapped
to a huge value and wrote out of bounds — reachable through the directory
browser, whose readdir entries can fill the buffer before the trailing
parent/roots appends. Route every accumulator through a clamping
http_appendf() helper that pins pos at the buffer size on truncation. Adds a
forked-child regression test that browses a directory wide enough to fill the
buffer.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The macOS digest path invoked `shasum -a CBM_SZ_256` — a literal string
(not a macro; string literals do not expand) naming an invalid algorithm, so
the digest always failed on macOS, and the self-update path then treated a
could-not-verify result as non-fatal and installed the download regardless.
Use the correct algorithm, and install only a positively-verified download:
a mismatch, a missing checksum entry, or an unavailable hash tool now all
abort. Adds a known-vector digest regression test.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
base_branch is spliced into `git diff --name-only "<base>"...HEAD`. The
existing check rejected shell metacharacters but not a leading '-', so a
value like --output=<path> was passed to git as an option and wrote the diff
to an arbitrary file. A real git ref never starts with '-'; reject it
alongside the metacharacter check.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The graph-UI server binds to loopback only and reflected a localhost Origin
for CORS, but did not inspect the Host header — so a request routed to the
local port under a foreign name (a rebinding DNS record, a proxy) was served
normally, reaching the state-changing endpoints. Parse the Host header and
refuse any request whose Host is not a loopback address before dispatch; a
request with no Host (local HTTP/1.0 tooling) still proceeds. Adds a
live-socket regression test.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A source path never legitimately contains a newline or carriage return, and
on the Windows search filelist those bytes are the per-entry record separator
(NUL can't be used there — it breaks PowerShell Get-Content). Skip any indexed
path containing one so a crafted path cannot inject an extra file into the
scan set.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
safe_stat rejected POSIX symlinks (lstat + S_ISLNK) but on Windows returned
wide_stat directly, which follows junctions / reparse points — so a junction
inside an indexed tree could be walked into and files outside the project
root pulled into the index. Check FILE_ATTRIBUTE_REPARSE_POINT and skip such
entries, mirroring the POSIX symlink skip.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
get_code_snippet resolved a result path and confirmed it stayed under the
project root before reading, but the search_code source-attach path built
the same kind of path and read it with no such check — so a result whose
indexed path resolves outside the root (a `..` segment, or a symlink /
junction followed during discovery) could be read back into the response.
Extract the existing containment check into a shared cbm_path_within_root()
and route both read sinks through it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The import path allocated the destination from the plaintext original_size
field in artifact.json and passed that value to the decoder truncated
through int. A large real database already wraps that cast (a full-kernel
index is ~14 GB), and a doctored field could additionally desync the decoder
capacity from the actual buffer. Read the decompressed size from the zstd
frame's own content-size header, require the metadata field to match it, cap
the total, and run the decoder with a size_t capacity equal to the
allocation. cbm_zstd_decompress now takes size_t lengths and returns int64_t.
Adds a regression test that a mismatched original_size is rejected.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The result projection is materialized per row into fixed-width stack
arrays (CBM_SZ_32 columns) in execute_return_simple and its siblings,
indexed by the parsed RETURN item count — which the parser left
unbounded. A RETURN/WITH clause with more than 32 projection items
wrote past those arrays. Reject an over-wide projection at parse time,
with a fork-isolated regression test driving a 48-column RETURN.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An 81k-file TypeScript corpus previously never finished indexing. Three
stacked defects, found by stack-sampling the live grind and reading the
supervisor's quarantine records:
1) walk_defs pushed children via index-based ts_node_child(i), which is
O(i) per call in tree-sitter — O(n^2) per wide node. A 3.5 MB file
with ~580k flat comment siblings needed ~1.7e11 iterator steps and
hung extraction past the supervisor's quiet-timeout. All six descent
sites (including the default loop) now collect children linearly with
a TSTreeCursor (wd_collect_children); small nodes keep the direct
index path. Guard: extract_wide_flat_file_is_linear (400k comment
siblings — the monster's exact shape — bounded at 30s).
2) The sequential cross-LSP driver handed the FULL def list to a full
per-file registry build+finalize — O(files x defs); 74% of samples
sat in build_qn_index. The parallel resolve worker's dispatch
(module-def filter -> shared prebuilt registries -> filtered
fallback) is now extracted into cbm_pxc_dispatch_file and drives
BOTH pipelines — one path, one semantics. The shared registries
live in a caller-owned arena that outlives the calls pass
(resolved_calls borrow registry strings; freeing at pass end was a
use-after-free ASan caught in pass_calls). Rust is exempt from the
def filter: Cargo-manifest cross-crate resolution needs defs from
OTHER workspace crates that the own+imports filter drops (this
starvation predated the unification on the parallel path). Guard:
pipeline_seq_ts_cross_uses_shared_registry (full builds 40 -> 0,
cross-file edge preserved).
3) The crash supervisor's recovery re-ran the worker SINGLE-THREADED to
keep one exact marker. At scale that fell into the sequential crawl,
was killed as a hang mid-pass, and the stale extraction marker got
four innocent files quarantined, one 15-minute retry at a time.
Recovery (and the terminal partial run) now re-run PARALLEL; the
marker is an append journal ("S <rel>" / "D <rel>", written by
extraction and both resolve drivers), the suspect set is the open-S
in-flight set, and a file is quarantined only when it recurs across
two consecutive failed runs — one per round, oldest open S first;
disjoint consecutive sets stop the loop rather than blame an
innocent. Guard: index_recovery_parallel_quarantines_crasher
(single-threaded spawn count must stay zero; verified RED against
the old loop).
Also: parse_ts_type_text gains a dynamic work budget proportional to
the input (1M + 64x source bytes, CBM_TS_TYPE_BUDGET override, warn
once then degrade to unknown) replacing unbounded recursion cost.
Measured on microsoft/TypeScript (81,398 files): infinite -> 33s cold
full index, 295k nodes / 779k edges, zero dangling edges, zero files
skipped — the monster file itself now indexes cleanly. Full 9-language
sweep green with no quality flags; kernel unchanged at 390s / 8.5M
nodes / 0 dangling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel