cbm_resolve_cache_dir() returns a pointer to a static thread-local buffer. The
cache prune called it for every directory the walk visited, so it rewrote a buffer
other code was entitled to still be holding — an aliasing bug, not a cosmetic
inefficiency.
The Windows guards job caught it as two REDs: a daemon cold-storm regression and
a following "index did not run" setup failure. Both are consistent with the cache
path being rewritten underneath a live reader during indexing.
Resolve once at the discovery entry point into a thread-local copy and compare
against that, so the prune costs one resolver call per walk rather than one per
directory, and holds no pointer into shared storage.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two loose ends from the bounds work.
The decompression side already took size_t so a >2 GiB capacity could not wrap
through int, but compression still took int for both the source length and the
destination capacity, and the artifact export cast a size_t database size down
to reach it. A database past 2 GiB would have handed the encoder a negative
length. Both lengths and the bound helper are size_t now, and the function
returns int64_t like its decompressing counterpart.
The discovery walk now prunes the cache directory by absolute path. A custom
CBM_CACHE_DIR may sit inside a repository — tests do it routinely — and walking
into it pulls every other project's graph database into this project's file
list. This is the narrow form of a concern that was briefly implemented as
refusing any root that contained the cache; refusing a whole root was too blunt,
and not walking the cache is what the concern actually asks for.
The test fails without the prune: a .go file planted under the cache is
otherwise discovered, and "cache" is not in the built-in skip list, so the
assertion is not vacuous.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A pass over places where a bound was either miscalculated or missing.
UI response builders (src/ui/http_server.c) mixed the clamping helper
http_appendf with raw cursor indexing. Route the per-line separators and
quotes through the helper too, so the cursor cannot leave the buffer once
output saturates it, and terminate explicitly afterwards — the helper pins
the cursor and then writes nothing, which would otherwise leave the "%s"
reply with no terminator in range. The log buffer was budgeted at
LOG_LINE_MAX + 10 per line while JSON escaping can double every byte, so
size it for the escaped worst case. Job status entries now escape their path
and error fields, which were interpolated raw.
Cypher (src/cypher/cypher.c): the WHERE grammar descends once per nested '('
and once per NOT, so parse depth followed the query text rather than
anything bounded. Track depth on parser_t and refuse past 256 levels with a
parse error. Bounding parse depth also bounds the resulting tree, so the
recursive evaluator inherits the limit.
Gitignore (src/discover/gitignore.c): '**' retries the remainder at every
position and consecutive groups multiply, so match cost is exponential in
the number of groups. Thread a step budget through the matcher and give up
past 20000 steps, reporting no-match so a pathological pattern fails to
ignore rather than ignoring the wrong files. glob_match keeps its name so
recursion_whitelist.h still describes the functions that recurse; the new
non-recursive wrapper that seeds the budget is glob_match_bounded.
Call extraction (internal/cbm/extract_calls.c): extract_fp_callee recursed
once per applied argument, so stack use followed the parse-tree depth of the
indexed file. Rewritten as a left-spine loop. Reassigning node/nk before
continuing leaves the fall-through cases where the recursive form left them,
so behaviour is unchanged and the extraction suite is untouched.
CALLS props (src/pipeline/pass_calls.c): `cap - pos - PAIR_LEN` wraps once
pos reaches cap - PAIR_LEN and stops bounding the copy below. Guard
additively, matching the closing write in the same function.
Shell-bound paths (src/foundation/str_util.*): the three git shell-out sites
each wrap a repo path in cmd.exe-compatible double quotes, where %VAR%,
!VAR! and ^ remain active. Two carried private copies of the check that
rejects those three on Windows and the third used the bare validator.
Promote the stricter form to cbm_validate_shell_path_arg and route all three
through it, so the copies cannot drift apart again.
Tests cover the buffer bounds, the parse-depth refusal, matcher termination
and the validator. The bounds tests run in a forked child so a violation
surfaces as a signal instead of silently.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A directory with more immediate subdirectories than the fixed 512-frame
walk stack made the whole discovery walk hard-fail with files=0 and a
misleading 'check repo_path' hint — dotnet/runtime carries 855 sibling
dirs in one JIT regression folder and could not be indexed at all. The
stack now grows geometrically from the same initial capacity; the parent
frame is a popped copy, so growth cannot dangle it. Covered by a 600-dir
fanout reproduction.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.
Daemon lifecycle:
- daemon start/stop/status subcommands. `daemon start` launches a
PERMANENT daemon (spawn shape is byte-exact argv; survives idle
periods and session ends) and reports an already-active daemon
instead of failing. Permanence is honored at every stop latch:
last-committed-client disconnect, host initial-client window,
coordinator release, and application final-session close — a
permanent daemon also keeps admitting new sessions after its last
one closes.
- daemon stop refuses while sessions are active and lists the blocking
peers (pid/role) that must finish first; an idle daemon drains
through the activation-shutdown machinery with the ACK ordered after
connection interrupts. A second stop is idempotent. The wire ops are
no-cohort first-frame requests with peer fingerprint authentication,
so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
keep their local supervised path). A cold CLI run that had to spawn a
temporary daemon prints a hint that `daemon start` removes the
per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
emits a visible, rate-limited notice (Claude-dialect systemMessage
plus stderr for other dialects) and always exits 0 — augmentation is
never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
older-build permanent daemon (strict semantic-version triples only;
dev builds never auto-drain) and the build-conflict message names
`cbm daemon stop` as the manual escape hatch.
Windows IPC/runtime (real-VM verified):
- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
path used to destroy its listening pipe instance on every 20 ms poll
timeout; a client attaching in the teardown window was severed or left
on an orphaned pipe object whose HELLO no server handle could ever
read, absorbing the connect until the client's own timeout expired.
The pending connect now survives poll timeouts and nothing is
destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
server handle can discard a just-sent response before the peer reads
it (POSIX stream sockets never lose buffered data on close). A bounded
cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
immediate on interrupted connections) now precedes close in
runtime_worker_finish and runtime_reject_inline, so hello-conflict,
capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
transport half-close, so close_begin now announces departure with an
explicit frame (ordered after APPLICATION_CANCEL, before the local
interrupt); the server releases the client's admission on receipt
instead of waiting for the handle to close. Admission-drop timing is
now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
request publication under the send lock, best-effort sends the active
token's APPLICATION_CANCEL frame, then interrupts local I/O; the
server cancels MCP/subprocess work promptly. Contract tests accept
both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
the requester's license to act on "snapshotted and draining", so every
connection interrupt is now initiated before the ACK is sent; a
session could previously get one more request serviced after the
requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
timer granularity) burned ~1.6 s against permanently obstructed paths,
stalling hello rejections past the client's timeout. The retry budget
is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
nonblocking send() on Windows is absorbed wholesale into AFD kernel
buffering regardless of SO_SNDBUF, so send deadlines and interrupts
could never engage against a slow peer (and the full payload was
pinned in nonpaged pool). Slicing restores a deterministic
backpressure point; a test hook pins SO_SNDBUF for the deadline and
interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
deadline, explicit connection states, refusal to free a server while
a listener-owned connection is active.
Windows long-path support:
- Central path-aware wide conversion (canonicalize via GetFullPathNameW
and prepend the extended-length prefix for absolute paths >=240) at
the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
store opens, and the daemon build-fingerprint/log paths. Deep managed
installs (a 64-hex generation directory routinely exceeds MAX_PATH)
now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
ancestry validators now operate in the extended-length namespace;
the launcher path is canonicalized (and prefixed when deep) once at
entry so every downstream exact-string comparison stays
form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
GetFileAttributesW) so non-ASCII install paths survive argv[0]
resolution.
Windows launcher install/uninstall transaction:
- FileRenameInfoEx names are NUL-terminated in an over-allocated
buffer. FileNameLength governs per the contract, but filter drivers
read FileName as NUL-terminated and appended adjacent heap bytes to
created names — a flaky, garbage-suffixed rename target. Both the CLI
and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
.cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
chars so the bare rename target stays under the FileRenameInfoEx
NT-conversion ceiling at guard depths; 64 bits still uniquely
identify the generation.
- When the running launcher's mapped generation backings pin .cbm
against rename, the backings are relocated to activation-<pid>-N
.retired tombstones beside the install (a mapped image may be renamed,
never deleted; the launcher's liveness-guarded sweep reclaims stale
tombstones). Every relocation is recorded, and a FAILED uninstall
reverses the moves after restoring .cbm — via MoveFileExW with
extended-length paths on both arguments, since the deep generation
target is beyond the handle-based rename's bare-path reach — so a
restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
out so the tree is shallow enough for the detached cleanup's rd, and
the cleanup's working directory strips the extended-length prefix
(CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
(CopyFileW destinations, CREATE_NEW tombstones, probe directories)
are explicitly owner-stamped so the exact-owner validators hold on
runner images; guard fixtures stamp hand-built trees the same way.
Diagnostics, tests and infra:
- diagnostics: discovery is now an always-delivered JSON control record
(new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
paths containing spaces; placement honors $TMPDIR with /tmp fallback
via a diagnostics-local helper; the soak parser reads the JSON record;
documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
log levels, TMPDIR-with-spaces, and native Windows output-contract
assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
parents with production-shaped ancestry (LocalAppData on Windows, via
th_secure_runtime_parent_new) — the runtime ancestry validation
correctly refuses temp roots whose ancestors grant mutation rights to
Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
drive the documented startup-owner publication flow before reading
generation-bound endpoint addresses. This turns the 26 Windows
failures previously visible in CI's full-test job green without
weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
contract including failed-uninstall restore and immediate reinstall
after uninstall; new daemon lifecycle and reworked hook-augment
guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
(it was set by CI but never read, so perf suites ran everywhere);
four throughput/bench suites are classified as perf, the heavy
store_arch suite moved to the slow-timeout tier, and two
wall-clock-sensitive assertions were rewritten as invariant checks
with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
contract, soak daemon-recovery contract, path-safety helper, the
Windows VM worktree-sync contract wired into scripts/test.sh, and
vm/win.sh guards building its clean embedded-UI product in an
isolated BUILD_DIR so it cannot clobber the incremental test build.
provision-windows.sh now installs Node.js for the guards UI build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two review items applied maintainer-side to save a roundtrip:
1. Restore both vendored ObjectScript scanner.c files to the pristine
upstream bytes (a formatter pass had rewritten indentation/braces).
Vendored grammar files stay byte-for-byte upstream except for the
include repoint documented in MANIFEST.md - style churn breaks
upstream diffability on re-vendor.
2. The Studio Export XML sniff in detect_file_language used a raw
fopen on a repository path; cbm_fopen keeps it working on
non-ASCII Windows paths (project rule, same class as #996).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Add ObjectScript (InterSystems IRIS / Caché) as a supported language,
covering the UDL class format (.cls), MAC/INT routines (.mac/.int/.rtn),
include/macro files (.inc), and IRIS Studio Export XML.
Definition extraction (extract_defs.c): Class, Method, ClassMethod,
Property, Parameter, Index, Trigger (with body text), XData, Storage,
and Query members as graph nodes; base classes from the Extends clause.
Call dispatch resolution (extract_calls.c) — four ObjectScript patterns
that are structurally invisible to text search:
1. ##class(Pkg.Class).Method() explicit cross-class call
2. ..Method() relative-dot self-call (the dominant
intra-class form; large impact on
CALLS completeness)
3. $$$Macro macro expansion via a per-project
table built from .inc files
4. type inference from %New/%OpenId + declared return types
Ensemble production topology (pass_ensemble_routing.c): EnsembleItem
nodes per production component and ROUTES_TO edges resolved from
ProductionDefinition XData, plus WorkMgr .Queue("##class(X).method")
dispatch — all parsed statically at index time, no live IRIS required.
Language detection (language.c): .mac/.int/.rtn map to ObjectScript
routine directly; .cls (shared with Apex) and .inc (shared with BitBake)
are disambiguated by content, defaulting to the existing language on any
doubt so neither Apex nor BitBake detection regresses.
The two new per-project tables (macros, return types) are threaded
through a new internal cbm_extract_file_ex() so the public
cbm_extract_file() signature is unchanged.
The tree-sitter grammars for ObjectScript UDL and routine are vendored
in internal/cbm/vendored/grammars/objectscript_{udl,routine}/ from
https://github.com/intersystems/tree-sitter-objectscript (MIT, ABI 15).
Refs #462
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
Completes the #963 coverage taxonomy: alongside the best-effort FAILURE
signals (parse_partial, skipped), the report now lists what was
deliberately NOT indexed — a deterministic, by-design class, never mixed
into the failure surfaces.
- discover: file skips are classified by reason (gitignore / cbmignore /
skip-list / ignored-suffix / fast-pattern / size-cap) and collected via
the new cbm_discover_ex2 (per-file entries capped at 2000 with the
uncapped total reported — truncation is explicit, never silent; whole
excluded subtrees stay exhaustive as before). Unsupported-extension
files are deliberately NOT recorded: no grammar for an extension is
not an ignore decision, and listing every README/asset would drown
the signal
- pipeline: ignored files captured per run (full + incremental) and
persisted into the index_coverage table as not_indexed_file rows,
excluded dirs as not_indexed_dir; incremental runs rewrite the
by-design set fresh from discovery instead of carrying old rows
- store: the deleted-file prune exempts not_indexed_* (deliberately
unindexed paths never have hash rows); the missed shadow graph keeps
showing FAILURES only — the UI's report-an-edge-case callout can
never fire for gitignored paths, and a project whose only coverage
rows are by-design gets no miss graph at all
- mcp: index_repository response gains not_indexed_files {files, count,
truncated, note} (dirs stay under excluded); index_status's coverage
report gains a not_indexed section (dirs + files + note) and no longer
lumps by-design rows into skipped; tool descriptions updated — the
by-design class is presented as deliberate and deterministic (change
the ignore rules and re-index), with no best-effort hedging
- tests: e2e guard (gitignored file + dir reported with reasons on both
surfaces, failure surfaces stay clean, missed graph excludes them,
rows survive an incremental re-index, ignored constructs genuinely
absent from the graph); store unit test covers the prune exemption
and shadow-graph exclusion
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
safe_stat rejected POSIX symlinks (lstat + S_ISLNK) but on Windows returned
wide_stat directly, which follows junctions / reparse points — so a junction
inside an indexed tree could be walked into and files outside the project
root pulled into the index. Check FILE_ATTRIBUTE_REPARSE_POINT and skip such
entries, mirroring the POSIX symlink skip.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A .cbmignore negation pattern (e.g. "!obj/") whose rule is the last
match for a directory now un-skips built-in skip-list dirs, so projects
can index sources that live under names like obj/, target/, or docs/
(issue #500). Distills the design and test intents of PR #543 onto the
current 3-state cbm_gitignore_match_result API: instead of a second
inverted-polarity matcher, should_skip_directory consults the existing
last-match-wins result (<0 = negated) inside the built-in branch and
falls through so .gitignore/global/local rules still apply.
Non-negatable safety core: .git, node_modules, and the worktree-internal
dirs (.worktrees, .claude-worktrees) can never be un-skipped — a
repo-committed .cbmignore must not defeat OOM/safety skips (#489, #802).
Un-skipped dirs are walked normally and therefore no longer appear in
the excluded-subtree list (#411 coherence).
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Follow-up to #683 (@Naam). The worktree info/exclude fix read the `.git` gitlink
file and its `commondir` with raw fopen(); on Windows raw fopen uses the ANSI
code page and fails to open non-ASCII (e.g. CJK) repo paths, so a worktree
indexed under such a path would silently not honor info/exclude /
core.excludesFile. Switch both reads to cbm_fopen() (the UTF-8 -> _wfopen
wrapper), matching the rest of discover.c (e.g. read_core_excludes_file at 248).
No behavior change on POSIX (raw fopen already opens UTF-8 byte paths there);
this is a Windows-only correctness/consistency fix, verified by the full test
suite staying green and cross-platform CI. discover.c now has no raw fopen.
Refs #682, #683
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Standard fopen on Windows uses the ANSI/system codepage and silently
fails to open files whose absolute path contains non-ASCII characters
(accented, CJK, etc.). This causes the pipeline to skip parsing those
files entirely — File nodes are created by directory traversal (which
already uses wide Win32 APIs) but tree-sitter never receives the source,
so no Function/Class/Method nodes are extracted.
Add cbm_fopen(path, mode) to compat_fs:
- Windows: converts the UTF-8 path to UTF-16LE via cbm_utf8_to_wide
(existing helper in win_utf8.h) and calls _wfopen.
- POSIX: delegates to fopen (no-op change, paths are already UTF-8).
Replace every fopen call that operates on user project file paths:
- src/pipeline/pass_parallel.c (primary indexing path)
- src/pipeline/pass_definitions.c
- src/pipeline/pass_calls.c
- src/pipeline/pass_semantic.c
- src/pipeline/pass_usages.c
- src/pipeline/pass_lsp_cross.c
- src/pipeline/pass_pkgmap.c
- src/pipeline/pass_k8s.c
- src/pipeline/pass_envscan.c
- src/pipeline/path_alias.c (tsconfig detection)
- src/pipeline/artifact.c
- src/discover/language.c (content-sniff for language detection)
- src/discover/gitignore.c
- src/discover/discover.c
- src/discover/userconfig.c
- src/mcp/mcp.c (read_file_lines, adr.md reader)
Fixes#700
Signed-off-by: ShauryaaSharma <shauryasofficial27@gmail.com>
In a linked worktree, <repo>/.git is a regular file ("gitdir: <path>")
rather than a directory. cbm_discover_ex gated all per-clone ignore
sources on .git being a directory, so inside a worktree it skipped
.git/info/exclude and the global core.excludesfile entirely and walked
into directories the user had excluded there (e.g. build/, twister-out/).
Committed .gitignore still applied via the issue #510 path, which is why
the gap was easy to miss.
Resolve the git common dir up front: when .git is a gitlink file, parse
'gitdir:', then follow <gitdir>/commondir to the shared dir where git
keeps info/exclude and config (falling back to the gitdir when no
commondir file is present). info/exclude and config are then read from
that common dir, so a worktree behaves like an ordinary checkout.
Adds discover_worktree_info_exclude (fails before this change) and
discover_worktree_committed_gitignore.
Signed-off-by: Nahim El Atmani <2959826+Naam@users.noreply.github.com>
QUALITY_ANALYSIS gap #1: .claude-worktrees was indexed (nearly tripling this
repo's indexed surface + polluting queries with stale duplicate symbols).
Discovery skipped .claude and .worktrees but not .claude-worktrees. Add it to
ALWAYS_SKIP_DIRS. Flips the discovery-hygiene invariant green.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_gitignore_merge copied src patterns into dst one strdup at a time and
returned void on failure, leaving dst with a partial set of patterns and no
signal to the caller. If the dropped pattern was the critical exclude (the
.git/info/exclude entry that prevents an OOM walk), the bug could silently
recur with some patterns applied and others dropped.
Make the merge atomic: on any allocation failure roll back the partial copies
so dst is left exactly as it was, and return bool so callers can tell. The
.gitignore caller now degrades gracefully (exclude patterns skipped, as if the
file were absent) instead of ending up with a half-merged matcher.
Adds a reproduce-first test that injects a strdup failure mid-merge via a test
seam and asserts dst is unchanged; it is red without the rollback.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Drop the repo-local .git/config pass in resolve_global_excludes_path() so
an untrusted repo cannot point core.excludesFile at an arbitrary file (e.g.
~/.ssh/id_rsa) and have cbm open and read it merely by being indexed. git is
safe here because it only trusts a repo's .git/config after a deliberate
clone/init; the indexer has no such trust step. Global excludes now resolve
only from the user-owned ~/.gitconfig and $XDG_CONFIG_HOME/git/config. A
repo's own ignores remain handled by .gitignore / .cbmignore.
Add a guard test asserting a repo-local .git/config whose excludesFile points
outside the repo is ignored, not read.
Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* fix(discover): honor .git/info/exclude during file discovery
index_repository only loaded .gitignore when building the exclusion
matcher. .git/info/exclude — the per-clone exclude file that Git treats
as authoritative — was never read, so paths excluded only there were
walked in full. On repos with Sandcastle worktrees this caused the
indexer to traverse 661K files / 170 GB and OOM (issue #489).
Add cbm_gitignore_merge() to append patterns from one matcher into
another. In cbm_discover_ex, after loading .gitignore, also load
.git/info/exclude and merge its patterns in. If only the exclude file
exists, use it directly. No downstream call paths change.
Add three unit tests for cbm_gitignore_merge and two integration tests
that reproduce the issue scenario (exclude-only and exclude-stacked-
with-gitignore).
Signed-off-by: ShauryaaSharma <shauryasofficial27@gmail.com>
* style(discover): fix clang-format alignment in cbm_gitignore_merge
Signed-off-by: ShauryaaSharma <shauryasofficial27@gmail.com>
---------
Signed-off-by: ShauryaaSharma <shauryasofficial27@gmail.com>
deploy/ and deployed/ build artifact directories were in FAST_SKIP_DIRS,
which only applies in fast/moderate indexing modes. In the default full
mode they were indexed alongside source files, creating duplicate
Class/Method rows with stale data. Any MATCH (c:Class {name:'Foo'})
without a file_path filter returned multiple rows — one from source,
one from the build copy.
Move deploy and deployed to ALWAYS_SKIP_DIRS so they are excluded in all
indexing modes.
Regression test: discover_deploy_excluded_full_mode in test_discover.c.
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
Removes the heaviest vendored grammar (66 MB) along with its language
wiring and test fixtures; the supported-language count moves to 158
across README, site, and manifest. The grammar manifest additionally
records the canonical-source decisions for the five
registry-disagreement grammars and the per-directory license files
restored earlier this week.
index_repository silently dropped entire subtrees (ALWAYS_SKIP dirs in all modes; FAST_SKIP dirs like tools/scripts/bin in moderate/fast) with no record — users couldn't tell what wasn't indexed. The discover walk now records skipped directory rel-paths; cbm_discover_ex returns them; the pipeline holds + exposes them; and the index response carries a compact summary: "excluded":{"dirs":[up to 25],"count":N,"truncated":bool}. Behavior-neutral for indexing. Adds a reproduction in test_integration.c.
Helm chart helpers (.tpl) and Go templates produced no semantic graph — just a
flat key flood. Wire the existing gotemplate grammar for real structure:
- .tpl files map to the gotemplate language (Helm _helpers.tpl).
- {{ define "chart.fullname" }} becomes a Function node named after the
template (the name is a string literal, not an identifier, so it's resolved
in a dedicated walker branch).
- {{ include "x" . }} and {{ template "x" . }} resolve to the referenced
template name instead of the bare `include` builtin, so they form CALLS
edges to the define'd Function. template_action is added to the call set.
Lets you trace which charts include which named templates. Tests cover define
-> Function and include/template -> CALLS, plus the .tpl mapping.
Refs #338
Vendor the MIT-licensed cfmleditor/tree-sitter-cfml grammars and wire both
CFML dialects through the pipeline:
- .cfc components -> cfscript grammar (CBM_LANG_CFSCRIPT): a JS-like grammar, so
function/method declarations, calls and imports extract with the shared JS/TS
arrays. Components' functions become Function nodes.
- .cfm templates -> cfml tag grammar (CBM_LANG_CFML): tag-based (HTML-derived).
Embedded <cfscript> functions extract via function_declaration; tag-based
<cffunction name="..."> nodes are handled by a dedicated walker branch that
reads the name attribute (cf_function_tag has no name field). Shared scanner
headers are vendored at vendored/common/ so the grammar's ../../common/
includes resolve without editing vendored sources.
Adds an ASCII case-insensitive compare for CFML's case-insensitive attributes.
Tests cover both dialects and the extension mapping. LICENSEs vendored.
Closes#38
Vendor the MIT-licensed tree-sitter-qmljs grammar (a TypeScript superset with
declarative ui_* nodes) and wire .qml files through the full pipeline: language
enum, extension + display-name mapping, grammar shim, and a language spec that
reuses the JS/TS function/call/branch arrays and adds QML class/field/import
types (ui_inline_component, ui_property, ui_signal, ui_import).
QML files now produce Function nodes (incl. JS functions inside QML objects),
class/interface/enum nodes, imports and CALLS edges, so call chains through a
Qt project's .qml files can be traced. LICENSE vendored alongside the grammar.
Closes#42
Laravel Blade templates (*.blade.php) fell through to the single-extension
lookup and were mis-classified as PHP, so the PHP extractor tried to parse
Blade syntax. Add a built-in compound-extension table (checked before the
single-extension fallback and alongside user config) mapping .blade.php to
CBM_LANG_BLADE, so Blade files are discovered and parsed with the Blade
grammar. Combined with the now-non-blocking discovery gate (c29e6d5), this
resolves the "agent stuck on Blade files" report. Plain .php is unaffected.
.mjs (ES modules) and .cjs (CommonJS) were absent from the extension
table, so those files were never assigned a language — they weren't
indexed (no graph nodes) and search_code returned 0 for any pattern in
them. Add .mjs/.cjs → JavaScript and .mts/.cts → TypeScript.
Verified end-to-end: a function in a .mjs file is now extracted and found
by search_code. Closes#197.
- cbm_arena_alloc returns NULL on a NULL arena (both arena.c copies)
instead of dereferencing it — defense-in-depth against the NULL-arena
type-allocation path that could crash the LSP type layer.
- discover: add "vendored" to the always-skip directory list.
Distilled from #374, taking the parts not already on main. The PR's
ts_lsp tuple-arena allocation, the C/C++ template formal-count clamp,
and the type_rep unbound-param preservation it also carried all landed
independently in v0.7.0 / via #322 / #360, so only these two
defensive improvements remain. Relates to #390.
Replace ANSI Windows APIs (FindFirstFileA, CreateFileA, etc.) with
wide-char (W) counterparts. Add UTF-8/UTF-16 conversion helpers.
Fix stat() calls in discover pipeline to use _wstat64.
Fixes#357
The per-file and cross-file LSP type-aware call/usage resolution now runs in every index mode (full/moderate/fast) instead of being gated behind a separate CBM_MODE_ADVANCED. Remove the mode from both index-mode enums, the g_lsp_enabled toggle and its save/restore dance, and the per-mode gating in discover, pipeline and the incremental path.
Now that the LSP walkers are O(n), the callee-attribution accuracy (e.g. ~91k call overrides on roslyn, ~60k on kubernetes) is worth the latency in every mode. Fast mode still filters files and skips similarity/semantic edges, so it stays faster than full while now also benefiting from LSP resolution.
A 'mode':'advanced' request falls back to the full default — identical behaviour to the old advanced mode, so existing callers are unaffected. This also fixes the test_incremental.c failures, which were asserting LSP-resolved results that the gated incremental path never produced.
New CBM_MODE_ADVANCED=3. Both per-file LSP (type-aware resolution
during extract) and the cross-file LSP pass (during resolve) are now
OPT-IN: FULL and below run tree-sitter extraction + registry textual
resolution only; ADVANCED adds the full LSP layer.
Rationale: profiling showed LSP changes the kubernetes graph by only
~0.01% of edges (it mostly re-confirms textual resolution) while
costing significant memory + time. Making it opt-in keeps the default
FULL path lean (3.0 GB vs 5.6 GB peak RSS on K8s) without meaningfully
worse output, and lets users pay for LSP precision only when wanted.
Mechanism:
- cbm.c: g_lsp_enabled atomic toggle wraps the entire per-file LSP
block (Go/C/C++/PHP/Python/TS/JS/C#) + the C/C++ preprocessor-pass
LSP call. Default true so ad-hoc extraction / tests keep historical
behaviour. cbm_set_lsp_enabled / cbm_get_lsp_enabled in cbm.h.
- pipeline.c: sets the toggle (save→set→restore so it never leaks to
other callers) from p->mode before extraction; gates the cross-LSP
all_defs collection + registry build on advanced; gates the
sequential lsp_cross dispatch. Fixed the moderate_only comparison
(== FAST, not > MODERATE) so ADVANCED is not treated as lighter.
- pipeline_incremental.c: same toggle save/set/restore.
- discover.h/.c: added CBM_MODE_ADVANCED to discover's enum + all 4
filter comparisons treat ADVANCED like FULL (full discovery, no
fast-filtering). Without this, ADVANCED (numerically 3) was
mis-detected as fast-mode and dropped ~half the files.
- mcp.c: parse mode "advanced" + tool schema enum/description.
Dual-mode K8s bench: FULL 46.7s (lsp_overrides=0, 3.0 GB) vs
ADVANCED 48.8s (lsp_overrides=80,818, 5.6 GB). Graphs within 0.01%.
All 3,615 tests pass.
The .m extension was missing from EXT_TABLE, causing detect_file_language
to return CBM_LANG_COUNT (Unknown) before the content-based disambiguation
logic (cbm_disambiguate_m) could ever run. This made .m files invisible
to the indexer.
Add {".m", CBM_LANG_MATLAB} to EXT_TABLE so the initial lookup succeeds,
allowing the subsequent cbm_disambiguate_m call to correctly determine
whether the file is MATLAB, Objective-C, or Magma.
Fixes#296
Adds Pine Script (TradingView's indicator/strategy DSL) as a first-class
language using the project's existing language-add pipeline:
- scripts/new-languages.json: register pine (display "PineScript",
ts_func tree_sitter_pine, repo kvarenzn/tree-sitter-pine, extension
.pine, has_scanner=true, module_root source_file)
- internal/cbm/vendored/grammars/pine/: vendored from
https://github.com/kvarenzn/tree-sitter-pine via scripts/vendor-grammar.sh
— parser.c, scanner.c, tree_sitter/ headers. License is ISC (declared
in upstream package.json; written into vendored LICENSE for clarity
since upstream has no LICENSE file)
- internal/cbm/grammar_pine.c: 4-line wrapper generated by
scripts/generate-lang-code.py wrappers
- internal/cbm/cbm.h: CBM_LANG_PINE before CBM_LANG_COUNT
- internal/cbm/lang_specs.c: 7 node-type arrays (func / class /
module / call / var / branch / assign) cross-checked against
kvarenzn's grammar.js, plus the spec table entry
- src/discover/language.c: .pine -> CBM_LANG_PINE extension dispatch
and "PineScript" name
scripts/audit-grammar-security.sh on the vendored grammar: no
dangerous includes (only string/stdint/stdbool/stdlib), no system /
popen / exec / socket / mmap / dlopen calls, no constructor/destructor
attributes, no inline asm. The 148-line scanner.c only handles
INDENT/DEDENT/NEWLINE tokens via the standard tree-sitter lexer API.
Distilled from #273 rather than vendored verbatim because the
original PR's parser.c was a plain 46k-line drop without provenance,
no LICENSE attribution, and bundled an unrelated Makefile change
(scripts/embed-frontend.sh -> sh scripts/embed-frontend.sh) that
didn't belong to a Pine Script feature. The node-type choices in
the lang spec match the original PR's selections, all of which were
verified to be real grammar.js rules.
Full suite: 2840 passed, 0 failed under ASan + UBSan.
Closes#273.
Co-authored-by: vinay-veerappa <vinay.veerappa@gmail.com>
Add automated extension-to-language mapping tests for all new grammars
that have file extensions. Removes conflicting extensions:
- .blade.php: compound extension, handled by userconfig system
- .env*: filenames, not extensions (no auto-discovery for dotfiles)
- .sv/.svh: already mapped to Verilog, SystemVerilog uses same grammar
Nested .gitignore support (fixes#178):
- Load per-subdirectory .gitignore during walk, match paths relative to
the gitignore's directory via local_rel_path()
- Root and nested gitignores stack independently
- Owned gitignores collected and freed at walk end (avoids use-after-free
from borrowed pointers on the iterative stack)
Security:
- Run security-strings/install/network + ClamAV + Windows Defender on
ALL binary variants (standard + UI), not just standard
- Whitelist UI bundle URLs (React, Three.js, Google Fonts, Tailwind, W3C)
Co-Authored-By: dLo999 <dLo999@users.noreply.github.com>
- Parallelize pass_similarity and pass_semantic_edges via worker pool with
thread-local edge buffers; sequential final merge since gbuf is not
thread-safe. Adds cbm_lsh_query_into() as a thread-safe variant with
caller-provided candidate buffer.
- Add activatable profiling subsystem (CBM_PROFILE=1 env or --profile flag)
for step-level timing of extract, resolve, corpus build, vector phases,
and sqlite dump. Zero overhead when disabled.
- Fix cbm_index_mode_t enum mismatch between pipeline.h (FULL=0, MODERATE=1,
FAST=2) and discover.h (FULL=0, FAST=1). mode=fast silently no-op'd
fast-discovery filtering because discover.c compared against the wrong
value. Linux kernel fast mode went 1:40 -> 3:11 as a result; now back to
1:40. Broaden the filter guard to mode != CBM_MODE_FULL so MODERATE and
FAST both get aggressive discovery.
- Clamp cbm_sem_combined_score output to [0, 1]. The proximity multiplier
returns up to 1.10 as a same-file boost which could push the final
cosine score above 1.0.
- Short-circuit semantic scoring when MinHash jaccard >= 0.95. Exact
near-clones are already emitted as SIMILAR_TO edges; returning 0 here
avoids flooding SEMANTICALLY_RELATED with cross-service copy-paste
boilerplate and frees the edge budget for genuine vocabulary-bridged
relations.
- Validate search_graph semantic_query as an array of strings and return
a clear error for a single-string input. Update the tool description
to spell out the requirement explicitly with an example.
- JSON-escape user-controlled strings (callee names, call arguments,
URL paths, import local_name) in call/argument properties. Introduces
cbm_json_escape() in foundation/str_util.
- Skip SQLite pending_byte_page (file offset 0x40000000) during raw page
writes in sqlite_writer to avoid corrupting databases that cross the
1 GiB boundary.
- Migrate pretrained vector blob from UniXcoder (51K tokens) to
nomic-embed-code (40856 tokens x 768d int8). Includes the extraction
script under scripts/extract_nomic_vectors.py.
Martin Vogel