mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-21 05:59:35 +08:00
main
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
65a93fab40 |
fix(grammar): accept non-ASCII letters in Pascal identifiers (#2211)
Delphi code that calls 1C's OLE API through late-bound Variants uses
Cyrillic member names (v8_Base.<Cyrillic>.<Cyrillic>('1', Now),
NM.<Cyrillic>()). Every such name produced a tree-sitter ERROR node, so the
unit was reported as parse_partial and a procedure whose own name is
Cyrillic was dropped from the graph.
Root cause is upstream tree-sitter-pascal (Isopod, the pin we vendor):
its identifier token is ASCII-only, /[&]?[a-zA-Z_]+[0-9_a-zA-Z]*/, and
there is no other token a non-ASCII letter can lex as.
Fix: pascal becomes a self-maintained fork of the pin we already vendored
(Isopod/tree-sitter-pascal @ 042119eca2e1, v0.10.2 = upstream master),
following the arkts model. One patch
(tools/tree-sitter-pascal/patches/unicode-identifiers.patch) changes only
that token to /[&]?[\p{L}_][\p{L}\p{M}\p{Nd}_]*/. Keywords are still
extracted from this `word` token, so begin/end/procedure lex as before
(`begin` glued to Cyrillic letters is one identifier; `&begin` stays an
escaped identifier). Node types are unchanged. The \p{} classes compile
to numeric code-point ranges: the vendored parser.c has 0 non-ASCII bytes.
Regenerated with tree-sitter-cli 0.25.10 --abi 14 (ABI unchanged, no
external scanner; tree_sitter/parser.h is byte-identical to chialisp's).
Fidelity: the UNPATCHED grammar regenerated with 0.25.10 gives the same
symbol/token/field/production counts as upstream's shipped parser and
byte-identical parse trees on upstream's 88 corpus cases and its three
examples/*.pas (37,773 tree lines for codetoolmanager.pas); the patched
parser gives the same identical trees on all of those ASCII inputs and
passes all 88 corpus cases.
Proof (7 Pascal files: the report's snippet, a Cyrillic procedure with
keyword-boundary cases, an ASCII control, a broken unit, upstream's three
examples; fresh caches): parse_partial 3 -> 1 (only the genuinely broken
unit remains), nodes 581 -> 582 (the Cyrillic-named procedure), edges
1480 -> 1482 (+1 DEFINES, +1 SEMANTICALLY_RELATED); per-file function
counts for the ASCII files are unchanged.
Registered like arkts: MANIFEST.md fork row + note, audit FORKS map,
THIRD_PARTY.md, vendored checksums. LICENSE stays upstream's MIT,
byte-identical.
Tests: parse_coverage gains the report's snippet and a Cyrillic-named
procedure (not partial, definitions extracted), an ASCII control, and a
guard that broken Pascal is still flagged; the fork carries
test/corpus/unicode_identifiers.txt.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
1b79558975 |
fix(grammar): accept a lone '&' in JSX strings and text (#1736)
A plain '&' inside JSX -- a URL query in an attribute (href="https://fonts.googleapis.com/css2?family=Inter&display=swap"), or text such as <p>Tom & Jerry</p> -- produced a tree-sitter ERROR node in .js, .jsx and .tsx files. The file was flagged parse_partial in the coverage view, and the error recovery turned words from the URL (create-next-app -> create/next/app) into identifiers that resolved to unrelated symbols as false USAGE edges. Root cause is upstream tree-sitter-javascript, which the tsx dialect builds on: unescaped_{double,single}_jsx_string_fragment is /([^"&]|&[^#A-Za-z])+/, so '&' is only accepted before a space/digit or as a complete html_character_reference (& / &). jsx_text is scanned by the external scanner, which stops at every '&', and nothing in _jsx_child accepted a '&' that is not a reference. So '&' + letter in a string, and any lone '&' in JSX text, had no valid token. Fix: javascript and tsx become self-maintained forks of the pins we already vendored (javascript @ 58404d8cf191; typescript tsx @ 75b3874edb2d on its javascript base @ 3a837b6f3658), following the arkts model. One patch (tools/tree-sitter-javascript/patches/ jsx-lone-ampersand.patch) is applied to both javascript bases: - a one-character '&' token at lexical precedence -1, aliased to the existing string_fragment (in strings) and jsx_text (in text); the lower precedence keeps & / & lexing as html_character_reference - the string-fragment regexes exclude a '&' directly before the closing quote, so "a&" no longer runs past the end of the string Node types are unchanged (same type/named set). The external scanners stay upstream bytes. Parsers regenerated with tree-sitter-cli 0.25.10 (ABI 15; tsx moves from ABI 14, and its tree_sitter/ headers now match arkts). Without the patch both fork dirs regenerate upstream exactly. The upstream corpus passes unchanged: javascript 116/116; tsx gives the same results as unpatched upstream. Proof on vercel/next.js examples/ (canary 725705bd, 1,722 js/jsx/tsx files): parse_partial 92 -> 47 (tsx 33 -> 0, jsx 2 -> 0, js 12 -> 2; the two left are Babel do-expressions and Flow types), no newly flagged file, code-graph nodes 14,739 -> 14,739, edges 24,773 -> 24,657: the 115 USAGE edges removed were all false cross-example edges from URL fragments, plus a 4 SIMILAR_TO / +3 SEMANTICALLY_RELATED reshuffle from the corrected ASTs. Registered like arkts: MANIFEST.md fork rows + notes (ABI tally recounted), audit FORKS map, THIRD_PARTY.md, vendored checksums. LICENSEs stay upstream's MIT byte-identical. Tests: parse_coverage gains a .js/.jsx/.tsx matrix (double/single-quoted attribute, '&' before the quote, '&' + letter/space/digit in text, plus &/& controls) and a guard that broken JSX is still flagged. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogelanddarkverbito
|
706bb2ce81 |
feat(lang): add Chialisp support with a first-party tree-sitter grammar
Chialisp (.clsp/.clib/.clinc) is the Chia smart-coin language. The only
public grammar, Quexington/tree-sitter-chialisp, cannot parse it: on the five
reference files from chia-blockchain@main it produced 3-34 ERROR nodes each.
Three defects account for all of them, and each desynchronised the rest of the
file rather than losing one node:
* comments were `/;.*\r\n/` -- CRLF-only, while real files are LF;
* `(include foo.clib)` was rejected because of the dot in the filename;
* `(defconstant NAME <expr>)` accepted only a primitive after the name.
So the grammar is written here instead, in tools/tree-sitter-chialisp/, and
files as first-party: its vendored LICENSE is the project's own, byte-identical
to the repository root, carrying no third-party copyright.
It is deliberately GENERIC -- source_file / list / symbol / string / number /
hex / dot / comment, modelling the clvm_tools reader rather than the Chialisp
form vocabulary. `mod`, `defun`, `defconstant` and `include` are ordinary head
symbols; which lists are definitions is decided in extract_defs.c. A dialect
that adds a form (cl-21 -> cl-23 -> cl-26 each added several) therefore needs
no regenerated parser. Acceptance: all five reference files parse with zero
ERROR and zero MISSING nodes; the three defects above are pinned as tests.
Extraction, and the four decisions in it:
* `Constant` is admitted to the cross-file name registry
(cbm_label_is_registry_symbol). Chialisp's condition codes live as
`(defconstant ...)` in a shared .clib and are referenced by name from every
puzzle; labelled but unregistered they are seeded nowhere, and roughly half
of a Chialisp graph would be present as nodes yet never the target of an
edge. All three registry-seeding sites route through that one predicate, so
they cannot diverge. No other language emits `Constant`, so the blast radius
is exactly Chialisp. Pinned by a two-file graph test that goes red when the
label is removed from the predicate.
* `export` / `namespace` are NOT definition heads. `(export foo)` names a
function already defined in the same file, so treating it as a def head
mints a second node for one symbol. They stay in the not-a-call filter.
* A parameter list is a `list` too, but it binds rather than invokes. Without
that gate `(defun check_conditions (HEIGHTLOCK conditions) ...)` mints a
phantom call to HEIGHTLOCK, and real Chialisp is mostly such definitions.
* `(include *standard-cl-26*)` is a dialect directive, not a file dependency;
recording it would invent a dependency on a file that does not exist.
Three helpers that the defs, calls and unified walks all need -- the quote-data
gate, comment-skipping named-child access, and the Chialisp def-head set -- live
in helpers.c rather than being copied per translation unit. They previously
existed as byte-identical duplicates that had already drifted: extract_defs.c
skipped comments when reading a def head while its mirror in extract_unified.c
did not, so a comment between a head and its name desynchronised defs from call
scope. Both now use the same accessor.
Ledger: 161 -> 162 grammars across the seven published surfaces, ABI-14 86 -> 87,
first-party 13 -> 14, CBM_LANG_COUNT 164 -> 165.
Distilled from PR #919 by darkverbito, whose extraction logic -- the CLVM
not-a-call vocabulary, the quoted-data gate, the filename-derived `mod` name and
the include/embed import handling -- is kept and is the substance of this change.
The grammar that PR vendored is not used.
Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
032ebf5327 |
fix(vendor): ship arkts upstream MIT byte-identical and register the fork
The vendored LICENSE carried tree-sitter-typescript's MIT text plus an appended derivative notice. That is not what MIT asks of a derivative -- it asks that upstream's notice travel with the code -- and the appended text meant the provenance audit could not byte-verify the file against upstream, so arkts landed as NO-MANIFEST-ENTRY and failed license-gate. The vendored LICENSE is now byte-identical to upstream's, and arkts is registered in the audit's FORKS map, which is defined for exactly this case: a self-maintained fork whose vendored LICENSE must match its upstream. That is a routing entry, not an exemption -- check_upstream still byte-verifies against the real upstream repository, so the fact is checked rather than asserted. Audit now reports arkts as IDENTICAL. Our copyright for the ArkTS additions is not lost, it moves to where our SOURCE is: the grammar.js header (which now states all three copyrights explicitly) and THIRD_PARTY.md, which ships inside the release archives. The edit to scripts/audit-license-provenance.py is a maintainer-only enforcement-surface change, made deliberately and stated here rather than slipped in; a contribution may never supply its own audit entry. No generated parser or source changes -- comments, licence text, manifest prose and one routing entry only. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0df990aa60 |
Add ArkTS (.ets) language support via first-party tree-sitter-typescript fork
Routing .ets through the plain TypeScript grammar produced no definition node for any @Component struct (0/19 across the OpenHarmony sample corpus), parsed struct members as statement labels and build() as a call expression, and registered ArkUI builtins (Column, Row, ListItem, ...) as user function definitions with 28% fabricated cross-file CALLS edges. This adds a dedicated ArkTS grammar that keeps everything TypeScript already handles and models the ArkTS syntax TS cannot parse. Grammar (tools/tree-sitter-arkts/, generated parser vendored to internal/cbm/vendored/grammars/arkts/, ABI 15 via tree-sitter-cli 0.25.10): first-party derivative of tree-sitter/tree-sitter-typescript v0.23.2 @ 75b3874edb2d (the commit our vendored typescript/tsx come from) on the tree-sitter-javascript v0.23.1 base @ 3a837b6f3658 (the version upstream's package-lock pins). LICENSE carries the upstream MIT text verbatim plus a derivative notice; the external scanner is a symbol-rename trampoline over a byte-identical copy of the already-vendored typescript scanner header. ArkTS additions: decorated struct component declarations with class-shaped bodies; UI-DSL trailing-closure component calls with attribute chains after the closing brace; decorators on function declarations (@Builder, @Extend(T), @Styles); @Extend/@Styles leading-dot attribute-chain bodies; anonymous stateStyles style blocks; import lazy named imports. Wiring: CBM_LANG_ARKTS enum + lang spec (struct_declaration as a class kind labeled Struct, public_field_definition fields), .ets extension, LANG_NAMES/userconfig entries, grammar shim, and TS-family joins across the extractors and the name-resolution guards (parse_es_imports, JS keywords/test-file/func-kind/scope tables, TS heritage/param/receiver/ namespace/type-ref/channel/usage paths, js_ts_family and the weak-method- match suppression). Field definitions now also carry their decorators, so @State/@Prop/@Link members are findable with their roles. Tests: grammar corpus (35 cases), per-language regression + label golden + import net + extension cases, seven ArkTS extraction tests including the phantom-builtin regression (ArkUI builtins must never be definitions), and capability-ledger / call-node-manifest / call-argument-matrix rows. On the 14-file OpenHarmony sample corpus the TS-grammar baseline had 12/14 files with parse errors (231 ERROR nodes); the arkts grammar parses 14/14 clean, and an end-to-end index yields all 19 components as Struct nodes, 68 methods, 119 fields, and zero phantom builtin definitions. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
18fa9979ee |
Replace vmem with mimalloc global allocator, add extraction-phase prescan, fix __init__.py QN collision
Memory management: - Vendor mimalloc v2.1.9 as global allocator (MI_OVERRIDE=1 in prod) - New mem.h/mem.c: RSS-based budget tracking via mi_process_info() - Remove vmem.c/vmem.h (mmap-based budget tracking) - Remove slab tier2 bump allocator (~300 LOC); >64B goes to mimalloc - Slab tier1 pages from malloc (= mimalloc) instead of vmem - Arena blocks from malloc instead of vmem - Budget raised from 35% to 50% RAM (no more untracked C++ heap) Extraction-phase prescan (eliminates disk re-reads): - HTTP call sites: keyword check + URL extraction during extraction - HTTP routes: decorator + source-based extraction during extraction - Config file refs: regex scan during extraction - httplinks: 41.8s → 13ms on Linux kernel (3,212x faster) - configlink: 41.4s → 0.8s on Linux kernel (54x faster) - Linux kernel fast-mode total: 2m38s → 1m18s Bug fixes: - __init__.py Module QN no longer collides with Folder QN - index.ts same fix for JS/TS packages - 13 regression tests for QN collision at FQN + extraction layers - search_graph/search_code default limit raised from 10 to 500k - Resolve all clang-tidy, cppcheck, and clang-format warnings Repo cleanup: - tree-sitter-form, tree-sitter-magma moved to tools/ - .gitignore: build/, node_modules/, graph-ui/dist/, TEST_PLAN.md |