mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-11-14 16:52:09 +08:00
Colby MchenryandClaude Opus 5.5
c12a7b6a74
The index stores a name in any script as one full-text token, and `codegraph query 로그인` found it, but every tokenizer between a question and the index knew only ASCII: extractSearchTerms split on [^a-zA-Z0-9] and kept 3+ characters, context's extractSymbolsFromQuery matched ASCII patterns only, and explore's exploreQueryTokens and the Flow's flowTokens required a leading [A-Za-z_$] and 3+ characters. So `explore 寻路服务` answered "No relevant code found", `context 로그인` printed only its header and `explore 入口 末端` had no Flow, while their ASCII twins worked. Shared helpers in search/query-utils now read words in any script whole: letters, digits and combining marks, cut where Han, kana or Hangul meets another script (NavMeshAgent类型 is NavMeshAgent and 类型, never agent类型), at least 2 characters in Han, kana or Hangul and 3 elsewhere, so Russian на and из stay out. extractSearchTerms and the explore and Flow tokens use them. Context adds names with a non-ASCII letter as whole runs, and a mixed name like get用户信息 as written, with no n-grams, so a partial Chinese word still never becomes a search of its own (PR #2005). The ASCII-only patterns see a word with accented or other non-ASCII letters blanked out, so état is no longer read as tat. A Han, kana or Hangul token counts as a deliberately named identifier, as such a name has no case or underscore to show it, so a query of such names is neither damped as prose nor flagged low-confidence. Validation: new cjk-query-tokens tests (12) cover CJK, mixed, Cyrillic and accented words, reference copies of the old tokenizers over an ASCII corpus, and explore and context end to end on Korean, Chinese and Japanese fixtures; 9 fail on main's source. Explore and context A/B with 214 ASCII queries (gin 76, Alamofire 79, express 59; 49 lead with a Flow): all byte-identical. Of 12 non-ASCII queries, 6 changed on gin and 4 on express, as intended: état, Señal and naïve are read whole, and handleHTTPRequest处理 now names handleHTTPRequest. Tokenizing a 10,000-character query takes at most 1.8 ms, against 0.1-0.3 ms before. Reimplements PR #516 by @andy-sg and PR #1262 by @yimingll. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>