cbm_writer_open used a raw ANSI-CP fopen for the hand-rolled SQLite
writer - the one file-creating call on the dump chain without a
UTF-8-to-wide conversion. On a non-ASCII Windows profile
(C:\Users\Kovacs Janos) extract and resolve succeeded but the dump
failed with pipeline.err phase=dump: fopen could not create graph.db
under the profile-derived cache dir while cbm_mkdir_p (already wide)
had happily created the directory around it. Routed through cbm_fopen,
matching #700 (pass readers) and #973 (canonicalization); POSIX is
byte-transparent.
New Windows guard test_non_ascii_cache_dump.py indexes an ASCII repo
into a non-ASCII CBM_CACHE_DIR and requires a query_graph readback
with Function count > 0 - proving the DB was written to and reopened
from the non-ASCII cache, not merely that no error surfaced. The
existing non-ASCII guard only varied the repo path against an ASCII
cache, which is why this class went unseen.
The USERPROFILE-sourced env encoding question (Layer 2 - _environ is
ANSI-CP, not UTF-8) is deliberately not bundled; it needs verification
on a real Windows box and will be filed separately if the guard alone
does not settle it.
Closes#996
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three path-canonicalization sites routed UTF-8 paths through the ANSI
CRT (_access + _fullpath): the repo_path canonicalization in
handle_index_repository, project-name derivation in fqn.c, and
resolve_canonical_path (the containment check). ANSI decoding is
locale-dependent by construction — on a CJK system codepage (the
reporter runs Big5) the UTF-8 bytes of a CJK repo path re-decode into
different characters, corrupting canonicalization while our CI (en-US
codepage) sails through: that is why the existing non-ASCII guards
stayed green while v0.9.0 crashed in the field.
New cbm_canonical_path: realpath on POSIX (unchanged behavior); on
Windows a wide GetFileAttributesW existence check + GetFullPathNameW,
never touching the ANSI codepage. resolve_canonical_path now also
requires existence on Windows, aligning it with the POSIX realpath
semantics it mirrors.
Guard: the Windows non-ASCII CLI guard gains the reporter's exact
shape — Traditional-Chinese directory (雷達測試), flag-form
--repo-path + --mode fast, real supervised spawn. Honest caveat: CI
cannot switch the system codepage, so the guard validates the
mechanics under en-US; the locale-specific corruption is fixed by
construction (no ANSI path decoding remains on these sites).
Closes#973
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
query_graph now defaults to TOON text; the guard's definition-count
probe parsed it as JSON, swallowed the parse failure, and reported the
ASCII baseline as having zero definitions (SETUP FAIL on the
test-windows-guards leg). Scripted consumers use the format="json"
escape hatch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#423, #20. A non-ASCII repo path passed to
`codebase-memory-mcp cli index_repository "<json>"` was mangled on Windows, so the
command failed with "repo_path is required" instead of indexing the real directory.
POSIX is unaffected (argv is already UTF-8). Two boundaries had to be fixed:
1. argv read: main() took only the narrow `int main(int argc, char **argv)`, so the
CRT handed it argv in the active ANSI code page. On Windows main() now rebuilds
argv from the wide command line (GetCommandLineW + CommandLineToArgvW) and
converts each element to UTF-8 (cbm_wide_to_utf8).
2. worker spawn: the index supervisor (and the UI index thread) spawned the worker
via CreateProcessA, which re-interpreted the UTF-8 command line through the ANSI
code page -- re-mangling the path at the parent->worker boundary even after (1).
Both spawns now build a wide command line (cbm_utf8_to_wide) and use CreateProcessW.
Without (2) the default (supervisor-enabled) path stayed broken while only the
in-process path worked, so this is the complete end-to-end fix, not just argv.
Promotes tests/windows/test_cli_non_ascii_arg.py from an opt-in known-red to a green
Windows guard (scripts/test-windows.ps1 + the test-windows-guards CI job). The guard
now exercises the DEFAULT supervised path (it drops the suite's CBM_INDEX_SUPERVISOR=0
so it actually crosses the worker-spawn boundary). Verified on native Windows: RED
before the fix, GREEN after (a non-ASCII repo indexes, nodes>0). Also adds -lshell32
explicitly (main.c now uses CommandLineToArgvW) and refreshes RED_TEST_ANALYSIS.md.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Rebased onto current main and reworked in response to review. Three of the four
Windows reds were fixed upstream since the branch was cut at b075f05, so they are
now green regression guards; the fourth stays a genuine known-red.
- test_non_ascii_path.py (#636/#357): green guard - fixed by #700 (per-pass
readers now route through cbm_fopen -> _wfopen). Re-verified green on main.
- test_hook_augment.py (#618): green guard - fixed by #619 (cbm_is_walkable_abs_path
accepts drive-letter X:/ cwd). Re-verified green on main.
- test_ui_drive_listing.py (#548): rewritten. The fix exposes drives via a new
roots[] field, not the dirs[] array the old test asserted (which would stay red
against fixed code). Now asserts every fixed drive is in roots and browsable.
Re-verified green on main (drives C:/D:/E:).
- test_cli_non_ascii_arg.py (#423/#20): unchanged - main() is still narrow-argv
with no wide command line, so this remains genuinely red (the keeper).
- scripts/test-windows.ps1: split green guards (gate CI) from opt-in known-reds;
add -GuardsOnly; run indexing in-process (CBM_INDEX_SUPERVISOR=0) so a guard
reflects the path/hook/drive fix under test, not the index-worker spawn path.
- .github/workflows/_test.yml: new test-windows-guards job builds the product+UI
binary (scripts/build.sh --with-ui) and runs the guards with -GuardsOnly so
#700/#619/#548 stay enforced on Windows CI.
- RED_TEST_ANALYSIS.md: refreshed to record the landed fixes and current status.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reference the Windows umbrella issue #394 in the analysis so every open
Windows-relevant issue is accounted for: its open children (#227/#367, the
mapped/SMB-drive class) are in the ruled-out table and its other children are
already fixed upstream. No test or production change.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Extends the Windows red-test suite with two more reproduced, Windows-specific
failures. No production fixes.
- windows_hook_augment_emits_context (integration, #618): the PreToolUse
Grep/Glob augmenter `hook-augment` emits empty stdout for every payload on
Windows. src/cli/hook_augment.c gates on POSIX-style absolute paths
(cwd[0] == '/' and the walk-up loop's dir[0] == '/'), which a Windows
drive-letter cwd never satisfies, so the graph augmentation never fires. A
control search_graph confirms the symbol is indexed.
- windows_ui_picker_reaches_all_drives (integration, #548): the UI directory
picker's GET /api/browse?path=/ returns no entries and never enumerates
logical drives, so drives other than the system drive (D:\, E:\) cannot be
selected. handle_browse in src/ui/http_server.c uses opendir without a
GetLogicalDriveStrings root case. Needs a UI build (cbm-with-ui) and >1 drive;
otherwise it reports a precondition (exit 2).
Also records additional ruled-out seed areas (get_code_snippet sanitizes
non-UTF-8 to U+FFFD #530.3; stdio handshake/flush works #513/#530.1/#635; mapped
subst-drive indexing keeps the DB #227/#367) and cross-platform items left out
of this Windows-only PR (#530.2 nested gitignore, #530.5 .git/info/exclude,
#530.4 libgit2 build, #581 memory soak).
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Adds Windows-only red tests and analysis for native Windows failures found
during a Windows red-test campaign. This change contains no production fixes.
- windows_non_ascii_repo_path_preserves_definitions (integration): byte-identical
TypeScript fixtures indexed under non-ASCII parent paths (Latin-1 accents,
Cyrillic, CJK, Greek) extract zero definitions and only File/Folder nodes
(5 nodes / 4 edges) versus the ASCII baseline (12 nodes / 20 edges / 5
definitions). The pipeline source readers open files with fopen() on a UTF-8
path, which the Windows CRT interprets in the active ANSI code page; directory
discovery already uses the wide API, so files are listed but never parsed.
- windows_cli_non_ascii_repo_path_is_honored (integration): the documented
`cli index_repository` entrypoint rejects a non-ASCII repo_path because main()
does not read the wide command line, so argv arrives in the ANSI code page.
Both reproduce at the product surface (real MCP process, real stdio, real SQLite
DB), are deterministic, and pass on Linux/macOS. A PowerShell runner builds the
binary and runs the suite; standard-library Python only. See
tests/windows/RED_TEST_ANALYSIS.md for environment, commands, ruled-out seed
areas, and suspected fix locations.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Martin Vogel