Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fixes surfaced while bringing the shared-coordination-daemon branch
green across macOS, the Linux containers, and the real Windows-ARM64 VM,
plus a new stability guard that caught one of them.
- daemon(win): the CLI teardown transition latched a failed release as
permanent, but a Windows participant-state release must briefly
try-hold the shared startup/legacy gates and legitimately collides
with a concurrent one-shot's teardown; the release is retriable by
contract (it always retains the transition), so parallel one-shot
commands no longer report "CLI coordination cleanup failed" despite
succeeding. Found by the new stability guard's churn section.
- cli/main(win): install/update/uninstall and the Augment hook-script
removal derive the managed launcher's identity from its plain
drive-form path (cli_windows_plain_utf8), not the \\?\ extended-length
form, so agent-config ownership matches on Windows and uninstall no
longer leaves the MCP entry or owned hook scripts behind.
- security: the `daemon start --open` browser launch is now shell-free
(ShellExecuteW on Windows, cbm_exec_no_shell elsewhere) instead of
system(); the CORS origin check spells out its two literal loopback
URLs so the static URL audit sees a complete value. Both clear the
Layer-1 allow-list audit.
- coordination: version-cohort lock retries now sleep a per-process
jittered interval — fixed-period retries can phase-lock two
participants so one starves. The activation-quiesce test's
observation window now covers worst-case candidate staging (three
tamper-defense hashes over a ~1 GB sanitizer binary exceed the old
30 s budget on container I/O).
- smoke/soak: the agent-config smoke matches Windows config paths
through their escaped-backslash quoted form; the soak parser skips the
update-available banner before the JSON summary and gained a one-shot
CLI admission-churn phase inside the RSS/FD leak window. The build-dir
safety fixture creates a real symlink/junction (MSYS2 ln -s otherwise
deep-copies) so its traversal-refusal contract is genuinely exercised,
and the symlinked-agent-roots test demotes its planted links to an
unprivileged owner when run as root so the refusal is really tested.
- tests(win): a new daemon-stability guard exercises the parameter
surface, hook fail-open with its rate-limited notice, start-twice and
occupied-port handling, busy-stop refusal against a live MCP session,
kill -9 crash recovery, and sequential/parallel one-shot churn.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.
Daemon lifecycle:
- daemon start/stop/status subcommands. `daemon start` launches a
PERMANENT daemon (spawn shape is byte-exact argv; survives idle
periods and session ends) and reports an already-active daemon
instead of failing. Permanence is honored at every stop latch:
last-committed-client disconnect, host initial-client window,
coordinator release, and application final-session close — a
permanent daemon also keeps admitting new sessions after its last
one closes.
- daemon stop refuses while sessions are active and lists the blocking
peers (pid/role) that must finish first; an idle daemon drains
through the activation-shutdown machinery with the ACK ordered after
connection interrupts. A second stop is idempotent. The wire ops are
no-cohort first-frame requests with peer fingerprint authentication,
so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
keep their local supervised path). A cold CLI run that had to spawn a
temporary daemon prints a hint that `daemon start` removes the
per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
emits a visible, rate-limited notice (Claude-dialect systemMessage
plus stderr for other dialects) and always exits 0 — augmentation is
never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
older-build permanent daemon (strict semantic-version triples only;
dev builds never auto-drain) and the build-conflict message names
`cbm daemon stop` as the manual escape hatch.
Windows IPC/runtime (real-VM verified):
- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
path used to destroy its listening pipe instance on every 20 ms poll
timeout; a client attaching in the teardown window was severed or left
on an orphaned pipe object whose HELLO no server handle could ever
read, absorbing the connect until the client's own timeout expired.
The pending connect now survives poll timeouts and nothing is
destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
server handle can discard a just-sent response before the peer reads
it (POSIX stream sockets never lose buffered data on close). A bounded
cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
immediate on interrupted connections) now precedes close in
runtime_worker_finish and runtime_reject_inline, so hello-conflict,
capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
transport half-close, so close_begin now announces departure with an
explicit frame (ordered after APPLICATION_CANCEL, before the local
interrupt); the server releases the client's admission on receipt
instead of waiting for the handle to close. Admission-drop timing is
now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
request publication under the send lock, best-effort sends the active
token's APPLICATION_CANCEL frame, then interrupts local I/O; the
server cancels MCP/subprocess work promptly. Contract tests accept
both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
the requester's license to act on "snapshotted and draining", so every
connection interrupt is now initiated before the ACK is sent; a
session could previously get one more request serviced after the
requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
timer granularity) burned ~1.6 s against permanently obstructed paths,
stalling hello rejections past the client's timeout. The retry budget
is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
nonblocking send() on Windows is absorbed wholesale into AFD kernel
buffering regardless of SO_SNDBUF, so send deadlines and interrupts
could never engage against a slow peer (and the full payload was
pinned in nonpaged pool). Slicing restores a deterministic
backpressure point; a test hook pins SO_SNDBUF for the deadline and
interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
deadline, explicit connection states, refusal to free a server while
a listener-owned connection is active.
Windows long-path support:
- Central path-aware wide conversion (canonicalize via GetFullPathNameW
and prepend the extended-length prefix for absolute paths >=240) at
the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
store opens, and the daemon build-fingerprint/log paths. Deep managed
installs (a 64-hex generation directory routinely exceeds MAX_PATH)
now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
ancestry validators now operate in the extended-length namespace;
the launcher path is canonicalized (and prefixed when deep) once at
entry so every downstream exact-string comparison stays
form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
GetFileAttributesW) so non-ASCII install paths survive argv[0]
resolution.
Windows launcher install/uninstall transaction:
- FileRenameInfoEx names are NUL-terminated in an over-allocated
buffer. FileNameLength governs per the contract, but filter drivers
read FileName as NUL-terminated and appended adjacent heap bytes to
created names — a flaky, garbage-suffixed rename target. Both the CLI
and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
.cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
chars so the bare rename target stays under the FileRenameInfoEx
NT-conversion ceiling at guard depths; 64 bits still uniquely
identify the generation.
- When the running launcher's mapped generation backings pin .cbm
against rename, the backings are relocated to activation-<pid>-N
.retired tombstones beside the install (a mapped image may be renamed,
never deleted; the launcher's liveness-guarded sweep reclaims stale
tombstones). Every relocation is recorded, and a FAILED uninstall
reverses the moves after restoring .cbm — via MoveFileExW with
extended-length paths on both arguments, since the deep generation
target is beyond the handle-based rename's bare-path reach — so a
restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
out so the tree is shallow enough for the detached cleanup's rd, and
the cleanup's working directory strips the extended-length prefix
(CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
(CopyFileW destinations, CREATE_NEW tombstones, probe directories)
are explicitly owner-stamped so the exact-owner validators hold on
runner images; guard fixtures stamp hand-built trees the same way.
Diagnostics, tests and infra:
- diagnostics: discovery is now an always-delivered JSON control record
(new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
paths containing spaces; placement honors $TMPDIR with /tmp fallback
via a diagnostics-local helper; the soak parser reads the JSON record;
documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
log levels, TMPDIR-with-spaces, and native Windows output-contract
assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
parents with production-shaped ancestry (LocalAppData on Windows, via
th_secure_runtime_parent_new) — the runtime ancestry validation
correctly refuses temp roots whose ancestors grant mutation rights to
Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
drive the documented startup-owner publication flow before reading
generation-bound endpoint addresses. This turns the 26 Windows
failures previously visible in CI's full-test job green without
weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
contract including failed-uninstall restore and immediate reinstall
after uninstall; new daemon lifecycle and reworked hook-augment
guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
(it was set by CI but never read, so perf suites ran everywhere);
four throughput/bench suites are classified as perf, the heavy
store_arch suite moved to the slow-timeout tier, and two
wall-clock-sensitive assertions were rewritten as invariant checks
with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
contract, soak daemon-recovery contract, path-safety helper, the
Windows VM worktree-sync contract wired into scripts/test.sh, and
vm/win.sh guards building its clean embedded-UI product in an
isolated BUILD_DIR so it cannot clobber the incremental test build.
provision-windows.sh now installs Node.js for the guards UI build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_writer_open used a raw ANSI-CP fopen for the hand-rolled SQLite
writer - the one file-creating call on the dump chain without a
UTF-8-to-wide conversion. On a non-ASCII Windows profile
(C:\Users\Kovacs Janos) extract and resolve succeeded but the dump
failed with pipeline.err phase=dump: fopen could not create graph.db
under the profile-derived cache dir while cbm_mkdir_p (already wide)
had happily created the directory around it. Routed through cbm_fopen,
matching #700 (pass readers) and #973 (canonicalization); POSIX is
byte-transparent.
New Windows guard test_non_ascii_cache_dump.py indexes an ASCII repo
into a non-ASCII CBM_CACHE_DIR and requires a query_graph readback
with Function count > 0 - proving the DB was written to and reopened
from the non-ASCII cache, not merely that no error surfaced. The
existing non-ASCII guard only varied the repo path against an ASCII
cache, which is why this class went unseen.
The USERPROFILE-sourced env encoding question (Layer 2 - _environ is
ANSI-CP, not UTF-8) is deliberately not bundled; it needs verification
on a real Windows box and will be filed separately if the guard alone
does not settle it.
Closes#996
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three path-canonicalization sites routed UTF-8 paths through the ANSI
CRT (_access + _fullpath): the repo_path canonicalization in
handle_index_repository, project-name derivation in fqn.c, and
resolve_canonical_path (the containment check). ANSI decoding is
locale-dependent by construction — on a CJK system codepage (the
reporter runs Big5) the UTF-8 bytes of a CJK repo path re-decode into
different characters, corrupting canonicalization while our CI (en-US
codepage) sails through: that is why the existing non-ASCII guards
stayed green while v0.9.0 crashed in the field.
New cbm_canonical_path: realpath on POSIX (unchanged behavior); on
Windows a wide GetFileAttributesW existence check + GetFullPathNameW,
never touching the ANSI codepage. resolve_canonical_path now also
requires existence on Windows, aligning it with the POSIX realpath
semantics it mirrors.
Guard: the Windows non-ASCII CLI guard gains the reporter's exact
shape — Traditional-Chinese directory (雷達測試), flag-form
--repo-path + --mode fast, real supervised spawn. Honest caveat: CI
cannot switch the system codepage, so the guard validates the
mechanics under en-US; the locale-specific corruption is fixed by
construction (no ANSI path decoding remains on these sites).
Closes#973
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
query_graph now defaults to TOON text; the guard's definition-count
probe parsed it as JSON, swallowed the parse failure, and reported the
ASCII baseline as having zero definitions (SETUP FAIL on the
test-windows-guards leg). Scripted consumers use the format="json"
escape hatch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#423, #20. A non-ASCII repo path passed to
`codebase-memory-mcp cli index_repository "<json>"` was mangled on Windows, so the
command failed with "repo_path is required" instead of indexing the real directory.
POSIX is unaffected (argv is already UTF-8). Two boundaries had to be fixed:
1. argv read: main() took only the narrow `int main(int argc, char **argv)`, so the
CRT handed it argv in the active ANSI code page. On Windows main() now rebuilds
argv from the wide command line (GetCommandLineW + CommandLineToArgvW) and
converts each element to UTF-8 (cbm_wide_to_utf8).
2. worker spawn: the index supervisor (and the UI index thread) spawned the worker
via CreateProcessA, which re-interpreted the UTF-8 command line through the ANSI
code page -- re-mangling the path at the parent->worker boundary even after (1).
Both spawns now build a wide command line (cbm_utf8_to_wide) and use CreateProcessW.
Without (2) the default (supervisor-enabled) path stayed broken while only the
in-process path worked, so this is the complete end-to-end fix, not just argv.
Promotes tests/windows/test_cli_non_ascii_arg.py from an opt-in known-red to a green
Windows guard (scripts/test-windows.ps1 + the test-windows-guards CI job). The guard
now exercises the DEFAULT supervised path (it drops the suite's CBM_INDEX_SUPERVISOR=0
so it actually crosses the worker-spawn boundary). Verified on native Windows: RED
before the fix, GREEN after (a non-ASCII repo indexes, nodes>0). Also adds -lshell32
explicitly (main.c now uses CommandLineToArgvW) and refreshes RED_TEST_ANALYSIS.md.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Rebased onto current main and reworked in response to review. Three of the four
Windows reds were fixed upstream since the branch was cut at b075f05, so they are
now green regression guards; the fourth stays a genuine known-red.
- test_non_ascii_path.py (#636/#357): green guard - fixed by #700 (per-pass
readers now route through cbm_fopen -> _wfopen). Re-verified green on main.
- test_hook_augment.py (#618): green guard - fixed by #619 (cbm_is_walkable_abs_path
accepts drive-letter X:/ cwd). Re-verified green on main.
- test_ui_drive_listing.py (#548): rewritten. The fix exposes drives via a new
roots[] field, not the dirs[] array the old test asserted (which would stay red
against fixed code). Now asserts every fixed drive is in roots and browsable.
Re-verified green on main (drives C:/D:/E:).
- test_cli_non_ascii_arg.py (#423/#20): unchanged - main() is still narrow-argv
with no wide command line, so this remains genuinely red (the keeper).
- scripts/test-windows.ps1: split green guards (gate CI) from opt-in known-reds;
add -GuardsOnly; run indexing in-process (CBM_INDEX_SUPERVISOR=0) so a guard
reflects the path/hook/drive fix under test, not the index-worker spawn path.
- .github/workflows/_test.yml: new test-windows-guards job builds the product+UI
binary (scripts/build.sh --with-ui) and runs the guards with -GuardsOnly so
#700/#619/#548 stay enforced on Windows CI.
- RED_TEST_ANALYSIS.md: refreshed to record the landed fixes and current status.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reference the Windows umbrella issue #394 in the analysis so every open
Windows-relevant issue is accounted for: its open children (#227/#367, the
mapped/SMB-drive class) are in the ruled-out table and its other children are
already fixed upstream. No test or production change.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Extends the Windows red-test suite with two more reproduced, Windows-specific
failures. No production fixes.
- windows_hook_augment_emits_context (integration, #618): the PreToolUse
Grep/Glob augmenter `hook-augment` emits empty stdout for every payload on
Windows. src/cli/hook_augment.c gates on POSIX-style absolute paths
(cwd[0] == '/' and the walk-up loop's dir[0] == '/'), which a Windows
drive-letter cwd never satisfies, so the graph augmentation never fires. A
control search_graph confirms the symbol is indexed.
- windows_ui_picker_reaches_all_drives (integration, #548): the UI directory
picker's GET /api/browse?path=/ returns no entries and never enumerates
logical drives, so drives other than the system drive (D:\, E:\) cannot be
selected. handle_browse in src/ui/http_server.c uses opendir without a
GetLogicalDriveStrings root case. Needs a UI build (cbm-with-ui) and >1 drive;
otherwise it reports a precondition (exit 2).
Also records additional ruled-out seed areas (get_code_snippet sanitizes
non-UTF-8 to U+FFFD #530.3; stdio handshake/flush works #513/#530.1/#635; mapped
subst-drive indexing keeps the DB #227/#367) and cross-platform items left out
of this Windows-only PR (#530.2 nested gitignore, #530.5 .git/info/exclude,
#530.4 libgit2 build, #581 memory soak).
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Adds Windows-only red tests and analysis for native Windows failures found
during a Windows red-test campaign. This change contains no production fixes.
- windows_non_ascii_repo_path_preserves_definitions (integration): byte-identical
TypeScript fixtures indexed under non-ASCII parent paths (Latin-1 accents,
Cyrillic, CJK, Greek) extract zero definitions and only File/Folder nodes
(5 nodes / 4 edges) versus the ASCII baseline (12 nodes / 20 edges / 5
definitions). The pipeline source readers open files with fopen() on a UTF-8
path, which the Windows CRT interprets in the active ANSI code page; directory
discovery already uses the wide API, so files are listed but never parsed.
- windows_cli_non_ascii_repo_path_is_honored (integration): the documented
`cli index_repository` entrypoint rejects a non-ASCII repo_path because main()
does not read the wide command line, so argv arrives in the ANSI code page.
Both reproduce at the product surface (real MCP process, real stdio, real SQLite
DB), are deterministic, and pass on Linux/macOS. A PowerShell runner builds the
binary and runs the suite; standard-library Python only. See
tests/windows/RED_TEST_ANALYSIS.md for environment, commands, ruled-out seed
areas, and suspected fix locations.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Martin Vogel