Completes the move started on Windows. `update` prints the install script's
command and exits 0 on every platform; the download/extract/chmod/exec sequence
is compiled out of release builds entirely, and the MCP background thread that
ran cbm_popen("curl ... api.github.com ...") on first tool call is gone.
The installers now place themselves beside the binary, which closes the last
gap: `update` referenced a raw.githubusercontent URL purely because an
install.sh-installed user had no local copy to point at. Two details make that
safe.
The source is the install script from the archive we JUST checksum-verified,
never "$0" -- which does not exist under `curl | bash`, and would pin the user
to the OLD installer forever. Since the script ships inside the verified
archive, it inherits that verification instead of needing its own.
And it is published by atomic rename, never written over the live path. Bash
reads a script incrementally by byte offset, so overwriting the file it is
executing continues reading NEW bytes at the OLD offset: silent, bizarre
corruption. cp to a temp name then mv -f swaps the directory entry while the
running shell keeps its original inode. Windows follows the same rule via
Copy-Item + Move-Item even though PowerShell parses up front.
Deliberately NOT done: spawning the new installer to delete and replace its
predecessor. Fetch remote code -> drop to disk -> spawn -> self-delete is a
textbook stager, and self-deletion is among the most heavily weighted
heuristics there is. It also gains nothing -- `"$DLBIN" install` already runs
the NEW release's install logic, because the binary owns the install.
Net: zero install or download URLs remain in the shipped binary, and the
allow-list entry is retired with it. Both scripts scan CLEAN (0/61) with the
self-placement code in them.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Windows already handed updating to install.ps1, because a running .exe cannot
replace its own image. The remaining platforms now do the same, for a different
and better reason.
An in-process updater is structurally a downloader: it fetches a remote archive,
verifies it, extracts it, marks the result executable and runs it. That composite
lived in every shipped artifact -- along with the release URLs it fetched -- to
serve a command most users run a handful of times. The install scripts already
do all of it, are idempotent (so re-running one IS the update), and run while cbm
is not running.
`update` now prints the exact command for the running platform and exits 0:
unix bash "<dir>/install.sh" (or the curl one-liner when not adjacent)
windows powershell -ExecutionPolicy Bypass -File "<dir>\install.ps1"
Both scripts ship inside their release archive, so the printed path resolves
next to the binary. Flags are still parsed and validated, so `update --dry-run
--ui` keeps rejecting typos rather than silently accepting them.
The updater itself is EXCLUDED from release builds rather than left to dead-code
elimination: build_update_url, download_verify_install, extract_and_install_binary,
checksum fetch, detect_os/detect_arch and the cli_download_* helpers now compile
only under CBM_CLI_ENABLE_TEST_API, which release builds do not define. The C
suite still covers the flow through the activation test seam. Verified against
the release binary: `releases/latest/download` is gone entirely.
Also removes the MCP background update check. It ran on the first tool call and
did:
cbm_popen("curl -sf --max-time 5 -H 'Accept: application/vnd.github+json' "
"'https://api.github.com/repos/.../releases/latest' 2>/dev/null")
An embedded shell command plus a startup network callout, in every session, to
tell users a newer version exists. It shelled out, depended on curl being
installed, and phoned home from every agent session. The install scripts report
versions; this did not need to be in the server.
Smoke follows the same collapse. Phases 6c and 14 asserted a Windows-specific
handoff and a POSIX in-process replacement; they now assert one platform-neutral
contract selected by UPDATE_SCRIPT, and 14a's byte-identity check compares the
driver against ITSELF before and after -- comparing against "$BINARY" passed on
Windows only because Windows has no signing step, while the POSIX fixture
ad-hoc re-signs its copy and so differed before `update` ever ran.
The Linux glibc guard is RELOCATED, not retired. The standard linux asset links
glibc 2.38+ and breaks Debian 11, RHEL 8 and Ubuntu 20.04, so the installer must
fetch the static "-portable" build. That constraint moved from the binary to
install.sh, and the smoke now guards it where the behaviour lives.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Release run 30385988475 lost three Windows smoke jobs to one phenomenon on two
different surfaces. A Windows file that was just written or just executed can
refuse deletion and rename for a moment while a scanner reads it or while the OS
finishes reaping a child. Both conditions clear on their own within moments.
It reads as nondeterminism because it is timing: windows-2025 passed both amd64
variants while windows-latest -- the SAME image, windows-2025-vs2026 -- failed
both, and windows-11-arm failed too.
PRODUCT: uninstall abandoned a live installation.
With one binary per platform, uninstall must remove the very image it is
executing, and on Windows a running image can only be renamed, never
overwritten. That rename got a single attempt, so a transient sharing violation
surfaced to the user as
error: failed to remove ...; completed configuration/index cleanup may remain
with the install left half-torn-down. install.ps1 already retries this exact
operation ten times for this exact reason; the C path now does the same, and
only for sharing/access/lock violations, so a file that is genuinely held still
fails closed instead of spinning.
Proven with a failure-injection seam rather than by racing a real scanner:
3 injected failures must still succeed, 64 must still fail AND leave the target
in place. Deterministic, so it does not need CI to learn whether it works.
HARNESS: the smoke killed itself silently.
Phases that install a binary then `rm -rf` the directory hit the same lock. Under
`set -euo pipefail` that returns non-zero and ends the run WITHOUT printing
anything -- the logs jump straight from "OK 13h" to job cleanup with no FAIL
line, which is what made this expensive to read.
None of those 16 removals is an assertion; they are fixture cleanup, and
_smoke.yml already states the principle ("a temp dir on an ephemeral runner must
never fail the job") that smoke-test.sh did not honour. They now go through
smoke_rmtree, which retries so the disk is actually reclaimed -- these fixtures
hold ~300 MB binaries and runners are disk-tight -- then warns and continues.
Verified: Windows VM full smoke ALL PASSED (previously died after 13h), macOS
full smoke ALL PASSED, activation_transaction + cli 259 passed on Windows, all
10 shell contracts pass, lint-ci clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Windows smoke guards that an invalid PATH seam must fail closed rather than
silently falling back to the live HKCU\Environment\Path. That guard went green
in a way that meant nothing: the seam still refused, and install still printed
"PATH configuration failed", but the process exited 0.
Windows release builds used to compile a separate managed-install path. With one
binary per platform they compile the shared cbm_cmd_install, which since 30b5f12
downgraded ANY failed dry-run plan check to CLI_OK. That commit was fixing a real
complaint -- the dry-run summary was being skipped entirely on Windows -- but it
bundled the exit status into the fix. Keeping the summary is right; reporting
success is not. --dry-run exists to answer "would this install work?", so a
caller that only sees the exit code could not tell a refused PATH probe from a
clean plan.
The summary and the triage note still print; only the status changes. A real
install already failed non-zero here, so the fail-closed behaviour itself was
never at risk -- just its observability.
The smoke also still encoded the launcher-era portable-vs-managed split, which
does not survive one-binary-per-platform:
* 6b expected uninstall to be REFUSED. That refusal existed because a portable
extracted bundle was a different artifact from the launcher-managed install
it would have torn down. The extracted binary now IS the installed one.
* 6c expected update to be REFUSED, contradicting the contract Phase 14a
asserts against a real update: Windows hands off to install.ps1, exits 0 and
prints the command. 6c now pins that same handoff.
* 14a compared binaries with cmp, which is absent from the Windows MSYS shell,
so it reported "different" for two copies of one file. Hashes instead, the
way Phase 12 already verifies the release archive.
* 14f seeded the MCP entry at the RETIRED binary. The old launcher-managed
update rewrote that entry in-process; the handoff does not, so uninstall
correctly declined to remove an entry owned by a different installation and
the phase demanded the one thing it must never do. It now seeds the
installed binary, which is what install.ps1 leaves a real user holding.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every ui-variant smoke job failed on every platform in dry run
30226483380 — and it turns out a ui-variant Phase 15 had never
legitimately passed anywhere: the old probe (a) never passed --ui=true,
but the HTTP UI is a persisted DEFAULT-OFF setting, so a fresh profile
(every runner, every smoke HOME) can never serve; (b) fed stdin an
empty file, but the UI does not pin the process — stdio EOF ends it
cleanly (rc=0) before any poll sees it serve; (c) gave a 10s window to
a server measured to bind ~6s after launch on a FAST host; (d) POSTed
an MCP initialize at /rpc, which speaks the UI's own narrow query
protocol, not MCP.
The rebuilt probe mirrors the drive-listing guard's proven invocation
(--ui=true --port=N equals-form, held-open stdin), waits 30s, and
verifies the API surface with the stateless GET /api/ui-config.
Validated red->green against the release-shaped artifact: the full
darwin ui artifact lane now passes 15a/15b legitimately.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dry run 30206811293: macos-15-intel release smoke failed B3 with a
visibly correct 'semantic: 50' table — BSD grep returns no-match for
the ENTIRE output when any line carries a NUL, control byte, or
invalid UTF-8 (verified: printf poisoned input -> grep -qE fails), so
one bad byte in one row silently unmatches every anchored check.
Close the class at the emission choke point: control bytes and
invalid UTF-8 force the quoted form; quoting escapes controls as
\u00XX and replaces invalid sequences with U+FFFD (RFC 3629
validation, NUL-safe). Tool output is valid UTF-8 by construction.
Unit test proven RED-on-revert (raw bytes emitted) and green with the
fix. The smoke's B3 failure path now also dumps od -c bytes so a
future red names the exact byte.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The staged venue-unification bundle: _smoke.yml onto the shared
wrappers with an extracted-artifact input (CBM_SMOKE_ARTIFACT_DIR),
soak-legs.sh as the one canonical soak entry (quick + query-leak legs
across all venues), protected per-user TEMP roots shared across
venues, clean-disk preflights before every VM/Docker run, win.sh and
run.sh routed through the canonical scripts, soak.yml retired, and
the venue-parity contract (whitelist walker: workflows may provision
or call canonical entries, nothing else).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The fuzz harness swallowed the target's exit status through '|| true', so
a SIGSEGV counted as a pass; the status now propagates (a planted crash
yields 139) and payloads derive from a logged, replayable seed, with a
missing python3 failing the gate instead of generating zero mutations.
The Windows guard runner classified unknown exit codes as skips; the
contract is now explicit (0 green, 1 red, 2 precondition-skip, anything
else a failure) and an all-skip run fails as verifying nothing.
The smoke suite gains crash-class detection on the phase-9b tolerance
paths and phase 11 kill handling (rc >= 128 or a missing jsonrpc banner
fails), a free-port pick plus readiness poll for the UI phase, and a
phase-15b failure that actually exits nonzero.
The soak reader resynchronizes on late JSON-RPC responses by draining to
the matching id, and the analysis fails when fewer than 60 percent of
snapshot attempts produced rows — a vacuous analysis previously passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase 9b's fixture installs spawn ephemeral daemons whose mapped
generation backings pin files in the fixture HOME; the cleanup rm then
races the daemon's asynchronous drain and loses nondeterministically on
Windows (Device or resource busy on a generations backing — the same
class the final-phase daemon retirement already fixed for the smoke
cache, two rounds green then one red on identical code). The retirement
is now a shared helper with a cheap not-running probe, called before
every fixture cleanup that follows an install and by the final phase; a
daemon that will not retire stays a hard failure.
The 9b fixtures also route LOCALAPPDATA under the fixture HOME: their
installs staged launcher pairs into the real runner profile, outside
the fixture's isolation and the harness's cleanup reach.
Verified: full VM smoke green end to end (all 9b subphases, phase 17
through the shared helper).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The 40 hidden Windows failures fixed in the previous commit reached CI
unseen because the local Windows leg never validly ran the C suite:
MSYS2's shared /tmp fails the secure-ancestry validation (refusals, not
signal) and the ssh driver piped everything through `tail -40`. This
closes those holes and hardens the gates that let an install failure
scroll past.
- vm-run-tests.sh (new, used by win.sh test / ubsan-test /
trap-ubsan-test): gives the suites CI's exact protected per-user temp
root (owner-stamped, protected current-SID DACL, mirroring the
workflow's pwsh step), shapes the runner's build directory like a real
user checkout (drive-root trees inherit an Authenticated-Users Modify
ACE that profile-rooted checkouts do not have, which the activation
source-directory policy correctly refuses; DACL re-rooted two-step —
directory protect then child /reset, since inheritance flags are
directory-only and a /T re-root leaves files with empty deny-all
DACLs), streams the FULL output, and refuses to report success unless
the runner printed its completion summary.
- smoke: Phase 8 no longer swallows the install exit code. On Windows it
now requires the managed install to SUCCEED and to publish an exact
two-link canonical launcher — a staging refusal used to scroll past as
tolerated noise while the downstream config assertions kept passing.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The account daemon legitimately outlives its last client for a moment
while it drains, holding its log and config DB open. POSIX rm does not
care, but on Windows an open file blocks deletion, so the caller's
cleanup of the smoke cache raced the daemon's asynchronous shutdown and
failed with "Device or resource busy" after every phase had passed —
first visible on windows-latest once the agent-config phases stopped
failing earlier, and reproduced identically on the local Windows VM.
The smoke now ends with an explicit phase: stop the daemon through its
own lifecycle command, wait (bounded) until status reports not-running,
and give Windows one beat for the final handle close. A daemon that
will not retire is now a smoke FAILURE in its own right instead of a
cleanup accident. Verified red-to-green on the VM and green on macOS.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fixes surfaced while bringing the shared-coordination-daemon branch
green across macOS, the Linux containers, and the real Windows-ARM64 VM,
plus a new stability guard that caught one of them.
- daemon(win): the CLI teardown transition latched a failed release as
permanent, but a Windows participant-state release must briefly
try-hold the shared startup/legacy gates and legitimately collides
with a concurrent one-shot's teardown; the release is retriable by
contract (it always retains the transition), so parallel one-shot
commands no longer report "CLI coordination cleanup failed" despite
succeeding. Found by the new stability guard's churn section.
- cli/main(win): install/update/uninstall and the Augment hook-script
removal derive the managed launcher's identity from its plain
drive-form path (cli_windows_plain_utf8), not the \\?\ extended-length
form, so agent-config ownership matches on Windows and uninstall no
longer leaves the MCP entry or owned hook scripts behind.
- security: the `daemon start --open` browser launch is now shell-free
(ShellExecuteW on Windows, cbm_exec_no_shell elsewhere) instead of
system(); the CORS origin check spells out its two literal loopback
URLs so the static URL audit sees a complete value. Both clear the
Layer-1 allow-list audit.
- coordination: version-cohort lock retries now sleep a per-process
jittered interval — fixed-period retries can phase-lock two
participants so one starves. The activation-quiesce test's
observation window now covers worst-case candidate staging (three
tamper-defense hashes over a ~1 GB sanitizer binary exceed the old
30 s budget on container I/O).
- smoke/soak: the agent-config smoke matches Windows config paths
through their escaped-backslash quoted form; the soak parser skips the
update-available banner before the JSON summary and gained a one-shot
CLI admission-churn phase inside the RSS/FD leak window. The build-dir
safety fixture creates a real symlink/junction (MSYS2 ln -s otherwise
deep-copies) so its traversal-refusal contract is genuinely exercised,
and the symlinked-agent-roots test demotes its planted links to an
unprivileged owner when run as root so the refusal is really tested.
- tests(win): a new daemon-stability guard exercises the parameter
surface, hook fail-open with its rate-limited notice, start-twice and
occupied-port handling, busy-stop refusal against a live MCP session,
kill -9 crash recovery, and sequential/parallel one-shot churn.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.
Daemon lifecycle:
- daemon start/stop/status subcommands. `daemon start` launches a
PERMANENT daemon (spawn shape is byte-exact argv; survives idle
periods and session ends) and reports an already-active daemon
instead of failing. Permanence is honored at every stop latch:
last-committed-client disconnect, host initial-client window,
coordinator release, and application final-session close — a
permanent daemon also keeps admitting new sessions after its last
one closes.
- daemon stop refuses while sessions are active and lists the blocking
peers (pid/role) that must finish first; an idle daemon drains
through the activation-shutdown machinery with the ACK ordered after
connection interrupts. A second stop is idempotent. The wire ops are
no-cohort first-frame requests with peer fingerprint authentication,
so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
keep their local supervised path). A cold CLI run that had to spawn a
temporary daemon prints a hint that `daemon start` removes the
per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
emits a visible, rate-limited notice (Claude-dialect systemMessage
plus stderr for other dialects) and always exits 0 — augmentation is
never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
older-build permanent daemon (strict semantic-version triples only;
dev builds never auto-drain) and the build-conflict message names
`cbm daemon stop` as the manual escape hatch.
Windows IPC/runtime (real-VM verified):
- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
path used to destroy its listening pipe instance on every 20 ms poll
timeout; a client attaching in the teardown window was severed or left
on an orphaned pipe object whose HELLO no server handle could ever
read, absorbing the connect until the client's own timeout expired.
The pending connect now survives poll timeouts and nothing is
destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
server handle can discard a just-sent response before the peer reads
it (POSIX stream sockets never lose buffered data on close). A bounded
cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
immediate on interrupted connections) now precedes close in
runtime_worker_finish and runtime_reject_inline, so hello-conflict,
capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
transport half-close, so close_begin now announces departure with an
explicit frame (ordered after APPLICATION_CANCEL, before the local
interrupt); the server releases the client's admission on receipt
instead of waiting for the handle to close. Admission-drop timing is
now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
request publication under the send lock, best-effort sends the active
token's APPLICATION_CANCEL frame, then interrupts local I/O; the
server cancels MCP/subprocess work promptly. Contract tests accept
both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
the requester's license to act on "snapshotted and draining", so every
connection interrupt is now initiated before the ACK is sent; a
session could previously get one more request serviced after the
requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
timer granularity) burned ~1.6 s against permanently obstructed paths,
stalling hello rejections past the client's timeout. The retry budget
is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
nonblocking send() on Windows is absorbed wholesale into AFD kernel
buffering regardless of SO_SNDBUF, so send deadlines and interrupts
could never engage against a slow peer (and the full payload was
pinned in nonpaged pool). Slicing restores a deterministic
backpressure point; a test hook pins SO_SNDBUF for the deadline and
interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
deadline, explicit connection states, refusal to free a server while
a listener-owned connection is active.
Windows long-path support:
- Central path-aware wide conversion (canonicalize via GetFullPathNameW
and prepend the extended-length prefix for absolute paths >=240) at
the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
store opens, and the daemon build-fingerprint/log paths. Deep managed
installs (a 64-hex generation directory routinely exceeds MAX_PATH)
now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
ancestry validators now operate in the extended-length namespace;
the launcher path is canonicalized (and prefixed when deep) once at
entry so every downstream exact-string comparison stays
form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
GetFileAttributesW) so non-ASCII install paths survive argv[0]
resolution.
Windows launcher install/uninstall transaction:
- FileRenameInfoEx names are NUL-terminated in an over-allocated
buffer. FileNameLength governs per the contract, but filter drivers
read FileName as NUL-terminated and appended adjacent heap bytes to
created names — a flaky, garbage-suffixed rename target. Both the CLI
and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
.cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
chars so the bare rename target stays under the FileRenameInfoEx
NT-conversion ceiling at guard depths; 64 bits still uniquely
identify the generation.
- When the running launcher's mapped generation backings pin .cbm
against rename, the backings are relocated to activation-<pid>-N
.retired tombstones beside the install (a mapped image may be renamed,
never deleted; the launcher's liveness-guarded sweep reclaims stale
tombstones). Every relocation is recorded, and a FAILED uninstall
reverses the moves after restoring .cbm — via MoveFileExW with
extended-length paths on both arguments, since the deep generation
target is beyond the handle-based rename's bare-path reach — so a
restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
out so the tree is shallow enough for the detached cleanup's rd, and
the cleanup's working directory strips the extended-length prefix
(CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
(CopyFileW destinations, CREATE_NEW tombstones, probe directories)
are explicitly owner-stamped so the exact-owner validators hold on
runner images; guard fixtures stamp hand-built trees the same way.
Diagnostics, tests and infra:
- diagnostics: discovery is now an always-delivered JSON control record
(new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
paths containing spaces; placement honors $TMPDIR with /tmp fallback
via a diagnostics-local helper; the soak parser reads the JSON record;
documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
log levels, TMPDIR-with-spaces, and native Windows output-contract
assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
parents with production-shaped ancestry (LocalAppData on Windows, via
th_secure_runtime_parent_new) — the runtime ancestry validation
correctly refuses temp roots whose ancestors grant mutation rights to
Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
drive the documented startup-owner publication flow before reading
generation-bound endpoint addresses. This turns the 26 Windows
failures previously visible in CI's full-test job green without
weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
contract including failed-uninstall restore and immediate reinstall
after uninstall; new daemon lifecycle and reworked hook-augment
guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
(it was set by CI but never read, so perf suites ran everywhere);
four throughput/bench suites are classified as perf, the heavy
store_arch suite moved to the slow-timeout tier, and two
wall-clock-sensitive assertions were rewritten as invariant checks
with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
contract, soak daemon-recovery contract, path-safety helper, the
Windows VM worktree-sync contract wired into scripts/test.sh, and
vm/win.sh guards building its clean embedded-UI product in an
isolated BUILD_DIR so it cannot clobber the incremental test build.
provision-windows.sh now installs Node.js for the guards UI build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Windows dry-run failure prints an error (which passes the
install/skill/mcp/agent check) but no "dry-run" indicator, and the
smoke never echoed the captured output on THIS branch, so the actual
error stayed hidden. Echo it. (Root: Windows install routes through
cli_windows_managed_install, which errors on launcher/payload source
verification before emitting a dry-run plan - the next run will name
which check fails.)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
pr-smoke's mcp_run discarded the frontend's stderr (2>/dev/null), so a
first-session daemon start/connect failure - which with the daemon
architecture surfaces ONLY on stderr, stdout being reserved for
JSON-RPC - showed up as a bare "no initialize response (id:1)" with
nothing to diagnose (exactly what an isolated ubuntu pr-smoke failure
just produced). Keep the frontend stderr and, on an id:1/id:2 miss,
dump it alongside the durable cbm-daemon.log and daemon-conflicts.ndjson
tails from the cache root. No behaviour change on success.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The PR CI caught two consumers the format migration missed — both parsed
tool output by its old shape and failed silently or with a wrong count:
- hook-augment built its PreToolUse additionalContext by reading the
legacy 'results' object-array from search_graph format:json; after the
json-tree reshape that key no longer exists, so the hook emitted nothing
(caught end-to-end by tests/windows/test_hook_augment.py). The parser
now walks groups -> qn_prefix/file + column-ordered rows. A new local
test feeds the parser the LIVE server envelope, so any future drift
between response shape and hook parser fails in the normal suite on
every platform — not only in the Windows CI guard. Reproduced RED
(ctx NULL, the exact CI failure) before the fix, GREEN after.
- smoke-test.sh carried six old-shape parsers (callers[N], rows[N]{,
clusters[N], semantic[N]{, and the assert_toon_table helper); the tree
headers never matched, so trace verification read 0 callers. All six
updated to the tree contract; the full smoke script passes end-to-end
locally.
cli suite 222 passed; lint clean; scripted smoke ALL PASSED.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The 8e gate compared uname -s to the bare literal MINGW64_NT, which
never matches the real value (MINGW64_NT-10.0-...), so the POSIX
branch silently ran on Windows and looked for the extensionless shim
that #929 no longer installs. Branch by platform prefix: Windows
expects the .cmd shim and the absence of the legacy extensionless
twin; POSIX unchanged. Content checks (never blocks, delegates to
hook-augment) now run on both.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the compact-output pass across the remaining query tools
(consumers are LLM agents; every response byte is context tokens):
- trace_path: callees[N]{qn,hop}/callers[N]{qn,hop} tables (risk/test/
args columns per flags); the per-hop JSON key envelope was 84% of the
payload. 3,589B -> 1,634B on a representative trace.
- query_graph: rows[N]{cols}: table (columns double as the header);
format:"json" restores columns/rows arrays.
- search_code compact mode: results/raw/dirs tables; drops the
field (duplicate of the qn tail). 2,488B -> 1,135B for 17 hits.
- search_code full mode: per-hit source capped at a 60-line window
anchored on the first match, with source_start/source_truncated
markers — uncapped whole-symbol dumps ran to 142KB per response; the
complete symbol stays one get_code_snippet call away.
- get_architecture: default (no aspects) is now a summary (languages,
packages, entry_points + always-on totals/label/type counts) with an
aspects_hint — the old default rendered EVERYTHING including the full
file_tree (94KB -> 4.5KB -> 2.8KB with TOON tables); all sections
emit as TOON tables, aspects:[...] and ["all"] keep full access
(52KB vs 94KB even for the explicit full dump).
- get_code_snippet: drops the property-blob enrichment (41% of the
response; signature/docstring are literally in the source; fp/sp/bt
never belonged there). Metrics remain reachable via search_graph
fields=[...].
- list_projects: per-project 12-field git block (mostly null) replaced
by the branch name; name/root/nodes/edges/size kept.
- search_graph description: in/out documented as TOTAL all-edge degree,
NOT caller/callee counts (agent evals mis-read it as callers).
- format:"json" escape hatch on every converted tool.
Guards: tool_output_byte_budgets pins absolute byte ceilings on default
search/trace outputs (a property blob sneaking back into row emission
blows them immediately); snippet_enriched_properties inverted to pin
the no-spill contract. Suite 5,984 green; smoke updated to the TOON
contract end-to-end.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Tool responses land in an LLM agent's context window, where the legacy
per-node JSON objects were dominated by data no agent can use: the
similarity/semantic pipeline intermediates fp (~450B minhash hex), bt
(body-token bag) and sp (structural profile) plus 14 complexity metrics
were grafted onto every result by enrich_node_properties — ~64% of every
node emission, ~370 tokens per search hit.
search_graph now defaults to TOON (Token-Oriented Object Notation,
toonformat.dev): scalars as 'key: value' lines and results as a
'results[N]{qn,label,file,lines,in,out}:' header + one row per hit.
Published benchmarks show equal-or-better LLM retrieval accuracy at
40-60% fewer tokens than JSON; rows now also carry line ranges, which
the legacy output lacked.
- fields:[...] opts into per-node property columns (complexity,
signature, docstring, ...); fp/sp/bt are blocklisted even there
- format:"json" restores the legacy verbose objects byte-identically;
include_connected forces it (nested neighbor lists)
- BM25 and semantic modes emit the same table shape (rank/score column)
- semantic-only calls no longer run the unfiltered regex search that
silently prepended up to 200 enriched nodes (102.8KB -> 4.2KB)
- default limit 200 -> 50 (BM25 100 -> 50): cheap first page, page via
has_more/offset as before
- new emitter module src/mcp/compact_out.{c,h} (string builder + TOON
scalar/table emission with spec quoting rules)
Measured on a real index (14-hit regex search): 20,793B -> 1,327B
(-94%); BM25 19,667B -> 5,081B; semantic 102,837B -> 4,181B. A typical
discovery session drops ~50.9K -> ~4.7K tokens combined with the
follow-up slices.
Guards: tool_search_graph_toon_never_leaks_internal_fields (sentinel
fp/sp/bt never emitted, even when requested), rewritten
tool_search_graph_includes_node_properties pins default-lean/fields-
opt-in/json-escape-hatch; suite 5983 passed, smoke green.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Check 8d still locked the matcher to the pre-#963 'Grep|Glob' and failed
the pr-smoke leg on all platforms. The matcher now includes Read (the
augmenter injects the coverage note when a not-fully-indexed file is
read; structurally non-blocking, so the issue-#362 gate hazard cannot
recur). The guard now pins the exact new matcher and still rejects
Search/catch-all creep.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
install.ps1 kept detecting arch=amd64 on windows-11-arm because it runs under x64
emulation, where neither $env:PROCESSOR_ARCHITECTURE nor .NET OSArchitecture (on
Windows PowerShell 5.1) reports the real Arm64. Add a CBM_ARCH env override to
install.ps1 (wins over auto-detect; also a genuine escape hatch for emulated
invocations) and have smoke Phase 13 pass the authoritative DL_ARCH. Deterministic
-- no reliance on in-process detection under emulation. Same emulated-arch class
as #907 (uname) and #908.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The persistent windows-11-arm smoke Phase 12a failure was a 404, not a network
error: it requested codebase-memory-mcp-windows-amd64.zip on the arm64 leg. Cause:
DL_ARCH came from `uname -m`, which on windows-11-arm is an emulated x86_64 MSYS2
uname reporting "x86_64" -> wrong (amd64) archive -> 404 (server has arm64). Phase
14 worked because the binary's own detect_arch() is native. Prefer SMOKE_ARCH
(passed from the smoke workflow's matrix.arch) over uname; fall back to uname for
local runs. This is the real cause the earlier curl/proxy/ipv4 attempts masked --
the 404 was swallowed by 2>/dev/null until #905 surfaced it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase 12a swallowed curl's stderr (2>/dev/null), so the persistent windows-arm64
"curl download failed" was undiagnosable. Two changes: (1) add --noproxy '*' so
curl never routes the local 127.0.0.1 test server through a proxy env var -- a
strong candidate since the app's own WinHTTP downloader reaches the server in
Phase 14 while only msys2 curl fails, instantly; (2) print curl's stderr on
failure so the actual reason is visible if it persists. Harmless on all
platforms (the smoke server is always local).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Companion to the raw-JSON deprecation warning committed on this branch (adds the
smoke coverage that was meant to land with it).
Migrate scripts/smoke-test.sh from raw-JSON `cli <tool> '{...}'` invocations to
FLAG form for every tool (index_repository / search_graph / trace_path /
get_graph_schema / query_graph incl. all Cypher checks / get_architecture /
search_code / delete_project), with identical results verified against a prior
run; flag form passes the Cypher query as one argv token, so the old
JSON-escaping helper is removed. Add dedicated guards for an integer flag
(--limit / --depth), a bare boolean (--exclude-entry-points), a repeated array
flag (--semantic-query -> JSON array), piped stdin, --args-file, and per-tool
--help (RC 0 + shows a flag; unknown tool -> RC!=0), plus the deprecation guard
(raw JSON warns on stderr; flag form does not). Non-CLI phases (incl. Phase 10
binary security) unchanged. Full smoke: ALL PASSED.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The "tampered arm64 binary is SIGKILLed (137)" premise is empirically false on
current macOS CI runners for an ad-hoc-signed CLI binary -- the binary has no
CS_KILL/hardened-runtime flag, so a tampered code page is not killed: it
executes the garbage and crashes with SIGILL (exit 132, run 28365724001), not
137. (remove-signature and corrupt-blob both ad-hoc re-sign on exec and run to
exit 0.) So no runtime exit code is a deterministic guard here, and
"tamper -> crash" is near-tautological (zeroed code crashes regardless of
signing).
Assert the real, deterministic integrity invariant instead: `codesign
--verify` REJECTS a tampered copy (the CodeDirectory page hashes no longer
match the modified code), while the untampered binary verifies cleanly (10a).
It is a pure userspace hash check -- no tampered code is executed. The copy is
separate, so the original binary stays intact for the 10e re-sign test.
Refs: github.com/garrytan/gstack#997, github.com/nodejs/node#40827
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Refines the previous 10c change on this branch. Garbling only the signature
blob still ran (exit=0 on dry-run 28360363173): since macOS 11 a binary with
a missing/invalid signature is ad-hoc re-signed on exec by newer macOS and
RUNS, so neither remove-signature nor a corrupt blob triggers the kill.
The reliable "tampered binary is SIGKILLed (137)" trigger is tampering the
SIGNED CODE while leaving the valid signature attached: the kernel validates
each executed page against the intact CodeDirectory hash, finds the mismatch,
and kills the process before user code runs. Zero the entry-point
instructions (LC_MAIN entryoff, extracted dynamically) plus a span of early
__text on a SEPARATE copy, leaving the Mach-O header + load commands intact so
it still parses. The original binary is untouched, so the later 10e re-sign
step stays valid. x86_64 keeps remove-signature (code signing is not enforced
there).
Refs: github.com/garrytan/gstack#997, github.com/nodejs/node#40827
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Smoke test 10c (Phase 10 binary security E2E) verifies that an arm64 binary
with an invalid code signature is SIGKILLed (exit 137). It did
`codesign --remove-signature` then ran the binary, expecting the kernel to
kill it. But since macOS 11, a binary with NO LC_CODE_SIGNATURE is ad-hoc
re-signed on exec by newer macOS and RUNS (exit 0) -- so the test went flaky
and then consistently red on updated CI runner images (dry-runs 28350650225
and 28354735368 both failed only here; every other job, including the full
cross-platform test matrix, passed).
Corrupt the signature blob in place instead, leaving the LC_CODE_SIGNATURE
load command intact: AMFI then sees "signed but invalid" and rejects the
binary before any user code runs (deterministic 137). Only the signature
blob is garbled (not the code), so the later 10e re-sign step stays valid --
it replaces the blob and the code is untouched. x86_64 keeps
remove-signature (code signing is not enforced there).
Refs: github.com/garrytan/gstack#997, github.com/nodejs/node#40827
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
mimalloc bumped to v3.3.2 (MI_MALLOC_VERSION 30302), which adds static-CRT malloc override on Windows. Makefile: define the real MI_MALLOC_OVERRIDE=1 ONLY on MinGW (enables the static-CRT override entry points — the generic Windows fix for #424); on Unix keep static-link-order override (defining MI_MALLOC_OVERRIDE there compiles alloc-override.c's forwarding defs and crashes macOS with 'mi_free: invalid pointer' since system libs keep the system allocator). The old -DMI_OVERRIDE=1 was never read by mimalloc source — kept as a prod marker. smoke-test.sh: hard-fail on any 'mimalloc: error' in stderr. Regenerated scripts/vendored-checksums.txt for the new tree (security-vendored.sh passes; no dangerous calls in vendored mimalloc).
The standard linux release binary dynamically links glibc 2.38+ and
GLIBCXX_3.4.32, so it fails to start on Debian 11, Ubuntu 20.04/22.04,
RHEL/Rocky 8/9, Amazon Linux 2, etc. — yet install.sh, the npm and PyPI
wrappers, and the binary's own self-update all fetched it by default,
contradicting the "single static binary" promise.
Point every linux install + self-update path at the fully-static
"-portable" asset (gcc -static), which has no glibc floor. macOS/Windows
are unaffected and unchanged.
- install.sh, pkg/npm/install.js, pkg/pypi _cli.py: select -portable on linux
- src/cli/cli.c: self-update download URL AND checksum archive name both use
-portable on linux (they must match or the update fails checksum verify)
- scripts/smoke-test.sh: assert linux self-update targets the -portable asset
- scripts/ci/check-glibc-compat.sh: new guard — runs the binary inside
debian:bullseye (glibc 2.31) and asserts it starts
- _smoke.yml: run the guard on the portable binary in smoke-linux-portable
Reproduced: standard binary -> "GLIBC_2.38 not found" on glibc 2.31;
portable binary runs cleanly.
The C++ heap-corruption crash only manifests in the production binary (MI_OVERRIDE=1), so the smoke job (which runs the prod binary) is where it reproduces — most reliably on Windows. A ~10k-line templated header exercises the parse-churn path; indexing it must succeed.
Phase 16's FIFO writer-close surfaced as POLLHUP rather than a clean POLLIN+EOF, which the server's poll()-based read loop does not treat as shutdown, so the probe left the process alive and the smoke job failed on every platform. Switch to a plain '< /dev/null' immediate EOF — the server exits reliably (~1s) via its normal getline-EOF path, and it is fully portable (POSIX + MSYS2 bash), so no OS gate is needed. Verified end-to-end: bash scripts/smoke-test.sh reaches 'ALL PASSED'.
Phase 16 of smoke-test.sh launches the stdio MCP server with its stdin held open (a running instance), closes stdin to trigger shutdown, and asserts the process terminates within ~6s — guarding the orphaned-background-process failure mode behind #406 from the prod binary on every platform. POSIX-gated (Windows uses job objects; mkfifo is unavailable on MSYS2).
The cypher unsupported-function check captured stdout, but the CLI prints the parse error to stderr (which cli() redirects into $CLI_STDERR) and exits non-zero, leaving stdout empty — so the assertion always failed once the run reached the smoke stage. Verify the exit status and the stderr channel where the binary actually reports the loud failure. Confirmed locally: full smoke-test.sh passes.
cyp_first_cell built the query_graph JSON by interpolation, so a query with a double-quoted string literal (e.g. replace(f.name, "a", "A")) produced invalid JSON and an empty result (FAIL: replace empty). Escape embedded double-quotes before building the JSON.
#272: '&' is neutralised by the single-/double-quoting of the grep/Select-String/
git commands, so it no longer fails validation. A relaxed validate_search_path_arg
(cbm_validate_shell_arg minus '&') is used for search_code + detect_changes
root_path/file_pattern; all other shell metacharacters stay rejected and
base_branch keeps the strict check.
#282: search_code defaulted regex=false, so 'foo|bar' matched the literal pipe
and silently returned 0 results. The result now carries a warnings[] array that
flags a literal '|' under regex=false (with the regex=true hint), so the trap is
visible instead of looking like a legitimate no-match. (The related invalid-regex
silent-empty case was already fixed in #283.)
Also: every search_code result now reports elapsed_ms, and a perf warning fires
when a search exceeds 5s (also logged as search.slow) so slow calls are visible.
Tests for both issues + the multi-word/invalid-regex no-regression; generous
search_code smoke coverage (basic+elapsed_ms, literal-| warning, '&' acceptance).
Closes#272Closes#282
Martin Vogel