Review on #2199: cbm_test_runtime_init ran more than a hundred lines
before the only EXIT trap, so under `set -e` a failure in the fixture
mktemp or its cygpath conversion left the private root behind, and the
fixture trap ran smoke_rmtree ahead of the runtime cleanup.
Arm `trap 'cbm_test_runtime_cleanup "$BINARY"' EXIT` immediately after
a successful init, as soak-test.sh and memlab.sh do. The fixture trap
that replaces it now retires the private daemon first and removes the
fixtures second, so no earlier cleanup step stands between the exit and
the runtime cleanup; on Windows the retirement is also what unblocks
the fixture rm.
tests/test_smoke_runtime_isolation_contract.sh pins the early trap to
the window between the init call and the first fixture; the pin fails
against the previous head.
Part of #1696.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Anton Standrik <astandrik@yandex-team.ru>
scripts/smoke-test.sh retires the account daemon from seven call sites,
but its wrappers sandbox only HOME/TMPDIR/CBM_CACHE_DIR and only
CBM_RUNTIME_DIR moves the daemon rendezvous, so every retirement reached
the operator's live daemon. Source scripts/test-runtime.sh so every
product process runs under a harness-owned runtime and cache, and clean
that root up from the EXIT trap.
Add tests/test_smoke_runtime_isolation_contract.sh, which fails before
this change, and wire it into scripts/test.sh.
Part of #1696.
Signed-off-by: Anton Standrik <astandrik@yandex-team.ru>
Indexing broke on x86-64 Linux: the index worker refused to start and
SIGKILLed its own process group, so every venue reported nothing but
"index worker ended with killed (exit=-1, signal=9)".
glibc places a thread's static TLS block inside the stack allocation it
makes for that thread. This branch's new thread-local caches -- a
512-slot field cache (~24 KB), per-depth tree cursors (~4 KB) and two
parked CBMArenas (~8 KB; an arena carries blocks[256] + block_sizes[256])
-- took static TLS from 47,905 to 93,513 bytes. The 64 KB parent-death
watchdog thread then could not be created at all, and a worker without
process-tree containment correctly refuses to index.
Four layers, because there were four things wrong:
- The big thread-locals move to the heap behind a thread-local POINTER,
allocated once per thread and reused, each with a correct fallback if
allocation fails (the field cache and cursor pool are caches; a miss
is slower, never wrong). Static TLS is back to 52,233 bytes.
- PARENT_WATCHDOG_STACK_SIZE 64 KB -> 256 KB. "The watchdog only polls,
a tiny stack suffices" was true when written and false once TLS
reached ~50 KB, because the TLS block comes out of that same
allocation.
- cbm_thread_create retries with the default stack on EINVAL. A stack
size is a hint about what a thread needs; the platform refusing the
hint is not a reason to fail to create the thread.
- tests/test_thread_stack_tls_contract.sh (Step 5a of scripts/test.sh)
keeps static TLS a quarter clear of the smallest requested stack,
reading that smallest size from the sources so a new small stack
tightens the gate automatically.
Two diagnostics that made this findable at all, and that stay:
smoke-test.sh prints the worker log its failure message names (CI deletes
that sandbox with the job, so the one artifact naming the cause was
unreadable), and worker_containment_unavailable() now reports WHICH of
its conditions failed, with pid/pgrp/ppid.
Verified in the amd64 container, the venue nothing on the local ladder
covers: baseline main indexes 9039 nodes/15067 edges, this branch before
the fix refuses to start the worker, and after it indexes 9039/15067.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fix-forward after rebasing onto main (545 commits, 26 conflicted-file
resolutions). Where main had shipped a behaviour in the meantime, main
wins; the PR keeps its output contract on top of it:
- search_code: the scan runs under main's supervised, deadline- and
output-bounded runner and feeds the branch's streaming classifier from
the runner's output file; the POSIX wrappers map grep's no-match to 0,
so every non-zero exit fails closed (no exit-1 special case).
- trace_path: strategy/confidence precede args in header and rows (#1542),
in both the tree table and the json legs.
- cypher: relationship expansion is never capped (a cap falsified
aggregates) and unlabeled MATCH scans every candidate; `truncated` is
raised only by a real ceiling — a clamped hop range is probed one hop
beyond the cap and reports truncated only when a candidate exists there.
The plain BFS and the trail BFS both probe one row past max_results so
saturation is observable instead of silent.
- search_code full mode and the final json use main's U+FFFD sanitizer
instead of '?' replacement.
- search_graph json keeps an empty `groups` array for semantic-only calls;
list_projects json echoes offset/limit next to next_offset and accepts
main's include_details spelling for the stats projection.
- discovery budget: main added get_file_outline, compare_graphs, manage_adr
set_sections and the debug/include_details parameters; their descriptions
are trimmed to the lean style and the ceiling moves from 15 to 18 KiB.
Tests that asserted a JSON default now request format:"json"; the
fail-closed search test uses an unreadable regular file (a missing or
non-regular indexed path is deliberately not a scan operand on main); the
non-regular fixture gives its real sources File nodes, as every indexed
project has; list_projects total counts listable projects (the 0-byte
ghost is never a row, so it is not a page either).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The rebase onto main brought the Bash-enabled PreToolUse matcher
(Grep|Glob|Bash), which the exec-form tests still counted as Grep|Glob.
The Windows lifecycle test seeded the .cmd shell spelling as the converged
identity; the shell-free registration converges every owned shell spelling
onto one exec-form entry per matcher instead. The test now asserts that,
plus their removal on uninstall, and hands cbm_install_agent_configs the
'/'-spelled binary path cbm_cmd_install registers (cbm_mkdtemp returns a
native '\' temp root on Windows, which uninstall never derives).
Smoke 8aq proved Claude's gate registration by grepping the shim name out
of settings.json; the exec form carries no shim name, so it now checks the
binary-plus-'hook-augment' PreToolUse hook. 9d's shim-name marker would
otherwise pass vacuously after uninstall; the 'hook-augment' argument is an
owned marker there too.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The 8al-node probe imports the generated Pi extension in bare Node with an
empty node_modules. That was a sound stand-in for the Pi host only while
the extension imported nothing but node: builtins; the first package
import (`typebox`, Pi's schema library for tool parameters — a real Pi
dependency, verified against pi-coding-agent 0.84.4 on #1809) can never
resolve there, so the probe fails on a correct extension and would pass a
wrong one just as readily.
Give the probe what the host gives the extension: a Proxy-backed
`typebox` stub in the probe's node_modules (any Type.X(...) returns a
descriptor, so the probe never tracks which helpers the emitter uses — it
only exercises the tool lifecycle), plus an explicit allowlist: every
import specifier in the generated file must be a node: builtin or a
package on PI_HOST_PACKAGES, otherwise the probe fails naming the
specifier. That turns the guarantee the empty node_modules gave by
accident (no surprise dependency) into an asserted contract.
Verified locally by running only the 8al-node block against extensions
generated by two binaries:
main@5802ccdd extension, old probe: OK; new probe: OK
#1809 extension, old probe: ERR_MODULE_NOT_FOUND typebox (the CI red)
#1809 extension, new probe: OK
#1809 extension + an injected 'lodash' import, new probe:
FAIL 8al-node: ... packages the Pi host does not provide: lodash
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Grok Build (xAI's grok CLI/TUI) already reached codebase-memory-mcp through
its Claude/Cursor compat layer, but a Grok-only user got nothing. Detect
$GROK_HOME (default ~/.grok) or a grok CLI and install:
- an owned [mcp_servers.codebase-memory-mcp] managed block in config.toml;
a same-name foreign table is left byte-identical because a second header
would make Grok reject the whole file, while the known owned shape written
by `grok mcp add` is adopted. No env forwarding: grok 1.0.5 passes the
full parent environment to stdio servers (verified), unlike Codex (#1562).
- an owned rules file under $GROK_HOME/rules/ (always scanned, all projects)
- the shared skill under $GROK_HOME/skills/
- Scout/Verify/Auditor agents with named-server mcpInheritance and the tier
allowlist spelled out as the server__tool ids Grok's search_tool/use_tool
dispatcher accepts (inheritance filters per server, never per tool)
- hooks withheld: Grok's passive events discard stdout and PreToolUse honors
only deny/rewrite decisions, so there is no context-output contract.
Verified against grok 1.0.5: inspect discovers the server, agents, skill and
rules; mcp doctor handshakes and discovers 15 tools. --clients=grok token,
README/npm/llms/landing-page matrices (43 -> 44 surfaces), and smoke coverage.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Align the exact Claude PreToolUse smoke contract and documentation with the Bash-enabled search matcher.
Signed-off-by: Jake Abendroth <contact@jakea.net>
Five field reports in the 24 hours after v0.10.0 all pointed at the same thing:
gates that were right in principle refused real, ordinary setups, and then
failed to say why. Per the consolidated strictness decision, each gate keeps the
protection that matters and drops the part that was refusing legitimate users —
and every refusal now names what it refused and how to proceed.
**Daemon image gate: npx and every ephemeral install path (#1539, #1383).**
The admission check treated "the peer's image hashes differently" and "the
peer's image cannot be examined at all" as one failure. The second is what
`npx codebase-memory-mcp` always produces (ephemeral cache path,
unfingerprintable), so every npx-invoked client was rejected — and, because the
client never reported it, agents saw a transport that closed mid-handshake with
zero bytes on stdout. Reported by @wassolles with the admission path already
read and the fix space mapped.
An unverifiable image is now admitted: the rendezvous HELLO immediately above it
has already proven semantic version, build fingerprint, and protocol/store/
feature ABI, and the image check was trading that real proof for an unavailable
one. It logs daemon.client_image_unverifiable_admitted so the weaker check is
never invisible. A fingerprint MISMATCH — the tamper case the gate exists for —
still rejects hard. Separate test seams keep the two modes testable apart.
**Client bootstrap failures are no longer silent (#1539).**
An MCP client that cannot reach the daemon now emits a JSON-RPC error on stdout
naming the reason, plus the same text on stderr. Previously the reason sat in
bootstrap_result.message and the process exited having written nothing at all.
**POSIX activation: group-writable ancestors (#1535, discussion #1526).**
activation_directory_secure required no group or other write bit on the install
directory AND every ancestor. WSL2 ships ~ and ~/.local at 0775, as do several
distro skeletons and any site using a shared primary group, so install.sh failed
for a large fraction of Linux users — reporting a policy refusal as "activation
transaction I/O failed", which sent reporters after disk errors and filesystem
types. Root-caused by @AmirF194 in a clean ubuntu container; @shochdoerfer and
@iandol confirmed independently.
World-writable ancestors are still refused (any local user could swap a path
component mid-transaction). Group-writable ancestors are now warned about and
admitted. The LEAF directory stays strictly owner-private — that is where the
binary is published, and group write there would let another account replace the
executable between validation and exec. Refusals now name the directory, its
mode, and which rule refused.
**The obsolete ui/standard chooser (#1538, from discussion #1526).**
v0.10.0 consolidated to one archive per platform with the UI always embedded,
but `update` still offered a variant choice: "ui" could only 404, and "standard"
quietly WAS the UI build. Reported by @iandol upgrading 0.9.0 -> 0.10.0. The
chooser, its --standard/--ui flags, and the ui- URL plumbing are removed, along
with the CBM_VARIANT=ui remnant in the npm installer.
Already-released 0.9.x binaries cannot be fixed retroactively, so the release
workflow now publishes byte-identical ui-*-named alias assets — their updaters
work again with no user action. The aliases are uploaded AFTER the VirusTotal
gate: they are the same bytes as archives it already cleared, and uploading them
earlier would duplicate every object in the scan set and the provenance manifest.
**macOS install noise and attribution (#1537).**
install.sh silenced the "No such xattr: com.apple.quarantine" line, which is
what happens when a curl-downloaded archive carries no quarantine attribute —
harmless, and it became the title of a bug report about an unrelated failure.
The session-stop refusal now points at `daemon status` to list the client
processes actually holding the daemon, instead of asserting sessions exist and
leaving the reader to guess. Reported by @listepo.
**Riders.** hatchling is pinned in pkg/pypi (an unpinned backend resolved fresh
inside `python -m build` is what emitted Metadata-Version 2.5 and broke the
v0.10.1 publish); SECURITY.md's supported-versions table moves to 0.10.x.
Tests: separate seams for unverifiable vs mismatched peer images with a test per
outcome; activation refusal must name directory + mode + rule; a group-writable
ancestor must stage successfully. The update tests drop the flag that no longer
exists. Verified against each reporter's environment shape.
**Open security alerts (all three, OSSF Scorecard).**
- HIGH, binary artifact: an 8.8 MB compiled Go ELF wrapper had been committed at
pkg/go/codebase-memory-mcp by accident. Removed, and both it and its .exe
sibling are gitignored so `go build` in that directory cannot repeat it.
- HIGH, GHSA-2v37-7h3g-55p8: nanoid < 3.3.17 loops forever when a custom
generator is called with size 0. It reaches us transitively (postcss -> vite),
so it is pinned through the existing graph-ui overrides block rather than
promoted to a direct dependency; the lockfile resolves 3.3.18.
- MEDIUM, unpinned pip command: the publish step installed build/twine by
version only, leaving the whole transitive graph resolved at run time.
pkg/pypi/requirements-publish.txt now hash-pins the complete toolchain (316
hashes), generated on a linux/amd64 python:3.12 image so the wheels match what
ubuntu-latest resolves, and the step runs pip with --require-hashes. Verified
by installing from it in that same image.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three regressions shipped in 0.10.0 share one failure shape: an empty result
with a success status, indistinguishable from "nothing found" for the LLM
clients that are cbm's primary consumers.
1) structuredContent {} on the whole tree-format surface (#1522 bug 1).
#1488 replaced the duplicated payload with an EMPTY structuredContent
object while every tool still declared a blanket permissive outputSchema.
Spec-honoring clients (Claude Code among them) treat structuredContent as
THE result when a schema is declared, so search_graph, trace_path,
query_graph, get_architecture, search_code, and detect_changes all rendered
as literally "{}" on their DEFAULT format, on every platform. The corrected
contract: no tool declares an outputSchema (tool output is
format-parameter-polymorphic — no static schema is truthful), JSON-object
payloads keep their parsed structuredContent, errors keep
structuredContent.error, and text-shaped payloads carry NO structuredContent
key at all — which also preserves #1375's no-duplication win.
2) Frontend queue overflow killed the session (found by the #1522 sweep).
Any 7+ requests pipelined in one stdin burst — an agent issuing parallel
tool calls does exactly this — overflowed the 8-frame frontend queue, which
failed the whole session: rc=1 with ZERO bytes of output, every buffered
response lost. A full queue is now backpressure: the stdin reader blocks
until the worker drains (bounded by the same stop/fail flags every teardown
path already sets); only a single frame larger than the entire 12 MiB byte
budget — which could never be admitted — remains a hard failure.
3) config get printed "" with exit 0 for every unset and every unknown key
(#1522 bug 2). list printed stored-or-DEFAULT while get printed
stored-or-EMPTY, and no subcommand validated key names, so a typo was
indistinguishable from a correctly-read setting. One config-key table now
drives help, list, get, set, and reset: get prints the stored value or the
key's real default (the same fallback the runtime readers use), and unknown
keys error with exit 1 on get, set, and reset alike.
Tests — each RED on the pre-fix tree and RED again on revert:
* test_mcp.c: text results carry no structuredContent key; tools/list
declares no outputSchema; the tool-table guard now binds all three
branches (absent / parsed-object / error) for every registered tool.
* test_daemon_frontend.c: the over-capacity contract flips from
"session fails" to "backpressure without loss" — the held first request
plus all 32 over-capacity frames are answered and the run closes cleanly.
* test_cli.c: the config command contract — defaults, round-trip, reset,
and unknown-key rejection on all three subcommands.
* smoke-test.sh: Phase 3z rewritten to the corrected structuredContent
contract, new Phase 3z1 (default-format replies usable in schema-honoring
clients, no outputSchema advertised), 3z2 (24 pipelined calls all
answered), 3z3 (config defaults + unknown-key rejection) — all asserted
against the SHIPPED artifact, where #1488's smoke phase previously pinned
the empty-object behavior as correct.
Verified end-to-end on the locally built production binary: all 15 tools
declare no schema; text tools return ABSENT structuredContent, object tools
populated, error envelopes intact; 7/24/64-deep pipelined bursts all answered
with rc=0; config get prints real defaults and exits 1 on unknown keys.
hook_augment (structuredContent.projects) and index_resilience
(structuredContent.status) consume object payloads and are unaffected.
Fixes#1522.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Making REQUIRE_UI unconditional broke all three pr-smoke legs:
FAIL 15a: SMOKE_REQUIRE_UI=1 but this binary serves no embedded UI assets
The PR lane runs scripts/build.sh WITHOUT --with-ui on purpose, so an npm
frontend build does not land on every product PR. Demanding embedded assets
there asserts a property that lane deliberately does not produce.
The guard still exists where it means something: scripts/ci/smoke-artifact.sh
builds --with-ui, packages the real archive and smokes the EXTRACTED result, so
it exports SMOKE_REQUIRE_UI=1 and a frontend-less binary fails there. That is
the lane whose whole job is release fidelity.
Deliberately NOT fixed by adding --with-ui to pr.yml: that would put `npm ci &&
npm run build` on three runners for every product PR to re-prove something the
dry-run and release lanes already gate. PR CI cost is not free and this branch
should not quietly raise it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the collapse to a single shipped composition and replaces the
zero-tolerance VirusTotal gate with a narrow, disclosed policy.
Packaging and installers
- package-release.sh loses --variant; archives are codebase-memory-mcp-<os>-<arch>
with exactly four members. install.sh/install.ps1 lose --ui/--standard.
- The extractor drops CBMUIPK pack parsing and --archive-scope; its scan-set and
association manifests (which the gate depends on) are unchanged otherwise.
- npm/PyPI/Go wrappers: the runtime "set" is one file again. The Windows lock
and race fixes from #1495/#1496 are kept; only multi-file set membership goes.
This also fixes `pip install` on Windows, which rejected the fifth archive
member against a hardcoded four-name allowlist.
- The wrappers' post-download probe moves from --verify-runtime-assets (removed)
to --version, which proves the same thing: the binary executes.
VirusTotal gate
- Exactly ONE detection is tolerated, and only when the engine is Microsoft AND
the label ends in `!ml`. Two or more engines, any non-`!ml` label, any other
vendor, any suspicious verdict and every infrastructure error still block.
- A tolerated object prints TOLERATED:, never OK:, and its counts are recorded
in vt-results.tsv exactly as a blocked one would be.
- append-vt-notes.sh mirrors the policy. It previously hard-failed on any
malicious count, so loosening only the gate would have passed the scan and
then died at note publication. The notes now DISCLOSE a tolerated detection
and link to SECURITY.md rather than claiming "0 malicious" for everything.
Rationale for the tolerance is in the gate itself: the verdict is not a property
of our bytes. It inverts across architectures and link modes, moves between
sibling artifacts of one build, and lands in different variant buckets for the
same source. The same `!ml` family hits llama.cpp, GitHub's own `gh`, Microsoft's
own Go toolchain and Anthropic's Claude installer.
The zero-tolerance contract becomes test_vt_gate_policy_contract.sh, asserting
the full matrix: 1x Microsoft !ml passes and reports TOLERATED; a Microsoft
signature label, a non-Microsoft engine, two engines, a suspicious verdict and
every malformed-response case still block. Its tripwire is narrowed to the
reverted endpoint-verification mechanism rather than the words "false positive",
so it no longer fires on a deliberate in-gate policy branch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Externalizing the integration templates (#1492/#1493) and the UI bundle
(#1501/#1503) was done to reduce the Microsoft `Wacatac.B!ml` surface. It did
not work: across dry runs the flagged artifact count stayed at ~3 and the
detections merely moved between artifacts.
Dissection of run 31286803592 shows there is no structural cause to fix. The
verdicts split across every axis at once — linux-amd64 (dynamic) flagged while
linux-amd64-portable (static) is clean, but linux-arm64 (dynamic) clean while
linux-arm64-portable (static) is flagged. The two macOS binaries have identical
segment structure and split clean/flagged. Siblings from one build landed in
different variant buckets (.B vs .C). Entropy is low everywhere
(code_vectors.bin 4.166, grammar tables 3.464 bits/byte, against 7.5-8.0 for
packed payloads), so the packed-payload hypothesis is excluded too.
So the complexity bought nothing, and installation goes back to being
self-contained: one binary that carries its own UI and agent integration
templates, with no adjacent data file that has to resolve before `install`
works. Only the UI-capable composition ships from now on, under the historical
unsuffixed archive name.
Removed: src/ui/asset_pack.{c,h}, asset_pack_stub.c, asset_manifest_stub.c,
scripts/pack-ui-assets.mjs, src/cli/integration_assets.{c,h},
assets/cbm-integrations.json, scripts/gen-integrations-hash.sh, the
--verify-runtime-assets probe (nothing adjacent left to verify), and the
composition gates A6/A7 whose property is now deliberately inverted.
Restored: scripts/embed-frontend.sh, src/ui/embedded_{assets.h,stub.c}, the
compiled-in hook/adapter template bodies, and the embed/EMBED_OBJS build path.
Kept from the reverted commits, re-applied by hand where a wholesale file
restore would have dropped them:
- cbm_module_path_utf8() in both self-path sites. GetModuleFileNameA renders
through the ANSI code page and mangles non-ASCII install paths.
- the /__cbm/ui-readiness HMAC proof, secure_random and cbm_hmac_sha256, so
`daemon start --open` still waits for a genuine CBM listener.
- X-Content-Type-Options: nosniff on served assets.
- the MinGW noexecstack gate, -lbcrypt, and the cppcheck/zip CI fixes.
Archives are now codebase-memory-mcp-<os>-<arch>[-portable] with exactly four
members (binary, LICENSE, installer, THIRD_PARTY_NOTICES.md). That restores the
names every static package manifest already points at — aur, chocolatey,
homebrew, scoop, winget and glama were all broken by the -ui- rename.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
pr-smoke failed on all three platforms: the binary's install/uninstall render
templates from the integration asset, and every place that stages the binary
without the asset made those operations fail closed with "integration assets
missing". The asset must travel with the binary in EVERY layout, not just the
release archive. Five staging paths were missing it:
- scripts/build.sh — stage cbm-integrations.json next to the freshly built
binary, so `build/c/codebase-memory-mcp install` works straight out of a
build tree (dev, and the base for the smoke fixture).
- scripts/smoke-local.sh — include the asset in the fixture tarball and its
required-sidecar check. Member set and ORDER now mirror package-release.sh
exactly (binary, cbm-integrations.json, LICENSE, install.sh, notices); the
fixture was smoking an archive layout we never actually ship.
- install.sh / install.ps1 — after installing the binary, copy the asset beside
it in the install dir. `install` already publishes a verified copy to
~/.cbm/assets/<version>/, but a later install/uninstall run from the install
dir resolves the asset NEXT TO THE BINARY first, so without the adjacent copy
that lookup misses and a re-install or uninstall fails on a machine that just
installed successfully. Best-effort atomic rename, same shape as the existing
install.sh/ps1 sidecar copy. install.ps1 stays pure ASCII.
- scripts/smoke-test.sh Phase 14 — this phase hand-stages the binary into a
fresh HOME without going through install, so nothing populates ~/.cbm/assets;
stage the asset next to each staged copy so the uninstall it drives resolves.
Verified: scripts/smoke-local.sh on the standard binary now runs clean through
all 16 phases — zero "integration assets missing" / hook_script_uninstall /
agent-cleanup-failed lines (was 5). windows-bundle, smoke-fixture, exec-bit and
no-embedded-scripts contracts pass; install.ps1 has zero non-ASCII bytes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every non-JSON tool result shipped its payload twice: content[0].text, and an
identical copy as structuredContent {"text": <entire payload>}. Measured on a
20k-node query_graph, the reply was 2.05x the payload it carried. Half of every
large answer was redundant bytes — half the 10 MiB transport budget, and double
the tokens billed to every LLM caller on every call.
Measured, same query, same fixture:
before 8,529,990 bytes content.text 4,154,932 structuredContent.text SAME
after 4,265,047 bytes content.text 4,154,932 structuredContent {}
Exactly 50.0% smaller with content.text byte-identical: the payload is fully
delivered, only the second copy is gone. The 10 MiB ceiling now also admits
roughly twice the rows before #1375's limit applies.
Nothing is lost. structuredContent exists to carry STRUCTURE, and a string
re-wrapped in a one-key object has none — a client reading
structuredContent.text learned exactly what content[0].text already told it. The
empty object still satisfies the declared outputSchema, which is
{"type":"object","additionalProperties":true} and never required a text field.
Two cases are deliberately NOT changed:
* JSON payloads. structuredContent stays the PARSED object. That is the
spec's structured+serialized pattern rather than waste, and it is what our
own hook-augment consumes (structuredContent.projects from list_projects).
* Errors. structuredContent.error is kept: bounded, small, and the only
machine-readable form of a failure a client gets.
Guarded at both levels, because the defect was invisible per-tool — each result
looked reasonable alone, and only measuring the wire showed half of it was
redundant:
* tests/test_mcp.c enumerates the TOOL TABLE itself, so a new tool is covered
the moment it is registered, with no test edit. A guard pinned to
query_graph would not have caught search_graph, and would not catch whatever
is added next. It fails if no tool produced a non-JSON payload, so it cannot
report a green it never earned — which it did on the first attempt, catching
that {} args make every tool error out and assert nothing.
* scripts/smoke-test.sh Phase 3z asserts the same property on the SHIPPED
binary. This is a wire-format contract: what a real client receives from the
real artifact is not something a from-source test can prove.
Both revert-checked: restoring the duplication reddens the unit guard at the
strcmp, the plain-text guard, the search_graph expectation, and the smoke phase
(which names search_graph and get_architecture).
tests/test_mcp.c expectations that pinned the old duplicating shape are updated
rather than relaxed — the format changed on purpose.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The comment above MIMALLOC_OVERRIDE_DEFINE claimed Unix "relies on
static-link-order override". That was never true. mimalloc emits strong
malloc/free definitions only when MI_MALLOC_OVERRIDE is set — the define its
source gates alloc-override.c on — and the build set it for MinGW only.
MI_OVERRIDE, which the build does set everywhere, is this project's own
prod/test marker that mimalloc never reads. With the override body compiled
out there were no strong symbols for link order to prefer, so ordinary malloc
went to libc.
Measured, not inferred. The shipped v0.9.1-rc.1 artifacts report 0/6
allocator-owned size classes on linux-arm64 glibc AND musl-static, and the
same A/B on ubuntu-arm64 here:
main warn mem.allocator.not_owned owned_classes=0/6
with this fix info mem.allocator.owned classes=all
Every purge/reclaim option cbm_mem_init sets was therefore inert on Linux,
applying only to the bound sqlite/tree-sitter populations — the same class of
defect as #581, where committed memory ratcheted for months because nothing
asserted the wiring on a real artifact.
macOS stays off deliberately and permanently: enabling the override there
compiles alloc-override.c's forwarding definitions, and under the two-level
namespace this binary's free becomes mi_free while system libraries keep
allocating from the system allocator, so the first pointer crossing that
boundary aborts with "mi_free: invalid pointer". ELF's flat namespace has no
such split, which is why Linux can have this and macOS cannot.
Three parts:
* Makefile.cbm switches MI_MALLOC_OVERRIDE and a new CBM_MEM_GLOBAL_OVERRIDE
together in one place, for MinGW and Linux. The latter tells our own sources
what to expect and goes into CFLAGS_PROD only, because MIMALLOC_CFLAGS_TEST
builds mimalloc with -DMI_OVERRIDE=0 and no override define: a test binary
has no global override BY CONSTRUCTION and must never be told to expect one.
Deriving that expectation from the platform instead would make every Linux
test run warn about a correctly configured build.
* mem.c splits the startup audit by what the build actually asked for. Where
no override was requested, ordinary malloc reaching libc is the design, so
it reports the measured ownership at INFO and names what IS bound. A warning
that fires on every run of a correct build is not a tripwire, it is noise —
it trains readers to ignore the one line that catches #581. The genuine
warning still fires wherever an override was requested and did not take.
* smoke-test.sh Phase 1b pins the SHIPPED artifact's wiring per platform and
fails in BOTH directions: Windows/Linux must own all classes, macOS must
not. This cannot live in a unit test — a from-source test build never has
the override — which is exactly why the defect survived so long.
Verified on ubuntu-arm64: prod owns all classes; the test build reports
bound-populations-only at INFO (0/6) and the mem suite stays green; smoke
Phase 1b passes, and fails with the expected message when the same tree is
built with the override forced off. On macOS: prod reports
bound_populations_only at INFO, smoke Phase 1b passes, mem suite 51/51.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the move started on Windows. `update` prints the install script's
command and exits 0 on every platform; the download/extract/chmod/exec sequence
is compiled out of release builds entirely, and the MCP background thread that
ran cbm_popen("curl ... api.github.com ...") on first tool call is gone.
The installers now place themselves beside the binary, which closes the last
gap: `update` referenced a raw.githubusercontent URL purely because an
install.sh-installed user had no local copy to point at. Two details make that
safe.
The source is the install script from the archive we JUST checksum-verified,
never "$0" -- which does not exist under `curl | bash`, and would pin the user
to the OLD installer forever. Since the script ships inside the verified
archive, it inherits that verification instead of needing its own.
And it is published by atomic rename, never written over the live path. Bash
reads a script incrementally by byte offset, so overwriting the file it is
executing continues reading NEW bytes at the OLD offset: silent, bizarre
corruption. cp to a temp name then mv -f swaps the directory entry while the
running shell keeps its original inode. Windows follows the same rule via
Copy-Item + Move-Item even though PowerShell parses up front.
Deliberately NOT done: spawning the new installer to delete and replace its
predecessor. Fetch remote code -> drop to disk -> spawn -> self-delete is a
textbook stager, and self-deletion is among the most heavily weighted
heuristics there is. It also gains nothing -- `"$DLBIN" install` already runs
the NEW release's install logic, because the binary owns the install.
Net: zero install or download URLs remain in the shipped binary, and the
allow-list entry is retired with it. Both scripts scan CLEAN (0/61) with the
self-placement code in them.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Windows already handed updating to install.ps1, because a running .exe cannot
replace its own image. The remaining platforms now do the same, for a different
and better reason.
An in-process updater is structurally a downloader: it fetches a remote archive,
verifies it, extracts it, marks the result executable and runs it. That composite
lived in every shipped artifact -- along with the release URLs it fetched -- to
serve a command most users run a handful of times. The install scripts already
do all of it, are idempotent (so re-running one IS the update), and run while cbm
is not running.
`update` now prints the exact command for the running platform and exits 0:
unix bash "<dir>/install.sh" (or the curl one-liner when not adjacent)
windows powershell -ExecutionPolicy Bypass -File "<dir>\install.ps1"
Both scripts ship inside their release archive, so the printed path resolves
next to the binary. Flags are still parsed and validated, so `update --dry-run
--ui` keeps rejecting typos rather than silently accepting them.
The updater itself is EXCLUDED from release builds rather than left to dead-code
elimination: build_update_url, download_verify_install, extract_and_install_binary,
checksum fetch, detect_os/detect_arch and the cli_download_* helpers now compile
only under CBM_CLI_ENABLE_TEST_API, which release builds do not define. The C
suite still covers the flow through the activation test seam. Verified against
the release binary: `releases/latest/download` is gone entirely.
Also removes the MCP background update check. It ran on the first tool call and
did:
cbm_popen("curl -sf --max-time 5 -H 'Accept: application/vnd.github+json' "
"'https://api.github.com/repos/.../releases/latest' 2>/dev/null")
An embedded shell command plus a startup network callout, in every session, to
tell users a newer version exists. It shelled out, depended on curl being
installed, and phoned home from every agent session. The install scripts report
versions; this did not need to be in the server.
Smoke follows the same collapse. Phases 6c and 14 asserted a Windows-specific
handoff and a POSIX in-process replacement; they now assert one platform-neutral
contract selected by UPDATE_SCRIPT, and 14a's byte-identity check compares the
driver against ITSELF before and after -- comparing against "$BINARY" passed on
Windows only because Windows has no signing step, while the POSIX fixture
ad-hoc re-signs its copy and so differed before `update` ever ran.
The Linux glibc guard is RELOCATED, not retired. The standard linux asset links
glibc 2.38+ and breaks Debian 11, RHEL 8 and Ubuntu 20.04, so the installer must
fetch the static "-portable" build. That constraint moved from the binary to
install.sh, and the smoke now guards it where the behaviour lives.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Release run 30385988475 lost three Windows smoke jobs to one phenomenon on two
different surfaces. A Windows file that was just written or just executed can
refuse deletion and rename for a moment while a scanner reads it or while the OS
finishes reaping a child. Both conditions clear on their own within moments.
It reads as nondeterminism because it is timing: windows-2025 passed both amd64
variants while windows-latest -- the SAME image, windows-2025-vs2026 -- failed
both, and windows-11-arm failed too.
PRODUCT: uninstall abandoned a live installation.
With one binary per platform, uninstall must remove the very image it is
executing, and on Windows a running image can only be renamed, never
overwritten. That rename got a single attempt, so a transient sharing violation
surfaced to the user as
error: failed to remove ...; completed configuration/index cleanup may remain
with the install left half-torn-down. install.ps1 already retries this exact
operation ten times for this exact reason; the C path now does the same, and
only for sharing/access/lock violations, so a file that is genuinely held still
fails closed instead of spinning.
Proven with a failure-injection seam rather than by racing a real scanner:
3 injected failures must still succeed, 64 must still fail AND leave the target
in place. Deterministic, so it does not need CI to learn whether it works.
HARNESS: the smoke killed itself silently.
Phases that install a binary then `rm -rf` the directory hit the same lock. Under
`set -euo pipefail` that returns non-zero and ends the run WITHOUT printing
anything -- the logs jump straight from "OK 13h" to job cleanup with no FAIL
line, which is what made this expensive to read.
None of those 16 removals is an assertion; they are fixture cleanup, and
_smoke.yml already states the principle ("a temp dir on an ephemeral runner must
never fail the job") that smoke-test.sh did not honour. They now go through
smoke_rmtree, which retries so the disk is actually reclaimed -- these fixtures
hold ~300 MB binaries and runners are disk-tight -- then warns and continues.
Verified: Windows VM full smoke ALL PASSED (previously died after 13h), macOS
full smoke ALL PASSED, activation_transaction + cli 259 passed on Windows, all
10 shell contracts pass, lint-ci clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Windows smoke guards that an invalid PATH seam must fail closed rather than
silently falling back to the live HKCU\Environment\Path. That guard went green
in a way that meant nothing: the seam still refused, and install still printed
"PATH configuration failed", but the process exited 0.
Windows release builds used to compile a separate managed-install path. With one
binary per platform they compile the shared cbm_cmd_install, which since 30b5f12
downgraded ANY failed dry-run plan check to CLI_OK. That commit was fixing a real
complaint -- the dry-run summary was being skipped entirely on Windows -- but it
bundled the exit status into the fix. Keeping the summary is right; reporting
success is not. --dry-run exists to answer "would this install work?", so a
caller that only sees the exit code could not tell a refused PATH probe from a
clean plan.
The summary and the triage note still print; only the status changes. A real
install already failed non-zero here, so the fail-closed behaviour itself was
never at risk -- just its observability.
The smoke also still encoded the launcher-era portable-vs-managed split, which
does not survive one-binary-per-platform:
* 6b expected uninstall to be REFUSED. That refusal existed because a portable
extracted bundle was a different artifact from the launcher-managed install
it would have torn down. The extracted binary now IS the installed one.
* 6c expected update to be REFUSED, contradicting the contract Phase 14a
asserts against a real update: Windows hands off to install.ps1, exits 0 and
prints the command. 6c now pins that same handoff.
* 14a compared binaries with cmp, which is absent from the Windows MSYS shell,
so it reported "different" for two copies of one file. Hashes instead, the
way Phase 12 already verifies the release archive.
* 14f seeded the MCP entry at the RETIRED binary. The old launcher-managed
update rewrote that entry in-process; the handoff does not, so uninstall
correctly declined to remove an entry owned by a different installation and
the phase demanded the one thing it must never do. It now seeds the
installed binary, which is what install.ps1 leaves a real user holding.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every ui-variant smoke job failed on every platform in dry run
30226483380 — and it turns out a ui-variant Phase 15 had never
legitimately passed anywhere: the old probe (a) never passed --ui=true,
but the HTTP UI is a persisted DEFAULT-OFF setting, so a fresh profile
(every runner, every smoke HOME) can never serve; (b) fed stdin an
empty file, but the UI does not pin the process — stdio EOF ends it
cleanly (rc=0) before any poll sees it serve; (c) gave a 10s window to
a server measured to bind ~6s after launch on a FAST host; (d) POSTed
an MCP initialize at /rpc, which speaks the UI's own narrow query
protocol, not MCP.
The rebuilt probe mirrors the drive-listing guard's proven invocation
(--ui=true --port=N equals-form, held-open stdin), waits 30s, and
verifies the API surface with the stateless GET /api/ui-config.
Validated red->green against the release-shaped artifact: the full
darwin ui artifact lane now passes 15a/15b legitimately.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dry run 30206811293: macos-15-intel release smoke failed B3 with a
visibly correct 'semantic: 50' table — BSD grep returns no-match for
the ENTIRE output when any line carries a NUL, control byte, or
invalid UTF-8 (verified: printf poisoned input -> grep -qE fails), so
one bad byte in one row silently unmatches every anchored check.
Close the class at the emission choke point: control bytes and
invalid UTF-8 force the quoted form; quoting escapes controls as
\u00XX and replaces invalid sequences with U+FFFD (RFC 3629
validation, NUL-safe). Tool output is valid UTF-8 by construction.
Unit test proven RED-on-revert (raw bytes emitted) and green with the
fix. The smoke's B3 failure path now also dumps od -c bytes so a
future red names the exact byte.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The staged venue-unification bundle: _smoke.yml onto the shared
wrappers with an extracted-artifact input (CBM_SMOKE_ARTIFACT_DIR),
soak-legs.sh as the one canonical soak entry (quick + query-leak legs
across all venues), protected per-user TEMP roots shared across
venues, clean-disk preflights before every VM/Docker run, win.sh and
run.sh routed through the canonical scripts, soak.yml retired, and
the venue-parity contract (whitelist walker: workflows may provision
or call canonical entries, nothing else).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The fuzz harness swallowed the target's exit status through '|| true', so
a SIGSEGV counted as a pass; the status now propagates (a planted crash
yields 139) and payloads derive from a logged, replayable seed, with a
missing python3 failing the gate instead of generating zero mutations.
The Windows guard runner classified unknown exit codes as skips; the
contract is now explicit (0 green, 1 red, 2 precondition-skip, anything
else a failure) and an all-skip run fails as verifying nothing.
The smoke suite gains crash-class detection on the phase-9b tolerance
paths and phase 11 kill handling (rc >= 128 or a missing jsonrpc banner
fails), a free-port pick plus readiness poll for the UI phase, and a
phase-15b failure that actually exits nonzero.
The soak reader resynchronizes on late JSON-RPC responses by draining to
the matching id, and the analysis fails when fewer than 60 percent of
snapshot attempts produced rows — a vacuous analysis previously passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase 9b's fixture installs spawn ephemeral daemons whose mapped
generation backings pin files in the fixture HOME; the cleanup rm then
races the daemon's asynchronous drain and loses nondeterministically on
Windows (Device or resource busy on a generations backing — the same
class the final-phase daemon retirement already fixed for the smoke
cache, two rounds green then one red on identical code). The retirement
is now a shared helper with a cheap not-running probe, called before
every fixture cleanup that follows an install and by the final phase; a
daemon that will not retire stays a hard failure.
The 9b fixtures also route LOCALAPPDATA under the fixture HOME: their
installs staged launcher pairs into the real runner profile, outside
the fixture's isolation and the harness's cleanup reach.
Verified: full VM smoke green end to end (all 9b subphases, phase 17
through the shared helper).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The 40 hidden Windows failures fixed in the previous commit reached CI
unseen because the local Windows leg never validly ran the C suite:
MSYS2's shared /tmp fails the secure-ancestry validation (refusals, not
signal) and the ssh driver piped everything through `tail -40`. This
closes those holes and hardens the gates that let an install failure
scroll past.
- vm-run-tests.sh (new, used by win.sh test / ubsan-test /
trap-ubsan-test): gives the suites CI's exact protected per-user temp
root (owner-stamped, protected current-SID DACL, mirroring the
workflow's pwsh step), shapes the runner's build directory like a real
user checkout (drive-root trees inherit an Authenticated-Users Modify
ACE that profile-rooted checkouts do not have, which the activation
source-directory policy correctly refuses; DACL re-rooted two-step —
directory protect then child /reset, since inheritance flags are
directory-only and a /T re-root leaves files with empty deny-all
DACLs), streams the FULL output, and refuses to report success unless
the runner printed its completion summary.
- smoke: Phase 8 no longer swallows the install exit code. On Windows it
now requires the managed install to SUCCEED and to publish an exact
two-link canonical launcher — a staging refusal used to scroll past as
tolerated noise while the downstream config assertions kept passing.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The account daemon legitimately outlives its last client for a moment
while it drains, holding its log and config DB open. POSIX rm does not
care, but on Windows an open file blocks deletion, so the caller's
cleanup of the smoke cache raced the daemon's asynchronous shutdown and
failed with "Device or resource busy" after every phase had passed —
first visible on windows-latest once the agent-config phases stopped
failing earlier, and reproduced identically on the local Windows VM.
The smoke now ends with an explicit phase: stop the daemon through its
own lifecycle command, wait (bounded) until status reports not-running,
and give Windows one beat for the final handle close. A daemon that
will not retire is now a smoke FAILURE in its own right instead of a
cleanup accident. Verified red-to-green on the VM and green on macOS.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fixes surfaced while bringing the shared-coordination-daemon branch
green across macOS, the Linux containers, and the real Windows-ARM64 VM,
plus a new stability guard that caught one of them.
- daemon(win): the CLI teardown transition latched a failed release as
permanent, but a Windows participant-state release must briefly
try-hold the shared startup/legacy gates and legitimately collides
with a concurrent one-shot's teardown; the release is retriable by
contract (it always retains the transition), so parallel one-shot
commands no longer report "CLI coordination cleanup failed" despite
succeeding. Found by the new stability guard's churn section.
- cli/main(win): install/update/uninstall and the Augment hook-script
removal derive the managed launcher's identity from its plain
drive-form path (cli_windows_plain_utf8), not the \\?\ extended-length
form, so agent-config ownership matches on Windows and uninstall no
longer leaves the MCP entry or owned hook scripts behind.
- security: the `daemon start --open` browser launch is now shell-free
(ShellExecuteW on Windows, cbm_exec_no_shell elsewhere) instead of
system(); the CORS origin check spells out its two literal loopback
URLs so the static URL audit sees a complete value. Both clear the
Layer-1 allow-list audit.
- coordination: version-cohort lock retries now sleep a per-process
jittered interval — fixed-period retries can phase-lock two
participants so one starves. The activation-quiesce test's
observation window now covers worst-case candidate staging (three
tamper-defense hashes over a ~1 GB sanitizer binary exceed the old
30 s budget on container I/O).
- smoke/soak: the agent-config smoke matches Windows config paths
through their escaped-backslash quoted form; the soak parser skips the
update-available banner before the JSON summary and gained a one-shot
CLI admission-churn phase inside the RSS/FD leak window. The build-dir
safety fixture creates a real symlink/junction (MSYS2 ln -s otherwise
deep-copies) so its traversal-refusal contract is genuinely exercised,
and the symlinked-agent-roots test demotes its planted links to an
unprivileged owner when run as root so the refusal is really tested.
- tests(win): a new daemon-stability guard exercises the parameter
surface, hook fail-open with its rate-limited notice, start-twice and
occupied-port handling, busy-stop refusal against a live MCP session,
kill -9 crash recovery, and sequential/parallel one-shot churn.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.
Daemon lifecycle:
- daemon start/stop/status subcommands. `daemon start` launches a
PERMANENT daemon (spawn shape is byte-exact argv; survives idle
periods and session ends) and reports an already-active daemon
instead of failing. Permanence is honored at every stop latch:
last-committed-client disconnect, host initial-client window,
coordinator release, and application final-session close — a
permanent daemon also keeps admitting new sessions after its last
one closes.
- daemon stop refuses while sessions are active and lists the blocking
peers (pid/role) that must finish first; an idle daemon drains
through the activation-shutdown machinery with the ACK ordered after
connection interrupts. A second stop is idempotent. The wire ops are
no-cohort first-frame requests with peer fingerprint authentication,
so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
keep their local supervised path). A cold CLI run that had to spawn a
temporary daemon prints a hint that `daemon start` removes the
per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
emits a visible, rate-limited notice (Claude-dialect systemMessage
plus stderr for other dialects) and always exits 0 — augmentation is
never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
older-build permanent daemon (strict semantic-version triples only;
dev builds never auto-drain) and the build-conflict message names
`cbm daemon stop` as the manual escape hatch.
Windows IPC/runtime (real-VM verified):
- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
path used to destroy its listening pipe instance on every 20 ms poll
timeout; a client attaching in the teardown window was severed or left
on an orphaned pipe object whose HELLO no server handle could ever
read, absorbing the connect until the client's own timeout expired.
The pending connect now survives poll timeouts and nothing is
destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
server handle can discard a just-sent response before the peer reads
it (POSIX stream sockets never lose buffered data on close). A bounded
cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
immediate on interrupted connections) now precedes close in
runtime_worker_finish and runtime_reject_inline, so hello-conflict,
capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
transport half-close, so close_begin now announces departure with an
explicit frame (ordered after APPLICATION_CANCEL, before the local
interrupt); the server releases the client's admission on receipt
instead of waiting for the handle to close. Admission-drop timing is
now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
request publication under the send lock, best-effort sends the active
token's APPLICATION_CANCEL frame, then interrupts local I/O; the
server cancels MCP/subprocess work promptly. Contract tests accept
both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
the requester's license to act on "snapshotted and draining", so every
connection interrupt is now initiated before the ACK is sent; a
session could previously get one more request serviced after the
requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
timer granularity) burned ~1.6 s against permanently obstructed paths,
stalling hello rejections past the client's timeout. The retry budget
is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
nonblocking send() on Windows is absorbed wholesale into AFD kernel
buffering regardless of SO_SNDBUF, so send deadlines and interrupts
could never engage against a slow peer (and the full payload was
pinned in nonpaged pool). Slicing restores a deterministic
backpressure point; a test hook pins SO_SNDBUF for the deadline and
interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
deadline, explicit connection states, refusal to free a server while
a listener-owned connection is active.
Windows long-path support:
- Central path-aware wide conversion (canonicalize via GetFullPathNameW
and prepend the extended-length prefix for absolute paths >=240) at
the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
store opens, and the daemon build-fingerprint/log paths. Deep managed
installs (a 64-hex generation directory routinely exceeds MAX_PATH)
now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
ancestry validators now operate in the extended-length namespace;
the launcher path is canonicalized (and prefixed when deep) once at
entry so every downstream exact-string comparison stays
form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
GetFileAttributesW) so non-ASCII install paths survive argv[0]
resolution.
Windows launcher install/uninstall transaction:
- FileRenameInfoEx names are NUL-terminated in an over-allocated
buffer. FileNameLength governs per the contract, but filter drivers
read FileName as NUL-terminated and appended adjacent heap bytes to
created names — a flaky, garbage-suffixed rename target. Both the CLI
and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
.cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
chars so the bare rename target stays under the FileRenameInfoEx
NT-conversion ceiling at guard depths; 64 bits still uniquely
identify the generation.
- When the running launcher's mapped generation backings pin .cbm
against rename, the backings are relocated to activation-<pid>-N
.retired tombstones beside the install (a mapped image may be renamed,
never deleted; the launcher's liveness-guarded sweep reclaims stale
tombstones). Every relocation is recorded, and a FAILED uninstall
reverses the moves after restoring .cbm — via MoveFileExW with
extended-length paths on both arguments, since the deep generation
target is beyond the handle-based rename's bare-path reach — so a
restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
out so the tree is shallow enough for the detached cleanup's rd, and
the cleanup's working directory strips the extended-length prefix
(CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
(CopyFileW destinations, CREATE_NEW tombstones, probe directories)
are explicitly owner-stamped so the exact-owner validators hold on
runner images; guard fixtures stamp hand-built trees the same way.
Diagnostics, tests and infra:
- diagnostics: discovery is now an always-delivered JSON control record
(new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
paths containing spaces; placement honors $TMPDIR with /tmp fallback
via a diagnostics-local helper; the soak parser reads the JSON record;
documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
log levels, TMPDIR-with-spaces, and native Windows output-contract
assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
parents with production-shaped ancestry (LocalAppData on Windows, via
th_secure_runtime_parent_new) — the runtime ancestry validation
correctly refuses temp roots whose ancestors grant mutation rights to
Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
drive the documented startup-owner publication flow before reading
generation-bound endpoint addresses. This turns the 26 Windows
failures previously visible in CI's full-test job green without
weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
contract including failed-uninstall restore and immediate reinstall
after uninstall; new daemon lifecycle and reworked hook-augment
guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
(it was set by CI but never read, so perf suites ran everywhere);
four throughput/bench suites are classified as perf, the heavy
store_arch suite moved to the slow-timeout tier, and two
wall-clock-sensitive assertions were rewritten as invariant checks
with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
contract, soak daemon-recovery contract, path-safety helper, the
Windows VM worktree-sync contract wired into scripts/test.sh, and
vm/win.sh guards building its clean embedded-UI product in an
isolated BUILD_DIR so it cannot clobber the incremental test build.
provision-windows.sh now installs Node.js for the guards UI build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Windows dry-run failure prints an error (which passes the
install/skill/mcp/agent check) but no "dry-run" indicator, and the
smoke never echoed the captured output on THIS branch, so the actual
error stayed hidden. Echo it. (Root: Windows install routes through
cli_windows_managed_install, which errors on launcher/payload source
verification before emitting a dry-run plan - the next run will name
which check fails.)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
pr-smoke's mcp_run discarded the frontend's stderr (2>/dev/null), so a
first-session daemon start/connect failure - which with the daemon
architecture surfaces ONLY on stderr, stdout being reserved for
JSON-RPC - showed up as a bare "no initialize response (id:1)" with
nothing to diagnose (exactly what an isolated ubuntu pr-smoke failure
just produced). Keep the frontend stderr and, on an id:1/id:2 miss,
dump it alongside the durable cbm-daemon.log and daemon-conflicts.ndjson
tails from the cache root. No behaviour change on success.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The PR CI caught two consumers the format migration missed — both parsed
tool output by its old shape and failed silently or with a wrong count:
- hook-augment built its PreToolUse additionalContext by reading the
legacy 'results' object-array from search_graph format:json; after the
json-tree reshape that key no longer exists, so the hook emitted nothing
(caught end-to-end by tests/windows/test_hook_augment.py). The parser
now walks groups -> qn_prefix/file + column-ordered rows. A new local
test feeds the parser the LIVE server envelope, so any future drift
between response shape and hook parser fails in the normal suite on
every platform — not only in the Windows CI guard. Reproduced RED
(ctx NULL, the exact CI failure) before the fix, GREEN after.
- smoke-test.sh carried six old-shape parsers (callers[N], rows[N]{,
clusters[N], semantic[N]{, and the assert_toon_table helper); the tree
headers never matched, so trace verification read 0 callers. All six
updated to the tree contract; the full smoke script passes end-to-end
locally.
cli suite 222 passed; lint clean; scripted smoke ALL PASSED.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
