trending/codebase-memory-mcp
1
0
Fork 0
mirror of https://github.com/DeusData/codebase-memory-mcp.git synced 2026-11-09 09:11:00 +08:00
Code Issues Packages Projects Releases Wiki Activity
2,366 Commits 627 Branches 56 Tags
506151f6c19cf5b80e9ea91ccecc674bb5d253ee
Commit Graph
29 Commits
Author SHA1 Message Date
Martin Vogel 39e02544cc fix(smoke): UI presence is the caller's claim, not every lane's assumption 
Making REQUIRE_UI unconditional broke all three pr-smoke legs:

  FAIL 15a: SMOKE_REQUIRE_UI=1 but this binary serves no embedded UI assets

The PR lane runs scripts/build.sh WITHOUT --with-ui on purpose, so an npm
frontend build does not land on every product PR. Demanding embedded assets
there asserts a property that lane deliberately does not produce.

The guard still exists where it means something: scripts/ci/smoke-artifact.sh
builds --with-ui, packages the real archive and smokes the EXTRACTED result, so
it exports SMOKE_REQUIRE_UI=1 and a frontend-less binary fails there. That is
the lane whose whole job is release fidelity.

Deliberately NOT fixed by adding --with-ui to pr.yml: that would put `npm ci &&
npm run build` on three runners for every product PR to re-prove something the
dry-run and release lanes already gate. PR CI cost is not free and this branch
should not quietly raise it.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-08-09 18:17:42 +02:00
Martin Vogel a4336dc40a feat(release): ship one archive set and tolerate a single Microsoft !ml verdict 
Completes the collapse to a single shipped composition and replaces the
zero-tolerance VirusTotal gate with a narrow, disclosed policy.

Packaging and installers
  - package-release.sh loses --variant; archives are codebase-memory-mcp-<os>-<arch>
    with exactly four members. install.sh/install.ps1 lose --ui/--standard.
  - The extractor drops CBMUIPK pack parsing and --archive-scope; its scan-set and
    association manifests (which the gate depends on) are unchanged otherwise.
  - npm/PyPI/Go wrappers: the runtime "set" is one file again. The Windows lock
    and race fixes from #1495/#1496 are kept; only multi-file set membership goes.
    This also fixes `pip install` on Windows, which rejected the fifth archive
    member against a hardcoded four-name allowlist.
  - The wrappers' post-download probe moves from --verify-runtime-assets (removed)
    to --version, which proves the same thing: the binary executes.

VirusTotal gate
  - Exactly ONE detection is tolerated, and only when the engine is Microsoft AND
    the label ends in `!ml`. Two or more engines, any non-`!ml` label, any other
    vendor, any suspicious verdict and every infrastructure error still block.
  - A tolerated object prints TOLERATED:, never OK:, and its counts are recorded
    in vt-results.tsv exactly as a blocked one would be.
  - append-vt-notes.sh mirrors the policy. It previously hard-failed on any
    malicious count, so loosening only the gate would have passed the scan and
    then died at note publication. The notes now DISCLOSE a tolerated detection
    and link to SECURITY.md rather than claiming "0 malicious" for everything.

Rationale for the tolerance is in the gate itself: the verdict is not a property
of our bytes. It inverts across architectures and link modes, moves between
sibling artifacts of one build, and lands in different variant buckets for the
same source. The same `!ml` family hits llama.cpp, GitHub's own `gh`, Microsoft's
own Go toolchain and Anthropic's Claude installer.

The zero-tolerance contract becomes test_vt_gate_policy_contract.sh, asserting
the full matrix: 1x Microsoft !ml passes and reports TOLERATED; a Microsoft
signature label, a non-Microsoft engine, two engines, a suspicious verdict and
every malformed-response case still block. Its tripwire is narrowed to the
reverted endpoint-verification mechanism rather than the words "false positive",
so it no longer fires on a deliberate in-gate policy branch.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-08-09 16:16:08 +02:00
Martin Vogel d58afe562d revert(release): re-embed runtime assets into the single shipped binary 
Externalizing the integration templates (#1492/#1493) and the UI bundle
(#1501/#1503) was done to reduce the Microsoft `Wacatac.B!ml` surface. It did
not work: across dry runs the flagged artifact count stayed at ~3 and the
detections merely moved between artifacts.

Dissection of run 31286803592 shows there is no structural cause to fix. The
verdicts split across every axis at once — linux-amd64 (dynamic) flagged while
linux-amd64-portable (static) is clean, but linux-arm64 (dynamic) clean while
linux-arm64-portable (static) is flagged. The two macOS binaries have identical
segment structure and split clean/flagged. Siblings from one build landed in
different variant buckets (.B vs .C). Entropy is low everywhere
(code_vectors.bin 4.166, grammar tables 3.464 bits/byte, against 7.5-8.0 for
packed payloads), so the packed-payload hypothesis is excluded too.

So the complexity bought nothing, and installation goes back to being
self-contained: one binary that carries its own UI and agent integration
templates, with no adjacent data file that has to resolve before `install`
works. Only the UI-capable composition ships from now on, under the historical
unsuffixed archive name.

Removed: src/ui/asset_pack.{c,h}, asset_pack_stub.c, asset_manifest_stub.c,
scripts/pack-ui-assets.mjs, src/cli/integration_assets.{c,h},
assets/cbm-integrations.json, scripts/gen-integrations-hash.sh, the
--verify-runtime-assets probe (nothing adjacent left to verify), and the
composition gates A6/A7 whose property is now deliberately inverted.

Restored: scripts/embed-frontend.sh, src/ui/embedded_{assets.h,stub.c}, the
compiled-in hook/adapter template bodies, and the embed/EMBED_OBJS build path.

Kept from the reverted commits, re-applied by hand where a wholesale file
restore would have dropped them:
  - cbm_module_path_utf8() in both self-path sites. GetModuleFileNameA renders
    through the ANSI code page and mangles non-ASCII install paths.
  - the /__cbm/ui-readiness HMAC proof, secure_random and cbm_hmac_sha256, so
    `daemon start --open` still waits for a genuine CBM listener.
  - X-Content-Type-Options: nosniff on served assets.
  - the MinGW noexecstack gate, -lbcrypt, and the cppcheck/zip CI fixes.

Archives are now codebase-memory-mcp-<os>-<arch>[-portable] with exactly four
members (binary, LICENSE, installer, THIRD_PARTY_NOTICES.md). That restores the
names every static package manifest already points at — aur, chocolatey,
homebrew, scoop, winget and glama were all broken by the -ui- rename.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-08-09 13:06:42 +02:00
Martin Vogel 8018561cfe fix(release): externalize runtime assets and harden VT verification 
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-08-08 17:35:05 +02:00
Martin Vogel 1c16786b4b fix(smoke): stage cbm-integrations.json in the Windows smoke path too 
The Windows pr-smoke uses test-infrastructure/vm/vm-smoke.sh, the Windows analog
of smoke-local.sh, which had the same three staging gaps the Unix fix closed:
its required-sidecar check, fixture copy, and zip member list all omitted
cbm-integrations.json, so the served archive lacked the asset and install/
uninstall failed closed with "integration assets missing".

Plus one Windows-specific spot: vm-smoke.sh copies the smoke binary into
$SMOKE_DIR and runs smoke-test.sh against it there, but the asset only reached
$FIXTURE_DIR — so Phase 14 (which hand-stages the binary and drives an
uninstall, resolving the asset next to the binary) missed it. Now staged next to
the binary in $SMOKE_DIR as well. build.sh already places it next to
build/c/*.exe, so the non-artifact source is $(dirname BINARY_SRC); artifact
mode takes the release copy.

Member set + order in the zip mirror scripts/package-release.sh (Windows).
vm-smoke.sh lives under test-infrastructure/, outside the license gate's scan
set, so the cbm-integrations.json/LICENSE adjacency raises no ScanCode finding
there. Syntax-checked; vm-worktree-manifest, smoke-fixture and windows-bundle
contracts pass. The Unix smoke already runs clean through all 16 phases; this
mirrors that staging onto the one path I cannot exercise locally.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-08-08 05:31:53 +02:00
Martin Vogel ba5724319c fix(ci): accept the documented test-amd64 leg spelling 
run.sh --help advertises 'amd64|test-amd64|tsan-amd64', but only 'amd64'
was dispatched, so the documented spelling failed with
"unknown leg 'test-amd64'". Accept both.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-08-06 02:54:23 +02:00
Martin Vogel 3e06362160 fix(ci): isolated VM checkout inherits the base's real origin URL 
`git clone --local` points origin at the base PATH, so a later
`git fetch origin <branch>` inside the isolated tree would resolve against
whatever the shared base happens to hold - i.e. another session's state, the
exact coupling the per-run checkout exists to remove. Inherit the base's
origin URL at clone time so the isolated tree fetches from GitHub itself.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-08-06 02:12:01 +02:00
Martin Vogel cad2b02cee fix(ci): give the Windows VM leg a per-run checkout and a HEAD assertion 
The VM's last piece of shared state was the working tree itself. /c/cbm is
ONE checkout and update/sync REPLACE it (git reset --hard + clean -fdx), so a
second session syncing its branch swaps the code under a running leg.

Observed 2026-08-06: a gate run on main compiled tests/test_workspace.c and
src/foundation/workspace.c - files that do not exist in main - and died on
'sys/wait.h' not found. The VM was sitting at another session's commit
7fa5b07. That is worse than a red: a confident verdict about the wrong tree.

- CBM_CI_RUN_ID gives the run its own checkout, /c/cbm-run-<id>, created with
  a LOCAL clone so .git/objects are hardlinked (seconds, near-zero disk).
  Unset keeps the historical single-tree behaviour byte-for-byte.
- CBM_VM_EXPECT_HEAD=<sha> asserts the tree is the intended one BEFORE and
  AFTER the suites, because the swap can land mid-flight.
- win.sh drop-run-checkout removes THIS run's tree (refuses in shared mode -
  a run may only delete what it created; another session's tree is never ours
  to remove).

Verified: bash -n and shellcheck -S error clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-08-06 02:09:21 +02:00
Martin Vogel 5444da4166 fix(ci): isolate local-CI runs per run id, and tidy what each run creates 
Concurrent local-CI legs clobbered each other. Container names were already
unique per `compose run --rm`, but the mutable state was not:

- Docker: `cbm-build` is ONE named volume shared by every service and every
  concurrent run, so two legs wrote the same /src/build - objects and
  test-logs included. The parallel scheduler then dies reading a suite log
  another run replaced ('cannot read suite log .../extraction.log').
- Windows VM: one checkout (/c/cbm) plus a FIXED log path (/tmp/win-test.log)
  and a shared build dir; -PruneStale could also delete a live run's temp root.

Each run now carries a unique id (pid+epoch, overridable via CBM_CI_RUN_ID):

- docker-compose: the build volume takes its name from CBM_CI_BUILD_VOLUME,
  defaulting to today's `cbm-build` so the single-run path is unchanged.
- run.sh: derives the id, points the build volume at it, and removes that
  volume on success. CBM_CI_SHARED_BUILD=1 opts back into the shared volume.
- vm-run-tests.sh: per-run log; and when a CALLER sets CBM_CI_RUN_ID (i.e.
  declares concurrency) a per-run BUILD_DIR too, removed on success.

ccache and the fixture cache stay SHARED deliberately - ccache is
concurrency-safe and content-verified, and sharing them is what keeps an
isolated run fast instead of cold.

Cleanup follows one rule: a run tidies what it created, EXCEPT when it failed -
then the artifacts are the post-mortem and the run prints how to inspect and
drop them. CBM_CI_KEEP=1 keeps them regardless.

Verified: bash -n on both scripts; `docker compose config` valid, and with
CBM_CI_BUILD_VOLUME set the volume resolves to the per-run name.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-08-06 01:45:56 +02:00
Martin Vogel a54ea95719 fix(windows): ship one binary — remove the launcher stub flagged as a dropper 
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.

That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.

So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF.  now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.

Windows now matches Linux and macOS: ONE binary per platform.

  * packaging, install.ps1, npm and PyPI wrappers all carry a single binary
  * the launcher/payload ABI contract and ~2500 lines of stub state machinery
    are deleted
  * every daemon start, CLI call and hook fire loses a process spawn, a named
    pipe handshake and an stdio relay
  * test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
    now asserts no shipped surface can reintroduce a launcher/payload pair,
    and that install.ps1 retires the running binary before publishing

Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-28 17:03:01 +02:00
Martin Vogel 0c0111eb33 fix(smoke): header-only fixture-server readiness polls 
Hosted Windows runners reset the readiness poll's full-archive GET
mid-body (WinError 10054 on the server, ~99 aborted transfers per
job) while the identical stack passes on the VM; every windows smoke
job died at 'fixture server did not serve' with a healthy server. A
HEAD proves the artifact routes without transferring the body per
poll; the download phases still own full-body transport and report
phase-precise if that is ever broken.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-27 04:45:13 +02:00
Martin Vogel c400dc1665 ci: hosted runners cannot enable Defender RTP — revert runner-side gating 
Dry run 30225685743 failed at the ensure-defender step on every
Windows job: the hosted images policy-lock real-time protection OFF
(WinDefend starts, Set-MpPreference accepts, RTP stays off —
untamperable from the job).

Revert to the pre-gate runner behavior: no ensure-defender step in
runner jobs, the release Defender scan is on-demand and fail-soft
again. The local VM preflight KEEPS the fail-closed Defender-ON gate
(same canonical script) — AV-interaction coverage is a deliberate
local superset where the platform allows it.

The venue-parity contract now enforces the INVERSE for runners
(ensure-defender must not gate hosted jobs) and still requires the VM
preflight gate; VM README documents the verified posture delta.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-27 01:56:33 +02:00
Martin Vogel 0a40fe8bea feat(ci): local-remote parity lanes — artifact-flow smoke, glibc floor, Defender-ON everywhere 
Closes the closable local-vs-remote venue gaps so that, outside arch
legs / YAML glue / release plumbing, every local red is a remote red
and vice versa on shared legs:

- scripts/package-release.sh: THE canonical archive step (names +
  five-file Windows bundle layout defined once); _build.yml's eight
  inline archive blocks become calls to it.
- scripts/ci/smoke-artifact.sh: the artifact-flow smoke lane — build,
  package, extract, then the canonical wrapper in artifact mode; wired
  as run.sh smoke-artifact (compose service), win.sh smoke-artifact,
  and directly runnable on macOS. Archive-layout bugs now surface
  locally instead of in a release dry run.
- glibc-floor leg (Dockerfile.glibc22 + compose + run.sh): portable
  binary smokes on ubuntu-22.04/glibc 2.35; the dynamic binary must
  refuse there (2.38+ floor by design).
- Defender-ON parity (user directive): scripts/ci/ensure-defender.ps1
  enables + VERIFIES real-time protection, fail-closed; runs in every
  Windows CI job (_test x2, _soak x3, _smoke, pr.yml) AND in the VM
  preflight; _smoke.yml's scan engine-failure soft-skip becomes a red
  gate. Expected cost: slower Windows jobs (AV scanning during
  install/build/test I/O); the next dry run proves the runner side.
- Contracts: launcher-bundle five-file check retargeted onto
  package-release.sh + per-archive canonical-call association;
  venue-parity contract requires the new lanes, counts one
  ensure-defender step per Windows job, adds --help probes for the new
  entries. Extended contract fails on the pre-change tree (verified:
  20 violations on HEAD).
- VM README: ephemerality/Defender posture documented — utmctl has no
  snapshot verb, so per-run revert stays a manual qcow2 option; the
  sweep preflight remains the standing mechanism.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-26 22:37:48 +02:00
Martin Vogel 3fc93d12da ci: unify smoke/soak/test venues onto canonical leg entries 
The staged venue-unification bundle: _smoke.yml onto the shared
wrappers with an extracted-artifact input (CBM_SMOKE_ARTIFACT_DIR),
soak-legs.sh as the one canonical soak entry (quick + query-leak legs
across all venues), protected per-user TEMP roots shared across
venues, clean-disk preflights before every VM/Docker run, win.sh and
run.sh routed through the canonical scripts, soak.yml retired, and
the venue-parity contract (whitelist walker: workflows may provision
or call canonical entries, nothing else).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-26 22:23:05 +02:00
Martin Vogel 636dd3f92c fix(test-infra): repair the native Windows soak transport and release checks 
The native Windows soak could never start its server, so that gate had
never actually measured anything. The root cause was the harness, not the
daemon: an MSYS filesystem FIFO feeding a native Windows process delivers
empty stdin (reproduced 5/5), while a Bash anonymous coprocess carries the
full JSON-RPC initialize exchange to the same protected payload, and the
failing daemon log showed an orderly client disconnect rather than a
daemon-side eviction. The soak now uses a named coprocess on native
Windows only, duplicating its endpoints to stable fd3/fd4 and closing the
originals so closing fd3 still delivers EOF; POSIX keeps the FIFO path.
The coproc syntax sits inside eval because macOS system Bash 3.2 must
still parse this file even though only MSYS2 Bash 5 executes that branch.

With transport repaired the next exact failure surfaced: native Windows
Python cannot open an MSYS /c/... diagnostics path. Both consumers now
pipe the file through stdin, the pattern already established elsewhere in
the repo, and the soak recovery contract forbids reintroducing a direct
native-Python open.

Also on the release path:

- The smoke fixture server no longer fsyncs before atomically publishing
  its port. This is ephemeral readiness signalling, not crash-durable
  state, and the macOS Intel runner failed inside that durability sync.
  When the contract does fail it now reports the observable state --
  waited, exit status, port file, staged temp files, interpreter, startup
  log -- because the previous verdict named nothing on the one runner we
  cannot reproduce locally.
- The POSIX publication test hook no longer compiles into Windows builds,
  fixing an exact -Werror unused-variable failure. The setter keeps a
  parameter-consuming Windows stub because it is public API.
- The Wine leg assembles the real release layout (payload plus canonical
  launcher) and version-checks both, running the launcher through cmd so
  it has a Windows-visible parent. The unsupported Wine soak is removed:
  Wine stays a fast compile/package/version check, and native Windows is
  authoritative for daemon, locking, ACL and process-lifetime semantics.
- run.sh soak-windows routes to the native Windows VM, which validates the
  payload, builds the protected per-user temp root, stamps ACLs, and
  refuses success without a completion summary.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-25 00:37:05 +02:00
Martin Vogel 62821375c4 fix: harden daemon release paths and verification 
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-24 16:08:42 +02:00
Martin Vogel 568e613141 test-infra: overlap the local ladder, open the serial tail, keep container I/O off virtiofs 
Local runs left most of the machine idle: the serial tail ran sixteen
suites one at a time on otherwise-idle cores, the three platform legs
were launched by hand (usually sequentially), and every container-leg
object write crossed the virtiofs bind mount.

- run-tests-parallel.sh tail scheduling in two phases: the FLEX suites
  (timing-shaped but free of the shared per-account daemon runtime
  namespace) run CBM_TAIL_JOBS-wide (default 2), then the EXCL group —
  daemon-family plus the suites that drive daemon one-shots or
  supervisor rendezvous — runs strictly sequentially on a machine
  exactly as quiet as the old fully-serial tail gave it. The wave was
  already fed longest-first by the shard dealing order, so the drain-out
  no longer ends on a heavy straggler.
- ladder.sh: one maintained entry point for the full local push gate
  with the legs overlapped — lint, the Linux container suite, and the
  Windows VM suite in the background, the macOS suite in the
  foreground, one verdict per leg, logs kept per leg. A missing
  prerequisite fails its leg loudly instead of silently skipping.
- win.sh test-par now runs through vm-run-tests.sh (--par mode): the
  full parallel harness under the CI-shaped protected temp root with
  complete output. It previously ran under the MSYS-shared /tmp and
  piped through `tail -25` — the same truncated-blindness class that
  hid 40 Windows failures from the `test` command.
- docker-compose: build artifacts and the incremental fixture cache
  move to named volumes on the container VM's native filesystem. Object
  writes over the virtiofs bind mount are the container legs' largest
  avoidable I/O cost, and the fixture cache now survives across
  container runs.
- test_mem(win): the first valid full-parallel VM run proved working-set
  trimming beats the re-touch mitigation (19 MB resident of a 256 MB
  double-touch at 18 parallel suites). The RSS probe now VirtualLocks a
  64 MB span — locked pages are exempt from trimming, making the
  measurement pressure-immune — with bounded touch-and-sample retries
  when the lock is unavailable. Red-to-green under the same 18-job load.
- cli: the portable install's staging error now appends the activation
  refusal note (predicate, SID, object) like the managed path already
  does — a bare "activation transaction I/O failed" on a CI-only
  failure is undiagnosable without it.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-23 01:33:58 +02:00
Martin Vogel 664058e86b test-infra(win): CI-shaped VM harness, managed-install smoke gate, calibrated suite tier 
The 40 hidden Windows failures fixed in the previous commit reached CI
unseen because the local Windows leg never validly ran the C suite:
MSYS2's shared /tmp fails the secure-ancestry validation (refusals, not
signal) and the ssh driver piped everything through `tail -40`. This
closes those holes and hardens the gates that let an install failure
scroll past.

- vm-run-tests.sh (new, used by win.sh test / ubsan-test /
  trap-ubsan-test): gives the suites CI's exact protected per-user temp
  root (owner-stamped, protected current-SID DACL, mirroring the
  workflow's pwsh step), shapes the runner's build directory like a real
  user checkout (drive-root trees inherit an Authenticated-Users Modify
  ACE that profile-rooted checkouts do not have, which the activation
  source-directory policy correctly refuses; DACL re-rooted two-step —
  directory protect then child /reset, since inheritance flags are
  directory-only and a /T re-root leaves files with empty deny-all
  DACLs), streams the FULL output, and refuses to report success unless
  the runner printed its completion summary.
- smoke: Phase 8 no longer swallows the install exit code. On Windows it
  now requires the managed install to SUCCEED and to publish an exact
  two-link canonical launcher — a staging refusal used to scroll past as
  tolerated noise while the downstream config assertions kept passing.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-23 00:53:51 +02:00
Martin Vogel ece838c525 test: widen ThreadSanitizer coverage and add native ARM64 Windows UBSan 
Close the cross-platform sanitizer gaps that were leaving real
concurrency and undefined-behavior bugs uncaught, and fix a data race
the first widened run surfaced.

- tsan: the data-race gate ran three suites (mem, slab_alloc, parallel)
  over no real threaded production code. It now covers every threaded
  surface that runs clean and stable under TSan: the parallel-extraction
  worker pool (parallel, worker_pool, pipeline), the filesystem watcher,
  the embedded HTTP server (httpd), diagnostics sampling, the MCP server
  and mutation guard, subprocess supervision, and the runnable
  daemon-coordination paths (daemon, daemon_application). daemon_runtime
  (deadlocks under TSan+fork), daemon_ipc and daemon_frontend
  (test-harness synchronization, not production) are excluded with the
  reasons recorded in the Makefile.
- tsan: fixed a genuine data race the widened gate immediately found —
  cbm_lsp_max_walk_depth's lazy cache was read and written by parallel
  LSP-extraction workers without synchronization. A data race is
  undefined behavior even when every worker computes the same value, so
  the cache slot is now a relaxed atomic: a plain load on the hot path,
  and a first-touch double-compute simply stores the same value.
- tsan(ci + local): the test-tsan job now runs on Linux amd64, Linux
  arm64, AND native ARM64 macOS (the threading code is shared, so a race
  is usually caught on all three, but scheduler differences let each
  surface one the others miss). The local ladder gained `run.sh tsan`
  and `tsan-amd64` plus the matching compose services. TSan's shadow
  memory aborts under modern high-entropy ASLR, so the containers run
  under `setarch -R` with an unconfined seccomp profile (the personality
  syscall is otherwise blocked) and the CI Linux legs lower
  vm.mmap_rnd_bits first; amd64 TSan cannot run under x86_64-on-ARM
  translation and is a real-hardware/CI gate only (documented in
  run.sh).
- ubsan(win/arm64): native ARM64 Windows had no sanitizer at all —
  AddressSanitizer ships no aarch64-w64-windows-gnu runtime. UBSan in
  trap mode (-fsanitize-trap=undefined) needs no runtime library, so it
  instruments natively and turns undefined behavior into an
  illegal-instruction trap; -fstack-protector-strong adds stack-smash
  coverage the heap tools miss. The GitHub windows-11-arm leg switches
  from unsanitized to this, and vm/win.sh gains trap-ubsan-build /
  trap-ubsan-test for local iteration (reproduce under the emulated
  x86_64 UBSan to see which check fired). The whole codebase builds and
  runs clean under it.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-22 17:48:54 +02:00
Martin Vogel 30550fbf75 test-infra(vm): restore the executable bit on win.sh lost in a merge resolution 
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-21 15:32:42 +02:00
Martin Vogel 0baddf7887 daemon: permanent lifecycle, daemon-backed CLI/hooks, real-Windows hardening, long-path launcher transactions 
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.

Daemon lifecycle:

- daemon start/stop/status subcommands. `daemon start` launches a
  PERMANENT daemon (spawn shape is byte-exact argv; survives idle
  periods and session ends) and reports an already-active daemon
  instead of failing. Permanence is honored at every stop latch:
  last-committed-client disconnect, host initial-client window,
  coordinator release, and application final-session close — a
  permanent daemon also keeps admitting new sessions after its last
  one closes.
- daemon stop refuses while sessions are active and lists the blocking
  peers (pid/role) that must finish first; an idle daemon drains
  through the activation-shutdown machinery with the ACK ordered after
  connection interrupts. A second stop is idempotent. The wire ops are
  no-cohort first-frame requests with peer fingerprint authentication,
  so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
  keep their local supervised path). A cold CLI run that had to spawn a
  temporary daemon prints a hint that `daemon start` removes the
  per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
  emits a visible, rate-limited notice (Claude-dialect systemMessage
  plus stderr for other dialects) and always exits 0 — augmentation is
  never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
  older-build permanent daemon (strict semantic-version triples only;
  dev builds never auto-drain) and the build-conflict message names
  `cbm daemon stop` as the manual escape hatch.

Windows IPC/runtime (real-VM verified):

- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
  path used to destroy its listening pipe instance on every 20 ms poll
  timeout; a client attaching in the teardown window was severed or left
  on an orphaned pipe object whose HELLO no server handle could ever
  read, absorbing the connect until the client's own timeout expired.
  The pending connect now survives poll timeouts and nothing is
  destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
  server handle can discard a just-sent response before the peer reads
  it (POSIX stream sockets never lose buffered data on close). A bounded
  cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
  immediate on interrupted connections) now precedes close in
  runtime_worker_finish and runtime_reject_inline, so hello-conflict,
  capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
  transport half-close, so close_begin now announces departure with an
  explicit frame (ordered after APPLICATION_CANCEL, before the local
  interrupt); the server releases the client's admission on receipt
  instead of waiting for the handle to close. Admission-drop timing is
  now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
  request publication under the send lock, best-effort sends the active
  token's APPLICATION_CANCEL frame, then interrupts local I/O; the
  server cancels MCP/subprocess work promptly. Contract tests accept
  both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
  the requester's license to act on "snapshotted and draining", so every
  connection interrupt is now initiated before the ACK is sent; a
  session could previously get one more request serviced after the
  requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
  prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
  timer granularity) burned ~1.6 s against permanently obstructed paths,
  stalling hello rejections past the client's timeout. The retry budget
  is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
  correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
  Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
  nonblocking send() on Windows is absorbed wholesale into AFD kernel
  buffering regardless of SO_SNDBUF, so send deadlines and interrupts
  could never engage against a slow peer (and the full payload was
  pinned in nonpaged pool). Slicing restores a deterministic
  backpressure point; a test hook pins SO_SNDBUF for the deadline and
  interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
  deadline, explicit connection states, refusal to free a server while
  a listener-owned connection is active.

Windows long-path support:

- Central path-aware wide conversion (canonicalize via GetFullPathNameW
  and prepend the extended-length prefix for absolute paths >=240) at
  the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
  store opens, and the daemon build-fingerprint/log paths. Deep managed
  installs (a 64-hex generation directory routinely exceeds MAX_PATH)
  now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
  ancestry validators now operate in the extended-length namespace;
  the launcher path is canonicalized (and prefixed when deep) once at
  entry so every downstream exact-string comparison stays
  form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
  GetFileAttributesW) so non-ASCII install paths survive argv[0]
  resolution.

Windows launcher install/uninstall transaction:

- FileRenameInfoEx names are NUL-terminated in an over-allocated
  buffer. FileNameLength governs per the contract, but filter drivers
  read FileName as NUL-terminated and appended adjacent heap bytes to
  created names — a flaky, garbage-suffixed rename target. Both the CLI
  and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
  .cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
  chars so the bare rename target stays under the FileRenameInfoEx
  NT-conversion ceiling at guard depths; 64 bits still uniquely
  identify the generation.
- When the running launcher's mapped generation backings pin .cbm
  against rename, the backings are relocated to activation-<pid>-N
  .retired tombstones beside the install (a mapped image may be renamed,
  never deleted; the launcher's liveness-guarded sweep reclaims stale
  tombstones). Every relocation is recorded, and a FAILED uninstall
  reverses the moves after restoring .cbm — via MoveFileExW with
  extended-length paths on both arguments, since the deep generation
  target is beyond the handle-based rename's bare-path reach — so a
  restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
  out so the tree is shallow enough for the detached cleanup's rd, and
  the cleanup's working directory strips the extended-length prefix
  (CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
  (CopyFileW destinations, CREATE_NEW tombstones, probe directories)
  are explicitly owner-stamped so the exact-owner validators hold on
  runner images; guard fixtures stamp hand-built trees the same way.

Diagnostics, tests and infra:

- diagnostics: discovery is now an always-delivered JSON control record
  (new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
  paths containing spaces; placement honors $TMPDIR with /tmp fallback
  via a diagnostics-local helper; the soak parser reads the JSON record;
  documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
  log levels, TMPDIR-with-spaces, and native Windows output-contract
  assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
  parents with production-shaped ancestry (LocalAppData on Windows, via
  th_secure_runtime_parent_new) — the runtime ancestry validation
  correctly refuses temp roots whose ancestors grant mutation rights to
  Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
  drive the documented startup-owner publication flow before reading
  generation-bound endpoint addresses. This turns the 26 Windows
  failures previously visible in CI's full-test job green without
  weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
  contract including failed-uninstall restore and immediate reinstall
  after uninstall; new daemon lifecycle and reworked hook-augment
  guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
  (it was set by CI but never read, so perf suites ran everywhere);
  four throughput/bench suites are classified as perf, the heavy
  store_arch suite moved to the slow-timeout tier, and two
  wall-clock-sensitive assertions were rewritten as invariant checks
  with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
  contract, soak daemon-recovery contract, path-safety helper, the
  Windows VM worktree-sync contract wired into scripts/test.sh, and
  vm/win.sh guards building its clean embedded-UI product in an
  isolated BUILD_DIR so it cannot clobber the incremental test build.
  provision-windows.sh now installs Node.js for the guards UI build.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-21 15:32:15 +02:00
Martin Vogel d826374577 test-infra(vm): remove embedded SSH key, pin host identity and installer 
The Windows-VM bootstrap embedded a personal SSH public key and the
drivers connected with StrictHostKeyChecking=no. Key material and host
identity are now strictly operator-supplied and live outside the
repository (~/.claude/cbm-vm/, never committed):

- windows-bootstrap.ps1 takes the operator's public key as a mandatory
  -SshPublicKeyPath argument (validated single Ed25519 line) and prints
  the VM host-key fingerprint to pin. Generating your own key is
  documented in the script and README:
  ssh-keygen -t ed25519 -N "" -C <label> -f ~/.claude/cbm-vm/id_ed25519
- win.sh and provision-windows.sh verify the pinned
  CBM_VM_HOST_KEY_SHA256 fingerprint before every connection
  (StrictHostKeyChecking=yes against a keyscanned known_hosts, Ed25519
  only, IdentitiesOnly), validate branch names before remote
  interpolation, and fail remote pipelines loudly (set -e -o pipefail).
- The MSYS2 installer download is pinned to a versioned release URL with
  SHA-256 verification instead of an unpinned latest.
- win.sh gains a sync subcommand mirroring the uncommitted worktree onto
  the VM; its manifest logic is guarded by
  tests/test_vm_worktree_manifest.sh, wired into scripts/test.sh.
  guards now runs the maintained PowerShell driver against a clean
  product build; smoke-install drives vm-smoke.sh.
- run.sh drops the stale duplicate windows-vm/mac-vm legs (the former
  carried its own unpinned ssh path) in favor of the vm/ drivers.

The previously committed public key has been rotated off the VM and is
no longer authorized anywhere.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-19 23:21:53 +02:00
Martin Vogel 08b7ff54d2 test-infra(vm): CI-identical smoke plus ubsan/pageheap/parallel drivers 
Add vm-smoke.sh, which stages the release pair (launcher + payload) in a
profile-rooted temp dir and runs scripts/smoke-test.sh exactly as the CI
windows pr-smoke step does, so the real-Windows leg exercises the same
managed-install path. Extend win.sh with ubsan-build/ubsan-test (UBSan works
under x64-on-ARM emulation; ASan does not), a PageHeap toggle for OS-level
heap checking of the native runner, and a full-suite parallel run. Document
the honest sanitizer limits and the daily loop in the README.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-19 16:56:02 +02:00
Martin Vogel aa90491792 fix(cli): stage Windows install transactions under a per-user root 
The activation transaction and the release-descriptor probe validate the
full ancestor chain of everything they stage: every path component must
have a trusted owner and grant mutation rights only to trusted identities.
That walk is deliberate TOCTOU defense and stays strict. The staging
directory, however, came from the process temp root, which routinely lives
in a tree that legitimately fails the walk through no fault of ours: under
msys2 TMP resolves inside the msys install tree (C:\msys64\tmp), and on
GitHub runners inside the runner work directory, both of which carry broad
inherited grants such as Authenticated Users modify on an upper component.
The staged pair then failed validation ("launcher descriptor candidate is
unsafe" / the smoke Phase 8 --version probe refusal) even though the staged
files themselves were private.

Add cli_windows_staging_root(): stage install and update transactions under
%LOCALAPPDATA%\codebase-memory-mcp, whose ancestor chain is user-owned end
to end, so the strict walk passes by construction instead of by environment
luck. Falls back to USERPROFILE\AppData\Local and finally the process temp
root when the environment is degraded.

Also make the staging observable where it previously failed blind:
cli_windows_stage_private_file reports which transaction step failed with
the OS error code, and cbm_mkdtemp warns once if the private security
descriptor cannot be applied and it falls back to default directory
security. A blind boolean here cost several blind CI round-trips.

win.sh: scp targets need native C:/ paths on Windows OpenSSH.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-19 14:32:05 +02:00
Martin Vogel 343582aa93 test-infra(vm): script-driven Windows VM leg for the local 3-OS gate 
The Windows VM becomes a first-class, reproducible local-CI leg driven
entirely over the local ssh link by maintained scripts:

- provision-windows.sh: idempotent host-side provisioner — msys2 base,
  CLANGARM64 (native, fast) + CLANG64 (CI arch) toolchains, repo
  clone/update, full build, smoke check. One command recovers a fresh VM
  or a lost disk.
- win.sh: daily driver — status/update/build/test/guards/smoke-install/
  sh/push-file/asan-build. smoke-install runs the managed-install E2E
  with stderr visible, the class the CI smoke probe structurally hides.
- windows-bootstrap.ps1: reduced to the one-time in-VM door-opener
  (OpenSSH + host key + GitHub-runner default-owner policy mirror), with
  the ISO delivery trick documented; toolchain work moved to the
  host-side provisioner. The previous winget path does not work over ssh
  and clipboard/shared folders do not exist on Windows-ARM guests.
- README.md: full lifecycle — VM creation, clone-before-drivers rollback
  point, SPICE guest tools, bootstrap, provisioning, daily loop, and the
  honest limits (no ASan on aarch64-windows; ASan also faults under
  x64-on-ARM emulation, so Windows sanitizer coverage stays on CI).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-19 14:14:11 +02:00
Martin Vogel a6a17e3a24 test-infra: fetch the Windows ISO from Microsoft's official page 
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-18 20:57:43 +02:00
Martin Vogel 43b3ae722d test-infra: Windows VM gets all cores - no artificial limits 
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-18 18:53:57 +02:00
Martin Vogel d09867d096 test-infra: size the Windows VM generously - vCPUs are scheduled, not pinned 
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-18 18:47:58 +02:00
Martin Vogel 37956940da feat(test-infra): opt-in windows-vm and mac-vm legs 
Two failure classes stay invisible to the container legs: kernel-real
Windows ACL/token/owner semantics (Wine only compile-checks) and the
GitHub-runner macOS environment (a developer Mac differs in temp
ancestry, ACLs, and policies - runner-only failures never reproduce
locally). Add real-VM legs for both, strictly opt-in: run.sh detects
the tooling and the named VM and prints setup guidance with a distinct
exit code when absent - explicitly requested legs are never silently
skipped.

windows-vm: UTM + CrystalFetch (both free), driven over ssh; the
in-VM bootstrap (vm/windows-bootstrap.ps1) installs the CLANG64
toolchain, enables sshd, and mirrors the runner's
NoDefaultAdminOwner=0 policy so freshly created objects are owned by
BUILTIN\Administrators - the exact class the daemon's owner gates must
survive.

mac-vm: Cirrus Labs' runner-equivalent image via tart (free OSS),
booted headless, worktree synced over key-based ssh, scripts/test.sh
executed in the runner-class environment.

vm/README.md documents both setups end to end.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
 2026-07-18 18:46:02 +02:00