mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-12 09:47:20 +08:00
ee9833df2c72b750378273897ffec9410fc2c4f2
776
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
1670414211 |
fix(release): skip_tests silently skipped smoke and soak too
Run 30504814537 built all eight platforms successfully with skip_tests=true, and
reported:
test: skipped (intended)
smoke: skipped (NOT intended)
soak: skipped (NOT intended)
release-draft: running
GitHub propagates "skipped" TRANSITIVELY down the needs graph. `build` carried
`!cancelled() && !failure()` so it overrode the skipped `test` and ran; `smoke`
had no `if:` and `soak` only checked soak_level, so both inherited the skip.
Nothing failed. Nothing announced it. The run then proceeded toward publishing
binaries that had never been smoke-tested or soaked -- the exact opposite of what
skip_tests was for, which was to skip a phase already proven green, not to
quietly stop testing the artifacts.
The second half of the defect is worse than the first: `release-draft` guarded
itself with a bare `!cancelled() && !failure()`, which is fail-OPEN. A skipped
job is neither cancelled nor failed, so a skipped smoke sails straight through
it. The draft was one gate away from being cut from unverified binaries, and the
only reason it was not is that the run was cancelled by hand.
Fixes, both directions:
- smoke and soak get `!cancelled() && !failure()`, so a deliberately skipped
ancestor no longer skips them;
- release-draft now requires `needs.smoke.result == 'success'` and enumerates
`needs.soak.result` explicitly, so a SKIPPED smoke BLOCKS the draft, and a
legitimately skipped soak (soak_level=none) stays distinguishable from a soak
that never ran.
tests/test_release_gate_chain_contract.sh pins both properties: every job
downstream of the optional phase must carry the override, and the draft must
demand smoke success rather than smoke non-failure. Verified in both directions
-- it passes on this workflow and fails with 2 violations when the previous
smoke/soak conditions are restored.
Nothing was published: the release remained a draft (isDraft=true,
publishedAt=null) and the run is cancelled.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
5a479facf7 |
fix(release): invoke the new CI scripts via bash, and pin the exec-bit class
Completes the previous commit, which carried only the two mode changes because
the call-site edits were not staged when it landed (--amend is denied in this
repo, so this is additive rather than a rewrite).
Call sites now go through `bash`, which is what most of this repo already does and
which cannot break if a mode bit is lost to a patch application or a non-POSIX
checkout:
scripts/package-release.sh -> bash scripts/ci/check-binary-composition.sh
.github/workflows/release.yml -> bash scripts/ci/append-vt-notes.sh
With the 100755 modes from the previous commit, both sides are now correct, and
either alone would have been sufficient.
tests/test_script_exec_bit_contract.sh pins the class so it cannot recur: any
tracked .sh whose COMMITTED mode is non-executable must not appear as the first
word of a command in workflows, scripts, test-infrastructure or the Makefiles.
The committed mode is the thing that matters and the thing no local run can
check -- the working copy having the bit is exactly why this shipped.
Verified in both directions: passes on this tree, and fails on the exact defect
when the mode and the call site are reverted together. It joins backslash
continuations before analysing, because its own first draft reported a false
positive on
... && bash \
test-infrastructure/vm/vm-run-tests.sh --soak
where the interpreter sits on the preceding line. A contract that cries wolf
teaches people to ignore contracts, so that had to be right before it could be
useful.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
1f674c805e |
harden: remove capability that should never have shipped
Microsoft's ML flagged the rc.1 release binaries. The decisive evidence is that the SAME sha256 went from 0/62 clean to Microsoft-detected in about an hour with no byte change, so the verdict lives partly in scanner-side state and no code change can promise a clean result. What code CAN do is stop shipping things that have no business in a release artifact, which is worth doing on its own merits and incidentally widens the classifier margin. Every claim below is verified against a built binary by the new gate, not by reading source. Executable stack (the worst of the findings). vendored/nomic/code_vectors_blob.S is the only assembly in the build and carried no .note.GNU-stack. An unannotated object makes ld assume the worst for the whole link, so EVERY Linux release we have ever shipped had GNU_STACK RWE. Adds the note (cause) plus ELF-only -Wl,-z,noexecstack (outcome); the gate fails the release if it returns. Test seams are now opt-in, never opt-out. TEST_SEAMS=1 defines CBM_ENABLE_TEST_SEAMS; without it the crash-orphan probe -- which forks a child that ignores SIGTERM and loops forever, then writes its pid to a caller-supplied path -- and the lease-ownership marker compile to trivial stubs, so call sites are untouched and the binary holds no fork, no signal handler and no env-var string. Opt-IN is the point: forgetting the flag yields a clean binary rather than a leaky one. scripts/test.sh requests it in the leg that consumes it, and tests/test_worker_watchdog.sh now asserts the capability up front instead of dying later with an opaque "Killed: 9". The daemon's background version check is gone. It spawned curl against api.github.com/repos/.../releases/latest on the first eligible session of every run to say "a newer version exists" -- a release URL and an outbound request in every shipped binary, for something the install scripts already report. The INJECTABLE SEAM survives: update_ops is still honoured, the fakes in tests/test_daemon_application.c still cover notice/ownership/cancellation/replay, and with no provider application_update_subscribe_locked returns early so no generation ever starts. "No network request by default" is now structural. Dead capability out of release builds. The tar.gz/zip extraction block (gzip_decompress through cbm_extract_binary_from_zip, plus its cli.h declarations) moves under CBM_CLI_ENABLE_TEST_API -- verified self-contained, zero uses of any helper outside it, only callers the excluded updater and tests/test_cli.c. Downloading an archive, decompressing it, picking an executable out of it and marking it executable is the canonical dropper composite; it is now absent rather than merely unreachable. SQLite is built with -DSQLITE_OMIT_LOAD_EXTENSION (no caller of load_extension anywhere in src/ or internal/), removing that API surface and part of the dlopen/dlsym surface. Temp files and environment scanning (S2/S3). Predictable paths in mcp.c, artifact.c and diagnostics.c are created privately and exclusively and written through the returned descriptor; pass_envscan.c no longer descends symlinked directories out of the project root, and its fixed 512-byte path buffers no longer truncate into pointer arithmetic that could land outside the buffer. Build-time entropy. mimalloc's version banner baked __DATE__/__TIME__ into every binary, so two builds of identical source seconds apart could never share a hash and no release could inherit a false-positive determination made about its predecessor. Local patch removes it (marked to survive refreshes), -Wdate-time makes any future use a build error, and -Wl,--no-insert-timestamp stops the PE header carrying the link clock. scripts/ci/check-binary-composition.sh is the proof that each removal stays removed, wired into package-release.sh after strip so the local artifact-flow smoke enforces exactly what CI does. It asserts absences plus a CANARY string, so handing it a compressed, stubbed or empty file fails instead of passing vacuously, and a missing tool is a hard error -- a skipped assertion must never look like a satisfied one. Two build-system traps found by that gate, both of which had silently defeated a fix: the product binary is compiled in one shot from sources, so a flag flip did not rebuild it (now tracked by a .build-config stamp that also removes the binary, making it independent of mtime granularity); and prod_sqlite3.o / prod_mimalloc.o depended on a single named source, so SQLITE_OMIT_LOAD_EXTENSION and the mimalloc patch BOTH compiled to nothing on the first incremental build. Source review would have called them done. Deliberately NOT changed. Three seams stay in release artifacts because scripts/smoke-test.sh runs against the real artifact and needs them: CBM_TEST_CRASH_ON and CBM_TEST_HANG_ON inject the faults that prove supervisor recovery, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID is what keeps the PATH smoke from writing the tester's actual PATH. The gate treats those as an allowlist, so a NOVEL seam still fails. The true no-UI standard build is deferred rather than rushed: src/ui/* is in PROD_SRCS and four files outside src/ui reference UI symbols, including the daemon that serves the UI, so that assertion reports instead of failing until the split lands -- a gate everyone knows is red teaches people to ignore gates. No grammar is removed. ObjectScript accounts for essentially all binary growth since the last provably-clean release (+21.4MB rodata, +1.1MB text from two four-line shims), which made it the obvious ablation candidate, but a dry run performed twelve real Defender endpoint scans across standard/UI and amd64/arm64 with ObjectScript, the daemon and the expanded hooks all present and every scan was clean. Nothing there is a deterministic trigger, so cutting a community-contributed language would spend a real feature on unproven margin. Lean is not a candidate either: at 99.6MB of source it is by far the largest grammar, but it shipped in v0.9.0 which scanned 20/20 clean, so removing it would produce a novel unscanned profile instead of restoring a known-good one. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3e343a3eb2 |
ci(release): revert the VT gate to zero tolerance
The pre-release ML false-positive tolerance (single-engine Microsoft "!ml" verdicts downgradable with Defender endpoint evidence, #1340) is reverted by owner decision: cbm does not ship binaries carrying a VirusTotal detection, demonstrably false or not. A "trojan" badge on a release asset is a reputation cost the project is not willing to price in, however good the accompanying evidence. The gate returns to its original form: any detection, by any engine, on any artifact, on any version blocks the release. The endpoint verification tool and the evidence side-channel are removed with it; the notes renderer keeps its extracted-script form but only ever states a verified "0 detections". False positives are resolved upstream instead: verify the bytes on a real Defender endpoint, submit a Microsoft false-positive report for the exact hashes, wait for the detection to clear, then RE-RUN the failed verify job -- which does not rebuild, so the cleared hashes are the shipped hashes. tests/test_vt_gate_zero_tolerance_contract.sh pins the decision: clean passes; 1 malicious and 1 suspicious each block across stable, -rc., -pre and -alpha versions; plus a tripwire for the specific reverted evidence mechanism returning. Loosening this gate again has to consciously delete that contract. The skip_tests dispatch input and the script-extracted notes step survive the revert -- both are orthogonal to gating policy. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
48fc942418 |
ci(release): double-verify ML antivirus false positives instead of re-rolling builds
Release run 30464288732 was blocked by the VirusTotal gate: three linux-amd64
binaries flagged 1/62 by Microsoft's Wacatac.B!ml -- fully stripped binaries
(0 symbols, verified on the exact artifacts), the state that scanned clean in
the two previous cycles. Meanwhile a real Defender endpoint (engine
1.1.26060.3008, signatures 1.455.410.0 updated the same day, RTP on) scans
the identical bytes clean. Four cycles of evidence now say the same thing:
this verdict is an unstable ML decision boundary, not a property of the code,
and no build-side lever moves it durably -- stripping, downloader removal and
metadata changes each "worked" only until a later build flipped it back.
So stop treating the flag as buildable-away and verify it honestly instead:
check-virustotal.sh may downgrade BLOCKED to TOLERATED only when ALL hold:
- pre-release version (-rc./-pre/-alpha/-beta); stable releases never
- every failing file flagged by exactly ONE engine
- that engine is Microsoft and the verdict ends in "!ml" (never a
signature name)
- hash-pinned Defender ENDPOINT evidence is attached to the draft release
(defender-endpoint-verification.txt) proving Microsoft's shipping
product, signature-updated at scan time, reports the exact bytes clean
av-endpoint-verify.sh (new) produces that evidence: downloads the draft
assets, scans them on the local Windows VM endpoint, refuses to attest if
RTP is off or Defender itself detects, uploads the hash-pinned result.
The gate prints the exact command when evidence is missing; re-running the
failed verify job does not rebuild, so the bytes stay fixed.
append-vt-notes.sh (new, extracted from inline YAML per venue-parity) then
renders the release-notes table honestly: a tolerated file reads "1/62 ML
false positive, endpoint-verified clean", never "0 detections".
tests/test_vt_gate_tolerance_contract.sh pins all nine decision directions
against a stubbed VT API and release store -- clean pass, stable-never,
missing/stale/DETECTED evidence, signature-named verdict, non-Microsoft
engine, multi-engine -- so the tolerance provably fails closed.
Also: release.yml gains skip_tests for re-releases of an already test-green
tree (build/smoke/soak/verify always run; lint failures still gate via
!cancelled() && !failure()).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
ae52db3147 |
test: make three Windows nondeterminisms deterministic
All three cost real release cycles, and none is fixed by widening a budget. daemon_application_cancels_physical_job_only_after_final_session waited for the SUBSCRIBER COUNT to reach 2, then cancelled both sessions and asserted the physical job had started exactly once. The job starts asynchronously after subscription, so both cancels could land first and leave starts == 0 -- arguably the correct outcome. It now waits for the state the assertions actually require. Verified 47/47 on Windows, the only platform it ever failed. The parallel harness refused outright when the suite leader had already exited, because taskkill /T cannot walk a tree from a dead PID. But the leader can exit between the timeout decision and that call, so the harness itself lost a race: a natural exit at the wrong moment failed the whole wave. It now proves cleanup the only way still available -- nothing parented to that PID -- and its contract asserts the PROPERTY rather than the phrase "tree cleanup" it used to grep for. That string pin is what broke when the guard was reworded while behaving correctly; the contract now checks rc==2 AND that the descendant really did survive, which would also catch a guard that claims to fail closed while leaking. extract_wide_flat_file_is_linear took ONE sample per size, so the ratio carried the noise of both. On a loaded Windows VM linear code measured 51x against a 40x bound (184ms -> 9387ms). Best-of-N instead: timing noise only ever adds time, so the minimum is the cheapest good estimate of the noise-free cost. The bound is deliberately unchanged -- it sits where linear (~20x) and quadratic (~128x) are each >=2x away, so raising it would move the test toward the very signal it exists to catch. Now measures 19.1x on Windows, 21.4x on macOS. Also: the smoke's `cli` helper redirected stderr to a file and discarded it, so any of the 10 bare `VAR=$(cli ...)` assignments could kill the run under `set -euo pipefail` printing NOTHING. One such abort cost a full Windows cycle just to locate and still could not be attributed. It now surfaces the command and its stderr. Neutral wording on purpose: one call site expects a non-zero exit and must not read as a failure. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
9ecabe0a6f |
refactor(cli): move self-update out of the product on every platform
Windows already handed updating to install.ps1, because a running .exe cannot
replace its own image. The remaining platforms now do the same, for a different
and better reason.
An in-process updater is structurally a downloader: it fetches a remote archive,
verifies it, extracts it, marks the result executable and runs it. That composite
lived in every shipped artifact -- along with the release URLs it fetched -- to
serve a command most users run a handful of times. The install scripts already
do all of it, are idempotent (so re-running one IS the update), and run while cbm
is not running.
`update` now prints the exact command for the running platform and exits 0:
unix bash "<dir>/install.sh" (or the curl one-liner when not adjacent)
windows powershell -ExecutionPolicy Bypass -File "<dir>\install.ps1"
Both scripts ship inside their release archive, so the printed path resolves
next to the binary. Flags are still parsed and validated, so `update --dry-run
--ui` keeps rejecting typos rather than silently accepting them.
The updater itself is EXCLUDED from release builds rather than left to dead-code
elimination: build_update_url, download_verify_install, extract_and_install_binary,
checksum fetch, detect_os/detect_arch and the cli_download_* helpers now compile
only under CBM_CLI_ENABLE_TEST_API, which release builds do not define. The C
suite still covers the flow through the activation test seam. Verified against
the release binary: `releases/latest/download` is gone entirely.
Also removes the MCP background update check. It ran on the first tool call and
did:
cbm_popen("curl -sf --max-time 5 -H 'Accept: application/vnd.github+json' "
"'https://api.github.com/repos/.../releases/latest' 2>/dev/null")
An embedded shell command plus a startup network callout, in every session, to
tell users a newer version exists. It shelled out, depended on curl being
installed, and phoned home from every agent session. The install scripts report
versions; this did not need to be in the server.
Smoke follows the same collapse. Phases 6c and 14 asserted a Windows-specific
handoff and a POSIX in-process replacement; they now assert one platform-neutral
contract selected by UPDATE_SCRIPT, and 14a's byte-identity check compares the
driver against ITSELF before and after -- comparing against "$BINARY" passed on
Windows only because Windows has no signing step, while the POSIX fixture
ad-hoc re-signs its copy and so differed before `update` ever ran.
The Linux glibc guard is RELOCATED, not retired. The standard linux asset links
glibc 2.38+ and breaks Debian 11, RHEL 8 and Ubuntu 20.04, so the installer must
fetch the static "-portable" build. That constraint moved from the binary to
install.sh, and the smoke now guards it where the behaviour lives.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
0a22fb14df |
fix(windows): absorb transient file locks when retiring a running image
Release run 30385988475 lost three Windows smoke jobs to one phenomenon on two
different surfaces. A Windows file that was just written or just executed can
refuse deletion and rename for a moment while a scanner reads it or while the OS
finishes reaping a child. Both conditions clear on their own within moments.
It reads as nondeterminism because it is timing: windows-2025 passed both amd64
variants while windows-latest -- the SAME image, windows-2025-vs2026 -- failed
both, and windows-11-arm failed too.
PRODUCT: uninstall abandoned a live installation.
With one binary per platform, uninstall must remove the very image it is
executing, and on Windows a running image can only be renamed, never
overwritten. That rename got a single attempt, so a transient sharing violation
surfaced to the user as
error: failed to remove ...; completed configuration/index cleanup may remain
with the install left half-torn-down. install.ps1 already retries this exact
operation ten times for this exact reason; the C path now does the same, and
only for sharing/access/lock violations, so a file that is genuinely held still
fails closed instead of spinning.
Proven with a failure-injection seam rather than by racing a real scanner:
3 injected failures must still succeed, 64 must still fail AND leave the target
in place. Deterministic, so it does not need CI to learn whether it works.
HARNESS: the smoke killed itself silently.
Phases that install a binary then `rm -rf` the directory hit the same lock. Under
`set -euo pipefail` that returns non-zero and ends the run WITHOUT printing
anything -- the logs jump straight from "OK 13h" to job cleanup with no FAIL
line, which is what made this expensive to read.
None of those 16 removals is an assertion; they are fixture cleanup, and
_smoke.yml already states the principle ("a temp dir on an ephemeral runner must
never fail the job") that smoke-test.sh did not honour. They now go through
smoke_rmtree, which retries so the disk is actually reclaimed -- these fixtures
hold ~300 MB binaries and runners are disk-tight -- then warns and continues.
Verified: Windows VM full smoke ALL PASSED (previously died after 13h), macOS
full smoke ALL PASSED, activation_transaction + cli 259 passed on Windows, all
10 shell contracts pass, lint-ci clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
740d9b4fc5 |
test(smoke): retire the Windows Phase 14 missing-executable requirement
The fixture contract pinned the literal string
STALE_CMD="$UPDATE_HOME/retired-install/codebase-memory-mcp.exe"
so that Windows Phase 14 would carry an MCP entry naming a missing executable.
Two separate things retired that intent, and because the requirement only ever
matched the STRING it kept passing while the property it stood for was gone.
First, the fixture itself now copies a binary to the retired path (added with
the launcher removal), so that path stopped being missing no matter what this
string said.
Second, Windows update is a handoff to install.ps1. Nothing rewrites the entry
in-process the way the launcher-managed update did, so an entry naming a foreign
path just survives to uninstall -- which correctly refuses to remove a config
entry owned by a DIFFERENT installation. Phase 14f was then asserting that
uninstall performs exactly the removal it must never perform, and it failed for
that reason.
The requirement now pins the installed binary, which is what install.ps1 leaves
a real Windows user holding. The missing-executable classification is not
dropped, it is covered by named unit tests rather than by this fixture:
cli_editor_mcp_preserves_unrecorded_posix_absolute_entries_without_probe and
cli_editor_mcp_preserves_unsafe_windows_drive_probe, the latter covering the
Windows missing-drive case specifically.
This contract runs as step zero of every test leg, so the stale requirement
failed all 30 test jobs of release run 30384603265 -- on Windows, macOS and
Linux alike -- from one violation.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
5f2c146866 |
test(daemon_ipc): pin the Windows startup interleaving instead of polling for it
Two Windows tests needed the startup thread to be observed HOLDING the startup lock, and looked for it by polling the lock for BUSY up to 200 times. That state has no lower bound: whenever the rendezvous handoff does not block, acquire -> handoff -> release completes in microseconds, so the window can close between two polls and startup_observed asserts on a coin flip. This blocked release run 30374135923 on test-windows (windows-2025, CLANG64, x86_64). Budget-tuning the poll was tried and rejected -- a wider budget only lowers the failure rate, and a transient window is never a fixed test. Instead the production path exposes a gate that fires once the startup lock is held and before the handoff runs, matching the hook pattern already used for POSIX publication. The test parks the startup thread there and releases it when it is ready, so the interleaving is pinned by construction. The acquired flag is monotonic, so the observation cannot be missed; the remaining bound only catches a thread that never started at all. The gate is inert unless a test installs it. Also fixes an unrelated use-after-free this exposed. ipc_forever_wait_server closed the connection and NULLed it from the server thread, while the test was about to call cbm_daemon_ipc_connection_interrupt on the pointer it had already loaded -- and receive_frame can return on its own before that call lands. It crashed roughly one Windows run in five, independent of the change above. Ownership now stays with the test, which closes after joining, so interrupt can only ever see a live connection. Verified on real Windows: 5 consecutive daemon_ipc runs, no failure and no crash. macOS 46 passed x5 under ASan/UBSan; Linux 6606 passed, 0 failed. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b0eae97c03 |
fix(windows): keep install.ps1 pure ASCII so PowerShell can parse it
install.ps1 did not run. It failed to PARSE, dying with a cascade of syntax
errors before its first statement, and it has been in that state on main since
|
||
|
Martin Vogel
|
a54ea95719 |
fix(windows): ship one binary — remove the launcher stub flagged as a dropper
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
46b77c3e3b |
test(daemon-app): wait for the stable terminal state, not the draining window
The fresh-request guard test waited for 'job terminal AND >= N/2 subscribers still attached'. Publish flips terminal and releases the blocked prior requests in the same locked step, so that conjunction is a transient window whose width is thread-wakeup scheduling: the busy-spin caught it by luck, a 1ms-yield poll (#1307) sampled past it on fast x64 runners, and no budget can pin it (release runs 30305464193 and 30309182389 failed it from both directions). The production guard never consults subscriber counts — application_find_active_job_locked skips any terminal job — and the test's downstream assertions (starts==2, destroys==2, stale/fresh response separation) catch a terminal-job reuse in every interleaving. So wait only for the stable end-state (active jobs == 0) and drop the racy helper. 47/47 locally. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
8268af4188 |
test(daemon-app): yield in the terminal-job wait like every sibling helper
app_wait_for_terminal_job_with_subscribers was the only app_wait_for_* helper polling without cbm_usleep(1000): a sleepless spin pins a core for up to 10s and can starve the daemon threads it is polling on scarce-CPU runners. Release run 30305464193 hit exactly this on windows-11-arm (test_daemon_application.c:2671 ASSERT(terminal_with_prior_subscribers)); the suite passes 47/47 locally with the yield in place. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a279a28878 |
test(watcher): key the blocked-git spawn-marker wait to a starved-runner budget
The blocked-git child's PID-marker wait was a fixed 5s. On the 4-vCPU windows-11-arm release runners that budget was missed twice in a row (run 30300256783, attempts 1+2: test_watcher.c:1038 ASSERT(marker_ready)) while every faster leg stayed green — spawn latency under CPU starvation, not a watcher defect. Raise the four spawn-marker waits to a shared 20s budget (WATCHER_TEST_SPAWN_MARKER_BUDGET_MS). The wait stays bounded: a genuinely wedged spawn still fails, at 20s instead of 5s. Termination, completion, and join budgets are untouched. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
052a7a80ea |
fix(contract): tolerate transient port-file read denials in the 0h self-test
On Windows the fixture server's atomic port publish (os.replace) — or a first-touch AV scan of the fresh file, with Defender ACTIVE on the VM — briefly denies the read: is_file() is true, read_text() raises PermissionError, and the poll loop crashed the whole contract with a bare traceback (observed on the VM once the 0e/0h needle fixes let the chain reach this self-test at all). Treat an unreadable-yet port file as not-ready and keep polling inside the existing deadline. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c400dc1665 |
ci: hosted runners cannot enable Defender RTP — revert runner-side gating
Dry run 30225685743 failed at the ensure-defender step on every Windows job: the hosted images policy-lock real-time protection OFF (WinDefend starts, Set-MpPreference accepts, RTP stays off — untamperable from the job). Revert to the pre-gate runner behavior: no ensure-defender step in runner jobs, the release Defender scan is on-demand and fail-soft again. The local VM preflight KEEPS the fail-closed Defender-ON gate (same canonical script) — AV-interaction coverage is a deliberate local superset where the platform allows it. The venue-parity contract now enforces the INVERSE for runners (ensure-defender must not gate hosted jobs) and still requires the VM preflight gate; VM README documents the verified posture delta. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
def698427e |
fix(contract): route the 0h VM-soak needle through the canonical soak entry
The venue unification moved the VM soak sequence and its completion guard into scripts/soak-legs.sh; the smoke-fixture contract still pinned the old direct soak-test.sh call in vm-run-tests.sh, failing Step 0h — and with it every test leg on every venue of dry run 30224804737 (the test wave's 20 reds share this one cause). Masked until now: the contract chain runs sequentially and Step 0e failed first on the pre-fix tree, so 0h was never reached; post-fix verification used --suites mode, which skips the chain. All contract steps 0a-0j now verified green standalone in bulk. Invariant preserved, anchor moved: the VM wrapper must route --soak through the canonical entry under its protected temp root, and the canonical entry must completion-guard every leg on the soak summary (grep for '=== soak-test: PASSED ===' lives in soak-legs.sh:101). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b5b96e7006 |
fix(tests): leak-free teardown for the disconnect-cancel runtime test
LSan (Linux amd64 leg): 200 bytes — the runtime client (192) plus its IPC connection (8) — leaked whenever an earlier stage of daemon_runtime_disconnect_cancels_blocked_non_index_child... failed: both close blocks were gated on the request thread having completed, so a failed run skipped the close entirely. Teardown now closes unconditionally: close_begin interrupts the transport (forcing the blocked call to complete), the join becomes safe, and close_finish frees the client on every path. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
ad713bce93 |
fix(mcp): guarantee valid UTF-8 in tree-format output
Dry run 30206811293: macos-15-intel release smoke failed B3 with a visibly correct 'semantic: 50' table — BSD grep returns no-match for the ENTIRE output when any line carries a NUL, control byte, or invalid UTF-8 (verified: printf poisoned input -> grep -qE fails), so one bad byte in one row silently unmatches every anchored check. Close the class at the emission choke point: control bytes and invalid UTF-8 force the quoted form; quoting escapes controls as \u00XX and replaces invalid sequences with U+FFFD (RFC 3629 validation, NUL-safe). Tool output is valid UTF-8 by construction. Unit test proven RED-on-revert (raw bytes emitted) and green with the fix. The smoke's B3 failure path now also dumps od -c bytes so a future red names the exact byte. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a20a50df77 |
fix(ci): executable bits on canonical entry scripts
git-apply of the staged unification recorded the new canonical entries as 100644; workflows execute them directly (run: scripts/ci/...), so a fresh CI checkout would fail at ci-ok/shard-union/soak/package steps with permission denied — the same class the parity worktree's own preflight just tripped over locally. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0a40fe8bea |
feat(ci): local-remote parity lanes — artifact-flow smoke, glibc floor, Defender-ON everywhere
Closes the closable local-vs-remote venue gaps so that, outside arch legs / YAML glue / release plumbing, every local red is a remote red and vice versa on shared legs: - scripts/package-release.sh: THE canonical archive step (names + five-file Windows bundle layout defined once); _build.yml's eight inline archive blocks become calls to it. - scripts/ci/smoke-artifact.sh: the artifact-flow smoke lane — build, package, extract, then the canonical wrapper in artifact mode; wired as run.sh smoke-artifact (compose service), win.sh smoke-artifact, and directly runnable on macOS. Archive-layout bugs now surface locally instead of in a release dry run. - glibc-floor leg (Dockerfile.glibc22 + compose + run.sh): portable binary smokes on ubuntu-22.04/glibc 2.35; the dynamic binary must refuse there (2.38+ floor by design). - Defender-ON parity (user directive): scripts/ci/ensure-defender.ps1 enables + VERIFIES real-time protection, fail-closed; runs in every Windows CI job (_test x2, _soak x3, _smoke, pr.yml) AND in the VM preflight; _smoke.yml's scan engine-failure soft-skip becomes a red gate. Expected cost: slower Windows jobs (AV scanning during install/build/test I/O); the next dry run proves the runner side. - Contracts: launcher-bundle five-file check retargeted onto package-release.sh + per-archive canonical-call association; venue-parity contract requires the new lanes, counts one ensure-defender step per Windows job, adds --help probes for the new entries. Extended contract fails on the pre-change tree (verified: 20 violations on HEAD). - VM README: ephemerality/Defender posture documented — utmctl has no snapshot verb, so per-run revert stays a manual qcow2 option; the sweep preflight remains the standing mechanism. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
afe38ab8e3 |
fix(contract): retarget launcher-bundle contract onto canonical smoke wrappers
The venue unification moved release-smoke staging/execution out of _smoke.yml into the canonical wrappers (vm-smoke.sh / smoke-local.sh), so Step 0e's needle checks against inline YAML failed on every venue. Re-anchor each pinned invariant where the behavior now lives: - canonical-exe execution + profile-rooted fixtures + update fixture identification: asserted in vm-smoke.sh / smoke-local.sh, plus the workflow must call the wrapper on the extracted artifact - version/install-audit preflight needles updated to the artifact-dir text shape (still workflow provisioning) - RUNNER_TEMP: absolute ban replaced by a provision/scan allowlist walker + an explicit no-exe-under-RUNNER_TEMP backstop No pinned security property is dropped; the retargeted contract fails on the pre-unification architecture (verified against HEAD). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3fc93d12da |
ci: unify smoke/soak/test venues onto canonical leg entries
The staged venue-unification bundle: _smoke.yml onto the shared wrappers with an extracted-artifact input (CBM_SMOKE_ARTIFACT_DIR), soak-legs.sh as the one canonical soak entry (quick + query-leak legs across all venues), protected per-user TEMP roots shared across venues, clean-disk preflights before every VM/Docker run, win.sh and run.sh routed through the canonical scripts, soak.yml retired, and the venue-parity contract (whitelist walker: workflows may provision or call canonical entries, nothing else). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e5c79418ca |
revert(install): restore the activation-transaction diagnostics to main
The refusal notes changed the error text the Windows launcher guards match against, and test-windows-guards went red on this PR after they landed. They were investigation instrumentation and they succeeded -- they turned 'staging transaction open failed (status -3, os 0)' into ERROR_DISK_FULL and named the exact site -- but a diagnostic that shifts a matched error string does not belong in a release-blocking change. Worth redoing deliberately on its own, with the guards' expectations updated in the same commit. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e9167d6a83 |
diag(guards): report the reinstall exit code
The guard reported only a version banner, so a silent nonzero exit was indistinguishable from a rejected precondition. The install command works in isolation and across a plain uninstall-then-reinstall, so the exit code is the remaining unknown -- a crash code would say so directly. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
babc3d837c |
revert(guards): drop the unverified launcher-guard edits
Neither the fresh portable pair nor the ownership stamp fixed the immediate managed reinstall, so both were guesses and do not belong on a release-blocking PR. The guard's failure is pre-existing on main -- byte-identical output there, and unchanged with the #581 fix disabled -- so this branch carries no regression and no speculative test edits either. What the investigation did establish: the reinstall's install command fails with 'managed install requires a verified adjacent codebase-memory-mcp.exe launcher' when run directly, and emits no diagnostic on the stream the guard captures, which is why the guard reports only a version banner. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e547de5d06 |
fix(guards): stamp the staged reinstall pair with current-user ownership
The adjacent launcher was present but refused as unverified: copied files inherit Administrators ownership on runner images, and the launcher's exact-owner validators require the current user, which is what stamp_fixture_owner_current exists to simulate. Without it the guard failed on the product's ownership check rather than on the immediate-reinstall contract. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
82f36713bb |
fix(guards): stage a fresh portable pair for the immediate-reinstall check
The immediate-reinstall guard reused the portable pair from work/portable-install, whose adjacent launcher the earlier managed installs consume. Managed install then refuses correctly -- 'requires a verified adjacent codebase-memory-mcp.exe launcher' -- so the guard was failing on the product behaving as designed rather than on the contract it protects. Reproduced by running the install command directly (exit 1, that exact error) and confirmed identical on origin/main, so this was never a regression from this branch. Stages its own pair, leaving the assertions untouched. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
5d97f82f23 |
test(mem): assert the map contract the allocator can actually honour
The probe assumed the walk would attribute over half of a 12 MB allocation. mimalloc v3 exposes only the main heap, abandoned pages and the calling thread's theap -- there is no API to enumerate every theap -- so on Windows the probe's blocks are unreachable through all three and the walk saw ~190 KB. Asserts the triple documented in mem.h instead: what the walk cannot see, the residual must carry. Either attribution covers the probe or the committed total grew by at least as much, so memory can never go missing from the map unnoticed. Bucket attribution is checked where the walk did reach the probe. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
73a15f93ac |
diag: add a memory map that reports what it cannot see
Growth investigations were guessing, because the diagnostics snapshot only carried process totals: RSS and committed bytes say memory grew, never where. The map adds allocator-live bytes and a size-class histogram to each snapshot, so a distinctive class identifies the allocation without per-callsite instrumentation. The design constraint is honesty about coverage. Walking the allocator reaches the aggregate heap plus this thread's heap and its abandoned pages, which is not the whole process, and in a build where malloc does not route through the allocator at all it reaches nothing. A per-subsystem tally would report zeros there and read as a clean bill of health. So every snapshot carries three independent totals and an explicit residual -- os_committed, allocator-live, residual = the difference -- and the triple localises growth instead of implying it: live grows leak on a walked heap; buckets say which size residual grows, flat another thread, retention, or non-allocator memory both flat, RSS grows mappings or stacks, not the heap That guard paid for itself immediately. Pointed at the open Windows growth in #581 it reported live=0 with zero blocks and the residual carrying all of it -- not 'the heap is innocent' but 'this walk saw nothing', which is the correct reading for a process demonstrably holding tens of megabytes. Together with the growth surviving a build with the allocator override compiled out, that says the Windows allocations under investigation never reach the allocator, so an allocator-based map cannot localise them and the next instrument has to sit lower down. Covered by a test that allocates a known volume in a known size class and requires the map to attribute it there, so the map can never regress into silently reporting zeros. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0d46fffbaa |
fix(test-infra): bind the smoke fixture server without a reverse-DNS lookup
The macOS Intel leg failed the fixture contract on every run while every other platform passed. Removing the readiness fsync did not change it; the hardened diagnostics named the real cause on the first remote run: waited=30.0s, exit_status=alive, port_file=absent, staged_temp_files=none, empty startup log -- the process was healthy but had never reached publish_port. http.server.HTTPServer.server_bind() resolves socket.getfqdn(host). On a host whose resolver does not answer for the bind address that call blocks for the resolver timeout, so the constructor never returns and no port is ever published. The fixture now binds through a subclass that keeps the threading server but skips the FQDN resolution, which only feeds CGI-style variables this fixture never serves. Proven both directions locally by forcing socket.getfqdn to hang: the subclass publishes its port immediately, the stock server never does. That forcing hook is kept as a permanent contract guard, so the reverse-DNS dependency cannot return without turning the gate red on every platform rather than on one runner nobody can reproduce. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
636dd3f92c |
fix(test-infra): repair the native Windows soak transport and release checks
The native Windows soak could never start its server, so that gate had never actually measured anything. The root cause was the harness, not the daemon: an MSYS filesystem FIFO feeding a native Windows process delivers empty stdin (reproduced 5/5), while a Bash anonymous coprocess carries the full JSON-RPC initialize exchange to the same protected payload, and the failing daemon log showed an orderly client disconnect rather than a daemon-side eviction. The soak now uses a named coprocess on native Windows only, duplicating its endpoints to stable fd3/fd4 and closing the originals so closing fd3 still delivers EOF; POSIX keeps the FIFO path. The coproc syntax sits inside eval because macOS system Bash 3.2 must still parse this file even though only MSYS2 Bash 5 executes that branch. With transport repaired the next exact failure surfaced: native Windows Python cannot open an MSYS /c/... diagnostics path. Both consumers now pipe the file through stdin, the pattern already established elsewhere in the repo, and the soak recovery contract forbids reintroducing a direct native-Python open. Also on the release path: - The smoke fixture server no longer fsyncs before atomically publishing its port. This is ephemeral readiness signalling, not crash-durable state, and the macOS Intel runner failed inside that durability sync. When the contract does fail it now reports the observable state -- waited, exit status, port file, staged temp files, interpreter, startup log -- because the previous verdict named nothing on the one runner we cannot reproduce locally. - The POSIX publication test hook no longer compiles into Windows builds, fixing an exact -Werror unused-variable failure. The setter keeps a parameter-consuming Windows stub because it is public API. - The Wine leg assembles the real release layout (payload plus canonical launcher) and version-checks both, running the launcher through cmd so it has a Windows-visible parent. The unsupported Wine soak is removed: Wine stays a fast compile/package/version check, and native Windows is authoritative for daemon, locking, ACL and process-lifetime semantics. - run.sh soak-windows routes to the native Windows VM, which validates the payload, builds the protected per-user temp root, stamps ACLs, and refuses success without a completion summary. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
62821375c4 |
fix: harden daemon release paths and verification
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
50d0cc8c7d |
test: free the harness-built return-type table in the parity runners
The parity harness drives the pipeline passes with a bare ctx and never ran the production teardown, so the return-type table pass_calls builds leaked — latent until the fixture gained typed Java methods, and visible only under Linux LeakSanitizer. Both runners now free it exactly the way pipeline.c does. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
4a3aac6478 |
fix(pipeline): one Interface edge split in both venues, explicit overrides
The parallel per-file resolve hardcoded INHERITS for every base-class relation while the sequential semantic pass made the Interface split, so any corpus large enough for the parallel path had its explicit implements demoted: elasticsearch carried 11,179 INHERITS edges pointing at Interface nodes and 3 IMPLEMENTS. Both venues now share one decision keyed off the target node's label — the graph truth the edge attaches to. OVERRIDE edges existed only for Go's implicit interface satisfaction. A serial full-graph tail run by both pipelines now matches class methods against explicit bases by name and emits Method-to-Method OVERRIDE edges for the explicit languages, with Go excluded to avoid double cover. The venue parity fixture gains a Java interface/implements/extends trio — IMPLEMENTS parity used to compare 0 == 0 and guarded nothing — plus an exact-count test so the parity can never go vacuous again. At scale: elasticsearch IMPLEMENTS 3 to 11,182 and OVERRIDE 0 to 49,695, TypeScript 0 to 4,496 and 594, dotnet/runtime 0 to 12,953 and 45,667, with INHERITS plus IMPLEMENTS sums conserved exactly. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
03dc9c916a |
fix(discover): grow the walk stack instead of failing wide repositories
A directory with more immediate subdirectories than the fixed 512-frame walk stack made the whole discovery walk hard-fail with files=0 and a misleading 'check repo_path' hint — dotnet/runtime carries 855 sibling dirs in one JIT regression folder and could not be indexed at all. The stack now grows geometrically from the same initial capacity; the parent frame is a popped copy, so growth cannot dangle it. Covered by a 600-dir fanout reproduction. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b98f6a99c4 |
test: sanitizer-keyed budgets and a causality-based httpd interruption proof
The build system now stamps CBM_SANITIZED_BUILD for any non-empty SANITIZE, so timing budgets key off build-system truth instead of compiler-specific __SANITIZE_ADDRESS__ probes that miss trap-UBSan and TSan; the two bench suites use it. The httpd interrupt test proved 'the interrupt worked' with a wall-clock bound, which is a lottery under sanitizers. A send-deadline test hook pins the deadline out of reach so the join itself is the causality proof, and the stop watchdog becomes a true ten-second hang detector. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
87f7c0d784 |
daemon: deterministic job interleavings and queued explicit index admission
Two changes to the index coordinator, developed together and validated as one state: Determinism seams. A cancel racing a worker start was only observable on a loaded runner; now the interleaving is forced by construction: a test hold parks every job thread before its first pre-start cancel check, and a background-initialize completion counter turns sleep-then-assert negatives into positive-signal waits. The cancel-before-worker-start case is pinned by a dedicated test; three auto-index admission tests now wait for the admission tail instead of sleeping through it. Queued admission. An explicit index_repository request that hits the physical job limit used to surface a raw busy error. The request thread already blocks for the whole index, so it now queues: a cancellable, shutdown-aware wait-and-resubscribe loop replaces the error path, with a wait counter as the parked-state observable. The programmatic coordinator API keeps its non-blocking busy answer for callers with their own retry policy (watcher, auto-index). The job-limit test asserts the new contract: parked without a second worker start, admitted and completed once the occupying job finishes. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
2af28c9d48 |
test: size the daemon-application wait budget for TSan's slowdown tail
The suite's observation waits shared a 2000 ms budget that is a hang detector, not a latency assertion — every waiter returns the moment its condition holds, so green runs never pay it. Under ThreadSanitizer's 5-20x slowdown on a loaded 3-core macOS CI runner the cancel-delivery wait lost the tail of that distribution once in seven otherwise-green TSan rounds (the cancel itself had succeeded; only the fake worker's counter observation timed out). The budget is now 10000 ms. Verified: nine consecutive daemon_application runs under native macOS TSan with the widened budget, all green, zero ThreadSanitizer warnings. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
568e613141 |
test-infra: overlap the local ladder, open the serial tail, keep container I/O off virtiofs
Local runs left most of the machine idle: the serial tail ran sixteen suites one at a time on otherwise-idle cores, the three platform legs were launched by hand (usually sequentially), and every container-leg object write crossed the virtiofs bind mount. - run-tests-parallel.sh tail scheduling in two phases: the FLEX suites (timing-shaped but free of the shared per-account daemon runtime namespace) run CBM_TAIL_JOBS-wide (default 2), then the EXCL group — daemon-family plus the suites that drive daemon one-shots or supervisor rendezvous — runs strictly sequentially on a machine exactly as quiet as the old fully-serial tail gave it. The wave was already fed longest-first by the shard dealing order, so the drain-out no longer ends on a heavy straggler. - ladder.sh: one maintained entry point for the full local push gate with the legs overlapped — lint, the Linux container suite, and the Windows VM suite in the background, the macOS suite in the foreground, one verdict per leg, logs kept per leg. A missing prerequisite fails its leg loudly instead of silently skipping. - win.sh test-par now runs through vm-run-tests.sh (--par mode): the full parallel harness under the CI-shaped protected temp root with complete output. It previously ran under the MSYS-shared /tmp and piped through `tail -25` — the same truncated-blindness class that hid 40 Windows failures from the `test` command. - docker-compose: build artifacts and the incremental fixture cache move to named volumes on the container VM's native filesystem. Object writes over the virtiofs bind mount are the container legs' largest avoidable I/O cost, and the fixture cache now survives across container runs. - test_mem(win): the first valid full-parallel VM run proved working-set trimming beats the re-touch mitigation (19 MB resident of a 256 MB double-touch at 18 parallel suites). The RSS probe now VirtualLocks a 64 MB span — locked pages are exempt from trimming, making the measurement pressure-immune — with bounded touch-and-sample retries when the lock is unavailable. Red-to-green under the same 18-job load. - cli: the portable install's staging error now appends the activation refusal note (predicate, SID, object) like the managed path already does — a bare "activation transaction I/O failed" on a CI-only failure is undiagnosable without it. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
f1beaec4d6 |
test: cache the incremental fixture and index the CI-shaped corpus everywhere
The incremental suite cloned fastapi from the network on every run and indexed a DIFFERENT corpus per environment: CI used a sparse checkout without docs/ and tests/ (the assertion thresholds are sized for it) while local runs indexed the full tree — twice the files for the same assertions, a per-run network dependency, and the suite's dominant cost on the sanitized container leg. The fixture is now cloned from the network at most once per machine into a persistent cache (committed with an atomic rename so a torn download can never pose as a valid cache; CBM_TEST_FIXTURE_CACHE overrides the location), every run local-clones from it offline, and docs/ plus tests/ are trimmed everywhere — the portable equivalent of CI's historical sparse profile, so all environments now index the same corpus the assertions were written against. Same 163 tests, same assertions: 74s -> 31s native, with the container leg's cut scaling with the halved indexing volume, and the "network offline" flake source reduced to first-run-per-machine. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
f51745533c |
fix(win): reconcile the C suite with the managed-launcher contract
The first full Windows x64 CI run of this branch (earlier contract-step walls had always aborted the leg) surfaced 40 failures: the managed launcher's fail-closed install/update/uninstall gates and the suite's POSIX-modeled activation tests were both new in this branch and had never been reconciled, and they hid three real production bugs. Production fixes: - ui: config publishes with POSIX rename semantics. MoveFileExW's legacy replace cannot supersede a destination a reader holds open (the name lingers until the last handle closes, even with FILE_SHARE_DELETE), so a config save racing an open reader failed. The temp file is opened with DELETE access and renamed in-hand (bare-path target, NUL-padded rename record) before close, falling back to MoveFileExW on any rename-by-handle failure so non-NTFS config homes keep the previous behavior. - subprocess(win): the child's redirected log handle now shares write access. POSIX places no lock on the redirected log, so a second producer may append while the child runs; the missing FILE_SHARE_WRITE mandatory-locked every other writer out, which the index-supervisor callback-isolation test legitimately exercises. - cli(win): the portable uninstall path composes the managed binary name with its .exe suffix; it could never find (or remove) the installed binary before. - activation: security-predicate refusals are now diagnosable. The transaction's predicates refuse without a usable OS last-error, which collapsed five distinct refusal classes into one blind "status -3, os 0" at every caller. Each refusing predicate records what refused — predicate, untrusted SID, and object path — first-wins so the deepest cause survives, and the staging error line appends it. This is what identified every Windows refusal in this batch, including the real install-failure class on drive-root-rooted payload directories (Authenticated Users holds inherited Modify there; profile-rooted directories do not). Windows test dispatch (release binaries unchanged): - Test builds (CBM_CLI_ENABLE_TEST_API, the existing test-object macro) compile the portable activation flow on Windows, reachable solely through the installed activation test seam and announced loudly on stderr; release binaries compile exclusively the managed transaction, so the dispatch polarity fails closed. The suite's install/update/ uninstall calls route through a seam-if-absent wrapper, keeping the shared activation and agent-config semantics unit-covered on Windows; POSIX arms keep exercising the real activation machinery. - Two new release-gate tests assert the fail-closed contract with the seam OFF: a portable payload may not self-update/self-uninstall, and install refuses to adopt or replace a foreign file at the canonical launcher path even under --force. - Test-bug fixes the first valid Windows runs exposed: the version- cohort crash test reused one conflict record across the probe and the turnover acquire (whose entry memset erased the probe's detail before the assertions read it); the activation staged-path assertion now checks containment rather than a separator (the \\?\ namespace performs no forward-slash translation); the ui atomic test captures, cleans up, then asserts so a failure cannot poison later tests; the PATH-visibility guard is POSIX-only (Windows configures PATH through the user registry, not a shell rc). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
34f9ad1b3d |
fix(daemon,win): CI-green fixes and a daemon-stability guard
Fixes surfaced while bringing the shared-coordination-daemon branch green across macOS, the Linux containers, and the real Windows-ARM64 VM, plus a new stability guard that caught one of them. - daemon(win): the CLI teardown transition latched a failed release as permanent, but a Windows participant-state release must briefly try-hold the shared startup/legacy gates and legitimately collides with a concurrent one-shot's teardown; the release is retriable by contract (it always retains the transition), so parallel one-shot commands no longer report "CLI coordination cleanup failed" despite succeeding. Found by the new stability guard's churn section. - cli/main(win): install/update/uninstall and the Augment hook-script removal derive the managed launcher's identity from its plain drive-form path (cli_windows_plain_utf8), not the \\?\ extended-length form, so agent-config ownership matches on Windows and uninstall no longer leaves the MCP entry or owned hook scripts behind. - security: the `daemon start --open` browser launch is now shell-free (ShellExecuteW on Windows, cbm_exec_no_shell elsewhere) instead of system(); the CORS origin check spells out its two literal loopback URLs so the static URL audit sees a complete value. Both clear the Layer-1 allow-list audit. - coordination: version-cohort lock retries now sleep a per-process jittered interval — fixed-period retries can phase-lock two participants so one starves. The activation-quiesce test's observation window now covers worst-case candidate staging (three tamper-defense hashes over a ~1 GB sanitizer binary exceed the old 30 s budget on container I/O). - smoke/soak: the agent-config smoke matches Windows config paths through their escaped-backslash quoted form; the soak parser skips the update-available banner before the JSON summary and gained a one-shot CLI admission-churn phase inside the RSS/FD leak window. The build-dir safety fixture creates a real symlink/junction (MSYS2 ln -s otherwise deep-copies) so its traversal-refusal contract is genuinely exercised, and the symlinked-agent-roots test demotes its planted links to an unprivileged owner when run as root so the refusal is really tested. - tests(win): a new daemon-stability guard exercises the parameter surface, hook fail-open with its rate-limited notice, start-twice and occupied-port handling, busy-stop refusal against a live MCP session, kill -9 crash recovery, and sequential/parallel one-shot churn. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0baddf7887 |
daemon: permanent lifecycle, daemon-backed CLI/hooks, real-Windows hardening, long-path launcher transactions
Daemon lifecycle and Windows correctness, verified on a real Windows 11 ARM64 VM through the maintained test-infrastructure/vm drivers, plus the macOS and Linux arm64 suites and the container lint gate. Daemon lifecycle: - daemon start/stop/status subcommands. `daemon start` launches a PERMANENT daemon (spawn shape is byte-exact argv; survives idle periods and session ends) and reports an already-active daemon instead of failing. Permanence is honored at every stop latch: last-committed-client disconnect, host initial-client window, coordinator release, and application final-session close — a permanent daemon also keeps admitting new sessions after its last one closes. - daemon stop refuses while sessions are active and lists the blocking peers (pid/role) that must finish first; an idle daemon drains through the activation-shutdown machinery with the ACK ordered after connection interrupts. A second stop is idempotent. The wire ops are no-cohort first-frame requests with peer fingerprint authentication, so stop/status never conflict with an exact-build admission gate. - One-shot CLI commands now execute through the daemon (index workers keep their local supervised path). A cold CLI run that had to spawn a temporary daemon prints a hint that `daemon start` removes the per-command startup tax; a warm daemon is recycled silently. - Hooks are connect-only fail-open: with no daemon present the hook emits a visible, rate-limited notice (Claude-dialect systemMessage plus stderr for other dialects) and always exits 0 — augmentation is never allowed to block the caller's tool use. - Version skew: a newer-build client automatically drains an older-build permanent daemon (strict semantic-version triples only; dev builds never auto-drain) and the build-conflict message names `cbm daemon stop` as the manual escape hatch. Windows IPC/runtime (real-VM verified): - ipc(win): persistent pending overlapped ConnectNamedPipe. The accept path used to destroy its listening pipe instance on every 20 ms poll timeout; a client attaching in the teardown window was severed or left on an orphaned pipe object whose HELLO no server handle could ever read, absorbing the connect until the client's own timeout expired. The pending connect now survives poll timeouts and nothing is destroyed while a client could be attaching. - ipc(win): drain-before-close for final responses. Closing a named-pipe server handle can discard a just-sent response before the peer reads it (POSIX stream sockets never lose buffered data on close). A bounded cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX, immediate on interrupted connections) now precedes close in runtime_worker_finish and runtime_reject_inline, so hello-conflict, capacity and disconnect acknowledgements reliably reach the peer. - runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no transport half-close, so close_begin now announces departure with an explicit frame (ordered after APPLICATION_CANCEL, before the local interrupt); the server releases the client's admission on receipt instead of waiting for the handle to close. Admission-drop timing is now identical to POSIX shutdown() semantics on every platform. - runtime(win): client close cancellation. close_begin serializes with request publication under the send lock, best-effort sends the active token's APPLICATION_CANCEL frame, then interrupts local I/O; the server cancels MCP/subprocess work promptly. Contract tests accept both correct outcomes (interrupted transport or decoded CANCELLED). - runtime: activation acknowledgement ordering. The activation ACK is the requester's license to act on "snapshotted and draining", so every connection interrupt is now initiated before the ACK is sent; a session could previously get one more request serviced after the requester observed the ACK. - service(win): deadline-bounded private-file prepare. The conflict-log prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms timer granularity) burned ~1.6 s against permanently obstructed paths, stalling hello rejections past the client's timeout. The retry budget is now a 250 ms deadline; transient share collisions still retry. - subprocess(win): cmd.exe /C payload encoder quotes metacharacters correctly (root cause of the git-on-Windows failure cluster). - watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native Windows stop/unwatch cancellation test with exact-image verification. - httpd: send_all writes in bounded 64 KiB slices. A single giant nonblocking send() on Windows is absorbed wholesale into AFD kernel buffering regardless of SO_SNDBUF, so send deadlines and interrupts could never engage against a slow peer (and the full payload was pinned in nonpaged pool). Slicing restores a deterministic backpressure point; a test hook pins SO_SNDBUF for the deadline and interrupt tests. - ui/http: shutdown lifecycle — interrupt checks, response-wide send deadline, explicit connection states, refusal to free a server while a listener-owned connection is active. Windows long-path support: - Central path-aware wide conversion (canonicalize via GetFullPathNameW and prepend the extended-length prefix for absolute paths >=240) at the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite store opens, and the daemon build-fingerprint/log paths. Deep managed installs (a 64-hex generation directory routinely exceeds MAX_PATH) now index, stage and activate correctly. - activation transaction: its own file APIs and the component-walking ancestry validators now operate in the extended-length namespace; the launcher path is canonicalized (and prefixed when deep) once at entry so every downstream exact-string comparison stays form-consistent. - Executable self-resolution uses the wide APIs (GetModuleFileNameW, GetFileAttributesW) so non-ASCII install paths survive argv[0] resolution. Windows launcher install/uninstall transaction: - FileRenameInfoEx names are NUL-terminated in an over-allocated buffer. FileNameLength governs per the contract, but filter drivers read FileName as NUL-terminated and appended adjacent heap bytes to created names — a flaky, garbage-suffixed rename target. Both the CLI and the launcher rename helpers are fixed. - Uninstall retires state via rename-aside (.cbm -> .cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex chars so the bare rename target stays under the FileRenameInfoEx NT-conversion ceiling at guard depths; 64 bits still uniquely identify the generation. - When the running launcher's mapped generation backings pin .cbm against rename, the backings are relocated to activation-<pid>-N .retired tombstones beside the install (a mapped image may be renamed, never deleted; the launcher's liveness-guarded sweep reclaims stale tombstones). Every relocation is recorded, and a FAILED uninstall reverses the moves after restoring .cbm — via MoveFileExW with extended-length paths on both arguments, since the deep generation target is beyond the handle-based rename's bare-path reach — so a restored install keeps its generation backings and stays runnable. - After a committed uninstall the retired tree's backings are relocated out so the tree is shallow enough for the detached cleanup's rd, and the cleanup's working directory strips the extended-length prefix (CreateProcessW lpCurrentDirectory silently ignores prefixed paths). - Files created under Administrators-default-owner directories (CopyFileW destinations, CREATE_NEW tombstones, probe directories) are explicitly owner-stamped so the exact-owner validators hold on runner images; guard fixtures stamp hand-built trees the same way. Diagnostics, tests and infra: - diagnostics: discovery is now an always-delivered JSON control record (new cbm_log_control) that survives CBM_LOG_LEVEL suppression and paths containing spaces; placement honors $TMPDIR with /tmp fallback via a diagnostics-local helper; the soak parser reads the JSON record; documented in docs/CONFIGURATION.md. Red-first coverage for suppressed log levels, TMPDIR-with-spaces, and native Windows output-contract assertions. - tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint parents with production-shaped ancestry (LocalAppData on Windows, via th_secure_runtime_parent_new) — the runtime ancestry validation correctly refuses temp roots whose ancestors grant mutation rights to Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and drive the documented startup-owner publication flow before reading generation-bound endpoint addresses. This turns the 26 Windows failures previously visible in CI's full-test job green without weakening any validation. - tests(win): the launcher guard covers the full permanent-launcher contract including failed-uninstall restore and immediate reinstall after uninstall; new daemon lifecycle and reworked hook-augment guards run the start/recycle/stop flow end to end. - tests: CBM_SKIP_PERF is now actually consumed by the test runner (it was set by CI but never read, so perf suites ran everywhere); four throughput/bench suites are classified as perf, the heavy store_arch suite moved to the slow-timeout tier, and two wall-clock-sensitive assertions were rewritten as invariant checks with coarse hang-detector backstops. - build/test infra: build-dir safety contract, UI dev-proxy security contract, soak daemon-recovery contract, path-safety helper, the Windows VM worktree-sync contract wired into scripts/test.sh, and vm/win.sh guards building its clean embedded-UI product in an isolated BUILD_DIR so it cannot clobber the incremental test build. provision-windows.sh now installs Node.js for the guards UI build. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
d826374577 |
test-infra(vm): remove embedded SSH key, pin host identity and installer
The Windows-VM bootstrap embedded a personal SSH public key and the drivers connected with StrictHostKeyChecking=no. Key material and host identity are now strictly operator-supplied and live outside the repository (~/.claude/cbm-vm/, never committed): - windows-bootstrap.ps1 takes the operator's public key as a mandatory -SshPublicKeyPath argument (validated single Ed25519 line) and prints the VM host-key fingerprint to pin. Generating your own key is documented in the script and README: ssh-keygen -t ed25519 -N "" -C <label> -f ~/.claude/cbm-vm/id_ed25519 - win.sh and provision-windows.sh verify the pinned CBM_VM_HOST_KEY_SHA256 fingerprint before every connection (StrictHostKeyChecking=yes against a keyscanned known_hosts, Ed25519 only, IdentitiesOnly), validate branch names before remote interpolation, and fail remote pipelines loudly (set -e -o pipefail). - The MSYS2 installer download is pinned to a versioned release URL with SHA-256 verification instead of an unpinned latest. - win.sh gains a sync subcommand mirroring the uncommitted worktree onto the VM; its manifest logic is guarded by tests/test_vm_worktree_manifest.sh, wired into scripts/test.sh. guards now runs the maintained PowerShell driver against a clean product build; smoke-install drives vm-smoke.sh. - run.sh drops the stale duplicate windows-vm/mac-vm legs (the former carried its own unpinned ssh path) in favor of the vm/ drivers. The previously committed public key has been rotated off the VM and is no longer authorized anywhere. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
85c2d99480 |
test(daemon): raise the frontend announce deadline above the claim budget
The backpressure/EOF fixtures waited only 10s for the forked daemon to announce readiness, but the daemon's own startup includes the cohort-claim path whose internal budget is HOST_DAEMON_CLAIM_TIMEOUT_MS (12s). Under the 4-core oversubscription of CI runners a slow-but-legitimate startup blew past 10s and was SIGKILLed before it could announce, so daemon_frontend backpressure failed almost every ubuntu run while passing on fast machines. Raise the announce deadline to 30s, comfortably above the daemon's own worst-case budget. Verified: the backpressure failures no longer reproduce under a local CI-fidelity 4-core run. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
fb54f54e60 |
test(windows): define the missing sha256_file helper in the launcher guard
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3325b31b37 |
test(daemon): skip the build-fingerprint hash in tests (root-cause fix)
Every spawned worker/daemon re-hashes the entire executable image for the exact-build fingerprint; for the ASan test-runner (hundreds of MB) that is tens of seconds per spawn on constrained CI runners and was the sole cause of the whole daemon-family readiness-timeout flake class (index_supervisor, daemon_frontend, cli, and much of the Windows daemon surface). Add a test-only seam under CBM_CLI_ENABLE_TEST_API: when CBM_TEST_BUILD_FINGERPRINT holds a valid stub the hash is skipped. test_main.c sets it once; every forked child and re-exec'd worker inherits it, so exact-build match AND mismatch behaviour is preserved (a mismatch test still passes a different fingerprint via argv). Never compiled into production. Local mac run: index_supervisor 6/6, the four daemon suites 111/0, exact-build test green. The generous frontend catastrophic-alarm values are kept as backstops. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
8d465bc9ce |
test(daemon): widen worker/frontend hang-guard deadlines for slow ARM CI
The macOS legs went green with the prior calibration but the 3-core ubuntu-24.04-arm runner is slower still: each spawned worker re-hashes the entire ASan test-runner image (hundreds of MB) for the exact-build fingerprint before signalling readiness, which there takes tens of seconds. Raise the hang-guard deadlines generously - index_supervisor 60s ready / 90s terminal, frontend 30-90s, cli 45s. These are hang guards, not benchmarks: large values only slow genuine-hang detection, never mask a regression. (Root-cause fix - a test-only fingerprint injection seam that skips the hash - is tracked for follow-up.) Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |