Found while proving the previous commit, in the same step, and the worse
of the two defects because it fails OPEN. The classification was
if printf '%s\n' "$FILES" | grep -qE '^(src/|internal/|...)'; then
under the runner's `bash -eo pipefail`. grep -q exits on its first match;
once the list is larger than the pipe buffer printf is still writing,
dies of SIGPIPE, and pipefail turns the MATCH into a failed pipeline. The
else branch then writes product=false, and pr-smoke and memwaste are
skipped -- on exactly the PRs that change the most files. The previous
commit's message flagged this as a follow-up; it is fixed here instead.
The list now reaches grep through a here-string, so there is no pipe to
break. The regex is unchanged.
Proof, by executing the workflow's own run block under
`bash -eo pipefail` with a stub gh, ten runs each:
849 KB list, first line src/a.c before: product=false 10/10
after: product=true 10/10
849 KB list, docs only product=false before and after
small docs-only list product=false
422 fallback on the real #2246 product=true
fallback, nonexistent merge ref rc 128, no output written
It takes roughly 1,500 changed files to cross the buffer, which is why
ordinary PRs never showed it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The `changes` job decides whether a PR touches product code, and ci-ok
requires it. It asks GitHub for the PR's file list. An earlier fix moved
it off the .diff endpoint, which rejects PRs over 20k lines, onto the
paginated pulls/files endpoint -- but that endpoint renders the diff
server-side as well, and for a PR that regenerates a vendored parser it
answers
HTTP 422: Sorry, this diff is taking too long to generate.
Observed on #2246 (a 42 MB sql/parser.c): `changes` red, therefore ci-ok
red, for a reason that has nothing to do with the contribution. The
compare endpoint fails the same way, so there is no API route to the
file list for such a PR. Every grammar refresh would hit this.
When the API call fails, the job now takes the list from git instead.
The PR merge ref's first parent is the base, so a name-only diff across
that one merge commit is exactly the PR's changed files. The fetch is
depth 2 with --filter=blob:none into a bare repository under RUNNER_TEMP:
commits and trees only, no blobs, nothing checked out and nothing from
the PR executed. No new action, no new permission, and the API path and
the classification regex are untouched, so ordinary PRs behave as before.
A ::notice:: line records when the fallback was used. If the fetch fails
too, the step exits non-zero and writes no output -- the gate fails
closed, as it did before.
Proof, by executing the workflow's own run block under
`bash -eo pipefail` with a stub gh:
API path, docs-only list rc 0 product=false
fallback, real #2246 rc 0 product=true, the same five
files `git diff --numstat`
reports for the PR; 0.85 s,
188 KB fetched
fallback, nonexistent merge ref rc 128 no output written
Contract tests that read pr.yml -- security_gate_fail_closed,
smoke_fixture_contract, windows_bundle_contract, venue_parity_contract --
pass.
Not addressed here, noted for a follow-up: the classification pipes the
list into `grep -q` under pipefail, which can report a match as a
failure once the list outgrows the pipe buffer.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dependabot split one upgrade across three PRs -- init (#2185), analyze
(#2187) and upload-sarif (#2186) -- but init and analyze share a
configuration file and CodeQL refuses to read one written by a different
version:
Loaded a configuration file for version '4.38.0',
but running version '4.37.9'
So #2185 and #2187 each go red on their own and neither can reach a green
required check, which means they cannot be merged one after the other
either: whichever lands first leaves the pair mismatched. The action also
warns about it directly -- "Not all workflow steps that use
github/codeql-action actions use the same version".
Both refs in codeql.yml therefore move in one commit, to the same pinned
SHA b96794f015dfd88f77b49b1c93e0fa7110f94c63 that both bot PRs target.
scorecard.yml's upload-sarif is deliberately not touched here: it runs in
a separate workflow that shares no configuration with init/analyze, which
is why #2186 is green on its own and can land as its own PR.
Supersedes #2185 and #2187.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The memory-core linter is a ratchet: raw malloc/calloc/realloc/free/strdup
sites may not grow, because allocations belong on one accounted path
(src/foundation/mem_core.h) rather than spread across hundreds of call
sites. This branch's own work had added 35 raw sites across nine files --
the parallel k8s prep, the configlink key matching, the worker pool's
counted path, the Go shared stdlib registry, the spill namespace copy,
the route service array and the namespace-name arrays.
All of them now use cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/
cbm_mem_strdup. One was a latent bug rather than a style point:
pass_k8s freed a buffer that k8s_read_file had allocated with raw malloc,
so converting only the free would have mismatched the allocator -- the
read function and every free of its result are converted together.
The Windows free-space path widens its argument through the memory core
instead of cbm_utf8_to_wide(), which allocates with raw malloc.
Two files improved enough to tighten their baselines, which the ratchet
requires so the gain cannot be quietly given back: sqlite_writer.c 92->74
(the index-cell arena), pass_k8s.c 13->6 and registry.c 19->17.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/fuzz.sh drives the libFuzzer targets in two shapes from one entry:
a fixed --runs from a fixed --seed, so a CI smoke is a function of the code
and not a lottery (O9), and a time-boxed --max-total-time for exploration.
Checked-in seeds are copied into a working corpus under build/fuzz/, so a
run never rewrites the repository; a crash leaves its reproducer behind and
the script exits non-zero. Wired as a smoke into dry-run.yml and as
exploration into nightly-soak.yml.
The venue-parity contract rightly refused both workflow steps: a venue may
provision or plumb, but only a CANONICAL leg entry may exercise the
product, and neither script was registered. Registering them — rather than
working around the contract — is the fix, since both own their leg end to
end (build, corpus or binary, artifact paths, exit code) exactly as
soak-legs.sh owns the soak. _memwaste.yml joins the policed venue walk for
the same reason, and both scripts join the layer-5 interface probes.
Those probes immediately earned their keep, finding two real defects:
`fuzz.sh --help` exited 2 because the positional target consumed the flag,
and memwaste.sh read a typo'd flag as the repository path — which would
have indexed nothing and reported an empty lane as a clean one.
test_shell_line_endings.sh now degrades where the checkout has no
reachable repository metadata: the Linux container bind-mounts the working
tree, and a git worktree's .git is a pointer to the host repository, so
`git ls-files` cannot run there and the contract failed with "matched no
files — the glob set is broken", killing the leg before it compiled. It is
a repository property, so it stays gating in every venue that has the
metadata (macOS host, the Windows VM's real checkout at C:\cbm with 117
files, every hosted-CI checkout). test_version_metadata_contract.sh already
guards the same way for the same reason; the alternatives considered and
rejected are recorded at the case.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Answers a question the existing tooling could not: of everything we
allocate, how much is never used?
Lanes, all driven by scripts/memwaste.sh:
event every allocation observed through the memory core — size, age,
churn, and whether the bytes were ever written
access SanitizerCoverage load/store instrumentation, so "allocated and
never touched" is measured rather than inferred
scaling k vs 2k replicas through clang profile counters, to catch
super-linear work per file
padding allocator slack per size class
ratchet the numbers recorded per venue so a regression is visible
The event lane needed a foreign-free counter: on Windows the override sees
frees for blocks it never allocated (they come from the CRT), and counting
those as untracked tripped the soundness gate. mem_events now separates
them (14 daemon / 5 worker on the Go corpus).
Two portability fixes the lanes surfaced, both real:
- the enable decision must not be cached before `environ` exists. Under
-fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our
wrapper before glibc sets up environ, so the lane silently disabled
itself and produced no dump at all on Linux.
- the access map needs 48 bits: arm64 Linux hands out addresses above
2^47, which a 47-bit map dropped on the floor.
mem_profile is retired into this: it measured a strict subset and had its
own sampling story.
Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on
all three platforms report-only, and pr.yml/release.yml carry the wiring.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The sysctl that lifts the AppArmor restriction is best-effort, and the test
runner prints per-shard aggregates only -- 'N passed, M skipped', never
individual test names. So if the sysctl stopped working, the #1830 namespace
test would quietly return to skipping and the log would stay green and look
identical. That is the same false assurance that let a security relaxation ship
behind a test which never executed.
Probe it the way the test does (unprivileged, no sudo) and print the answer, so
there is one greppable line of evidence per ubuntu leg. Deliberately not a gate:
a runner that cannot lift the restriction should still run everything else.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three maintainer decisions from the challenger read on the #1830 unit.
1. DELETE THE CACHE (not just fix its comment).
posix_ancestor_overflow_uid() memoised via pthread_once behind a comment
claiming "immutable /proc state". Both halves were false: unshare(CLONE_NEWUSER)
rewrites /proc/self/uid_map, and pthread_once state survives fork() already
marked done. A process that forked and then entered a namespace kept the
parent verdict and refused a directory it should have accepted.
It was harmless only because every caller today runs in a freshly exec'd
process -- the daemon double-fork execs, workers go through
cbm_subprocess_spawn's fork+exec, worker_pool.c never forks, and src/ contains
no setns or unshare at all. That made a security decision depend on an
invariant nothing enforced. The next fork-without-exec caller would have
inherited a stale verdict silently.
Now derived fresh per call: two small /proc reads against an openat + fstat +
fchmod + ACL check per path component in the same walk.
With a test that would have caught the original bug and needs no namespace.
daemon_ipc_posix_overflow_uid_is_never_cached_issue1830 counts real
derivations through the production entry point and asserts the count rises on
every ancestor check. Re-introducing a cache fails it by name on every
platform. Proven both ways on linux/arm64 as uid 1001: cache re-introduced
49 passed 1 failed; removed 50 passed 0 skipped.
Worth knowing why that test is not redundant: with the re-exec probe in
place, the userns smoke test PASSES even with the cache restored, because the
probe is a fresh process and immune to it by construction. The two tests are
complementary -- the smoke test binds the namespace wiring, this one binds
the absence of the cache.
2. RUN THE GUARD ON THE CORE MATRIX.
Ubuntu 23.10+ ships kernel.apparmor_restrict_unprivileged_userns=1, so the
#1830 smoke test skipped on every 24.04 leg and its only real-namespace
coverage sat on the two ubuntu-22.04 legs. Those are broad-matrix only, so
the test ran in NO pull request, and GitHub retires those images on
2027-04-17 -- after which it would have become a permanent silent skip, and a
skip is not a red. A deliberate relaxation of a security check was guarded by
a test that ran nowhere a change would be reviewed, on a clock.
_test.yml now lifts the restriction with one sysctl on every ubuntu leg
(hosted runners have passwordless sudo), best-effort so a runner that refuses
simply returns to skipping. The O10 skip-whitelist comment no longer presents
the AppArmor restriction as immovable.
3. KEEP THE TOLERANCE, CORRECT THE SAFETY ARGUMENT.
The comment claimed that in a single-uid map "nobody reachable can have
created it or can mutate it". The overflow uid is what EVERY unmapped host
uid maps to, not only host root (user_namespaces(7)), so overflow-owned does
not prove root-created: on a shared host another local user's sticky
directory is indistinguishable from root-owned /tmp from inside the namespace.
The tolerance stays, because there is no in-namespace discriminator that
could tell them apart -- and narrowing it would refuse the legitimate
root-owned 01777 /tmp that #1830 exists to accept, i.e. it would revert the
fix in practice. What changes is the claim: the real guarantee is that no
principal reachable from inside the namespace can create or mutate such an
ancestor, and the leaf is never tolerated as overflow (0700, euid-rechecked),
so a hostile host-side ancestor owner is bounded to denial of service and
socket-path control.
macOS daemon_ipc 53 passed 2 skipped; linux/arm64 50 passed 0 skipped;
lint-ci clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#2189 gave codeql-gate the security-events:read it needs to see code-scanning
alerts, and granted it at the two callers I knew about: pr.yml and release.yml.
There is a third. dry-run.yml calls the same reusable workflow and was left
granting only the workflow default, so every dispatch of it now fails at
startup -- before a single job runs, with no job output to explain why.
Observed immediately: run 34725401182 on 7d74337e, startup_failure.
The contract test did not catch it because it iterated a hardcoded
("pr.yml", "release.yml"). A test that checks only the callers you remembered
cannot catch the one you forgot, and it reports green while doing so -- the
same shape of false assurance as the 403-to-"0 alerts" bug this whole contract
exists to prevent.
So the list is now derived: every workflow containing
`uses: ./.github/workflows/_security.yml` must grant security-events:read and
actions:read, with a floor of three callers so a silent deletion is noticed
too. Reverting only dry-run.yml's grant now fails the test by name.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The codeql-gate has never checked anything. It declared no permissions block,
so it inherited the workflow's contents:read and the code-scanning alert API
answered 403. gh writes the API error BODY to stdout, so
ALERTS=$(gh api '.../code-scanning/alerts?state=open' --jq 'length' 2>/dev/null || echo "0")
left ALERTS holding
{"message":"Resource not accessible by integration",...,"status":"403"}0
every [ ... -gt ... ] then failed as a non-integer comparison, the if took its
false branch, and the step printed "CodeQL gate passed (0 alerts)" and exited 0.
Observed in the log of a GREEN run (34700514102, job 103571322660) while the
repository had an open high-severity alert. The `|| echo "0"` reads as defensive
and is the exact opposite: it converts "I cannot see the alerts" into "there are
no alerts", which is the one direction a security gate must never fail in.
Grant the job security-events:read (and actions:read, which the CodeQL-run poll
needs for the same reason), and replace the fallback with a helper that treats a
non-zero gh exit or a non-numeric body as a hard failure. The helper returns
rather than exits because it runs inside a command substitution, where an exit
would end only the subshell and let the caller continue with an empty count --
the same silent pass in a new costume; both call sites check the status.
tests/test_security_gate_fail_closed.sh pins both halves and fails with six
assertions against the previous workflow.
Also bumps graph-ui's vitest to ^4.1.11 (GHSA-82fw-gwwq-j7x9, path traversal via
@vitest/mocker, CVSS 5.9). Development scope, and graph-ui ships in no release
artifact, so nothing released was ever exposed -- but it was the sole open alert
and the gate above is worthless while it stands unresolved. Raising the floor
past the vulnerable range also stops a fresh install resolving back into it.
graph-ui: npm ci clean, 47 tests in 12 files pass on 4.1.11.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
apt.llvm.org rotates its per-major noble snapshot channels; the
llvm-toolchain-noble-22 channel transiently emptied, breaking every lane that
installs clang from it. #2124 already moved Dockerfile.msan (clang-22 ->
clang-21) for the same reason. The two GitHub-hosted lanes that install from
apt.llvm.org were still on 22 and would break identically on the next
rotation: _test.yml test-diag (ASan+UBSan) and _lint.yml lint-mem
(clang-analyzer, clang-tidy). Repin both to clang-21 / clang-tidy-21 /
llvm-toolchain-noble-21 (the channel #2124 verified apt.llvm.org serves on
amd64+arm64). The macOS LSan lane's Homebrew llvm@22 pin is intentional and
unrelated to apt churn (left unchanged); the clang-format-20 CI-lint lane is
untouched.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The SHA dependabot pinned (efb35369) is upstream's v3.0.3 tag, a major bump
from the v2 line: 3.0.0 moves the action runtime from Node 20 to Node 24 and
changes nothing else. The pin comment still read v2, which would misdirect the
next reader of this file; label it with the version the SHA actually is.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The test-lsan-macos leg installs the unversioned `llvm` formula, which
tracks Homebrew's current stable. That moved from 22.1.8 to 23.1.0 today,
so the leg's compiler now depends on which bottle the runner image draws:
run 33745916176 (PR #1811) poured 23.1.0 and died at compile on a new
Clang 23 diagnostic (-Wunused-but-set-global under -Werror); the run an
hour earlier poured 22.1.8 and built fine. Same code, two verdicts.
Pin the formula (and the `brew --prefix` lookup) to llvm@22 — the version
every green run of this leg has used so far. Moving to a newer major
becomes a deliberate edit of these two lines instead of an accident of
the runner image.
The diagnostic itself is fixed independently in #2027 so main compiles
under both majors; this change is about determinism of the leg, not the
warning.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The gate waited 90 x 30s = 45 min for CodeQL to finish on the PR head. That
is shorter than CodeQL actually takes on this repository, so the gate has
been failing runs that had not failed.
Measured on PR #1426, head 7b72652a: the CodeQL SAST workflow completed with
conclusion=success at 17:46:05, having started at 15:41:44 -- 124 minutes.
The gate step ran 16:52:58 to 17:38:44 and reported "BLOCKED: CodeQL timeout"
7 minutes and 21 seconds before the scan it was waiting for succeeded.
Two open contributor pull requests are red from exactly this: #1426 and
#1769, both with CodeQL completed=success on their head and every other
check green.
Three further PRs (#1703, #1741, #1742) are also red on codeql-gate alone,
but from a different cause: the CodeQL run on their head is
completed=cancelled, so the gate saw a non-success conclusion and correctly
exited 1 without waiting. This change does not help those and is not
intended to; they need a fresh scan, most likely having been superseded by
concurrency cancel-in-progress in codeql.yml.
300 x 30s = 150 min covers the measured 124 min with margin. The job already
declares timeout-minutes: 240, so the wait still cannot outlive its own job.
No trigger, permission or gating change: codeql-gate blocks exactly what it
blocked before, and a genuine CodeQL failure still exits 1 immediately rather
than waiting out the budget. Only the absence of a verdict waits longer.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dependabot split this bump across two PRs -- #1898 bumps init, #1900 bumps
analyze -- but codeql-action requires every step in a workflow to run the
same version. Either PR alone produces a mismatch and analyze fails with:
Loaded a configuration file for version '4.37.4', but running version '4.37.9'
so neither can be green on its own, and merging either would leave main's
CodeQL job broken for every subsequent PR. Bump both pins in one commit.
Pin verified: cdf488f595d80d6e07e03d4674febd5ab45fa938 is the commit that
the annotated tag v4.37.9 dereferences to in github/codeql-action.
Supersedes #1898 and #1900.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three fixes from the v0.10.7 release incident (2026-08-18):
1. build (and smoke/soak) required 'not failed' instead of explicit success.
failure() does not cover a needed job that TIMED OUT (conclusion
'cancelled'), so lint hitting its 15-min timeout cascaded test into
'skipped' and the pipeline published with the whole test matrix and
asan-soak silently skipped. build now requires lint success plus either
test success or the sanctioned skip_tests input; smoke/soak require build
success explicitly.
2. The tag is inputs.version verbatim: dispatching a bare '0.10.7' published
a release the installers can never resolve (they fetch
releases/download/v<version>/...), and under immutable releases the
mis-named tag cannot be retagged or its name reused. A preflight job now
refuses any non-v-prefixed version before anything runs.
3. Lint's 15-min timeout was one slow-runner day away from cancelling a
normally-5-min job; raised to 30 so only a genuine hang can hit it.
Release-path only (workflow_dispatch); adds one ~5s preflight job; no PR-CI
gating, cost, or trigger changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The v0.10.6 release run failed deterministically at verify:
BLOCKED: expected scan object is missing:
objects/scan-3099e91c...--codebase-memory-mcp.exe
exclude-rescanned-selected-objects.sh (added after v0.10.5, first exercised
by this release) deliberately deletes the selected executables from the
surface-scan directory — their bytes were already scanned as candidates and
re-submitting identical bytes re-rolls a probabilistic classifier — and
writes binaries/virustotal-withheld.tsv. But check-virustotal.sh still
received the pre-withhold scan-set listing all sixteen objects and failed
closed on the first missing file. The rework's two halves never talked.
The gate now accepts an optional VT_WITHHELD manifest (strict parse: v1
marker, the stated reason required, sha256-keyed rows): an expected-set row
whose hash the manifest vouches for is exempt from the on-disk and
action-output contracts, while everything else keeps the strict path.
Fail-closed properties preserved and extended:
- no VT_WITHHELD -> byte-for-byte previous behavior (candidate
stage and dry-run call sites are unaffected;
verified against the original failure)
- withheld object present -> blocked (inconsistent staging)
- hash outside the set -> blocked (spurious withhold)
- everything withheld -> blocked (scan would cover nothing)
- mismatched object name -> blocked
vt-results.tsv keeps its exact shape (scanned objects only) — the release
notes table already uses the candidate results, and the withheld manifest is
now preserved with the rest of the evidence artifacts. release.yml passes
VT_WITHHELD only in the verify stage, right after the withhold step.
Verified offline with a fixture reproducing the release failure verbatim
plus the four negative cases above; the positive case passes staging and
association validation and proceeds to VT polling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The verify pass submitted every extracted object, selected executables included,
on the stated grounds that "VirusTotal is content-addressed, so identical bytes
return the analysis it already holds instead of re-running 70+ engines".
That is measurably false. On v0.10.5 all EIGHT re-submissions produced a NEW
analysis - same VirusTotal file-id, timestamp 47 minutes later:
candidate: file-id=2c00f485... ts=1786795957 (12:12:37Z)
verify : file-id=2c00f485... ts=1786798758 (12:59:18Z)
Re-analysing identical bytes re-rolls a probabilistic classifier, and Microsoft's
ML engine answered differently within that hour, in BOTH directions:
82750cd1 (linux-amd64) microsoft-ml -> clean
6d3c5be6 (darwin-arm64) clean -> microsoft-ml
The published notes are generated from the candidate scan, so v0.10.5 shipped a
table calling linux-amd64 flagged when VirusTotal had it clean, and darwin-arm64
clean when VirusTotal was reporting Trojan:Script/Wacatac.B!ml. Every hash in
that table links to the page that contradicted it. Corrected in place after
publication; this removes the cause.
The second scan proved nothing the first did not. Identity is settled by hash
before this step runs: verify-release-selection.py reconciles every published
container to the selected bytes, and checksums.txt binds the same digests
publicly. A re-scan adds no assurance - only another roll.
What still gets scanned is exactly what the candidate pass never saw: install.sh,
install.ps1, LICENSE, THIRD_PARTY_NOTICES.md, the MCPB manifest.json and the
unpacked UI assets. install.sh and install.ps1 are the highest-consequence
non-executable bytes we publish - users pipe them straight into a shell - and
that coverage is untouched. Measured on the v0.10.5 object set: 16 objects in,
8 withheld, 8 still scanned.
The withheld set is recorded as evidence (cbm-virustotal-withheld-v1) naming each
sha256 and pointing at virustotal-candidate-results.tsv, so the published
evidence still accounts for every shipped object.
Fails closed three ways, each with an actionable message: no object matches a
selected sha (the containers do not carry the recorded bytes), everything is
withheld (the surface scan would be a no-op), or the selection names no shas at
all. The zero-match grep is wrapped rather than left to pipefail, because a
guard that aborts silently is not a guard - found by testing the guards rather
than assuming them.
Also drops 8 VirusTotal submissions per release.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The ui-* archives are byte-identical copies of the canonical ones, published
after verify so they inherit the hash-bound VirusTotal verdicts. They were
absent from checksums.txt, and publish-legacy-aliases.sh documented that as
intentional: "checksums.txt covers the canonical names current installers
request."
That reasoning has a hole. The aliases exist only for 0.9.x updaters (#1538),
and those verify the NAME they asked for. So the alias fixed the 404 and moved
the failure one step later - the updater downloads the archive, cannot find its
name in checksums.txt, and refuses:
warning: codebase-memory-mcp-ui-darwin-arm64.tar.gz not found in checksums.txt
error: refusing to install an unverified download
Reported by AmooAti in #1134. Confirmed on the live v0.10.4 release: eight ui-*
archives published, zero of them listed. Every pre-0.10 user who answered the
old variant chooser with "ui" is hard-blocked from updating by any path.
The same digest is now emitted under the legacy name before the attestation
step, so the attested artifact covers both names. No new bytes and no new scan
surface: an alias is a copy, so its sha256 is by construction the one already
computed.
The rule lives in scripts/ci/append-legacy-alias-checksums.sh rather than inline
in the workflow, because the venue-parity contract requires it: a venue may
provision, plumb artifacts, or call a canonical leg script, and text
transformation is none of those. Keeping it beside publish-legacy-aliases.sh
also puts the two halves of the alias rule in one place, which matters because
they must stay in step - .tar.gz and .zip only, never an already-ui-* name. It
fails closed when it matches nothing, since a name with no asset is as broken as
an asset with no name.
Validated against the real v0.10.4 checksums file: the generated set is exactly
the eight ui-* assets that release published - no phantom names, none missing -
and the empty case exits non-zero.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two things, both found by the dry-run and by sweeping what it exposed.
1. The three-candidate change was incomplete. The dry-run failed at
stage-release-candidates.py, which carries its OWN copy of the transform
validation that the previous commit only fixed in the selector:
stage-release-candidates: candidate linux-amd64/debug-stripped has
invalid transform: 'strip-debug'
Sweeping for that assumption found it in six places, not two:
stage-release-candidates.py, select-release-candidates.py,
verify-release-selection.py, append-vt-notes.sh and two contract-test
fixtures — as hardcoded 16s, `len(TARGETS) * 2`, two-entry VARIANTS tuples
and two-key truth tables. All are now derived from len(VARIANTS).
Field prefixes needed care: the variant NAME keeps its hyphen because it is
the on-disk directory, while the evidence columns use underscores, so
`debug-stripped` reads `debug_stripped_sha256`. Every lookup now goes through
an explicit FIELD_KEY map instead of interpolating the variant directly.
The selection contract test now covers the truth table EXHAUSTIVELY: three
variants x two tolerated classifications is exactly eight combinations, and
there are exactly eight targets, so every case is exercised once.
2. Full-surface VirusTotal scanning is restored. This PR had moved scanning
upstream to the candidates and deleted the post-package pass, which silently
narrowed coverage from everything we ship to executables only. The 42 runtime
files across the 14 containers — install.sh, install.ps1, LICENSE,
THIRD_PARTY_NOTICES.md, the MCPB manifest.json and the unpacked UI assets —
were still extracted, structurally verified and strings-audited, but no
longer scanned at all. install.sh and install.ps1 are the highest-consequence
non-executable bytes we publish; users pipe them straight into a shell.
The verify job scans every extracted object again, under the same policy.
Re-submitting the selected executables alongside them is close to free
because VirusTotal is content-addressed and answers for identical bytes from
its own record — the same property that made the analysis-id equality check
untenable two commits ago.
The gate-chain contract asserted the opposite ("duplicate post-package
VirusTotal path remains"). That assertion is inverted: the pass is required,
and it is not a duplicate, since it covers a strictly larger set. README and
SECURITY.md updated from "archive containers are checksummed rather than
redundantly rescanned" to state the full covered surface.
All five release/VT contract tests pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Release evidence from run 31744302624 shows the tolerated Microsoft `!ml`
verdict is close to a coin flip per byte image rather than a property of the
code. Across the eight targets the stripped and unstripped candidates of the
SAME linker output disagreed on four, and in both directions:
linux-amd64 stripped microsoft-ml unstripped clean
darwin-arm64 stripped clean unstripped microsoft-ml
linux-arm64 stripped clean unstripped microsoft-ml
If the classifier were keying on something intrinsic to our code the siblings
would agree; they do not. So each variant is close to an independent draw, and
5 of 16 candidates drew the flag.
Two draws is not always enough. On that run linux-amd64-portable came back
microsoft-ml on BOTH candidates, leaving no clean binary to ship for that
target. A third independent draw at a ~31% observed per-candidate hit rate takes
the both/all-flagged case from roughly 1-in-10 per target to roughly 1-in-30.
The third candidate is `--strip-debug` (Apple: `-S`): debug information removed,
symbol table kept. Behaviourally identical to the other two — same linker
output, only metadata differs — but a distinct byte image, which is all
VirusTotal needs to scan it as its own file. Verified on the real v0.10.4
candidates: linux-amd64 gives three distinct hashes (294,634,656 /
294,623,208 / 293,746,096 bytes) and darwin-arm64 likewise, with the ad-hoc
signature verifying after strip.
Selection is unchanged in spirit and now ordered: smallest artifact first
(stripped, debug-stripped, unstripped), take the first CLEAN one, and only if
every candidate drew the tolerated verdict ship the smallest flagged one. A
hard verdict on any candidate still blocks the release before selection.
Cost is 24 objects per release instead of 16.
Derivation enforces that all three hashes differ — identical candidates would be
one draw wearing three hats, and the selector would believe it had alternatives
it does not have. Public claims in README, SECURITY.md and docs/index.html
updated from "both stripped and unstripped" to the three candidates.
All three release contract tests pass, including the native derivation test
which exercises the real strip and codesign path on this host.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The previous commit pinned the linux/amd64 asset hash directly, which
verified the asset this job needs but hard-coded the platform: the
uname-based selection was replaced by a fixed filename, so moving the job
to another runner OS or architecture would have needed a code change and
a second pinned hash.
Pin the SHA-256 of the release's own checksums file instead, and verify
whichever asset the runner selects against it. Upstream publishes that
file for the whole release, covering linux, darwin and windows on both
amd64 and arm64, so one pinned value now covers every platform and the
uname-based selection is restored.
Match the asset by exact filename, since a substring match would also
accept the .sbom.json and .sigstore.json lines for the same asset, and
fail closed when an asset is absent from the checksums file rather than
installing it unverified.
Verified end to end against the pinned release: the checksums file
matches its pinned hash; asset selection resolves to a listed asset for
linux/amd64, linux/arm64, darwin/amd64 and darwin/arm64; the linux/amd64
asset verifies and extracts to a statically linked x86-64 ELF; and
appending one byte to the downloaded asset makes verification FAIL, so
the gate binds rather than passing vacuously.
Co-Authored-By: Andrew Hundt <ATHundt@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The publish-mcp-registry job fetched mcp-publisher from the `latest`
release and piped curl directly into tar. Whatever upstream published at
that moment therefore executed inside the job that holds the MCP Registry
publish credential, with no opportunity to verify it first.
Pin the release to v1.8.1, download to a file, verify its SHA-256 against
the checksum published in registry_1.8.1_checksums.txt for that same
release, and only extract once the hash matches. The job runs on
ubuntu-latest, so the linux/amd64 asset replaces the uname-derived
selection.
Reported by Andrew Hundt in #1245.
Verified: fetched the pinned asset (7,339,841 bytes, matching the release
asset size), confirmed its SHA-256 against upstream's checksums file, and
extracted a valid statically linked x86-64 ELF.
Co-Authored-By: Andrew Hundt <ATHundt@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
v0.10.3 published, then failed:
MCPB package '...codebase-memory-mcp-darwin-amd64.mcpb' is not publicly
accessible (status: 404)
publish-mcp-registry and publish-final both needed only publish-registries, so
they ran in PARALLEL — and publish-final is the job that un-drafts the release.
The registry validates every package URL it is handed by fetching it, and a
draft release's assets are not publicly readable. The registry lost the race by
five seconds; the identical URL served 200 once the release went public, and the
job passed on a plain re-run.
The registry now needs publish-final. This keeps the documented intent exactly:
publish-final still does NOT depend on the registry, so a registry-preview
outage can never block shipping — the registry simply runs after the assets it
validates exist to the outside world.
The gate-chain contract now pins BOTH directions, because either one alone is a
bug: the registry must depend on publish-final, and publish-final must never
depend on the registry. Verified by reverting the ordering — the contract goes
red with the exact v0.10.3 failure, and green again with it restored.
This is the second latent defect the .mcpb path produced on its first real
release (after the security-strings audit treating a manifest as a binary).
Both were invisible until a stable release ran; an end-to-end bundle gate is
worth adding before the next one.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every release now carries .mcpb one-click-install bundles alongside the
archives, and the MCP Registry entry lists them with per-file sha256:
- package-release.sh (canonical) builds codebase-memory-mcp-<target>.mcpb
for darwin/windows and the STATIC linux builds — manifest.json + the same
staged (stripped, gated) binary + LICENSE + THIRD_PARTY_NOTICES.md. The
glibc-dynamic linux targets stay archive-only: a dynamic binary defeats
the one-click promise.
- _build.yml / release-draft: bundles flow through provenance attestation,
checksums.txt, cosign signing and the release asset list; checksums.txt
is also preserved as a same-run artifact for the registry job.
- verify: the canonical scan matrix grows to 14 containers; MCPB manifests
are validated (parse, binary server, entry_point member, command binds
the entry point). Bundle binaries dedupe to the archive scan objects, so
the VT gate gains only the three distinct manifest.json files.
- publish-mcp-registry: gen-mcpb-registry-entries.sh appends one mcpb
package entry per bundle (release-asset URL + fileSha256 from the
attested checksums) to server.json before mcp-publisher runs.
Idempotent; a checksums file without bundles is a hard failure.
- contracts: Step 0o pins the bundle shape at its producer on every leg,
Step 0p pins the registry entries against the live server.json, and the
extractor contract covers the 14-container matrix incl. broken-manifest
fail-closed cases. The linux test image gains zip for the packager.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Five field reports in the 24 hours after v0.10.0 all pointed at the same thing:
gates that were right in principle refused real, ordinary setups, and then
failed to say why. Per the consolidated strictness decision, each gate keeps the
protection that matters and drops the part that was refusing legitimate users —
and every refusal now names what it refused and how to proceed.
**Daemon image gate: npx and every ephemeral install path (#1539, #1383).**
The admission check treated "the peer's image hashes differently" and "the
peer's image cannot be examined at all" as one failure. The second is what
`npx codebase-memory-mcp` always produces (ephemeral cache path,
unfingerprintable), so every npx-invoked client was rejected — and, because the
client never reported it, agents saw a transport that closed mid-handshake with
zero bytes on stdout. Reported by @wassolles with the admission path already
read and the fix space mapped.
An unverifiable image is now admitted: the rendezvous HELLO immediately above it
has already proven semantic version, build fingerprint, and protocol/store/
feature ABI, and the image check was trading that real proof for an unavailable
one. It logs daemon.client_image_unverifiable_admitted so the weaker check is
never invisible. A fingerprint MISMATCH — the tamper case the gate exists for —
still rejects hard. Separate test seams keep the two modes testable apart.
**Client bootstrap failures are no longer silent (#1539).**
An MCP client that cannot reach the daemon now emits a JSON-RPC error on stdout
naming the reason, plus the same text on stderr. Previously the reason sat in
bootstrap_result.message and the process exited having written nothing at all.
**POSIX activation: group-writable ancestors (#1535, discussion #1526).**
activation_directory_secure required no group or other write bit on the install
directory AND every ancestor. WSL2 ships ~ and ~/.local at 0775, as do several
distro skeletons and any site using a shared primary group, so install.sh failed
for a large fraction of Linux users — reporting a policy refusal as "activation
transaction I/O failed", which sent reporters after disk errors and filesystem
types. Root-caused by @AmirF194 in a clean ubuntu container; @shochdoerfer and
@iandol confirmed independently.
World-writable ancestors are still refused (any local user could swap a path
component mid-transaction). Group-writable ancestors are now warned about and
admitted. The LEAF directory stays strictly owner-private — that is where the
binary is published, and group write there would let another account replace the
executable between validation and exec. Refusals now name the directory, its
mode, and which rule refused.
**The obsolete ui/standard chooser (#1538, from discussion #1526).**
v0.10.0 consolidated to one archive per platform with the UI always embedded,
but `update` still offered a variant choice: "ui" could only 404, and "standard"
quietly WAS the UI build. Reported by @iandol upgrading 0.9.0 -> 0.10.0. The
chooser, its --standard/--ui flags, and the ui- URL plumbing are removed, along
with the CBM_VARIANT=ui remnant in the npm installer.
Already-released 0.9.x binaries cannot be fixed retroactively, so the release
workflow now publishes byte-identical ui-*-named alias assets — their updaters
work again with no user action. The aliases are uploaded AFTER the VirusTotal
gate: they are the same bytes as archives it already cleared, and uploading them
earlier would duplicate every object in the scan set and the provenance manifest.
**macOS install noise and attribution (#1537).**
install.sh silenced the "No such xattr: com.apple.quarantine" line, which is
what happens when a curl-downloaded archive carries no quarantine attribute —
harmless, and it became the title of a bug report about an unrelated failure.
The session-stop refusal now points at `daemon status` to list the client
processes actually holding the daemon, instead of asserting sessions exist and
leaving the reader to guess. Reported by @listepo.
**Riders.** hatchling is pinned in pkg/pypi (an unpinned backend resolved fresh
inside `python -m build` is what emitted Metadata-Version 2.5 and broke the
v0.10.1 publish); SECURITY.md's supported-versions table moves to 0.10.x.
Tests: separate seams for unverifiable vs mismatched peer images with a test per
outcome; activation refusal must name directory + mode + rule; a group-writable
ancestor must stage successfully. The update tests drop the flag that no longer
exists. Verified against each reporter's environment shape.
**Open security alerts (all three, OSSF Scorecard).**
- HIGH, binary artifact: an 8.8 MB compiled Go ELF wrapper had been committed at
pkg/go/codebase-memory-mcp by accident. Removed, and both it and its .exe
sibling are gitignored so `go build` in that directory cannot repeat it.
- HIGH, GHSA-2v37-7h3g-55p8: nanoid < 3.3.17 loops forever when a custom
generator is called with size 0. It reaches us transitively (postcss -> vite),
so it is pinned through the existing graph-ui overrides block rather than
promoted to a direct dependency; the lockfile resolves 3.3.18.
- MEDIUM, unpinned pip command: the publish step installed build/twine by
version only, leaving the whole transitive graph resolved at run time.
pkg/pypi/requirements-publish.txt now hash-pins the complete toolchain (316
hashes), generated on a linux/amd64 python:3.12 image so the wheels match what
ubuntu-latest resolves, and the step runs pip with --require-hashes. Verified
by installing from it in that same image.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The v0.10.1 publish failed twice, for two stacked defects:
1. `python -m build` resolves the UNPINNED hatchling backend fresh inside its
isolated build env, and current hatchling emits Metadata-Version 2.5 —
which the pinned twine==6.2.0 rejects as "'2.5' is not a valid metadata
version". Deterministic, and a time bomb: v0.10.0 published cleanly days
ago on the same pins. Verified locally on identical artifacts: twine 6.2.0
rejects, twine 7.0.0 passes. The pin moves to 7.0.0 and `twine check`
now runs at build time so a metadata regression fails BEFORE upload.
2. The job was not idempotent, breaking its own design comment ("if publish
fails, the release stays in draft so we can re-run"). Attempt 1 published
npm 0.10.1 and then died at twine; the re-run 403'd on its own success
("cannot publish over the previously published versions") and the release
wedged in draft. npm publish now skips when the exact version already
exists on the registry, and twine uploads with --skip-existing — both
registries treat immutable prior success as done, not as a collision.
Workflow-only diff. Unblocks re-dispatching the wedged v0.10.1 release.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The first full dry run after #1508 failed in all three smoke-linux-portable
legs: _smoke.yml still expanded a variant matrix and extracted
codebase-memory-mcp-ui-<os>-<arch>.tar.gz — a name the build no longer
produces. PR CI never sees this job (pr.yml calls the smoke wrappers directly),
so the miss only surfaced in the dry-run/release path this workflow serves.
The matrix loses its variant dimension, all three legs extract the unsuffixed
archive, and the positional/SMOKE_VARIANT plumbing is replaced by
SMOKE_REQUIRE_UI=1: these legs smoke the SHIPPED artifact, so a binary serving
no embedded UI is a defect here, exactly like scripts/ci/smoke-artifact.sh.
Verified locally: venue-parity and smoke-fixture contracts pass; YAML parses.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the collapse to a single shipped composition and replaces the
zero-tolerance VirusTotal gate with a narrow, disclosed policy.
Packaging and installers
- package-release.sh loses --variant; archives are codebase-memory-mcp-<os>-<arch>
with exactly four members. install.sh/install.ps1 lose --ui/--standard.
- The extractor drops CBMUIPK pack parsing and --archive-scope; its scan-set and
association manifests (which the gate depends on) are unchanged otherwise.
- npm/PyPI/Go wrappers: the runtime "set" is one file again. The Windows lock
and race fixes from #1495/#1496 are kept; only multi-file set membership goes.
This also fixes `pip install` on Windows, which rejected the fifth archive
member against a hardcoded four-name allowlist.
- The wrappers' post-download probe moves from --verify-runtime-assets (removed)
to --version, which proves the same thing: the binary executes.
VirusTotal gate
- Exactly ONE detection is tolerated, and only when the engine is Microsoft AND
the label ends in `!ml`. Two or more engines, any non-`!ml` label, any other
vendor, any suspicious verdict and every infrastructure error still block.
- A tolerated object prints TOLERATED:, never OK:, and its counts are recorded
in vt-results.tsv exactly as a blocked one would be.
- append-vt-notes.sh mirrors the policy. It previously hard-failed on any
malicious count, so loosening only the gate would have passed the scan and
then died at note publication. The notes now DISCLOSE a tolerated detection
and link to SECURITY.md rather than claiming "0 malicious" for everything.
Rationale for the tolerance is in the gate itself: the verdict is not a property
of our bytes. It inverts across architectures and link modes, moves between
sibling artifacts of one build, and lands in different variant buckets for the
same source. The same `!ml` family hits llama.cpp, GitHub's own `gh`, Microsoft's
own Go toolchain and Anthropic's Claude installer.
The zero-tolerance contract becomes test_vt_gate_policy_contract.sh, asserting
the full matrix: 1x Microsoft !ml passes and reports TOLERATED; a Microsoft
signature label, a non-Microsoft engine, two engines, a suspicious verdict and
every malformed-response case still block. Its tripwire is narrowed to the
reverted endpoint-verification mechanism rather than the words "false positive",
so it no longer fires on a deliberate in-gate policy branch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Externalizing the integration templates (#1492/#1493) and the UI bundle
(#1501/#1503) was done to reduce the Microsoft `Wacatac.B!ml` surface. It did
not work: across dry runs the flagged artifact count stayed at ~3 and the
detections merely moved between artifacts.
Dissection of run 31286803592 shows there is no structural cause to fix. The
verdicts split across every axis at once — linux-amd64 (dynamic) flagged while
linux-amd64-portable (static) is clean, but linux-arm64 (dynamic) clean while
linux-arm64-portable (static) is flagged. The two macOS binaries have identical
segment structure and split clean/flagged. Siblings from one build landed in
different variant buckets (.B vs .C). Entropy is low everywhere
(code_vectors.bin 4.166, grammar tables 3.464 bits/byte, against 7.5-8.0 for
packed payloads), so the packed-payload hypothesis is excluded too.
So the complexity bought nothing, and installation goes back to being
self-contained: one binary that carries its own UI and agent integration
templates, with no adjacent data file that has to resolve before `install`
works. Only the UI-capable composition ships from now on, under the historical
unsuffixed archive name.
Removed: src/ui/asset_pack.{c,h}, asset_pack_stub.c, asset_manifest_stub.c,
scripts/pack-ui-assets.mjs, src/cli/integration_assets.{c,h},
assets/cbm-integrations.json, scripts/gen-integrations-hash.sh, the
--verify-runtime-assets probe (nothing adjacent left to verify), and the
composition gates A6/A7 whose property is now deliberately inverted.
Restored: scripts/embed-frontend.sh, src/ui/embedded_{assets.h,stub.c}, the
compiled-in hook/adapter template bodies, and the embed/EMBED_OBJS build path.
Kept from the reverted commits, re-applied by hand where a wholesale file
restore would have dropped them:
- cbm_module_path_utf8() in both self-path sites. GetModuleFileNameA renders
through the ANSI code page and mangles non-ASCII install paths.
- the /__cbm/ui-readiness HMAC proof, secure_random and cbm_hmac_sha256, so
`daemon start --open` still waits for a genuine CBM listener.
- X-Content-Type-Options: nosniff on served assets.
- the MinGW noexecstack gate, -lbcrypt, and the cppcheck/zip CI fixes.
Archives are now codebase-memory-mcp-<os>-<arch>[-portable] with exactly four
members (binary, LICENSE, installer, THIRD_PARTY_NOTICES.md). That restores the
names every static package manifest already points at — aur, chocolatey,
homebrew, scoop, winget and glama were all broken by the -ui- rename.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel