Fix-forward after rebasing onto main (545 commits, 26 conflicted-file
resolutions). Where main had shipped a behaviour in the meantime, main
wins; the PR keeps its output contract on top of it:
- search_code: the scan runs under main's supervised, deadline- and
output-bounded runner and feeds the branch's streaming classifier from
the runner's output file; the POSIX wrappers map grep's no-match to 0,
so every non-zero exit fails closed (no exit-1 special case).
- trace_path: strategy/confidence precede args in header and rows (#1542),
in both the tree table and the json legs.
- cypher: relationship expansion is never capped (a cap falsified
aggregates) and unlabeled MATCH scans every candidate; `truncated` is
raised only by a real ceiling — a clamped hop range is probed one hop
beyond the cap and reports truncated only when a candidate exists there.
The plain BFS and the trail BFS both probe one row past max_results so
saturation is observable instead of silent.
- search_code full mode and the final json use main's U+FFFD sanitizer
instead of '?' replacement.
- search_graph json keeps an empty `groups` array for semantic-only calls;
list_projects json echoes offset/limit next to next_offset and accepts
main's include_details spelling for the stats projection.
- discovery budget: main added get_file_outline, compare_graphs, manage_adr
set_sections and the debug/include_details parameters; their descriptions
are trimmed to the lean style and the ceiling moves from 15 to 18 KiB.
Tests that asserted a JSON default now request format:"json"; the
fail-closed search test uses an unreadable regular file (a missing or
non-regular indexed path is deliberately not a scan operand on main); the
non-regular fixture gives its real sources File nodes, as every indexed
project has; list_projects total counts listable projects (the 0-byte
ghost is never a row, so it is not a page either).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The rebase onto main brought the Bash-enabled PreToolUse matcher
(Grep|Glob|Bash), which the exec-form tests still counted as Grep|Glob.
The Windows lifecycle test seeded the .cmd shell spelling as the converged
identity; the shell-free registration converges every owned shell spelling
onto one exec-form entry per matcher instead. The test now asserts that,
plus their removal on uninstall, and hands cbm_install_agent_configs the
'/'-spelled binary path cbm_cmd_install registers (cbm_mkdtemp returns a
native '\' temp root on Windows, which uninstall never derives).
Smoke 8aq proved Claude's gate registration by grepping the shim name out
of settings.json; the exec form carries no shim name, so it now checks the
binary-plus-'hook-augment' PreToolUse hook. 9d's shim-name marker would
otherwise pass vacuously after uninstall; the 'hook-augment' argument is an
owned marker there too.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Step 0o (MCPB bundle contract) and Step 0q (release candidate derivation)
hand the composition gate a stub compiled straight from a .c file with the
compiler's defaults. That stub exists to exercise packaging, format detection
and the needle scans; it is not linked with -z now and never ships. A1d
measured eager binding on it anyway, so on a toolchain that binds lazily by
default (test-diag: clang-22) Step 0o died deterministically with
A1d-bind-now stage/codebase-memory-mcp: GOT section(s) .got.plt fall
outside PT_GNU_RELRO [15824,16384) — they stay WRITABLE after relocation
The property A1d protects is a property of the release link (-z now in
Makefile.cbm), so the assertion is scoped to it: a caller that feeds the gate
a packaging fixture declares so with CBM_COMPOSITION_FIXTURE=1 and A1d is
reported n/a — printed on every run as a named exemption, never a silent
skip. Every other assertion still runs on the fixture unchanged, and release
derivation never sets the variable.
Verified in the Linux arm64 container (Ubuntu 24.04, gcc 13.3): a stub
linked -z lazy reproduces the CI failure text on the gate without the
variable and reports n/a with it; both contracts pass under a CC wrapper
that links lazily (the shape that was red), and unchanged on macOS.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
When a suite leader has already exited, taskkill /T cannot walk its tree,
so run-test-wave.py proves cleanup by asking Windows whether anything is
still parented to the dead PID. That probe was timed with --kill-grace,
which the parallel gate passes as 1s. A cold powershell.exe + CIM start
routinely costs more than that on a runner, so TimeoutExpired -- a fact
about interpreter start-up, not about the process tree -- was read as
"assume the worst" and failed an already-clean shard with exit 2.
Give the probe its own stable-state budget (15s) plus one retry, keeping
the fail-closed semantics: an exhausted or failed probe still counts as
"cannot prove absence". The refusal now names which of the two happened,
an unfinished probe or a counted set of live descendants, instead of
reporting both as a leaked tree.
The contract test pins this structurally -- no sleeps, no timing
thresholds: the probe must be bounded by the module constant rather than
by the kill-grace argument, must retry a timed-out probe, must never read
an unusable probe as absence, and must report the two failures
distinctly. Its Windows leader-exit case now waits on the scheduler's
refusal rather than on a deadline shorter than the new budget.
Refs #1741
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The 8al-node probe imports the generated Pi extension in bare Node with an
empty node_modules. That was a sound stand-in for the Pi host only while
the extension imported nothing but node: builtins; the first package
import (`typebox`, Pi's schema library for tool parameters — a real Pi
dependency, verified against pi-coding-agent 0.84.4 on #1809) can never
resolve there, so the probe fails on a correct extension and would pass a
wrong one just as readily.
Give the probe what the host gives the extension: a Proxy-backed
`typebox` stub in the probe's node_modules (any Type.X(...) returns a
descriptor, so the probe never tracks which helpers the emitter uses — it
only exercises the tool lifecycle), plus an explicit allowlist: every
import specifier in the generated file must be a node: builtin or a
package on PI_HOST_PACKAGES, otherwise the probe fails naming the
specifier. That turns the guarantee the empty node_modules gave by
accident (no surprise dependency) into an asserted contract.
Verified locally by running only the 8al-node block against extensions
generated by two binaries:
main@5802ccdd extension, old probe: OK; new probe: OK
#1809 extension, old probe: ERR_MODULE_NOT_FOUND typebox (the CI red)
#1809 extension, new probe: OK
#1809 extension + an injected 'lodash' import, new probe:
FAIL 8al-node: ... packages the Pi host does not provide: lodash
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The vendoring table claims our copy of a grammar matches the pinned
upstream commit. A local patch makes that claim false unless the patch
is written down, so a future re-vendor would drop the fix without
anyone noticing.
The row covers both changed lines, because both are the same defect in
the same file: OP_SYMBOL_SUPPRESSOR at line 131 and eat_operators at
line 514.
The row also names the two upstream commits that already carry these
changes, and the table intro gains one clause for the case. Upstream
fixed eat_operators in fb63a7004f07 on 2026-04-06 (their issue #558)
and the OP_SYMBOL_SUPPRESSOR entry in 6ab8d1d74ebd on 2026-08-10. Our
pin 8abb3e8b3325 is from 2026-03-20 and predates both, so these are
backports rather than local inventions.
That distinction changes what a re-vendor should do. The table's
instruction is "must re-apply these", which is right for the other four
rows -- upstream never took those. For this row it would mean
hand-applying a change the newer file already has. The row now says to
delete it instead, and the intro allows for that case.
MANIFEST.md is itself listed in scripts/vendored-checksums.txt, so its
own recorded checksum moves with the edit.
Refs #1892
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
scanner.c line 131 shifts a literal 1UL by 32 to build the entry that
suppresses the bang of a try!. unsigned long is 64 bits on Linux and
macOS but 32 bits on Windows, so there the shift count equals the width
of the type, which is undefined.
1ULL is 64 bits on every target this project builds for.
Nothing reports it today. The expression is a compile-time constant, so
no sanitizer sees it run, and Makefile.cbm:719 builds vendored grammars
with -w, which switches off -Wshift-count-overflow.
Upstream tree-sitter-swift made the same change in 6ab8d1d74ebd, after
the commit this grammar is pinned to.
Refs #1892
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
The Swift scanner keeps a 64-bit mask of the symbols that suppress a
match -- the rule that stops `try!` emitting its `!` as a token of its
own. It tests one bit per candidate:
uint64_t suppressing_symbols = OP_SYMBOL_SUPPRESSOR[full_match];
for (uint64_t suppressor = 0; suppressor < TOKEN_COUNT; suppressor++) {
if (!(suppressing_symbols & 1 << suppressor)) {
The mask is uint64_t but the literal `1` is an int, so the shift is an
int shift. TOKEN_COUNT is larger than 32, so once suppressor reaches 31
the shift runs past the width of the type. That is undefined behavior,
and every bit above 31 is tested against a value the standard does not
define.
Nothing caught it because nothing in the tree reached the suppressor
path. Any Swift force-unwrap does: `cached!` is enough.
UBSan reports it as:
scanner.c:514:47: runtime error: left shift of 1 by 31 places
cannot be represented in type 'int'
`1ULL` makes the literal as wide as the mask it is tested against.
The new test in tests/test_extraction.c cannot go red on its own. The
normal test build prints the UBSan message and carries on, which is why
this survived. The Windows CLANGARM64 leg runs UBSan in trap mode, and
there the same shift is an illegal-instruction crash -- so the test
exists to make sure that leg keeps parsing a force-unwrap at all.
scripts/vendored-checksums.txt records the new hash for the one changed
file, as scripts/security-vendored.sh --update writes it. Layer 8 of the
security gate compares vendored content against that manifest, so the
edit and its recorded hash belong in the same commit.
Found while adding Swift URL extraction in #1892 / #1976, and split out
of that PR so the vendored change can be reviewed on its own.
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
Refresh the complete generated Perl grammar bundle from upstream v1.2.1 at c3e17b31179bf8f658c9f37c7a3ea6a202212d5a, raising it from ABI 14 to supported ABI 15. Preserve the existing license and record the source and per-file provenance.
Add deterministic coverage for a valid format followed by a named sub, while retaining a malformed-source control.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Adds -Wl,-z,relro and -Wl,-z,now to ELF_HARDENING_FLAGS, and asserts the
resulting property on the produced artifact rather than on the compiler's
willingness to accept a flag.
The shipped Linux binaries measurably lacked the property. Built the way
.github/workflows/_build.yml builds them (STATIC=1, gcc 13.3 / GNU ld 2.42,
Ubuntu 24.04 aarch64), PT_GNU_RELRO ends at 0x11DA0000 in both cases:
without -z now: .got 0x11d9ec20+0x13c8, .got.plt 0x11d9ffe8+0x40
-> .got.plt ends at 0x11DA0028, 40 bytes PAST the window.
Those GOT slots stayed writable for the whole process
lifetime, in every Linux binary shipped to date.
with -z now: .got.plt folds into .got, the whole GOT sits inside the
window and is re-mapped read-only after startup.
Both flags are free at runtime for a static binary -- everything resolves at
link time, so there is no lazy binding left to pay for. -z relro is already
this toolchain's default and changes nothing here; it is named anyway so the
property stops depending on one distro's spec file, since the musl/portable
and glibc-floor images are different toolchains and a default is not a
guarantee.
A1c/A1d in scripts/ci/check-binary-composition.sh assert the OUTCOME on the
binary via readelf, for the same reason A1 exists beside the .note.GNU-stack
annotation: a flag the linker accepts is not evidence the artifact gained
anything. PR #1138 demonstrated that directly -- its -pie was silently
discarded under -static, and its -D_FORTIFY_SOURCE=2 silently DOWNGRADED
Ubuntu's default of 3, both while every compile probe passed.
The readers use awk rather than a short-circuiting reader on purpose: a
reader that exits at the first match leaves the upstream taking EPIPE, and
under `set -o pipefail` the satisfied case is reported as the failing one.
Deliberately NOT taken from #1138: -D_FORTIFY_SOURCE=2 (a downgrade on the
release platform), -fPIE/-pie (inert under STATIC=1), -fstack-protector-strong
(real but unmeasured cost, and #1138 injected it into SQLite, tree-sitter and
mimalloc -- the indexing hot loop), and the $(shell) probe machinery (~10
compiler spawns per Makefile parse, making the binary a function of the
builder's toolchain).
Co-authored-by: PR9000 <119280965+PR9000@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Seven packaging surfaces declared 0.8.1 while v0.10.8 was the shipped
release — nine releases of drift in metadata published to package
registries, and the third version-metadata drift this month.
pkg/go was not cosmetic: it hardcodes its version into the release URL
it downloads from, so `go install` fetched v0.8.1 binaries.
Chocolatey had to move as a unit. The nuspec is version-only but
chocolateyInstall.ps1 pins the asset sha256, so bumping the nuspec alone
would ship a package declaring 0.10.8 while installing 0.8.1. The new
checksum is the v0.10.8 windows-amd64.zip digest, verified by downloading
the published asset and hashing it, not merely copied from checksums.txt.
The gate (tests/test_version_metadata_contract.sh, scripts/test.sh Step
0x) checks against the newest stable v-prefixed tag, because nothing in
the tree declares the version: CBM_VERSION is injected at build time and
the release is dispatched with a version input. That absence is exactly
why these files drifted. Mutual consistency across the surfaces needs no
git and always runs, so a shallow clone degrades the gate loudly instead
of passing silently.
Homebrew (0.10.3), Scoop and AUR (0.8.1) are left pinned: each embeds
per-release asset checksums, so bumping them is a distribution chore
rather than a metadata edit. They are still gated — internally
consistent, and never naming a release that does not exist — and each
carries a recorded reason for being frozen.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The line read `7x ABI-13, 86x ABI-14, 65x ABI-15`, which sums to 158 against
161 vendored grammars. Counting `#define LANGUAGE_VERSION` in each parser.c
gives 9 / 79 / 73 = 161, so the line was already wrong before Chialisp; bumping
the ABI-14 bucket by one, as the Chialisp change did, would have carried a
four-off error forward under a fresh timestamp.
Recount with Chialisp included: 9 / 80 / 73 = 162, which matches both the
grammar-directory count and the language-count contract. The regeneration
command is now recorded next to the numbers so the next person does not have to
rediscover it:
grep -h '#define LANGUAGE_VERSION' internal/cbm/vendored/grammars/*/parser.c \
| sort | uniq -c
Vendored-integrity gate and the language-count contract both re-run green.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Chialisp (.clsp/.clib/.clinc) is the Chia smart-coin language. The only
public grammar, Quexington/tree-sitter-chialisp, cannot parse it: on the five
reference files from chia-blockchain@main it produced 3-34 ERROR nodes each.
Three defects account for all of them, and each desynchronised the rest of the
file rather than losing one node:
* comments were `/;.*\r\n/` -- CRLF-only, while real files are LF;
* `(include foo.clib)` was rejected because of the dot in the filename;
* `(defconstant NAME <expr>)` accepted only a primitive after the name.
So the grammar is written here instead, in tools/tree-sitter-chialisp/, and
files as first-party: its vendored LICENSE is the project's own, byte-identical
to the repository root, carrying no third-party copyright.
It is deliberately GENERIC -- source_file / list / symbol / string / number /
hex / dot / comment, modelling the clvm_tools reader rather than the Chialisp
form vocabulary. `mod`, `defun`, `defconstant` and `include` are ordinary head
symbols; which lists are definitions is decided in extract_defs.c. A dialect
that adds a form (cl-21 -> cl-23 -> cl-26 each added several) therefore needs
no regenerated parser. Acceptance: all five reference files parse with zero
ERROR and zero MISSING nodes; the three defects above are pinned as tests.
Extraction, and the four decisions in it:
* `Constant` is admitted to the cross-file name registry
(cbm_label_is_registry_symbol). Chialisp's condition codes live as
`(defconstant ...)` in a shared .clib and are referenced by name from every
puzzle; labelled but unregistered they are seeded nowhere, and roughly half
of a Chialisp graph would be present as nodes yet never the target of an
edge. All three registry-seeding sites route through that one predicate, so
they cannot diverge. No other language emits `Constant`, so the blast radius
is exactly Chialisp. Pinned by a two-file graph test that goes red when the
label is removed from the predicate.
* `export` / `namespace` are NOT definition heads. `(export foo)` names a
function already defined in the same file, so treating it as a def head
mints a second node for one symbol. They stay in the not-a-call filter.
* A parameter list is a `list` too, but it binds rather than invokes. Without
that gate `(defun check_conditions (HEIGHTLOCK conditions) ...)` mints a
phantom call to HEIGHTLOCK, and real Chialisp is mostly such definitions.
* `(include *standard-cl-26*)` is a dialect directive, not a file dependency;
recording it would invent a dependency on a file that does not exist.
Three helpers that the defs, calls and unified walks all need -- the quote-data
gate, comment-skipping named-child access, and the Chialisp def-head set -- live
in helpers.c rather than being copied per translation unit. They previously
existed as byte-identical duplicates that had already drifted: extract_defs.c
skipped comments when reading a def head while its mirror in extract_unified.c
did not, so a comment between a head and its name desynchronised defs from call
scope. Both now use the same accessor.
Ledger: 161 -> 162 grammars across the seven published surfaces, ABI-14 86 -> 87,
first-party 13 -> 14, CBM_LANG_COUNT 164 -> 165.
Distilled from PR #919 by darkverbito, whose extraction logic -- the CLVM
not-a-call vocabulary, the quoted-data gate, the filename-derived `mod` name and
the include/embed import handling -- is kept and is the substance of this change.
The grammar that PR vendored is not used.
Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Seven surfaces disagreed with reality and with each other: README.md,
docs/index.html, docs/llms.txt and pkg/npm/README.md said 158, while
server.json, the chocolatey nuspec and scripts/package-release.sh said 159.
161 grammars are vendored.
21 individual claims corrected. The count is now gated by
tests/test_language_count_contract.sh, so the next language to land cannot
ship a stale figure.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The count was hand-maintained on seven separate surfaces and had drifted apart
from reality AND from itself: README, docs/index.html, docs/llms.txt and the npm
README said 158, while server.json, the chocolatey nuspec and package-release.sh
said 159.
Derived from the vendored grammar directories rather than CBM_LANG_COUNT,
because the registry contains entries with no parser at all: CBM_LANG_NIM
(grammar removed 2026-06-12) and CBM_LANG_OBJECTSCRIPT_EXPORT have no grammar
directory, no tree_sitter function and no MANIFEST row, yet files still route to
them by extension. Publishing CBM_LANG_COUNT would claim languages we do not
parse. A vendored grammar directory demonstrably exists and is checksum-gated.
Accepted trade-off, stated in the test: this UNDERCOUNTS languages that share
one grammar (TypeScript/TSX, objectscript_udl/_routine). We publish the number
we can prove rather than the larger number we cannot.
The contract binds four things so it cannot go quietly vacuous:
- the grammar directory count is sanity-bounded, so a layout change fails loudly
instead of making every check below it trivially true
- MANIFEST.md's declared figure must already agree with the directory count
- every registered surface must state exactly that number, and a registered
surface that states NO number fails (the wording changed, or it should be
deregistered)
- any UNREGISTERED file stating a language count fails, so an eighth surface
cannot reintroduce the drift this exists to prevent
Files that legitimately say "<N> languages" about something else — benchmark
corpus subsets, probe scopes, the version-pinned winget manifests — are listed
with what their number actually counts, so no exemption is unexplained.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Vendor AndreasMaierDe/tree-sitter-plsql @ 28aebef209be (MIT, ABI 14, no
external scanner) and wire CBM_LANG_PLSQL end to end:
- extraction onto existing labels only: package spec/body -> Class,
procedure/function (create/definition/declaration) -> Function,
file -> Module; callee extraction for ref_call incl. the
package-qualified pkg.name form; identifier usage vocabulary with the
upstream 'parameter' kind (a ref_call ARGUMENT wrapper) carved out of
the whole-binding rule
- discover: .pks/.pkb/.pck/.pls/.plb/.plsql/.fnc/.trg/.bdy/.tps/.tpb
(.sql stays SQL and .prc stays FORM, both asserted negatively)
- ledgers: CALL_WITHOUT_REFERENCE_VOCAB(PLSQL), call-node manifest
ref_call row (historical total 219 -> 220), call-argument matrix B row
- tests: extraction triple (package body + package-qualified call,
standalone function, honest negative pin of the upstream
CREATE TYPE ... AS OBJECT ERROR-node limitation), grammar-regression
case, label golden Class:1,Function:1,Module:1, MUST_EXTRACT_DEFS,
extension mapping, bug-repro board battery row
- MANIFEST.md (community-niche verdict, local include-quote patch row),
THIRD_PARTY.md section, refreshed vendored checksums
PL/SQL support was contributed by Oguz (@ouzsrcm) in #1033; re-vendored
from upstream per vendoring policy with his language wiring distilled on
top.
Co-authored-by: Oğuz <oguzhan.saricam@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Grok Build (xAI's grok CLI/TUI) already reached codebase-memory-mcp through
its Claude/Cursor compat layer, but a Grok-only user got nothing. Detect
$GROK_HOME (default ~/.grok) or a grok CLI and install:
- an owned [mcp_servers.codebase-memory-mcp] managed block in config.toml;
a same-name foreign table is left byte-identical because a second header
would make Grok reject the whole file, while the known owned shape written
by `grok mcp add` is adopted. No env forwarding: grok 1.0.5 passes the
full parent environment to stdio servers (verified), unlike Codex (#1562).
- an owned rules file under $GROK_HOME/rules/ (always scanned, all projects)
- the shared skill under $GROK_HOME/skills/
- Scout/Verify/Auditor agents with named-server mcpInheritance and the tier
allowlist spelled out as the server__tool ids Grok's search_tool/use_tool
dispatcher accepts (inheritance filters per server, never per tool)
- hooks withheld: Grok's passive events discard stdout and PreToolUse honors
only deny/rewrite decisions, so there is no context-output contract.
Verified against grok 1.0.5: inspect discovers the server, agents, skill and
rules; mcp doctor handshakes and discovers 15 tools. --clients=grok token,
README/npm/llms/landing-page matrices (43 -> 44 surfaces), and smoke coverage.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Layer 8 (scripts/security-vendored.sh) failed because the seven new arkts
files were untracked in the checksum baseline. Regenerated with --update
after reviewing the diff: the only additions are the arkts grammar's own
files, plus MANIFEST.md's hash changing because the preceding commit edited
it. Nothing else was re-blessed.
Verified independently rather than taken on trust:
arkts/_common_scanner.h sha256 == typescript/_common_scanner.h sha256
(da66ef2bd14a...), i.e. the external scanner body is the already-reviewed
typescript one, and arkts/scanner.c is a 21-line symbol-rename trampoline
that allocates nothing and delegates scan straight through.
All three security-static layers now pass locally: static allow-list, UI
audit, and vendored integrity -- the last also reporting no subprocess,
network, or dangerous calls anywhere in vendored grammar code.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The vendored LICENSE carried tree-sitter-typescript's MIT text plus an
appended derivative notice. That is not what MIT asks of a derivative -- it
asks that upstream's notice travel with the code -- and the appended text
meant the provenance audit could not byte-verify the file against upstream,
so arkts landed as NO-MANIFEST-ENTRY and failed license-gate.
The vendored LICENSE is now byte-identical to upstream's, and arkts is
registered in the audit's FORKS map, which is defined for exactly this case:
a self-maintained fork whose vendored LICENSE must match its upstream. That
is a routing entry, not an exemption -- check_upstream still byte-verifies
against the real upstream repository, so the fact is checked rather than
asserted. Audit now reports arkts as IDENTICAL.
Our copyright for the ArkTS additions is not lost, it moves to where our
SOURCE is: the grammar.js header (which now states all three copyrights
explicitly) and THIRD_PARTY.md, which ships inside the release archives.
The edit to scripts/audit-license-provenance.py is a maintainer-only
enforcement-surface change, made deliberately and stated here rather than
slipped in; a contribution may never supply its own audit entry.
No generated parser or source changes -- comments, licence text, manifest
prose and one routing entry only.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Follow-up to the PR #1105 review: the unit test proves the config predicate but
would still pass even if the daemon-side gate were removed. Add a process-level
regression that drives the real binary and proves the gate actually prevents the
watcher from being built, started or registered when disabled.
- tests/test_watcher_disabled.sh (wired into scripts/test.sh as Step 5e) drives
the binary against an isolated CBM_CACHE_DIR + tiny git fixture. With
watcher_enabled=false it asserts: watcher.disabled reason=config emitted,
watcher.start absent, no watcher.watch registration, auto_index still runs, and
manual index_repository is served. A watcher_enabled=true positive control
asserts watcher.start + watcher.watch appear, so the test fails if the gate is
removed (verified by neutering the gate in host_state_prepare and observing the
failure).
No assertion is decided by a timeout. The watcher lives in the daemon, which
logs to $CBM_CACHE_DIR/logs/cbm-daemon.log in a fixed order: watcher.disabled
(host_state_prepare) -> daemon.start -> the thread's own watcher.start -> the
join -> daemon.stop. Each run therefore retires its daemon and waits for that
CLOSED lifecycle before asserting, so absence is a fact rather than a race.
Every wait is a bounded poll on an asserted state following the soak harness's
wait_for_daemon_stop doctrine, and an exhausted budget fails loudly with the
reason instead of passing quietly; session stdin is held open through a FIFO
until the last JSON-RPC response lands, not for an interval.
Registration is asynchronous and unreachable until the project DB exists
(application_refresh_watch_locked requires application_regular_db_exists), so a
closed lifecycle alone would not carry the negative — the session could simply
have ended first. The disabled+auto_index run waits for that DB, i.e.
registration's own precondition; the enabled control waits for watcher.watch
itself, so its absence in the disabled runs is evidence rather than noise.
Honors CBM_TEST_BINARY like tests/test_worker_watchdog.sh, so the BUILD_DIR=
sanitizer and container legs test the binary they built.
- docs: CONFIGURATION.md + README.md document that watcher_enabled is read once
when the daemon starts — the daemon outlives individual MCP sessions, so
reconnecting a client is not enough; `daemon stop` retires it. This corrects
the earlier claim that auto_watch is also startup-read: auto_watch is consulted
per registration, and the two keys are now contrasted on read time as well as
on scope (session registration vs whether the subsystem starts at all).
No production code change in this commit: watcher.start and watcher.watch already
exist as production logs.
Signed-off-by: soren <nguiasoren@gmail.com>
Align the exact Claude PreToolUse smoke contract and documentation with the Bash-enabled search matcher.
Signed-off-by: Jake Abendroth <contact@jakea.net>
scripts/test.sh labels its pre-build contracts as a running 0a..0u sequence.
The new blob-generator contract was slotted in as "0f2" to sit next to the
other build-tooling contract, which breaks the one convention the list has.
Append it as 0v instead; these steps all run before the build, so position
carries no meaning beyond the label.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
vendored/nomic/code_vectors_blob.S is a GENERATED file that was hand-edited.
1f674c80 added the ELF .note.GNU-stack section to it after finding that every
Linux binary we had ever shipped requested an executable stack: an object
carrying no such note tells GNU ld nothing about its stack requirement, and ld
then assumes the worst for the entire link. That fix went into the artifact
only. scripts/extract_nomic_vectors.py still emitted the Mach-O branch alone,
so the next regeneration would have overwritten the tracked .S, dropped both
the ELF and COFF branches, and put GNU_STACK RWE back into every release --
silently, because nothing compared the generator against its own output.
write_blob_s() now emits the tracked wrapper verbatim, and the new contract
test fails if the two ever drift again. The template is compared rather than
executed so the test does not need the torch/transformers import the extraction
script pulls in. The write is pinned to UTF-8 because the WHY comment carries
an em dash and the default encoding is locale-dependent.
check-binary-composition.sh already fails a release whose binary has an
executable stack; this closes the same hole at edit time, where the diff is
still small enough to read.
Reported and originally fixed by Anand Aiyer in #1151, which also carried the
COFF branch. That PR sat unreviewed for a month while the .S half was
rediscovered and landed independently without credit.
Co-authored-by: Anand Aiyer <anand@aiyer.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The *.sh eol=lf rule landed via #1314, but the git hooks
(scripts/git-hooks/commit-msg, scripts/hooks/pre-commit) are extensionless
and were still at the mercy of core.autocrlf. Add their explicit entries,
plus the contract test from #1272 wired as scripts/test.sh Step 0t: every
shell entrypoint (*.sh + both hook directories) must carry an eol=lf
attribute, with a matched-zero-files guard so a broken glob can never pass
vacuously.
Distilled from #1272: the diagnosis (CRLF checkouts breaking shebangs under
WSL/MSYS), the hook-file coverage, and the guard design are @xumian520's;
verified RED without the *.sh rule (105 uncovered entrypoints) and green
with it (107 files).
Co-Authored-By: xumian520 <126989134+xumian520@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The v0.10.6 release run failed deterministically at verify:
BLOCKED: expected scan object is missing:
objects/scan-3099e91c...--codebase-memory-mcp.exe
exclude-rescanned-selected-objects.sh (added after v0.10.5, first exercised
by this release) deliberately deletes the selected executables from the
surface-scan directory — their bytes were already scanned as candidates and
re-submitting identical bytes re-rolls a probabilistic classifier — and
writes binaries/virustotal-withheld.tsv. But check-virustotal.sh still
received the pre-withhold scan-set listing all sixteen objects and failed
closed on the first missing file. The rework's two halves never talked.
The gate now accepts an optional VT_WITHHELD manifest (strict parse: v1
marker, the stated reason required, sha256-keyed rows): an expected-set row
whose hash the manifest vouches for is exempt from the on-disk and
action-output contracts, while everything else keeps the strict path.
Fail-closed properties preserved and extended:
- no VT_WITHHELD -> byte-for-byte previous behavior (candidate
stage and dry-run call sites are unaffected;
verified against the original failure)
- withheld object present -> blocked (inconsistent staging)
- hash outside the set -> blocked (spurious withhold)
- everything withheld -> blocked (scan would cover nothing)
- mismatched object name -> blocked
vt-results.tsv keeps its exact shape (scanned objects only) — the release
notes table already uses the candidate results, and the withheld manifest is
now preserved with the rest of the evidence artifacts. release.yml passes
VT_WITHHELD only in the verify stage, right after the withhold step.
Verified offline with a fixture reproducing the release failure verbatim
plus the four negative cases above; the positive case passes staging and
association validation and proceeds to VT polling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The verify pass submitted every extracted object, selected executables included,
on the stated grounds that "VirusTotal is content-addressed, so identical bytes
return the analysis it already holds instead of re-running 70+ engines".
That is measurably false. On v0.10.5 all EIGHT re-submissions produced a NEW
analysis - same VirusTotal file-id, timestamp 47 minutes later:
candidate: file-id=2c00f485... ts=1786795957 (12:12:37Z)
verify : file-id=2c00f485... ts=1786798758 (12:59:18Z)
Re-analysing identical bytes re-rolls a probabilistic classifier, and Microsoft's
ML engine answered differently within that hour, in BOTH directions:
82750cd1 (linux-amd64) microsoft-ml -> clean
6d3c5be6 (darwin-arm64) clean -> microsoft-ml
The published notes are generated from the candidate scan, so v0.10.5 shipped a
table calling linux-amd64 flagged when VirusTotal had it clean, and darwin-arm64
clean when VirusTotal was reporting Trojan:Script/Wacatac.B!ml. Every hash in
that table links to the page that contradicted it. Corrected in place after
publication; this removes the cause.
The second scan proved nothing the first did not. Identity is settled by hash
before this step runs: verify-release-selection.py reconciles every published
container to the selected bytes, and checksums.txt binds the same digests
publicly. A re-scan adds no assurance - only another roll.
What still gets scanned is exactly what the candidate pass never saw: install.sh,
install.ps1, LICENSE, THIRD_PARTY_NOTICES.md, the MCPB manifest.json and the
unpacked UI assets. install.sh and install.ps1 are the highest-consequence
non-executable bytes we publish - users pipe them straight into a shell - and
that coverage is untouched. Measured on the v0.10.5 object set: 16 objects in,
8 withheld, 8 still scanned.
The withheld set is recorded as evidence (cbm-virustotal-withheld-v1) naming each
sha256 and pointing at virustotal-candidate-results.tsv, so the published
evidence still accounts for every shipped object.
Fails closed three ways, each with an actionable message: no object matches a
selected sha (the containers do not carry the recorded bytes), everything is
withheld (the surface scan would be a no-op), or the selection names no shas at
all. The zero-match grep is wrapped rather than left to pipefail, because a
guard that aborts silently is not a guard - found by testing the guards rather
than assuming them.
Also drops 8 VirusTotal submissions per release.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
audit-license-provenance.py ran `curl https://www.apache.org/licenses/LICENSE-2.0.txt`
every time the gate executed and byte-compared the result against
vendored/nomic/LICENSE, with capture_output and no error check. Any fetch
failure therefore produced an empty string, compared unequal, and reported:
vendored/nomic: DIFFERS [apache.org canonical LICENSE-2.0.txt]
PROVENANCE AUDIT FAILED: 1 unexplained verdict(s)
which is indistinguishable from a real licence discrepancy.
A gating verdict must be a pure function of the tree, not of whether a web
server answered. This one was neither reproducible nor attributable: it
reddened `security / license-gate` on PR #1337 - a branch touching cli.h,
hook_augment.c and test_cli.c, and no licence at all - and left that
contributor blocked for over two weeks on a signal that had nothing to do with
their change.
Verified before pinning: our vendored copy is byte-identical to the upstream
canonical text, 11358 bytes on both sides, diff clean. The Apache-2.0 text is
immutable and versioned, so a digest is the honest way to express "this is that
text". A mismatch now means our vendored copy changed, which is exactly - and
only - what this audit exists to detect.
The audit passes locally with no network access on the nomic entry.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The ui-* archives are byte-identical copies of the canonical ones, published
after verify so they inherit the hash-bound VirusTotal verdicts. They were
absent from checksums.txt, and publish-legacy-aliases.sh documented that as
intentional: "checksums.txt covers the canonical names current installers
request."
That reasoning has a hole. The aliases exist only for 0.9.x updaters (#1538),
and those verify the NAME they asked for. So the alias fixed the 404 and moved
the failure one step later - the updater downloads the archive, cannot find its
name in checksums.txt, and refuses:
warning: codebase-memory-mcp-ui-darwin-arm64.tar.gz not found in checksums.txt
error: refusing to install an unverified download
Reported by AmooAti in #1134. Confirmed on the live v0.10.4 release: eight ui-*
archives published, zero of them listed. Every pre-0.10 user who answered the
old variant chooser with "ui" is hard-blocked from updating by any path.
The same digest is now emitted under the legacy name before the attestation
step, so the attested artifact covers both names. No new bytes and no new scan
surface: an alias is a copy, so its sha256 is by construction the one already
computed.
The rule lives in scripts/ci/append-legacy-alias-checksums.sh rather than inline
in the workflow, because the venue-parity contract requires it: a venue may
provision, plumb artifacts, or call a canonical leg script, and text
transformation is none of those. Keeping it beside publish-legacy-aliases.sh
also puts the two halves of the alias rule in one place, which matters because
they must stay in step - .tar.gz and .zip only, never an already-ui-* name. It
fails closed when it matches nothing, since a name with no asset is as broken as
an asset with no name.
Validated against the real v0.10.4 checksums file: the generated set is exactly
the eight ui-* assets that release published - no phantom names, none missing -
and the empty case exits non-zero.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Release notes now carry one row per product — the bytes actually published, with
their VirusTotal verdict — instead of a column per candidate.
Several candidates are scanned so the selector has an alternative when an opaque
classifier flags one, but a reader installing cbm cares about what they receive,
not about what we discarded. The rejected candidates' verdicts remain in the
published evidence TSVs for anyone auditing the selection, and they stay useful
to US as a development signal; in a changelog they are noise, and listing a
"microsoft-ml" verdict next to a binary nobody can download invites exactly the
wrong conclusion.
Also removes a claim that had become FALSE: the section asserted that every scan
had a minimum of 50 decisive engines. That floor was removed deliberately (it is
VirusTotal fleet availability, not a property of our binary) and the last release
observed 31. Published notes would have stated something untrue. The observed
range is still reported, now over the shipped binaries.
The disclosure of a tolerated Microsoft `!ml` is preserved and narrowed to the
binaries that ship. The contract test now pins both directions: a flagged binary
that SHIPS must be disclosed, and flagged candidates that were rejected must not
appear at all. Its fixture gained a target that draws the tolerated verdict on
all three candidates, so the disclosure branch is actually exercised — previously
every flagged candidate was rejected in favour of a clean sibling, leaving that
path untested.
README and SECURITY.md updated: they promised notes "link every candidate result".
All five release/VT contract tests pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The v0.10.4 release (run 31755436916) failed at the very last step of `verify`,
after all 69 other jobs had passed — full test matrix, all eight builds, smoke
and soak:
publish-vt-evidence: wrong evidence marker in binaries/vt-results.tsv
The results format was bumped to `cbm-virustotal-results-v2` in #1596, but
publish-vt-evidence.sh still demanded `-v1`. It was the only straggler: the
gate, the selector, the notes renderer and all three contract fixtures were
already on v2.
It survived because that PR had removed the script's only caller, leaving it
dead code that nothing exercised. Restoring the full-surface scan restored the
caller too, and the stale expectation surfaced in the worst possible place — at
the end of a real release rather than in a dry-run, since `verify` does not run
in dry-runs.
Also adds the guard that would have caught it: every marker publish-vt-evidence.sh
validates must be one the gate actually writes. Revert-checked — reintroducing
v1 fails with "publisher expects marker never written by the gate:
cbm-virustotal-results-v1".
The scan itself passed. This is purely the evidence-publishing step.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two things, both found by the dry-run and by sweeping what it exposed.
1. The three-candidate change was incomplete. The dry-run failed at
stage-release-candidates.py, which carries its OWN copy of the transform
validation that the previous commit only fixed in the selector:
stage-release-candidates: candidate linux-amd64/debug-stripped has
invalid transform: 'strip-debug'
Sweeping for that assumption found it in six places, not two:
stage-release-candidates.py, select-release-candidates.py,
verify-release-selection.py, append-vt-notes.sh and two contract-test
fixtures — as hardcoded 16s, `len(TARGETS) * 2`, two-entry VARIANTS tuples
and two-key truth tables. All are now derived from len(VARIANTS).
Field prefixes needed care: the variant NAME keeps its hyphen because it is
the on-disk directory, while the evidence columns use underscores, so
`debug-stripped` reads `debug_stripped_sha256`. Every lookup now goes through
an explicit FIELD_KEY map instead of interpolating the variant directly.
The selection contract test now covers the truth table EXHAUSTIVELY: three
variants x two tolerated classifications is exactly eight combinations, and
there are exactly eight targets, so every case is exercised once.
2. Full-surface VirusTotal scanning is restored. This PR had moved scanning
upstream to the candidates and deleted the post-package pass, which silently
narrowed coverage from everything we ship to executables only. The 42 runtime
files across the 14 containers — install.sh, install.ps1, LICENSE,
THIRD_PARTY_NOTICES.md, the MCPB manifest.json and the unpacked UI assets —
were still extracted, structurally verified and strings-audited, but no
longer scanned at all. install.sh and install.ps1 are the highest-consequence
non-executable bytes we publish; users pipe them straight into a shell.
The verify job scans every extracted object again, under the same policy.
Re-submitting the selected executables alongside them is close to free
because VirusTotal is content-addressed and answers for identical bytes from
its own record — the same property that made the analysis-id equality check
untenable two commits ago.
The gate-chain contract asserted the opposite ("duplicate post-package
VirusTotal path remains"). That assertion is inverted: the pass is required,
and it is not a duplicate, since it covers a strictly larger set. README and
SECURITY.md updated from "archive containers are checksummed rather than
redundantly rescanned" to state the full covered surface.
All five release/VT contract tests pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Release evidence from run 31744302624 shows the tolerated Microsoft `!ml`
verdict is close to a coin flip per byte image rather than a property of the
code. Across the eight targets the stripped and unstripped candidates of the
SAME linker output disagreed on four, and in both directions:
linux-amd64 stripped microsoft-ml unstripped clean
darwin-arm64 stripped clean unstripped microsoft-ml
linux-arm64 stripped clean unstripped microsoft-ml
If the classifier were keying on something intrinsic to our code the siblings
would agree; they do not. So each variant is close to an independent draw, and
5 of 16 candidates drew the flag.
Two draws is not always enough. On that run linux-amd64-portable came back
microsoft-ml on BOTH candidates, leaving no clean binary to ship for that
target. A third independent draw at a ~31% observed per-candidate hit rate takes
the both/all-flagged case from roughly 1-in-10 per target to roughly 1-in-30.
The third candidate is `--strip-debug` (Apple: `-S`): debug information removed,
symbol table kept. Behaviourally identical to the other two — same linker
output, only metadata differs — but a distinct byte image, which is all
VirusTotal needs to scan it as its own file. Verified on the real v0.10.4
candidates: linux-amd64 gives three distinct hashes (294,634,656 /
294,623,208 / 293,746,096 bytes) and darwin-arm64 likewise, with the ad-hoc
signature verifying after strip.
Selection is unchanged in spirit and now ordered: smallest artifact first
(stripped, debug-stripped, unstripped), take the first CLEAN one, and only if
every candidate drew the tolerated verdict ship the smallest flagged one. A
hard verdict on any candidate still blocks the release before selection.
Cost is 24 objects per release instead of 16.
Derivation enforces that all three hashes differ — identical candidates would be
one draw wearing three hats, and the selector would believe it had alternatives
it does not have. Public claims in README, SECURITY.md and docs/index.html
updated from "both stripped and unstripped" to the three candidates.
All three release contract tests pass, including the native derivation test
which exercises the real strip and codesign path on this host.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The build+VT dry-run (run 31744302624) blocked all eight targets on:
BLOCKED: objects/3eb486a4... completed with only 48/75 decisive engines (< 50)
That object is the windows-arm64 STRIPPED candidate with ZERO detections. It
was classified "hard" — the same bucket as a real malware verdict — purely
because 48 engines answered instead of 50. Its own sibling scanned clean with
66.
How many engines return a decisive result is VirusTotal fleet availability on
the day, not a property of our binary. These artifacts are ~300 MB and many
engines skip or time out at that size, so the count sits near the floor and
moves run to run: on this run one object landed at 48 and the other fifteen
spread 59-68. A floor there makes shipping a lottery decided by someone else's
infrastructure, which is exactly what our CI-determinism rule forbids, and it is
not something we can influence by fixing anything on our side.
Classification now depends only on what engines FOUND:
hard - two or more engines, any non-Microsoft engine, any label that is
not `!ml`, or anything suspicious. Still blocks the release.
microsoft-ml- exactly one Microsoft `!ml`. Tolerated and disclosed.
clean - nothing flagged.
The decisive-engine count is still recorded per candidate in the results
manifest and printed as a NOTE when it is below the reference value, so the
evidence is unchanged; only its power to block is gone. The selector keeps
rejecting an incoherent response (total < completed).
Public claims updated to match in README.md, SECURITY.md and docs/index.html —
all three promised "at least 50 decisive engines" as a publication requirement,
which would no longer have been true.
Contract test updated: a clean verdict now stands regardless of engine count,
and the below-reference NOTE must still be reported. All three release contract
tests pass (vt-gate-policy, release-gate-chain, candidate-derivation).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel