Why: the test runner isolated HOME and CBM_CACHE_DIR but not
CBM_RUNTIME_DIR, and neither of the first two moves the daemon
rendezvous. Its key is a constant product hash (cbm_daemon_rendezvous_key,
"c888acc3ae367a1e"), and with no explicit parent and CBM_RUNTIME_DIR unset
cbm_daemon_bootstrap_endpoint_new falls back to the account-wide default
parent (/private/tmp on macOS, /tmp on Linux, LocalAppData on Windows).
Every endpoint a test resolved without an explicit parent was therefore
byte-identical to the developer's live daemon socket
(/private/tmp/cbm-daemon-<uid>/cbm-c888acc3ae367a1e.sock), whatever HOME
said.
The only in-runner product caller is CLI activation
(cli_activation_production_context_init, reached from cbm_cmd_install,
cbm_cmd_uninstall and update). Only a process-global seam kept it off the
default: SUITE(cli) sets that seam on entry and clears it on exit. On the
default endpoint, activation admits into the live daemon's cohort. When
the cohort is ours or busy, or the active cache fingerprint is unreadable,
activation prints "Stopping active CBM sessions" and drains them. That is
the same class of failure as an old binary's install, which stopped a
user daemon and six MCP sessions. One test outside SUITE(cli), or one
fixture that drops the variable, was enough to reach it.
daemon_bootstrap_runtime_dir_env_relocates_rendezvous did exactly that:
it restored by unsetting CBM_RUNTIME_DIR.
Fix:
- tests/test_main.c: tf_setup_cache_sentinel creates a fresh per-run
runtime parent (th_secure_runtime_parent_new "run", under /tmp on POSIX
and LocalAppData on Windows) and sets CBM_RUNTIME_DIR to it for the
whole run. It replaces any inherited value, the rule
scripts/test-runtime.sh applies to the shell harness. The existing
atexit cleanup removes the directory.
- src/daemon/bootstrap.c, CBM_ENABLE_TEST_SEAMS only: once a test runner
arms the guard, an endpoint request with no explicit parent and no
CBM_RUNTIME_DIR is refused with a stderr diagnostic instead of
resolving the default. No product binary arms it.
- tests/repro/repro_main.c arms the same guard. No reproduction needs the
rendezvous.
- tests/test_daemon_bootstrap.c: the relocation test restores the value
it found instead of unsetting it.
- tests/test_helpers.h: th_secure_runtime_base() names the anchor that
th_secure_runtime_parent_new already used.
Proof:
- Before, on origin/main 1f9b4db1: runner_isolation is RED 3/3 at its
first assertion (CBM_RUNTIME_DIR is NULL during the run).
- After: green 3/3. The endpoint resolved through the product resolver
lies inside /private/tmp/cbm-runtime-run-XXXXXX/, never at
/private/tmp/cbm-daemon-<uid>. All 43 activation-guard scopes logged
by the suites below are fixture or cli-suite directories, and the
guard never fired. The per-run directory is gone after exit, and the
live daemon and its runtime directory are unchanged.
- Revert: dropping the setenv gives RED (value is NULL). Disarming the
guard gives RED (refused_without).
Tests:
- runner_isolation: runner_isolation_daemon_runtime_dir_is_private,
runner_isolation_refuses_default_daemon_runtime
- Green on macOS: cli (328), mcp + mcp_mutation_guard (344),
daemon_runtime + daemon_application (106), daemon + daemon_version +
daemon_frontend + daemon_bootstrap + daemon_ipc (117), runner_isolation
+ activation_transaction + version_cohort + project_lock +
index_supervisor + private_file_lock + lock_registry (73). All skips
are existing Windows- or Linux-only platform skips.
- make -f Makefile.cbm lint-ci passes. test-repro-runner builds.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
extraction::extract_spill_round_trip_keeps_every_field and
parallel::parallel_spill_mode_builds_the_same_graph failed on any machine
with less than 10 GB free: cbm_result_spill_open refuses to spill onto a
nearly full disk (mem.spill.refused reason=low_disk), so the store never
opened. Three pipeline tests took the same refused-store path there and
passed without spilling at all (backpressure_futile_nap_disengages,
over_budget_after_futility_fails_whole_and_preserves_db,
spill_resolves_namespace_imports_like_memory). A verdict must be a pure
function of code, test, platform and seed, not of how full the disk is.
Add a free-space pin to result_spill that exists only in test-seam
builds (CBM_ENABLE_TEST_SEAMS). Production still asks the disk, and its
10 GB floor and half-of-free cap are unchanged. The test runner pins
64 GiB at the top of main, before worker re-exec dispatch, so every
suite and re-exec'd index worker sees the same disk. A new test,
extraction::extract_spill_refuses_below_the_free_disk_floor, pins the
guard itself: one byte under the floor refuses, the floor opens.
Verified with cbm_fs_free_bytes temporarily forced to 9 GB: both tests
RED 3/3 before, green 3/3 after, RED again with the runner pin removed;
the refusal test goes RED with the production guard disabled.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Replace raw malloc/free with cbm_alloc/cbm_free (CBM_MEM_CLASS_EXTRACT)
and a growable CBM_DYN_ARRAY for matches. Remove the 4KiB line-copy
cap; lines are split in place instead of copied. Remove the
DOCLINK_MAX_REFS(256) cap; the ref vector now grows, is sorted by
target id, then collapsed. Track reference kind (bare, relative,
rooted) so /x never attempts a directory join and ./x never falls
back to root. Report per-file skips via cbm_pipeline_add_file_error
and cbm_log_warn, and check cbm_pipeline_check_cancel in the per-file
loop.
Add negative-control tests for all of the above, plus CRLF handling,
fenced-code-block pinning, bare URLs, backtick-basename exactness,
and ".." escaping.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Clément Deust <clement.deust@gmail.com>
Markdown docs reference other repo files constantly (a coding-standards
doc links the modules it governs, a README points at entry points) but
none of that surfaced as edges, so fan-in queries were blind to
documentation hubs. In an A/B retrieval benchmark I ran in August on a
docs-heavy repo, the graph's top fan-in answer was off by roughly 17x:
the most-referenced file was a standards doc with 173 references and
zero inbound edges.
New pre-dump pass, pass_doclinks.c, modeled on pass_configlink.c: three
strategies emit REFERENCES_FILE edges between existing File nodes only
(unresolvable targets are dropped, the pass never invents nodes):
inline links, backtick paths, bare path mentions. Targets resolve
against the referencing file's directory and the repo root; repeated
references collapse into one edge carrying strategy, confidence and
count. Registered in the pre-dump sequence after configlink and in the
incremental post-passes; REFERENCES_FILE added to the skill's edge-type
list and the structural/language contract tests.
Tests mirror test_configlink.c (real files in a tmpdir, File nodes in a
gbuf, run the pass, assert edges): inline link, backtick, bare mention,
http/anchor ignored, anchor-suffixed file link, dedupe with count,
relative-vs-root resolution, unresolvable-target guard, NULL repo_path
skip.
A companion change adds the same linking for shell files (source lines
and script invocations); split out to keep each change reviewable.
Validation: scripts/build.sh clean; focused serial runner (doclinks,
configlink, pipeline, edge_structural, lang_contract) 340 passed under
ASan/UBSan; cppcheck clean on the new files with the repo's flags.
Signed-off-by: Clément Deust <clement.deust@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every item here was measured on a real corpus before and after, not
reasoned about. Go corpus, event lane, first run -> now: 245.5 M
allocations -> 134.8 M, 190 GB cumulative -> 29.3 GB, 158 GB never
written -> 9.2 GB, 132 M churn pairs -> 35.7 M, 93 GB of allocator byte
work -> 0.84 GB. Wall clock: Go 40.7 s -> 19.3 s, linux kernel 239.6 s ->
194.6 s, and all ten LSP languages faster with none slower.
Graph output is unchanged, proven rather than assumed: per-class multiset
digests over the full kernel index (8,563,471 nodes identical, 18/20 node
classes and 18/24 edge classes byte-identical, the rest differing only in
the classes a same-binary re-run also perturbs through worker merge order).
The changes, each with its own reason:
arena an exact arena that is appended to later restarts growth at
8 KB instead of the 64 KB default — the cross-file pass adds a
few resolved calls to a compacted result, so the default block
was ~60 KB untouched per result, 0.5 GB of the Go worker peak.
cbm_arena_init_lazy defers the first block for arenas that are
usually never used at all.
scratch per-thread reuse instead of allocate-and-free per unit of
work: result_compact (8 MB kept), the cross-file pass (4 MB),
extract_defs' walk stack, and the sqlite index cells (a 1 MB
arena replacing 12.1 M malloc/free pairs on Go). Walk frames
moving into scratch is why pass_k8s now opens the keep window
explicitly: without it every YAML file opened a 512 KB scratch
on the main thread, +3 GB.
sqlite index cells are written in place — building the record and
copying it into the cell was a second allocation per entry,
16 M record buffers of pure churn on Go.
registry is_test_qn and common_prefix_len in one pass, per-bucket test
flags computed once instead of per candidate comparison, and
the reachability keys arena allocated lazily.
minhash the unique-trigram set is sized from the trigram count and
skips repeats, and LSH queries reuse one seen-set per worker
instead of allocating per query.
parallel lsp_surface rows and the k8s manifest prep now run across
workers (graph writes stay sequential and ordered), the file
node is hoisted out of register_and_link_def, and configlink
prepares dependency imports once with a length prefilter.
go_lsp one sealed stdlib registry shared by all workers, published
by CAS, instead of one per worker.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Follow-up to the re-exec fix, from the challenger read the frozen scope
requires for security-boundary units. Three findings, all addressed here; two
direction questions go to the maintainer separately.
1. src/daemon/ipc.c stated an invariant that is FALSE. The comment above
posix_ancestor_overflow_uid() read "Derived once from immutable /proc
state". /proc/self/uid_map is not immutable -- unshare(CLONE_NEWUSER) is
exactly what rewrites it -- and pthread_once state survives fork() already
marked done. The hazard was written up in the tests, which is the one place
an ipc.c maintainer will never look, while production shipped the opposite
claim. The comment now states the real guarantee: the cache is safe only
because every caller runs in a freshly exec'd process, and it names what
would break it.
An audit of every process-creation site in src/ backs that: the daemon
double-fork execs (daemon/bootstrap.c), cbm_fork_with_retry execs
(foundation/subprocess.c), index workers go through cbm_subprocess_spawn,
worker_pool.c never forks, and src/ contains no setns or unshare at all.
No production path forks without exec and then reaches this check.
2. The probe now _exit()s instead of returning. The runner is built with
-fsanitize=address, so returning runs the atexit chain, and a future leak
anywhere in the prologue would exit 23 -- which the parent would have read
as a verdict on a security test that has none.
3. The parent now treats 0/1/2/3 as the closed protocol and names anything
else with the code it actually saw. A signal or a sanitizer exit is a broken
harness, not a refusal, and must not be able to look like one.
The probe guard also now matches its caller (linux AND test seams) instead of
compiling dead code in a non-seams build.
VERIFICATION. The challenger was right that the earlier proof was incomplete:
it showed the new test goes green, never that the test still binds the #1830
relaxation. Stubbing posix_compute_ancestor_overflow_uid() to
POSIX_NO_OVERFLOW_UID -- the relaxation removed -- and running as uid 1001:
tolerance off, new probe 48 passed, 1 failed (refused, exit 1)
tolerance on, new probe 49 passed, 0 skipped
so the smoke test is revert-sensitive to the production decision, not merely to
the test change. Worth recording: both deterministic #1830 tests PASSED with
the tolerance stubbed off, because they drive the test-seam override rather
than the real derivation. They bind the decision table; only this smoke test
binds the wiring.
macOS daemon_ipc 53 passed 1 skipped; linux/arm64 49 passed 0 skipped;
lint-ci clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The single-uid userns smoke test forked a child, unshared CLONE_NEWUSER,
wrote a 1:1 uid_map, then called cbm_daemon_ipc_private_directory_secure()
directly in that child. The call cannot give the right answer there.
posix_ancestor_overflow_uid() in src/daemon/ipc.c derives the overflow uid
once per process under pthread_once, and the comment above it calls the
/proc state "immutable". unshare(CLONE_NEWUSER) is precisely what changes
/proc/self/uid_map, and pthread_once state survives fork(). Seven earlier
call sites in this suite run the derivation first, so by the time the test
forks, the child has already inherited the HOST answer -- no overflow uid --
and refuses the directory no matter what its namespace says.
The test therefore asserted a product behaviour it never executed, and failed
as a product refusal (exit 1) while the product was correct: production
daemons start inside their namespace and derive the right answer on first use.
Re-exec /proc/self/exe into a new --userns-secure-probe mode so the decision
is made by a process that STARTED in the namespace, which is also the only
shape production ever takes. A failed exec gets its own exit code 3 and a
named FAIL, so a broken harness can never be misread as a quiet refusal.
Verified both directions on linux/arm64 in the container leg as uid 1001:
reverted, 48 passed / 1 failed on "expected WEXITSTATUS(status) == 1"; with
the fix, 49 passed, 0 skipped.
The non-root half of that recipe is now written down at the test, because it
is the part that fooled me first: as root the mapped range covers uid 0, so
root-owned /tmp stays 1:1 inside the namespace and never turns overflow. The
first run of this reproduction passed with the fix REVERTED for exactly that
reason. Only a non-root uid reaches the branch the test exists to cover.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Indexing accepts whatever a repository contains. A tree carrying a vendored
monorepo, a generated dump, or a runaway build directory is discovered in
full, and the first sign of trouble is a host under memory pressure with
nothing that attributes it to indexing.
Add two opt-in limits evaluated during discovery against accepted source
files only: index_max_files and index_max_source_mb. Both default to off, so
nothing changes until an operator sets one. Crossing a limit fails the whole
attempt with a structured resource_limit_exceeded result naming the resource,
the observed value and the limit; no partial graph is published, and an
existing serving index keeps answering.
Limits are read from the CLI-managed _config.db and are not MCP request
arguments. A supervised parent replaces any caller-supplied policy before
spawning its worker, and the worker rejects a missing or incomplete contract,
so the CLI, the daemon and the supervised worker all enforce the same
decision.
Both keys reach an operator through the existing config get/set/list/reset
with no new subcommand. `set` suppresses its own generic message for them
because the policy writer names the precise reason -- so that writer speaks
on every failure it can return, including a validated value whose write then
fails on a database that cannot be written. Exiting non-zero in silence is
not an acceptable answer from a CLI.
The two shell regressions that hand-roll the supervisor's worker argv carry
that contract as well. Without it the worker exits before either guard can
observe anything, and the guard would go quietly vacuous.
Signed-off-by: 刘冲 <mail@liuchong.dev>
daemon_runtime_process_fingerprint_never_hashes_replacement_path (POSIX
variant) copied /bin/cat to a temp file named "image" and executed the
copy. On Ubuntu 25.10+/26.04 and any other uutils system, /bin/cat is a
multi-call binary: a copy invoked under the name "image" prints
"coreutils: unknown program 'image'" and exits 1, so the fixture died at
ASSERT(setup). Reproduced on WSL Ubuntu 26.04 (uutils coreutils 0.8.0).
The Windows variant of the same test already avoids this class of donor:
it copies the test runner itself and parks it in a
__cbm_runtime_image_holder argv mode. The POSIX variant now does the
same. The holder mode gains a POSIX branch that blocks reading stdin
until EOF — exactly the behaviour the cat donor provided — and
runtime_test_spawn_blocked_executable passes the holder argument, so its
target must now be a copy of this runner (it has exactly one caller).
The /bin/echo replacement donor is gone too: it was only ever hashed,
never executed, and on a uutils system cat and echo can be the same
multi-call bytes, which would have broken the fixture's
original != replacement precondition. The replacement is now a small
data file, mirroring the Windows variant.
No signing change is needed on macOS: the image copy stays
byte-identical to the runner, and the identical-copy fixture already
proves such a copy executes.
daemon_runtime suite on WSL Ubuntu 26.04: 44 passed / 0 failed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: OhOkThisIsFine <102485413+OhOkThisIsFine@users.noreply.github.com>
Score every Function/Method/Class node during indexing and persist the
result as a numeric "importance" key inside the node's existing
properties_json:
importance = sqrt(num_refs) * priv * generic * distinct * test_penalty
num_refs is the incoming CALLS + USAGE degree. The multipliers demote
private (leading-underscore) and generic names (defined in >= 5 distinct
files), promote distinctive identifiers (snake_case or camelCase, len >=
8), and demote test scaffolding -- symbols in a test file per the graph's
own cbm_is_test_path() classifier, or targets of an incoming TESTS edge.
Weighted degree only; a transitive (PageRank) refinement is deliberately
not built here until it can be shown to beat this on a fixed judgment
set.
No schema change and no CBM_INDEX_FORMAT_VERSION bump: the score lives in
the existing properties_json TEXT column, so no index is forced to
rebuild and indexes written by older builds simply lack the key.
Three failure modes this shape invites, all of them silent, are addressed
directly:
* Cost. The generic-name multiplier needs |{files a name is defined in}|.
Computing that per NODE with a pairwise file-path scan is O(k^3) in a
same-name group of size k -- ruinous on real corpora, where a single
Java name group reaches k > 4000. Here the count is computed once per
distinct NAME and memoized, and file paths are deduplicated through a
hash set, so total distinct-file work is linear in the graph. Measured
cold full index of a 666 MB Java corpus, same -O2 binary: 85 s with the
pass disabled, 80 s with this implementation, 481 s with the per-node
shape.
* Registration. PREDUMP_PASS_COUNT is now derived from the pass table
with sizeof instead of being hand-written. A hand-written count that
lags an appended entry silently skips the last-registered pass with
every test still green.
* Incremental. The incremental post-pass sequence runs the pass too, over
a graph rehydrated from the store whose nodes already carry the key.
The write-back overwrites an existing key in place instead of appending
a second one, which would otherwise corrupt properties_json into
{"importance":1.0,...,"importance":2.0}.
tests/test_importance.c covers the formula and binds all three failure
modes; the complexity suite gains a GATED counter asserting the pass's
same-name-group work stays linear (~2x, not ~4x) under a corpus doubling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: petercoxphoto <20230727+petercoxphoto@users.noreply.github.com>
Maintainer follow-up on top of the contributor's rebased commit,
implementing the Half-B strategy decision from the 2026-08-21 review:
- Linux: when the self-resolved path fails the executable check after an
atomic rename-over, hand back the /proc/self/exe magic link. The link
still executes the in-memory OLD build, which is the only spawn the
worker's build-fingerprint gate accepts. Deliberately NOT the saved
launch path: after a replacement that path holds a DIFFERENT build, so
preferring it swaps an ENOENT for a fingerprint refusal.
- macOS: no magic link exists; fail closed. The supervisor logs
index.supervisor.no_self_path and indexing resumes on daemon restart
instead of spawning a missing binary. A rename-over that leaves a VALID
binary still resolves (and the fingerprint gate refuses downstream, with
its own coverage).
- The prefer-saved-launch-path branch from the original head is dropped
per the same ruling.
Also fixes both Half-A review items:
- Aliasing in cbm_http_server_set_binary_path: `path` may alias
g_binary_path on a repeated call. The overlapping snprintf cannot
actually fire today — resolve's leading out[0]='\0' zeroes the shared
buffer first — but that same reset means the call silently DISCARDS the
stored path it was asked to re-resolve, and the in-place write is one
reset-reorder away from real UB. Resolve into a local first: correct
input, no aliasing trap.
- Coverage asymmetry: the probe and driver ran only on Linux while the
production edit covered all of POSIX. Both now run on macOS too, and the
Linux probe first proves the "(deleted)" state via readlink so its
assertions cannot pass vacuously on an intact image.
Revert-check: with main's http_server.c restored, the driver fails at
WEXITSTATUS 44 (resolver returned a dead path) on macOS; with the guard it
passes 63/63.
Co-authored-by: sunmingqiang <sunmingqiang@bytedance.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Prefer the saved executable launch path when it still points to an
executable, and reject a deleted /proc/self/exe path before spawning
index workers.
Add a Linux integration regression that forks, atomically replaces the
running test binary on disk, and verifies that subprocess resolution
uses the saved launch path instead of the now-deleted self path.
Signed-off-by: David Musk <davidmusk93@users.noreply.github.com>
Signed-off-by: sunmingqiang <sunmingqiang@bytedance.com>
Finding #1669 took an 11-corpus A/B across two release binaries. This
suite makes that bug class fail a unit test in seconds, on every
platform, from tiny corpora.
Method: build k and 2k REPLICATED module copies, run the full in-process
pipeline on both, assert counter RATIOS. Independent copies mean every
extensive quantity — nodes, edges, Σ per-file registry defs — must grow
linearly (ratio ~2). A files x corpus coupling makes per-file work itself
grow with k and lands at ratio ~4. Ratios expose the exponent regardless
of absolute scale, so 60-120 files suffice.
Verdicts are pure functions of (code, input): gates ride ONLY on
deterministic work counters and data-product counts, never on wall time.
Throughput (nodes/s, edges/s) is information-only, written to
private/benchmarks/complexity-<ts>.json (local, gitignored; skipped
under CBM_SKIP_PERF where rates are meaningless).
Two corpus shapes, both needed:
- Independent modules (java/py/go/ts templates): catches cross-module
contamination and dedup breakage. The #1669 bug is GREEN here — fully
closed modules filter perfectly, which is exactly why it survived.
- The growing shared package (bigpkg): one Java package whose file count
scales with k — the real-repo shape (files concentrate in large
packages). The JVM namespace filter branch makes per-file work track
package size, so this corpus is the honest #1669 reproducer:
ratio 4.00 RED on the pre-fix tree, 4.00 RED for a module-scoped
overlay, 2.00 GREEN for the own-file overlay. It discriminated the
correct fix design before the fix was written.
Both legs of every pair exceed MIN_FILES_FOR_PARALLEL(50): below it the
sequential path runs, which builds no shared registries and would be the
wrong code path to gate (its per-file cost is bounded by the 50-file
ceiling).
Recorded but deliberately NOT gated, with reasons at the case:
tail_candidates and fallback_rows are legitimately superlinear under
replication until those scans are bounded, and measured ~1 ns/unit.
Every ratio gate carries a non-vacuousness floor on the base counter so
broken counter wiring fails loudly instead of green-washing
(cbm_pxc_count_perfile_defs feeds the overlay path into the same
counter the fallback path already used; wired for Java, extend to the
TS overlay when touching ts_lsp).
Dynamic coverage: languages iterate CBM_LANG_COUNT; embedded templates
cover the LSP-hybrid languages, and tests/fixtures/complexity/<lang>/
dirs are auto-discovered so a new language joins the guard by dropping
fixtures. Uncovered languages are listed in the report with the reason.
The local report additionally carries per-language node/edge counts with
ratios and a per-pass elapsed_ms table per run (captured via a TEE log
sink during the in-process pipeline runs) — trend data for humans, still
never a gate.
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Six reports describe the same dead end: the indexing worker dies, the
supervisor points at `logs/.worker-<pid>.log`, and the file is 0 bytes.
Nothing was ever flushed, so not one of #1070, #1130, #1132, #1133,
#1145 or #1450 is reproducible or attributable -- the hint says "crashed
on a file" and the file is never named.
Root cause is buffering, not the crash. The worker's stderr is redirected
to that log file by the supervisor, and every diagnostic goes through
`fprintf(stderr, ...)` with no flush. The C standard only promises stderr
is "not fully buffered"; the Windows CRT gives a redirected stderr FULL
buffering, which is why five of the six reports are Windows. A worker
that aborts, is SIGKILLed by the OOM killer, or is terminated after a
hang takes its whole buffer with it.
- cbm_log_set_crash_durable(): setvbuf(_IONBF) plus a per-line flush in
emit_line. Both, deliberately -- setvbuf covers every writer to the
stream including the plain fprintf startup errors that explain a
worker which never got as far as logging, and the flush covers the
case where setvbuf is refused because the stream was already written
to. Enabled for the worker role only, claimed in main() before the
process writes anything.
- cbm_index_worker_log_begin(): a startup header written and flushed as
the worker's first act -- version, build fingerprint, pid, repo path,
and the worker's own arguments. A control record rather than an info
line, so CBM_LOG_LEVEL cannot restore the 0-byte log.
- Under a crash-durable log the parallel extract pass logs every file
it starts, not just the first two rounds of workers, so the log ends
with the files that were in flight when the worker died. One line per
file, never per node; unchanged for every non-worker caller.
This fixes no crash. It converts six unreproducible reports into reports
we can act on, and every future one arrives with the run identified.
Test: a worker started with a fully buffered stderr -- the state the
Windows CRT hands it, forced on POSIX so the contract binds on all three
legs -- writes diagnostics and is then SIGKILLed. The retained log must
be non-empty, carry the header with the version, pid, repo path and args,
and still hold the line written after it. Reverting the production change
leaves that log at 0 bytes.
The probe kills rather than aborts because Darwin's abort() runs the
stdio cleanup handler and flushes the very buffer the repro depends on
stranding: under abort the reverted build still produced a populated log
and the test was quietly toothless. SIGKILL runs no cleanup, and is
literally #1070's death (signal=9).
Refs #1070
Refs #1130
Refs #1132
Refs #1133
Refs #1145
Refs #1450
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The CLI resolves each client's config home from its own environment
variable before falling back to $HOME — cbm_codex_config_dir() honours
$CODEX_HOME and ignores the home_dir it is handed. A test that redirects
only HOME therefore does not isolate those clients: it resolves, reads
and WRITES the developer's real configuration.
On a machine with $CODEX_HOME exported (any Codex or Orca user) 27 of
269 cli tests failed for that reason alone. Every one of them asserted a
successful rc while the CLI was reporting genuine failures against the
ambient config: an unbalanced managed marker left in the real
config.toml, an AGENTS.md that is a symlink and is refused by the
O_NOFOLLOW writer, and user-modified agent profiles that are correctly
preserved. The behaviour under test passed in all of them; only the rc
was poisoned. Worse than the red: the suite mutated that real config,
installing and removing SKILL.md under the user's Codex home.
The individual tests that already neutralize CODEX_HOME are exactly the
ones that pass, and the file carries a comment admitting one suite was
"green only via env leaked from earlier suites". Rather than extend those
ad-hoc per-test lists again, neutralize the whole inventory once in the
runner, and reuse the inventory the shell fixtures are already required
to satisfy.
tests/test_smoke_fixture_contract.sh grows one check so the two cannot
drift: a client isolated in the smoke scripts but forgotten in the C
runner now fails the contract.
Verified on macOS arm64 with $CODEX_HOME and $OPENCODE_CONFIG_DIR
exported: cli went from 241 passed / 28 failed to 268 passed / 0 failed,
with no change outside tests/.
Signed-off-by: Gabriel Gonzalez Pineda <64bits@gmail.com>
Adds cbm_workspace_classify_root, the breadth half of the workspace
boundary. It answers "is this path obviously too broad or too sensitive to
index as one root", not "is this caller allowed to index it" — that second
question needs the user-level grant store, since it is the only input a
caller cannot write. The header says so, because a reader who mistakes one
for the other would draw the wrong conclusion about what is enforced.
Three rules, each doing what the others cannot:
- Depth below the volume. Two components on POSIX refuses every top-level
tree in one rule with no list to maintain. Windows and UNC count
drive/share-relative and require one, because there the first component is
already user space ("D:/repos") — the system trees that sit at the same
depth ("C:/Windows") are covered by name instead, which is tractable
because that set is small and stable.
- The home directory itself, which is two deep on macOS and Linux and so is
invisible to depth.
- Credential directory names, matched against every component so both
"…/.ssh" and "…/.ssh/keys" are refused. This list is deliberately additive
so extending it needs no design authority.
A path that is or contains the cache directory is refused outright: indexing
such a tree would pull every other project's graph database into this
project's index.
Classification order is load-bearing and commented as such. The home
directory normally contains the cache directory, so testing the cache first
reported every home as "holds the cache" and made it non-overridable, which
contradicts the intent that a person may override it. Depth precedes the
cache test for the same reason: "/Users" satisfies both and "too broad" is
the reason that helps the reader.
Takes home and cache as parameters rather than reading the environment, so
the policy is a pure function and the tests need no filesystem. Twelve table
tests; three of them failed on the first implementation and drove the
ordering fix above.
Not yet wired to any caller — deliberately, so the policy can be reviewed on
its own.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.
Daemon lifecycle:
- daemon start/stop/status subcommands. `daemon start` launches a
PERMANENT daemon (spawn shape is byte-exact argv; survives idle
periods and session ends) and reports an already-active daemon
instead of failing. Permanence is honored at every stop latch:
last-committed-client disconnect, host initial-client window,
coordinator release, and application final-session close — a
permanent daemon also keeps admitting new sessions after its last
one closes.
- daemon stop refuses while sessions are active and lists the blocking
peers (pid/role) that must finish first; an idle daemon drains
through the activation-shutdown machinery with the ACK ordered after
connection interrupts. A second stop is idempotent. The wire ops are
no-cohort first-frame requests with peer fingerprint authentication,
so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
keep their local supervised path). A cold CLI run that had to spawn a
temporary daemon prints a hint that `daemon start` removes the
per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
emits a visible, rate-limited notice (Claude-dialect systemMessage
plus stderr for other dialects) and always exits 0 — augmentation is
never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
older-build permanent daemon (strict semantic-version triples only;
dev builds never auto-drain) and the build-conflict message names
`cbm daemon stop` as the manual escape hatch.
Windows IPC/runtime (real-VM verified):
- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
path used to destroy its listening pipe instance on every 20 ms poll
timeout; a client attaching in the teardown window was severed or left
on an orphaned pipe object whose HELLO no server handle could ever
read, absorbing the connect until the client's own timeout expired.
The pending connect now survives poll timeouts and nothing is
destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
server handle can discard a just-sent response before the peer reads
it (POSIX stream sockets never lose buffered data on close). A bounded
cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
immediate on interrupted connections) now precedes close in
runtime_worker_finish and runtime_reject_inline, so hello-conflict,
capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
transport half-close, so close_begin now announces departure with an
explicit frame (ordered after APPLICATION_CANCEL, before the local
interrupt); the server releases the client's admission on receipt
instead of waiting for the handle to close. Admission-drop timing is
now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
request publication under the send lock, best-effort sends the active
token's APPLICATION_CANCEL frame, then interrupts local I/O; the
server cancels MCP/subprocess work promptly. Contract tests accept
both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
the requester's license to act on "snapshotted and draining", so every
connection interrupt is now initiated before the ACK is sent; a
session could previously get one more request serviced after the
requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
timer granularity) burned ~1.6 s against permanently obstructed paths,
stalling hello rejections past the client's timeout. The retry budget
is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
nonblocking send() on Windows is absorbed wholesale into AFD kernel
buffering regardless of SO_SNDBUF, so send deadlines and interrupts
could never engage against a slow peer (and the full payload was
pinned in nonpaged pool). Slicing restores a deterministic
backpressure point; a test hook pins SO_SNDBUF for the deadline and
interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
deadline, explicit connection states, refusal to free a server while
a listener-owned connection is active.
Windows long-path support:
- Central path-aware wide conversion (canonicalize via GetFullPathNameW
and prepend the extended-length prefix for absolute paths >=240) at
the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
store opens, and the daemon build-fingerprint/log paths. Deep managed
installs (a 64-hex generation directory routinely exceeds MAX_PATH)
now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
ancestry validators now operate in the extended-length namespace;
the launcher path is canonicalized (and prefixed when deep) once at
entry so every downstream exact-string comparison stays
form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
GetFileAttributesW) so non-ASCII install paths survive argv[0]
resolution.
Windows launcher install/uninstall transaction:
- FileRenameInfoEx names are NUL-terminated in an over-allocated
buffer. FileNameLength governs per the contract, but filter drivers
read FileName as NUL-terminated and appended adjacent heap bytes to
created names — a flaky, garbage-suffixed rename target. Both the CLI
and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
.cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
chars so the bare rename target stays under the FileRenameInfoEx
NT-conversion ceiling at guard depths; 64 bits still uniquely
identify the generation.
- When the running launcher's mapped generation backings pin .cbm
against rename, the backings are relocated to activation-<pid>-N
.retired tombstones beside the install (a mapped image may be renamed,
never deleted; the launcher's liveness-guarded sweep reclaims stale
tombstones). Every relocation is recorded, and a FAILED uninstall
reverses the moves after restoring .cbm — via MoveFileExW with
extended-length paths on both arguments, since the deep generation
target is beyond the handle-based rename's bare-path reach — so a
restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
out so the tree is shallow enough for the detached cleanup's rd, and
the cleanup's working directory strips the extended-length prefix
(CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
(CopyFileW destinations, CREATE_NEW tombstones, probe directories)
are explicitly owner-stamped so the exact-owner validators hold on
runner images; guard fixtures stamp hand-built trees the same way.
Diagnostics, tests and infra:
- diagnostics: discovery is now an always-delivered JSON control record
(new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
paths containing spaces; placement honors $TMPDIR with /tmp fallback
via a diagnostics-local helper; the soak parser reads the JSON record;
documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
log levels, TMPDIR-with-spaces, and native Windows output-contract
assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
parents with production-shaped ancestry (LocalAppData on Windows, via
th_secure_runtime_parent_new) — the runtime ancestry validation
correctly refuses temp roots whose ancestors grant mutation rights to
Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
drive the documented startup-owner publication flow before reading
generation-bound endpoint addresses. This turns the 26 Windows
failures previously visible in CI's full-test job green without
weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
contract including failed-uninstall restore and immediate reinstall
after uninstall; new daemon lifecycle and reworked hook-augment
guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
(it was set by CI but never read, so perf suites ran everywhere);
four throughput/bench suites are classified as perf, the heavy
store_arch suite moved to the slow-timeout tier, and two
wall-clock-sensitive assertions were rewritten as invariant checks
with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
contract, soak daemon-recovery contract, path-safety helper, the
Windows VM worktree-sync contract wired into scripts/test.sh, and
vm/win.sh guards building its clean embedded-UI product in an
isolated BUILD_DIR so it cannot clobber the incremental test build.
provision-windows.sh now installs Node.js for the guards UI build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every spawned worker/daemon re-hashes the entire executable image for
the exact-build fingerprint; for the ASan test-runner (hundreds of MB)
that is tens of seconds per spawn on constrained CI runners and was the
sole cause of the whole daemon-family readiness-timeout flake class
(index_supervisor, daemon_frontend, cli, and much of the Windows
daemon surface). Add a test-only seam under CBM_CLI_ENABLE_TEST_API:
when CBM_TEST_BUILD_FINGERPRINT holds a valid stub the hash is skipped.
test_main.c sets it once; every forked child and re-exec'd worker
inherits it, so exact-build match AND mismatch behaviour is preserved
(a mismatch test still passes a different fingerprint via argv). Never
compiled into production. Local mac run: index_supervisor 6/6, the four
daemon suites 111/0, exact-build test green. The generous frontend
catastrophic-alarm values are kept as backstops.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The runner executed all 104 suites sequentially (~10 min locally, the bulk
of every CI test leg). The suites are process-isolated already (per-process
mkdtemp HOME sentinel; two full parallel runs produced zero cross-suite
failures), so the serialization was pure convention.
- test-runner --list-suites: prints every registered suite, one per line,
emitted by the SAME macro table that executes suites — the list cannot
drift from the run set by construction.
- scripts/run-tests-parallel.sh: runs each suite as its own process
(jobs = CPU count; CBM_TEST_PAR_JOBS overrides) under a ZERO-LOSS
CONTRACT: a union guard fails the gate if the set of suites that
produced a result differs from --list-suites (nothing can be silently
dropped, a newly added suite is picked up automatically); per-suite
pass/fail/skip are summed into the sequential runner's exact summary
format; any suite crash, failure, or omission exits nonzero. Per-suite
wall times are printed for balance tracking.
- make test-par: the parallel target. make test (sequential) is unchanged
and remains the escape hatch (CBM_TEST_SEQUENTIAL=1 in test.sh).
- scripts/test.sh: builds, then routes through test-par — every CI test
leg gets the speedup with zero workflow-topology change (same jobs,
same gates, same billing; the legs just finish sooner).
- The stack_overflow suite is split into a/b/c (7+6+7 of its 20 tests,
pure re-registration): as one suite it was the wall-clock critical path
of any parallel run — every other suite finished underneath its ~4
minutes.
Measured locally (Apple Silicon, ASan runner): sequential ~10 min vs
parallel 232 s, totals identical (6,361 passed / 0 failed / 1 skipped),
union guard clean. The TSan leg keeps its dedicated subset runner
(sequential) — out of scope here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Run the test process under a private temporary HOME and clear inherited CBM_CACHE_DIR so production index paths and legacy fixtures resolve to the same isolated cache.
Make lang-contract fixtures honor an explicit cache override and remove the sentinel tree at process exit, including after early assertion returns.
Validated on Windows with the full runner: 5828 passed, 22 skipped; a 500-project user cache had zero files added or removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: tmonestudio <tmonestudio@users.noreply.github.com>
Replace the plan-01 placeholder with the full Perl LSP test suite covering the
ten foundational resolution scenarios from 22-RESEARCH.md, plus the extract_perl
/ find_resolved / require_resolved / find_resolved_with_strategy helpers cloned
from test_php_lsp.c:
1. method via bless-assignment 6. use parent MRO
2. constructor class-method type 7. use base MRO
3. static package call 8. Exporter import (use Mod qw(f))
4. $self method dispatch 9. require fallback
5. @ISA inheritance 10. unresolvable receiver -> zero edges
Assertions match the resolver's actual QN scheme (module_qn.subname, dotted —
the Perl package governs dispatch, not the emitted QN), and the negative test
confirms the zero-edge guarantee for untyped scalar and unindexed package
receivers. Register suite_perl_lsp in test_main.c so scripts/test.sh runs it.
All 10 perl_lsp tests pass (3563 passed, 1 pre-existing unrelated failure:
search_code_multi_word in tests/test_mcp.c).
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Files whose parse tree contains tree-sitter ERROR/MISSING regions were
silently indexed as if complete — constructs inside those regions are
absent from the graph with no signal anywhere (classic trigger: the
preprocessor-blind #ifdef-split-brace pattern in C).
- cbm_extract_file: set parse_incomplete and record the 1-based line
ranges of the top-most error regions (bounded at 64), serialized as
"start-end,start-end" into the result arena
- pipeline: record such files under the new phase "parse_partial"
(distinct from skips — the file IS indexed) in both the parallel and
sequential extraction paths; stamp the File node with
{parse_incomplete:true, error_ranges} on full and incremental runs
via a new cbm_gbuf_set_node_props (upsert survivor rules bypass)
- mcp: report parse_partial {files, count, truncated, note} separately
from skipped[] (skipped_count semantics unchanged); the per-run
logfile lists parse_partial rows; the index_repository tool
description documents the signal and the query_graph pattern to find
flagged files
- tests: new parse_coverage suite (RED-first, 7 cases incl. false-
positive guards and the region cap) plus an end-to-end resilience
guard asserting response fields, logfile, File-node marker, and the
advertised query_graph query
The signal is explicitly best-effort: a flag means constructs in the
listed ranges were dropped (prefer grep there); the absence of a flag
is NOT a completeness guarantee.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The supervised-worker fast path deliberately skips the pipeline
teardown (the OS reclaims the process wholesale) and pairs that with
_Exit, which bypasses atexit/LeakSanitizer. The production worker in
run_cli has both halves; the test binary's worker emulation
(tf_maybe_run_index_worker) only had the first: it returned normally
through main(), LeakSanitizer ran at exit, reported the intentionally
unfreed pipeline and forced exit code 1 — so the supervisor read a
healthy index as worker_failed and IDX832 went red on exactly the
platforms whose ASan builds run a leak pass (all four Linux legs;
macOS/Windows ASan have none). Reproduced and verified fixed in an
ubuntu:22.04 arm64 container: reap outcome=clean, mcp suite 140/0.
Mirror the production exit: deliver the response file, fflush, _Exit.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes the gap flagged in the distill review (test_security.c: "NOT proven
here: the full UI repro — listening socket + MSYS2 handle walk under a
single-threaded server"). Test-only; no production code changes.
Guard 1 — deterministic socket-isolation probe (any Windows, RED-able):
popen_isolates_listening_socket (tests/test_security.c) opens a real
inheritable AFD/listening socket — the exact handle class that deadlocked
git in #798 — then spawns THIS binary through cbm_popen (a cmd.exe
grandchild, git's spawn shape) in a new __cbm_sockprobe re-exec mode
(tests/test_main.c). The child reports via exit code whether that socket
handle is live in its address space: isolated spawn -> getsockopt fails ->
exit 0 (GREEN); raw-_popen regression leaks it transitively through
cmd.exe -> getsockopt succeeds -> exit 42 (RED). Verified RED with a local
spawn-inherits-all revert, GREEN with the isolated spawn.
Guard 2 — end-to-end liveness under live UI sockets (tests/test_httpd.c):
- ui_server_list_projects_responds_under_watchdog: POST /rpc list_projects
to the running single-threaded UI server with a client SO_RCVTIMEO
watchdog; a wedge -> no 200 -> hard FAIL, never an infinite CI hang.
- git_context_resolve_no_hang_under_live_ui_sockets: while the UI server
holds live listening/AFD handles in-process, run cbm_git_context_resolve
(the exact path list_projects takes) on a worker thread under a
WaitForSingleObject(30s) watchdog; a hang -> FAIL. SKIPs where git cannot
init a repo via system().
All three run in the existing security/httpd suites on the test-windows CI
leg; no new build wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018g9f9jr5S46BfKaDvDykBp
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Martin Vogel