Codex sanitizes stdio MCP subprocess environments to the names listed in
env_vars. Since #1645 CBM_RUNTIME_DIR relocates the daemon rendezvous, so a
Codex subprocess that does not receive it looks for the daemon in the DEFAULT
location and never finds it — the same silent client/daemon split
CBM_CACHE_DIR caused in #1562. Both names decide WHICH daemon a process talks
to and are now forwarded unconditionally (forward-if-present semantics);
behavioural knobs (log level, workers, budgets) deliberately stay
unforwarded — that broader list remains #1664's open enhancement question.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
gotspatel's live opencode.json stores our entry with backslashes
(`C:\...\codebase-memory-mcp.exe`) while the installer compares its own path
with forward slashes — the same file on disk, refused over the separator
spelling, so op=mcp_install failed on a correctly-installed machine (and on
Windows the dead-path probe rightly reported the binary PRESENT, which turned
the mismatch into a hard refusal).
Ownership comparison now treats `\` and `/` as equal everywhere and folds
case on Windows only, where the filesystem is case-insensitive; POSIX
byte-exactness otherwise holds. The annotated entry that names this binary is
recognised as already satisfied and preserved byte-for-byte.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
PowerShell 5.1's `Set-Content -Encoding UTF8` writes a BOM, so real
Windows-authored Hermes configs start with EF BB BF — and both edit ops
failed content-independently (the reporter's 26-byte reproduction is their
23-byte file plus exactly this BOM; reproduced RED on macOS with the same
bytes, so the platform was never the variable).
The document read now validates past a leading BOM and yaml_doc_init treats
it as a prologue: the first line's structure starts after it, the key lookup
still sees our own section when the BOM immediately precedes it (guarded by a
dedicated no-duplicate-section test), and every edit splices interior ranges,
so the BOM survives writes byte-for-byte. Non-document inputs — keys, entry
blocks, identity scalars — keep the strict no-BOM rule.
Also makes the moved-entry cli test fixture platform-correct: the Windows
dead-path probe can only prove a fixed-drive path absent, so the Windows
branch uses one; a POSIX-shaped path is refused there by design.
Fixes#1656.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two install failures with the same root theme — the entry we write is a
compatibility contract with the agent's parser, and both sides of that
contract needed repair:
goose (#1675): ExtensionConfig::Stdio declares `name` as a required serde
field with no default, and goose's loader silently drops entries that fail to
deserialize — install reported success and the extension was invisible. The
goose block now carries `name: codebase-memory-mcp`; the non-goose YAML
schema stays name-free. A CBM_CLI_ENABLE_TEST_API seam asserts the exact
block bytes per schema.
annotated MCP entries (#1630, the deferred field-merge): dbd20eaa recognised
an entry the client annotated ("enabled": true beside our command/type) but
could only leave it untouched, because replacing the whole entry would drop
the client's keys. config_json_like gains
cbm_json_like_replace_field_raw_if_unchanged — splice ONE member's value,
preserving every other byte (comments, ordering, client keys) — and the
upsert flow uses it on the two AUTHORIZED repair channels only:
- a relocating update (the entry names the previous managed binary), and
- the existing Windows dead-path probe, which previously fell back to a
wholesale rewrite and lost the annotations.
POSIX keeps its doctrine unchanged: a config-supplied path is never trusted,
so a moved-looking entry without that authority is preserved byte-for-byte
and install fails loudly (cli_editor_mcp_preserves_unrecorded_posix_absolute_
entries_without_probe holds). All repair/refusal paths are covered by tests
proven RED on the unfixed flow.
Fixes#1675.
Fixes#1630.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Four legal-YAML constructs from the reporters' real Hermes configs made
`install` fail permanently (any one of them aborted mcp_install and/or
pre_llm_hook_install):
- exact empty flow collections as values (`plugins: []`, `tool_choice: {}`)
— now validated key-only in both the mapping-body and sequence document
scans, mirroring the #1673 empty-mapping exception; non-empty flow
collections stay rejected.
- block sequences at the same indent as their mapping key (column-0 `- item`)
— item lines directly after a value-less key are structure, not malformed
keys, in the root walker, the key matcher, and the sequence mapping-range
walker.
- double-quoted scalars continued across lines with a trailing `\` — the doc
loader now precomputes per-line continuation flags; continuation lines are
value bytes every structural walker skips, and a document ending inside an
open continuation stays an error.
- mid-word quote characters in plain scalars (`LET'S`) — quotes are scalar
indicators only at a node start (range start, after `:`, after `-`),
exactly like the #1639 anchor/alias rule; real quoted values keep their
protection.
Byte-identity alone also froze users on canonicals older releases wrote:
galaxy's entry had `command:` unquoted, and the goose block gained `name:`
(#1675), so the existing entry was declared FOREIGN forever. An entry under
our key now repairs when it parses as a known prior shape (single command
line, or the pre-name goose block) with a codebase-memory-mcp[.exe] command
basename; anything else stays FOREIGN and the file untouched.
End-to-end: both reporters' full configs (iandol 15.6 KB, galaxy 15.2 KB) now
install with zero agent_config errors, every original line byte-preserved,
and the goose upgrade path rewrites the old block in place. Each construct
carries a distilled regression test proven RED on the unfixed editor.
Fixes#1631.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
ts_eval_expr_type and ts_signature_for_call are mutually recursive: resolving
a call evaluates its argument expressions once per lookup path (method
dispatch + namespace fallback), and in tsc-compiled spread files the first
argument is itself the next nested Object.assign(...) call — the same subtree
re-evaluates once per enclosing level, 2^n total. The TS suite's
objectSpreadRepeatedComplexity.js (3.6 KB, 5 nodes) measured 20.4 s; with the
memo its eval cost is zero within measurement noise of a one-file control,
and the microsoft/TypeScript corpus drops 37.5 -> ~24 s warm (nodes
byte-identical, edges within the known scheduler jitter).
Expression types are position-pure within a file pass (one node = one scope
path; the per-file walk is single-threaded and deterministic), so one eval
per node is the correct semantics, not a cache trade-off. The memo is a
per-file, arena-backed, linear-probe table keyed on TSNode.id. Results
produced under a depth-cap or budget bail are never stored: both bail sites
bump a degradation counter, and a store only happens when the subtree
completed clean — a degraded UNKNOWN can therefore never shadow a later full
evaluation.
The regression guard asserts work, not wall-clock: the nested-Object.assign
shape must complete without exhausting the deterministic eval budget, read
back through a new CBM_ENABLE_TEST_SEAMS accessor pair. The seam lives in the
lsp_all unity object, so GRAMMAR_CFLAGS_TEST/TSAN now carry the seams define
(test artifacts always have seams; prod never does). Verified RED without the
memo (budget exhausted, suite 47.8 s) and green with it (3.2 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The no-skips lint gate (scripts/check-no-test-skips.sh) rightly rejected the
throughput-report test's two SKIP() calls:
- CBM_SKIP_PERF=1 is deliberate operator configuration, not a hidden
environment failure: reporting is off by request, so the test PASSes with a
stderr note instead of skipping.
- an uncreatable report dir IS an environment failure and now FAILs with the
remedy in the message (set CBM_COMPLEXITY_REPORT_DIR), per the policy text.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Finding #1669 took an 11-corpus A/B across two release binaries. This
suite makes that bug class fail a unit test in seconds, on every
platform, from tiny corpora.
Method: build k and 2k REPLICATED module copies, run the full in-process
pipeline on both, assert counter RATIOS. Independent copies mean every
extensive quantity — nodes, edges, Σ per-file registry defs — must grow
linearly (ratio ~2). A files x corpus coupling makes per-file work itself
grow with k and lands at ratio ~4. Ratios expose the exponent regardless
of absolute scale, so 60-120 files suffice.
Verdicts are pure functions of (code, input): gates ride ONLY on
deterministic work counters and data-product counts, never on wall time.
Throughput (nodes/s, edges/s) is information-only, written to
private/benchmarks/complexity-<ts>.json (local, gitignored; skipped
under CBM_SKIP_PERF where rates are meaningless).
Two corpus shapes, both needed:
- Independent modules (java/py/go/ts templates): catches cross-module
contamination and dedup breakage. The #1669 bug is GREEN here — fully
closed modules filter perfectly, which is exactly why it survived.
- The growing shared package (bigpkg): one Java package whose file count
scales with k — the real-repo shape (files concentrate in large
packages). The JVM namespace filter branch makes per-file work track
package size, so this corpus is the honest #1669 reproducer:
ratio 4.00 RED on the pre-fix tree, 4.00 RED for a module-scoped
overlay, 2.00 GREEN for the own-file overlay. It discriminated the
correct fix design before the fix was written.
Both legs of every pair exceed MIN_FILES_FOR_PARALLEL(50): below it the
sequential path runs, which builds no shared registries and would be the
wrong code path to gate (its per-file cost is bounded by the 50-file
ceiling).
Recorded but deliberately NOT gated, with reasons at the case:
tail_candidates and fallback_rows are legitimately superlinear under
replication until those scans are bounded, and measured ~1 ns/unit.
Every ratio gate carries a non-vacuousness floor on the base counter so
broken counter wiring fails loudly instead of green-washing
(cbm_pxc_count_perfile_defs feeds the overlay path into the same
counter the fallback path already used; wired for Java, extend to the
TS overlay when touching ts_lsp).
Dynamic coverage: languages iterate CBM_LANG_COUNT; embedded templates
cover the LSP-hybrid languages, and tests/fixtures/complexity/<lang>/
dirs are auto-discovered so a new language joins the guard by dropping
fixtures. Uncovered languages are listed in the report with the reason.
The local report additionally carries per-language node/edge counts with
ratios and a per-pass elapsed_ms table per run (captured via a TEE log
sink during the in-process pipeline runs) — trend data for humans, still
never a gate.
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Finding the 0.10.x indexing regression took an 11-corpus A/B across two
release binaries, a subset-scaling series, and a per-pass exponent fit.
None of that should have been necessary: the pass that carried it,
cross-file LSP, was already timed and already logged its def count. The
numbers were there, just never normalised into something a reader could
judge.
Three additions, all shipped without a flag except the detailed curve:
- parallel.resolve.cross_lsp_cost — cross-LSP cost NORMALISED per file,
next to defs_total. Wall time cannot separate "big repo" from
"superlinear pass"; us_per_file can. Measured on one Java tree:
files=3710 defs=102845 us_per_file=35129 us_per_file_per_kdef=341
files=14833 defs=339866 us_per_file=106722 us_per_file_per_kdef=314
Per-file cost tripled while per-kdef stayed flat — the fingerprint of
work proportional to the whole corpus (files x defs). One grep on two
differently sized repos now answers what previously took a two-binary
bench.
- parallel.resolve.scan_cost — candidates visited per tail-match lookup,
plus fallback_rows, which the code has counted since #1085 but exposed
only to a test. On the same tree the tail scan reached 242M candidate
visits (n^2.04), worth seeing even though it proved cheap in wall time.
- cbm_scale_probe (foundation/profile.h) — samples cumulative elapsed at
1/8, 1/4, 1/2 and 1 of a pass's items and fits k in T ~ n^k, warning
once k reaches 1.35 in shipped builds. Wired into parallel_extract and
parallel_resolve.
The probe's limits are documented rather than oversold: it catches growth
WITHIN a run, and would NOT have caught this bug, whose per-item cost is
constant-but-large within any single run (it reported 1.26 while the
cross-corpus exponent was 1.86). That is exactly why us_per_item and
us_per_file are emitted alongside it.
Tests are deterministic by construction: the exponent fit is a pure
function fed synthetic points, and the checkpoint bookkeeping is asserted
directly. A test that proved the detector by generating a real quadratic
workload would be asserting on the scheduler.
No product behaviour changes; diagnostics only.
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Since #1360 routed ordinary malloc/new through mimalloc on Linux, the
arena policy governs every allocation in the process rather than just the
bound sqlite/tree_sitter populations. cbm sets arena_eager_commit=0, so
mimalloc commits sub-ranges with mprotect(PROT_READ|PROT_WRITE) over a
PROT_NONE reservation, and each partial commit SPLITS the reserved VMA.
Measured on the Go corpus, Linux arm64, shipped binaries:
v0.9.0 10 mappings, at ANY worker count
v0.10.5 ~22k mappings, peak; the count tracks CONCURRENCY
(999 at 1 worker, 8460 at 4, 11965 at 18)
Two consequences, both of which #1654 reported from a 96-CPU/376 GB host:
the mmap/mprotect churn serialises on the kernel's per-process mmap_lock,
and the VMA count climbs toward vm.max_map_count, after which mmap fails
for ANY size -- so mimalloc reported it could not allocate 10 KB while
`free -g` still showed 246 GB available.
mimalloc's own default for this option is 2, meaning "eager-commit arenas
only on an OS that overcommits (i.e. linux)", precisely because commit is
free there until pages are touched. Overriding it to 0 opted Linux out of
the default written for Linux. Restore it on Linux only; every other
platform keeps the lazy setting, where commit is NOT free and the
upfront-memory reason still holds (Windows especially, #581).
Measured effect, same corpus and host, baseline build vs this build:
mappings 22450 -> 17312 (-23%)
wall 92.4s -> 92.6s (unchanged)
peak RSS 19.14 -> 19.22 GB (unchanged)
This is a partial mitigation, not a cure: the remaining ~17k mappings are
individual 64 KB-3 MB extraction buffers, each taking its own mmap (the
worker reserves ~40 GB of address space for ~19 GB of RSS). Pooling those
is the durable fix and is deliberately left out of this change.
Guard: mem_arena_eager_commit_follows_platform_commit_cost pins the
platform split so the Linux default cannot be silently opted out again.
Reproduction and controlled 2x2 (only vm.max_map_count varied) are
recorded on #1654.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
LeakSanitizer on CI caught all five new tests leaking their CBMFileResult:
Indirect leak of 24 byte(s) ... ts_tree_new
cbm_extract_file_ex cbm.c:1256
do_extract test_parse_coverage.c:39
test_dockerfile_missing_final_newline_not_flagged_issue1610:272
SUMMARY: AddressSanitizer: 706504 byte(s) leaked in 189 allocation(s)
Every pre-existing test in this suite calls cbm_free_result before PASS; the new
ones did not. The local run could not have found it - LeakSanitizer reports
"detect_leaks is not supported on this platform" on macOS arm64, so this class
of defect is CI-only here.
Each test now captures what it asserts, frees, and only then decides, so the
early-FAIL paths do not leak either. The cross-grammar loop prints its
diagnostic before freeing so the failure message keeps naming the grammar.
While correcting the guard, a first attempt left ASSERT_TRUE(flagged ||
has_ranges || true) in real_error_before_eof_still_flagged - always true, and it
would have silently disarmed the guard that stops the EOF suppression from being
over-broad. Removed. The guard is re-proven binding: forcing
cbm_is_eof_terminator_miss to return true makes EIGHT tests fail, including both
guards, and restoring it returns the suite to green.
parse_coverage 14 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A file that does not end with a newline leaves the grammar's mandatory line
terminator MISSING. cbm_collect_error_regions counted that node, so the file was
reported parse_partial with the last line as its error range.
It is not a miss. The node is ZERO-WIDTH and sits at EOF: the parser consumed no
source for it, so by construction nothing was dropped - no construct can live in
a zero-byte span - and every real instruction above it parsed normally. Proven
by dumping the tree: the reporter's two-line Dockerfile yields
(source_file (from_instruction ...) (entrypoint_instruction ...) (MISSING "\n"))
with both instructions intact and the MISSING node spanning bytes 73-73.
It was never Dockerfile-specific. Stripping the trailing newline from the 156
linkable grammar fixtures flips 13 of them to has_error, and SIX produce regions:
dockerfile, tcl, fish, gomod, hyprlang - and makefile, which is a genuinely
different case (its ERROR has WIDTH; the recipe really is lost).
Worse, the ones that stayed silent did so for no principled reason. ini, fsharp,
beancount, requirements, gitignore, sshconfig and kconfig omit the same
terminator, but theirs is a HIDDEN node and hidden nodes are invisible to
ts_node_child(). Whether a user was told their file was partially parsed came
down to whether that grammar's author declared the terminator visible.
The cost was not cosmetic: a phantom parse_partial writes a "<project>::missed"
shadow row, and until #1609 that row made the project fail cross-repo validation
as BOTH source and target. A single absent byte could remove an entire
repository from cross-repo intelligence with no error shown anywhere.
The suppression is deliberately narrow - zero-width AND at EOF. A MISSING or
ERROR node with width still counts even at EOF, and anything before EOF is
untouched. Both callers pass the raw root, so one source_len is correct for
both; verified rather than assumed, since root is bound once and never
reassigned.
Reported by @vitaliy-shatskiy, who could not share the original file and instead
rebuilt the property from scratch with a byte-exact script - an editor would
have silently re-added the newline and hidden it. Their isolation matrix ruled
out BOM, CRLF vs LF, exec-form vs shell-form and file length before we looked at
it once.
Reproduce-first, revert-checked: the Dockerfile and cross-grammar tests fail on
the previous tree and pass with the fix; forcing the new predicate to return
false brings the identical REDs back. Two guards pin the boundary and hold in
both directions - a width-bearing failure at EOF (makefile) and a real
mid-file ERROR in a file that ALSO lacks its final newline (built from
C_IFDEF_SPLIT, the fixture this suite already proves is flagged).
parse_coverage 14, extraction 276, language 217, infrascan 3,
grammar_regression 1 - 511 passed, 0 failed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`update` hands off to install.sh (install.ps1 on Windows) and prints the command
to run. It built that command from cbm_detect_self_path - the BINARY's directory
- and treated "I resolved my own location" as "the installer is beside me".
Those are different questions. install.sh is placed beside the binary by
install.sh itself, but a binary that was moved, packaged by a distro, or built
from source has no installer next to it. We printed the path anyway:
bash "/home/<user>/.local/bin/install.sh"
/usr/bin/bash: /home/<user>/.local/bin/install.sh: No such file or directory
Reported on discussion #1560 (#1632) by a user who was already three releases
deep in install trouble and had just been told, by us, to run a file that does
not exist.
`update` exists to tell someone how to proceed. Ending the interaction on a
command that cannot run is the one outcome it must not produce - and the
fallback text was already there and already correct, naming install.sh as
shipping in the release archive without asserting a path.
The probe goes through cbm_path_info_utf8 so a non-ASCII install directory
resolves on Windows, and rejects a DIRECTORY of that name, because `bash <dir>`
is not a command either. A symlink still counts: it is reported rather than
followed, and the shell runs it perfectly well.
The Windows branch gets the same treatment; it had the identical assumption
about install.ps1.
Reproduce-first and revert-checked: with the probe forced to return true - the
old behaviour - the new test fails with "a directory with no installer must not
be named as one", and passes once it is restored. cli: 276 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Indexing writes an internal "<name>::missed" miss-graph row into the SAME db as
the primary project whenever a file parses partially. cr_store_has_exact_project
required `count == 1` over ALL rows returned by cbm_store_list_projects, which
does not filter those rows - so any project that had ever recorded a parse miss
failed validation, as SOURCE and as TARGET, and the whole feature reported:
project is not indexed
for a project that plainly was. There is no user-level workaround: a partial
parse is not something the operator controls, and re-indexing reproduces the
shadow row.
This is the same defect mcp.c fixed for list_projects in #1044 ("requiring
n == 1 over ALL rows made every project with a miss graph vanish"); the
cross-repo site never learned it. The fix ports that primary-row filter.
The single-primary requirement itself is deliberately kept: it is what proves
the db belongs to the project we were asked about rather than being a shared or
mislabelled store. Only "::" shadow rows stop counting toward it.
Reported by vitaliy-shatskiy in #1609, whose diagnosis named the exact function
and the exact reason.
Reproduce-first, and revert-checked both ways:
- the new test fails on origin/main with `ASSERT(!(result.failed))`
(tests/test_cross_repo.c), for the behaviour under test rather than a setup
error;
- it passes with the fix;
- reverting ONLY src/pipeline/pass_cross_repo.c and keeping the test brings
the identical RED back, so the test binds to the production change.
The existing pair without shadow rows is the control: those tests already prove
that path returns edges, so this cannot pass vacuously on a fixture that never
matched.
cross_repo 8 passed; pipeline 249 passed; store_edges 25 and store_nodes 67
passed - no collateral change.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`codebase-memory-mcp uninstall --help` removed the binary and every agent
configuration (#1038). It did the destructive thing to someone asking what the
command does.
The top-level dispatcher matches the subcommand at argv[1] and forwards the rest,
so its own --help check at src/main.c:1047 never sees argv[2]. Nothing downstream
looked either, and cbm_cmd_uninstall went straight to parse_auto_answer.
The guard is checked FIRST, before parse_auto_answer, so a `-y` elsewhere on the
line cannot auto-confirm the destruction we are trying to prevent. It prints real
usage, states plainly that the command is destructive, and points at --dry-run.
--help is the flag a person types precisely BECAUSE they are unsure what a
command does. It must never be the thing that destroys their install.
cli suite: 271 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The daemon/CLI rendezvous directory is created under %LOCALAPPDATA% (Windows) or
/tmp -- /private/tmp on macOS -- and every ancestor of it must pass the
private-directory walk. That ancestry is not always acceptable, and when it is
not, EVERY invocation fails, `config list` included, so the settings surface
cannot be reached either:
codebase-memory-mcp: secure daemon endpoint could not be created
#1623 narrowed the Windows side of this by admitting AppContainer package and
capability SIDs on ancestors, and named the remainder explicitly: a live local
group, Authenticated Users inherited from a secondary volume root, and orphaned
unresolvable SIDs still refuse, and "those need CBM_RUNTIME_DIR or a separate
change". #1621 is the POSIX shape of the same dead end -- /private/tmp/cbm-daemon-<uid>
refused with no way to move it.
There was no way to move it in a shipped build. The only relocation hook,
CBM_TEST_DAEMON_RUNTIME_PARENT, is compiled out unless CBM_ENABLE_TEST_SEAMS is
defined, so a test build started while the shipped build did not; CBM_CACHE_DIR
is no help either, because it moves the cache and never the rendezvous.
CBM_RUNTIME_DIR names the parent directory the rendezvous is created under. It
does NOT relax the check: the directory it names goes through exactly the same
validation as the default -- ancestors owned by you or root, not world-writable,
no allow-ACL; the rendezvous directory itself still forced to owner-only -- and a
value that fails is refused rather than silently replaced by the default. The
operator only chooses an ancestry that passes. cbm_safe_getenv never truncates,
so no half of an over-long value can become a runtime parent.
The override is resolved in cbm_daemon_bootstrap_endpoint_new(), the one function
every product endpoint goes through: the daemon, the MCP client, the local CLI,
the index worker, and the install/update/uninstall activation path in cli.c. No
call site can silently keep the default, and the detached daemon inherits the
value with the rest of its environment. An explicit parent still wins, so the
compile-time test seam and the lifecycle guards' isolated namespace behave
exactly as before.
Approach and variable name from #1576 by Leonardo trindade miranda, resolved one
layer lower so the activation path is covered too.
Refs #1574
Refs #1621
Co-Authored-By: Leonardo trindade miranda <tmonestudio@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Six reports describe the same dead end: the indexing worker dies, the
supervisor points at `logs/.worker-<pid>.log`, and the file is 0 bytes.
Nothing was ever flushed, so not one of #1070, #1130, #1132, #1133,
#1145 or #1450 is reproducible or attributable -- the hint says "crashed
on a file" and the file is never named.
Root cause is buffering, not the crash. The worker's stderr is redirected
to that log file by the supervisor, and every diagnostic goes through
`fprintf(stderr, ...)` with no flush. The C standard only promises stderr
is "not fully buffered"; the Windows CRT gives a redirected stderr FULL
buffering, which is why five of the six reports are Windows. A worker
that aborts, is SIGKILLed by the OOM killer, or is terminated after a
hang takes its whole buffer with it.
- cbm_log_set_crash_durable(): setvbuf(_IONBF) plus a per-line flush in
emit_line. Both, deliberately -- setvbuf covers every writer to the
stream including the plain fprintf startup errors that explain a
worker which never got as far as logging, and the flush covers the
case where setvbuf is refused because the stream was already written
to. Enabled for the worker role only, claimed in main() before the
process writes anything.
- cbm_index_worker_log_begin(): a startup header written and flushed as
the worker's first act -- version, build fingerprint, pid, repo path,
and the worker's own arguments. A control record rather than an info
line, so CBM_LOG_LEVEL cannot restore the 0-byte log.
- Under a crash-durable log the parallel extract pass logs every file
it starts, not just the first two rounds of workers, so the log ends
with the files that were in flight when the worker died. One line per
file, never per node; unchanged for every non-worker caller.
This fixes no crash. It converts six unreproducible reports into reports
we can act on, and every future one arrives with the run identified.
Test: a worker started with a fully buffered stderr -- the state the
Windows CRT hands it, forced on POSIX so the contract binds on all three
legs -- writes diagnostics and is then SIGKILLed. The retained log must
be non-empty, carry the header with the version, pid, repo path and args,
and still hold the line written after it. Reverting the production change
leaves that log at 0 bytes.
The probe kills rather than aborts because Darwin's abort() runs the
stdio cleanup handler and flushes the very buffer the repro depends on
stranding: under abort the reverted build still produced a populated log
and the test was quietly toothless. SIGKILL runs no cleanup, and is
literally #1070's death (signal=9).
Refs #1070
Refs #1130
Refs #1132
Refs #1133
Refs #1145
Refs #1450
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`cli <tool>` resolved its JSON arguments from stdin whenever stdin was not a
terminal and no --args-file, raw-JSON positional or --flag form was supplied,
and cli_slurp_stream reads to EOF. An ordinary automation caller never sends
that EOF: Node's child_process.spawn defaults to stdio ['pipe','pipe','pipe']
and the parent must call child.stdin.end() explicitly, which almost nobody does
for a command it is not writing to. fd 0 then stays open with no writer and the
read never returns. The reporter's shell script was still parked in fread(0)
fourteen minutes later.
`list_projects` advertises "properties":{} — stdin could never have carried
anything it accepts, so the read was pure deadlock with nothing to gain. Gate
the stdin path on the tool's input_schema actually declaring properties, in a
seam (cbm_cli_args_from_stdin_allowed) that main.c's resolution chain calls.
An unknown tool also stops blocking: dispatch rejects it by name and no stdin
content can change that verdict. Tools that do declare properties keep the
documented `echo '<json>' | cli <tool>` channel untouched, and interactive runs
are unchanged because isatty(0) already short-circuited them.
Measured on the pre-fix binary, stdin held open by a writer that never writes,
15-second watchdog: `cli list_projects` was killed at 15s with 0 bytes of
output, while `cli list_projects < /dev/null` returned in 3s with 90 bytes.
After the fix the same open-pipe invocation returns in 3s with 90 bytes.
Regression tests (tests/test_cli.c): the reported hang, the piped-argument
channel that must survive, the interactive path, unknown/NULL tool names, and a
sweep asserting the gate tracks the advertised schema across the whole tool
table so a future zero-argument tool cannot reintroduce this silently.
Revert-check: with the schema gate reverted to the pre-fix `return
!stdin_is_tty`, both tests fail on the list_projects assertion and the suite
reports 2 failed; with the gate applied, 272 passed / 0 failed.
Known limit, not addressed here: a tool that DOES declare properties, invoked
with its required flag omitted (`cli index_status` without --project), still
blocks on an open pipe. Distinguishing "arguments are coming" from "nobody will
ever write" needs a bounded wait, which is a public-surface decision (deadline
value, and the cross-platform pipe-readiness probe Windows would need) rather
than a mechanical fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
OpenCode writes "enabled": true beside the "command" and "type" we write. Our
ownership check required the entry's key set to match EXACTLY
(config_json_like.c: member_count != found_count), so a three-key entry with two
recognised keys was classified as FOREIGN - and we refused to touch an entry we
had written ourselves. install then failed with:
error: agent_config agent=OpenCode op=mcp_install path=.../opencode.json
Confirmed on two independent configs: Linux (#1630) and Windows (#1582). In
gotspatel's file EVERY MCP server carries the key - mssql, forgetful,
chrome-devtools and ours - so this is OpenCode's normal shape, not an unusual
hand-edit. Anyone who has ever toggled a server on or off in the UI was hit.
Two of my own hypotheses were wrong before the reporters' files settled it: it is
not JSONC comment parsing, and it is not the .jsonc targeting that #1575 fixed.
#1575 fixed WHICH file we open; this happens after, on what we find inside.
The distinction now reported is MATCH_WITH_EXTRAS, and the caller treats it as
ALREADY SATISFIED - success, without touching the file. That is deliberate and it
is the safe half of the fix: cbm_json_like_upsert_entry REPLACES an entry
wholesale, so writing our canonical shape over an annotated entry would silently
delete the client's keys. A refusal the user can see beats a deletion they
cannot. Doing nothing is also correct on the merits: the entry already names this
binary with the right type, which is the entire content of the install.
Merging our fields into an annotated entry while preserving the rest is the
fuller fix and stays tracked in #1630. This makes the common case work without
risking anyone's configuration tonight.
Ownership is NOT loosened otherwise: an entry whose command points at a different
binary is still foreign and still refused, byte-identically, and that direction
is pinned by its own test.
Tests use the reporters' actual entry shape. cli suite: 272 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
yaml_range_has_unsupported rejected `&` and `*` anywhere in a plain scalar, with
no positional test at all. So a value containing prose asterisks - `use
*emphasis* here`, a kaomoji, a glob inside a description - was refused as if it
were an alias.
That is one of four constructs that made a real 16 KB hand-maintained Hermes
config permanently un-editable by us (#1631). It is not YAML we cannot model; it
is YAML we declined to read.
An indicator only counts where a NODE begins. The test is now positional and
judged by what PRECEDES the character, because the scanned range covers a whole
line including its key: in `command: &shared` the `&` is an anchor even though
`command:` came first. A node begins at the range start, after a mapping colon,
or after a block-sequence dash. Everything else is text.
`{` and `}` keep their existing treatment. Empty flow mappings (`key: {}`) are
the next item in #1631 and carry their own semantics; bundling them here would
mix a positional correction with a value-semantics change.
Root-caused by @rg6304, who reproduced it in isolation, read this source, and
corrected the maintainer's hypothesis - the constructs I had guessed (inline
comments, anchors, `---` separators) appear nowhere in the failing file.
TESTING NOTE, recorded because it nearly went wrong: the first version of the
regression test put the asterisk in an untouched foreign section and passed with
the fix REVERTED - the editor only validates the range it writes, so the test
proved nothing. Rewritten onto the entry block, it now fails without the fix
(`== -1, expected 0`) and passes with it. The companion test pins the other
direction: a LEADING `*` is still an alias and still refused, byte-identically.
config_yaml_edit + yaml suites: 117 passed, 0 failed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
CI lint flagged two continuation lines from the previous commit. Formatting
collapsed the runtime-directory call onto one line, so the contract pin that
asserted ancestor=false there stopped matching; the pin now keys on the
formatted text. Both revert directions re-verified: capability-only fails, and
relaxing the runtime directory to ancestor=true fails.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Six independent machines in #1533 and #1574 cannot run cbm at all, and none of
them is exotic: a domain-joined UAC-filtered admin, a secondary volume, orphaned
ACEs from an uninstalled application, and an AppContainer package SID belonging
to a shipping desktop application.
The ancestor walk was identity-blind to all of it. win_directory_component_secure
demanded that no ancestor DACL entry grant any private-mutation bit to any SID
outside the trusted set, so a single ACE anywhere up %LOCALAPPDATA% refused the
endpoint before logging started. Every mode failed, config list included, so the
product could not even be reconfigured out of it, and CBM_CACHE_DIR does not help
because the runtime directory is %LOCALAPPDATA%\cbm-daemon-<hash>, whose ancestor
chain relocating the cache never touches.
Ancestor components now also tolerate AppContainer identities: package SIDs
(S-1-15-2-*) and capability SIDs (S-1-15-3-*), under identifier authority 15.
The boundary is ancestor-only. The private runtime directory is validated with
ancestor=false and keeps demanding the exact current user with a protected DACL;
the flag is threaded explicitly through win_file_security_secure and
win_file_acl_secure rather than inferred, so the strict path cannot acquire the
tolerance by accident.
Why these identities are admissible on an ancestor: a process cannot choose which
AppContainer it runs in. The identity is stamped by the OS at process creation
from the package it was launched from, so such an ACE cannot be exercised by
arbitrary local code the way a live local group can. What it does permit is the
packaged application itself — that is the residual risk this accepts, and it is
the same trust already extended to whoever installed that package.
BOTH forms are covered deliberately. The most common real ACE of this shape is
S-1-15-2-*, a package SID; on reported machines it resolves through the registry
AppContainer mappings to Anthropic Claude Desktop, which many of our users run and
cannot be asked to uninstall. Covering only capability SIDs leaves exactly that
case failing.
This narrows a deliberate policy: the strict gate was chosen on purpose and a
middle ground was previously declined. It is reopened here by explicit maintainer
decision, and narrowed as far as the evidence allows rather than relaxed wholesale
— a live local group, Authenticated Users on a secondary volume, and orphaned
unresolvable SIDs all still refuse. Those need CBM_RUNTIME_DIR or a separate
change; orphan tolerance in particular needs LookupAccountSid bound first and has
an offline-domain-controller caveat, so it is not bundled in here.
Approach and the ancestor-only boundary follow @mlandolfi90's PR #1447, extended
from capability SIDs to package SIDs.
Co-Authored-By: mlandolfi90 <mlandolfi90@users.noreply.github.com>
Contract-pinned in tests/test_windows_bundle_contract.sh and revert-checked in
both directions: reducing it to capability-only fails, and passing ancestor=true
for the runtime directory fails.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Release notes now carry one row per product — the bytes actually published, with
their VirusTotal verdict — instead of a column per candidate.
Several candidates are scanned so the selector has an alternative when an opaque
classifier flags one, but a reader installing cbm cares about what they receive,
not about what we discarded. The rejected candidates' verdicts remain in the
published evidence TSVs for anyone auditing the selection, and they stay useful
to US as a development signal; in a changelog they are noise, and listing a
"microsoft-ml" verdict next to a binary nobody can download invites exactly the
wrong conclusion.
Also removes a claim that had become FALSE: the section asserted that every scan
had a minimum of 50 decisive engines. That floor was removed deliberately (it is
VirusTotal fleet availability, not a property of our binary) and the last release
observed 31. Published notes would have stated something untrue. The observed
range is still reported, now over the shipped binaries.
The disclosure of a tolerated Microsoft `!ml` is preserved and narrowed to the
binaries that ship. The contract test now pins both directions: a flagged binary
that SHIPS must be disclosed, and flagged candidates that were rejected must not
appear at all. Its fixture gained a target that draws the tolerated verdict on
all three candidates, so the disclosure branch is actually exercised — previously
every flagged candidate was rejected in favour of a clean sibling, leaving that
path untested.
README and SECURITY.md updated: they promised notes "link every candidate result".
All five release/VT contract tests pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Ertan