Commit Graph
1663 Commits
Author SHA1 Message Date
Martin Vogel d1761e40cb fix(daemon): stamp the token-user SID as owner on private-namespace objects
The Windows SECURITY_ATTRIBUTES used for every private-namespace
creation (runtime dirs, rendezvous/startup/participant/lifetime/cohort
lock files, log files, pipes) carried a DACL but no owner, so created
objects took the token DEFAULT owner. Admin-group tokens can default
that to BUILTIN\Administrators - standard policy on Windows Server,
including CI runners - while every strict validation gate
(win_file_owner_secure(require_current_user=true),
private_win_owner_only_dacl) demands the exact token-user SID. Result:
the daemon rejected objects it had just created itself; the client
refused the daemon's rendezvous record, retried silently for its full
30 s window, and each clientless generation clean-stopped, looping
until the frontend gave up. The legacy kernel mutex was the one object
that already force-set its owner post-create - it is the template this
change generalizes.

Both descriptor builders now set the owner explicitly at creation, so
objects are born satisfying the same strict gates on every default-
owner policy. The gates themselves are unchanged.

The Windows client connect path also stops failing mute: terminal
rendezvous refusals, deadline expiry (with last status), and pipe
server-identity rejections now log their reason once, matching the
launcher-side refusal diagnostics.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 17:19:50 +02:00
Martin Vogel c634c86d1a ci(windows): protected per-user temp root + explicit owner on staged guards
Completes the handoff task for the Windows harness: the native test job
ran with the MSYS-shared /tmp and the runner's inherited
LocalAppData\Temp ACLs, which the daemon's strict private-filesystem
checks reject by design. The test-windows job now creates a per-user
root under the profile with an owner-stamped, protected current-SID
DACL and routes TEMP/TMP (native form) and TMPDIR (POSIX form) through
it before scripts/test.sh runs.

The guards script already hardened its root DACL but ownership is never
inherited on Windows: bundle copies created by the admin-group runner
token can come out Administrators-owned, and the launcher's exe policy
requires the exact current user as owner. Stamp the current SID on the
staged bundle dir and both executables explicitly.

Job topology, gates, and triggers are unchanged; the new step is a
sub-second pwsh block in the existing test-windows job.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 16:58:10 +02:00
Martin Vogel 5dfce38daa fix(launcher): name the object and check behind every security refusal
The launcher answered every ownership/ACL problem with one generic
"launcher ownership or access policy is unsafe", which made the CI
guard failures (and any field report) undiagnosable: the same message
covers an untrusted owner SID, a mutation-capable ACE, a reparse point,
a hard-link clone, and a plain open failure across the whole ancestry
walk. Record the failing check at its site - offending path, owner or
ACE SID in SDDL string form, rights mask, Win32 error - and print it
once alongside the refusal. No policy changes; refusals stay refusals.

The interactive MCP smoke now also dumps daemon-conflicts.ndjson next
to the daemon log tail on failure, so an admission conflict names
itself instead of surfacing as a bare 30 s client timeout.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 16:57:58 +02:00
Martin Vogel 712d689efb fix(test): serialize the daemon-family suites in the parallel runner
The six suites that spawn coordinated worker subprocesses or bind local
endpoints (index_supervisor, daemon_application, daemon_runtime,
daemon_frontend, daemon_bootstrap, daemon_ipc) ran in the parallel wave,
where the saturated 3-4-core CI runners starve their fixed readiness
deadlines: index_supervisor waits at most 3 s for a worker marker while
the worker is a full re-exec of the ASan runner plus the admission
handshake. Both macOS legs failed the same four readiness assertions
deterministically and ubuntu-latest added four daemon_frontend and one
daemon_runtime timing failures, while an idle machine passes the same
suites 6/6 in the wave.

Moving them to the quiet serial tail follows the runner's existing rule
for this class (cli/subprocess/watcher/...) and also keeps the shared
per-account coordination namespace free of cross-suite admission
traffic. Same suites, same tests, same union-guarded totals - only the
schedule changes. Locally re-verified: 6742 passed, 0 failed, 2 skipped
(120 suites, 17 serial-tail).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 16:45:06 +02:00
Martin Vogel f7e23d8483 docs(vendored): record the SQLite MAX_PATHNAME patch
The Unix-VFS MAX_PATHNAME bump from 512 to 4096 shipped with the daemon
work but left no paper trail, so the next amalgamation refresh would
silently revert it. Document it in vendored/sqlite3/PATCHES.md with the
rationale and re-apply procedure, and point to it from THIRD_PARTY.md.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 16:44:56 +02:00
Martin Vogel 7696b1486a fix: resolve remaining platform CI regressions
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 15:41:57 +02:00
Martin Vogel 3bd9e872ac Merge origin/main into feat/shared-coordination-daemon
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>

# Conflicts:
#	Makefile.cbm
2026-07-18 15:28:23 +02:00
Martin Vogel ad2874bcff fix: close remaining daemon coordination races
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 15:26:38 +02:00
Martin Vogel 7d6cdb23ef Merge pull request #1164 from DeusData/feat/test-sharding
Parallel test execution: per-suite processes with a zero-loss gate contract
2026-07-18 15:21:09 +02:00
Martin Vogel f35b10b79b fix: harden cross-platform daemon startup smoke
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 14:38:07 +02:00
Martin Vogel bc3594b728 fix(test): pin XDG_CONFIG_HOME in the three env-coupled cli tests
The parallel runner's isolated-process execution exposed three cli tests
that were passing for the wrong reason on Linux: cbm_zed_config_dir (and
the VS Code/vendor path resolvers) prefer $XDG_CONFIG_HOME over
$HOME/.config, CI runners export it, and an isolated `test-runner cli`
process therefore resolved config paths OUTSIDE the test fixture — the
tests only stayed green in the all-suites single process because an
earlier suite's env mutations leaked into them (hidden cross-suite
coupling, deterministic failure in isolation, macOS unaffected since its
resolvers do not consult XDG).

- cli_detect_agents_finds_zed: save/set XDG_CONFIG_HOME to the fixture's
  .config for the detection call, restore after.
- cli_vscode_profile_mcp_uninstall: same pin alongside its existing
  HOME/PATH/APPDATA handling.
- cli_durable_profiles_follow_current_vendor_paths: XDG_CONFIG_HOME joins
  its unset-and-restore env list.

All 222 cli tests pass locally with a hostile XDG_CONFIG_HOME exported.
No production code changed; no assertion weakened.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 14:10:06 +02:00
Martin Vogel 5d8a9421fb Merge remote-tracking branch 'origin/main' into feat/shared-coordination-daemon
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>

# Conflicts:
#	src/mcp/mcp.c
#	tests/test_cli.c
2026-07-18 13:48:06 +02:00
Martin Vogel 95659b3955 test-par: serialize the mcp suite too (both alarm-deadline tests live there)
Flake audit: the whole corpus has exactly two wall-clock test deadlines —
the EOF-shutdown alarm(5) and the issue-832 supervisor-spawn alarm(60),
both in the mcp suite (the latter already flaked once under heavy local
load). Serializing mcp costs ~6s of tail and closes the starvable-deadline
class completely; every remaining parallel-wave suite is deadline-free.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 13:39:42 +02:00
Martin Vogel b0a1287794 test(so): document the POSIX GLR-cap branch as a bounded smoke, per falsification
Follow-up correcting d8a78e8, whose message claimed a broken merge cap
crashes "within the first second" so the 10s alarm could not mask it.
Falsification disproved that claim AND found the guard was vacuous on
POSIX from the start:

- With CBM_TS_STACK_MERGE_MAX_DEPTH deleted outright, the ORIGINAL
  unbounded test stayed GREEN (220s full grind, no crash) on macOS ASan.
- Small thread stacks (2 MB, then 512 KB) did not change that.
- Instrumenting the vendored recursion showed why: the recursive merge
  path is NEVER ENTERED for this input in the observable window — max
  depth stays 0. There is nothing for a stack bound to catch here.

So the cap-regression DETECTION lives in the Windows in-process branch
(real ~1 MB native stack, where #913 actually manifested — exercised by
the windows CI leg). The POSIX branch is, and always was, a bounded
no-crash smoke of the pathological parse; the 10s alarm makes it
deterministic (12-15s across three runs, was 2-218s machine-dependent)
without any detection loss, because there was no POSIX detection to
lose. The experimental pthread machinery is dropped; the test comment
now carries the falsification data so no future reader mistakes the
POSIX branch for a cap guard.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 13:31:18 +02:00
Martin Vogel d8a78e8cf4 fix(test): deterministic ceiling for the GLR merge-cap regression guard
perl_glr_deep_parse_recursion_capped wandered between 2s and 218s for the
IDENTICAL input: past the merge cap the ambiguity-exploded GLR parse grinds
at an environment-dependent rate (allocator/scheduler state in the forked
child), and the test paid for however long that grind ran. It was the
critical path of every parallel run and the dominant chunk of the
sequential one.

The guard's actual signal is the CRASH, not the grind: when
CBM_TS_STACK_MERGE_MAX_DEPTH regresses, the recursive stack-merge
overflows within the first second at this depth (same profile as the
sibling pre-guard probe, rc=139 in under a second). The forked child now
arms a 10s alarm whose handler exits cleanly — a real SIGSEGV/SIGBUS
still terminates the child by signal first and stays visible to
WIFSIGNALED, so the RED path is untouched; only the pointless remainder
of the grind is skipped. Windows (in-process branch) is unchanged.

Suite b: 13s across three consecutive runs (was 2-218s). Full parallel
run: 97s wall, 6,374 passed / 0 failed / 1 skipped, union guard clean —
vs 351s sequential on the same machine (3.6x).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 13:12:20 +02:00
Martin Vogel 410c8d70c8 fix: stabilize Windows coordination startup
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 13:10:55 +02:00
Martin Vogel 5fdbd7b1f4 Merge main (tree-format program #1154) into the parallel-test branch 2026-07-18 12:48:48 +02:00
Martin Vogel 94c8e96889 Merge pull request #1154 from DeusData/feat/toon-round2
Tree-format tool output: prefix-grouped rows, exactly-once pagination, real change impact, cycle detection
2026-07-18 12:48:25 +02:00
Martin Vogel 89c7c98f00 fix: support standard Windows profile ACLs
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 12:26:17 +02:00
Martin Vogel c8e07a2f9f fix(test-par): CRLF-proof the suite list, serialize deadline-sensitive suites
First CI validation round of the parallel runner, three findings:

- Windows failed ALL 104 suites with "0 passed, 1 failed" each: the CRT
  emits --list-suites lines as CRLF, so every dispatched name carried a
  trailing CR ("arena\r" is an unknown suite). The union guard caught it
  loudly, exactly as designed. The driver now strips CR when writing the
  suite list.

- The ubuntu-gcc legs had 3 real failures in the cli suite: it spawns
  subprocesses with fixed deadlines, and a fully saturated 4-core runner
  starves those deadlines into flakes. Deadline-sensitive suites (cli,
  subprocess, watcher, incremental, httpd, ui, index_resilience, and the
  stack_overflow family) now run SEQUENTIALLY after the parallel wave on a
  quiet machine — same suites, same tests, same gates, only the schedule
  differs; the union guard checks the combined result set.

- Failing suites now print every FAIL site with context (the tail-30 of a
  long suite log hid which tests actually failed on CI).

Local: totals unchanged (6,361 passed / 0 failed / 1 skipped), union guard
clean.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 12:24:07 +02:00
Martin Vogel ad58e71bea fix: stabilize cross-platform daemon coordination
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 11:48:59 +02:00
Martin Vogel bd939fc49a perf(test): run suites as parallel processes — same gates, ~2.5x faster
The runner executed all 104 suites sequentially (~10 min locally, the bulk
of every CI test leg). The suites are process-isolated already (per-process
mkdtemp HOME sentinel; two full parallel runs produced zero cross-suite
failures), so the serialization was pure convention.

- test-runner --list-suites: prints every registered suite, one per line,
  emitted by the SAME macro table that executes suites — the list cannot
  drift from the run set by construction.
- scripts/run-tests-parallel.sh: runs each suite as its own process
  (jobs = CPU count; CBM_TEST_PAR_JOBS overrides) under a ZERO-LOSS
  CONTRACT: a union guard fails the gate if the set of suites that
  produced a result differs from --list-suites (nothing can be silently
  dropped, a newly added suite is picked up automatically); per-suite
  pass/fail/skip are summed into the sequential runner's exact summary
  format; any suite crash, failure, or omission exits nonzero. Per-suite
  wall times are printed for balance tracking.
- make test-par: the parallel target. make test (sequential) is unchanged
  and remains the escape hatch (CBM_TEST_SEQUENTIAL=1 in test.sh).
- scripts/test.sh: builds, then routes through test-par — every CI test
  leg gets the speedup with zero workflow-topology change (same jobs,
  same gates, same billing; the legs just finish sooner).
- The stack_overflow suite is split into a/b/c (7+6+7 of its 20 tests,
  pure re-registration): as one suite it was the wall-clock critical path
  of any parallel run — every other suite finished underneath its ~4
  minutes.

Measured locally (Apple Silicon, ASan runner): sequential ~10 min vs
parallel 232 s, totals identical (6,361 passed / 0 failed / 1 skipped),
union guard clean. The TSan leg keeps its dedicated subset runner
(sequential) — out of scope here.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 11:05:17 +02:00
Martin Vogel cc5f71bdff fix(hook,smoke): teach the last two consumers the tree output shape
The PR CI caught two consumers the format migration missed — both parsed
tool output by its old shape and failed silently or with a wrong count:

- hook-augment built its PreToolUse additionalContext by reading the
  legacy 'results' object-array from search_graph format:json; after the
  json-tree reshape that key no longer exists, so the hook emitted nothing
  (caught end-to-end by tests/windows/test_hook_augment.py). The parser
  now walks groups -> qn_prefix/file + column-ordered rows. A new local
  test feeds the parser the LIVE server envelope, so any future drift
  between response shape and hook parser fails in the normal suite on
  every platform — not only in the Windows CI guard. Reproduced RED
  (ctx NULL, the exact CI failure) before the fix, GREEN after.

- smoke-test.sh carried six old-shape parsers (callers[N], rows[N]{,
  clusters[N], semantic[N]{, and the assert_toon_table helper); the tree
  headers never matched, so trace verification read 0 callers. All six
  updated to the tree contract; the full smoke script passes end-to-end
  locally.

cli suite 222 passed; lint clean; scripted smoke ALL PASSED.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 10:36:08 +02:00
Martin Vogel 98a28473db fix: harden cross-platform daemon startup
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 10:15:22 +02:00
Martin Vogel b8a75d142c fix: stabilize cross-platform daemon launch
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 03:16:37 +02:00
Martin Vogel cb896a3de4 fix: harden cross-platform daemon startup
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 02:43:14 +02:00
Martin Vogel 4693b625e0 fix: stabilize cross-platform daemon smoke
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 02:22:37 +02:00
Martin Vogel dd87676865 Merge origin/main into feat/shared-coordination-daemon
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 01:29:45 +02:00
Martin Vogel 60390affb8 fix(mcp): close the legacy-generation cursor gap; json = the tree model everywhere
Fresh review pass over the whole branch; three classes of fix:

Cursor safety — the 'legacy' generation gap:
  A pre-migration DB (indexed by an older binary, opened read-only) has no
  store_meta, so its generation reads 'legacy' — and 'legacy' == 'legacy'
  across rebuilds, which silently defeats stale-cursor detection for exactly
  those DBs: a stale (hop,id) watermark would resume on wrong node ids.
  Cursors are now neither minted (truncated + raise-limit hint instead) nor
  accepted (cursor_unsupported teaching error) under a legacy generation.
  The standard flow was verified safe empirically: every index run seeds
  store_meta via cbm_store_upsert_project in the flush path, and a rebuilt
  DB file gets a fresh random db_uid, so aliasing is impossible there. The
  guard is unreachable through the public API (upsert always seeds) — it
  protects only genuinely old DBs, hence no synthetic test.

format:'json' now emits the tree model on EVERY tool (the last five):
  - BM25 query mode: {total, cols, rows[[qn,label,file,lines,rank]]} —
    rank order preserved, ranked output is never regrouped.
  - semantic: {semantic: {cols, rows[[qn,label,file,score]]}}.
  - search_code compact/full: {cols, rows} with nested match-line arrays;
    full appends a per-row source object (same guarded, windowed reader);
    raw_matches likewise {cols, rows}. files mode stays a plain list.
  - get_architecture: all twelve sections emit {cols, rows} via one shared
    section helper; cluster row cells carry their nested lists as arrays.
  - include_connected became a real  column in BOTH encodings
    (';'-joined cell in the text tree, nested name array in json) — the
    include-connected-forces-JSON coupling is gone, and the last legacy
    per-node-object emitters (emit_search_results, enrich_node_properties)
    are deleted.

Teaching-error parity: detect_changes now rejects an unknown
with the valid values (it silently fell back to inbound; trace_path already
errored — the caller would have misread the result's semantics).

Full suite 6,373 passed / 0 failed; lint clean. Live-verified on the
self-index: bm25/search_code/architecture json shapes, the connected
column, and cursor pagination.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 01:26:12 +02:00
Martin Vogel 83c137d2a5 feat: complete shared daemon lifecycle
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 01:26:08 +02:00
Martin Vogel 47e4d25dac fix(mcp): review polish — honest totals, complete counts, quoted cells
Fresh-eyes review of the phase-2/3 work; five defects, each against our own
stated principles (no silent caps, exact counts, same model in both
encodings):

- trace callees_total/callers_total counted test-file rows even when
  include_tests=false hides them from the table — the scalar disagreed with
  the enumerable set (a field-eval agent read callers_total=175 against a
  handful of visible rows and stopped trusting the tool). Totals now apply
  the same filter as the emitters; description documents the transitive
  (within-depth) semantics. Guard test: filtered=1 vs include_tests=2.
- cycles clamped the SCC list at 100 silently, losing the true count.
  cycles_total is now always exact and cycles_omitted is emitted when the
  display clamp engages (both encodings).
- detect_changes accepted a 'limit' param (referenced by its own omitted-
  hint) that the schema never declared — undiscoverable. Declared.
- detect_changes format:json lacked the impacted_modules rollup the tree
  encoding carries — model parity restored via a shared rollup helper.
- tree search extra-field cells were appended raw: a signature with spaces
  shifted every following column. Cells now route through the shared
  emitters (space-triggered quoting, '-' placeholder); the properties test
  pins the quoted form.

Full suite 6,373 passed / 0 failed; lint clean.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 00:40:54 +02:00
Martin Vogel 130a82bd10 feat(mcp): real impact analysis + cycle detection (multi-source BFS, SCC)
Phase 3 of the graph-analysis overhaul. Two new capabilities on a shared
multi-source traversal substrate, plus a context-bomb guard.

detect_changes — rebuilt into real blast-radius analysis:
  The old handler parsed and echoed depth but never traversed: impacted was
  just the symbols DEFINED in changed files. It now resolves the git diff to
  those symbols, then runs ONE multi-source BFS to the transitive impact set.
  - default direction inbound = the blast radius (transitive callers that may
    need review); outbound/both available.
  - seeds are excluded from the result, and a changed file reached from
    another changed file is not counted as extra impact (that is not
    downstream fallout).
  - merge_base SHA pins the exact diff base for reproducibility.
  - impacted_modules: a 2-segment (src/mcp, tests/...) rollup — a quotient
    view of where the radius lands, always complete (overflow bucketed, never
    silently dropped). Per-symbol rows cap at limit (default 200, nearest
    hops first) with impacted_total exact and impacted_omitted surfaced.
  On a 7-file core diff: 1469 impacted symbols, 64KB unclipped -> 10.7KB with
  the rollup + cap. format:json returns the same model as structured JSON.

cbm_store_bfs_multi (store): one recursive CTE anchored on ALL seeds via a
  temp table — never seed_count separate walks over overlapping hubs.
  Seed-exclusion NOT IN, MIN(hop) across the set, canonical (hop,id) order,
  a memory-safety ceiling that reports *truncated instead of silently
  capping. Reproduce-first: seed-exclusion + min-hop, and truncation.

get_architecture aspects:[cycles] — SCC condensation:
  Iterative Tarjan (heap stacks, no recursion overflow) over the CALLS graph
  reports circular dependencies (strongly-connected components of size > 1).
  cbm_store_fetch_call_edges feeds it with a 400k-edge scan budget that
  reports partial coverage rather than truncating silently. Opt-in ONLY —
  never via a default or 'all' call, since it scans the whole call graph.
  On this repo it surfaces 26 real cycles (e.g. the C-LSP type-evaluation
  recursion) in ~5KB. Reproduce-first: a 3-node cycle is found, an acyclic
  chain is not, and a default call does not run the scan.

get_code_snippet: cap the returned line span at 500 (a structural Module/File
  node spans its whole file — an unclipped read returned ~400KB in one call).
  source_clipped flags it; start_line/end_line still allow a targeted re-read.

Full suite 6,372 passed / 0 failed; lint clean.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 00:13:50 +02:00
Martin Vogel 3f5ae18852 feat(mcp): complete the tree-format migration — TOON pruned, json = stringified tree
- cbm_toon_* renamed to cbm_tree_* (the emitters have emitted tree syntax
  since the format flip; the name now matches).
- format enums reduced to [tree, json], default tree; dead tree/toon flags
  removed; tool RESPONSE descriptions rewritten for the grouped-tree
  contract (group prefix printed once, full qn = prefix + '.' + name).
- format:'json' is now the json-stringified tree: the SAME grouped model
  serialized as JSON — {total, cols, groups:[{qn_prefix, file,
  rows:[[column-ordered cells]]}], has_more} for search, per-leg
  {cols, groups} for trace (risk/args columns follow the flags),
  query_graph's columns+rows shape already complied. Per-row key envelopes
  and the legacy property blob are gone (blob-never-spills pinned by test);
  requested  add columns in both encodings.
- include_connected keeps the per-node-object shape (nested neighbors) —
  the one remaining legacy consumer, conversion tracked.
- legacy verbose emitters deleted (bfs_to_json_array).

Full suite 6,367 passed / 0 failed; lint clean.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-17 22:58:22 +02:00
Martin Vogel c5bffb7fe6 feat(mcp): tree-format tool output, exactly-once pagination, lean defaults
Round-2 overhaul of the tool-output surface, built eval-first (byte harness +
no-context field-eval agents, always vs the v0.9.0 release binary):

Format — tree syntax everywhere:
- compact_out emits tree tables: 'key: N  (cols: a b c)' headers, indented
  space-delimited rows, '-' placeholders, space-aware quoting. Every table
  in every tool converts mechanically.
- Grouped prefix-factored output is the DEFAULT for search_graph and
  trace_path: rows grouped under their shared (qn-prefix, file) printed
  once, row qn = group prefix + '.' + name (stated in-band). Measured:
  192-hit search 2,929B (was 4,811 TOON / 238,466 v0.9.0), 82-row trace
  2,256B (was 4,234 / 8,094). Field eval: 5/5 answers correct, QN
  reconstruction verified, no comprehension loss.
- detail:'ids' tier: bare-qn enumeration for wide sweeps (-44%).

Pagination — exactly-once, stateless:
- store_meta generation (per-DB random uid + mutation counter bumped in
  cbm_store_upsert_project — every index run's choke point): cursors can
  never silently validate against a rebuilt DB.
- trace cursors: opaque token (leg, generation, params-hash, (hop,id)
  watermark); pages drain callees then callers with exact
  callees_total/callers_total on every page; teaching errors for
  params-mismatch and staleness. Guard: 12 callees paged 5+5+2, every row
  on exactly one page.
- deterministic ordering: BFS ORDER BY hop,id; canonical (hop,id) sort +
  min-hop dedup across same-name seeds (hop fed risk_labels wrong before);
  search (name,id)/(rank,id) tie-breaks make offset pages stable.

Lean defaults + teaching errors:
- get_graph_schema no longer advertises blocked internal fields (fp/sp/bt).
- index_repository coverage lists: counts + 5 examples + one-line pointer
  notes (full record was always in the logfile); 6.0KB -> 1.5KB here,
  53KB observed on a large repo. index_status gates the git context block
  behind verbose:true.
- trace limit param (<=5000) + truncated:true — never a silent truncation;
  invalid direction now errors with the valid values; requesting core
  columns via fields hints instead of emitting empty cells; cypher
  resolves f.file/f.qn/f.path aliases.

Trace-correctness fixes (reproduce-first, RED verified):
- bfs_collect_edges 4KB id-string overflow: past ~1000 visited nodes the
  edge SQL failed to prepare and EVERY trace edge silently vanished; now a
  per-connection temp-table join (star guard: 1200/1200 edges).
- bfs_union_same_name: min-hop across seeds + edge dedup.

Guards: gating tool-output regression suite (context-explosion detector:
blocked-field leaks, repeated-key envelopes, oversized notes + calibrated
byte ceilings + semantic floors). Full suite 6,367 passed / 0 failed.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-17 22:32:53 +02:00
Martin Vogel e678b2b6ac Merge pull request #1137 from DeusData/fix/1083-wal-journal-size-limit
fix(store): bound the WAL (journal_size_limit) + warn on checkpoint starvation (#1083 part 1)
2026-07-16 22:29:34 +02:00
Martin Vogel 7235a4db14 Merge pull request #1136 from DeusData/fix/codeql-objectscript-wider-type
fix(grammars): widen uint8_t loop counters in objectscript scanner (CodeQL #69-#72)
2026-07-16 21:12:33 +02:00
Martin Vogel 00719da86f Merge pull request #1135 from DeusData/fix/1071-cpp-macro-type-arg
fix(extract): don't flag in-body function-like macro calls as parse gaps (#1071)
2026-07-16 20:38:41 +02:00
Martin Vogel 87e3efec25 Merge pull request #1049 from DeusData/fix/988-py-alias-import
fix(py-lsp): resolve calls through aliased imports
2026-07-16 19:43:18 +02:00
Martin Vogel 9dfe52f16e fix(store): warn when a checkpoint can't reset a large WAL (starvation)
Add a checkpoint-starvation warning to cbm_store_checkpoint: read the
wal_checkpoint_v2 out-params (frames in the WAL vs frames reclaimed) and, when
the WAL is large (>= ~1 GiB of frames) but the PASSIVE checkpoint couldn't reset
it because readers hold marks, log store.wal.starving. This is the operator
visibility the reclaim limit can't provide on its own — it surfaces the
starvation (and its scale) before the -wal file fills the disk. Portable
(no file stat / no path handling), fires only under genuine starvation.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-16 19:39:15 +02:00
Martin Vogel 0e00ef5702 feat: coordinate concurrent CBM sessions
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-16 19:20:46 +02:00
Martin Vogel 4a3a9b745f fix(store): bound the WAL with journal_size_limit so a starved log is reclaimed
On-disk write connections set no journal_size_limit, so the -wal file is never
physically shrunk: all our checkpoints are PASSIVE (TRUNCATE is deliberately
avoided because its ftruncate(fd,0) can SIGBUS a sibling mmap'd process on
macOS), and PASSIVE never ftruncates. Under checkpoint starvation the WAL then
grows without bound (reported: 115 GB, filled the drive) and even a later
successful checkpoint leaves the file at its peak size (#1083).

Set journal_size_limit = 256 MiB on write connections so SQLite truncates the
-wal back to that bound whenever a checkpoint can finally reset the log —
reclaiming disk opportunistically during a session, not only on clean close.
256 MiB is far above the healthy WAL (~4 MiB under the default 1000-page
autocheckpoint), so normal indexing never triggers truncate/regrow churn; the
limit only fires after abnormal growth. No TRUNCATE checkpoint is introduced.

Scope: this is the *reclaim* half of #1083. It does not prevent the peak under
continuous reader contention (a checkpoint can't reset the log while a reader
holds a mark) — the write-amplification driver (concurrent full re-indexes of
one repo) is serialized by the cross-process pipeline lock in #892.

Test: a file-backed write store now reports journal_size_limit = 256 MiB
(RED on main: -1 / unlimited).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-16 18:53:35 +02:00
Martin Vogel 29d05589dc fix(grammars): widen uint8_t loop counters in objectscript scanner (CodeQL)
CodeQL cpp/comparison-with-wider-type (4 high-severity alerts, #69-#72) flagged
two loops in the vendored ObjectScript scanner (objectscript_common.h, in both
the _routine and _udl grammar copies) where a uint8_t counter is compared
against an int length: the reverse_marker scan and the html_marker_buffer
reversal. Both lengths are hard-bounded by MARKER_BUFFER_MAX_LEN (30), so the
uint8_t counter can never wrap and the flagged infinite loop is unreachable in
practice — but widening the counters to int removes the pattern and clears the
gate. Provably behavior-neutral (int holds every uint8_t value; length <= 30).

Recorded as a local modification in the vendored grammar MANIFEST so it is
re-applied on the next re-vendor.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-16 18:34:23 +02:00
Martin Vogel a75f4b6a3b fix(extract): don't flag in-body function-like macro calls as parse gaps
A function-like macro invocation whose argument is a type token — e.g.
ALLOC(int, n) — makes tree-sitter's C/C++ grammar emit an ERROR node (it parses
the type in expression position), which cbm_collect_error_regions recorded as a
parse_partial coverage gap. But the macro is #defined in the same file and the
call sits inside an already-extracted function body, so nothing is actually
missing from the graph — it's a benign call the grammar can't parse without the
preprocessor (#1071, systematic across allocation-macro-style codebases).

Subtract an error region only when it both (a) contains an invocation of a
file-defined function-like macro and (b) is fully enclosed by an extracted
Function/Method body. Condition (b) keeps a TOP-LEVEL macro invocation that
expands to a definition still flagged (#949: the generated def isn't in the
original span), and fails safe — a benign top-level call stays flagged rather
than a real gap being hidden.

Tests: a type-arg macro call inside a function no longer reports parse_partial;
a real in-body syntax error still does; #949/#946 preserved.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-16 18:19:54 +02:00
Martin Vogel 0ec24787fa Merge remote-tracking branch 'origin/main' into HEAD
# Conflicts:
#	tests/test_lang_contract.c
2026-07-16 17:37:46 +02:00
Martin Vogel a688d85cc8 Merge pull request #1093 from PR9000/freebsd-hw-physmem
fix(foundation): use HW_PHYSMEM for physical memory detection on FreeBSD
2026-07-16 17:33:20 +02:00
Pedro Ramos 62808da92e tests: add platform_system_info regression test
Signed-off-by: Pedro Ramos <131530838+pr9000@users.noreply.github.com>
2026-07-16 04:10:26 +01:00
Martin Vogel 5d150ea051 Merge pull request #1109 from DeusData/fix/601-cypher-exec-deadline
fix(cypher): add a wall-clock execution deadline to abort runaway queries (#601)
2026-07-16 04:56:50 +02:00
Martin Vogel 3d6aa6791d Merge pull request #1048 from DeusData/fix/999-ci-config-infra-routes
fix(pipeline): stop CI/tooling config URLs from minting infra Route nodes
2026-07-16 04:19:42 +02:00
Martin Vogel 6bd11ffb58 fix(cypher): add a wall-clock execution deadline to abort runaway queries
An unbounded whole-graph OPTIONAL MATCH or a GROUP BY that produces roughly one
group per node does O(bindings x groups) work in execute_return_agg and can run
for minutes on a large graph before emitting a single row. The 100k row ceiling
never fires (no rows are produced), so query_graph just hangs with no partial
result and no error (#601).

Arm a monotonic wall-clock budget (default 30s) at query entry and check it
(throttled, every 1024 iterations) in the scan, relationship-expansion and
aggregation hot loops. On expiry the query aborts with a clear, actionable error
instead of hanging. A thread-local test hook forces the budget so the guard is
covered by a deterministic reproduce-first test; a companion test confirms the
default budget does not false-positive on a normal query.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-16 03:09:27 +02:00
Martin Vogel 90d7315cf9 Merge pull request #1108 from DeusData/fix/964-1077-fqn-collision
fix(fqn): preserve full filename in File-node QNs so sibling files don't collide
2026-07-16 02:52:00 +02:00