mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-09 09:11:00 +08:00
8af291deb48acda7568300577b670da415da978e
27
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
6bf6e34e43 |
perf(store): evict idle adjacency layers, budget them, own memory class
The daemon keeps adjacency layers (store/graph_mem.h) resident across
requests -- 237 MB for the kernel -- with no end of life but LRU pressure.
- Idle eviction: a layer nobody has held for CBM_GRAPH_MEM_IDLE_EVICT_MS
(10 minutes, measured from its last release) is evicted on the next
request: cbm_mcp_handle_tool sweeps after releasing the request store, and
every acquire sweeps too. No timer thread. Ten minutes outlasts the gaps
between an agent's graph calls within one task, while a session that moved
on stops pinning the memory; a wrong guess costs one rebuild (1.2-1.5 s on
the 17 GB kernel index, tens of ms on ordinary repos). The clock is a test
seam (cbm_graph_mem_set_clock_for_testing), so tests move time instead of
waiting.
- Worker budget: application_worker_memory_slice_locked hands index workers
the aggregate minus the resident layer bytes (cbm_graph_mem_cached_bytes),
at least one byte per active job so a slice never reads as "no cap".
- Memory class: layers are charged to the new CBM_MEM_CLASS_GRAPH_CACHE
("graph_cache" in census and class logs) instead of STORE.
Responses stay byte-identical to the baseline on django and redis (tools
and Cypher) and on the kernel trace/hub and extra BFS/degree cases.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
859294561a |
feat(index): the documentation pass follows the index mode by default
Doc-section -> code candidates (doc_link_candidates, #2574/#2610) are the documentation-in-the-graph feature of this release. They now run by default for full indexes and not for moderate or fast ones; an explicit `doc_candidates` argument, or a `doc_candidates` setting the user set, overrides the default either way in every mode. - pipeline: the switch is unset/on/off; unset means on only when the effective mode is full. - MCP: forwards only an explicit choice (argument, then a user-set config value) to daemon jobs and workers; daemon auto-index and watcher/UI jobs now carry a user-set value too (their workers have no config, so `doc_candidates false` was ignored there). - fast + doc_candidates on runs the semantic pass's corpus phases and the doc phase only: candidates, and still no SIMILAR_TO or SEMANTICALLY_RELATED edges. - config: `doc_candidates` joins the config key table (default `auto`); `config set doc_candidates ...` was refused as an unknown key before. README, docs/CONFIGURATION.md and the MCP schema describe the default. Proof (macOS arm64, django fb11376): full 11,304 rows · moderate 0 · fast 0 · full+off 0 · moderate+on 1 · fast+on 1 (moderate/fast discovery skips docs/) -- nodes and edges are identical between on and off in every mode; fast+on adds 0 semantic edges. - New tests (RED on the old code): pipeline pipeline_doc_candidates_default_follows_mode, pipeline_doc_candidates_explicit_wins_over_mode, pipeline_doc_candidates_fast_on_legacy_incremental; mcp tool_index_repository_doc_candidates_follow_mode, tool_index_repository_doc_candidates_forwarded_only_when_explicit; daemon_application daemon_application_programmatic_index_forwards_set_doc_candidates. - mcp 374 passed (4 Windows-only skips), daemon_application 73 passed, pipeline/store_arch/cli/runtime_settings/index_policy/pipeline_frozen green on the release tree. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a19d7a45fb |
Use memory core for Lens response ownership
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0392068d49 |
Merge main into CBM Lens Beta
Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
6bda6e26de |
fix(watcher): report a tree-polled non-git root as strategy "tree"
Since #1948 a non-git root can be polled by a file-tree scan when watch_non_git is on. index_status still reported such a root as strategy "none", which the enum documents as "not a git project: never polled", so a root that is actively polled looked unwatched. publish_status now maps a tree-polled root to the new CBM_WATCHER_STRATEGY_TREE, rendered as "tree" (the value the watcher.baseline log already uses). "none" keeps its meaning: a registered non-git root that is never polled. CONFIGURATION.md lists the four values. Test: daemon_application_index_status_reports_tree_strategy_issue2167 fails without the production change and passes with it; suites daemon_application, watcher and mcp: 507 passed, 4 platform skips. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3794085492 |
fix(watcher): show watch state in index_status, correct auto-sync docs (#2167)
The background watcher only watches the project an open MCP session is
rooted in, and drops the watch when the last session for that project
closes. A repository indexed with `cli index_repository` (or from a
session rooted elsewhere) is never watched, but README promised
"Auto-sync keeps it fresh after that" for every index_repository target
and the MCP instructions said "Indexes auto-refresh". Nothing in the
status output said which projects were actually watched, so a stale
CLI-indexed repo looked like a watcher bug.
What gets watched is unchanged. Instead:
- index_status gains a `watch` object: `watched`, plus `strategy`,
`poll_interval_ms` and `last_scan_at` when watched, or a `reason`
(not_session_project, cli_session, auto_watch_off, watcher_disabled,
not_registered, no_watcher) and a re-index hint when not. It does not
touch the #1561 `freshness` key.
- The watcher publishes per-project strategy, cadence and last completed
scan time through atomics and exposes them via
cbm_watcher_project_info(); the daemon answers index_status through a
session-scoped provider.
- README, docs/CONFIGURATION.md, docs/index.html, the index_status tool
description and the MCP instructions now state the real scope.
`watcher_enabled` already shows in `config list` since v0.11.0 (
|
||
|
Martin Vogel
|
5a9f6e7637 |
Merge pull request #2494 from DeusData/fix/issue-1300-silent-worker-exit
fix(index): report clean worker exits without a response |
||
|
Martin Vogel
|
7acb5eac83 |
fix(index): name a worker that exits cleanly without a response (#1300)
The supervisor creates the worker's response file before spawning it. A worker that exited 0 without writing that file left it empty, and slurp_worker_response read the empty file as "" with status OK. The run then counted as a SUCCESS with an empty response (the daemon marked the job successful). The clean-exit branch of worker_terminal_log also deleted the worker log, the only record of where the run stopped. Only an oversized response was treated as a failure. - index_supervisor: an empty response file is WORKER_RESPONSE_READ_EMPTY. A clean exit with no readable response sets response_missing, keeps the worker log, records its path, and records the last structured event the log holds (text or JSON format, last 64 KiB) as last_phase. It is logged as index.supervisor.no_response with the log path and the last phase. The outcome stays CLEAN, so the disposition is FALLBACK: an explicit error, with no in-process fallback and no recovery re-run of a worker that did not crash. - mcp: cbm_mcp_index_worker_no_response_failure builds the error (reason "no_response", last_phase, worker_log, hint). It is used for the first attempt and for a recovery run. - daemon application: the job failure message says the worker exited cleanly without writing a response, and gives the last phase and the log to inspect. - pipeline_incremental: the legacy per-file purge loop now logs incremental.purge.start (changed/deleted counts) and incremental.purge.progress. Progress lines carry counts only, use a ceiling stride (at most ~20 lines plus the final one) and never log one line per file. Before this, nothing was logged between incremental.edge_snapshot and incremental.purge. Tests (RED before, green after, RED on revert): - index_supervisor_clean_exit_without_response_is_named_failure_issue1300 (a real worker subprocess logs a phase and exits 0 without a response) - daemon_application_clean_exit_without_response_is_named_failure_issue1300 - incr_purge_reports_bounded_progress_issue1300 - index_supervisor_clean_exit_without_response_response_issue1300 (the contract of the response builder) Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
28b41c35b7 |
feat(mcp): opt-in async index_repository with status polling (#2144)
index_repository blocks until the whole index completes. MCP clients with a per-call deadline (Copilot for IntelliJ) give up on a large repository; their cancel drops the daemon job's last subscriber, the daemon cancels the worker, and every retry starts over, so the index never finishes. Add an opt-in async mode and a status query to the same tool; the default synchronous behaviour is unchanged: - async: true starts the project's index job in the daemon, or joins the one already running for it, and returns at once with its state. The application holds one subscriber reference for an async job until its terminal publish, so the request returning, the client cancelling or the starting session closing no longer cancels it. Only daemon shutdown (or the final session of a non-permanent generation) ends it. Async requests never queue behind the physical job limit; a full daemon answers busy. async/status are stripped from the worker args, so an async request coalesces with an identical synchronous one. - status: true reports the running or last job for the project (queued/running/cancelling/succeeded/failed/cancelled, started_at, finished_at, error summary) from a small per-project record kept in the daemon's job registry after the job itself is reaped. An unknown project is an error; a project indexed before this daemon generation reports idle. No freshness key is added: index_status keeps the #1561 freshness object. - async + status together, non-boolean values, and async with cross-repo-intelligence are refused. In-process servers (index worker, embedders) refuse both modes: nothing there outlives the call. - On a temporary (non-permanent) daemon, async is refused when the request arrives on the one-shot `cli` tool channel and that session is the daemon's only live session: the daemon would stop and cancel the job the moment the command exits. The error points at `daemon start` (or an open MCP session). A long-lived MCP session stays a valid host even when it is the only one, which is the IDE case this exists for; status stays allowed everywhere. The daemon lifecycle is unchanged. - New allocations go through the memory core (cbm_alloc/cbm_calloc/ cbm_mem_strdup/cbm_free); blocks returned by non-core APIs are released with safe_free, so no file's raw allocator count grows. - A cut-short synchronous call now offers the async alternative: in the cancellation text the daemon or frontend delivers (including the -32800 reply to a cancelled index_repository), and, because a timed-out client never reads that reply, as a notice on the next index_repository or status call for the project. - The tool description and schema, the CLI help and the README explain the flags, the deadline problem and the polling pattern. No server-side timeout or budget is introduced. Tests (deterministic, held fake worker, bounded observation waits): async returns before completion and status observes running then succeeded; an async job survives its session closing while a sync job in the same shape is still cancelled; validation errors; the cancellation reply and the next-call notice carry the advice; a lone one-shot client of a temporary daemon is refused async while status and an MCP session on the same daemon are allowed; in-process refusal; schema; notice helper. Refs #2031 Fixes #2144 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
92abefa3f5 |
Merge pull request #1723 from liuchong/feat/index-resource-discovery
feat(index): add opt-in discovery resource limits |
||
|
Bernhard Jackiewicz
|
4372d9fada |
feat(ui): integrate repository evidence and change analysis on one daemon
Add source-backed repository navigation, selection context and a shared change analysis workspace for symbols, working-tree changes and Git comparisons. Persist agent events and project logs through the existing daemon with SQLite schema 4, bounded polling, retention and restart-safe deduplication. Repair compact RPC decoding, path-search races, source pagination and identity, large map responses, live error propagation and explicit local diagnosis flows. Include the real-repository browser runners and focused regression coverage. Validation: 2473 frontend tests, 203 acceptance checks, production build, style/promise gates, 5 reference-boundary checks and 7 local hook tests pass. The complete repository-map browser flow also passes after daemon restart. Global C lint and cross-platform release validation remain open and are recorded in the accompanying handoff report. Signed-off-by: Bernhard Jackiewicz <bj@techport.io> |
||
|
Martin Vogel
|
cc6743e9f3 |
Merge pull request #2060 from TheStranjer/fix/follow-own-symlinked-agent-roots
fix(cli): follow agent-root symlinks owned by the invoking user |
||
|
刘冲
|
21ae02154a |
feat(index): add opt-in discovery resource limits
Indexing accepts whatever a repository contains. A tree carrying a vendored monorepo, a generated dump, or a runaway build directory is discovered in full, and the first sign of trouble is a host under memory pressure with nothing that attributes it to indexing. Add two opt-in limits evaluated during discovery against accepted source files only: index_max_files and index_max_source_mb. Both default to off, so nothing changes until an operator sets one. Crossing a limit fails the whole attempt with a structured resource_limit_exceeded result naming the resource, the observed value and the limit; no partial graph is published, and an existing serving index keeps answering. Limits are read from the CLI-managed _config.db and are not MCP request arguments. A supervised parent replaces any caller-supplied policy before spawning its worker, and the worker rejects a missing or incomplete contract, so the CLI, the daemon and the supervised worker all enforce the same decision. Both keys reach an operator through the existing config get/set/list/reset with no new subcommand. `set` suppresses its own generic message for them because the policy writer names the precise reason -- so that writer speaks on every failure it can return, including a validated value whose write then fails on a database that cannot be written. Exiting non-zero in silence is not an acceptable answer from a CLI. The two shell regressions that hand-roll the supervisor's worker argv carry that contract as well. Without it the worker exits before either guard can observe anything, and the guard would go quietly vacuous. Signed-off-by: 刘冲 <mail@liuchong.dev> |
||
|
TheStranjer
|
8622d6a0ed |
fix(cli): make following user-owned symlinks an opt-in per call site
Review of the previous head found the global rule unsound for one caller: cbm_mkdir_p also creates <repo_path>/.codebase-memory inside the checkout being indexed, and git creates symlinks owned by whoever cloned, so a repository shipping `.codebase-memory -> ~/.ssh` would have been followed by the artifact writer. "User-owned" says nothing about "user-intended" once the path root is a checkout. cbm_mkdir_p keeps the strict walk (root-owned links only). The new cbm_mkdir_p_ex(path, mode, policy) with CBM_MKDIR_FOLLOW_OWNED also follows a symlink owned by the invoking user, and only callers whose paths are rooted in the user's own configuration opt in: the installer's agent roots, skills, instruction files and hook directories, the JSON-like config editor's parent creation, the cache root, <cache>/logs, the UI config directory and the DB parent. The artifact directory inside the checkout and the ~/.local/bin creations stay strict; the activation transaction re-walks the latter with its own no-follow policy anyway. What a follow lands on is now checked too: the opened target must be a directory owned by root or the invoking user and not world-writable unless sticky, so a link swapped between the fstatat and the openat cannot steer the walk into a directory an attacker controls or one where anyone can pre-plant entries. This applies to root-owned links as well; macOS /private/tmp is root-owned and sticky and still passes. Refusal restores the first openat's errno. Tests: the plain walk refuses an own link that the opted-in walk follows; targets that are world-writable without the sticky bit are refused; as root, a foreign-owned target and a user-owned link are both refused, each followed by restoring ownership and walking again; the identical `.codebase-memory -> <user-owned dir>` link is refused by the plain walk and followed once opted in; cbm_artifact_export refuses a checked-in artifact-dir link and leaves its target empty; the installer test gains the euid-0 leg with home, link and target owned by a non-root uid. Under euid 0 the tests demote their links to a non-root owner, since root-owned links are trusted by both walks. Refs #1722 Signed-off-by: TheStranjer <791672+TheStranjer@users.noreply.github.com> |
||
|
Martin VogelandClaude Fable 5.1
|
21591d5116 |
fix(daemon): slice the worker memory budget by active jobs, not by the job limit
The daemon handed every index worker aggregate/physical_job_limit (a
fixed quarter by default) regardless of how many jobs were running, so a
lone worker on a host like #1864 was capped at a quarter of the memory
the daemon itself was allowed — and with the budget now failing whole
(decision A) that fixed slice would have turned an idle host into a
guaranteed over_memory_budget. Decision 2 (#1997, #832): the slice is
decided at spawn time as aggregate / jobs active right then (the job
being spawned included), so a lone worker may use the whole aggregate
and concurrent jobs split it. The explicit CBM_MEM_BUDGET_MB override
keeps its precedence inside the worker (
|
||
|
Martin Vogel
|
9846c3f171 |
fix(daemon): compare session roots separator-equivalently without respelling the policy root
The first cut on this branch (
|
||
|
Martin Vogeland刘冲
|
c1b9c451c7 |
fix(daemon): keep one separator spelling for the session root on Windows
On Windows cbm_canonical_path answers in the platform's native backslash form, and the daemon stored that spelling verbatim for the client's session_root and allowed_root (cbm_mcp_server_set_session_context) and for the auto-index job's repo_path (application.c canonicalizes with the bare call at three sites). Every tool handler, however, normalizes separators after canonicalizing. One directory therefore had two names, and the exact comparison in application_index_args_equal never matched: an explicit index_repository request for the session root was refused as OPTIONS_CONFLICT instead of joining the auto-index job already running for it. Normalize the separator spelling of both session roots in set_session_context, and canonicalize plus normalize at the three application.c sites through one helper, application_canonical_root(). Separators fold on every platform, so the new test daemon_session_context_keeps_one_spelling_of_a_root binds wherever the suite runs; it fails on main with "C:\repos\cbm" != "C:/repos/cbm". Distilled from #1726 with co-author credit. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> Co-authored-by: 刘冲 <mail@liuchong.dev> |
||
|
Ertan
|
9d4ac2b98b |
fix(daemon): accept Unicode session roots on Windows
Signed-off-by: Ertan <ertan.kucukoglu@gmail.com> |
||
|
Martin Vogel
|
3a8c0d9758 |
fix: enforce workspace policy in daemon background work
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
8018561cfe |
fix(release): externalize runtime assets and harden VT verification
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
5efa57492a |
fix(daemon): answer an oversized MCP reply with an error, not process death (#1375)
A single query_graph reply larger than one 10 MiB frame killed the MCP server:
exit code 1, no JSON-RPC error, nothing on stderr, no crash report. Under a host
that does not respawn the server, every tool from it was gone for the rest of
the session — the reporter lost their tooling mid-investigation to one dead-code
query and finished by driving the binary over stdio by hand.
Root cause is a conflation, not the limit itself. application_mcp_request
checked only response_length > UINT32_MAX and passed anything smaller on, so an
oversized reply reached the transport, which rejected the frame. The frontend
worker cannot tell that rejection from a dead socket, and for a dead socket it
deliberately _Exit()s the process — closing the kernel IPC handle is the only
portable way to cancel daemon session ownership from a thread blocked in stdio.
Correct for a broken transport; a fatal over-reaction to a large answer.
The substitution therefore has to happen in application_mcp_request, which is
the last layer still holding the request id. Replies that fit are returned
byte-identical; a reply that cannot be framed becomes a JSON-RPC -32603 naming
both numbers and what to do ("narrow the projection, add LIMIT, or paginate"),
because the failure it replaces was a silent exit and an opaque "internal error"
would be no improvement.
Reproduced first, on a locally generated 20k-node fixture rather than the
reporter's repo (external code stays unverified until reviewed):
before LIMIT 10000 -> ok, 8,529,990 bytes
LIMIT 20000 -> SERVER DIED exit=1 stderr=''
after LIMIT 20000 -> JSON-RPC -32603, and a follow-up query on the SAME
session succeeds — which is the property that actually matters
The decision and the substitution live in one function on purpose. The first
version of the test built the error object directly and PASSED with the size
check removed: it proved the error was well-formed while the server still died.
Testing application_framable_response instead binds the branch — verified by
removing the guard, which reddens it at ASSERT(replaced != big).
Not addressed here, deliberately: every reply still carries its payload twice
(content[0].text and structuredContent.text are verbatim duplicates, ~2.05x
measured), which is why the ceiling is reached at ~4 MB of real content. That is
a wire-format question affecting every tool and every caller's token bill, so it
belongs in its own change with a decision behind it, not smuggled into a crash
fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
SS-42
|
fb7a6358ac |
fix(mcp): keep ADR reads nonblocking
Signed-off-by: SS-42 <noreply@incogni.to> |
||
|
Martin Vogel
|
1f674c805e |
harden: remove capability that should never have shipped
Microsoft's ML flagged the rc.1 release binaries. The decisive evidence is that the SAME sha256 went from 0/62 clean to Microsoft-detected in about an hour with no byte change, so the verdict lives partly in scanner-side state and no code change can promise a clean result. What code CAN do is stop shipping things that have no business in a release artifact, which is worth doing on its own merits and incidentally widens the classifier margin. Every claim below is verified against a built binary by the new gate, not by reading source. Executable stack (the worst of the findings). vendored/nomic/code_vectors_blob.S is the only assembly in the build and carried no .note.GNU-stack. An unannotated object makes ld assume the worst for the whole link, so EVERY Linux release we have ever shipped had GNU_STACK RWE. Adds the note (cause) plus ELF-only -Wl,-z,noexecstack (outcome); the gate fails the release if it returns. Test seams are now opt-in, never opt-out. TEST_SEAMS=1 defines CBM_ENABLE_TEST_SEAMS; without it the crash-orphan probe -- which forks a child that ignores SIGTERM and loops forever, then writes its pid to a caller-supplied path -- and the lease-ownership marker compile to trivial stubs, so call sites are untouched and the binary holds no fork, no signal handler and no env-var string. Opt-IN is the point: forgetting the flag yields a clean binary rather than a leaky one. scripts/test.sh requests it in the leg that consumes it, and tests/test_worker_watchdog.sh now asserts the capability up front instead of dying later with an opaque "Killed: 9". The daemon's background version check is gone. It spawned curl against api.github.com/repos/.../releases/latest on the first eligible session of every run to say "a newer version exists" -- a release URL and an outbound request in every shipped binary, for something the install scripts already report. The INJECTABLE SEAM survives: update_ops is still honoured, the fakes in tests/test_daemon_application.c still cover notice/ownership/cancellation/replay, and with no provider application_update_subscribe_locked returns early so no generation ever starts. "No network request by default" is now structural. Dead capability out of release builds. The tar.gz/zip extraction block (gzip_decompress through cbm_extract_binary_from_zip, plus its cli.h declarations) moves under CBM_CLI_ENABLE_TEST_API -- verified self-contained, zero uses of any helper outside it, only callers the excluded updater and tests/test_cli.c. Downloading an archive, decompressing it, picking an executable out of it and marking it executable is the canonical dropper composite; it is now absent rather than merely unreachable. SQLite is built with -DSQLITE_OMIT_LOAD_EXTENSION (no caller of load_extension anywhere in src/ or internal/), removing that API surface and part of the dlopen/dlsym surface. Temp files and environment scanning (S2/S3). Predictable paths in mcp.c, artifact.c and diagnostics.c are created privately and exclusively and written through the returned descriptor; pass_envscan.c no longer descends symlinked directories out of the project root, and its fixed 512-byte path buffers no longer truncate into pointer arithmetic that could land outside the buffer. Build-time entropy. mimalloc's version banner baked __DATE__/__TIME__ into every binary, so two builds of identical source seconds apart could never share a hash and no release could inherit a false-positive determination made about its predecessor. Local patch removes it (marked to survive refreshes), -Wdate-time makes any future use a build error, and -Wl,--no-insert-timestamp stops the PE header carrying the link clock. scripts/ci/check-binary-composition.sh is the proof that each removal stays removed, wired into package-release.sh after strip so the local artifact-flow smoke enforces exactly what CI does. It asserts absences plus a CANARY string, so handing it a compressed, stubbed or empty file fails instead of passing vacuously, and a missing tool is a hard error -- a skipped assertion must never look like a satisfied one. Two build-system traps found by that gate, both of which had silently defeated a fix: the product binary is compiled in one shot from sources, so a flag flip did not rebuild it (now tracked by a .build-config stamp that also removes the binary, making it independent of mtime granularity); and prod_sqlite3.o / prod_mimalloc.o depended on a single named source, so SQLITE_OMIT_LOAD_EXTENSION and the mimalloc patch BOTH compiled to nothing on the first incremental build. Source review would have called them done. Deliberately NOT changed. Three seams stay in release artifacts because scripts/smoke-test.sh runs against the real artifact and needs them: CBM_TEST_CRASH_ON and CBM_TEST_HANG_ON inject the faults that prove supervisor recovery, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID is what keeps the PATH smoke from writing the tester's actual PATH. The gate treats those as an allowlist, so a NOVEL seam still fails. The true no-UI standard build is deferred rather than rushed: src/ui/* is in PROD_SRCS and four files outside src/ui reference UI symbols, including the daemon that serves the UI, so that assertion reports instead of failing until the split lands -- a gate everyone knows is red teaches people to ignore gates. No grammar is removed. ObjectScript accounts for essentially all binary growth since the last provably-clean release (+21.4MB rodata, +1.1MB text from two four-line shims), which made it the obvious ablation candidate, but a dry run performed twelve real Defender endpoint scans across standard/UI and amd64/arm64 with ObjectScript, the daemon and the expanded hooks all present and every scan was clean. Nothing there is a deterministic trigger, so cutting a community-contributed language would spend a real feature on unproven margin. Lean is not a candidate either: at 99.6MB of source it is by far the largest grammar, but it shipped in v0.9.0 which scanned 20/20 clean, so removing it would produce a novel unscanned profile instead of restoring a known-good one. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
87f7c0d784 |
daemon: deterministic job interleavings and queued explicit index admission
Two changes to the index coordinator, developed together and validated as one state: Determinism seams. A cancel racing a worker start was only observable on a loaded runner; now the interleaving is forced by construction: a test hold parks every job thread before its first pre-start cancel check, and a background-initialize completion counter turns sleep-then-assert negatives into positive-signal waits. The cancel-before-worker-start case is pinned by a dedicated test; three auto-index admission tests now wait for the admission tail instead of sleeping through it. Queued admission. An explicit index_repository request that hits the physical job limit used to surface a raw busy error. The request thread already blocks for the whole index, so it now queues: a cancellable, shutdown-aware wait-and-resubscribe loop replaces the error path, with a wait counter as the parked-state observable. The programmatic coordinator API keeps its non-blocking busy answer for callers with their own retry policy (watcher, auto-index). The job-limit test asserts the new contract: parked without a second worker start, admitted and completed once the occupying job finishes. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0baddf7887 |
daemon: permanent lifecycle, daemon-backed CLI/hooks, real-Windows hardening, long-path launcher transactions
Daemon lifecycle and Windows correctness, verified on a real Windows 11 ARM64 VM through the maintained test-infrastructure/vm drivers, plus the macOS and Linux arm64 suites and the container lint gate. Daemon lifecycle: - daemon start/stop/status subcommands. `daemon start` launches a PERMANENT daemon (spawn shape is byte-exact argv; survives idle periods and session ends) and reports an already-active daemon instead of failing. Permanence is honored at every stop latch: last-committed-client disconnect, host initial-client window, coordinator release, and application final-session close — a permanent daemon also keeps admitting new sessions after its last one closes. - daemon stop refuses while sessions are active and lists the blocking peers (pid/role) that must finish first; an idle daemon drains through the activation-shutdown machinery with the ACK ordered after connection interrupts. A second stop is idempotent. The wire ops are no-cohort first-frame requests with peer fingerprint authentication, so stop/status never conflict with an exact-build admission gate. - One-shot CLI commands now execute through the daemon (index workers keep their local supervised path). A cold CLI run that had to spawn a temporary daemon prints a hint that `daemon start` removes the per-command startup tax; a warm daemon is recycled silently. - Hooks are connect-only fail-open: with no daemon present the hook emits a visible, rate-limited notice (Claude-dialect systemMessage plus stderr for other dialects) and always exits 0 — augmentation is never allowed to block the caller's tool use. - Version skew: a newer-build client automatically drains an older-build permanent daemon (strict semantic-version triples only; dev builds never auto-drain) and the build-conflict message names `cbm daemon stop` as the manual escape hatch. Windows IPC/runtime (real-VM verified): - ipc(win): persistent pending overlapped ConnectNamedPipe. The accept path used to destroy its listening pipe instance on every 20 ms poll timeout; a client attaching in the teardown window was severed or left on an orphaned pipe object whose HELLO no server handle could ever read, absorbing the connect until the client's own timeout expired. The pending connect now survives poll timeouts and nothing is destroyed while a client could be attaching. - ipc(win): drain-before-close for final responses. Closing a named-pipe server handle can discard a just-sent response before the peer reads it (POSIX stream sockets never lose buffered data on close). A bounded cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX, immediate on interrupted connections) now precedes close in runtime_worker_finish and runtime_reject_inline, so hello-conflict, capacity and disconnect acknowledgements reliably reach the peer. - runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no transport half-close, so close_begin now announces departure with an explicit frame (ordered after APPLICATION_CANCEL, before the local interrupt); the server releases the client's admission on receipt instead of waiting for the handle to close. Admission-drop timing is now identical to POSIX shutdown() semantics on every platform. - runtime(win): client close cancellation. close_begin serializes with request publication under the send lock, best-effort sends the active token's APPLICATION_CANCEL frame, then interrupts local I/O; the server cancels MCP/subprocess work promptly. Contract tests accept both correct outcomes (interrupted transport or decoded CANCELLED). - runtime: activation acknowledgement ordering. The activation ACK is the requester's license to act on "snapshotted and draining", so every connection interrupt is now initiated before the ACK is sent; a session could previously get one more request serviced after the requester observed the ACK. - service(win): deadline-bounded private-file prepare. The conflict-log prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms timer granularity) burned ~1.6 s against permanently obstructed paths, stalling hello rejections past the client's timeout. The retry budget is now a 250 ms deadline; transient share collisions still retry. - subprocess(win): cmd.exe /C payload encoder quotes metacharacters correctly (root cause of the git-on-Windows failure cluster). - watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native Windows stop/unwatch cancellation test with exact-image verification. - httpd: send_all writes in bounded 64 KiB slices. A single giant nonblocking send() on Windows is absorbed wholesale into AFD kernel buffering regardless of SO_SNDBUF, so send deadlines and interrupts could never engage against a slow peer (and the full payload was pinned in nonpaged pool). Slicing restores a deterministic backpressure point; a test hook pins SO_SNDBUF for the deadline and interrupt tests. - ui/http: shutdown lifecycle — interrupt checks, response-wide send deadline, explicit connection states, refusal to free a server while a listener-owned connection is active. Windows long-path support: - Central path-aware wide conversion (canonicalize via GetFullPathNameW and prepend the extended-length prefix for absolute paths >=240) at the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite store opens, and the daemon build-fingerprint/log paths. Deep managed installs (a 64-hex generation directory routinely exceeds MAX_PATH) now index, stage and activate correctly. - activation transaction: its own file APIs and the component-walking ancestry validators now operate in the extended-length namespace; the launcher path is canonicalized (and prefixed when deep) once at entry so every downstream exact-string comparison stays form-consistent. - Executable self-resolution uses the wide APIs (GetModuleFileNameW, GetFileAttributesW) so non-ASCII install paths survive argv[0] resolution. Windows launcher install/uninstall transaction: - FileRenameInfoEx names are NUL-terminated in an over-allocated buffer. FileNameLength governs per the contract, but filter drivers read FileName as NUL-terminated and appended adjacent heap bytes to created names — a flaky, garbage-suffixed rename target. Both the CLI and the launcher rename helpers are fixed. - Uninstall retires state via rename-aside (.cbm -> .cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex chars so the bare rename target stays under the FileRenameInfoEx NT-conversion ceiling at guard depths; 64 bits still uniquely identify the generation. - When the running launcher's mapped generation backings pin .cbm against rename, the backings are relocated to activation-<pid>-N .retired tombstones beside the install (a mapped image may be renamed, never deleted; the launcher's liveness-guarded sweep reclaims stale tombstones). Every relocation is recorded, and a FAILED uninstall reverses the moves after restoring .cbm — via MoveFileExW with extended-length paths on both arguments, since the deep generation target is beyond the handle-based rename's bare-path reach — so a restored install keeps its generation backings and stays runnable. - After a committed uninstall the retired tree's backings are relocated out so the tree is shallow enough for the detached cleanup's rd, and the cleanup's working directory strips the extended-length prefix (CreateProcessW lpCurrentDirectory silently ignores prefixed paths). - Files created under Administrators-default-owner directories (CopyFileW destinations, CREATE_NEW tombstones, probe directories) are explicitly owner-stamped so the exact-owner validators hold on runner images; guard fixtures stamp hand-built trees the same way. Diagnostics, tests and infra: - diagnostics: discovery is now an always-delivered JSON control record (new cbm_log_control) that survives CBM_LOG_LEVEL suppression and paths containing spaces; placement honors $TMPDIR with /tmp fallback via a diagnostics-local helper; the soak parser reads the JSON record; documented in docs/CONFIGURATION.md. Red-first coverage for suppressed log levels, TMPDIR-with-spaces, and native Windows output-contract assertions. - tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint parents with production-shaped ancestry (LocalAppData on Windows, via th_secure_runtime_parent_new) — the runtime ancestry validation correctly refuses temp roots whose ancestors grant mutation rights to Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and drive the documented startup-owner publication flow before reading generation-bound endpoint addresses. This turns the 26 Windows failures previously visible in CI's full-test job green without weakening any validation. - tests(win): the launcher guard covers the full permanent-launcher contract including failed-uninstall restore and immediate reinstall after uninstall; new daemon lifecycle and reworked hook-augment guards run the start/recycle/stop flow end to end. - tests: CBM_SKIP_PERF is now actually consumed by the test runner (it was set by CI but never read, so perf suites ran everywhere); four throughput/bench suites are classified as perf, the heavy store_arch suite moved to the slow-timeout tier, and two wall-clock-sensitive assertions were rewritten as invariant checks with coarse hang-detector backstops. - build/test infra: build-dir safety contract, UI dev-proxy security contract, soak daemon-recovery contract, path-safety helper, the Windows VM worktree-sync contract wired into scripts/test.sh, and vm/win.sh guards building its clean embedded-UI product in an isolated BUILD_DIR so it cannot clobber the incremental test build. provision-windows.sh now installs Node.js for the guards UI build. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
83c137d2a5 |
feat: complete shared daemon lifecycle
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0e00ef5702 |
feat: coordinate concurrent CBM sessions
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |