Commit Graph
29 Commits
Author SHA1 Message Date
Martin Vogel 5a9c72d8f8 test(watcher): pin the pending-free drain on a held state, not a git probe
watcher_unwatch_drains_pending_free failed three times on macos-14 CI
with `index_call_count == 0, expected 1` at its dirty-poll precondition.
That precondition needs every git subprocess of one poll (rev-parse
HEAD, status, submodule foreach) to succeed. A transient probe failure
(spawn refused under runner load, a non-zero exit) makes that poll a
fail-closed no-op by design: #937 leaves the baseline uncommitted and
the next poll retries. Several of those paths return without a log
line, which matches the CI output exactly: watch, baseline, FAIL, with
no watcher.changed and no watcher.git.failed. Holding a spawn failure
with cbm_subprocess_force_spawn_eagain_for_testing reproduces that
output deterministically. This is a test race, not a production defect.

The precondition was also unrelated to the contract under test. A poll
that stopped draining would neither crash nor leak, because
cbm_watcher_free frees the same deferred list. So the old test could not
see a broken drain at all.

Add a test-only seam, cbm_watcher_test_pending_free_count, and assert
the stable states the watcher holds: 0 parked before unwatch, 1 after,
0 after the next poll with no index work, and still 0 after a further
poll. No assertion depends on a git probe succeeding. Dirty-poll
reindexing stays covered by watcher_detects_dirty_worktree and
watcher_modify_tracked_file.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-26 00:27:53 +02:00
Martin Vogel e321b1fcea Merge branch 'main' into fix/watcher-index-failure-backoff 2026-09-21 07:40:04 +02:00
Martin Vogel 9e1faa7327 fix(watcher): exclude the indexer's own artifact dir from the dirty signature
After every publish the pipeline re-exports <root>/.codebase-memory/ (graph.db.zst + artifact.json) whenever an artifact already lives there: once a session indexed with persistence, or when the team committed it and every linked worktree checks it out. The watcher's dirty signature folded those files' (size, mtime) in, so each successful reindex rewrote them, the next poll saw a new dirty state, and the daemon re-triggered itself forever: index.supervisor.reap outcome=clean immediately followed by watcher.changed strategy=git, 100+ times in 15 minutes with index workers pinned (#1953). Nothing under .git moved and no source changed, which is why the reporters blamed their long hyphenated worktree branch names (the #1254 theory its own reporter retracted); the regression test builds that exact worktree and shows the branch is irrelevant.

Scope git status with the same :(exclude).codebase-memory pathspec the exporter's own clean-tree probe already uses for the same reason. Nothing under that directory is ever an index input (discovery skips it), so a change there can never require a reindex; the pathspec is CWD-relative, so a monorepo sub-package watch excludes its own artifact directory. Untracked and committed artifacts both looped before; a real edit and a real HEAD move are still detected.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-08 00:11:39 +02:00
Shane McCarronandClaude Opus 5 086422020f fix: address QA round 3
Documentation and claim accuracy only — no behavioural change. Two of the
three findings are corrections to inaccurate statements the round-2 commit
(61ccd905) introduced.

The gating test's framing was overstated, and this time the overstatement was
measured rather than argued. Replacing the backoff call with the pre-change
`ctx->now + interval_ms * US_PER_MS` leaves the watcher suite green at 82/82:
the index_backoff_* tests pass because they exercise the pure function
directly, which the reverted scheduler no longer calls, and the streak tests
pass because the counter is untouched. So no test in this suite fails when
this change's behaviour is removed. The test now records that, with the
figure, instead of claiming it demonstrates the backoff. It is a regression
guard on a deadline being assigned at all — real, but narrower than round 2
said.

Closing that gap needs either a controllable clock or a further accessor
exposing next_poll_ns. Both were judged out of scope: a third exported symbol
purely for testing, on a change already carrying two, is the API widening
earlier rounds flagged. The gap is recorded in the test, the PR description
and the round-3 note rather than papered over.

The memory-visibility comment claimed index_failures follows "the same
discipline every other per-project field in this struct already follows".
That is false: active_git is serialized by projects_lock and registered is an
atomic_bool. Narrowed to the poll-mutated fields, which are now enumerated.
The round-2 decision to document rather than lock still stands — the panel
verified the underlying premise — but it was resting on an overbroad claim.

watcher.h said the delay "doubles up to a fixed ceiling", which skips the
shift cap: the delay plateaus at interval_ms << INDEX_FAIL_SHIFT_MAX, and
that reaches the ceiling only for interval_ms >= 4688 ms. Every interval this
watcher generates is >= POLL_BASE_MS so the ceiling is always reached in
practice, but the function is exported and a caller may pass less.

Refs #2015

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FbrM52pmB1m2SR1vRRoNnf
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
2026-09-02 15:28:21 -05:00
Shane McCarronandClaude Opus 5 61ccd90511 fix: address QA round 2
Retracts an incorrect justification from the round-1 commit (4fc5999a) and
closes the coverage it was used to excuse.

MAJOR — the delay wire-up had no coverage, and the reason given for that was
wrong. Round 1 claimed the gating half was "structurally unobservable without
an injectable clock". It is not. cbm_watcher_touch zeroes next_poll_ns, so a
wire-up that never sets a deadline leaves it at zero and the callback fires on
every poll: simply NOT calling touch between polls observes gating
deterministically, no clock involved. Only the delay's MAGNITUDE needs one.

watcher_index_failure_backoff_gates_repolling_issue2015 covers it — one hard
failure, then five polls without touch asserting no re-fork, then a touch
proving the retry still gets through (so the test cannot pass by observing a
dead watcher). Mutation-verified: deleting the wire-up yields
"failing_index_calls == 6, expected 1".

MINOR — round 1's F1 was half-fixed. Of the two single-token mutations that
finding named, only the streak reset was covered; widening the
sustained-failure "==" to ">=" still passed. watcher_sustained_failure_logs_
once_issue2015 drives 14 consecutive failures through a cbm_log_set_sink and
asserts exactly one emission. Mutation-verified: ">=" yields
"sustained_log_hits == 5, expected 1".

All three mutations F1 named are now demonstrated caught rather than argued
about: reset removal (round 1), wire-up deletion, and the "==" widening.

MINOR — cbm_watcher_index_failure_count reads index_failures under
projects_lock while poll_project writes it lock-free from a state snapshot.
That is the discipline every other per-project field here already follows, so
this documents the visibility contract rather than making one field atomic in
isolation: the accessor is a diagnostic and test seam, not a synchronisation
point, and must not carry scheduling decisions without atomic accessors first.

MINOR — watcher.h still described the pre-F3 backoff contract. It now states
the clamp is a floor as well as a cap, why the floor exists for an exported
function whose reachable inputs cannot currently hit it, and that negative
inputs are treated as zero.

Still uncovered, now stated precisely rather than hand-waved: the delay's
MAGNITUDE. Nothing asserts that ten failures produce a five-minute delay
rather than a five-second one. Gating is proven; the numbers are not, and
that does need a controllable clock.

Refs #2015

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FbrM52pmB1m2SR1vRRoNnf
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
2026-09-02 14:56:05 -05:00
Shane McCarronandClaude Opus 5 4fc5999a85 fix: address QA round 1
F1 (major) — the failure state machine the backoff feeds had no test
coverage. Deleting the streak reset, or widening the sustained-failure
comparison, left all 7788 tests green: the index_backoff_* tests cover the
delay arithmetic, and nothing covered the counter that selects which delay
applies. A streak that never increments silently degrades the backoff to the
unbounded retry it exists to prevent.

Adds cbm_watcher_index_failure_count(), alongside the existing
cbm_watcher_watch_count() in watcher.h's "Introspection (for testing)"
section, and a test that drives a real git repo through fail/fail/succeed on
the failing_index_callback seam already used by the #937 test, asserting the
streak goes 0 -> 1 -> 2 -> 0. Verified by mutation: removing the reset turns
it red at the reset assertion while every index_backoff_* test stays green.

The delay-GATING half stays uncovered on purpose. Every integration test
calls cbm_watcher_touch first, which zeroes next_poll_ns, so whether the
scheduler honours the computed deadline is unobservable without an
injectable clock the watcher does not have. That harness is a larger change
than this fix.

F3 (minor) — cbm_watcher_index_backoff_ms was non-monotonic for an interval
above the ceiling: zero failures returned the interval, one failure returned
the smaller ceiling. Unreachable today (POLL_MAX_MS < the ceiling), but the
function is exported, so the clamp is now a floor as well as a cap and
backing off can never schedule sooner than the project's own cadence.

F2 (minor) needed no code change: a busy-skip following an unresolved
failure streak keeps the backed-off delay, which is deliberate — only a
success clears the streak, and treating rc > 0 as recovery would let a
project alternating fail/busy never back off at all. The PR description's
over-strong "busy-skip is behaviourally unchanged" claim is corrected there.

Refs #2015

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FbrM52pmB1m2SR1vRRoNnf
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
2026-09-02 14:18:02 -05:00
Shane McCarronandClaude Opus 5 76e54f7525 fix(watcher): back off re-forking a hard-failing index worker
A hard index failure (index_fn < 0) left the retry cadence untouched, so a
persistently failing project re-forked an index worker every poll for as
long as the daemon lived. Observed on 0.10.8/macOS: 2233 consecutive failed
workers over 4h43m, ~8/min, every one dying at the same worker-side
coordination seal. Read queries stayed healthy throughout, so nothing
surfaced to the user while no project could be indexed at all.

#937 deliberately leaves the baseline uncommitted on a failed reindex so the
change is retried rather than lost. That guarantee is kept; what changes is
the cadence. Each consecutive hard failure doubles the delay, capped at five
minutes, taking a permanently failing project from ~480 attempts/hour to
~12 while still recovering on its own once the cause clears. Busy-skip
(rc > 0) and success paths are untouched.

The delay arithmetic is a pure exported helper so it is unit-testable
without a clock. watcher.index.err now carries rc and the consecutive-
failure count, and a distinct watcher.index.sustained_failure names a
project whose failures are clearly not transient.

Note: itoa_buf returns a single shared per-thread buffer, so the two-value
log line uses local buffers rather than two itoa_buf calls.

Refs #2015

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FbrM52pmB1m2SR1vRRoNnf
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
2026-09-02 13:02:34 -05:00
Martin Vogelandbethzyy 1af49dfe58 fix(watcher): stop nested non-git dirs and monorepo siblings retriggering
Three related defects in how the watcher decides what counts as a change.

1. `git rev-parse --git-dir` walks UP the tree, so an ordinary folder that
   merely sits under an unrelated repository answered yes. The watcher then
   inherited that ancestor's dirty state -- permanently non-empty and about
   entirely different files -- and reindexed on every poll forever. A
   directory is now git-managed only if it carries its own .git (directory
   or gitlink file) or the ancestor actually tracks something inside it, so
   a real monorepo sub-package still qualifies while a scratch or ignored
   folder does not.

2. `git status` reports the whole repository regardless of -C, so every
   package in a monorepo reindexed whenever any sibling was edited. The
   status call is now scoped with a `-- .` pathspec.

3. Porcelain paths are REPOSITORY-relative but were stat'ed against
   root_path. For a project watched at the repository root the two coincide
   and the bug is invisible; for a subdirectory project every stat missed,
   silently degrading the dirty signature to text-only and losing the
   size/mtime component that makes an edit to an already-dirty file
   detectable. Paths now resolve through `rev-parse --show-cdup`.
   --show-cdup rather than --show-toplevel because MSYS/Cygwin git returns a
   translated absolute path from the latter that will not join onto the
   native path we hold; a relative hop composes on every platform.

Distilled from #1001 by bethzyy, whose diagnosis identified all three.
Main independently fixed that PR's two headline problems (the unbounded
non-git file count, and per-poll churn via a dirty-state signature) while
it waited, and the watcher was rewritten onto a supervised argv-spawn
harness in between, so this is a reimplementation against the new code
rather than a rebase of theirs.

Both new tests are verified binding: reverting the classification guard
reddens the nested-directory test, and removing the pathspec reddens the
monorepo one, each without disturbing the other.

Addresses the remaining parts of #713, #841 and #937.

Co-authored-by: bethzyy <bethzyy@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-20 16:43:38 +02:00
Martin Vogel 0baddf7887 daemon: permanent lifecycle, daemon-backed CLI/hooks, real-Windows hardening, long-path launcher transactions
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.

Daemon lifecycle:

- daemon start/stop/status subcommands. `daemon start` launches a
  PERMANENT daemon (spawn shape is byte-exact argv; survives idle
  periods and session ends) and reports an already-active daemon
  instead of failing. Permanence is honored at every stop latch:
  last-committed-client disconnect, host initial-client window,
  coordinator release, and application final-session close — a
  permanent daemon also keeps admitting new sessions after its last
  one closes.
- daemon stop refuses while sessions are active and lists the blocking
  peers (pid/role) that must finish first; an idle daemon drains
  through the activation-shutdown machinery with the ACK ordered after
  connection interrupts. A second stop is idempotent. The wire ops are
  no-cohort first-frame requests with peer fingerprint authentication,
  so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
  keep their local supervised path). A cold CLI run that had to spawn a
  temporary daemon prints a hint that `daemon start` removes the
  per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
  emits a visible, rate-limited notice (Claude-dialect systemMessage
  plus stderr for other dialects) and always exits 0 — augmentation is
  never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
  older-build permanent daemon (strict semantic-version triples only;
  dev builds never auto-drain) and the build-conflict message names
  `cbm daemon stop` as the manual escape hatch.

Windows IPC/runtime (real-VM verified):

- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
  path used to destroy its listening pipe instance on every 20 ms poll
  timeout; a client attaching in the teardown window was severed or left
  on an orphaned pipe object whose HELLO no server handle could ever
  read, absorbing the connect until the client's own timeout expired.
  The pending connect now survives poll timeouts and nothing is
  destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
  server handle can discard a just-sent response before the peer reads
  it (POSIX stream sockets never lose buffered data on close). A bounded
  cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
  immediate on interrupted connections) now precedes close in
  runtime_worker_finish and runtime_reject_inline, so hello-conflict,
  capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
  transport half-close, so close_begin now announces departure with an
  explicit frame (ordered after APPLICATION_CANCEL, before the local
  interrupt); the server releases the client's admission on receipt
  instead of waiting for the handle to close. Admission-drop timing is
  now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
  request publication under the send lock, best-effort sends the active
  token's APPLICATION_CANCEL frame, then interrupts local I/O; the
  server cancels MCP/subprocess work promptly. Contract tests accept
  both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
  the requester's license to act on "snapshotted and draining", so every
  connection interrupt is now initiated before the ACK is sent; a
  session could previously get one more request serviced after the
  requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
  prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
  timer granularity) burned ~1.6 s against permanently obstructed paths,
  stalling hello rejections past the client's timeout. The retry budget
  is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
  correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
  Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
  nonblocking send() on Windows is absorbed wholesale into AFD kernel
  buffering regardless of SO_SNDBUF, so send deadlines and interrupts
  could never engage against a slow peer (and the full payload was
  pinned in nonpaged pool). Slicing restores a deterministic
  backpressure point; a test hook pins SO_SNDBUF for the deadline and
  interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
  deadline, explicit connection states, refusal to free a server while
  a listener-owned connection is active.

Windows long-path support:

- Central path-aware wide conversion (canonicalize via GetFullPathNameW
  and prepend the extended-length prefix for absolute paths >=240) at
  the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
  store opens, and the daemon build-fingerprint/log paths. Deep managed
  installs (a 64-hex generation directory routinely exceeds MAX_PATH)
  now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
  ancestry validators now operate in the extended-length namespace;
  the launcher path is canonicalized (and prefixed when deep) once at
  entry so every downstream exact-string comparison stays
  form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
  GetFileAttributesW) so non-ASCII install paths survive argv[0]
  resolution.

Windows launcher install/uninstall transaction:

- FileRenameInfoEx names are NUL-terminated in an over-allocated
  buffer. FileNameLength governs per the contract, but filter drivers
  read FileName as NUL-terminated and appended adjacent heap bytes to
  created names — a flaky, garbage-suffixed rename target. Both the CLI
  and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
  .cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
  chars so the bare rename target stays under the FileRenameInfoEx
  NT-conversion ceiling at guard depths; 64 bits still uniquely
  identify the generation.
- When the running launcher's mapped generation backings pin .cbm
  against rename, the backings are relocated to activation-<pid>-N
  .retired tombstones beside the install (a mapped image may be renamed,
  never deleted; the launcher's liveness-guarded sweep reclaims stale
  tombstones). Every relocation is recorded, and a FAILED uninstall
  reverses the moves after restoring .cbm — via MoveFileExW with
  extended-length paths on both arguments, since the deep generation
  target is beyond the handle-based rename's bare-path reach — so a
  restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
  out so the tree is shallow enough for the detached cleanup's rd, and
  the cleanup's working directory strips the extended-length prefix
  (CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
  (CopyFileW destinations, CREATE_NEW tombstones, probe directories)
  are explicitly owner-stamped so the exact-owner validators hold on
  runner images; guard fixtures stamp hand-built trees the same way.

Diagnostics, tests and infra:

- diagnostics: discovery is now an always-delivered JSON control record
  (new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
  paths containing spaces; placement honors $TMPDIR with /tmp fallback
  via a diagnostics-local helper; the soak parser reads the JSON record;
  documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
  log levels, TMPDIR-with-spaces, and native Windows output-contract
  assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
  parents with production-shaped ancestry (LocalAppData on Windows, via
  th_secure_runtime_parent_new) — the runtime ancestry validation
  correctly refuses temp roots whose ancestors grant mutation rights to
  Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
  drive the documented startup-owner publication flow before reading
  generation-bound endpoint addresses. This turns the 26 Windows
  failures previously visible in CI's full-test job green without
  weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
  contract including failed-uninstall restore and immediate reinstall
  after uninstall; new daemon lifecycle and reworked hook-augment
  guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
  (it was set by CI but never read, so perf suites ran everywhere);
  four throughput/bench suites are classified as perf, the heavy
  store_arch suite moved to the slow-timeout tier, and two
  wall-clock-sensitive assertions were rewritten as invariant checks
  with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
  contract, soak daemon-recovery contract, path-safety helper, the
  Windows VM worktree-sync contract wired into scripts/test.sh, and
  vm/win.sh guards building its clean embedded-UI product in an
  isolated BUILD_DIR so it cannot clobber the incremental test build.
  provision-windows.sh now installs Node.js for the guards UI build.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-21 15:32:15 +02:00
Martin Vogel 83c137d2a5 feat: complete shared daemon lifecycle
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 01:26:08 +02:00
Martin Vogel 0e00ef5702 feat: coordinate concurrent CBM sessions
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-16 19:20:46 +02:00
Martin Vogel 9789a7c55e fix(watcher): reindex dirty repos once per distinct state, not per poll
A persistently dirty or untracked worktree re-triggered a full reindex
(and its DB/artifact rewrite) on every poll cycle - git_is_dirty()
answered 'is the tree dirty' where the watcher needs 'did the tree
CHANGE'. Idle repos with one uncommitted file produced sustained disk
write amplification (reported at 1 TB/day across several repos).

The watcher now tracks a dirty-state signature: FNV-1a over the
entries of git status --porcelain -uall -z plus each listed path's
(size, mtime_ns). Same signature -> no reindex; editing a dirty file,
adding/removing one, or reverting the tree to clean each produce a new
signature and trigger exactly one reindex. Submodule status is folded
in on POSIX, mirroring the old submodule dirty probe. 0 is reserved
for a clean tree; a dirty-at-baseline tree still reindexes once (the
watcher cannot know whether that state reached the DB, e.g. after a
server restart with a stale artifact).

Baselines (HEAD and signature) are now committed only after a
SUCCESSFUL reindex: check_changes stages observations in pending_*
fields, and busy-skips (index_fn > 0, e.g. pipeline lock held) or
failed runs leave the baselines untouched so the change is retried on
the next poll instead of being recorded as seen and silently lost -
the old code stored the new HEAD at check time, dropping any commit
that arrived while the pipeline was busy.

Closes #937

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-12 13:34:42 +02:00
Martin VogelandDustin Persek 96cb6d188c fix(ui): unwatch projects on UI delete; harden deferred-free OOM corner
DELETE /api/project removed the project's .db file but never told the
watcher; the zombie watch kept polling the repo and the next change
silently resurrected the deleted DB (#803). Wire the watcher into the
UI HTTP server (non-owned ref; created before the server, freed after
it) and unwatch after a successful unlink and on ENOENT (clears an
already-orphaned watch, still 404). Invalid name stays a plain 404
with no unwatch; a non-ENOENT unlink failure stays 500 and keeps the
watch. The cbm_file_exists pre-check is replaced by unlink-errno
handling, removing the TOCTOU window.

Distilled from #830, rebased over the defer_state_free refactor
(#817): the OOM hardening of the deferred-free path moves into
defer_state_free itself so BOTH unwatch and stale-root prune get it.
On pending-free realloc failure the helper no longer frees the state
immediately (a potential use-after-free against an in-flight poll_once
snapshot) — it logs watcher.unwatch.oom and returns failure, and the
callers keep the entry registered; the prune path retries on the next
poll cycle.

Carries six httpd regression tests from #830 driving the live UI
server with a real watcher wired in.

Closes #803.

Co-authored-by: Dustin Persek <dustin.persek@protonmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-04 11:56:43 +02:00
Martin Vogelandpcristin e5bb74916a feat(watcher): prune projects whose root stays missing (ENOENT-only, grace window)
Distilled from #738: the watcher now prunes a watched project whose root
directory has genuinely disappeared - deleting the cached DB (+wal/shm,
validated name, cache-dir-only paths) and removing the watch entry - so
vanished worktrees stop being watched forever (#286).

Hardened beyond the original PR to remove a data-loss hazard (the cached
DB can hold user-authored data such as the ADR, unrecoverable once
deleted):

- Only ENOENT/ENOTDIR stat failures count as missing. Any other failure
  (EACCES, EIO, transient mounts, macOS TCC revocation) resets the
  streak and logs watcher.root_stat_error with the errno; Windows
  (mingw/UCRT) maps not-found to ENOENT so the check holds there.
- Pruning requires BOTH >=3 consecutive missing polls AND a sustained-
  absence grace window since the streak's first miss (default 600s,
  override via CBM_WATCHER_PRUNE_GRACE_S), tracked with monotonic
  cbm_now_ms.
- Root reappearance resets streak + timestamp (watcher.root_restored).
- The pruned entry is released via the deferred-free list poll_once
  drains (one freeing model shared with cbm_watcher_unwatch).

Limitation: only currently-watched projects are pruned; stale DBs left
by older sessions are out of scope.

Co-authored-by: pcristin <xxxokzxxx@protonmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-03 20:40:22 +02:00
Gol3vka 1c602c6c34 fix(watcher): guard cbm_watcher_free and add deferred-free test
Signed-off-by: Gol3vka <gol3vka@163.com>
2026-06-26 09:38:05 +08:00
Gol3vka 7814d960f3 format src/watcher.c
Signed-off-by: Gol3vka <gol3vka@163.com>
2026-06-26 09:36:20 +08:00
Gol3vka 2bc4a539ac fix(watcher): defer state_free in unwatch to prevent UAF
Signed-off-by: Gol3vka <gol3vka@163.com>
2026-06-26 09:36:20 +08:00
Martin Vogel a2c9804397 Make the watcher git helpers work under Windows cmd.exe
The watcher shells out to git via cbm_popen, which runs through cmd.exe on Windows. The commands used POSIX-only constructs: single-quoted paths ('%s'), which cmd.exe does NOT strip — so git received a literal-quoted path, failed with 'cannot find the path', and the watcher fell back to strategy=none (no change detection); plus /dev/null and '| wc -l'. Use double-quoted paths (stripped by cmd.exe and POSIX sh alike), a platform null device (NUL / dev/null), count ls-files lines in C, and gate the submodule-foreach enhancement (whose inner shell command cmd.exe cannot pass) to POSIX. The watcher's git change-detection now works on Windows; this is what the (newly un-skipped) watcher tests exercise.
2026-06-06 19:05:40 +02:00
Martin Vogel c768b6154d fix(watcher): close residual gaps in #207
Follow-up to #207:
- cbm_watcher_new: NULL-check cbm_ht_create() and free w on OOM so
  later watch/unwatch/touch don't dereference a NULL hash table.
- cbm_watcher_watch_count: drop const from the signature and remove
  the const_cast around projects_lock. The function necessarily
  touches synchronization primitives so a const receiver was
  misleading; all current callers pass a non-const pointer.
2026-05-08 21:33:06 +02:00
Matthew Prock 8fbdb0fa20 fix(thread-safety): eliminate races in log mutex, watcher, and index threads (#207)
- http_server: replace lazy log mutex init with 3-state UNINIT/INITING/INITED CAS pattern; cbm_ui_log_init() called from main before thread creation, also still invoked lazily from cbm_ui_log_append() so early single-threaded logs are not dropped
- watcher: protect projects hash table with cbm_mutex_t (watch/unwatch/touch/watch_count/free); poll_once snapshots project pointers under lock then iterates without it so git I/O and indexing don't block watch/unwatch
- watcher: handle state_new() OOM in watch path
- compat_thread: add cbm_thread_detach() for POSIX and Windows; both join() and detach() clear the handle on success for consistent lifecycle tracking
- http_server: detach index job threads to prevent handle leaks
2026-05-08 21:28:00 +02:00
Martin Vogelandhalindrome 7f79ce53a1 Fix watcher: detect dirty state inside git submodules
The watcher's git_is_dirty() only ran git status on the parent repo,
missing uncommitted changes inside submodules. Add a second check via
git submodule foreach --recursive (portable, works on Apple Git which
lacks --recurse-submodules).

Fixes #71

Co-Authored-By: halindrome <halindrome@users.noreply.github.com>
2026-04-03 01:22:51 +02:00
Martin Vogel 3908624fad Fix remaining lint: cppcheck variableScope, dead code, const correctness
- platform.c: move nlen into if(env) block (variableScope)
- userconfig.c: move home into else block (variableScope)
- pipeline.c: remove dead rc!=0 check after httplinks removal
- cli.c: rmdir_recursive propagates cbm_rmdir errors, remove redundant local
- clang-tidy: whitelist cosmetic-only checks, enable all safety checks
- Thread-safe getenv via environ iteration (concurrency-mt-unsafe fix)
- Multiple files: add constants.h includes for named constants
2026-04-02 12:46:39 +02:00
Martin Vogel 7fa3acd0c6 WIP: strict linting + RAM-first pipeline (lint fixes pending) 2026-04-01 23:22:44 +02:00
Martin Vogel f26acd4ced Eliminate NOLINTNEXTLINE suppressions + iterative AST walkers
Remove ~480 dead per-line NOLINTNEXTLINE comments. Update .clang-tidy with
documented upstream bug references for disabled checks.

Convert key AST walkers to iterative with explicit stacks: walk_defs (with
context-aware stack frames for class nesting), walk_throws, walk_readwrites,
walk_calls, walk_usages, walk_env_accesses, extract_elixir_call, glob_match,
cbm_yaml_free, rmdir_recursive, resolve_func_name, walk_variables_iter,
count_branching. Reversed child push for left-to-right traversal.

Disable readability-implicit-bool-conversion globally — designed for C++,
flags all idiomatic C patterns (bool = x && y) with no config to allow them.

Fix clang-format, implicit-bool edge cases, add test_helpers.h.
2026-03-31 20:04:11 +02:00
Martin Vogel cd1417427c Add 8-layer security test suite + hardening
Code-level defenses:
- cbm_validate_shell_arg(): reject shell metacharacters before popen/system
- SQLite authorizer: block ATTACH/DETACH at engine level
- CORS localhost-only origin reflection (replaces wildcard *)
- Path containment: realpath() check in get_code_snippet
- process-kill restricted to server-spawned PIDs
- SHA256 checksum verification in update command

Security audit scripts (8 layers):
- L1: Static allow-list for dangerous calls + URLs
- L2: Binary string audit (URLs, payloads, credentials)
- L3: Network egress monitoring via strace (Linux)
- L4: Install output path + content validation
- L5: Smoke test hardening (clean shutdown, residual procs)
- L6: Graph UI audit (external domains, CORS, binding)
- L7: MCP robustness (23 adversarial JSON-RPC payloads)
- L8: Vendored integrity (checksums + dangerous call scan)

CI: parallel security-static job (no build needed), binary
layers in smoke jobs per-platform. Cleanup of test fixture
dirs in clean.sh + .gitignore.
2026-03-20 18:19:15 +01:00
Martin Vogel 85fc74fc31 Full Windows portability: proper implementations, not stubs
- handle_processes: GetProcessMemoryInfo + GetProcessTimes (Windows),
  getrusage (POSIX) — real process metrics on both platforms
- handle_process_kill: TerminateProcess via OpenProcess (Windows),
  kill(SIGTERM) (POSIX)
- index_thread_fn: CreateProcess + WaitForSingleObject (Windows),
  fork+exec+waitpid (POSIX)
- cbm_clock_gettime: QueryPerformanceCounter (Windows) — replaces
  all bare clock_gettime calls across 6 files
- cbm_nanosleep, cbm_strcasestr, cbm_mkdir: compat shims
- CBM_TLS: _Thread_local (C11 standard, works on all compilers)
- Replace bare strndup with cbm_strndup, guard POSIX headers
- Add mingw-w64 cross-compile to Docker test infrastructure
2026-03-18 18:47:43 +01:00
Martin Vogel 18fa9979ee Replace vmem with mimalloc global allocator, add extraction-phase prescan, fix __init__.py QN collision
Memory management:
- Vendor mimalloc v2.1.9 as global allocator (MI_OVERRIDE=1 in prod)
- New mem.h/mem.c: RSS-based budget tracking via mi_process_info()
- Remove vmem.c/vmem.h (mmap-based budget tracking)
- Remove slab tier2 bump allocator (~300 LOC); >64B goes to mimalloc
- Slab tier1 pages from malloc (= mimalloc) instead of vmem
- Arena blocks from malloc instead of vmem
- Budget raised from 35% to 50% RAM (no more untracked C++ heap)

Extraction-phase prescan (eliminates disk re-reads):
- HTTP call sites: keyword check + URL extraction during extraction
- HTTP routes: decorator + source-based extraction during extraction
- Config file refs: regex scan during extraction
- httplinks: 41.8s → 13ms on Linux kernel (3,212x faster)
- configlink: 41.4s → 0.8s on Linux kernel (54x faster)
- Linux kernel fast-mode total: 2m38s → 1m18s

Bug fixes:
- __init__.py Module QN no longer collides with Folder QN
- index.ts same fix for JS/TS packages
- 13 regression tests for QN collision at FQN + extraction layers
- search_graph/search_code default limit raised from 10 to 500k
- Resolve all clang-tidy, cppcheck, and clang-format warnings

Repo cleanup:
- tree-sitter-form, tree-sitter-magma moved to tools/
- .gitignore: build/, node_modules/, graph-ui/dist/, TEST_PLAN.md
2026-03-17 22:05:57 +01:00
Martin Vogel 0d216c7936 Resolve all clang-tidy, cppcheck, and clang-format warnings across 94 files
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
2026-03-15 22:37:34 +01:00
Martin Vogel 94a9a92f83 Add C-native pipeline with blocking clang-tidy, cppcheck, clang-format linting
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.

Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
2026-03-15 19:26:12 +01:00