Normalize Windows cache paths when reporting runtime settings and resolve
Git executables before spawning history queries. Keep executable lookup
results per thread and cover the ownership contract deterministically.
Support Git 2.43 while denying history-query network transports, and close
the activity database on daemon startup failure so Windows can clean up.
Validation: 463 native macOS tests, 421 Windows tests, 44 MSan UI tests,
and focused canonical lint passed. Production-only revert checks confirm
the repaired failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Merge validated environment overrides into a child-owned environment on POSIX and Windows, preserving caller PATH lookup for executable names. Forward each ObjectScript export parse timeout to the extractor.
Cover inherited and replaced values, empty values, duplicate overrides, parent isolation, invalid names, executable lookup, and timeout forwarding.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Rename the reading workspace to CBM Lens Beta, distinguish Beta with an amber badge, and keep build diagnostics out of the header. Put architecture history beside the active context and record meaningful exploration steps.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Follow-up to #2589.
No timeout decides a cli_scope test any more:
- The status probe and the parent's client connect repeat an attempt that
reaches a live but mute endpoint (a loaded runner answering slowly). A
reply or a definite refusal decides. Each attempt keeps the API's finite
5 s budget. This replaces the 15 s status budget and the single 5 s
connect.
- The host child is reaped without a deadline (it used to get SIGKILL after
60 s).
- The host child joins its version cohort with UINT64_MAX, which waits for
locks but answers a conflicting holder at once (it used to have a 45 s
deadline).
A wedged child is left to the harness's per-suite ceiling.
cli_scope_fixture_finish restored HOME, CBM_CACHE_DIR and SHELL even when
start had failed before saving them, which unset them for every later test
in the suite. It now restores only what start saved.
New tests:
- cli_scope_fixture_finish_keeps_env_after_an_early_failure fails on main
at ASSERT(home_kept);
- cli_scope_retries_a_mute_endpoint_until_it_answers checks the retry rule
on scripted attempt outcomes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cli_install_skip_binary_unchanged_in_host_namespace_quiesces_nothing
failed once on 2026-10-09 at ASSERT(ready), with nothing else in the log.
cli_scope_fixture_start returned false silently from any of its setup
steps, and a 90 s deadline decided the wait for the host child's
readiness.
The fixture now records and prints the step that failed: the fixture
directory, the host directories, the runner's own image fingerprint, the
pipes, the fork, the host's start, or the client connect, with the OS error
where one applies. The host child prints which of its own steps failed
before it answers 'E'.
The readiness wait has no deadline any more, because no timeout may decide
a test. The child answers 'R' or 'E', or the pipe reaches EOF when it dies.
A child that wedges is left to the harness's per-suite ceiling.
New test: cli_scope_fixture_names_its_failing_setup_step.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On macOS the signed-candidate preparation folded every step into one
boolean, so each failure printed the same line: "signed update candidate
preparation failed". A full /tmp read exactly like a broken codesign. That
is what a cli test hit on 2026-10-09, while a multi-GB index was filling a
nearly full disk.
Each step now records why it failed, and the update prints, for example:
error: signed update candidate preparation failed while writing the
unsigned candidate: No space left on device
The staging functions in activation_transaction.c keep the errno of the
failing write (or sync, close, read) through their cleanup, so a caller can
name it. The non-macOS path adds the OS error to its existing "failed to
stage verified update" line (POSIX only; Windows reports through
GetLastError). A failed fingerprint of the staged file now gets its own
message instead of "...: activation transaction completed".
A test seam makes the next staging write fail with a chosen errno. The new
test cli_update_names_the_failing_staging_step_and_os_error fails without
the cli.c change and passes with it, and checks that the old executable
stays in place. Also checked on real input: with /tmp on a small volume
that fills between two updates, the update prints the line above.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
rst_body cut a reST section's text at 500 bytes by setting a stop value
past the end, then backing off from out[RST_BODY_MAX] while that byte
looked like a UTF-8 continuation byte. That byte was never written, so
where the text ended (499 or 500 bytes) depended on what the arena memory
held, and when a multibyte character did not fit, bytes never written
could end up in the text.
Only whole characters are written now, and the text ends at the last one
written. The same django tree indexed twice gave doc_link_candidates rows
that differed in about 250 of 11,300 (through the section text's TF-IDF);
with this change two runs are identical, and so are the 6,594 sections.
The new test pins the cut at the boundary: a word that fills the 500 bytes,
a two-byte character that would cross them, a word cut at a character.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The doc-section candidate pass (#2574) made a Linux kernel index 2.8x
slower: 207 s before #2574, 580 s after, on the same machine. Every
candidate of every section got a dot product that merged the section's up
to 1024 terms with the function's, and then every candidate was sorted for
a top 10.
- The section's vector is held densely per worker, indexed by term and
stamped per section, so a candidate's dot product walks only the
function's (or file's) own terms. The same products are added in the
same ascending term order, so every score is the same float.
- Only the candidates the results read are sorted: the top
SEM_DOC_TOP_K (and the local ones down to CBM_SEM_DOC_MIN_SCORE), the
top CBM_SEM_DOC_FILE_K files. cmp_doc_cand is a total order, so the
sorted prefix is the full sort's prefix.
Kernel (perf-bench/linux, M5 Pro, same plain build): 579.7 s -> 391.5 s,
nodes and edges unchanged. Exactness, measured directly: a build that also
ran the old merge and the old full sort found 0 score, 0 shared-term and
0 order differences over 5,003 django sections.
Candidate generation itself (a section's key terms, df <= funcs/20) is
unchanged here: narrowing it would change the stored rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
81264814 built the production allocator as mimalloc's release build
(-DMI_DEBUG=0). That bought a large speed-up, but it raised what macOS
charges an index worker, its phys_footprint:
| peak phys_footprint | v0.11.0 | main |
|---|---:|---:|
| TypeScript | 1.25 GB | 1.94 GB |
| dotnet/runtime | 4.5 GB | 12.7 GB |
The 2026-10-09 bench missed most of this, because it sampled RSS.
The difference is one compile-time branch in mimalloc's unix decommit.
With MI_DEBUG or MI_SECURE>2, a purged range is mprotect(PROT_NONE)'d
and flagged needs_recommit. A release build leaves it "committed".
When the arena hands such a range out again, it is re-accounted as a
whole with MADV_FREE_REUSE (arena.c, _mi_os_reuse). macOS then charges
every page of it to the footprint, touched or not.
On macOS, take the recommit path in every build. Linux and Windows keep
upstream's behaviour; there MADV_DONTNEED already drops RSS at once.
Measured on Apple arm64 (macOS 26.6), main ec4fbd3a vs this change, cold
full indexes interleaved with main (C#: 2 runs each, in the same session
as main):
| corpus | footprint before | footprint after | change | wall time |
|---|---:|---:|---:|---|
| TypeScript | 1.98 GB | 1.09 GB | -45 % | unchanged |
| kubernetes | 2.92 GB | 1.88 GB | -35 % | unchanged |
| dotnet/runtime | 12.6 GB | 4.9 GB | -61 % | unchanged |
| linux kernel | 16.0 GB | 13.5 GB | -16 % | 574 -> 559 s |
The graph is byte-identical: TypeScript, kubernetes and django give
the same nodes, edges and CALLS-set hash.
dotnet/runtime is now close to v0.11.0's 4.5 GB.
Measured but not taken:
- the whole debug build, which costs ~5 % wall time on C#;
- MI_STAT=2, MI_PADDING=1 or MI_ENCODE_FREELIST=1 alone, none of which
moves the footprint;
- MIMALLOC_PURGE_DELAY=0, which is already set.
Vendored file changed in place (precedent 1f674c80); its hash in
scripts/vendored-checksums.txt is updated and security-vendored.sh passes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1a342553 made suffix_match ties deterministic by preferring the least
nested QN, then the smaller QN. Determinism was the point, and it stays.
But depth says nothing about which same-named definition a call means.
An audit of main against v0.11.0 (2026-10-10) traced three resolution
regressions to that one rule:
- django: about 5,600 `Model.objects.X(...)` calls tie between
`QuerySet.X` and same-named methods elsewhere. `Library.filter`,
`EngineHandler.all` and `HttpResponseBase.get` are each one segment
shallower, so the QuerySet share fell from 57 % to 24 %.
- Kotlin / Java: a call went to a property or field that merely shares
the callee's name. Kotlin `where(...)` went to `var where`, and Java
`esr.indexMode()` to a field whose QN lacks its class segment.
Kotlin CALLS edges on Variable/Field targets went 785 -> 1,648.
- C#: .NET reference assemblies (`src/libraries/<Asm>/ref/<Asm>.cs`,
`throw null` bodies) are shallower than the implementation, and "ref"
sorts before "src". 14.1 % of dotnet/runtime's CALLS landed on such
stubs (4.9 % in v0.11.0).
A tie now goes to:
1. the more plainly callable candidate: Function/Method, then any other
definition, then an unknown label, then Variable/Field;
2. for C# only, an implementation over a reference-assembly stub;
3. the lexicographically smaller QN.
This is still a pure function of the candidate set (O9). The kernel
cases behind 1a342553 keep their answers: `dev_name` is the only Function
among the fields, and include/linux sorts before tools/virtio.
Measured on the bench corpora (macOS arm64, main cb8b641f vs this
change, one cold full index each):
- django: `objects.X` -> QuerySet 23.7 % -> 41.4 %; CALLS -> Variable
1,022 -> 741
- Kotlin (Exposed): CALLS -> Variable 1,648 -> 480
- Java (elasticsearch): CALLS -> Variable/Field 62,313 -> 42,957
- C# (dotnet/runtime): CALLS -> ref/ stubs 143,799 -> 52,000;
CALLS -> Variable/Field 33,779 -> 28,662
- TypeScript: CALLS -> Variable 7,978 -> 7,341
The rest of the django gap needs Manager -> QuerySet typing, which no
tie order can supply.
Tests:
- registry_tie_prefers_callable_over_property
- registry_tie_ignores_nesting_depth
- registry_tie_prefers_csharp_implementation_over_ref_stub
All three are RED on main and RED again with the registry change
reverted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
CI's cppcheck 2.20 failed the lint job on rr_c_role: after the last dot was
cut to '\0', a second strrchr looked for the owner's dot, and cppcheck took
that result for the first one ("Condition 'prev' is always true",
knownConditionTrueFalse). The owner is now the segment found by walking back
from the last dot to the dot before it or to the start of the name: the same
owner, without a second search (Homebrew's 2.21 did not report it).
rst_c_member_owner covers the member path, which no test reached: the owner
running back to the start (point.x), to the dot before it (shapes.box.x),
through "struct" and -> (struct point->y), and a field taken for a struct
(box.corner.y, missing). It passes on the old and the new code alike, and
fails when the walk ignores the dot before the owner.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every process start fingerprints its own executable for the daemon build
identity, and the hook path pays for that before it can answer. With the
portable transform that was ~1.2 s of CPU for the ~300 MB binary, so on a
slower machine hook-augment hit its 2000 ms deadline and printed nothing
(#2441).
Two changes in src/foundation/sha256.c:
- cbm_sha256_update compresses whole blocks straight from the input instead
of copying every byte through the context buffer.
- Block compression runs on the CPU's SHA-256 instructions when it has them:
ARMv8 SHA2 (sysctl on macOS, HWCAP on Linux, IsProcessorFeaturePresent on
Windows) or x86 SHA-NI (CPUID). The backend is probed once; the portable
transform stays the fallback.
Measured on an Apple arm64 Mac (macOS 26.6), release builds of main 72a2c0bc
and of this change, hook-augment SessionStart in fresh isolated dirs, 9
interleaved runs, medians:
- hook-augment: wall 1.448 s -> 0.376 s, user CPU 1.162 s -> 0.097 s
- SHA-256 of the 303 MB binary: 1.18 s -> 0.095 s CPU per pass (the
block-direct update alone, portable rounds: 0.74 s)
- with CBM_HOOK_DEADLINE_MS=1000 (a slower machine): before, exit 0 with
empty stdout after 1.01 s; after, the full 188-byte answer in 0.37 s
Digests are unchanged: the build id the new binary logs equals the file's
SHA-256, and the hook output is byte-identical. A new test compares the
hardware and portable backends at every length 0..299 and at block-boundary
sizes up to 1 MiB under eight update chunkings, plus the NIST million-'a'
vector; it fails if the ARM schedule is broken or if Apple arm64 falls back
to portable. The x86 path was checked under qemu-x86_64 -cpu max (2,436
comparisons equal, million-'a' correct) and falls back to portable on a CPU
without SHA-NI. On a Windows arm64 VM the probe picks the hardware path:
0.094 s against 0.82 s portable for the 303 MB binary, digest equal to
sha256sum's.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
index_status now ends with a watch object (#2167) whose hint names index_repository. The checks searched from doc_links: to the end of the text, so the watch hint failed the no-hint check and could have satisfied the hint and index_repository checks of the error cases. dm_doc_links_block cuts the text at the next top-level key.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolves the definition walks next to each other: main's decl_namespace (#2120) and this branch's extra_props are both compacted and both counted in the result census.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolves the index_status tool description: main's watch state (#2167) and this branch's doc_links both stay.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Doc -> code candidates now carry a p judged for their kind, their document
format and where the document sits.
Format (sample 6, 234 blind-judged candidates): reStructuredText, AsciiDoc and
PDF sections each get their own curve (reST stored from tfidf 0.30, AsciiDoc
from 0.20, PDF not measured and not stored). A candidate that repeats an
exact MENTIONS link of the same section is not stored.
Position (sample 7, 490 blind-judged candidates from four monorepos): every
document gets a home folder -- its folder; a docs folder documents its
parent; a folder without code hands over to the nearest one with code; the
repository root means the whole project. Markdown candidates outside the
document's home were right 6 of 90 and are never stored; inside it they are
stored from tfidf 0.30 (0.33 / 0.53); for documents of the whole project
samples 4 and 7 pooled give 0.22 / 0.51 from 0.30.
New kinds, all in doc_link_candidates (evidence gains "kind" and
"position"; no new edge type, no schema change):
- local: up to two home functions ranked below the top five (0.32);
- file: a whole file, its vector the sum of its functions' terms and its
definitions' names, config keys included, cut to 64 terms (0.30, from
tfidf 0.40);
- folder: a README or index document -> its home folder (0.87).
get_code_snippet on a section lists every kind with its label;
get_file_outline shows the sections about the file or a folder holding it
(cbm_store_doc_candidates_for_path).
cbm_sem_p_2dp rounded 0.325 and 0.525 down (their float values sit 1.2e-6
below the half); its epsilon is 1e-4 on the x100 scale.
Replayed on the sample repositories, the build stores exactly the rows the
judged curves select (950 rows, 0 differences).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
pipeline_doc_candidates_published_and_kept_by_delta appended its
delta-route edit to "user_config.go" while the fixture writes
"User_config.go". On a case-insensitive filesystem (macOS, Windows)
that is the same file; on Linux it is a new file, closure repair
declines new files (correctly), the run falls back to a full rebuild,
the route assertion fails and the early return leaks the pipeline
(LeakSanitizer on the arm64 GCC leg). The edit now names the fixture
file exactly.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The candidate rows and their strings were allocated with calloc/strdup
in the semantic pass and in cbm_store_doc_candidates_get, and released
with free. They now go through the memory core in CBM_MEM_CLASS_STORE
(cbm_calloc, cbm_mem_strdup, cbm_free) on both sides, so the rows the
pass hands to the pipeline and the rows a read returns are accounted
like every other store buffer. make lint-memory-core: no file grew.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A documentation section that explains a function rarely names it, so
the deterministic doc links cannot connect them. The semantic pass now
ranks, for every Markdown section that can be about code, the functions
whose TF-IDF terms best match the section's heading and body, over the
functions' own corpus (function-pair scores are untouched):
- gate: no candidates for release-number or project-meta headings
(changelog, licence, contributing, links, contents, install, ...),
bodies under 20 words, or bodies that are mostly links;
- targets: functions and methods, never tests (by path or name);
- stored: a section's best five with tfidf >= 0.20, each with p, the
probability that the section is about the function, from 274
blind-judged candidates in five held-out repositories (three reading
rounds): 0.36 below 0.30, 0.42 below 0.40, else 0.55. Below 0.20 the
judged share was 0.06, so nothing is stored there.
These are candidates for an agent to verify, not graph edges: at best
about half are right, far below the bar for a link. They live in a side
table, doc_link_candidates (section and function qualified names, rank,
score, p, evidence), created when a generation is published, so older
indexes read as "no candidates". Full and legacy-partial reindexes
replace a project's rows; the delta route keeps the previous rows (its
proxy buffer has no section text or function tokens to recompute
them), and a row whose node is gone is skipped when read.
get_code_snippet shows them: on a Section, "possibly_about" (its
candidate functions); on a function or method, "possibly_described_in"
(the sections), at most five, highest p then score first, with a
candidates_note saying what they are.
Measured on fastapi (6,144 gated sections): 24,293 rows, identical to
the generator's dump; index time within noise; the database grows by
the table (37 MB there, with a 148-character project prefix on every
qualified name).
Tests: semantic:sem_doc_calibrated_p_bands,
store_arch:doc_candidates_replace_and_get (RED when a replace keeps old
rows), pipeline:pipeline_doc_candidates_published_and_kept_by_delta
(RED with test functions admitted, the heading gate off, the publish
write skipped, or the delta route replacing rows),
mcp:snippet_doc_candidates_both_directions (RED with stale rows shown).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The semantic pass read every string property of a function (body text,
signature, return type, decorators, docstring) with a reader that
stopped at the first '"' and kept escapes raw: a value holding an
escaped quote lost everything after it, and "\nreturn" became the token
"nreturn". cbm_sem_json_str honours the escapes: \" \\ \/ keep their
character, \n \t \r \b \f and \uXXXX become a separator.
The tokens change, so admissions change: on 13 repositories 1,340 ->
1,387 admitted SEMANTICALLY_RELATED pairs, 614 of them unchanged. Of
the dropped pairs that earlier samples had judged, 52 of 89 were
related (0.58); of the kept ones, 73 of 92 (0.79). A pre-registered
blind sample of 100 newly admitted pairs from 11 held-out repositories
(three reading rounds) judged 65 related (0.65, Wilson 0.55-0.74), so
the admitted set is 0.72 precise against 0.69 before. The same index
run twice gives identical pairs and scores.
"p" is refit on the new admitted population (bands at the quartiles of
the admitted scores, pooled where a higher band judged lower; added and
kept pairs weighted by their share): score < 0.773: 0.63, else 0.76
(was 0.56 / 0.64 / 0.73 / 0.86 in four bands).
Tests: sem_json_str_honours_escapes (RED with escapes ignored),
sem_calibrated_p_bands.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
When a function has more candidate partners than its budget of ten
SEMANTICALLY_RELATED edges, admission took the pairs in discovery order:
the first ten won, whatever their scores. Admission is now best-first
(cbm_sem_admit_best_first: descending score, ties in the canonical
order, still a pure function of the sorted inputs). On a blind judged
sample from held-out repositories, the pairs first-come admitted over
budget were 0 of 17 related; the pairs best-first admits instead, 7 of
20. Where no budget binds nothing changes (django 66 edges identical;
fastapi 426 -> 430 edges, 222 changed).
Every SEMANTICALLY_RELATED edge now carries "p": the probability that a
reader judges the pair related, by score band, from 120 blind-judged
admitted pairs (score < 0.778: 0.56, < 0.806: 0.64, < 0.852: 0.73,
else 0.86). p is computed from the score as stored (three decimals),
so one shown score always has one p. Scores and the 0.75 threshold are
unchanged; a scoring change (TF-IDF alone, or weights fitted on the
first sample) was judged on the same sample and did not beat the
shipped score by the pre-registered margin.
Tests: sem_admit_best_first_keeps_the_best, sem_calibrated_p_bands,
pipeline_semantic_edges_carry_p; each is RED with its behavior
reverted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The TF-IDF signal keyed each term by the token's POSITION in the
function, so any two functions of similar length shared their first
terms whatever their words: 0.20 of every SEMANTICALLY_RELATED score
measured length, not vocabulary. Terms are now keyed by corpus token
(cbm_sem_tfidf_terms: one term per distinct word, weight term
frequency x idf, ascending token index, which the sparse cosine
merges on).
Measured on a blind judged sample: 222 function pairs from six
held-out repositories, stratified by score band, read in three rounds.
The share of admitted SEMANTICALLY_RELATED edges a reader judges
related rises from 0.713 to 0.742, with about 19% fewer edges (fastapi
507 -> 426, django 64 -> 66). The same sample shows that no score
threshold reaches 0.90 precision, so the documented "~95% at 0.75" did
not hold before this change either.
The harness that measured it ships with the fix, off by default:
CBM_SEM_PAIR_SIGNALS=<path> writes one line per scored pair (both
qualified names, the score, every signal value, whether the pair was
admitted, both locations); CBM_SEM_PAIR_SIGNALS_FLOOR also records
pairs below the threshold, never admitting them. cbm_sem_combined_score
is now cbm_sem_signal_values + cbm_sem_combine (same arithmetic, same
order). CBM_SEMANTIC_INDEX_VERSION 3 -> 4: indexes built before score
pairs differently.
Tests: sem_tfidf_terms_compare_vocabulary_not_positions,
sem_combine_weighs_signals,
pipeline_semantic_pair_signals_never_change_the_graph; each is RED with
its behavior reverted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel