fetch_models.py sorted versions with float(), and Firefox Remote Settings
now publishes Nightly pre-releases ("1.0a1") alongside each release. The
first one crashed the whole fetch, so adding Swedish, or 21 of the 34 named
languages, silently did nothing and the button never appeared.
Only take records desktop Firefox would use on the release channel, pick
the newest complete version per direction instead of mixing files across
versions, support pairs that publish a separate source and target vocab
(Japanese), and keep one failing language from stopping the rest.
That opens 50 languages, up from the ~40 the docs claimed, including
Arabic, Japanese, Korean and Thai, which the docs said were unavailable.
All 50 downloaded, loaded and translated on the bergamot 0.4.5 engine in
the shipped image (NOMAD3). Chinese stays out: it is published as
zh-Hans/zh-Hant, which the proxy's two-letter codes cannot carry yet.
Translated pages now send Cache-Control: no-cache and put the installed
language set in the ETag. Kiwix's max-age=3600 kept the old language bar
for up to an hour after a language was added, and a normal refresh does
not reach the article frame.
Docs: step-by-step instructions and the full code table in the Supply
Depot section; sizes corrected to 45-140 MB per language.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
urlopen() follows redirects silently. Kiwix answers a book root such as
/content/<book> with a 302 to the ZIM's main page, so the proxy served
that page's HTML at the original, shallower URL. Every relative link and
stylesheet then resolved one directory too high: the page lost its
styling and every article link 404'd (e.g. atenolol/ -> /content/atenolol/).
Kiwix's /random is a 302 too and broke the same way.
Use an opener that returns 3xx to the browser like any other non-200
response, and keep an absolute Location on this origin rather than the
upstream container name.
Bumps the catalog image to 0.1.1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The model declares `'zim' | 'map' | 'dataset'` and IngestDrugDataJob writes a
'dataset' row for the openFDA drug labels, but the column was still
enum('zim','map') from the table's original migration. Every write failed:
Data truncated for column 'resource_type' at row 1
insert into `installed_resources` (... `resource_type`, ...) values
(..., 'openfda-drug-labels', 'dataset', ...)
The ingest itself succeeded — 261,671 rows landed in drug_labels — but the
install-state write-back at the end of the job was lost every time. Since
getCategoriesWithStatus() resolves a tier only when every resource id is
accounted for, and it already queries whereIn('resource_type', ['zim',
'dataset']), the missing row kept the Medicine category from ever resolving
its Standard or Comprehensive tier. Only the enum was left behind.
Verified against a live install: after the migration the previously failing
insert succeeds, and Medicine resolves installedTierSlug
'medicine-comprehensive'.
down() drops 'dataset' rows before narrowing the column, since MySQL would
otherwise truncate them to '' or fail under STRICT_TRANS_TABLES.
ollama/ollama:0.24.0 never cancels in-flight generation when a client
disconnects: the runner process is left spinning at 300%+ CPU with no
active requests and never recovers. NOMAD triggers this organically via
its own 60s RAG embedding timeouts, so slow CPU-only boxes can wedge
themselves after a single slow request. Reported with a standalone
container repro in #1321; the cancellation fix landed upstream between
0.24 and 0.33.
Regression-tested on 0.33.3 (RTX 5060 box):
- model listing via /api/tags (NOMAD backend detection intact)
- GPU acceleration (100% GPU placement, 82.8 tok/s on llama3.1:8b)
- model pull through the NOMAD download pipeline (NDJSON progress)
- embeddings via /api/embed with nomic-embed-text (768-dim)
- streaming chat end-to-end through the Command Center UI
- the disconnect hang itself: container idles at 0.06% CPU 20s after
a mid-generation client drop
The AMD path is unaffected: it always overrides to ollama/ollama:rocm.
The seeder does not sync container_image on existing rows, so this pin
reaches fresh installs; existing installs pick it up via the opt-in app
auto-update or a manual update.
Fixes#1321
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(translate): offline translation for the Information Library
Implements #1272. Ships as a Supply Depot app so users who do not need
translation never see it.
A path-preserving reverse proxy in front of kiwix-serve. /viewer, /skin/*,
/search, /random and /catalog/* are forwarded byte for byte; only text/html
under /content/ is rewritten. Preserving the path is the whole design:
viewer.js keeps the content iframe in sync with location.hash by comparing
contentWindow.location.pathname, so moving the path starts the ping-pong its
own comment warns about and forces us to patch Kiwix's JavaScript on every
Kiwix bump. Because the path never moves, no Kiwix JS is modified, and Kiwix's
header, search, library and random keep working because they are Kiwix.
Language is a cookie, so links inside the ZIM need no rewriting either.
Bergamot rather than the AI Assistant: measured on the same box and the same
passage, 12,818 words/s on CPU against llama3.1:8b at 8 words/s on an RTX 5060.
It is also more careful with names, where the LLM translated the product name
and a smaller model hallucinated a clause. No GPU, so it works on CPU-only
boxes where the LLM path is unusable.
Models come from Firefox Remote Settings, the endpoint Firefox itself uses,
not the mozilla/firefox-translations-models repo, which was archived on
2026-08-21. MPL-2.0, about 37 MB per direction.
Failure handling is deliberate throughout: a failed model fetch is non-fatal, a
translation error returns the original article with a comment rather than an
error page, and a box with no models still starts and still forwards the
library. Translation being unavailable must never make the Information Library
unreachable.
amd64 only, and pinned as such in the build workflow. Bergamot's intgemm
backend is x86 specific and there is no aarch64 wheel, so a multi-arch build
would produce an ARM image that pulls cleanly and dies at start.
Also adds .gitattributes pinning *.sh to LF. The repo stores LF but Windows
checks out CRLF, and a CRLF after a shebang makes the interpreter
unresolvable inside the image. CI builds on Linux so this never bit, but it
breaks a local image build and would have bitten the existing sidecar script
too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
* fix(translate): vary the cache on the language cookie
Found by clicking the language buttons in a browser rather than trusting curl.
Kiwix sends ETag and Cache-Control: max-age=3600, and the proxy forwarded both
unchanged on a response whose content now varies by cookie. The browser cached
the English article for an hour and served it straight back when the reader
picked a language, so the feature looked like it simply did not work. A shared
cache would have been worse, serving one reader's language to another.
Translatable responses now send Vary: Cookie, and their ETag is namespaced by
language so revalidation still works per language rather than being turned off.
Forwarded paths are untouched, so /viewer, /skin/* and /catalog/* stay byte for
byte identical to Kiwix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
* fix(translate): move off port 8480, which Vaultwarden already uses
Caught by seeing the two cards side by side in Supply Depot with the same port.
My port survey grepped ui_location for a bare number, and Vaultwarden's is
'https:8480' because it needs an https Open link, so it did not match and 8480
looked free. Moved to 8460, which fits the existing spacing between Meshtastic
Web on 8450 and Homebox on 8470.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
* fix(translate): pin the base image to python 3.10
The build fails outright on 3.12: bergamot publishes no wheel for it and pip
resolves no further than 0.1.2. The spike used 3.10 for this reason and I had
modernised it without checking. Caught by building the image rather than
reading it.
Moving off this pin means moving off the 2022 PyPI wheel to a build from
mozilla/translations, which is the follow-up already noted in the Dockerfile.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
* fix(translate): translate text that lives in a bare div
Sotoki-generated StackExchange ZIMs put every question excerpt in a
<div class="...excerpt">, so on a questions list the titles translated and the
descriptions underneath them did not. Reported against the ham radio ZIM.
div could not simply be added to BLOCK_TAGS: the outermost-block rule would
claim a top-level layout wrapper and send the whole page as one job. It is now
claimed speculatively and abandoned the moment anything block-level opens
inside it, so a wrapper yields to the blocks it contains and only a genuine
text leaf is translated.
Also moves the test harness's reporting block back to the end of the file. New
checks had been appended after it, so they executed but their failures were
never reported: verified that removing div from the set now fails 3 checks,
where before it reported success.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
* fix(translate): decode numeric character references before translating
Bergamot's HTML mode decodes named entities (`&` arrives as "&") but not
numeric ones. It treats `'` as literal text and escapes the ampersand on
the way out, so "If I'm in the wilderness" came back as
"Si I&#x27;m dans la nature" and rendered the escape sequence visibly to
the user. Confirmed by round-tripping fragments through the model directly.
Numeric refs are now decoded before the round trip. `&`, `<` and `>` are
deliberately left encoded: decoding those would inject real markup into the
fragment and fail the balance check that gates HTML mode, dropping the block to
plain text and losing its inline links.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(kb): add per-file active/inactive toggle for search inclusion
Implements the per-file toggle agreed on in #1119, in place of disabling
ZIM auto-discovery by default. Any indexed file (ZIM, upload, or other)
can now be excluded from RAG search results instantly, without deleting
or re-embedding its vectors.
- Vectors stay in Qdrant permanently; each point's payload carries an
active boolean. Toggling flips this payload in place, both directions,
with no re-embedding cost.
- Search filter is a denylist (must_not: active == false), not an
allowlist, so a point missing the field entirely is still found by
default -- search correctness never depends on backfill timing.
- embedAndStoreText() looks up a file's existing active state before
writing new points, so retries, force re-embeds, and replaced-content
reindexing never silently reactivate a file the user explicitly
toggled off.
- Applies uniformly to all non-admin_docs buckets (ZIM, uploads, other),
matching how the rest of this pipeline already treats buckets, rather
than special-casing ZIM.
New POST /api/rag/files/active endpoint and KB panel toggle switch,
following the existing updateFileCollection pattern.
Real vector deletion for genuinely deleted/replaced files remains
#1170's separate concern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* feat(kb): group personal KB uploads by collection with bulk toggle
Groups personal KB uploads into collapsible per-collection header
rows (with an "Uncategorized" bucket for files with no collection) to
reduce clutter in the file table. Each collection header also exposes
a bulk active/inactive toggle to flip every file in that collection at
once.
Also fixes three pre-existing bugs found in the collection-assignment
dropdown while building this: the popover was clipped by table cell
truncation, clipped again by the table wrapper's overflow, and the
combobox query was stale-filtering results on open.
Refs #1193
* fix(kb): widen the modal for the Active column and coerce active to a boolean
Both found by browser-verifying on NOMAD3.
The Active column adds ~57px, which put the table back over the modal width and
reintroduced the exact defect #1198 fixed: the Delete button falls off the right
edge at every viewport. Measured on the box at 1018px of table against 961px of
usable width at max-w-5xl. Widened to 6xl.
MySQL returns tinyint(1) as 0/1 rather than a boolean, and the raw 1 reached the
DOM as aria-checked="1". That is not a valid ARIA value, so a screen reader is
told nothing useful about whether the toggle is on. Coerced in the service so
every consumer sees a real boolean, with a !! guard in Switch as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
* fix(kb): validate the bulk collection toggle's input before sanitising
sanitizeCollectionName ran before the type check, so a form-encoded request
sending collection[] as an array reached .trim() on a non-string and produced
an uncaught TypeError, surfacing as a 500. The vision controller already
guards this the same way.
Also documents why requiring `active` to be a real boolean is load-bearing
rather than mere input hygiene: a cross-site HTML form can only send the string
"true", so that check is what keeps a bulk Qdrant write across every point in
the index out of reach of a form POST from another origin. Worth a comment,
because coercing it would look like a harmless usability fix.
Found in a security review of the release PRs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
---------
Co-authored-by: John Cortright <jcortright@zscaler.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The catalog-sync loop overwrote container_config and ui_location for every
non-user-modified curated service, resetting the published host port of an
install that had been moved off the catalog default. The row then disagreed
with the running container: the Open link pointed at a dead port and the
next recreate collided.
Preserve the live host ports for installed rows and apply every other
catalog change, so these installs keep receiving catalog updates instead of
being opted out of them.
Diagnosis and the original fix are @johno10661's work in #1005.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
A link tile rendered on a flat surface colour, which read as a stark white card
against the filled green app tiles. Tiles now take one of six colours from the
brand palette, chosen when creating or editing one, and the card is tinted from
it.
Tiles stay visually distinct from managed apps regardless of colour: the dashed
border and the external-link marker carry that, not the hue, and the tint is
deliberately light so the distinction survives every option.
Class names are written out in full rather than interpolated, because Tailwind
scans source text for literal class strings and would silently drop a computed
one, leaving the tile untinted. A test asserts that.
Added as a separate migration from the one introducing is_link_tile, since that
has already run on the test box and editing it in place would leave fresh and
upgraded installs with different schemas.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
Users want things they already run to appear on the dashboard: a NAS, a router
UI, a printer, a Pi, a container they started themselves. Adds a tile they
address themselves, with a name, one URL field, an optional description and an
icon from the curated set.
Chosen over adopting containers from the Docker daemon, which is what community
PR #1239 proposed. Adoption only reaches containers on this Docker host and has
to guess: it took the lowest published host port and assumed http, which picks
the wrong port for anything publishing several (Portainer 8000 against 9443,
Jellyfin 8096 against 8920) and breaks entirely for HTTPS. A tile the user
addresses has no guessing in it, covers the whole LAN rather than one Docker
host, and has no lifecycle that can break.
One URL field rather than separate host, port and path, because people paste
URLs and it gets https and sub-paths for free. A bare "192.168.1.50:8080" gains
http automatically, which is what most LAN devices need. normalizeCustomUrl and
getServiceLink already did exactly this work for the custom app URL override, so
both are reused rather than duplicated.
Stored as a service row with no container behind it: is_link_tile marks it,
custom_url carries the destination, and the service_name is namespaced
nomad_link_* so it can never collide with a curated entry. The seeder only
iterates its own DEFAULT_SERVICES and never prunes unknown rows, so a reseed
cannot touch or remove a tile.
The tile is deliberately styled differently from a managed app, outlined and
dashed with an external-link marker rather than filled, because if it looked the
same users would expect Start, Stop and Update and file bugs when those controls
are not there. Removing a tile arms a confirm first, since it is user-entered
data with no undo.
Icons come from a fixed 36 shared by the picker and the server-side validator,
so the two cannot disagree about what is selectable. DynamicIcon renders nothing
at all for a name it does not know, which would leave a silent hole in a tile.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
Salvages @kennethbrewer3's work from #836, which had drifted onto a stale base
(GitHub showed 231 files; the real change is the map UI plus a migration).
Marker visibility (individually and Hide All), per-marker icon and custom hex
colour with contrast-adjusted icon colour, search, sorting and a tree view over
the marker list, coordinate search and fly-to with URL parameters
(?lat=&lng=&zoom=), creating a marker from typed coordinates, and markers
rendered as SVGs anchored to true coordinates rather than approximately.
Name, notes and preset colour already existed; the new columns are
custom_color, icon, icon_color and visible.
Dropped from the original as stale-base noise rather than his work: unrelated
duplicate changes to the Ollama, Docker, download, RAG and ZIM services, the
API reference and getting-started docs, README/FAQ/CONTRIBUTING, the installer
script, and a revert of AppLayout to a version predating the compact sidebar.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
Closes#1179.
Surfaces the documents an answer was drawn from as a "Sources" list under the
assistant message, with the archive date where one is known, and persists it so
reopening a past conversation still shows what backed each answer.
Offline there is no second opinion to check an answer against, so provenance is
the only signal the user has that a claim came from a real document rather than
from the model. This is the other half of retrieval being allowed to decline:
one says when nothing relevant was found, this says what was found.
Citations are built from the chunks actually injected into the prompt, not from
everything retrieval returned. A chunk dropped by the budget planner never
reached the model, so citing it would credit the answer to a document it was
not based on -- the exact failure the list exists to prevent.
The archive title and date have been written into the vector payload by the ZIM
ingestion path all along and were simply never read back, so this works on
already-embedded content with no re-embed.
Rebased onto the RagPipelineService refactor, which landed after this was
written: the citation list now comes off the pipeline trace rather than the
inline retrieval block that used to live in the controller. Entries with
neither a path nor a title are skipped rather than shown as "Unknown source" --
a row the user cannot act on makes an answer look sourced when it is not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
Three changes that together let ARM hardware appear on the leaderboard honestly.
Shipping them separately would leave ARM half-supported either way: without the
image a Pi cannot submit at all, and without the architecture field it submits
but is indistinguishable from x86.
1. PIN THE OFFICIAL MULTI-ARCH SYSBENCH IMAGE
severalnines/sysbench publishes amd64 only, so ARM hosts could not run the
System Benchmark at all — not a graceful failure, the container simply cannot
execute. Apple Silicon could only run it under Rosetta emulation, which distorts
the measurement it is taking, and that is what drove a community macOS fork to
substitute a different benchmark and submit incomparable numbers.
Swaps to ghcr.io/crosstalk-solutions/nomad-sysbench (Debian 12 + sysbench
1.0.20+ds-5, built for linux/amd64 + linux/arm64). One digest covers both
architectures; verified that pulling the pinned manifest-list digest resolves to
arm64 on a Raspberry Pi 5 and amd64 on x86, and that RepoDigests reports the
same manifest-list digest on both — so a single allowlist entry serves both.
No rescoring: 1.0.17 -> 1.0.20 measured 1.25% apart on identical hardware with
identical flags (7170.18 vs 7259.56 events/sec), inside run-to-run noise and
~0.3% on a composite. Both digests are allowlisted server-side, so the fleet can
cross over gradually.
2. REPORT THE DIGEST ACTUALLY RESOLVED
The submission previously sent SYSBENCH_DIGEST, the constant the client was
compiled with. The leaderboard validates that field, but a constant attests to
how a client was BUILT rather than what it RAN, so any build inherits a valid
value simply by carrying the same source.
Now reads it back from the image. Uses RepoDigests (the manifest digest we
pulled by), never Id — Id is the config digest, differs per architecture, and
would never match the allowlist. Falls back to the constant if inspection yields
nothing usable, so a benchmark never fails over provenance metadata.
Still forgeable, and always will be with an open-source client. It moves the bar
from "no effort" to "deliberate", which is the distinction that matters when
judging whether a submission is a mistake or a choice.
3. RECORD CPU ARCHITECTURE AND OS
The leaderboard is a single board across instruction sets by design, with
disclosure as the fairness mechanism. Without an architecture field an ARM result
sits unlabelled beside x86 — exactly what the disclosure exists to prevent.
All three fields come from the Docker daemon, reusing the docker.info() call
_detectRunEnvironment already makes. That is deliberate: inside the admin
container os.arch() and si.osInfo() describe the CONTAINER, not the host being
benchmarked.
cpu_architecture Architecture x86_64 -> amd64, aarch64 -> arm64
os_version OSVersion '24.04' (already structured, no parsing)
os_name OperatingSystem 'Ubuntu 24.04.4 LTS' minus the version
run_environment is kept rather than replaced: "which distro" and "is this
virtualised" are different questions, and WSL2 is a real performance factor.
String handling lives in app/utils/platform_metadata.ts with unit tests, matching
the amd_hsa_override convention, so it is testable without a Docker daemon.
Unknown architectures pass through verbatim rather than being guessed at, and
os_name falls back to the full description whenever the version is missing or
absent from it — an over-long name is harmless, a wrong one is not.
Columns are nullable and the submission fields optional, so results recorded
before this shipped remain submittable.
Closes#1156
Refs #1151
* feat(drug-reference): offline FDA drug labels, conditions, and remedies
Adds an offline medical-reference feature with three coupled layers:
- Drug Reference: full-text search over openFDA drug-label indications,
a detail page per label, and a side-by-side single-drug comparison view.
A two-phase background pipeline downloads the openFDA label parts to the
storage volume (resumable) and ingests them into the search table.
- Conditions ("When to use what"): a curated spine of first-aid situations
that maps each situation to matching OTC drugs, each linking back to its
Drug Reference detail page.
- Curated remedies: hand-authored natural and home-remedy entries drawn from
US-government public-domain sources (NCCIH, CDC, MedlinePlus, FDA), shown
with their source links and the same safety disclaimers as the rest of the
feature.
The drug-reference and conditions layers are intentionally coupled: the drug
detail page shows the situations a drug treats, and the conditions controller
reads the same drug_labels table.
Safety surfaces ship as written. The amber SafetyBanner ("informational only,
not medical advice, not an FDA endorsement, not a drug-interaction checker, in
an emergency call emergency services") renders on the condition pages and the
search page; the detail and comparison pages carry their own "not a cross-drug
interaction checker" callout; and every page carries the openFDA CC0 source
citation and no-FDA-affiliation footer.
Wiring on this branch:
- start/routes.ts: the /drug-reference and /conditions page GETs plus their
/api/* groups.
- commands/queue/work.ts: the drug-download and drug-ingest queues, both at
concurrency 1. The two drug queues get a per-queue stall override
(lockDuration 1_800_000, maxStalledCount 3) because each part is one long
stream; every other queue keeps the existing 300000 default.
- inertia/pages/home.tsx: Drug Reference and "When to use what" tiles. The
icon and display_order are a starting point, open to change.
- types/kv_store.ts: the two drugReference.* keys the pipeline reads and writes.
- package.json: yauzl and stream-json (plus their @types), used by the ingest
job to stream the label JSON out of the downloaded zips.
The app reads the conditions and remedy data from the compiled TS constants in
app/data/; the repo-root collections/*.json files are the browseable mirrors.
The natural-remedies standalone test reads collections/natural_remedies.json to
assert the two stay in sync, so that file is also a test fixture.
The four standalone tests pass (drug_interactions, drug_ingest_status,
conditions, natural_remedies). tsc reports no errors in the feature code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(drug-reference): drop a fork-internal comment reference
* fix(drug-reference): address review — defer FULLTEXT, fallback remedies prop, strip fork refs
- migration: wrap the FULLTEXT ALTER in this.defer so it runs after the
deferred createTable (was silently swallowed, index never created)
- controller: add remedies:[] to the index() error fallback (required prop)
- strip fork-internal issue/spec references from ported comments
- tsconfig: exclude tests/standalone (node --experimental-strip-types only)
- correct the varchar(768) byte-math comment; extend remedy-spine test
* fix(drug-reference): make the interaction comparison readable at five drugs
The comparison view laid its columns out on an equal-fraction CSS grid
(repeat(N, minmax(0, 1fr))), so each added drug shrank every column; at the
five-drug maximum the FDA interaction text was squeezed into unreadable slivers.
Lay the columns out with flex instead: full-width and stacked on phones, then
fixed-width columns that scroll sideways from the sm: breakpoint up, so they
never shrink below a readable width. Theme the columns with the same palette as
the rest of the page (they were on stock gray), and give the headers a fixed
min-height so columns line up when drug names wrap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(collections): route a 'dataset' tier resource to the drug pipeline
Add an optional `type` discriminator to SpecResource ('zim' | 'dataset',
absent == 'zim'), so the tier installer can carry a DB-ingested resource
alongside ZIM files. ZimService.downloadCategoryTier branches on it: a
'dataset' resource dispatches the existing FDA download+ingest pipeline
instead of RunDownloadJob, guarded against duplicate dispatch by the drug
ingest status. Every existing manifest entry has no `type` and keeps the
exact ZIM path.
Widen InstalledResource.resource_type to include 'dataset' and exclude
dataset rows from the ZIM/map catalog-update scan (datasets aren't
filename-versioned; their freshness path is separate). No dataset rows are
written yet: the InstalledResource 'dataset' row on ingest-ready, the
manifest entry, install-gating, and the downloads-aggregator integration
are follow-up commits.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): rework into an opt-in medicine-standard tier dataset
Reshapes the offline FDA drug reference from a built-in feature into curated
content installed by selecting the Medicine / Standard tier, per maintainer
direction.
- install-state: the ingest writes an installed_resources 'dataset' row on
ready (version = the openFDA export_date), threaded installer to download to
ingest; the tier-status math and the home-tile gate read it. Manual ingests
write no row, so install-state stays tied to the curated path.
- manifest: declare the dataset in the medicine-standard tier (runtime fetches
the remote manifest, so this also needs to land upstream).
- install-gating: the drug-reference home tiles render only when installed.
- uninstall: DrugReferenceService.uninstall() stops the two drug queues, deletes
the on-disk parts, truncates drug_labels (schema kept), clears the KV markers,
and drops the install row. Best-effort, logged, scoped to drug data only.
- downloads: the download phase reports the canonical {percent, downloadedBytes,
totalBytes} shape as one drug-data card in the Active Downloads aggregator with
cancel/remove; the heavy ingest stays in the IngestStatus surface with an
Indexing handoff on the card.
- auto-update: a daily DrugAutoUpdateJob compares the manifest export_date and
re-downloads when newer, gated on installed + no active job.
typecheck clean; the drug standalone suites pass. Three points are flagged in
code for the maintainer: the InstalledResource 'dataset' approach, the tier
home, and the export_date string format.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(downloads): keep the drug download card live across all parts
The Active Downloads card filtered to the deterministic jobId, but the
download's continuations run under auto-generated jobIds (only part 0 uses the
deterministic one). So the card tracked part 0 and then vanished while parts
2..N kept downloading. The queue is concurrency 1, so collapse to whichever
single part is in flight and report the deterministic jobId: one card tracks
aggregate progress through the whole download and cancel/remove still routes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): co-locate a persistent safety note with affirmative remedy guidance
Add RemedySafetyNote at the head of every natural-remedy section — the two on
Drug Reference and the one on "When to use what" — so the "informational only,
not medical advice, seek real medical care in an emergency" framing appears with
the guidance itself, not only in the page-top banner. Replaces the terse
per-section caveat with the same amber alert language as SafetyBanner.
Addresses the upstream #1040 review request that the disclaimer be unmistakable
and present wherever affirmative self-care guidance appears, not a one-time banner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(drug-reference): fold dataset freshness into the content-auto-update path
The drug dataset auto-updated on its own daily cron that ignored the
contentAutoUpdate.* master switch, so it would refresh even with content
auto-update turned off, and it didn't ride the content-update path the way
ZIMs and maps do.
Move the export_date freshness/apply orchestration onto
DrugReferenceService.attemptAutoUpdate(), add
ContentAutoUpdateService.attemptDrugDataset() gated on the same enabled +
window config, and have the hourly ContentAutoUpdateJob drive both. Retire
the standalone DrugAutoUpdateJob. The ZIM/map attempt() path is unchanged.
Addresses the upstream #1040 request to wire the openFDA export_date check
into the content updater so the dataset updates alongside ZIMs and maps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): strengthen the remedy note to consult a clinician before combining with meds
Widen the affirmative-remedy safety note from "talk to a clinician before use"
to explicitly cover using a remedy AND combining one with a medication the user
already takes — the interaction case is the higher-risk path for an off-grid
user self-treating.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): gate affirmative remedy content behind an off-by-default flag
Add drugReference.remediesEnabled (default off), independent of the tier
install. When off, the server emits no remedy data at any boundary — the
drug-reference page prop, conditions show, and the /api/conditions/drugs
situation search — and the "Natural" filter is hidden, so installing the
medicine-standard tier lights up the verbatim FDA label search and the
condition-to-OTC matching but not the hand-authored self-care and herbal
sections. No user-facing toggle: it is flipped on after a clinician content-pass.
Implements the upstream #1040 split-by-risk request: the regulated label content
ships with the tier; the authored remedy guidance stays gated until sign-off.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
Phase 4 of the NOMAD Score v2 effort (stacked on #1089). Captures the raw
channel values the leaderboard scores from, computes the uncapped v2 score
in-app (byte-matching the leaderboard's server-side recompute), submits the
v2 payload, and surfaces it in the benchmark UI.
- Capture raws: per-channel sysbench values + O_DIRECT disk (W4), single/multi
thread CPU + memory, W6 consistency companions, and run-environment metadata
(run_environment, storage_path_type, gpu_compute_detected, #1016).
- AI hardening: reference model llama3.1:8b, num_predict=256, VRAM eviction
before the run + unload own model after, pre-flight disk check (~6.5GB, only
when the model is uncached), median-of-3 run (W7).
- Score: _calculateNomadScoreV2 (uncapped, reference->1000) byte-matches the
leaderboard recompute. nomad_score_v2 stays null unless a full run with AI.
- Payload: submitToRepository sends the v2 raws + score alongside legacy v1.
- UI: v2 headline score, legacy v1 as secondary, raws/env in details.
- Migration: 13 nullable columns; double (not knex float(8,2)) so millions-scale
raws keep full precision and don't break the leaderboard byte-match.
Reference constant REFERENCE_SCORES_V2.ai_tokens_per_second is still the stale
40.5 placeholder (TODO in source); locking it to the measured 13.2 is a
follow-up gated on the AMD iGPU fix (#1074). Live prod v2 submit is separately
blocked by the leaderboard TTFT floor (tracked in the leaderboard follow-up).
Verified on NOMAD3 (RTX 5060) dev env: migration applies, raws recorded,
v2 score computed, backend + benchmark.tsx typecheck clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(rag): add subject/collection organization to knowledge base
- Add nullable collection field to KbIngestState, propagated through the
embed job, RAG service, and Qdrant point payloads (indexed for filtering)
- Add upload-time category selection and per-file collection reassignment
in the Knowledge Base modal, with a filterable Stored Files table
- Add a 'Search in' collection filter to the chat interface, threaded
through to searchSimilarDocuments as an optional Qdrant filter
- Fix .docx extraction: previously routed through raw-text extraction
(garbage output for a ZIP-based XML format); adds a proper mammoth-based
extractor and a dedicated 'docx' file-type case
* feat(rag): support dynamic KB collection creation, rename, and removal
Extends collection organization with a Manage Collections UI: collections
are created on the fly when a file is assigned to a new name, can be
renamed (bulk-updates every tagged file and Qdrant point), and can be
removed (reassigns tagged files back to Uncategorized rather than
deleting anything).
* fix(rag): use dynamic collections query in chat search filter
chat/index.tsx still imported the static KB_COLLECTIONS constant for its
'Search in' dropdown, inconsistent with KnowledgeBaseModal.tsx which already
uses the live getKnowledgeCollections() query. Renamed/added collections
via the new Manage Collections UI weren't reflected in the chat filter.
* feat(rag): broaden preset tags and add creatable collection combobox
Replaces the survival-specific preset list with general-purpose starter
tags (recipes, diy, health, technology, finance, travel, hobbies,
reference, survival, energy) so the Knowledge Base reads well for
home-lab/reference use, not just prepping.
Adds sanitizeCollectionName() (trim, lowercase, length cap) applied on
every write path server-side, and a dependency-free CollectionCombobox
component replacing the plain <select> + window.prompt pattern for
tagging — autocompletes against presets + tags already in use, with a
'+ Create' option for anything new.
* chore(rag): remove .docx fix from this branch, split into #1100
Per review feedback, the .docx extraction fix is unrelated to the
collections feature and can merge independently. Moved to a standalone
PR (Crosstalk-Solutions/project-nomad#1100) off dev.
* chore: remove unrelated diff noise (lockfile, comments, indentation)
---------
Co-authored-by: John Cortright <jcortright@zscaler.com>
Score v2 Phase 1. Safe under v1 — no scoring, weight, reference, or
submission-payload changes; only the benchmark's failure behavior and
forensic metadata.
- Fail-on-parse-miss (W3): the four SCORED sysbench metrics (CPU
events/sec, memory ops/sec, disk read/write MiB/s) now THROW when the
regex misses or the value is <= 0, instead of silently returning 0.
A parse failure (e.g. an upstream image output-format change) now
fails the run with a clear error via the existing _runBenchmark
try/catch, rather than submitting a phantom zero sub-score. Secondary/
informational fields keep their existing defaults.
- Pin sysbench by digest (W3): severalnines/sysbench@sha256:64cd003b...
(was :latest), so a latest-tag format change can't break the parsers
fleet-wide. Digest validated on the NOMAD6 reference build.
- Record provenance (W7): sysbench_digest + ollama_version (from Ollama
/api/version, null-tolerant) stored on each result. New nullable
columns + additive migration.
Verified: pinned digest pulls/runs/parses on NOMAD3; migration applies
(columns present); typecheck clean.
Part of the NOMAD Score v2 effort.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Meshtastic Daemon was pulled from DEFAULT_SERVICES because it can't work
without hands-on setup (radio MAC address, etc.). The seeder never deletes,
so every early-access deployment keeps an orphaned nomad_meshtasticd row and
still shows the broken card.
Migration mirrors the legacy-Kolibri sunset: drop the row where installed=0,
flag is_deprecated where installed=1 (keeps it manageable, hides from catalog).
Runs automatically on each box's next update.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: replace legacy Kolibri image default with latest v19 image
* feat(supply-depot): add content migration instructions for Edu Platform Gen 1 to 2
Adds the MeshCore web client to the Supply Depot catalog (host port 8500),
alongside the existing Meshtastic apps. Uses aXistem's prebuilt image of Liam
Cottle's MeshCore client (MeshCore is a sibling LoRa mesh project to Meshtastic).
The image is stock nginx serving a static Flutter build over HTTP, but the
client reaches radios via Web Bluetooth / Web Serial, which browsers only allow
from a secure (HTTPS) context. So we serve it over HTTPS: a new preinstall hook
generates a self-signed cert + a small SSL nginx config into storage/meshcore-web,
both bind-mounted into the container (the config over the image's default.conf),
publishing 443. Same one-time browser-warning approach as Vaultwarden, whose
openssl cert generation is refactored into a shared _ensureSelfSignedCert helper.
Also adds a NOMAD-specific docs section + Manage>Docs anchor, and registers the
IconAntenna icon. Meshtastic Web left unchanged.
Validated on NOMAD3 (v1.33.0-rc.1): the image + SSL config + self-signed cert
serves the MeshCore Flutter app over HTTPS 200 with working SPA fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Let users override an app's "Open" link with a reverse-proxy or
local-DNS address (e.g. https://jellyfin.myhomelab.net). Falls back to
the default host+port when unset. Metadata-only — no container changes.
Add NOMAD-specific getting-started docs for all 9 curated Supply Depot apps,
the catalog/install fixes each one surfaced, and a way to reach the docs from
each app card.
Docs:
- New in-app Markdoc page admin/docs/supply-depot-apps.md covering all 9 apps
(Stirling PDF, File Browser, Calibre-Web, IT Tools, Excalidraw, Homebox,
Vaultwarden, Jellyfin, Meshtastic Web): first run/login, where data lives,
and offline behaviour. Registered in docs_service DOC_ORDER.
- Manage > Docs dropdown item linking each app to its section
(/docs/supply-depot-apps#<anchor>): anchor map in constants/supply_depot_docs.ts,
heading anchors via Markdoc {% #id %}, and hash-scroll on the docs page.
Install / catalog fixes:
- Stirling PDF: open straight to the tools (SECURITY_ENABLELOGIN=false; the old
v1 DOCKER_ENABLE_SECURITY flag was dead).
- File Browser: seed a known admin/nomad login (bcrypt) instead of a random
log-only password; scope visibility to content folders via mount selection and
move the DB out of the browsable root.
- Calibre-Web: bundle an empty Calibre library and seed it on install so setup
doesn't dead-end at db config (_runPreinstallActions__CalibreWeb).
- Homebox: swap the archived hay-kot image for the maintained sysadminsmedia fork.
- Vaultwarden: generate a self-signed cert on install and serve HTTPS by default
(_runPreinstallActions__Vaultwarden + ROCKET_TLS + ui_location https:8480), so
the web vault has the secure context it requires.
- Jellyfin: pre-create storage/media/{Movies,TV Shows,Music,Photos} so each
library points at its own subfolder, avoiding the overlapping-path issue that
silently hides content (_runPreinstallActions__Jellyfin).
- Seeder run() now also syncs ui_location for non-modified curated services, so a
catalog link/scheme/port change reaches existing installs on update.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch kb_ratio_registry.chunks_per_mb from DECIMAL(10,2) to UNSIGNED
INTEGER so the value mysql2 returns matches the `number` type declared
on the model. DECIMAL columns deserialize as strings by default, which
would break `=== 0` checks for video-only ZIMs and silently coerce
through arithmetic in Phase 2 consumers.
All seeds are whole numbers and the heuristic's real-world variance
(~±50%) makes sub-integer precision meaningless.
Foundation for the cost estimates and partial-stall detection that
Phase 2 will surface. No consumers yet — this PR just lays the table,
the seed rows, and the lookup helper so subsequent UI work has
estimates available without a per-ZIM benchmark.
## What lands
- New table `kb_ratio_registry` (pattern, chunks_per_mb, sample_count,
notes). Migration creates and seeds heuristic defaults from the RFC
appendix: devdocs (1100/MB), Wikipedia variants (270/MB), iFixit
(50/MB), Stack Exchange Q&A (200/MB), video-only ZIMs (0), plus a
catch-all fallback at 100/MB.
- `KbRatioRegistry` model with static `lookup()` and `estimateChunks()`.
- Pure helper `kb_ratio_lookup.ts` doing longest-prefix-match — a
specific entry (`wikipedia_en_simple_`) overrides a broader one
(`wikipedia_en_`). 9 unit tests covering the lookup boundary.
- `sample_count` starts at 0 (heuristic seed) and is reserved for
Phase 4 self-calibration to increment as observed ZIMs update each row.
## Not in scope
- Self-calibration on successful ingestion (Phase 4)
- UI consumers — Warning B (partial-embed stall) and the storage budget
meter / time estimates land in Phase 2.
## Tested
- Type-check clean
- 9 unit tests pass for `findChunksPerMb` and `estimateChunkCount`
- Migration applied on NOMAD3 via hot-patch; 9 seed rows verified in DB
Adds a persistent state machine for AI knowledge-base ingestion so the
scanner can distinguish "fully indexed", "user opted out", "failed", and
"stalled" from each other — none of which were derivable from the prior
binary "any chunks in Qdrant ⇒ embedded" check.
## What lands
- New table `kb_ingest_state` keyed by `file_path` with enum state column
(`pending_decision | indexed | browse_only | failed | stalled`).
Independent of `installed_resources` so it covers both curated downloads
and manually-uploaded KB files.
- New KV key `rag.defaultIngestPolicy` (string: `Always | Manual`).
Registered now but not consumed yet — JIT prompt + wizard step land in
Phase 3 of the RFC.
- `EmbedFileJob.handle` writes state on terminal outcomes:
- Success (final batch) → `indexed` + chunks count
- `UnrecoverableError` → `failed` + error message
- Retryable errors are left to BullMQ's existing retry path
- `scanAndSyncStorage` swaps the binary qdrant check for a state-aware
decision tree (see `decideScanAction`). Existing installs auto-backfill
on first scan: files with chunks in Qdrant but no state row become
`indexed`; new files start as `pending_decision`.
- `deleteFileBySource` drops the state row last, so removed files
disappear entirely instead of leaving an orphan that the next scan
would re-dispatch into nothing.
## What does NOT land here
- Ratio registry (separate PR) — needed for partial-stall detection and
cost estimates, but a separable concern.
- #880 follow-up initial-progress anchor (separate tiny PR).
- Phase 2 UI (status pill, per-card actions, conditional warnings).
- Phase 3 policy surfaces (wizard step, JIT prompt, guardrail modal).
- PR #886's bulk-action hookup — `_deletePointsBySource` / Re-embed All
/ Reset & Rebuild would also want to set state, but #886 isn't merged
yet; that wiring goes in a follow-up once #886 lands.
## Target
This is forward work for v1.40.0 (RFC #883). Branching off `rc` because
that's the current latest base and post-GA Jake will sync rc→dev; a
retarget at PR-open time is a fast-forward if requested.
## Tests
- 9 new unit tests for `decideScanAction` covering all five states plus
the no-row / chunks-present / chunks-missing combinations
- Type-check clean
- Smoke-tested end-to-end on NOMAD3 via hot-patch:
- Backfill: 5 ZIMs + 2 KB uploads with existing chunks in Qdrant all
came back `indexed` on first scan
- Pending dispatch: a video-only ZIM with no chunks (`lrnselfreliance`)
came back `pending_decision` and was correctly re-dispatched (Bull
deduped to its historical `:completed` jobId — bgauger's #886 fix
drains that)
- Delete hook: deleting a KB upload via `DELETE /api/rag/files`
removed both the disk file and the state row
Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
* feat(content): add custom ZIM library sources with pre-seeded mirrors
Users reported slow download speeds from the default Kiwix CDN. This adds
the ability to browse and download ZIM files from alternative Kiwix mirrors
or self-hosted repositories, all through the GUI.
- Add "Custom Libraries" button next to "Browse the Kiwix Library"
- Source dropdown to switch between Default (Kiwix) and custom libraries
- Browsable directory structure with breadcrumb navigation
- 5 pre-seeded official Kiwix mirrors (US, DE, DK, UK, Global CDN)
- Built-in mirrors protected from deletion
- Downloads use existing pipeline (progress, cancel, Kiwix restart)
- Source selection persists across page loads via localStorage
- Scrollable directory browser (600px max) with sticky header
- SSRF protection on all custom library URLs
Closes#576
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(content): recognize Wikipedia downloads from mirror sources
When Wikipedia is downloaded via a custom mirror instead of the default
Kiwix server, the completion callback now matches by filename instead
of exact URL. This ensures the Wikipedia selector correctly shows
"Installed" status and triggers old-version cleanup regardless of
which mirror was used.
Also handles the case where no Wikipedia selection exists yet (file
downloaded before visiting the selector), creating the record
automatically.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ZIM): use cheerio for custom mirror directory parsing
* fix(ZIM): use URL constructor for more robust joining
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Jake Turner <jturner@cosmistack.com>
Qdrant's upstream default enables anonymous telemetry to telemetry.qdrant.io,
which doesn't match NOMAD's offline-first "zero telemetry" posture. Adding
QDRANT__TELEMETRY_DISABLED=true to the container environment turns it off for
fresh installs and reinstalls.
Existing installs keep their current telemetry-enabled container until the
Qdrant service is force-reinstalled via the Knowledge Base panel or the next
container recreation — Docker bakes Env into containers at create time, so
env changes require a new container.
Closes#742
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add distance scale bar and user-placed location pins to the offline maps viewer.
- Scale bar (bottom-left) shows distance reference that updates with zoom level
- Click anywhere on map to place a named pin with color selection (6 colors)
- Collapsible "Saved Locations" panel lists all pins with fly-to navigation
- Full dark mode support for popups and panel via CSS overrides
- New `map_markers` table with future-proofed columns for routing (marker_type,
route_id, route_order, notes) to avoid a migration when routes are added later
- CRUD endpoints: GET/POST /api/maps/markers, PATCH/DELETE /api/maps/markers/:id
- VineJS validation on create/update
- MapMarker Lucid model
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removes the InstalledTier model and instead checks presence of files on-the-fly. Avoid broken state by handling on the server-side vs. marking as installed by client-side API call
Content Manager now shows Title and Summary columns from Kiwix metadata
instead of just raw filenames. Metadata is captured when files are
downloaded from Content Explorer and stored in a new zim_file_metadata
table. Existing files without metadata gracefully fall back to showing
the filename.
Changes:
- Add zim_file_metadata table and model for storing title, summary, author
- Update download flow to capture and store metadata from Kiwix library
- Update Content Manager UI to display Title and Summary columns
- Clean up metadata when ZIM files are deleted
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds a standalone Wikipedia selection section that appears prominently in both
the Easy Setup Wizard and Content Explorer. Features include:
- Six Wikipedia package options ranging from Quick Reference (313MB) to Complete
Wikipedia with Full Media (99.6GB)
- Card-based radio selection UI with clear size indicators
- Smart replacement: downloads new package before deleting old one
- Status tracking: shows Installed, Selected, or Downloading badges
- "No Wikipedia" option for users who want to skip or remove Wikipedia
Technical changes:
- New wikipedia_selections database table and model
- New /api/zim/wikipedia and /api/zim/wikipedia/select endpoints
- WikipediaSelector component with consistent styling
- Integration with existing download queue system
- Callback updates status to 'installed' on successful download
- Wikipedia removed from tiered category system to avoid duplication
UI improvements:
- Added section dividers and icons (AI Models, Wikipedia, Additional Content)
- Consistent spacing between major sections in Easy Setup Wizard
- Content Explorer gets matching Wikipedia section with submit button
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The migration file used a 10-digit timestamp (1769324448) while all other
migrations use 13-digit timestamps. When sorted numerically, this caused
the builder_tag ALTER TABLE migration to run before the benchmark_results
CREATE TABLE migration, breaking fresh installs.
Renamed: 1769324448 -> 1769324448000 (append 000 to match 13-digit format)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
